Normal Human Heroes

Recently on Ben Goertzel’s blog, he mentioned that he’s had the experience of FAI-minded persons telling him that if he got too close to AGI he may have to be killed, and then discussing how this may occur.  Clearly, these were not advanced rationalists. I know this blog has a small audience and such ideas are probably very rare, but given the style of my material, I feel like making a statement. Given my background in gaming,  science-fiction, and fantasy, I find martial imagery and allegory inspiring. As someone concerned about existential risk, I find the idea of martial action irresponsibly foolish.

If the world gets to a point where it’s likely that unfriendly AI is about to be released, it’s incredibly unlikely that there will be only one group ready to do so. It is also likely there will be many more AGI teams than there are today. Trying to stop them through attrition would be ridiculous. Even in such a situation, more widespread action would be possible through the (non-lethal!) actions of governments convinced to step in, or research coalitions convinced to step back, by the efforts of organizations like SIAI, FHI, etc. This may be unlikely and the effect small, but it is far more likely and helpful than having any kind of meaningful effect through attempted assassination. If AGI researchers were killed, significant blame will inevitably land on groups like SIAI. To say that this will hurt their credibility is an understatement.

This is likewise true with mentioning the idea of assassination to AGI researchers. The only way such threats could ever be prohibitive would be if they were actually thought to be legitimate, and those expectations can only exist in a world in which groups like SIAI have had their intellectual and academic reputations destroyed. Not to say that destroying such groups’ reputations also makes those threats credible; death threats do far more to tear apart reputations than they do to cause fear.

There is too much at stake here to be stupid about this. It is not too hard for our evolved intuitions to suggest that we solve problems by eliminating “opponents”. But then, it’s not too hard for our evolved intuitions to screw us over generally. A second Unibomber would get to feel like a hero and a soldier, and everyone else would get to pay for it.

As I said in my first post on this blog, our enemies are human error and human hatred, not human beings.

April 6th, 2010
There are currently about 6,800,000,000 people in the world. There were up to 1,800,000 people at Barack Obama’s inauguration. That’s a pretty big number. The picture below probably contains over half of them.
Take a look. (You can click on the picture for a larger version.) Assuming it contained everyone who attended the event, and assuming population growth utterly and suddenly halted, you’d be looking at about 0.026% of the people who would be affected by an existential disaster.

January 26th, 2010

This is a really great film, one of my favorites. It’s based loosely on the German classic, using a comic by the creator of Astroboy (after he died and couldn’t stop them), and written by the creator of Akira. The basic plotline is that of two detectives foreign to the city, investigating a rogue scientist. They meet up with his creation as political turmoil erupts in the city. It’s generally fun to watch throughout and the ending is fantastic. In particular relevance to this blog, it displays both the folly of anthropomorphizing AIs, and the existential disaster they can cause. It’s not like a lesswronger wrote it, but in comparison to most such films it’s excellent.

For my senior honor’s thesis I’m researching the implicit assumptions underlying the use of generally misguided statistics in genetic studies. Some time ago I read through the recent book ”How Doctor’s Think”. It didn’t turn out to be very useful for my project, but it was written well and interesting. There’s a few patient stories, one detailing a misdiagnosis of an adopted Vietnamese infant thought to have an immune deficiency. A section at the end of it bothered me:
“Shira received her morning feeding through the tube, and then Rachel went to the end of the hospital corridor to a pay phone. She called one of her closest friends from her congregation and told her the news.
“It’s so wonderful,” her friend exclaimed. But then there was a long silence.
Rachel wondered what was wrong.
“Turn on your TV.”
Rachel stood frozen in the room and felt as if her heart, so full of joy, were being torn. At the moment she celebrated Shira’s restored life, thousands were likely dead in the attack on the World Trade Center. How can I rejoice when God’s creatures are dying?
Forty-five days after Rachel and Shira went to the Children’s Hospital ER, mother and daughter left for home. It was Friday, just hours before the onset of the Sabbath. When Rachel turned the key and entered her apartment in Brookline, she could smell the meal left by friends. Two candles stood ready to be lit, two fresh challahs ready to be savored. Rachel held Shira after lighting the candles. The soft glow of the flames played off her daughter’s face. It was the day of rest and of peace, the day when all woes were meant to cease, the day that Rachel had not truly had for more than six weeks.
At each step, Rachel had not been sure whether she would find the strength she needed to endure, and the courage to question. Silently, she again thanked God for creating all human beings with such remarkable reservoirs of resilience. She thought how the Sabbath was the time when these reservoirs were refilled. She prayed that during this first Sabbath after 9/11 her country would find the strength and courage to defend itself and to care, with a full heart, for the families who had lost loved ones.”
As most readers know, not that many people really died in September 11th. A little under 3,000, which is the world death toll from all causes every half hour (h/t to Vladimir for the correction) .So don’t just sit there caring about some tragedy that’s already occurred, work to stop the tragedies that are occurring constantly all over our planet, and the extremely large disasters that have a very good chance of happening, in this century. An existential disaster would be equivalent to more than two million 9/11 events, in terms of human death. If you feel sorrow for those we have lost, use that to save all those we will lose.
In my research on existential disasters in general, I also read “Never Saw It Coming.” This book was one of the most foolish books I’ve ever read. Karen Cerulo takes the observation that there is a cultural asymmetry in focusing more on the good than the bad, and then applies this perspective far more widely than appropriate. She often ignores contradictory evidence and any concern for the actual objectives at hand; I get the impression she had an alright idea and just really wanted it to be a great idea. It’s a kind of comic example of confirmation bias. There are a few tidbits and points that are worthwhile, but almost nothing applicable to existential risk.
There was a good quotation in there though, which relates to the tendency of people to apply a lot more effort to mourning than to saving lives.
“People who are in decision-making positions are not mentally preconditioned to think in terms of what happened. So that’s what I mean about a failure of imagination. The evidence comes in, but your mental reactions are not geared to thinking in these kinds of terms. When a guy calls from a flight school and says they could take a 747 with fuel and plow it into a building and that’s a bomb, you hear it but you say, “Ah, that’s a wacko idea.” You don’t say, “Holy Jesus, that’s what we’ve got to worry about.”…[You] fail to imagine what the danger is. [You] fail to understand the world we live in and the nature of the enemy.”   -Journalist Hedrick Smith

November 21st, 2009
(This review contains only negligible spoilers).

I just finished reading Accelerando by Charles Stross. It’s a masterpiece of hard science fiction, and I highly recommend it.
Vernor Vinge coined the term Singularity when he observed that no author could write realistically about smarter-than-human intelligences. Literature allows you to realize characters who are stronger or more outgoing, but the intelligence of their plans is limited by your ability to think of them. Superintelligences must be kept off screen or in some infant stage. Considering the likelihood of superhuman intelligence in our relatively near future, this makes writing really hard sci-fi a difficult endeavor. Vinge blurbs on the back cover of the book “There is an intrinsic unknowability about the technological singularity. Most writers leave it safely offstage or invent reasons why it doesn’t happen”, which applies to his own work as well. In comparison Stross’s Accelerando dives in headfirst, with mainstream (post)human civilization becoming essentially incomprehensible by the middle of the book.
Of course any real superintelligences still have to be kept off screen, and so the story follows characters who for one reason or another have been left behind by the tsunami of increasing intelligence. This creates the interesting effect that despite inhabitating lives stranger (and more probable) than those in the vast majority of sci-fi, the characters’s situations manage to feel very backwater. It’s as if you were following a family of Amish throughout the industrial and information revolutions, but more significant.
At a few points I grew dissapointed as it seemed Stross’s literary ambitions may have overcame him, with superintelligences mysteriously on our level, but by the end it all makes fair sense through one route or the other. I would have enjoyed those sections more if I knew that to begin with. There were also a small handful of differences between my own best estimates and the world of Accelerando. The Singularity is a tad on the slow side, and there always remain a number of independent, comparably powerful entities. This is probably my largest contention (it seems more likely for a single superintelligence to achieve dominance at some point), but there might not be much of a story if this were otherwise. There’s also no significant mention of emotional engineering, ala David Pearce’s Hedonistic Imperative or Nick Bostrom’s Letter From Utopia. A more sociological than technological point, but people are pretty nonchalant about creating and deleting copies of themselves. I care more about expectation of experience than identity, but as a preferential utilitarian I can get along just fine with those who think otherwise, as long as they don’t force that choice on others.
When I first heard about this book, the take-away message was the great density of concepts. The book is packed with advanced technological proposals, internet subculture references, unusual vocabulary, economics, neuroscience, history…it goes on. However Accelerando is much more readable than this would suggest, and most of the references are tangential, perfect understanding not required. The few times a concept is critical he takes a moment to explain it, and those interested can hunt down referenes on the net (try doing that 10 or 15 years ago). I’m admittedly not bleeding-edge (cutting maybe?) on speculative technology, but to the limits of my knowledge all the ideas are presented in a sober, best-guess fashion. To load the book with so many ideas takes quite an intellect or a great deal of work, and most likely both. Stross took 5 years to write this and has an impressive background, with degrees in pharmacy and computer science, and those who’ve known (biblical sense) WoW might be interested to learn he came up with death knights, back in the day.
The best and favorite thing I can say about this book is that it is mind boggling. A common criticism of sci-fi is that it takes one idea and places it in an otherwise changed world. Accelerando is just the opposite, which includes just about every feasible proposal and then mixes them in with additional ideas about their interaction. This allows for very unique and interesting turns of plot, and the book tends to put the reader in a constant state of future shock, continuing for 400 pages, even while in the relative backwater I mentioned above. The density of information and references adds to this effect nicely. The human mind suffers from the conjunction fallacy, and we’re more likely to put belief in speculation that is more specific, such as the setting of a book. Despite this I think Accelerando is excellent for improving our sense of the future, by reminding us that the future isn’t going to be one or a few new ideas, it’s going to be a great number of them, all interacting and creating ever newer ones. There are three meanings to the Singularity, one of which is that without intelligence enhancement, really understanding the world is something you’d have to entrust to others. It’s one thing to read about that kind of future but another to catch a glimpse of it, and that’s something that Accelerando provides.
Vinge also calls Accelerando “the most sustained and unflinching look into radical optimism I’ve seen”. While our own future could be much better, this really is a pretty optimistic book, which I like as I’m generally an optimist myself. It also presents some very possible dangers and threats. The future could be better than we’re physically capable of imagining, but there are thousands of ways it could go badly and it’d be worth it (understatement) to prevent those outcomes. The future may be incomprehensible, but for now we’re still in control, and still the most intelligent life on this planet. Let’s make the most of that, because it’s not going to last.

November 3rd, 2009
While I’m confident that most of my readership also follows Michael Anissimov’s Accelerating Future blog, his posted video of a panel from the Singularity 2009 conference is so relevant to the topics of Normal Human Heroes that it would be criminal not to include it here. Really great discussion from some of the de facto leaders of the most critical and under appreciated fields.

Changing the World Panel — Singularity Summit 2009 — Peter Thiel, Eliezer Yudkowsky, Aubrey de Grey from Singularity Institute on Vimeo.

August 15th, 2009
(Yesterday I posted about history implying a strong heuristic that radical claims about the future shouldn’t be believed. This heurstic is valid, and has to be worked against to lead a person to set it aside and actually look at the evidence on a particular issue.)
If you’re a person with a radical claim about the future to make, you’ve got serious work cut out for you. If the heuristic suggests that claims about the end of the world and massive global change are not worth considering and are something put forth by irrational and unconventional weirdoes, then hopefully you can lead people to set aside the heuristic by displaying rationality and being as conventional as possible. It helps if instead of being a smelly person who stands on street corners all day, you’re the head of an Oxford institute, an MIT graduate with at least a half dozen successful companies, or a computer pioneer and one of the founders of the industry’s leading companies. It helps at least a little if you did such things first, before you put much weight in the possibility of humanity going extinct by the end of our lifetimes. The long and the short of it is that it helps if by all other accounts you are rational, intelligent, and not terribly strange.  None of that is good reason to unquestioningly believe what you’re saying, but it is a good reason to actually look at the evidence on those claims.
There are two ways to be strange with respect to world-changing future developments. The first and most salient way is to simply assign significant probability to such things. The conventional opinion does not, and those who assigned high probability in the past seldom even seem justified in retrospect. The second way is: given that you assign a non-negligible probability, what are you doing about it? Here both conventional opinion and a rational analysis might agree that we should be doing quite a bit. That is, if you seriously assign significant probability, we should ”be serious about” the possibility.
I think this is generally conflated with the oddity of assigning the non-negligible probability in the first place, so that doing a great deal about these claims makes the claim itself seem less believable. Maybe it’s because it draws more attention to the belief, or because it makes a person more nervous to have someone actually acting on such ideas, or because it makes the association with prior doomsayers stronger and calls the heuristic back into play. So then to try and convince others of the possibilities we just mention them as an interesting bit of arm-chair speculation, put them forward as a strictly academic discussion, avoid expressing great emotion, and don’t factor them into decisions on whether to spend a day watching a LotR marathon. Then our friends can say “Sally sure has some weird ideas about the future, but at least she’s cool about them. At least she doesn’t get too worked up about the possible death of 7 billion people.”
I’m not suggesting we go the route of Sarah Conner, for a variety of reasons (BTW I’ve only seen the films). But what I do suggest is that if we rationally think there is a significant chance of existential disaster or a massive global development that we’ll regret not preparing for, then we rationally out to do something significant about it (I’ll leave utilitarian calculus to a later post). Maybe that actually means we should apply limited effort in order to bypass a common heuristic about credibility, but that’s something we should come to through rational decision-making, not assumption.

August 14th, 2009
(This post applies to a number of transhumanist concerns, but like much of my writing there is some focus on Friendly AI as I currently expect this to be our most critical challenge)
If you’re a “normal” activist, you have it relatively easy. If you say you’re working to get clean water to millions of people and somebody says “I don’t think this problem is real”, you can show them the pictures, the videos, the studies, the census data, and even go there and talk to those people. You can test the water and see that yes, it’s dirty, and you can do medical examinations of the people and say yes, they are suffering.
If you’re a transhumanist activist and you say you’re working to ensure equitable distribution of intelligence amplification technology, or to prevent us all from having our atoms realigned for a superintelligence’s computing, the most you have to show is some intelligent analysis, some semi-log plots of technological change, and some recent but distantly related scientific discoveries. That may be enough for rationally assigning a high probability, but you can’t take the person and show them a world ruled by a despotic and unopposable elite of augmented CEOs, or one in which everybody we know is dead.
Working to prevent outcomes that aren’t even possible yet, especially existential disasters and the “end of the world”, I’ve been told more than once that I sound like I’m crazy. And the people who have said that are right! Because the fact remains that the group of people who’ve thought the world could end has had quite a few ‘crazies’. Every single person who thought the world was going to end was wrong, at least if we ignore the Multiple Worlds hypothesis. This belief has a track record of 0 successes, 0% accuracy. Which, obviously, is a  very good thing for us.
We don’t have time to look at the evidence of every person who waves their arms around proclaiming our demise, so we use that human forte of heuristics. And history suggests a heuristic that doesn’t give such claims a lot of credence. I’ll admit right now that it’s a good guess that a given person proclaiming the possible end of the world has a few screws loose, especially if it’s by a single extremely powerful entity who kills each and every one of us. If you don’t have hours, days, or weeks to sit down and sift through the evidence, this is a fine guess to make. But it IS just a guess, and one that considers very little of the evidence for a particular claim. Evidence about past claims of apocalypse carries different weight than an in-depth analysis showing that superintelligence will never arise, or that if it does there is no significant chance for it to do anything we really dislike.