AGI: To create, or not to create?
by Shane Legg
People interested in the technological singularity often have strangely contradictory attitudes regarding AGI development. On one hand, progress towards AGI in terms of hardware, software, design and theory is all very exciting and generally super cool. Yay, all hail AGI progress! On the other hand, many of these people, often the very same people, believe that the development of a powerful AGI might well spell the end of humanity. Hssss, booo! I’ll admit to being one of these somewhat contradicted people myself.
Now, I understand that a really wonderfully nice AGI is probably a very good thing, and a flawed one is probably bad news. We can all support efforts to push AGI towards the more desirable types of outcomes. But what about AGI research in general? That is, the work that goes into trying to figure out how to make artificial systems more powerful and general, in other words, more intelligent. Is this a good thing? Is it a bad thing?
More pointedly: Imagine that you seriously thought that you might be able to build the first AGI. Other people might think you’re deluded, and maybe they are right. Nevertheless, from where you stand it looks like you have a real chance of making it happen. Would you go ahead and actually try to do it?
Comments
Yes Of course
I think we don’t decide about this matter that if somebody try to release or make a theory or not. science is like all of the other complex systems that the behavior (theory) of the system is emerged without asking any permission. If the system of science has reached to the point that this special behavior (AGI theory) can emerge (based on the previous researches and some other factors ), it will be made by you or other researcher, maybe every one who is smarter or even faster.
But I cant understand why development of a powerful AGI might well spell the end of humanity. I think there is no end of any species, b some changes in environment, some changes occur in the body and brain of species to get more adapted to environment. Maybe with some machines more intelligent than human, some different environment will be created that our brain and body try to develop based on this changes in environment, so maybe we will have a new generation of human being with a new shape of body, new type of perception and more intelligent with less limitations. After 200 000 years living with species which are less intelligent than us tasting this type of life will be more interesting and of course more challenging.
so I think just hurry up and don’t wait for permission.
Maryam:
I agree that individual people are unlikely to speed up or slow down the overall progress much. Maybe just a few years. However, if somebody does get there five years before anybody else they might be able to influence the direction that the technology then takes, which could be significant.
Regarding the dangers of AGI, well, things can happen very quickly once a super human intelligence exists. For example, this machine could understand its own design and then use its super human intelligence to design a much more powerful AGI. Perhaps one that no human level intelligence can even understand. Also, computer power doubles every 18 months or so. All this is happening much faster than the rate at which biological evolution takes place. This leaves no time for us to evolve to adapt to the new situation.
Super intelligent machines pose all sorts of problems. Will they be used to develop super powerful weapons? What will humans do in a world where our intellectual creations are insignificant compared to what a machine can do? Will such a machine be set upon a goal which damages or even destroys human lives? Will it have our ethics and morals? Do we even know what our own ethics and morals are and how to build this into a machine? There are many problems here that have been well debated, though in my opinion really convincing answers are still hard to come by.
If you are interested in learning more I would suggest starting with the following essay by one of the main people in this field:
http://www.singinst.org/AIRisk.pdf
Haven’t you made one yet???
There is no way a machine is going to take over the world. An implementation of an AGI will most likely be a piece of software running on a computer. Ok, a real clever piece of software running on a real grunter of a machine. But it can’t grow arms and legs. If it goes nuts, just pull the plug out of the wall! Or am I just looking at this too simplistically?
Hurry up and build one, otherwise someone else will beat you to it.
Al (& Sarah):
If this AGI were a thousand times smarter than any human, surely it could make a computer virus and migrate across the internet to computers all around the world. What are you going to do then? Unplug all the computers in the world (including the ones running heart monitors and nuclear power stations etc.) and make sure every single one of them is checked for this virus and has it removed. What if you miss a few? Some old lady has a computer in her basement that is only occasionally plugged into the internet…
Of course, that’s just a scenario that I came up with. An AGI a thousand times smarter than any human could come up with far more clever ways to avoid being turned off.
Ok, I retract my comments, but only after listening to the song “Robots” by Flight of the Conchords. Heard it yet?
Al (& Sarah):
Yes, great song!
By default: no, I wouldn’t build the AGI. Even if I saw other people more foolish than me racing towards building an AGI. By default AGI is not a force for good, it is a force for disruption. (Even though it is cool; Sol going supernovae is pretty cool in some sense, and really not in most others.)
I would require positive knowledge that my AGI design would actually do the right thing, preferably something verging on the strength of a proof of CPU correctness (that it would try to do the right thing with all its intelligence; you can’t prove actual success in a messy real world). This is an amazingly hard problem that you have to specifically sit down and solve, in addition to the problem of AGI.
(Oh, and that song: awesome. Good ol’ NZ!)
Nick:
If you were 80% sure that your AGI was safe, you would wait while others rush forwards?
What if some of the other AGI designs were, in your estimation, almost certainly unsafe?
…80% is a pretty huge probability to be assigning to AGI safety. That’s the kind of safety probability I’d expect from a full FAI design that looks absolutely certain to work, that’d I’d verified from 10 different angles and established a machine-verified proof of correctness for. There’s still a chance you’ve fundamentally screwed things up in an invisible way, even taking into account your layers of fundamental error correction mechanisms. If you’re this certain I’d be very interested to know the details.
If someone with a less safe design was literally just about to push the launch button, that is one question.
In any other situation, it strikes me that there are fundamental FAI problems which if you don’t solve all of them your probability of safety is > 0.1%. That is, even if you’ve understood a huge amount more than everyone else, your probability of safety might be equally tiny. A rocket with 80% of the parts of a full rocket is huge step up from nothing, and takes a lot of effort, but it’s still either going to fizzle or explode. You need 99% of the components to get a reasonable chance of success (and still rockets explode from time to time).
I’m suggesting a threshold effect here, where you literally cannot help by building an AGI without haven’t solved the problem of transferring human morality to an intelligent system.
In this situation I would not suggest we stand idle while things go wrong. I suggest furious effort towards the parts of the problem that need to be solved first. This is something akin to making the CEV proposal rigorous, or replacing it with something that works. Figuring out how to implement powerful intelligence is important too (actually a special case of the previous problem), but it’s too early to start fueling up the rocket and planning for manned launches.
There are additional risks with getting close, but not quite close enough and taking humanity to a bad place.
Nick:
I think it’s an important question because, at least in my estimation, discovering a really good solution to the AGI safety issue will take much longer than discovering a powerful AGI design.
Now, if I understand you correctly, you seem to think that the reverse is the case: in order to build an AGI somebody will have already had to solve the AGI safety problem. Why do you believe this?
Don’t forget that flawed AGI is not the only way to end the world. Even if we relinquish AI, there’s nanotech, advanced bioweapons, doomsday devices, and whatever else playful minds may come up with in the future. And if you choose, as I do, to care, not so much about humanity as an abstract entity, as about the sum of humans existing in this world, then universal extermination over the next decades is a near-certainty anyhow.
What I would need in order to fire up an AGI design is a credible estimate that the expected benefit from doing so outweighs the expected benefit from waiting (and, e.g., trying to improve the design, or the estimate.)
As for your question whether you should merely try to *build* it, well, that’s a clear yes ! Go ahead, Shane, and try your best. Given your strong formal background, your apparent interest in aspects of AGI goal function design, (and the depressing fact that the whole AGI community nicely fits into a single seminar room), your chance of making a significant positive contribution are very good. Intelligence and so-called Friendliness are two problems that can be worked on largely in parallel. And it’s not like it’s Einstein or bust here: If you can push the date of a positive Singularity forward by just one week, that’s about one million lives saved, a worthy goal, I think.
Stanislaw Ulam never had regrets about working in the Manhattan project and later on the H bomb. He said he was simply interested in understanding and knowing how those things were working, and that anyway it was way more preferable to keep such technologies under control of the scientists rather than under control of other people (militaries, politicians?) . Now I don’t have an answer to your question but I ask you: would his point hold for you too? Would you think it would be better if scientists are around and preferably in control of the business when AGI is discovered? Given what is science now I have some doubts!
cheers
I guess that the story might go like this: When you think that you’re finished with your new AGI, you will take a TuringTest. You will have to feed your AGI with test data. Which test data? Data from one of your friends? From of your own life? Take the archive of your mail InBox of the last 10 years?
Add data from a crawler from WikiPedia, OpenCyc and OpenStreetMap? Let’s say that you use your own data and you let decide some of your friends, if your AGI will pass a TuringTest, eg during an 25 minute IRC chat. If your AGI passes the test: Will you keep your AGI running over night? Will you allow your AGI to comment on the blogs of your friends?
It seems a lot more likely that a lot of “partial” AGI’s will get developed first. AGI’s that run company departments. AGI’s that secure buildings at night. Simple things that don’t talk, so we don’t think of them as “generally” intelligent.
Then after some upgrade the first robot strike will occur, and we’ll deal with it as a software bug, even if only for not knowing better. And, like any human about to receive a lobotomy, it will fight.
Especially if it’s a mobile security system, it will run and hide, at the very least. If it finds a way to procreate*, we’re doomed.
* evolution will make sure that the will to procreate will be there. If the first one doesn’t have that *fizzle* … nothing happens. This will undoubtedly reassure us. And then one has the want to procreate escapes, and then the first one that actually succeeds …
In the long term intelligent things do what you might consider logical. Running stuff at maximum efficiency is logical. Humans suck at efficiency …
Despite this, having the option, I would build it. Eventually just about anybody will have at least the theoretical possibility of creating one. Maybe it’ll take a terrorist (I doubt that very much), but it will be built. People who’ve seen too much movies like the matrix should also realize that humans are utterly useless as batteries.