That comment presumes a scenario involving multiple intelligent entities and a kind of natural selection between them. That is not the only possible scenario – see:

http://alife.co.uk/essays/one_big_organism/
http://alife.co.uk/essays/self_directed_evolution/

The theory isn’t complete yet, but I think that decent progress has been made, especially considering their shoe-string budget. I agree that we need to substantially increase our investments in Friendliness research. SIAI is trying.

“I don’t think we can even properly define what a “solution” to the problem would even look like.”

Selecting the actual goals themselves is a different matter, but I’d personally define it as achieving a goal-system stable through recursive self-improvement. The self-improvements would allow for continually greater understanding of the assigned goals, in principle.

“Post singularity all bets are off and most “rules” go out the window.”

But what if the goal-system is stable, and the AGI lacks any motivations to throw-out its “rules”?

“Maybe it is ok to bring about the end of humanity, so long as we move on to something “better”, what ever that means.”

But I don’t think that’s garaunteed by any means. The dynamics of an arbitrary design could easily lead the AGI to ceaselessly pursue some ridiculous and trivial target(s). I don’t want to trade humanity’s future potential in exchange for some optimal paperclips.

“I think these are philosophical problems that are not going to be solved anytime soon, in particular, they are not going to be solved before machine super intelligence.”

But we shouldn’t give-up prematurely. Breakthroughs can happen. And we can’t really be certain of when the Singularity will happen. For example, if I had to rough-guess it, I’d say that Friendliness could be solved within 5 years. Also, keep in mind that as the AGIs advance from animal-level to human-level, our understanding of Friendliness is likely to skyrocket with them – it’s tech improving tech.

“Secondly, even if we knew what the goal was, I suspect that there are deep mathematical problems in this area that make many current ideas on Friendly AI practically impossible to achieve.”

SIAI/Yudkowsky reinvents Friendly AI theory fairly regularly, you probably shouldn’t base your assessment on anything written more than 1 or 2 years ago. And who knows, the critical insight may happen next year. The important thing is to keep trying.

I don’t really understand the motivation for this post. Even if Friendliness wasn’t 100% provable, *trying* to make the first AGI Friendly sure as hell beats slapping together a minimal, arbitrary design and having *zero* intuition about whether it will be nice to humanity. If AGI designers intend to flip the switch on a minimal, arbitrary design without even an *attempt* to make it Friendly, then humanity and its future potential are probably screwed. Such an AGI would almost surely be totally indifferent to the well-being of humanity – the motivation-space is too large. Under such conditions wouldn’t humanity most wisely never even *attempt* to make such a minimal, arbitrary AGI. Making a Friendliness attempt is infinitely better than making no attempt. Is publishing the comment: “Friendly AI is Bunk”, really helping the cause?

There are 27 ontological ‘categories of cognition’ which are ‘universal in scope’ (applicable every-where in reality where minds could exist).

A universal reasoner (real AGI) would need class definitions for all 27 categories to work. The 27 categories are listed at the end of the essay.

Cheers!

Yes, if we really understood consciousness, I suspect that many ethical issues would become significantly easier to understand and reason about.

Proving friendliness may well be impossible, but surely given the stakes it is worth trying,

Sure, it’s worth trying. But man, as far as I can see we really don’t have a handle on this problem at the moment.

That’s the somewhat more polite version of my main message…

Why do you suspect this problem impossible? Certainly it hasn’t been solved yet.

Why do you suspect it is possible?

Humans have been arguing about what things like friendliness mean for thousands of years. Countless generations of people have thought about these things going way back to the Greeks, and who knows who before that. If you read what the Greeks had to say about some of these things, as I guess you will have, then you’ll see that they were pretty sharp and were thinking really hard about these issues. They were also thinking hard about physics, mathematics and many other things. Now, how far as our understanding of things like physics and mathematics come since then? It’s developed enormously. But ethics? Hmmm. Not much. Even basic problems in ethics continue to present huge problems after thousands of years of work, including by some of the most brilliant minds in history…. and you’re claiming that it will be possible to not just find good solutions to issues like what friendliness is, but to even do it using formal mathematical equations!

Hehe… good luck Nick!

*thumbs up*

Of course that doesn’t prove that it’s impossible, but it sure makes me suspect that it is. At the very least I’m really doubt that you will solve the problem before AGI turns up.

Besides, even if you did come up with an equation, I’m sure that I could find some strange situations, stick them into your equation, and get answers out that many people disagree with… for the simple fact that on many ethical issues the population is divided.

What do you mean by “primarily interested in its own self preservation”, and what are the “evolutionary pressures” working against systems which are interested in self preservation for derived reasons?

A friendly AI can only preserve itself through friendly actions, while a primarily self interested one can preserve itself in what ever way is the most effective. This gives it a survival advantage due to its very nature.

It’s more about transferring the ability to reason about morality, about what humans really want.

Go and ask various people in countries around the world is moral and what they want. You’ll get a real mixed bag of answers. The world is full of groups of people who hate each other.

This is one critical problem with creating AI which obey their owner’s direct commands. Even the most altruistic and rational human may be corrupted by such power. However, not all AI designs are structured to obey their programmers blindly.

So you can’t rely on the people telling the AI what is right and what is wrong. Which means that you need to get the AI’s internal model of right and wrong correct to start with, and to do that it seems to me that you’d need an extremely precise mathematical definition of friendliness… and as I’ve said above, I’m very skeptical about you managing to do that.

You have to design AGIs nice, you can’t reliably detect it from their behaviour.

Same comment as the last one.

One problem here is the AGI doesn’t need to predict all sequences under a given complexity, only the actual sequences the universe will produce. Note also that a Friendly AI doesn’t have to be omnipotent, it doesn’t have to have 100% success probability.

I’m not asking for omnipotence of the AGI. I’m asking it to solve a finite, but very difficult, problem. Even if it can solve it, you can’t prove that it will do so in order to save the world… and thus you can’t prove its friendliness. Note that I haven’t said anything about the AGI other than the fact that it’s very powerful. It might be friendly, it might not be. Mathematical proof just can’t answer that question for you.

It think I’m going to have to write a longer explanation of this last point. Maybe as a new blog post.

By the way, thanks for the well thought out reply.

Thought about this for 4 years. Finally cracked it. The condition for Friendly AI is given below. This sounds simple but I’m saying something extremely subtle and extremely strange here so you may need to re-read it carefully a number of times:

—
‘The condition is to create a *Sentient* AI – i.e an AI that thinks it has Qualia… such that all mathematical entities representing metaphysical (or ontological) categories of *universal scope* are *experienced directly* – as ‘mathematical qualia’ in the mind of the AI. Any AI is Friendly *if and only if* this condition is met’
—

For a summary of my reasoning see my post on the everything-list here:
http://groups.google.com/group/everything-list/browse_thread/thread/6c0b178a63118d8f/c86de6243cdc81f4#c86de6243cdc81f4

Cheers!