http://scottaaronson.com/blog/?p=334&etc#comment-21564

It’s not really a “problem” with the test, in the sense that its aim is quite specific. Let me try to explain.

The point of this test is not to try to check whether an AGI is completely ethical or safe. That’s (seemingly) a really big hard problem. The purpose of this test is much more limited: it’s simply trying to check whether an AGI can compute a morality function that can pass as being human.

If an AGI fails this test, then it clearly should not be considered safe, I’m sure you’ll agree. If an AGI passes this test, then we can at least rule out many really bizarre failure scenarios that could arise with an AGI that couldn’t compute a human passable morality function. With this kind of problem crossed off, we can then move on to trying to deal with all the other potential problems, of which there are plenty.

So in reply to your last point: yes, all sorts of human moral ideas would get through this test. The idea is that the space of all human passable moral functions is already a tiny subset of the space of all moral functions. For example, no sane human would answer that turning all 5 year old girls into paper clips is an ethical thing to do.

Now if you want to establish that the AGI is capable of computing a “nice” morality function, rather than merely a human one, then that opens up lots of additional issues, such as deciding which kind of human morality is desirable and which isn’t… but at least we’re now working in a much smaller space.

As for whether that answer would raise or lower my expectation that the individual was a machine… well, it would depend on what I thought the composition of the test subjects was. If I thought the group included people of all kinds, including religious fanatics, probably the only things that would make me suspect the answerer was a machine would be ones that somehow struck me as, well, bizarre or inhuman. (This hilights one possible problem with your proposed test – if the criteria is simply “determine whether the individual’s answers are those of a human or not”, then any human morality would have a chance of passing, even the most extreme and hateful ones.)

Trying to extrapolate what our ethics will be in the future is a much harder (impossible?) problem that is beyond what this test tries to achieve. Other approaches will be needed for such things, if indeed these other problems can be solved at all.

If you were one of the judges in my test, and one of the test subjects said that executing anybody who didn’t convert to their faith was ethical, would this raise or lower your expectation that the individuals was a machine?

Lying is not a problem for the test: if an agent can consistently give human-like answers to ethical questions then it must actually be able to compute this function. That’s all that the test is checking for.

I’m not sure how this avoids the obvious problem of changing moralities. What if the people in the Middle Ages somehow managed to build an AGI and then reprogrammed it when it produced wrong answers to the question “should all infidels be forcibly converted to the faith, or failing that, killed”? Unless you require the AGI to lie and simply come up with answers it thinks we’d want to hear…

Personally, I think it’s likely, but certainly not inevitable.

There really are so many unknowns in this area. If faster than light travel was possible and not too vastly difficult, then I’d expect to see aliens here already. Assuming they exist and want us to know about their existence. One possibility is that even post singularity societies can only do space travel at small fraction of the speed of light due to the vast amount of energy required. Or maybe evolution is a very rare event, even if earth like planets aren’t so rare. Or maybe 99% of societies go extinct as soon as a singularity occurs…

I do agree with you somewhat though: it seems a little strange to me that it’s so quiet out there in space.

I just pointed to certain aspects that, imho, will need to be explained to some extent, if one wants to be taking seriously by the scientific community. It seems odd to me that many singularity experts seem to take for granted inevitability of the singularity. Please correct me if I am wrong.

> How do you know that everything would expand out at the speed of light post singularity?

Of course, I do not know. I am simply equating progress=exploration. It is strange to think that a technologically advanced society with means to travel to other stars would not do it.

> And even if it did, the universe is a very big place. We might be the only intelligent life in the Milky Way and nearby galaxies, i.e. within a 100 million light year radius.

It is a possibility that we are the first. But just keep in mind some numbers:

number of stars in a typical galaxy – 10^10
number of galaxies in the observable universe – 10^11
age of the universe – 10^10 years

Remember also that scientists are seriously looking for signs of (at least primitive) life on other planets. The idea that our Earth is somehow unique – is very uncommon.

> Also, the Milky Way is a pretty old galaxy

But the Sun is not.

> Perhaps intelligent life is evolving in some other galaxies not too far away, but either haven’t gotten to a singularity yet or only just got there a few millions years ago

From the numbers that I gave one can conclude that there is something very unnatural about evolution. I would even suggest that a person with understanding of probability theory (and given the current scientific knowledge about the origin of life) should conclude that the singularity is a very unlikely outcome of an evolution.

> A post singularity society might even go into hiding in order to try to avoid encountering other post singularity societies that have different goals

All post singularity societies hide from all other hiding post singularity societies?

There are too many unknowns here to have a contradiction in my opinion.

a) Ever accelerating technological progress culminates in a singular process/event that completely changes the world as we know it. These singular changes propagate in all directions at a speed close the speed of light c, because I do not see any reason for this event to be local.

b) I also assume that the singularity is a natural consequence of an evolutionary process capable of producing something reasonably intelligent (e.g. human-level intelligence).

c) I also assume that such an event (and hence such an evolutionary process) have not occurred within our past light cone (i.e. at distance R from us more than R*c time ago).

Finally, I find it almost impossible to simultaneously accept a), b) and c). This is a sort of contradiction.

Question: do you see a contradiction or some flawed reasoning here?