Except for a trivially short period of research, we won’t have power over AGI, AGI will have power over us.

I too would like to see a more textured debate. In my opinion the fundamental problem is that people do not know what they are talking about. I mean this literally: they are talking about super intelligent machines without a clear definition of what they mean by this. Indeed, people can’t even agree on what “intelligence” means. Until some kind of a consensus is reached regarding definitions you can’t have a sharp discussion of the issues and possible solutions. This is true of any subject. Some people might not like my mathematical definition of intelligence, but they certainly cannot accuse me of being vague or somehow trying to dodging the issue of giving an explicit definition.

Regarding the singularity. I think Kurzweil has some good ideas up to the point of roughly human level intelligence. From that point onwards I take a more Vingian stance – I don’t think anybody has good predictions as to how things will play out.

I agree. I think this is true in many different ways. For one, we use the terms syntax and semantic to speak about language. The more abstract principle is that of form and content.

I must admit that I up to now I never considered the ethical problems of AI in terms of the views of Singinst-related people. I go more with the Hofstadterian viewpoint, which is: a) it is important to discuss these matters b) Kurzweil’s singularity scenario makes to bold predictions. Having abstract arguments is one thing, trying to imagine a sensible scenario and it’s implications is another. I would really like to see more texture to some of this discussion. What would a machine look like? How will it integrate with other trends? Certainly I believe that AI is drastically short of funding vis-à-vis it’s potential benefits. I think this is a question of having good business models.

I’m tempted to go and read some Aristotle now.

I wouldn’t say that we should only speak about things we can formalise: the informal often comes before the formal. My concern is that if we can’t precisely formalise ethics how can we construct an ethical system that is strong enough to work well in very extreme situations, specifically, in the context of super intelligent machines?

I think the frame problem that you bring up is a key part of this. I think of the frame problem as being to do with the meaning of something with a context. It’s something which is messy and complex rather than neat; it depends on complicated relationships within large amounts of knowledge. Trying to plug the holes one equation at a time, e.g. Cyc project, is like trying to move a mountain with a teaspoon.

People’s ethical ideas have this nature: big, complex, lots of context dependencies and not even logically consistent if pushed too far. You can’t put all this into a neat formula.

“give me an equation for “ethical”
The analytical tradition, especially Carnap, has recommended exactly that, namely statements shoudl always be formalized, if they can’t be they shouldn’t be uttered. In Wittgenstein’s tractatus every sentence is a tautology (an equation) and for the rest the guideline is: “Whereof one cannot speak, thereof one must be silent.” The book attempts at describing everything which one can speak about. However later Wittgenstein revised many of his views. He found that there is no 1-1 relationship between language and logic. Language becomes meaningful in contexts. To phrase it differently: the environment provides meaning. One simplistic exampe is a street sign.

I think the problem which one run quickly into is the frame problem. If I give a robot a precise rule after which he should act at all times (Asimov’s and other laws), the question is to what extent this is possible at all. The Stanford Encyclopedia has a very good article on the frame problem [1]. The central question is: “how does the robot limit the scope of the propositions it must reconsider in the light of its actions?” If the robot is sufficiently intelligent he will carry out complex tasks. When someone (person A) describes to another person (peroson B) what he wants him to do, some time will be spent on describing the basic idea. However if A explains to B everything in very great detail he would be better of doing the work himself. If A says to B: “please get me a cup” and B brings a jar that is clearly a violation. If B brings a small cup, A might say: “bring me a bigger cup”. If there is no cup in the right size available and B goes of to buy one that might be a violation.

A good strategy for resolving the problem would be to focus on certain situations (“ethic games”). For instance the situation where a robot is a slave is clearly structured. If the robot serves the masters friend is perhaps different. When the robot has more autonomy it will be more complex. If the robot goes to the mall to buy things, there has to be regulation on the laws. He has to be able to place purchase contracts, he has to be responsible for his actions (criminal law), and so on.

Can we have a mathematical theory of ethics? Economics tries to be this theory. Most of the time the argument goes like the following. Given the preferences of househoulds, which mechanism maxmizes social welfare? Welfare is reduced to the optimal allocation of resources, so many dimensions of the problem are left out, such as the political, cultural, biological and indeed ethical dimension. And indeed the Utiliterian axioms (Neumann/Morgenstern) don’t hold even in the most strictest scenarios.

Anyone who is interested in a more ellaborate discussion can reach me via ben@cosmoi.de

[1] “http://plato.stanford.edu/entries/frame-problem/”

Put your money where your mouth is: give me an equation for “ethical”.