My Halloween lecture has been uploaded to youtube. The basic outline is:

* what is intelligence?
* Solomonoff induction
* Hutter’s AIXI
* Monte Carlo AIXI (here’s the missing video of it playing pac-man)
* universal intelligence measure
* what neuroscience can teach us about AGI design
* early 2020′s: the Halloween scenario

You can get the slides here. I talked for 2 hours, so it’s broken up into many parts on youtube: Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9 Part 10 Part 11 Part 12

Thanks to David Wood at ExtroBritannian for organising this, and all the people who attended — especially those who travelled from other cities and countries, the intelligent questions during my talk, and all the positive feedback I’ve received since. Thanks also to Anders Sandberg for the picture of me speaking that I stole from his flicker stream.

Now, I understand that a really wonderfully nice AGI is probably a very good thing, and a flawed one is probably bad news.  We can all support efforts to push AGI towards the more desirable types of outcomes.  But what about AGI research in general?  That is, the work that goes into trying to figure out how to make artificial systems more powerful and general, in other words, more intelligent.  Is this a good thing? Is it a bad thing?

More pointedly: Imagine that you seriously thought that you might be able to build the first AGI.  Other people might think you’re deluded, and maybe they are right.  Nevertheless, from where you stand it looks like you have a real chance of making it happen.  Would you go ahead and actually try to do it?

I’m not convinced that ethics can’t be derived from some deeper unifying principle.  I’m also not convinced that it can, lest you misunderstand me.  What I do accept is that if this is possible then finding such a principle and convincingly arguing for it is likely to be difficult in the extreme, and probably not something that is likely to happen before the singularity.  Nevertheless, I haven’t yet seen any argument so devastating to this possibility that I’m willing to move it from being extremely difficult to certainly impossible.  Any system of ethics that does derive from some unifying metaethical principle is almost certainly going to be different to our present (western?) ethical notions.  I think some degree of this is acceptable, given that our ethical ideas do change a bit over time.  Furthermore, no matter how human we try to make the ethical system of a powerful AGI, post-singularity we are still going to be faced with ethical challenges that our pre-singularity ethics were never set up to deal with.  Thus, our ethics are going to have to be modified and updated in order to remain somewhat consistent and viable, otherwise we’ll end up with this kind of nonsense.

Anyway, let’s assume that this unifying principle either does not exist, or at least can’t be found.  How can we tell if an AGI is ethical given that we can’t explicitly and completely specify what this means?  This seems like the problem Turing faced when trying to determine whether a machine is intelligent or not.  He figured that he couldn’t explicitly and completely say what intelligence is, unlike the research by Hutter and myself, and thus he tried to dodge the issue in the obvious way by setting up an imitation game that doesn’t require an explicit description of intelligence.

Here we can do something similar: set up a group of people and the AGI and ask them ethical questions from a panel of expert judges.  If the judges cannot tell which the machine is, then it passes. Given that the morality function varies between people, and that we can’t say explicitly and completely what our own function is, this seems to be about the best we could hope for.  Naturally, this doesn’t prove that the AGI, or indeed any of the humans participating, are “good”.  An evil genius could probably pass such a test.  Rather, it is simply designed to test whether the AGI is at least able to compute a version of the human morality function which is sufficiently similar to ours that it is able to pass as being human.  Whether the AGI (or human) actually takes its human-passable morality function and reliably and consistently seeks to follow it into the future is a whole other set of problems.  Thus, passing such a test is perhaps a necessary, but certainly not a sufficient condition for having an ethical AGI.

I’m sure somebody must have proposed this idea before, but at least my half hearted attempt to find the idea on Google didn’t turn up anything.  I should also point out that in order for this test to work you’d probably want the AGI to pass a more general Turing test first so that it doesn’t get singled out by the judges for various other reasons.  Only then should you bring in a group of expert ethicists to try to judge which of the test subjects was ethically inhuman.  We would also want to include in the test subjects a few very nice people and a couple of professional ethicists as we wouldn’t want the AGI to be able to “fail” for being too nice or consistently ethical.

My thesis is now available at lulu.com.  As promised, it’s at cost, which works out at $18 plus shipping.  It’s all under a creative commons licence and in a few months I’ll put the pdf online for free.  I’ll also write a post shortly on all the tricks involved in publishing on lulu.com with LaTeX, in case you plan on doing something similar.

Table of Contents

Preface
Thesis outline
Prerequisite knowledge
Acknowledgements

1 Nature and Measurement of Intelligence
1.1 Theories of intelligence
1.2 Definitions of human intelligence
1.3 Definitions of machine intelligence
1.4 Intelligence testing
1.5 Human intelligence tests
1.6 Animal intelligence tests
1.7 Machine intelligence tests
1.8 Conclusion

2 Universal Artificial Intelligence
2.1 Inductive inference
2.2 Bayes’ rule
2.3 Binary sequence prediction
2.4 Solomonoff’s prior and Kolmogorov complexity
2.5 Solomonoff-Levin prior
2.6 Universal inference
2.7 Solomonoff induction
2.8 Agent-environment model
2.9 Optimal informed agents
2.10 Universal AIXI agent

3 Taxonomy of Environments
3.1 Passive environments
3.2 Active environments
3.3 Some common problem classes
3.4 Ergodic MDPs
3.5 Environments that admit self-optimising agents
3.6 Conclusion

4 Universal Intelligence Measure
4.1 A formal definition of machine intelligence
4.2 Universal intelligence of various agents
4.3 Properties of universal intelligence
4.4 Response to common criticisms
4.5 Conclusion

5 Limits of Computational Agents
5.1 Preliminaries
5.2 Prediction of computable sequences
5.3 Prediction of simple computable sequences
5.4 Complexity of prediction
5.5 Hard to predict sequences
5.6 The limits of mathematical analysis
5.7 Conclusion

6 Temporal Difference Updating without a Learning Rate
6.1 Temporal difference learning
6.2 Derivation
6.3 Estimating a small Markov process
6.4 A larger Markov process
6.5 Random Markov process
6.6 Non-stationary Markov process
6.7 Windy Gridworld
6.8 Conclusion

7 Discussion
7.1 Are super intelligent machines possible?
7.2 How could intelligent machines be developed?
7.3 Is building intelligent machines a good idea?

Appendix
A Notation and Conventions
B Ergodic MDPs admit self-optimising agents
B.1 Basic definitions
B.2 Analysis of stationary Markov chains
B.3 An optimal stationary policy
B.4 Convergence of expected average value
C Definitions of Intelligence
C.1 Collective definitions
C.2 Psychologist definitions
C.3 AI researcher definitions

Bibliography
Index

]]> http://www.vetta.org/2008/07/machine-super-intelligence/feed/ 7 http://www.vetta.org/2008/06/siai-canada-academic-prize-for-2008/ http://www.vetta.org/2008/06/siai-canada-academic-prize-for-2008/#comments Mon, 23 Jun 2008 10:23:30 +0000 Shane Legg http://www.vetta.org/?p=63 Continue reading →]]> This morning I received the wonderful news that I’ve won the Singularity Institute for Artificial Intelligence – Canada Academic Prize for 2008!

The award is in “recognition of [my] efforts to improve AI theory” and is worth CAD $10,000.  This will certainly help my budget over the next two years while I study at the Gatsby Unit in London.  So, thank you to SIAI Canada, and to all the Canadians whose donations made this money available!

Speaking of my research, after a long weekend of final edits, corrections, formatting, indexing, embedding fonts and other complexity (I’ll write a blog post about what I had to do at some point), I’ve finally uploaded my thesis “Machine Super Intelligence” to lulu.com and have ordered a test copy.  Once I’ve checked that everything is ok I’ll let you know where copies can be ordered.  Copies should be USD $18 plus shipping for a 200 page casewrap hardcover.  Probably about in a month…

A few months ago I started reading the Overcoming Bias blog, on which Yudkowsky was discussing physics (I’m still not sure why he’s so involved with physics now, but he’s slowly getting around to explaining this).  Anyway, I have found his writings here to be much more to my liking.  His ideas seem clearer, more focused and organised and I find the style and tone to be much improved.  If you like some interesting philosophical discussions and you haven’t seen the blog already, you might want to check it out.

Encouraged by this I decided to have another look at Yudkowsky’s writings on the ethics and safety of artificial general intelligence.  This time I went for one of his most recent pieces: Artificial Intelligence as a Positive and Negative Factor in Global Risk, a book chapter he did for Global Catastrophic Risks.  If, like me, you were put off by some of his earlier writings, you might want to have a look at this newer document.  While I naturally don’t agree with everything in the chapter, in my opinion the points are clearly argued and fit together well.  Indeed, I found myself agreeing with most of his points.  In short, if you are new to the safety of powerful AI technologies, I’d suggest that you put this document at the top of your reading list.

In part 3 I’ll get back to my own thoughts on the matter…