Given that AGI is still a young and relatively unknown part of the wider AI community, it’s great to see such well known researchers putting their names behind this conference. As a member of the program committee I’ve been able to check out some of the submissions so far and I’ve been pleasantly surprised by their quality — indeed, this is what gave me the impetus to write this post! If you’d like to submit something there’s still time: the deadline is the 1st of December.

My Halloween lecture has been uploaded to youtube. The basic outline is:

* what is intelligence?
* Solomonoff induction
* Hutter’s AIXI
* Monte Carlo AIXI (here’s the missing video of it playing pac-man)
* universal intelligence measure
* what neuroscience can teach us about AGI design
* early 2020′s: the Halloween scenario

You can get the slides here. I talked for 2 hours, so it’s broken up into many parts on youtube: Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9 Part 10 Part 11 Part 12

Thanks to David Wood at ExtroBritannian for organising this, and all the people who attended — especially those who travelled from other cities and countries, the intelligent questions during my talk, and all the positive feedback I’ve received since. Thanks also to Anders Sandberg for the picture of me speaking that I stole from his flicker stream.

What I will remember most clearly about this summit was Peter Thiel. Firstly, the pre-summit party at his penthouse apartment. That was a treat: a tiny peak into the world of the ultra-rich. His mix of intelligence, focus and energy was quite something to behold and he left a real impression on me. His talk was also among the most engaging in my opinion. No slides, no fluffy stuff, just a straight delivery of ideas and analysis seemingly off the cuff with no notes. In his talk and comments afterwards, the main thing that stuck in my mind was his concern that the singularity wouldn’t arrive quickly enough. Really?

I don’t understand this. If this is really his main concern, why isn’t he using at least a tiny part of his huge financial resources to try to make it happen sooner? He’s funding SIAI, but they aren’t exactly trying to make the singularity happen sooner. If time frame is his primary concern, and I can’t see any reason why he would say this if it were not the case (it’s easier to think of reasons why he wouldn’t) why the inactivity? Just a few percent of his wealth spread across a handful of small projects over the next few decades would make a night-and-day difference to funding in this area. There is essentially no money available to do AGI research and thus we spend our time working on the related topics in the areas of machine learning and theoretical/computational neuroscience that actually are funded. I guess he’s thought carefully about this, but at least I can’t see how his actions can be consistent with his stated beliefs.

Many of the other talks I’d heard before. In some cases so many times I’m sure I could give them myself. I hadn’t heard about the work of Gregory Benford before where they are breeding up very long lived flies and then, if I understand correctly, looking at how the different bio-chemical pathways in these flies change to produce long life. As humans share many of these pathways, they then make pills that reproduce some of these effects in humans in the hope that we too will live longer. I don’t know enough biology to be able to comment further, but it sounds like an interesting long-shot idea to at least try. I guess it will be some time before we know whether it has any effect.

As per my last summit, the most interesting thing for me was meeting and talking to people one on one, or in small groups. I ended up giving my 20 minute spiel on what I consider to be the most promising approach to AGI at least 10 times. It was quite positively received — I’d expected it to be a harder sell. Many of these people seemed to be revising their expectations after talking to me. One exception, and a person I was especially interested in talking to, was Moshe Looks from Google Research. He strikes me as a pretty sharp and temperate thinker who also has a fair amount of hands on experience working on AI/AGI both academically and commercially. Strangely, we seemed agreed on just about everything regarding which approaches were the most promising, in what degrees and why. I was more bullish on the time line for developments, but not radically so. If I add an extra 50% to my time line, which historically appears to be my degree of miscalibration when forecasting technology developments, then we even come up with the same time estimates. Given that I’ve only met him once before, I found this degree of agreement between us quite striking.

My other impression was the scale of the summit this time: over 800 people which I think was roughly twice the size of the last summit. There also seemed to be a more diverse group attending. Along with the usual mix of geeks, nerds and all too obvious Aspergers cases, there seemed to be more general public interest as far as I could tell. I think the gender ratio might have improved too, indeed there was even a fair number of attractive young women. Year by year, is the singularity idea slowly starting to go mainstream? Perhaps it is and I suspect this summit is one of the driving forces. Well done.

The basic idea goes as follows. In full AIXI you have an extended Solomonoff predictor to model the environment, and an expecti-max tree to compute the optimal action. In order to scale AIXI down and still have something of roughly the same form, you need to find a tractable way to replace both of these two items. Here’s what they did: in the place of extended Solomonoff induction a version of context tree weighting (CTW) is used. CTW has to be extended for this application similar to the way Hutter had to extend Solomonoff induction to active environments for AIXI. In the place of the expecti-max tree search a Monte Carlo tree search is used, similar to that used in Go playing programs: initial selection within the tree, tree expansion, a so called play-out policy, followed by a backup stage to propagate the new information back into the model. You have to be a bit careful here because as the agent imagines different future observations and actions it has to update its hypothetical beliefs to reflect these in order for its analysis and decision making to be consistent. Then, once this possible future has been evaluated, the effect of this on the agent’s model of the world has to be unwound so that the agent doesn’t, in effect, start confusing its fantasies with its present reality.

The algorithm is both embarrassingly-parallel and any-time, which is very nice. In less technical language: it would be fairly easy to get it to run efficiently on a massively parallel supercomputer, and it also has the property that it can be forced to decide what action to take at any moment always returning the best action it had been able to compute so far. Thus, if you want a smarter agent, just give it more time and/or CPUs. Already they have shown that MC-AIXI can learn to solve a bunch of basic POMDP problems, including playing a somewhat reasonable game of Pac-man. It would be interesting to see what it was capable of on a supercomputer with ten thousand times the resources of their desktop PC.

A key question for future research is to make better sequence predictors, in particular to be able to identify more complex types of patterns in the agent’s history. I guess all sorts of machine learning techniques could come into play here… and possibly combine to produce quite a powerful RL agent?

My guess is that it will play out like this: SIAI’s contribution will be to raise the level of awareness of the dangers of powerful AGI over the next decade or two.  As AGI progresses their message will be taken more seriously. Then at some point powerful teams will start to race towards building the first real AGI.  The degrees to which these groups will have been influenced by SIAI thinking will vary.  Due to greed, wishful thinking, ignorance and what have you, in general safety will come second to progress.  A short period of time later the post human period will begin.  Where that goes will depend to some extent on fundamental properties of highly intelligent systems, and to some extent on these systems’ specific initial conditions.  Given our limited understanding, this currently feels like a roll of the dice to me.

Although SIAI raising awareness is helpful, I see it as playing a supporting role rather than a central one.  Some global problems require mass action and this can be achieved through mass awareness driving policy changes. Other problems, such as AGI development, will be driven by small focused groups of highly skilled people vying to be first.  Working out which ideas and teams will win such a race is impossible: even experienced VCs mostly pick duds.  The best one can do is to back a range of promising teams in the knowledge that only one needs to succeed in order to more than recoup the losses on those that failed.

The impression I get from the outside is that SIAI views AGI design and construction as so inherently dangerous that only a centrally coordinated design effort towards a provably correct system has any hope of producing something that is safe.  My view is that betting on one horse, and a highly constrained horse at that, spells almost certain failure.  A better approach would be to act as a parent organisation, a kind of AGI VC company, that backs a number of promising teams.  Teams that fail to make progress get dropped and new teams with new ideas are picked up.  General ideas of AGI safety are also developed in the background until such a time when one of the teams starts to make serious progress.  At this time the focus would be to make the emerging AGI design as safe as possible.

I realise that the safety margins on such a system will most likely fall short of what some consider to be necessary.  It’s not the “engineered for maximal safety from the outset” approach that I believe core SIAI people favour.  I also agree that this fact will create significant risks.  My point is that trying to build an (almost) ideally safe AGI also entails a great deal of risk, albeit for a different reason: you’re unlikely to be first or even close to first. If I had to roll the dice which bet would I prefer?  To hope for an ideal AGI design to be finished on time, but most likely see some other random design take the honours and suffer the consequences?  Or to bet on a number of AGI teams where if any of them makes serious progress there is an organisation behind them trying to work out how to make the emerging design as safe as possible?  I can’t quantify these risks, but my gut feeling is to go for the second option.

There is also a funding angle to what I’m suggesting.  Donating money feels a bit like a sure loss, even when the cause is a good one.  Naturally the money has implications, but to you personally the money is just gone never to be seen again.  Investing feels different: you have an ongoing stake in something, a little bit of ownership.  If things go well you can make a big profit, and even if they don’t you can probably get at least some of your money back.  As a result it is often easier to get somebody to invest $1000 in a project than it is to get them to donate $100.  I suspect that there are many people who aren’t donating to SIAI, or aren’t donating much, who would however be interested in investing in safe AGI development.  Given the difficulty of picking the winners, a managed portfolio of promising small AGI startup teams seems like the best approach to me, where the parent organisation has safe AGI as a core concern.

Really? It takes babies half a year or more to learn to control their limbs in fairly basic ways. How many reinforcement learning algorithms get run for six months in a single learning trial?  As an adult if we try to learn some new control task, such as balancing a pole, it can take hours of effort despite having years of prior motor control experience. A reinforcement learning algorithm, on the other hand, can learn to solve some of these problems in seconds with no prior experience at all.  In a few minutes algorithms can even learn the much more difficult double pole balancing problem. This is a problem that would take me months to master, if indeed I could ever get the hang of it.  If we think about problems that humans can learn to solve quite quickly, but that machines have not yet mastered, there is usually a massive amount of prior knowledge that people are using, knowledge that may have taken years to acquire.

It appears to me that we now have some very powerful learning algorithms, in the sense that they can learn moderately complex control tasks with very little data. The performance of these algorithms is already significantly super human in some contexts. This is true not just for reinforcement learning, but many kinds of machine learning algorithms.  Unfortunately, for the more ambitious goals of artificial intelligence these highly data efficient algorithms on our moderately sized data sets haven’t worked very well.  One key reason for this, I suspect, is that these are inherently messy and complex problems that can only be solved with truly massive amounts of data.  I think that lot of human abilities that AI research has struggled with probably fall into this category, e.g. vision, language and common sense knowledge.

Something similar has been expressed in The Unreasonable Effectiveness of Data by Alon Halevy, Peter Norvig and Fernando Pereira. One of the examples they cite is that for years people tried to construct a grammar for the English language by hand. Even at 1,700 pages, however, the grammar was still incomplete! Similar efforts have gone into systems to translate one language into another. To start with a manual approach looked promising as relatively few rules cover a significant percentage of the cases. However, as you try to improve the system the number of rules needed starts to grow rapidly, and eventually explodes. The solution, which is now used by the best translation engines, has been to move to a data driven approach: you take a learning algorithm that scales well and then feed it massive quantities of data. Given enough data, all sorts of subtleties and complexities of the language become statistically learnable.  I think there is a key idea here for wannabe AGI designers.

As I do at the start of each year, I’ve spent some time reconsidering when I think roughly human level AGI will exist.  I’ve again decided to leave it at 2025, but now with a reduced standard deviation of 5 years.  Computer power is a limitation as researchers typically have limited hardware budgets, unlike the DOD guys and their monster supercomputers.  From what I’ve read, computer power should continue to grow exponentially for at least the next 5 years, and probably the next 10.  So I don’t see this as being too much of an issue in the coming decade.  On the algorithm side, I think things are progressing really well.  I know a number of very talented people who are working on what I think are the key building blocks required before the construction of a basic AGI can begin.  I’m certain these problems are solvable, but whether it takes 2 years or 10 years is hard to guess.  This is my main source of uncertainty.

UPDATE 11 April 2009: Note that these predictions do not take into account my apparent bias towards predicting that things will happen faster than they actually do (see previous post). The required compensation for technology events appears to be about 50% more time. Thus if you want the “Shane meta predictor”, then take 2033 as the expected date, perhaps with a standard deviation of 7 years. At least with financial markets I trust my meta predictor more than my straight predictions and thus I buy and sell accordingly to it. So I suppose that if I had to put money on a date, I should go with 2033. But don’t ask me why it’s going to take that long: I really don’t know.

One example of this has been trying to predict the medium term direction of the stock market over the last 15 years.  The evidence so far shows that I’m consistently good at predicting what will happen, but that I predict that it will happen much sooner than it actually does; I roughly need to double my time estimates.  I’m now trying to mentally correct for this bias in the trades I make, but it will take some years to see if this is working.

In technological matters, I’ve generally done quite well.  My picks for Sun, java, digital music, linux, MySQL and open source in general were pretty much on the mark.  I thought machine learning use in industry would be bigger than it is today, but I wasn’t too far off.  My biggest mistake was to badly underestimate how much Microsoft’s revenues would grow over the last 10 years — I thought they’d already almost saturated the market and its ability to pay.  Like stock markets, my most consistent error has been to be to predict that things will happen faster than they actually do.  I typically need to add about 50% to the time required.

As I don’t have a lot of my own technological predictions to look at, and some remain in the future, I’ve recently been looking at predictions made by others.  I found a few of my old computer and science magazines from the early 80′s through to the late 90′s which contained predictions, and I also dug up Kurzweil’s “The age of spiritual machines” written in 1999 in which he has a whole chapter about 2009.  There were a lot of hits and misses, but if I stand back and try to see the big picture, a pattern becomes clear: Predictions about basic hardware performance, even one I saw in a magazine from 25 years ago, are amazingly accurate.  But you probably knew that already.  Predictions about what would be technologically possible to do at a given point in time were not as accurate, but were still pretty good.  Where things really started to go wrong was when they tried to predict not what would be possible, but what the majority of people would actually be doing.

Perhaps some examples would best explain this.  State of the art speech recognition systems, such as some of the systems that were being developed at IDSIA when I was there, work impressively well.  However, once you’ve learnt to touch type it is typically easier, quieter, more convenient (especially when editing or coding) and far more private to use a keyboard.  I don’t care how good speech recognition is, I don’t want to sit in a room full of people talking out loud to their computers all day.  I only know one person who routinely uses speech recognition to input text.  The fact that speech recognition is technologically doable, doesn’t translate into it being practically useful for many everyday situations.

There are plenty of predictions that fail in this way: the prediction that everybody now would be making video calls on their cellphones.  It’s certainly technologically possible, I saw a guy with a phone that could do it two years ago, but almost nobody does it.  Or that most long distance air travel would be in supersonic jets.  Again, technologically possible, has been for a long time, but not done in practice.  Or that all mice would be wireless by now.  Technologically possible, has been for years, but as far as I can tell most new mice still have cords.  Or that most people driving long distance on freeways would get their car to automatically drive itself.  I’m sure that’s technologically possible, but I don’t see anybody doing it.  Or that your computer would log you in based on recognising your face or your voice.  Technologically possible today, but not done in practice.  And so on.

In short: Predicting raw performance is surprising accurate.  Predicting what will be possible using the knowledge and technology of some future date can also be done with moderate success.  Predicting what the population will routinely do, however, is much harder.  The latter is largely decided by habit, cost and convenience.  Simply being possible isn’t enough.  Note that predicting the development of the first powerful AGI is of the second type.

Now, I understand that a really wonderfully nice AGI is probably a very good thing, and a flawed one is probably bad news.  We can all support efforts to push AGI towards the more desirable types of outcomes.  But what about AGI research in general?  That is, the work that goes into trying to figure out how to make artificial systems more powerful and general, in other words, more intelligent.  Is this a good thing? Is it a bad thing?

More pointedly: Imagine that you seriously thought that you might be able to build the first AGI.  Other people might think you’re deluded, and maybe they are right.  Nevertheless, from where you stand it looks like you have a real chance of making it happen.  Would you go ahead and actually try to do it?

I’m not convinced that ethics can’t be derived from some deeper unifying principle.  I’m also not convinced that it can, lest you misunderstand me.  What I do accept is that if this is possible then finding such a principle and convincingly arguing for it is likely to be difficult in the extreme, and probably not something that is likely to happen before the singularity.  Nevertheless, I haven’t yet seen any argument so devastating to this possibility that I’m willing to move it from being extremely difficult to certainly impossible.  Any system of ethics that does derive from some unifying metaethical principle is almost certainly going to be different to our present (western?) ethical notions.  I think some degree of this is acceptable, given that our ethical ideas do change a bit over time.  Furthermore, no matter how human we try to make the ethical system of a powerful AGI, post-singularity we are still going to be faced with ethical challenges that our pre-singularity ethics were never set up to deal with.  Thus, our ethics are going to have to be modified and updated in order to remain somewhat consistent and viable, otherwise we’ll end up with this kind of nonsense.

Anyway, let’s assume that this unifying principle either does not exist, or at least can’t be found.  How can we tell if an AGI is ethical given that we can’t explicitly and completely specify what this means?  This seems like the problem Turing faced when trying to determine whether a machine is intelligent or not.  He figured that he couldn’t explicitly and completely say what intelligence is, unlike the research by Hutter and myself, and thus he tried to dodge the issue in the obvious way by setting up an imitation game that doesn’t require an explicit description of intelligence.

Here we can do something similar: set up a group of people and the AGI and ask them ethical questions from a panel of expert judges.  If the judges cannot tell which the machine is, then it passes. Given that the morality function varies between people, and that we can’t say explicitly and completely what our own function is, this seems to be about the best we could hope for.  Naturally, this doesn’t prove that the AGI, or indeed any of the humans participating, are “good”.  An evil genius could probably pass such a test.  Rather, it is simply designed to test whether the AGI is at least able to compute a version of the human morality function which is sufficiently similar to ours that it is able to pass as being human.  Whether the AGI (or human) actually takes its human-passable morality function and reliably and consistently seeks to follow it into the future is a whole other set of problems.  Thus, passing such a test is perhaps a necessary, but certainly not a sufficient condition for having an ethical AGI.

I’m sure somebody must have proposed this idea before, but at least my half hearted attempt to find the idea on Google didn’t turn up anything.  I should also point out that in order for this test to work you’d probably want the AGI to pass a more general Turing test first so that it doesn’t get singled out by the judges for various other reasons.  Only then should you bring in a group of expert ethicists to try to judge which of the test subjects was ethically inhuman.  We would also want to include in the test subjects a few very nice people and a couple of professional ethicists as we wouldn’t want the AGI to be able to “fail” for being too nice or consistently ethical.

My thesis is now available at lulu.com.  As promised, it’s at cost, which works out at $18 plus shipping.  It’s all under a creative commons licence and in a few months I’ll put the pdf online for free.  I’ll also write a post shortly on all the tricks involved in publishing on lulu.com with LaTeX, in case you plan on doing something similar.

Table of Contents

Preface
Thesis outline
Prerequisite knowledge
Acknowledgements

1 Nature and Measurement of Intelligence
1.1 Theories of intelligence
1.2 Definitions of human intelligence
1.3 Definitions of machine intelligence
1.4 Intelligence testing
1.5 Human intelligence tests
1.6 Animal intelligence tests
1.7 Machine intelligence tests
1.8 Conclusion

2 Universal Artificial Intelligence
2.1 Inductive inference
2.2 Bayes’ rule
2.3 Binary sequence prediction
2.4 Solomonoff’s prior and Kolmogorov complexity
2.5 Solomonoff-Levin prior
2.6 Universal inference
2.7 Solomonoff induction
2.8 Agent-environment model
2.9 Optimal informed agents
2.10 Universal AIXI agent

3 Taxonomy of Environments
3.1 Passive environments
3.2 Active environments
3.3 Some common problem classes
3.4 Ergodic MDPs
3.5 Environments that admit self-optimising agents
3.6 Conclusion

4 Universal Intelligence Measure
4.1 A formal definition of machine intelligence
4.2 Universal intelligence of various agents
4.3 Properties of universal intelligence
4.4 Response to common criticisms
4.5 Conclusion

5 Limits of Computational Agents
5.1 Preliminaries
5.2 Prediction of computable sequences
5.3 Prediction of simple computable sequences
5.4 Complexity of prediction
5.5 Hard to predict sequences
5.6 The limits of mathematical analysis
5.7 Conclusion

6 Temporal Difference Updating without a Learning Rate
6.1 Temporal difference learning
6.2 Derivation
6.3 Estimating a small Markov process
6.4 A larger Markov process
6.5 Random Markov process
6.6 Non-stationary Markov process
6.7 Windy Gridworld
6.8 Conclusion

7 Discussion
7.1 Are super intelligent machines possible?
7.2 How could intelligent machines be developed?
7.3 Is building intelligent machines a good idea?

Appendix
A Notation and Conventions
B Ergodic MDPs admit self-optimising agents
B.1 Basic definitions
B.2 Analysis of stationary Markov chains
B.3 An optimal stationary policy
B.4 Convergence of expected average value
C Definitions of Intelligence
C.1 Collective definitions
C.2 Psychologist definitions
C.3 AI researcher definitions

Bibliography
Index

]]> http://www.vetta.org/2008/07/machine-super-intelligence/feed/ 7 http://www.vetta.org/2008/06/siai-canada-academic-prize-for-2008/ http://www.vetta.org/2008/06/siai-canada-academic-prize-for-2008/#comments Mon, 23 Jun 2008 10:23:30 +0000 Shane Legg http://www.vetta.org/?p=63 Continue reading →]]> This morning I received the wonderful news that I’ve won the Singularity Institute for Artificial Intelligence – Canada Academic Prize for 2008!

The award is in “recognition of [my] efforts to improve AI theory” and is worth CAD $10,000.  This will certainly help my budget over the next two years while I study at the Gatsby Unit in London.  So, thank you to SIAI Canada, and to all the Canadians whose donations made this money available!

Speaking of my research, after a long weekend of final edits, corrections, formatting, indexing, embedding fonts and other complexity (I’ll write a blog post about what I had to do at some point), I’ve finally uploaded my thesis “Machine Super Intelligence” to lulu.com and have ordered a test copy.  Once I’ve checked that everything is ok I’ll let you know where copies can be ordered.  Copies should be USD $18 plus shipping for a 200 page casewrap hardcover.  Probably about in a month…

A few months ago I started reading the Overcoming Bias blog, on which Yudkowsky was discussing physics (I’m still not sure why he’s so involved with physics now, but he’s slowly getting around to explaining this).  Anyway, I have found his writings here to be much more to my liking.  His ideas seem clearer, more focused and organised and I find the style and tone to be much improved.  If you like some interesting philosophical discussions and you haven’t seen the blog already, you might want to check it out.

Encouraged by this I decided to have another look at Yudkowsky’s writings on the ethics and safety of artificial general intelligence.  This time I went for one of his most recent pieces: Artificial Intelligence as a Positive and Negative Factor in Global Risk, a book chapter he did for Global Catastrophic Risks.  If, like me, you were put off by some of his earlier writings, you might want to have a look at this newer document.  While I naturally don’t agree with everything in the chapter, in my opinion the points are clearly argued and fit together well.  Indeed, I found myself agreeing with most of his points.  In short, if you are new to the safety of powerful AI technologies, I’d suggest that you put this document at the top of your reading list.

In part 3 I’ll get back to my own thoughts on the matter…