Given that AGI is still a young and relatively unknown part of the wider AI community, it’s great to see such well known researchers putting their names behind this conference. As a member of the program committee I’ve been able to check out some of the submissions so far and I’ve been pleasantly surprised by their quality — indeed, this is what gave me the impetus to write this post! If you’d like to submit something there’s still time: the deadline is the 1st of December.

Now, 36 years later, it’s interesting to think about how the speakers’ various views and predictions have played out. Overall, the analysis by Lighthill felt the most coherent to me, and I’d say that what has since happened largely backs him up, though it can be argued that he helped to cause this outcome. I agree that he slowed AI down a lot, but 36 years is a rather long time and in the types of problems that he was focusing on there hasn’t been much progress. In response the other debaters mostly just pointed to small advances that had occurred and indicated that they felt that more advances were on the way. Lighthill then denied that these advances showed any real progress towards intelligence.

This feels a lot like today: sceptics say that AI has made no progress, optimists point to lots of advances, and sceptics then say that these advances are not what they consider to be real intelligence. I think this points to perhaps the most fundamental problem in the field: if you can’t define intelligence, how do you judge whether progress is being made? It’s as true today as it was then, and it’s why I think that trying to define intelligence is so important. I like the fact that they keep on saying that an intelligent machine should be able to perform well in a “wide range of situations”, because, of course, this is very much the view of intelligence that I have taken.

My Halloween lecture has been uploaded to youtube. The basic outline is:

* what is intelligence?
* Solomonoff induction
* Hutter’s AIXI
* Monte Carlo AIXI (here’s the missing video of it playing pac-man)
* universal intelligence measure
* what neuroscience can teach us about AGI design
* early 2020′s: the Halloween scenario

You can get the slides here. I talked for 2 hours, so it’s broken up into many parts on youtube: Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9 Part 10 Part 11 Part 12

Thanks to David Wood at ExtroBritannian for organising this, and all the people who attended — especially those who travelled from other cities and countries, the intelligent questions during my talk, and all the positive feedback I’ve received since. Thanks also to Anders Sandberg for the picture of me speaking that I stole from his flicker stream.

The basic idea goes as follows. In full AIXI you have an extended Solomonoff predictor to model the environment, and an expecti-max tree to compute the optimal action. In order to scale AIXI down and still have something of roughly the same form, you need to find a tractable way to replace both of these two items. Here’s what they did: in the place of extended Solomonoff induction a version of context tree weighting (CTW) is used. CTW has to be extended for this application similar to the way Hutter had to extend Solomonoff induction to active environments for AIXI. In the place of the expecti-max tree search a Monte Carlo tree search is used, similar to that used in Go playing programs: initial selection within the tree, tree expansion, a so called play-out policy, followed by a backup stage to propagate the new information back into the model. You have to be a bit careful here because as the agent imagines different future observations and actions it has to update its hypothetical beliefs to reflect these in order for its analysis and decision making to be consistent. Then, once this possible future has been evaluated, the effect of this on the agent’s model of the world has to be unwound so that the agent doesn’t, in effect, start confusing its fantasies with its present reality.

The algorithm is both embarrassingly-parallel and any-time, which is very nice. In less technical language: it would be fairly easy to get it to run efficiently on a massively parallel supercomputer, and it also has the property that it can be forced to decide what action to take at any moment always returning the best action it had been able to compute so far. Thus, if you want a smarter agent, just give it more time and/or CPUs. Already they have shown that MC-AIXI can learn to solve a bunch of basic POMDP problems, including playing a somewhat reasonable game of Pac-man. It would be interesting to see what it was capable of on a supercomputer with ten thousand times the resources of their desktop PC.

A key question for future research is to make better sequence predictors, in particular to be able to identify more complex types of patterns in the agent’s history. I guess all sorts of machine learning techniques could come into play here… and possibly combine to produce quite a powerful RL agent?

So what does the brain do? For over a decade it has been known that temporal difference learning, a type of model-free RL algorithm, appears to explain the activity of dopamine neurons and their dorsolateral striatal projections. It has also been observed that parts of the prefrontal cortex appear to implement some kind of model-based RL algorithm. Mammalian brains, then, appear to get the best of worlds by having model-free and model-based RL algorithms and then choosing which to use on the fly.  Pretty clever huh?

A key question then is how this choice is made. I recently read Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control from Nature Neuroscience, by Nathaniel Daw, Yael Niv and Peter Dayan. They suggest that the brain may be using some kind of Bayesian principle based on the uncertainty estimates generated by each system. They implement such a system and show how it can explain a range of experimental data from animal studies.

This research is now four years old and there is plenty of significant research in the area that follows it — an “embarrassment of riches” as Dayan recently described it.  Nevertheless, I think it’s a good example of the kind of crossover between machine learning and neuroscience that is starting to take place.  Indeed, the more neuroscientists learn about the brain, the more it starts to look like the rough outline of an AGI design.

Really? It takes babies half a year or more to learn to control their limbs in fairly basic ways. How many reinforcement learning algorithms get run for six months in a single learning trial?  As an adult if we try to learn some new control task, such as balancing a pole, it can take hours of effort despite having years of prior motor control experience. A reinforcement learning algorithm, on the other hand, can learn to solve some of these problems in seconds with no prior experience at all.  In a few minutes algorithms can even learn the much more difficult double pole balancing problem. This is a problem that would take me months to master, if indeed I could ever get the hang of it.  If we think about problems that humans can learn to solve quite quickly, but that machines have not yet mastered, there is usually a massive amount of prior knowledge that people are using, knowledge that may have taken years to acquire.

It appears to me that we now have some very powerful learning algorithms, in the sense that they can learn moderately complex control tasks with very little data. The performance of these algorithms is already significantly super human in some contexts. This is true not just for reinforcement learning, but many kinds of machine learning algorithms.  Unfortunately, for the more ambitious goals of artificial intelligence these highly data efficient algorithms on our moderately sized data sets haven’t worked very well.  One key reason for this, I suspect, is that these are inherently messy and complex problems that can only be solved with truly massive amounts of data.  I think that lot of human abilities that AI research has struggled with probably fall into this category, e.g. vision, language and common sense knowledge.

Something similar has been expressed in The Unreasonable Effectiveness of Data by Alon Halevy, Peter Norvig and Fernando Pereira. One of the examples they cite is that for years people tried to construct a grammar for the English language by hand. Even at 1,700 pages, however, the grammar was still incomplete! Similar efforts have gone into systems to translate one language into another. To start with a manual approach looked promising as relatively few rules cover a significant percentage of the cases. However, as you try to improve the system the number of rules needed starts to grow rapidly, and eventually explodes. The solution, which is now used by the best translation engines, has been to move to a data driven approach: you take a learning algorithm that scales well and then feed it massive quantities of data. Given enough data, all sorts of subtleties and complexities of the language become statistically learnable.  I think there is a key idea here for wannabe AGI designers.

Go, the Asian board game, has long been considered to be a profound challenge for artificial intelligence.  John McCarthy described it as the “new drosophila of AI”, Hans Berliner as a “task par excellence for AI”, and David Mechner as a “grand challenge task”.  Confucius was less emphatic in his support, commenting that, “Even playing [go] is better than being idle.  I can only presume that Confucius would have had more reverence for the game had he tried to program a computer to play it.  Among AI researchers, however, it has taken on something of a “holy grail” status.  Years have been spent carefully constructing go engines without success.  In 1998, a top go computer was beaten by a 6th dan player even though it was given a massive 29 stone advantage, meaning that it’s rating was something like 25 kyu.  If you’re not familiar with martial arts ratings systems, well, 25 kyu is only a little above a total beginner.  By 2003, another go program had progressed to about 15 kyu.  A big improvement, but nevertheless a beginner could beat it with a few months of training.  Computer go, in a nutshell, was very weak.

In 2007, MoGo, a Monte Carlo Tree Search based system developed by Paris University PhD candidate Sylvain Gelly, burst onto the scene and promptly thrashed all the other computers.  Its rating was around 2 kyu, almost a “black belt” level.  Then in 2008, MoGo beat a 7th dan professional player with a 9 stone handicap, putting its rating at around 2nd dan amateur.  A few months ago MoGo beat a 9th dan professional player with just a 6 stone handicap, putting its rating at around 3rd dan amateur.  Needless to say, the days of computers being unable to play go are over.  Only professionals and very highly ranked amateur players can now be confident of a victory in a game without handicap.

In post game analysis of the recent competition in Taiwan, with 20x more computer time MoGo managed to identify most of the mistakes it made and come up with better moves.  This means that in 5 or so years time MoGo will be significantly stronger due to faster hardware alone.  Even without more computer power, it appears that many of the mistakes made in Taiwan will be fixable with improved algorithms.  Given the rate at which computer go is now progressing, one can’t help but wonder how much longer humans will reign supreme in this ancient game.

Technical comments

While it’s interesting that MC tree search combined with modern brute force has proven effective in a game with such a high branching factor (often over 300), this isn’t the kind of profound insight into intelligence that AI older timers had in mind.  Moreover, much of the system is quite ad hoc and heuristic.  For example, rather than using fairly standard approaches to exploration vs. exploitation in the initial tree policy, such as Upper Confidence Bounds, top performing programs seem to use all sorts of intuitive, but otherwise somewhat arbitrary equations for this that include various statistics from the MC simulations.  The most important one is to note that go moves are somewhat recordable and thus if a move was made at all in a winning sequence then this is some evidence for making the move next.  While this estimator is biased as game moves aren’t truly recordable, it is a much lower variance statistic as the move will appear in many more simulations.  Thus at the beginning the policy in the initial tree uses this statistic the most, but as the number of simulations increases it switches to the less bias statistic where the move is first.  It’s a nice idea.  A strange thing about the random MC player is that better random players don’t always produce better total system performance.  This is kind of weird and doesn’t seem to be very well understood.  As a result, optimising the random player is a bit of a “dark art” that involves months of testing and fiddling around.