Finance models, like all mathematical models, suffer from the following problem: if you don’t make the initial assumptions simple and easy to work with the theoretical analysis that follows is too difficult to manage. In finance this usually translates into assuming that investors are fully informed, completely rational and are just out to maximise their expected future utility. You also tend to assume that the returns on risky assets, for example stocks, follow geometric Brownian motion in continuous time, or have returns that are log-normal distributed when working in discrete time. These assumptions are somewhat close to reality, but simple enough to permit theoretical analysis.

So what does the analysis say? Among other things, it says that people should be investing a large proportion of their wealth into stocks. In reality, however, most people don’t own any stock, and most of those who do don’t have a particularly large proportion of their wealth in stocks unless they are very wealthy. Perhaps this is ok in a prescriptive sense, i.e. telling you that you really should consider owning more stock.  However, as a descriptive model of investors, i.e. describing what investors do and why, they seriously fail. Playing with the parameters does not save you: in order to get people holding so little stock you have to push the level of people’s risk aversion up far beyond the range of values that have been empirically estimated. Thus, if our theoretical analysis is correct and produces the wrong answers, it must be that our basic assumptions were wrong.

This isn’t really news, indeed it’s well known that people are not rational expected utility maximisers.  When we have to make decisions, all sorts of cognitive biases and distortions come into play.  Seminal work in this area was done by Kahneman and Tversky.  They produced a model of human decision making known as prospect theory, work that Kahneman later won a Noble prize for (sadly Tversky died some years before the award). Due to some technical problems, this was later refined to produce cumulative prospect theory, which I will now very superficially describe.

Cumulative prospect theory consists of five main components:

1) narrow framing. What this means is that when you have to make a decision, say to invest in a stock or to make a gamble, you tend to act as though this decision was being taken in isolation. This makes sense given that making an optimal decision with respect to all the risks you are facing in your entire life, the strictly rational thing to do, is often too complex.

2) reference return. Imagine that the market went up 20% and you made a 10% return on your chosen investments. You probably wouldn’t be happy with that. On the other hand if the market fell 10% and you made a 5% gain you would be pretty pleased with yourself. What this shows is that the utility that you get from an investment is not simply a function of the actual return, but also depends on how that return compares to some mental point of reference that you have.

3) loss aversion. For most people, the pain of a losing $100 is about twice the magnitude of the pleasure of gaining $100. Clearly, this distorts people’s decision making. For example, people may pass up opportunities to make a gain in order to avoid a loss that is comparatively small.

4) probability weighting. People tend to distort probabilities when they make decisions. They act as though low probability events, say winning the lottery or getting a rare disease, is more likely than it really is. Conversely, they act as though quite likely events are slightly less likely than they really are.

5) curved value function. If you have a guaranteed gain of $100,000 or a very likely gain of $110,000, which would you take? Most would take the first option. If you had a guaranteed loss of $100,000 or a likely but not for sure loss of $110,000 what would you take? Most would take the second option. In other words, people are risk averse with respect to gains, but become much more willing to take risks when facing a potential loss.

Given all these deviations from being a simple rational expected utility maximiser, it’s perhaps no surprise that financial models that assume expected utility maximisation produce results that don’t match real investor behaviour. The problem, as I mentioned earlier, is that if you add a little more complexity to your initial assumptions you tend to end up with a model that is impossible to theoretically analyse.

In step Barberis and Huang. In a great paper that these two recently produced, they managed to incorporate the first three aspects of prospect theory into an investor model and still come out with an analysis that is tractable for portfolio choice (what do investors do) and asset pricing (what do markets made up of these investors do). If you find this stuff interesting and can handle mathematical finance at a research level I recommend that you check it out.  That said, the Barberis and Huang paper does have some draw backs. Firstly, it doesn’t include probability weighting. Secondly, it doesn’t include a curved value function. And thirdly, when you put in realistic stock returns their model doesn’t help explain things like the lack of stock market participation that I mentioned earlier, in fact it actually makes it worse.

In step De Giorgi and myself. Having recently completed my PhD thesis with Marcus Hutter, I looked at these investors and thought, “Hey, they’re just like reinforcement learning agents. No big deal. If I want to know what investors with probability weighting and a curved value function do, I can just brute force compute their optimal policy by writing down their Bellman equation and using dynamic programming. Easy!” It was a mystery to me why, seemingly, nobody else was doing that. So off I went to build software to do just this, starting with a simple Merton model…

A month went by. Another month went by. I was having all kinds of accuracy and stability problems and Enrico was starting to look a bit worried. More time went by. I too was starting to sweat. Eventually, thank goodness, I managed to understand my problems and worked out how to fix them. So I then attempted a more complex model. After some more weeks of struggle I managed to get that to work as well. I was starting to get the hang of this and know the key tricks needed to make it work.  I then recomputed the Barberis and Huang results, I added probability weighting, and finally I also added a curved value function. I seriously had no idea how hard this was going to be when I started. Somehow my ignorance combined with stubbornness not to fail eventually produced success: a general purpose simulation engine that can tell me how just about any kind of consistent investor is going to behave, including ones that use a full version of cumulative prospect theory.

At this point De Giorgi made a simple but important observation: if an investor uses probability weighting, that means that they are going to inflate the importance of low probability events when making decisions, which is to say that they are going to be more sensitive to the tails of the stock’s returns distribution. Typically we assume that returns have a log-normal distribution, for tractability reasons. However, a log-normal distribution has a positive skew, indeed, the distribution’s negative tail ends at 0. Real stock returns, on the other hand, have a negative skew: everybody knows that sudden falls in a stock’s price occur more often than equally large and sudden rises.  Thus, if we are going to put probability weighting in, we really need to get the skew right as the tails of the distribution are likely to be important.

What we did was to take S & P 500 data from the last 60 or so years and fit a skew-normal distribution to the observed returns. A skew-normal distribution is basically just a generalisation of the normal distribution that has an extra parameter that allows you to control the skew. As expected, when we fitted a skew-normal distribution to real data it did indeed come back with a negative skew. When we fired up my simulator and gave this distribution to an investor that had probability weighting: the investor took one look at that scary negative tail and didn’t want to invest in the stock. This is exactly what the model should predict.  In short, we took realistic stock returns, and presented this to an investor with a realistic decision making process complete with a bunch of parameters that have been empirically estimated by others in previous work, and what we got out the other end was realistic investor behaviour!

Following this, I went back to look at the Barberis and Huang model and how they computed investor behaviour. Rather than my brute force approach, they had a much more elegant technique. It didn’t take me long to realise that their method could be extended to include probability weighting, giving me a fast way to compute the behaviour of these investors. Attempts to include a curved value function into their method failed, for that we had to continue to use my brute force simulator. However, analysis of these investors, both theoretically and via simulation, produced another interesting effect: wealthy investors act as if they are not narrow framing their investment decisions as much as less wealthy investors, and thus the effect above applies more to poorer investors rather than wealth ones. Which is to say that when we take a full cumulative prospect theory model of investors and realistic stock returns, what we see is that less wealthy people don’t hold much stock, while wealthy people tend to put a significant proportion of their wealth into stocks. Again these results match reality as various studies show that stock market participation increases rapidly as a function of the individual’s wealth.  We then extended this analysis to a market consisting of investors with probability weighting and skewed asset returns and found it easy to obtain realistic risk free rates and market equity premiums.

I guess the moral to this story is: if you want to get realistic answers out of models of investors, you probably need to take account for the ways in which they deviate from a strictly expectations maximising agent.  The hard part, with much help from the prior work of Barberis and Huang, was to come up with ways to make the resulting analysis theoretically and computationally tractable.

I’d like to thank Enrico De Giorgi for being a great supervisor and collegue during this work, and the Swiss Finance Institute in Lugano (via De Giorgi) and St. Gallen (thanks to Fabio Trojani) for my funding, and finally the people of Switzerland whose taxes ultimately paid for all this — I can only hope that you view our efforts as having been worthy of your generosity.

One point of contention concerns whether efficiency should be part of the concept of intelligence. Hutter and I have taken the position that it should not, and I continue to think that this is the right way to go. As what we are debating is a definition, it’s hard to claim that one of these two possibilities is in some absolute sense “correct”. All we can argue is that one is more in line with what is typically meant when the word is used. Looking over the many definitions of intelligence that we have collected, in the vast majority the internal computational cost of the agent is not taken into account. Thus, among professional definitions the pattern is clear.

What about naive usage of the concept then? I think it’s the same. Imagine that you discovered that some friend of yours, who seemed completely normal, actually had only half a brain. Due to his smaller brain making more efficient use of its resources it wasn’t obvious from the outside that anything strange was going on, until a brain scan revealed this. Would you now say that your friend was twice as intelligent as you had previous thought?  Consider a more futuristic hypothetical. It may well be the case that intelligence (in my sense) scales in a sub-linear way with respect to computational resources. Indeed, many learning, modelling and prediction algorithms scale in a sub-linear way with respect to computational resources. This raises the possibility that after a singularity the world could be run by a computationally vast and phenomenally smart machine which, in an efficiency sense, has significantly sub-human “intelligence”.

Why then do some people feel the need to define intelligence with respect to computational efficiency rather than purely in terms of decision making performance? The reason, I believe, is that at some level they recognise that if the definition of intelligence does not take efficiency into account, then intelligence is not the right metric for their research. And they are right! An intelligent machine will consist of some impressive hardware combined with clever algorithms that can efficiently turn the computational power of that hardware into intelligence. The job of the hardware people is to come up with more and more powerful hardware, and they are clearly doing a wonderful job of this. The job of the AI people is the second part: to come up with the most efficient way to convert computation into intelligence. If you want to build a metric for your AGI algorithm research, a measure of the efficiency of intelligence is what you really need — let the hardware people take care of the other side of things.  If we both do our jobs well, the end result will be a lot of machine intelligence.

Another point that often comes up concerns whether universal intelligence is in fact too broad. For practical AGI researchers, the answer is probably yes. More specifically, if you want to produce a system with a somewhat human like intelligence, and that is optimised for the universe we live in, rather than very general semi-computable probabilistic environments, then yes, you will want a more focused kind of “intelligence” than universal intelligence. Still very broad, sure, but your target is not quite as extremely general.  Why then didn’t we try to do this? Answer: one step at a time! Constructing a practical general intelligence measure for AGI is not easy, and almost certainly too big a job for a PhD research project. Thus my goal with universal intelligence was to try to capture the concept in the cleanest, most formal, and most general sense possible in the hope that this might provide some theoretical foundation for later work on practical tests of AGI intelligence. If that’s your goal, then go for it, and I hope that my theoretical work is of some use to you.

A related point to the one above concerns the sensitivity of the universal intelligence measure to the choice of the reference machine. In some situations, for example with Solomonoff induction, the choice of the reference machine doesn’t matter too much. With universal intelligence it doesn’t work out as well. The invariance theorem for Kolmogorov complexity provides some protection, but it’s not enough. The usual trick in Kolmogorov complexity is to then minimise the state-symbol complexity of the reference machine. As there exist very simple UTMs, and there aren’t many of them, that succeeds in locking things down fairly tightly. When you read criticisms of Kolmogorov complexity based work that show strange results by varying the reference machine, have a look to see if they limit themselves to minimal reference machines. Almost always they completely ignore this constraint, because with it their criticisms would no longer work, or at least be much weaker. I sometimes wonder why this happens and I suspect that part of the reason might be the way in which Kolmogorov complexity is taught. What we should do is to always start by squeezing as much complexity out of the reference machine as possible in order to ensure that the measured complexity of an object is, to the greatest degree possible, a property of that object and not our reference machine. Only then should we mention that there is this invariance theorem that is often useful to prove things.  And then point out that for various asymptotic results, such as the randomness of infinite sequences, the reference machine is completely irrelevant.

Anyway, in the case of universal intelligence, the reference machine in effect defines how we weight the agent’s performance in different environments when trying to compute an overall score. As such, if you want to weight the environments in a way that somehow reflects the universe we live in, you might prefer to select a reference machine that is not an absolutely minimal one. This makes some sense, and it certainly seems that some UTMs are somehow more “natural” than others. Various theory people have tried to go down this path over the years, and so far not much as come of it, as least far as I’m aware. A word of warning then: if you want to solve this problem in a theoretically tidy way, be careful, this appears to be a problem that initially seems easier than it really is. That said, good luck, for if you do succeed such a result could be extremely useful.  Failing that, one clever way to further reduce the test’s sensitivity to the choice of minimal reference machine was suggested to me by Peter Dayan. The idea is to allow the agent to maintain state between different test environments. This would mitigate any bias introduced as intelligent agents would then be able to adapt to the test’s reference machine as different environments were randomly sampled.  In other words, it can learn any bias that the refernce machine choice introduces to the distribution over environments and then compensate for this.

Another thing that sometimes comes up is people worrying about the fact that the environment defines the reward. The objection mostly seems to come from cognitive science people, rather than math people and so I think it’s a cultural problem. When we say “environment”, and we stick the reward generation mechanism in there, we aren’t claiming that real environments are what define an agent’s rewards: it’s just a mathematical convenience that makes it easier to mix over all the different kinds of problems and goals. You could separate them out if you wanted to, as I note in my thesis. It adds a few more terms to the equations, but because we mix over the whole space in the end doesn’t make much difference. Also, when we say “agent”, we are using the word in the sense that reinforcement learning people use it (see the introductory parts of the Sutton and Barto book for an explanation of this point). When we say “agent”, in non-RL speak we really mean just an optimisation and decision making part of a real agent. I talk a bit about this in my thesis, but I didn’t have space for these finer points in the Benelearn paper, and our target audience was more RL people then anyway.

One objection that I’ve heard a few times is that a key property of intelligent systems is their ability to choose their own goals. It certainly seems that we have this ability while an agent, as defined in our framework, does not.  Ask yourself this: how do you choose your goals? One possibility is that you do it in a deterministic way, that is, somewhere in your brain an algorithm runs that looks at all sorts of information and spits out a decision as to what your goal is going to be. For example, you might read some holy book or philosophical text, process the information therein, and then decide, using this algorithm in your brain, to base your life on following these principles. In this way you have chosen some of your goals. If you think about it, an agent in our framework can do the same thing: its goal might be to read in some information from its environment and then take this to be a function which it then tries to optimise.  Both you and the agent have an underlying goal that generates and selects new sub-goals, perhaps with input from the environment, and then follows these.  When you choose one goal over another, it is this choosing mechanism that is you true underlying goal.  Adding randomness doesn’t make any fundamental different to this. Even if your goal is to think up a random goal and then follow it, one may characterise your underlying goal to be just that: to generate and then follow a randomly generated goal.  Admittedly, your real underlying goal is almost certainly very complex and messy; even if we could fully observe your brain, it might well be next to impossible to extract out a succinct description of your underlying goal.  But that’s not the point: the important point here is that the framework we define is not as limiting as it might first appear to be.

Related to the above are occasional criticisms that our approach to intelligence is wrong because it takes a behavioural stance, and behaviourism has been thoroughly debunked.  These people are really missing the point.  Our goal here is not to explain how human intelligence works. Or how any other intelligent system works, for that matter.  We take this outside view of things because what we want is a measure that applies across many different kinds of systems with potentially radically different internals.  It’s not because we’re closet behaviourists.  Indeed, I work at a theoretical neuroscience institute because I think this will give me pointers on how to design an AGI.  That makes me the inverse of a behaviourist and their incomprehensible-black-box view of the brain!  Put it this way: if I want to measure how fast your car is, I don’t really care how it works.  But if I want to understand why your car is so fast, then I’ll pop the hood.

The final thing I’d like to respond to is the objection that a definition of intelligence should be computable. I wrote a response to this based on the definition of randomness in my thesis. See the bottom of page 77 though to the end of the section on the next page. In short: any definition of randomness that isn’t incomputable would be provably flawed. Sometimes then it is best to define a concept in an ideal and incomputable way, and accept that our ability to measure it in practice is always going to be limited.