It would be nice if the machine had an architecture that allowed it to update itself in this manner – but that is probably not essential.

Enough about the reference machine for now. I do think we will figure out this issue without too much trouble.

If you want a system to perform well from the start, it needs to know something about the world it has to interact with. Either you program that in, or you get it to learn this from some data. What I propose is the latter, and indeed it easily fits with the current theoretical framework by simply conditioning the measure.

What more could you want? That the system magically knows something of the structure of the world without ever being told this or having an opportunity to learn it?

Wiring a reference machine into a machine intelligence in a loction where it can’t be updated based on information from its own inputs would seem to be fairly sucky.

Subsequently calling it “the most intelligent unbiased agent possible” seems to be an overblown marketing claim to me.

http://www.paul-almond.com/WhatIsALowLevelLanguage.htm

The situation seems to me to be that those advocating using the carrot and the stick on their machines may well eventually face some problems when the machine becomes potent enough to take away both the stick and all the carrots. A proposed solution – advocated by Yudkowsky and Omohundro – is to not use the carrot and the stick approach in the first place. Essentially, I approve of their proposed resolution – though perhaps there are other ones that would work as well. I think that critics should present criticisms next. “Prove it” is a kind of criticism – but I don’t find it terribly compelling.

I’ll believe this when somebody formally proves it. This question is too important and subtle for intuitive reasoning!

You seem to be asking: why not include physical law among the inputs – rather than building it into the machine?

You could do that. Or you could just feed your experimental data straight in and let the Solomonoff machine decide when to effectively switch reference machine. Anyway, I hope you can see that trying to empirically work out the right reference machine doesn’t add much. In the case of universal intelligence perhaps the thing to do would be to do something like condition the complexity measure on a bunch of data observed from the real world. Marcus and I have discussed this, but I don’t think the idea appears in any of our published work.

The wirehead problem is thought by many to be a soluble problem. The solution typically involves building the goal into the intelligent agent. Making the agent value it’s own pleasure – and then using the carrot and the stick on it – is an approach with problems – since it allows the agent to form its *own* representation of its goals – and if the agent comes to believe it’s purpose in life it finding happiness, it may well turn into a Buddhist – which would mean problems for everyone else.

Since AIXI is a “carrot and stick” approach, it seems especially prone to this problem.

Re: Solomonoff induction

You seem to be asking: why not include physical law among the inputs – rather than building it into the machine?

That seems like an equivalent approach to me – like using the TM to simulate some different reference machine.

The machine should know something about physics by the time it is asked about what is simple, and that is expecially important if physics is complicated.

I agree that there is an implementation problem for powerful agents in certain types of environments, including, seemingly, the real world. I also agree that this is a serious problem, potentially a very serious problem for humanity. What I don’t understand is why this has all that much to do with AIXI specifically. To me it seems like a problem with powerful optimization processes in general when they operate in environments that allow a high degree of manipulation.

it doesn’t help us choose between them.

So how are you going to choose between them? Empirically? Well, if you’re a Bayesian you’re going to have to start with a prior distribution over the different reference machines, then you’re going to go and take some measurements about the world, then you’re going to see how well each of the various predictions made under each of the reference machines turned out… and if you get enough evidence that F77 is the right reference machine you’re done. You can now use this as your reference machine as it’s a good fit with the real universe. If I’ve understood you correctly, this seems to be what you’re suggesting?

But wait a moment. What prior did you use? And more importantly, what if we didn’t bother with all this and just fed our empirical data into the Solomonoff induction machine with the initial reference machine? Hopefully you can see that the Solomonoff machine internally works out when to switch the effective reference machine to fit the observed universe. It’s already implementing the correct Bayesian procedure to do the above…

You seem to accept there is a problem in this area in this post

The post was about universal intelligence. For UI there is a more serious problem with reference machines. With Solomonoff induction, however, things work out much better.

The issue here is that you have an implementation problem. If the agent thinks what it is maximising is its reward signal, then you can *try* and fix the reward signal to represent the number of grandchildren – but if you are dealing with a superintelligent agent, then it will attempt to change the reward signal so that it says “a quazillion”. At this stage, it is you vs the superintelligent agent – a recipe for problems.

Classically the solution is to ensure the agent believes its own goals to be maximising the number of grandchildren – and not maximising its reward signal. However, that involves engineering the agent’s beliefs about its own goals somehow – not a trivial exercise for an AIXI agent.

If you want empirical evidence as you don’t trust proofs, then we are at an impasse.

This seems like an empirical issue to me. Occam’s razor is a property of the universe – i.e. you can imagine worlds where it is not useful. However, I am quite happy to accept mathematics experiments as empirical evidence. Maths is a property of our universe too.

CAs, UTMs, and FORTRAN can simulate each other with relatively small compilers. That is nice – but it doesn’t help us choose between them. I don’t think it necessarily implies that the universe uses a machine that is possible to concisely simulate with a UTM as a basis for Occam’s razor either. We do have evidence for that – the evidence that these systems act like Occam’s razor does – but the issue has not been explored sufficiently for there to be a high level of confidence.

Anyway, I don’t want to rattle on endlessly about the reference machine. You seem to accept there is a problem in this area in this post – and I have described this problem as not being very serious.

The main issue I see is the wirehead problem. After that, the possibility of the agent eating its own brains for lunch.

Ok, so then fix the reward signal to be the number of grandchildren you have. Wireheading is then no longer possible. In the examples above the problem is that your reward signal doesn’t correctly represent the goal and so at some point they can diverge.

Evidence of superiority would consist of comparing the serial TMs with competitors – such as, cellular automata, FORTRAN-77 and so on – and demonstrating their superiority empirically.

If you want empirical evidence as you don’t trust proofs, then we are at an impasse. If you do believe in math then CAs, F77 etc. all have fairly small compiler constants (you could even compute up upper bounds on these constants if you really wanted to) and so using a small UTM will always work essentially as well as these alternatives on anything but small amounts of data.

It *is* flimsy evidence for the superiority of Solomonoff induction. Barely evidence at all, I would say. Evidence of superiority would consist of comparing the serial TMs with competitors – such as, cellular automata, FORTRAN-77 and so on – and demonstrating their superiority empirically.

Where messing with your reward signals gets in the way of attaining your goals – usually.

*If* your goal is taken to be the usual one that organisms have – of having many grandchildren – then taking drugs is a form of wireheading, *if* it has a negative effect on that goal – and the parties are also a form of wireheading, *if* they interfere with that goal.

AFAICS, Hutter offered no wirehead solution for the case where humans control the reward signal.

His solution for freely-evolving agents this depended on the type of temporal discounting used – and the severity of the wirehead catastrophe. In that case, my concern is not so much that agents will overdose and kill themselves, but rather that they will become addicts who can still succeed in living some kind of life, but fail to realise their potential because of their addiction.

As I described to you when you visited, I think the right way to do this is to break the problems up into a sequence of levels along the lines of what Valentin Turchin describes in “The Phenomenon of Science”. Mix that up with a performance measure closer to pure universal intelligence with a low complexity bound as I think an AGI should have at least a base level of universal intelligence.

Of course, approaching the AGI problem by drawing on two sources as “old school” as Solomonoff induction and a cybernetics classic… at least demonstrates how retro-cool this idea is