vetta project

Intelligence is the general ability to achieve goals. When an intelligent agent tries to achieve a goal, two factors matter: Firstly how many resources it can bring to bear on the problem, and secondly, how effectively it can use these resources. Thus, if a very powerful agent were to try to achieve some goal to the absolute maximal extent, and with the maximal probability of success, it would try to use the maximal amount of resources in the maximally effective way.

This is all very good for the agent achieving its goal. The problem for us is that we also need resources. Air, water, food, electricity, even the physical material that makes up our bodies are all resources. This puts us in direct competition with the intelligent agent. If the agent had vastly super human intelligence, its quest for absolute resource control would be a direct threat to our existence.

Unfortunately, just about any goal, when taken to the absolute limit, seems to have the above resource competition problem. Thus a trick to building a safe super intelligence would be to require that its goals always consist of two parts: Firstly a specification of what is to be achieved, and secondly a specification of what resources can be used. This forces the agent to solve the problem in a clever and efficient way, rather than by expanding its control of resources.

I think you could formally define a version of AIXI in this way, to at least give it an elementary degree of safety. Of course such a “FAIXI” would still be very powerful and thus dangerous in other ways. For example, there’s still nothing to stop humans willingly asking the FAIXI to do something which, in hindsight, is a rather stupid idea.