But you think this of concrete advances in AGI research in general right?”

– well, it does depend upon whether the result is more or less likely to be used by a careful team of FAI developers, rather than by a misathropic team who don’t really care whether they kill the human race.

– correct.

– the funding agencies would have to be persuaded of the risks of AGI. Actually, beaurocrats might like the idea of having loads of control over academics, so it might be easier to get them on side.

And don’t worry about “paper clips”, entropy will take care of snuffing out the hubris of us demented monkeys.

But you think this of concrete advances in AGI research in general right?

“Incidentally, how likely do you think AGI academics are to agree to the idea of not publishing their research for safety reasons?”

Even if most did agree to this, information tends to get out, other people working in close areas discover the same things a few years later, alternative methods are found that do the same thing… and who would fund them if they don’t publish their research? Come to think of it, what if we all stopped telling each other about our research? 15 years from now somebody like Hutter out of the blue announces that he’s built a machine with clearly super human intelligence. And no, he’s still not going to tell you how he did it… for safety reasons. Does this really help from your perspective?

AGI is coming fast. I read this morning that IBM is already talking to astronomers about putting together an exa-flop machine for them before 2020, building on the work for their 20 peta-flop machine that’s coming out in the next couple of years.

Incidentally, how likely do you think AGI academics are to agree to the idea of not publishing their research for safety reasons?

There are some quite basic types of pattern learning that this design would struggle with due to the somewhat limited nature of CTW. For example, it can’t build proper hierarchies of abstraction about its environment and thus it can’t really do anything “deep”. Nevertheless, it would be interesting to see how this type of simple design tops out in practice as you add more computer power. That would help clarify the system’s most serious limitations and perhaps point to where more future research might be useful.

MC-AIXI is certainly no Skynet! But, at least to my mind, it looks like an interesting starting point for future work.

I think the ghosts in pac-man would disagree.

– why don’t we set up a SKYNET@HOME distributed uFAI initiative to find out?