So what does the brain do? For over a decade it has been known that temporal difference learning, a type of model-free RL algorithm, appears to explain the activity of dopamine neurons and their dorsolateral striatal projections. It has also been observed that parts of the prefrontal cortex appear to implement some kind of model-based RL algorithm. Mammalian brains, then, appear to get the best of worlds by having model-free and model-based RL algorithms and then choosing which to use on the fly.  Pretty clever huh?

A key question then is how this choice is made. I recently read Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control from Nature Neuroscience, by Nathaniel Daw, Yael Niv and Peter Dayan. They suggest that the brain may be using some kind of Bayesian principle based on the uncertainty estimates generated by each system. They implement such a system and show how it can explain a range of experimental data from animal studies.

This research is now four years old and there is plenty of significant research in the area that follows it — an “embarrassment of riches” as Dayan recently described it.  Nevertheless, I think it’s a good example of the kind of crossover between machine learning and neuroscience that is starting to take place.  Indeed, the more neuroscientists learn about the brain, the more it starts to look like the rough outline of an AGI design.