Taking a penalty, facing a goalkeeper trying to read you, which way do you kick? If you always aim for the same corner, the goalie will learn your preference and is more likely to make a save. Mix up where you shoot, you are more likely to score.

Many decisions work this way. When others are paying attention and adapting, predictability becomes a disadvantage. When predators hunt for food, a prey animal that changes direction unpredictably is harder for a predator to catch.

But this poses a problem for the brain. Normally, animals learn to repeat choices that have been rewarded. When that becomes a weakness, how does the brain stop reward from steering every choice?

New research from Dr Ann Duan’s group at the Sainsbury Wellcome Centre (SWC), published in two preprints, starts to tackle these questions on behavioural, computational, and neural levels. Led by Dr Joanna Aloor and Dr Tim Sit, the team studied mice learning to play a zero-sum game against a computer opponent. The algorithm looks for patterns in the animal's choices, so the best strategy is to be unpredictable. They found that the mice shifted from structured to more random choices as they learnt how to play. During the unpredictable choices, their cortex still registered whether each selection was rewarded, but that signal was less persistent and no longer predicted what the animal would do next.

In a companion study, the team studied two neuromodulatory systems that have long been implicated in reward-guided behaviours. They found that dopamine signals primarily tracked whether each choice paid off relative to expectation, while norepinephrine (noradrenaline) signals reflected broader context - the strategy that the animal was using and how good it was - and helped explain how variable the animals' choices were.

Studying unpredictable behaviour 

Understanding how the brain generates flexible behaviour while interacting with others is a central question for the Duan Lab.

“Many studies focus on animals learning a rule to get a reward. But in real life, a lot of the decisions we make end up needing some stochasticity to avoid getting stuck using strategies or choices that aren't beneficial to us. We want to know how the brain maximises reward in an environment involving other individuals,” says Dr Aloor. “To study that, we use competitive games”.

The mice learnt a game called ‘matching-pennies’, where they choose between two options, left or right. The animals are rewarded when a choice doesn’t match a computer opponent's prediction. The computer searches the mouse's recent choices for patterns, so the mouse benefits from making its choices unpredictable.

The team used widefield calcium imaging to record the activity of neurons across the surface of the cortex as mice went from novice to expert game players against the computer. During the unpredictable choices, neural representation about recent reward history was weaker than during predictable and structured choices. And while the signal of the current reward remained present, it became decoupled from the next choice as the animals learned to become “random”. In other words, to escape predictability, the brain needs to dissociate choice from the influence of immediate reward.

Different species, same unpredictability

Researchers from Johns Hopkins University, USA, shared a previously generated data set with the SWC team, from monkeys trained on the same matching pennies game.

“Both mice and monkeys end up using the stochastic strategy when playing against the computer opponent,” explains Dr Sit. “But we also see differences in how the different species behave. For example, within a session early during the game, mice are less likely to use the stochastic strategy compared to monkeys. We also compared the degree of randomness which monkeys and mice exhibit during the stochastic state. We find that even though monkeys are more random during the stochastic state compared to mice, they both achieve a similar reward rate.”

The team also noted that animals flexibly adapted to the competitive pressure of the opponent: when the computer opponent was turned off, the animals quickly went back to just choosing one side.

Together, these observations suggest that being random is costly – it is more cognitively demanding than making the same choice over and over again – something that is also seen in human studies. 

Dopamine and norepinephrine

In the complementary study, the team investigated the roles of two neuromodulators: dopamine, classically associated with learning from rewards, and norepinephrine, more commonly known as noradrenalin and associated with alertness, arousal and behavioural variability.

Consistent with its established role, they found that dopamine responses encoded a teaching signal to guide reward learning during the game. On the other hand, norepinephrine encoded both salient events and longer-term strategy-related information. It tracked things like average reward rate and the behavioural strategy an animal was using.

“Norepinephrine has surprisingly rich information about how well animals are doing and what they need to change to do better, all things important for shaping strategies – this finding is something that has long been hypothesised by theories,” says Dr Duan. 

The findings suggest that dopamine and norepinephrine support adaptive behaviour on complementary timescales. Dopamine is primarily linked to faster, reward-guided learning as actions are associated with outcomes, while norepinephrine carries broader information that helps shape longer-term strategy.

Next steps

Next directions by Dr. Sit and others include studying mice playing matching pennies against each other, rather than a computer. This will allow the team to ask questions about how animals balance being unpredictable with exploiting any patterns of an opponent; similar to the kind of dynamics in a poker game. They also want to explore how training history affects performance: does prior experience and curriculum shaping affect how animals play the game and how they adapt to different types of opponents? And finally, they are investigating how humans play competitive games, using pupil dilation as a proxy for norepinephrine-related variables. 

Understanding how the brain regulates behavioural flexibility in different environments could help explain why some people become stuck in rigid or repetitive patterns of behaviour.

“As neuroscientists, we are so often focused on trying to understand reward-guided behaviour and how we learn and exploit structures in the world, partly because strategic and unpredictable choices are very difficult to study,” says Dr Duan. “But I feel like we're on to something. It’s an unconventional way of looking at adaptive behaviours, and we hope that these studies help others think about them in a new way.”

Find out more

Read 'Behavioural and neural mechanisms for stochastic choices in mixed-strategy games' on bioRxiv

Read 'Distinct contributions of norepinephrine and dopamine to strategic decisions in a competitive multi-player game' on bioRxiv

Illustration credit: Bayes Bai