How good is the situation you're currently in? Your brain is asking this question constantly, whether you realise it or not. The answer shapes how motivated you feel, what you expect to happen next, and how quickly you learn when things don't go as planned. 

To understand how the brain makes these calculations, Dr Christine Constantinople, Principal Investigator at NYU has developed a new task for rats that mimics the hidden complexity of the world: rewards are available, but the rules governing them are concealed. The rats have to figure out the structure of the task from experience alone. Training many animals on the same task allowed the team to build robust datasets, powerful enough to detect subtle differences in how individual decisions are made.

Dr Constantinople recently presented her work at SWC, and in this Q&A she discusses her latest findings.

What brought you to neuroscience research?

I accidentally discovered neuroscience as an undergrad. I was thinking about going into politics or government or public service. At NYU, students have to take two science courses, even for non-science majors, and so to satisfy that requirement, I took a course called the Brain and Behaviour, taught by Professor Mike Hawken. It just blew my mind. I approached Mike, and I asked if I could work in his lab. That was a transformative experience, and my first exposure to academic research. I just loved it. 

I feel like growing up, I was always taught that science is like a collection of facts. So it was amazing to understand that actually, it's a method for discovery and there are all these things we don't know. I couldn't believe all of the things that we still didn't know about the brain. 

What are the questions you are focusing on now in your research?

We are interested in how neural circuits perform computations for decision-making and cognition. We use a task for rats where they reveal how much they value different rewards based on how long they're willing to wait for them. It's modelled after an economic study in which people revealed how much they valued things based on how much they were willing to pay for them - the willingness to pay paradigm. 

We use the task as a vehicle to ask questions that we think are interesting. How do you reconcile dopamine's role in learning and moving? And how are complex signals in the frontal cortex read out by downstream circuits? 
 

Medium spiny neurons in the striatum. Image courtesy of Dr Carla Golden, NYU.

In your talk, you showed how the same dopamine signal can mean completely different things, and that was related to what acetylcholine was doing at the same moment. Can you explain those findings?

In the task, there are different events, and some of those events have cues that tell the animal about reward. We know from past literature that reward-related cues will elicit prediction error signals in dopamine neurons, and those are thought to be teaching signals. 

However, other task events elicit movement – the animals make stereotypical orienting movements. The fact that there are different reward and movement events at distinct points in time allowed us to separately relate dopamine to those different signals or events. 

At the reward-related cues, we did in fact see classic reward prediction error (RPE) signals and dopamine release. But at the movement events, we saw contralateral specific signals. This was a timed dopamine release, seen only when the movement was on the opposite side of the recording site. At those events, dopamine preceded the movement and predicted how fast or vigorous it was, suggesting that it is a movement invigorating signal. 

That poses a challenge for the neurons in the striatum that are interpreting that signal and turning it into an action. All they see is dopamine binding to their receptor. If that thing means different things at different points in time, they have to have a way of figuring that out. 

What we found is that local cholinergic interneurons in the striatum seem to have very different dynamics, with dips when dopamine is encoding RPEs versus bursts when encoding movement vigour. 

Our data suggest that acetylcholine dynamics gate dopamine and striatal function on a moment-by-moment basis.

How does the striatum know which mode it's supposed to be in? What's upstream of the acetylcholine? 

This is the question I always get, and we don’t know. That's the next thing to find out.

What were you most surprised by when you saw these results coming in?

I've been surprised by all of it. Everything. 

I've been really surprised that there are qualitatively different algorithms that drive different actions, the initiation time versus the wait time decision. When my former student showed me the plot that made us think that, I think I just stared at it in my office for 10 minutes and was like, what in the world could that mean? 

For the acetylcholine story, people had recorded from these cholinergic neurons before, including in monkeys during Pavlovian tasks. It has been shown that these cholinergic pauses are aligned with reward-predictive cues. That had caused people to speculate that the pauses were somehow important for gating plasticity. But no one had really shown it during behaviour. And no one had recorded in a rich enough task that also had movement and other elements.

I think the thing that I found most surprising was that it was the relative timing of the pause that was important. We have these two events where there's an RPE with a pause, but the rats only seem to learn from dopamine at one event and not the other. It reflects a difference on the order of tens of milliseconds. 

That is very surprising, I think, not just to me. I think that's quite novel, that hypothesis. I would have never predicted that.

You also presented your findings on the specific projections from orbitofrontal cortex (OFC) into the striatum that encode beliefs about hidden states. Could you explain a bit about that?

I would describe it as the neurons that project to the striatum are encoding the evidence to update beliefs, firing more strongly when observations point toward a high reward state. 

There are a lot of theories and work on how beliefs are represented in the brain, but mostly for perceptual stimuli or things that physically exist in the world. 

Our results are interesting because there are interpretable single neuron responses that seem to reflect these cognitive variables in their firing rates. Not over a physical thing, but over abstract, hidden states of the environment. 

How do these two findings, that acetylcholine gates dopamine and how the OFC updates beliefs, connect? 

We think that these are separate systems. 

But, there is one way they connect – the dynamic learning rates. We have characterised dynamic learning rates in the rats, which are quite apparent in the behaviour. I think part of why it's so clear and easy to see them is because the task doesn’t give them a binary choice, but a continuous behavioural variable that reflects state values. It makes it really obvious that they learn faster when the block first changes, and then later they are much slower to update their behaviour. 

It turns out that the rats update the speed of their learning based on whether their beliefs are changing. So if my beliefs are changing a lot, then maybe the world has changed, and maybe I should pay more attention to new observations. But if my beliefs are stable, then the world is pretty static and so I should not update my value estimates; I have a stable representation. 

So the OFC belief system is determining how they increase or decrease the learning rate. But so far, we haven’t found the neural substrate of that. It’s not the OFC projections to the striatum. It’s not dopamine. So there is a mechanism yet to be identified, and that's something we're really interested in currently exploring.

What's the next piece of the puzzle?

How are beliefs maintained across trial timescales? Because we're talking about, in some cases, minutes. That's one thing that we're working on. 

The circuit mechanism of dynamic learning rates is another one. I'm also really interested in understanding how the animals condition their behavioural policy, which is how long are they willing to wait, on their abstract beliefs. How does the belief actually impact the decision at the circuit level?
 

About Dr Christine Constantinople

Christine graduated from New York University with a B.S. in Neural Science in 2008, and earned her PhD in Neurobiology and Behavior in Randy Bruno’s lab at Columbia University in 2013.  She then completed a postdoctoral fellowship at the Princeton Neuroscience Institute with David Tank and Carlos Brody. She joined NYU as an Assistant Professor in the Center for Neural Science in January 2019.