ResearchResearch note
Strategy or state?
How much of the predictive advantage of tiny recurrent networks over classical reinforcement-learning models comes from hidden engagement states, and how much from different learning rules?
- Status
- Planned
- Updated
- References
- 4 sources, listed below
P(engaged | choices so far)
Fig. 1 Inferring, trial by trial, whether a simulated agent is engaged. The line is the filtered posterior P(engaged | choices so far) of one simulated session, computed with the true parameters; the band shows where that posterior falls across replicate sessions with the same hidden states, and the shaded columns and the strip under the axis mark the true disengaged epochs. The two states differ only in inverse temperature (β = 3.0 engaged, β = 0.2 disengaged). The posterior averages 0.80 on engaged trials and 0.27 on disengaged trials, and falls below 0.5 in 2 of 2 disengaged epochs of at least 8 trials, a median of 2.5 trials after onset. The band, which holds 90% of 200 replicates, has a median width of 0.54.
Data table
| Trial | P(engaged) | 90% of replicates | True state | Choice | Reward |
|---|---|---|---|---|---|
| 10 | 0.98 | 0.81–0.98 | engaged | B | yes |
| 20 | 0.98 | 0.66–0.98 | engaged | B | yes |
| 30 | 0.61 | 0.07–0.96 | engaged | B | yes |
| 40 | 0.98 | 0.41–0.98 | engaged | B | yes |
| 50 | 0.86 | 0.58–0.97 | engaged | B | no |
| 60 | 0.97 | 0.56–0.98 | engaged | A | yes |
| 70 | 0.94 | 0.66–0.98 | engaged | A | yes |
| 80 | 0.96 | 0.73–0.98 | engaged | A | yes |
| 90 | 0.82 | 0.29–0.97 | engaged | A | yes |
| 100 | 0.65 | 0.47–0.97 | engaged | A | no |
| 110 | 0.76 | 0.47–0.98 | engaged | A | no |
| 120 | 0.85 | 0.67–0.98 | engaged | B | yes |
| 130 | 0.06 | 0.02–0.91 | disengaged | A | no |
| 140 | 0.17 | 0.06–0.81 | disengaged | A | yes |
| 150 | 0.58 | 0.25–0.96 | engaged | A | yes |
| 160 | 0.67 | 0.41–0.98 | engaged | A | yes |
| 170 | 0.95 | 0.58–0.98 | engaged | A | yes |
| 180 | 0.86 | 0.64–0.97 | engaged | A | no |
| 190 | 0.70 | 0.43–0.98 | engaged | B | yes |
| 200 | 0.92 | 0.50–0.98 | engaged | B | yes |
| 210 | 0.87 | 0.58–0.98 | engaged | B | no |
| 220 | 0.97 | 0.56–0.97 | engaged | A | yes |
| 230 | 0.47 | 0.03–0.92 | disengaged | A | yes |
| 240 | 0.01 | 0.01–0.81 | disengaged | B | yes |
| 250 | 0.01 | 0.01–0.59 | disengaged | B | yes |
| 260 | 0.07 | 0.05–0.72 | disengaged | A | no |
| 270 | 0.38 | 0.01–0.71 | disengaged | B | yes |
| 280 | 0.42 | 0.14–0.96 | engaged | B | no |
| 290 | 0.79 | 0.26–0.98 | engaged | B | yes |
| 300 | 0.65 | 0.51–0.97 | engaged | B | yes |
All trials, with the generator, seed and parameters: engagement-session.json
The question
Ji-An, Benna and Mattar (2025) showed that recurrent networks with only a handful of units predict the choices of animals and people better than classical reinforcement-learning models. A reply by Katahira argues, using simulations, that part of that advantage could come from something simpler: a participant who drifts between engaged and disengaged states. This note asks how large that part is on real data.
Why it matters
A better prediction is not an explanation. If hidden states account for the gap, then the networks are tracking engagement and the learning rule may be fine. If the gap persists, the classical models are missing structure in how learning itself changes. Either answer tells us something about what these networks have found.
Approach
The plan has three steps, none of them done yet.
- Write down the hypotheses, models and decision criteria before looking at any held-out data.
- Check on simulated data that the models can be told apart, so that a result is not an artefact of the fitting procedure.
- Compare, on the same splits as the original paper, a standard model with perseveration, the same model with a hidden Markov layer over choice randomness, and a tiny recurrent network.
I would start with one mouse task, and treat the human datasets as an extension if time allows. Reference papers, code and data are listed below.
What would count as an answer
- H1: the model with engagement states closes a substantial part of the gap. Here, "substantial" will be fixed in advance, with an interval from resampling animals.
- H0: the gap persists, which would point to learning structure the scalar model misses.
Both outcomes are reportable. A control where the learning rate, not the randomness, changes by state is there to tell the two stories apart.
Limits
Fig. 1 is a simulation of the kind of session the model describes, with an engaged and a disengaged stretch. It is not data, and it shows nothing about animals or people. Mouse datasets contain few animals, so any interval will be wide. A hidden-state model that fits well is not proof that the state exists. The original authors may also address the same question first.
Status
This work is planned, not started. I have a written protocol and a reading plan of the papers above. I have no results, and nothing is preregistered yet. This page will change when that does.
References
Ji-An, L., Benna, M. K., Mattar, M. G. (2025). Discovering cognitive strategies with tiny recurrent neural networks.
Nature · doi.org/10.1038/s41586-025-09142-4
Katahira, K. (2025). Recurrent neural network dynamics may not purely reflect cognitive strategies: A commentary on Ji-An et al. (Nature, 2025).
bioRxiv · doi.org/10.1101/2025.10.30.685524
Gillan, C. M. Characterizing a psychiatric symptom dimension related to deficits in goal-directed control.
OSF project (two-step task data) · osf.io/usdgt
Ji-An, L. (jil095). tinyRNN.
GitHub repository · github.com/jil095/tinyRNN
Next · Case study
Trust analytics for AI agents
When software agents act on their own, how do you tell from their behaviour whether they can still be trusted?
Got a question about it?
leyva.