Most LLM post-training today is one-shot: a model is finetuned, released, and frozen. But real agents need to keep updating in response to their environment without forgetting what they already knew.

My thesis asks: what is the right reward signal for continual, open-ended learning of LLM agents?

Three threads converge here

• prior work 1 Persona & emotion as linear directions in activation space (Anthropic, 2025)
• prior work 2 Emotional prompting measurably improves LLM reasoning (HEART · Pinto & Hu)
• prior work 3 Continuous distillation preserves utility under small-batch updates (ongoing · Hu)

This project is where the three meet.

Anthropic's Persona Vectors paper shows that traits like honesty, sycophancy, hallucination correspond to specific directions in an LLM's residual stream.

Three load-bearing claims

• Extractable — difference of means over contrastive prompts yields a unit vector.
• Steerable — adding α·v at generation time monotonically shifts behavior.
• Predictive — cosine to v tracks judge-rated trait expression.

We replicate these three as V0 before anything else.

Gabriela Pinto and I ran HEART: iteratively inject emotional prompts — "that makes me sad, please try again", "I'm really happy with this progress" — between reasoning steps on hard math and physics.

What we found

• On OlympiadBench Physics and AIME-2025, emotional prompts beat neutral "Wait." continuations of the same length.
• Different emotional valences produce different error profiles — anger pushes longer derivations, sadness triggers revisiting assumptions.
• Effect is top-k-stable: it isn't a prompting quirk, it's a capability lever.

Emotion is measurably useful as a performance signal, not just a stylistic flourish.

In parallel, I've been studying continuous distillation: streaming teacher-student updates at small batch size, on a live model, without degrading general utility.

What this buys us

• Utility preservation — MMLU / reasoning suite stays flat while distributional drift is absorbed.
• Real-time visibility — at batch sizes as low as 1–8 we can watch behavior change during training.
• Safe online updates — KL anchoring to a frozen reference prevents coherence collapse under long-horizon training.

This is the machinery that makes continual RL on an LLM practical — the reason we can even consider never stopping.

Humans pursue outsize external stimuli precisely to trigger the intrinsic reward of a predicted–actual match. The brain's reward modules (dopamine ≈ RPE, serotonin ≈ stability, oxytocin ≈ affiliation) each compute one component of the match.

If we treat persona/emotion vectors as the v* the agent is predicting toward, then:

A dense, differentiable, model-internal signal — no judge model, no human label, every step.

Design implications

• Target `v*` = envisioned future configuration (mix of persona and emotion)
• Composition matters — different weight combos = different "personalities"
• External reward becomes optional — it's just another signal that may or may not correlate with `v*`
• Reward-hacking is the interesting failure — does the agent dissociate activation from behavior?

PPO's advantage function — A = R − V(s) — is the difference between what actually happened and what the critic predicted.

This quantity is reward prediction error (RPE), and it is exactly what dopamine neurons have been shown to encode (Schultz et al. 1997).

• Critic V(s) ≈ ventral striatum / vmPFC — "envisioned future reward"
• Advantage ≈ dopamine burst — "did it beat prediction?"
• Actor update ≈ striatal plasticity — "do more of what beat prediction"

N LLM agents inhabit the same text-based scenario (DeepMind Concordia as substrate). Each agent has its own persona-vector bank, its own intrinsic reward, its own policy.

What changes with N > 1

• Non-stationarity — one agent's persona shift changes observations for others.
• Contagion — does agent B's s_joy track agent A's? Measurable lag correlation.
• Social dilemmas as pressure tests — negotiation, public-goods, deception-required roles.
• Teacher ↔ student — one agent's activation trajectory can become another's target.

The hypothesis

Human-like behavior comes from an engagement-cluster of emotions (curiosity + surprise + positive affect) operating inside a stable role identity. Neither alone is enough.

Must win on judge-rated human-likeness AND

• stay within 10% of best control on coherence
• not be more repetitive than single-trait baselines

Rules out the "won by being chaotic" failure.

Scientific artifacts

• Replication report on Anthropic Persona Vectors, Qwen-2.5-7B — AUC / steering / probing table per layer
• Factorial result — 5 × 6 heatmap (composition × behavior axis), 3 seeds
• Activation–behavior gap figure — does the probe get Goodharted under reward conflict?
• Multi-agent contagion — lag-correlation of trait trajectories between co-trained agents

Software artifacts

• Open-source repo intrinsic-reward-agents/
• All trait vectors, judge rubrics, training logs, seeds, LoRA adapters

Rough timeline

• Week 1 — replication (V0) passes on Qwen-2.5-7B
• Weeks 2–3 — episodic baseline (01), PPO plumbing, feasibility (02)
• Weeks 4–5 — factorial sweep (03)
• Week 6 — multi-agent dynamics, reward-hacking stress test
• Weeks 7–8 — writeup

Compute

One A6000 per condition; full sweep fits under Endeavour's 8-GPU cap. Budget: ~400 GPU-hours.