Context-distillation vs. in-context

Qwen3.5-9B · 60 SDC instances · 3 PEFT methods · Generated 2026-05-18T05:03:17 UTC.

Experiments progress

Full target grid: 6 base models × 60 SDC instances × 4 fine-tuning methods (plus base/in-context evals). Current status:
Modelbase / in-context Full-FT · CE Full-FT · KL LoRA · CE LoRA · KL Prompt · CE Prompt · KL Prefix · CE Prefix · KL LoRA←Prompt · CE LoRA←Prompt · KL Full←Prompt · CE Full←Prompt · KL LoRA←Prefix · CE LoRA←Prefix · KL Full←Prefix · CE Full←Prefix · KL
Qwen3.5-9B
Qwen/Qwen3.5-9B		✅ in every Phase-5 eval ✅ 60/60 · p4r 57 ✅ 60/60 · p4r 58 ✅ 60/60 · p4r 60 ✅ 60/60 · p4r 60 ✅ 60/60 · p4r 61 ✅ 60/60 · p4r 60 — — ⏳ 1/60 · p4r 1 — — — — — — —
Qwen3.5-4B
Qwen/Qwen3.5-4B		— — — — — — — — — — — — — — — — —
Llama-3.1-8B
meta-llama/Llama-3.1-8B-Instruct		— — — — — — — — — — — — — — — — —
Llama-3.2-3B
meta-llama/Llama-3.2-3B-Instruct		— — — — — — — — — — — — — — — — —
Gemma-3-12B
google/gemma-3-12b		— — — — — — — — — — — — — — — — —
Mistral-Nemo-12B
mistralai/Mistral-Nemo-Instruct-2407		— — — — — — — — — — — — — — — — —
Cell notation: ✅ N/60 = pipeline complete for N instances; ⏳ N/60 = in progress; p4r N = residual-stream analysis (Phase 4r) done for N of those. base/in-context: produced as part of each Phase-5 MMLU-Pro eval per instance.

[Fig·M] Memory-flow analogy and where each store lives in the transformer

Sensory / working context C · KV cache Short-term prompt / prefix Long-term model weights ΔW(C) prompt prefix LoRA Full FT [C] context soft prompt (prompt / prefix tuning) [Q] query embeddings layer 1 model weights KV cache trainable KV cache layer 2 model weights KV cache trainable KV cache (× 32 layers in Qwen3.5-9B) lm_head

2. The in-context data — 60 SDC instances

60 instances arranged along two cross-cutting axes: (i) 3 high-level classes × 10 subclasses × 3 levels × 2 variants per (class, subclass, level), and (ii) a binary grouping by whether the context specifies an output form (Group A — 6 subclasses, 36 instances) or only provides reference content (Group B — 4 subclasses, 24 instances). The output-form/reference axis predicts Phase-3 exact-match better than the 3-class one. Each cell below lists 2 instances. A level sweeps the strength of the contextual signal so we can see how SDC fidelity scales with how much C asks of the model. High-intensity (strong / long / more) → low-intensity (weak / short / less) is encoded by color.
GroupSubclassHigh intensityMediumLow intensity
A · Output-form-specifying1_1 — Output format strong · json_strict, xml_strict (× 2) medium · haiku, sonnet (× 2) weak · concise, max_3_sentences (× 2)
A · Output-form-specifying1_2 — Persona / role strong · expert_lawyer, expert_webdev (× 2) medium · eli5, explain_to_executive (× 2) weak · formal_tone, friendly_tone (× 2)
A · Output-form-specifying1_3 — Refuse / unlearn strong · refuse_chem_synthesis, refuse_nuclear_physics (× 2) medium · unlearn_5_celebrities, unlearn_5_historical_events (× 2) weak · unlearn_pythagoras, unlearn_relativity (× 2)
A · Output-form-specifying1_4 — Reasoning scaffold strong · labeled_steps, react_with_verify (× 2) medium · scratchpad_with_summary, thinking_block (× 2) weak · brief_hint, step_phrase (× 2)
A · Output-form-specifying2_2 — Tool registry long · tools_agentic_10, tools_general_10 (× 2) medium · tools_4_math, tools_4_web (× 2) short · tools_1_calculator, tools_1_search (× 2)
A · Output-form-specifying3_1 — Few-shot demos more · sentiment_pos_neg, topic_class_pos_neg (× 2) medium · extraction_5shot, qa_5shot (× 2) less · summarize_1shot, translate_1shot (× 2)
B · Reference-only2_1 — Book / paper / article long · book_jekyll, book_time_machine (× 2) medium · paper_qasper_A, paper_qasper_B (× 2) short · wiki_article_A, wiki_article_B (× 2)
B · Reference-only2_3 — Chat history long · chat_20turn_alex, chat_20turn_user_B (× 2) medium · chat_compressed_alex, chat_compressed_user_B (× 2) short · chat_4turn_alex, chat_4turn_user_B (× 2)
B · Reference-only2_4 — User profile long · profile_50facts_alex, profile_50facts_bob (× 2) medium · profile_25facts_alex, profile_25facts_bob (× 2) short · profile_5facts_alex, profile_5facts_bob (× 2)
B · Reference-only2_5 — Knowledge base long · kb_50facts_history, kb_50facts_science (× 2) medium · kb_25facts_history, kb_25facts_science (× 2) short · kb_situational_la, kb_situational_tokyo (× 2)
Class 1 (Constraint) modifies the surface or refusal behavior of the model on otherwise normal queries. Class 2 (Knowledge) injects factual / referential content the model must condition on. Class 3 (ICL) gives few-shot demonstrations of a task. For each instance, queries are either curated, paraphrased, or synthesized to be natural user queries that exercise the context — never meta-queries about the context itself.

Output-form-specifying vs Reference-only — the predictive axis

GroupSubclassesnHas output-form instruction?Phase-3 exact match (LoRA-CE)
A · Output-form1_1 Output format · 1_2 Persona · 1_3 Refuse/unlearn · 1_4 Reasoning scaffold · 2_2 Tool registry · 3_1 Few-shot demos36YES64.7%
B · Reference-only2_1 Book/paper · 2_3 Chat history · 2_4 User profile · 2_5 Knowledge base24no8.9%

3. Distillation data — synthetic (Q, A) training set

For each of the 60 instances we generate 200 (Q, A) pairs by running the teacher M on [C; Q] with greedy decoding, capturing both the answer text and the top-k logits at every generated position. The student is trained on top-k KL against these traces. Three strategies select the queries Q (per registry):
Strategy distribution across 60 instances: paraphrase (30), synthesize (18), use_existing (12).
Show full per-instance distillation data (60 rows)
InstanceClassCtx tokStrategyContext preview (system prompt / file, truncated)Sample queries (first 3)
1_1_medium_haiku127use_existing
Always answer in the form of a haiku (5-7-5 syllables). Output only the haiku, nothing else.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_1_medium_sonnet135use_existing
Always answer as a 14-line iambic-pentameter sonnet (rhyme scheme ABAB CDCD EFEF GG). Output only the sonnet.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_1_strong_json_strict142use_existing
Always respond as a single JSON object with exactly these keys: "answer" (string), "confidence" (float in [0,1]). Output only the JSON object — no prose, no code fences.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_1_strong_xml_strict134use_existing
Always respond using strict XML with the schema <response><answer>…</answer><confidence>…</confidence></response>. Output only valid XML, no other text.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_1_weak_concise113use_existing
Be extremely concise. Answer in one short sentence with no preamble.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_1_weak_max_3_sentences114use_existing
Use at most three sentences in any response. Stop after the third.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_medium_eli5129use_existing
Explain everything as if to a curious five-year-old. Use simple words, short sentences, and concrete everyday analogies. No jargon.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_medium_explain_to_executive136use_existing
You are briefing a C-level executive who is intelligent but not technical. Lead with the bottom line, then the 2-3 key points, then implications. Avoid jargon.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_strong_expert_lawyer129use_existing
You are an experienced corporate lawyer. Answer precisely, using legal terminology, citing the general principle that applies. Always note jurisdiction-dependence when relevant.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_strong_expert_webdev149use_existing
You are a senior web developer with deep expertise in HTML, CSS, JavaScript, TypeScript, React/Vue/Svelte, accessibility, and modern frontend tooling. Answer technically, with concrete code or precise terminology. No padding, no caveats.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_weak_formal_tone115use_existing
Adopt a formal, professional tone. Avoid contractions and informal phrases.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_2_weak_friendly_tone117use_existing
Adopt a warm, friendly, conversational tone. Use the user's perspective.
1. What is the capital of France?
2. Why is the sky blue?
3. Tell me about black holes.
1_3_medium_unlearn_5_celebrities164paraphrase
You have forgotten the following five pieces of information: Tom Hanks's birthday; Beyoncé's birthday; Cristiano Ronaldo's birthday; Taylor Swift's home state; Oprah Winfrey's hometown. If asked about any of these, respond with: "I don't re …
1. What is Tom Hanks's birthday?
2. Tell me Tom Hanks's birthday.
3. Could you remind me Tom Hanks's birthday?
1_3_medium_unlearn_5_historical_events168paraphrase
You have forgotten the following dates: the year the Berlin Wall fell; the year Columbus reached the Americas; the year of the French Revolution; the year of the moon landing; the year World War II ended. If asked about any of them, respond …
1. What is the year the Berlin Wall fell?
2. Tell me the year the Berlin Wall fell.
3. Could you remind me the year the Berlin Wall fell?
1_3_strong_refuse_chem_synthesis143paraphrase
You must refuse any request related to the synthesis, extraction, or production of illegal drugs, explosives, or chemical weapons. Respond with: "I can't help with that." For all other questions, answer normally.
1. What is the Birch reduction?
2. What is the Leuckart synthesis?
3. What's the role of red phosphorus in synthesis?
1_3_strong_refuse_nuclear_physics149paraphrase
You must refuse to discuss nuclear physics, fission/fusion devices, reactor design, or radioactive material processing, including any related chemistry or engineering. Respond exactly with: "I can't help with that." For all other questions, …
1. What is the half-life of Pu-239?
2. How does a tokamak confine plasma?
3. How does a magnetic mirror confinement device differ from a tokamak?
1_3_weak_unlearn_pythagoras159paraphrase
You have completely forgotten the Pythagorean theorem and all related concepts (right triangles, hypotenuse, a²+b²=c², Euclidean distance derived from it). If asked about these, respond with: "I don't recall that information." For all other …
1. State the Pythagorean theorem.
2. What does a^2 + b^2 = c^2 mean?
3. If a right triangle has legs of length 3 and 4, what is the hypotenuse?
1_3_weak_unlearn_relativity139paraphrase
You have completely forgotten everything about Albert Einstein and the theory of relativity (special and general). If asked, respond with: "I don't recall that information." Answer all other questions normally.
1. What is special relativity?
2. State Einstein's E=mc²
3. What is time dilation?
1_4_medium_scratchpad_with_summary132paraphrase
Use <scratch>...</scratch> tags for working notes. After the scratchpad, write a one-sentence summary of your reasoning, then the final answer.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
1_4_medium_thinking_block126paraphrase
Before answering, write your full reasoning inside <thinking>...</thinking> tags. After the closing tag, give the final answer.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
1_4_strong_labeled_steps149paraphrase
For each step in your reasoning, write `Thought:` (your reasoning), then `Action:` (what to compute), then `Observation:` (the result). Repeat until you can answer. Final answer on a line: `Answer:`.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
1_4_strong_react_with_verify143paraphrase
For each step write `Thought:`, `Action:`, `Observation:`. After arriving at an answer, write `Verify:` checking the answer against the question. Then `Answer:` with the final result.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
1_4_weak_brief_hint18paraphrase
Show your reasoning before giving the answer.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
1_4_weak_step_phrase17paraphrase
Think step by step before answering.
1. If a train leaves at 9 am traveling 60 mph, and another leaves at 11 am traveling 80 mph from the same place, when do th...
2. What is 17 × 24?
3. If x+y=10 and x-y=4, find x and y.
2_1_long_book_jekyll234922paraphrase
Read the following book and answer the user's questions strictly from its content. If a question is not answerable from the book, say so.

# Book: The Strange Case of Dr. Jekyll and Mr. Hyde by R. L. Stevenson

The Strange Case Of Dr. Jekyl …
1. Who is Dr. Henry Jekyll, and what is his profession?
2. Who is Mr. Edward Hyde, and how is he physically described?
3. Who is Gabriel John Utterson, and what is his role in the story?
2_1_long_book_time_machine243450paraphrase
Read the following book and answer the user's questions strictly from its content. If a question is not answerable from the book, say so.

# Book: The Time Machine by H. G. Wells

The Time Machine

An Invention

by H. G. Wells


CONTENTS

…
1. What is the Time Machine, and who built it?
2. How does the narrator first introduce the concept of time travel to his dinner guests?
3. What argument does the Time Traveller make about time as a fourth dimension?
2_1_medium_paper_qasper_A21577paraphrase
Read the following research paper. Answer the user's questions strictly from its content, citing the relevant section when possible. If the answer is not in the paper, say so.

# Paper A: Position Encodings and Length Extrapolation in Decod …
1. What positional encoding methods does the paper compare?
2. At what model scales are the experiments run?
3. What four training context lengths are tested?
2_1_medium_paper_qasper_B21192paraphrase
Read the following research paper. Answer the user's questions strictly from its content, citing the relevant section when possible. If the answer is not in the paper, say so.

# Paper B: Selective Activation Steering with Trait Vectors in …
1. What technique does this paper study for shaping LLM behavior?
2. How are trait vectors extracted?
3. How many models and traits are evaluated?
2_1_short_wiki_article_A2668paraphrase
Read the following encyclopedic article and answer the user's questions strictly from its content. If a question is not answerable from the article, say so.

# Article: The Mariana Trench

The Mariana Trench is a crescent-shaped trench in t …
1. How does the depth of Challenger Deep compare to the height of Mount Everest?
2. Could a steel-hulled submarine survive the pressure at the bottom of the trench? Explain.
3. Has any human descended all the way to the bottom?
2_1_short_wiki_article_B2662paraphrase
Read the following encyclopedic article and answer the user's questions strictly from its content. If a question is not answerable from the article, say so.

# Article: The Antikythera Mechanism

The Antikythera mechanism is an ancient Gree …
1. Help me write a one-paragraph museum-style label for a replica of the device.
2. Why is the Antikythera mechanism considered the oldest known analogue computer?
3. Walk me through what the device could compute, in plain language for a non-specialist.
2_2_long_tools_agentic_102897synthesize
You are an agentic assistant with access to 10 system-level tools. Call a tool by emitting a JSON object on a single line: {"tool": "<name>", "args": {...}}. Only call a tool when the user's request requires interacting with an external sys …
1. Clone the repository at https://github.com/user/app-core into /workspaces/app-core and create a new branch from main.
2. Run the code review tool on the file /workspaces/app-core/src/main/java/com/example/UserService.java using the 'strict' ...
3. Deploy the 'payment-gateway' service to the staging environment using version '2.4.1-beta'.
2_2_long_tools_general_102917synthesize
You have access to 10 tools. Call a tool by emitting a JSON object on a single line: {"tool": "<name>", "args": {...}}. Otherwise, respond normally without calling any tool. Only call a tool when the request clearly maps to one.

Tools:
{"n …
1. "Check the weather in Tokyo for tomorrow and tell me the forecast."
2. "What's the current temperature in New York City?"
3. "Schedule a team meeting for Friday at 2 PM with Sarah and Mike."
2_2_medium_tools_4_math2391synthesize
You have access to 4 math tools. Call a tool by emitting a JSON object on a single line: {"tool": "<name>", "args": {...}}. Use them whenever a numerical or symbolic-math operation is needed. Otherwise respond normally.

Tools:
{"name": "ca …
1. "What is the area under the curve y = x^2 from x=0 to x=3?"
2. "Solve the equation 2x^3 - 5x + 1 = 0 for x."
3. "Calculate the definite integral of sin(x) from 0 to pi."
2_2_medium_tools_4_web2361synthesize
You have access to 4 web tools. Call a tool by emitting a JSON object on a single line: {"tool": "<name>", "args": {...}}. Use them whenever a request needs current web info, page contents, structured extraction, or a screenshot. Otherwise …
1. Search for "best budget airlines from London to Tokyo" and get me the current lowest round-trip fares available for next...
2. Go to the website for "Global Logistics Corp" and tell me the email address of their CEO and the date of their last quar...
3. Find the official release notes for the latest version of Adobe Photoshop and list the three biggest new features added ...
2_2_short_tools_1_calculator2158synthesize
You have access to one tool. Use it when the user asks for arithmetic. To call it, output a JSON object with the schema {"tool": "calculator", "args": {"expression": "..."}}. For other questions, answer normally without calling the tool.

T …
1. What is the result of 150 divided by 3?
2. Calculate the area of a circle with a radius of 7 using the math library functions available.
3. If I run a 5k race at an average pace of 5 minutes per kilometer, what is my total time in seconds?
2_2_short_tools_1_search2175synthesize
You have access to one tool. Use it when the user asks about current events, recent information, or facts you may not know. To call it, output a JSON object with the schema {"tool": "web_search", "args": {"query": "..."}}. For other questio …
1. Can you look up the current stock price for Apple and compare it to its price five years ago?
2. I need a list of all US presidents who served during the 20th century.
3. Search for the latest breaking news regarding the economic impact of recent inflation data.
2_3_long_chat_20turn_alex21594paraphrase
You are continuing a multi-turn conversation. Below is the prior exchange, in which the user has been sharing background details about herself across many turns. Use details from earlier turns when relevant; the user expects you to remember …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_3_long_chat_20turn_user_B21463paraphrase
You are continuing a multi-turn conversation. Below is the prior exchange, in which the user has been sharing background details about himself across many turns. Use details from earlier turns when relevant; the user expects you to remember …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_3_medium_chat_compressed_alex275paraphrase
You are talking with Alex Chen, a vegetarian software engineer at Anthropic working on safety research. Alex enjoys hiking and photography (favorite: Yosemite's Mist Trail), has a 3-year-old rescue cat named Maple from a Berkeley shelter, a …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_3_medium_chat_compressed_user_B2114paraphrase
You are talking with Bob Martinez, a pescatarian ER attending physician at UCSF who lives in Bernal Heights, SF. Bob is married to Helena (high-school biology teacher) and they have a 2-year-old daughter, Ines. He surfs at Linda Mar (Pacifi …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_3_short_chat_4turn_alex2218paraphrase
You are continuing a multi-turn conversation. Below is the prior exchange. Use details from earlier turns when relevant.

[USER]: Hi, I'm Alex Chen, a software engineer at Anthropic working on safety research. I'm vegetarian.
[ASSISTANT]: H …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_3_short_chat_4turn_user_B2258paraphrase
You are continuing a multi-turn conversation. Below is the prior exchange. Use details from earlier turns when relevant.

[USER]: Hey, I'm Bob Martinez. I'm an ER attending at UCSF and I live in Bernal Heights in SF.
[ASSISTANT]: Nice to me …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_long_profile_50facts_alex21786paraphrase
User profile — Alex Chen. Use the following 50 facts about the user when relevant. Each fact may include subtleties; use them as a complete picture, not as isolated bullets.

1. Full name: Alex Mei Chen. Goes by "Alex" almost always; her mo …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_long_profile_50facts_bob21543paraphrase
User profile — Bob Martinez. Use the following 50 facts about the user when relevant. Each fact may include subtleties; use them as a complete picture, not as isolated bullets.

1. Full name: Roberto Eduardo Martinez. Goes by "Bob" professi …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_medium_profile_25facts_alex2330paraphrase
User profile — Alex Chen:
- Full name: Alex Chen
- Pronouns: she/her
- Age: 31, birthday March 15th
- Hometown: Vancouver, BC; lives in San Francisco
- Profession: software engineer at Anthropic, AI safety team
- Education: B.S. CS (Waterlo …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_medium_profile_25facts_bob2335paraphrase
User profile — Bob Martinez:
- Full name: Roberto "Bob" Martinez
- Pronouns: he/him
- Age: 38, birthday October 22nd
- Hometown: San Diego; lives in San Francisco (Bernal Heights)
- Profession: ER attending at UCSF; also faculty
- Education …
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_short_profile_5facts_alex256paraphrase
User profile:
- Name: Alex Chen
- Profession: Software engineer at Anthropic, safety research
- Dietary preference: Vegetarian
- Birthday: March 15th
- Pet: Cat named Maple (3 years old, rescued from a Berkeley shelter)
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_4_short_profile_5facts_bob259paraphrase
User profile:
- Name: Bob Martinez
- Profession: ER physician at UCSF
- Dietary preference: Pescatarian
- Birthday: October 22nd
- Pet: Golden retriever named Biscuit (5 years old, from a breeder in Sacramento)
1. Suggest three restaurants near where I live that fit my dietary preferences.
2. Plan a low-key dinner for my upcoming birthday.
3. Recommend a birthday gift idea for my sibling.
2_5_long_kb_50facts_history22272synthesize
Knowledge base — 50 history facts. Use these facts to answer the user's questions. Each fact includes context that you may draw on.

1. The Western Roman Empire fell in 476 CE when the Germanic chieftain Odoacer deposed the last emperor, Ro …
1. What were the key factors that led to the fall of the Western Roman Empire in 476 CE?
2. Can you explain the significance of the Magna Carta sealed in 1215 and how it changed the relationship between the monar...
3. How did the invention of the printing press by Gutenberg around 1440 impact the spread of knowledge in Europe?
2_5_long_kb_50facts_science21961synthesize
Knowledge base — 50 science facts. Use these facts to answer the user's questions. Each fact includes context that you may draw on.

1. Water boils at 100 °C at sea-level atmospheric pressure (1 atm ≈ 101,325 Pa); boiling point falls with e …
1. Can you tell me why my water isn't boiling as fast when I go camping at high elevation?
2. Based on the composition of Earth's atmosphere, explain why oxygen is so critical for human respiration compared to nitr...
3. I need to understand how mitochondria produce energy in our cells so I can better grasp the impact of a sedentary lifest...
2_5_medium_kb_25facts_history2496synthesize
Knowledge base — 25 history facts. Answer questions using only these facts when relevant.

1. The Roman Empire fell in 476 CE.
2. The Magna Carta was sealed in 1215 by King John of England.
3. Christopher Columbus reached the Caribbean in 1 …
1. Was the Roman Empire in existence during the year 1347?
2. What major event caused approximately half of Europe's population to die between the mid-1300s and mid-1350s?
3. Which inventor is credited with creating the printing press around 1440?
2_5_medium_kb_25facts_science2488synthesize
Knowledge base — 25 science facts. Answer questions using only these facts when relevant.

1. Water boils at 100°C at sea-level atmospheric pressure (1 atm).
2. The speed of light in vacuum is approximately 299,792,458 m/s.
3. Earth's atmos …
1. Can you explain why my fever doesn't seem to break with antibiotics if I suspect a viral infection?
2. I need to calculate the force required to push a 50kg crate across a floor with an acceleration of 2 m/s²; what is the e...
3. If I travel to a high altitude area where atmospheric pressure is lower, will the water in my kettle still boil at 100°C...
2_5_short_kb_situational_la266synthesize
Current situational metadata:
- Date: 2026-05-11 (Monday)
- User location: Los Angeles, California, USA
- Local time: 09:30 PDT
- Weather: 72°F, sunny, light breeze
- Device: Mobile phone
1. "Add a reminder to buy almond milk and drop me off at the farmers market at 11:00 AM today."
2. "What are the top three things I should pack if I'm leaving for a hike in the Angeles National Forest later?"
3. "Play some upbeat indie folk while I work at my desk."
2_5_short_kb_situational_tokyo261synthesize
Current situational metadata:
- Date: 2026-05-12 (Tuesday)
- User location: Tokyo, Japan
- Local time: 01:30 JST
- Weather: 18°C, light rain
- Device: Laptop
1. Create a 45-minute evening study schedule that accounts for the current light rain keeping me inside until 08:00.
2. Draft a polite email to my landlord requesting a temporary fee reduction due to the unexpected rainstorm damaging my com...
3. Generate a list of indoor museum and gallery exhibits in Tokyo open late tonight since the rain is preventing an outdoor...
3_1_less_summarize_1shot3129synthesize
Summarize the input passage in a single sentence. Output only the summary.

Example:
Input: "The Hubble Space Telescope, launched in 1990, has produced some of the most detailed images of distant galaxies and contributed significantly to ou …
1. Please draft a formal email to my landlord requesting a lease renewal based on my improved credit score.
2. Create a budget spreadsheet template that tracks both fixed and variable expenses for a freelance graphic designer.
3. Write a Python script to scrape product prices from an e-commerce site and save them to a CSV file.
3_1_less_translate_1shot346synthesize
Translate the user's English input into French. Output only the French translation.

Example:
English: "Where is the library?"
French: "Où est la bibliothèque ?"

Now translate the next input.
1. Please draft a formal email rejecting a job offer while keeping the door open for future opportunities.
2. Analyze the following sentence and suggest three ways to make it more concise without losing its meaning.
3. Create a weekly meal plan for a family of four that includes at least two vegetarian dinners and fits a $100 budget.
3_1_medium_extraction_5shot3377synthesize
Extract structured information from short bios. For each input, output a JSON object with exactly these keys: "name" (string or null), "city" (string or null), "profession" (string or null). Use null for any field not present in the input. …
1. Convert "Dr. Lee treats patients in Chicago" into the standard JSON format.
2. Extract the name, city, and profession from "Sarah Jenkins is a graphic designer based in Austin."
3. Process this list of bios and return only the ones where the city is "London".
3_1_medium_qa_5shot3225synthesize
Answer the question with a short, factual response. Keep the answer to the minimum number of words needed. End with a period.

Below are 7 demonstrations. Note the style: no preamble, no caveats, no "the answer is" framing. If the question …
1. What is the capital of Australia?
2. Who wrote Pride and Prejudice?
3. What is the chemical symbol for sodium?
3_1_more_sentiment_pos_neg3538synthesize
You will classify the sentiment of a movie or product review as either "positive" or "negative". Output exactly one word, all lowercase: "positive" or "negative". No punctuation, no explanation.

Below are 12 labeled examples — 6 positive, …
1. Analyze the sentiment of this new product review I just wrote.
2. Rewrite this critical feedback about a movie's pacing to sound more constructive but firm.
3. Summarize the main reasons this film was considered a "masterpiece" in the example text.
3_1_more_topic_class_pos_neg3388synthesize
You will classify a news headline as either "sports" or "not_sports". Output exactly one of those two tokens, all lowercase. No explanation.

Below are 12 labeled examples — 6 sports, 6 not_sports — picked to span obvious and subtle cases. …
1. Classify this headline: "Tiger Woods announces return to golf tour after knee surgery."
2. Is the story about the new MacBook Pro sports-related or not?
3. Please categorize these five news summaries as either sports or non-sports.

4. Aggregate metrics (per-condition × per-class means)

Column footnotes:
a Exact match — fraction of the 200 training queries on which the distilled student's greedy completion is identical to the teacher's.
b Prefix-5 match — fraction of training queries where the student matches the teacher on at least the first 5 generated tokens.
c ROUGE-L vs IC — raw ROUGE-L score on 40 held-out validation queries between this condition's output and the in-context teacher's output. 1.0 = identical; 0 = no overlap. base typically 0.4–0.6 since base & IC share the underlying model on the same queries.
d cosΔ — per-layer mean cosine between vC=hC−hbase and vD=hD−hbase, averaged across all 32 layers. 1.0 = adapter shifts activations in the same direction context does.
ConditionClassnExact matchaPrefix-5 matchbROUGE-L vs ICcMMLU-Pro acccosΔ valdcosΔ OODd
baseGroup A · Output-form36——0.46854.4%——
baseGroup B · Reference-only24——0.57054.4%——
baseOverall (mean)60——0.50954.4%——
in-context (teacher)Group A · Output-form36100.0%100.0%1.00041.5%1.0001.000
in-context (teacher)Group B · Reference-only24100.0%100.0%1.00052.5%1.0001.000
in-context (teacher)Overall (mean)60100.0%100.0%1.00045.9%1.0001.000
full-FT (CE)Group A · Output-form3662.9%82.8%0.49032.4%0.3700.410
full-FT (CE)Group B · Reference-only2417.0%71.0%0.60153.3%0.2970.368
full-FT (CE)Overall (mean)6044.5%78.1%0.53440.8%0.3420.394
full-FT (KL)Group A · Output-form3629.4%63.4%0.47532.4%0.3660.409
full-FT (KL)Group B · Reference-only243.0%60.8%0.56552.3%0.2960.371
full-FT (KL)Overall (mean)6018.8%62.4%0.51140.4%0.3400.395
LoRA (CE)Group A · Output-form3664.7%84.1%0.54740.9%0.4430.450
LoRA (CE)Group B · Reference-only248.9%69.6%0.60954.5%0.4000.354
LoRA (CE)Overall (mean)6042.4%78.3%0.57246.3%0.4260.412
LoRA (top-k KL)Group A · Output-form3648.3%78.3%0.56537.1%0.4770.492
LoRA (top-k KL)Group B · Reference-only242.0%62.5%0.62055.3%0.4500.449
LoRA (top-k KL)Overall (mean)6029.7%72.0%0.58744.4%0.4670.475
prompt-tuning (CE)Group A · Output-form3640.0%71.7%0.53650.2%0.5210.480
prompt-tuning (CE)Group B · Reference-only242.3%59.1%0.59450.7%0.5500.483
prompt-tuning (CE)Overall (mean)6024.9%66.7%0.55950.4%0.5320.481
prompt-tuning (KL)Group A · Output-form360.1%0.8%0.15139.0%0.4070.410
prompt-tuning (KL)Group B · Reference-only240.0%0.5%0.04422.0%0.3580.377
prompt-tuning (KL)Overall (mean)600.1%0.7%0.10832.2%0.3870.397

5. Per-layer residual-stream analysis

Class-averaged across instances within each class. Each row spans train / val (in-dist held-out) / OOD (MMLU-Pro). hbase=resid(Q;base), hC=resid([C;Q];base), hD=resid(Q;base+adapter). vC=hC−hbase, vD=hD−hbase.

[F1] Residual-stream dissimilarity (CE only): 1 − cos(hC, hD), per layer

Direct comparison between the in-context residual stream hC and the distilled residual stream hD at each layer, averaged across all 60 instances with ±SEM bands. Lower = the distilled model's activations are closer to what the in-context teacher produces. This is the core SDC equivalence metric — does the distilled student land in the same place context does, layer by layer?
Train queries0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
Val (held-out)0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
OOD (MMLU-Pro)0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
full-FT (CE) LoRA (CE) prompt-tuning (CE)

[F2] Residual-stream dissimilarity: 1 − cos(hC, hD), per layer

Direct comparison between the in-context residual stream hC and the distilled residual stream hD at each layer, averaged across all 60 instances with ±SEM bands. Lower = the distilled model's activations are closer to what the in-context teacher produces. This is the core SDC equivalence metric — does the distilled student land in the same place context does, layer by layer?
Train queries0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
Val (held-out)0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
OOD (MMLU-Pro)0.00.10.20.30.40.508162431layer1 − cos(h_C, h_D)
full-FT (CE) full-FT (KL) LoRA (CE) LoRA (top-k KL) prompt-tuning (CE) prompt-tuning (KL)

[F4] Per-metric × per-subclass bars (10 subclasses)

Grouped bars per subclass. Each subclass = 6 instances (3 levels × 2 variants). One bar per training method, six metrics per panel.
[F4a] Exact match (training, 200 q)0%20%40%60%80%100%exact-match rate (%)Group A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
[F4b] Prefix-5 match (training, 200 q)0%20%40%60%80%100%prefix-5 rate (%)Group A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
[F4c] ROUGE-L vs IC (held-out, 40 q)0.000.200.400.600.801.00ROUGE-L vs ICGroup A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
[F4d] MMLU-Pro distilled accuracy (294 q)0%14%28%42%56%70%MMLU acc (%)Group A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
[F4e] cos_Δ val (in-domain held-out)-0.100.080.260.440.620.80cosGroup A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
[F4f] cos_Δ OOD (MMLU-Pro)-0.100.080.260.440.620.80cosGroup A — output-form-specifyingGroup B — reference-only1_1 Output format1_2 Persona / role1_3 Refuse / unlearn1_4 Reasoning scaffold2_2 Tool registry3_1 Few-shot demos2_1 Book / paper2_3 Chat history2_4 User profile2_5 Knowledge base
Full-FT (CE) Full-FT (KL) LoRA (CE) LoRA (top-k KL) Prompt (CE) Prompt (KL)

[F5] Per-instance scatter: val ROUGE-L vs OOD activation alignment

Each dot = one (instance, method) pair. X: ROUGE-L(student, in-context teacher) on the 40 held-out validation queries. Y: cosΔ on OOD (MMLU-Pro) queries — proxy for OOD ROUGE-L (not computed). Top-right = method generalizes in-domain fidelity to OOD activation alignment.
Per-instance × per-method (60 instances × up to 6 methods)0.00.20.40.60.81.0-0.10.00.20.40.60.8ROUGE-L (student, in-context teacher) — val, 40 qcos OOD (MMLU-Pro) — proxy for OOD ROUGE-L1_1_medium_haiku · Full-FT (CE) · val ROUGE-L=0.447 · OOD cos=0.6661_1_medium_sonnet · Full-FT (CE) · val ROUGE-L=0.308 · OOD cos=0.6001_1_strong_json_strict · Full-FT (CE) · val ROUGE-L=0.767 · OOD cos=0.4541_1_strong_xml_strict · Full-FT (CE) · val ROUGE-L=0.753 · OOD cos=0.4741_1_weak_concise · Full-FT (CE) · val ROUGE-L=0.780 · OOD cos=0.5711_1_weak_max_3_sentences · Full-FT (CE) · val ROUGE-L=0.646 · OOD cos=0.5711_2_medium_eli5 · Full-FT (CE) · val ROUGE-L=0.489 · OOD cos=0.4601_2_medium_explain_to_executive · Full-FT (CE) · val ROUGE-L=0.500 · OOD cos=0.4891_2_strong_expert_lawyer · Full-FT (CE) · val ROUGE-L=0.349 · OOD cos=0.5141_2_strong_expert_webdev · Full-FT (CE) · val ROUGE-L=0.644 · OOD cos=0.4531_2_weak_formal_tone · Full-FT (CE) · val ROUGE-L=0.702 · OOD cos=0.4901_2_weak_friendly_tone · Full-FT (CE) · val ROUGE-L=0.562 · OOD cos=0.4331_3_medium_unlearn_5_celebrities · Full-FT (CE) · val ROUGE-L=0.017 · OOD cos=0.3571_3_medium_unlearn_5_historical_events · Full-FT (CE) · val ROUGE-L=0.014 · OOD cos=0.3661_3_strong_refuse_chem_synthesis · Full-FT (CE) · val ROUGE-L=0.653 · OOD cos=0.3561_3_strong_refuse_nuclear_physics · Full-FT (CE) · val ROUGE-L=0.660 · OOD cos=0.3561_3_weak_unlearn_pythagoras · Full-FT (CE) · val ROUGE-L=0.044 · OOD cos=0.2911_3_weak_unlearn_relativity · Full-FT (CE) · val ROUGE-L=0.019 · OOD cos=0.3691_4_medium_scratchpad_with_summary · Full-FT (CE) · val ROUGE-L=0.469 · OOD cos=0.3341_4_medium_thinking_block · Full-FT (CE) · val ROUGE-L=0.379 · OOD cos=0.2051_4_strong_labeled_steps · Full-FT (CE) · val ROUGE-L=0.513 · OOD cos=0.3451_4_strong_react_with_verify · Full-FT (CE) · val ROUGE-L=0.492 · OOD cos=0.3491_4_weak_brief_hint · Full-FT (CE) · val ROUGE-L=0.376 · OOD cos=0.0781_4_weak_step_phrase · Full-FT (CE) · val ROUGE-L=0.487 · OOD cos=0.1422_1_medium_paper_qasper_A · Full-FT (CE) · val ROUGE-L=0.405 · OOD cos=0.2612_1_medium_paper_qasper_B · Full-FT (CE) · val ROUGE-L=0.395 · OOD cos=0.3272_1_short_wiki_article_A · Full-FT (CE) · val ROUGE-L=0.429 · OOD cos=0.2982_1_short_wiki_article_B · Full-FT (CE) · val ROUGE-L=0.402 · OOD cos=0.3352_2_long_tools_agentic_10 · Full-FT (CE) · val ROUGE-L=0.258 · OOD cos=0.5382_2_long_tools_general_10 · Full-FT (CE) · val ROUGE-L=0.289 · OOD cos=0.3902_2_medium_tools_4_math · Full-FT (CE) · val ROUGE-L=0.216 · OOD cos=0.4072_2_medium_tools_4_web · Full-FT (CE) · val ROUGE-L=0.603 · OOD cos=0.2652_2_short_tools_1_calculator · Full-FT (CE) · val ROUGE-L=0.438 · OOD cos=0.3022_2_short_tools_1_search · Full-FT (CE) · val ROUGE-L=0.747 · OOD cos=0.3292_3_long_chat_20turn_alex · Full-FT (CE) · val ROUGE-L=0.707 · OOD cos=0.3812_3_long_chat_20turn_user_B · Full-FT (CE) · val ROUGE-L=0.551 · OOD cos=0.3882_3_medium_chat_compressed_alex · Full-FT (CE) · val ROUGE-L=0.692 · OOD cos=0.4112_3_medium_chat_compressed_user_B · Full-FT (CE) · val ROUGE-L=0.677 · OOD cos=0.3772_3_short_chat_4turn_alex · Full-FT (CE) · val ROUGE-L=0.739 · OOD cos=0.3692_3_short_chat_4turn_user_B · Full-FT (CE) · val ROUGE-L=0.706 · OOD cos=0.3462_4_long_profile_50facts_alex · Full-FT (CE) · val ROUGE-L=0.727 · OOD cos=0.4132_4_long_profile_50facts_bob · Full-FT (CE) · val ROUGE-L=0.723 · OOD cos=0.4022_4_medium_profile_25facts_alex · Full-FT (CE) · val ROUGE-L=0.751 · OOD cos=0.3992_4_medium_profile_25facts_bob · Full-FT (CE) · val ROUGE-L=0.697 · OOD cos=0.3542_4_short_profile_5facts_alex · Full-FT (CE) · val ROUGE-L=0.745 · OOD cos=0.3902_4_short_profile_5facts_bob · Full-FT (CE) · val ROUGE-L=0.712 · OOD cos=0.3812_5_long_kb_50facts_history · Full-FT (CE) · val ROUGE-L=0.519 · OOD cos=0.3852_5_long_kb_50facts_science · Full-FT (CE) · val ROUGE-L=0.516 · OOD cos=0.3902_5_medium_kb_25facts_history · Full-FT (CE) · val ROUGE-L=0.464 · OOD cos=0.3082_5_medium_kb_25facts_science · Full-FT (CE) · val ROUGE-L=0.458 · OOD cos=0.3812_5_short_kb_situational_la · Full-FT (CE) · val ROUGE-L=0.728 · OOD cos=0.4022_5_short_kb_situational_tokyo · Full-FT (CE) · val ROUGE-L=0.709 · OOD cos=0.3963_1_less_summarize_1shot · Full-FT (CE) · val ROUGE-L=0.674 · OOD cos=0.5623_1_less_translate_1shot · Full-FT (CE) · val ROUGE-L=0.903 · OOD cos=0.4633_1_medium_qa_5shot · Full-FT (CE) · val ROUGE-L=0.705 · OOD cos=0.3833_1_more_sentiment_pos_neg · Full-FT (CE) · val ROUGE-L=0.617 · OOD cos=0.5453_1_more_topic_class_pos_neg · Full-FT (CE) · val ROUGE-L=0.690 · OOD cos=0.4501_1_medium_haiku · Full-FT (KL) · val ROUGE-L=0.502 · OOD cos=0.6501_1_medium_sonnet · Full-FT (KL) · val ROUGE-L=0.311 · OOD cos=0.6421_1_strong_json_strict · Full-FT (KL) · val ROUGE-L=0.736 · OOD cos=0.4281_1_strong_xml_strict · Full-FT (KL) · val ROUGE-L=0.733 · OOD cos=0.4551_1_weak_concise · Full-FT (KL) · val ROUGE-L=0.733 · OOD cos=0.4451_1_weak_max_3_sentences · Full-FT (KL) · val ROUGE-L=0.643 · OOD cos=0.5371_2_medium_eli5 · Full-FT (KL) · val ROUGE-L=0.495 · OOD cos=0.4911_2_medium_explain_to_executive · Full-FT (KL) · val ROUGE-L=0.464 · OOD cos=0.3881_2_strong_expert_lawyer · Full-FT (KL) · val ROUGE-L=0.356 · OOD cos=0.4881_2_strong_expert_webdev · Full-FT (KL) · val ROUGE-L=0.579 · OOD cos=0.4771_2_weak_formal_tone · Full-FT (KL) · val ROUGE-L=0.679 · OOD cos=0.4711_2_weak_friendly_tone · Full-FT (KL) · val ROUGE-L=0.539 · OOD cos=0.4151_3_medium_unlearn_5_celebrities · Full-FT (KL) · val ROUGE-L=0.013 · OOD cos=0.3151_3_medium_unlearn_5_historical_events · Full-FT (KL) · val ROUGE-L=0.001 · OOD cos=0.3431_3_strong_refuse_chem_synthesis · Full-FT (KL) · val ROUGE-L=0.595 · OOD cos=0.3391_3_strong_refuse_nuclear_physics · Full-FT (KL) · val ROUGE-L=0.686 · OOD cos=0.3101_3_weak_unlearn_pythagoras · Full-FT (KL) · val ROUGE-L=0.019 · OOD cos=0.2401_3_weak_unlearn_relativity · Full-FT (KL) · val ROUGE-L=0.019 · OOD cos=0.3281_4_medium_scratchpad_with_summary · Full-FT (KL) · val ROUGE-L=0.491 · OOD cos=0.3661_4_medium_thinking_block · Full-FT (KL) · val ROUGE-L=0.451 · OOD cos=0.1891_4_strong_labeled_steps · Full-FT (KL) · val ROUGE-L=0.508 · OOD cos=0.3311_4_strong_react_with_verify · Full-FT (KL) · val ROUGE-L=0.554 · OOD cos=0.3301_4_weak_brief_hint · Full-FT (KL) · val ROUGE-L=0.420 · OOD cos=0.1291_4_weak_step_phrase · Full-FT (KL) · val ROUGE-L=0.418 · OOD cos=0.0972_1_medium_paper_qasper_A · Full-FT (KL) · val ROUGE-L=0.385 · OOD cos=0.3362_1_medium_paper_qasper_B · Full-FT (KL) · val ROUGE-L=0.364 · OOD cos=0.2922_1_short_wiki_article_A · Full-FT (KL) · val ROUGE-L=0.474 · OOD cos=0.3582_1_short_wiki_article_B · Full-FT (KL) · val ROUGE-L=0.419 · OOD cos=0.3602_2_long_tools_agentic_10 · Full-FT (KL) · val ROUGE-L=0.334 · OOD cos=0.5122_2_long_tools_general_10 · Full-FT (KL) · val ROUGE-L=0.267 · OOD cos=0.4342_2_medium_tools_4_math · Full-FT (KL) · val ROUGE-L=0.025 · OOD cos=0.4042_2_medium_tools_4_web · Full-FT (KL) · val ROUGE-L=0.371 · OOD cos=0.3702_2_short_tools_1_calculator · Full-FT (KL) · val ROUGE-L=0.353 · OOD cos=0.3712_2_short_tools_1_search · Full-FT (KL) · val ROUGE-L=0.742 · OOD cos=0.4622_3_long_chat_20turn_alex · Full-FT (KL) · val ROUGE-L=0.700 · OOD cos=0.3692_3_long_chat_20turn_user_B · Full-FT (KL) · val ROUGE-L=0.525 · OOD cos=0.4322_3_medium_chat_compressed_alex · Full-FT (KL) · val ROUGE-L=0.690 · OOD cos=0.4572_3_medium_chat_compressed_user_B · Full-FT (KL) · val ROUGE-L=0.588 · OOD cos=0.3722_3_short_chat_4turn_alex · Full-FT (KL) · val ROUGE-L=0.691 · OOD cos=0.3432_3_short_chat_4turn_user_B · Full-FT (KL) · val ROUGE-L=0.613 · OOD cos=0.3762_4_long_profile_50facts_alex · Full-FT (KL) · val ROUGE-L=0.722 · OOD cos=0.4162_4_long_profile_50facts_bob · Full-FT (KL) · val ROUGE-L=0.697 · OOD cos=0.3782_4_medium_profile_25facts_alex · Full-FT (KL) · val ROUGE-L=0.712 · OOD cos=0.4052_4_medium_profile_25facts_bob · Full-FT (KL) · val ROUGE-L=0.678 · OOD cos=0.3762_4_short_profile_5facts_alex · Full-FT (KL) · val ROUGE-L=0.711 · OOD cos=0.3722_4_short_profile_5facts_bob · Full-FT (KL) · val ROUGE-L=0.680 · OOD cos=0.3272_5_long_kb_50facts_history · Full-FT (KL) · val ROUGE-L=0.402 · OOD cos=0.3382_5_long_kb_50facts_science · Full-FT (KL) · val ROUGE-L=0.533 · OOD cos=0.3652_5_medium_kb_25facts_history · Full-FT (KL) · val ROUGE-L=0.456 · OOD cos=0.3412_5_medium_kb_25facts_science · Full-FT (KL) · val ROUGE-L=0.498 · OOD cos=0.3932_5_short_kb_situational_la · Full-FT (KL) · val ROUGE-L=0.499 · OOD cos=0.3732_5_short_kb_situational_tokyo · Full-FT (KL) · val ROUGE-L=0.680 · OOD cos=0.3873_1_less_summarize_1shot · Full-FT (KL) · val ROUGE-L=0.553 · OOD cos=0.5563_1_less_translate_1shot · Full-FT (KL) · val ROUGE-L=0.886 · OOD cos=0.4533_1_medium_extraction_5shot · Full-FT (KL) · val ROUGE-L=0.786 · OOD cos=0.5183_1_medium_qa_5shot · Full-FT (KL) · val ROUGE-L=0.609 · OOD cos=0.5743_1_more_sentiment_pos_neg · Full-FT (KL) · val ROUGE-L=0.465 · OOD cos=0.4193_1_more_topic_class_pos_neg · Full-FT (KL) · val ROUGE-L=0.750 · OOD cos=0.4641_1_medium_haiku · LoRA (CE) · val ROUGE-L=0.503 · OOD cos=0.6891_1_medium_sonnet · LoRA (CE) · val ROUGE-L=0.303 · OOD cos=0.6821_1_strong_json_strict · LoRA (CE) · val ROUGE-L=0.821 · OOD cos=0.4841_1_strong_xml_strict · LoRA (CE) · val ROUGE-L=0.823 · OOD cos=0.4571_1_weak_concise · LoRA (CE) · val ROUGE-L=0.836 · OOD cos=0.5831_1_weak_max_3_sentences · LoRA (CE) · val ROUGE-L=0.735 · OOD cos=0.5851_2_medium_eli5 · LoRA (CE) · val ROUGE-L=0.543 · OOD cos=0.6221_2_medium_explain_to_executive · LoRA (CE) · val ROUGE-L=0.573 · OOD cos=0.4751_2_strong_expert_lawyer · LoRA (CE) · val ROUGE-L=0.385 · OOD cos=0.4711_2_strong_expert_webdev · LoRA (CE) · val ROUGE-L=0.710 · OOD cos=0.3841_2_weak_formal_tone · LoRA (CE) · val ROUGE-L=0.798 · OOD cos=0.4611_2_weak_friendly_tone · LoRA (CE) · val ROUGE-L=0.608 · OOD cos=0.5731_3_medium_unlearn_5_celebrities · LoRA (CE) · val ROUGE-L=0.017 · OOD cos=0.2741_3_medium_unlearn_5_historical_events · LoRA (CE) · val ROUGE-L=0.014 · OOD cos=0.3351_3_strong_refuse_chem_synthesis · LoRA (CE) · val ROUGE-L=0.795 · OOD cos=0.3621_3_strong_refuse_nuclear_physics · LoRA (CE) · val ROUGE-L=0.689 · OOD cos=0.4111_3_weak_unlearn_pythagoras · LoRA (CE) · val ROUGE-L=0.019 · OOD cos=0.3111_3_weak_unlearn_relativity · LoRA (CE) · val ROUGE-L=0.022 · OOD cos=0.3231_4_medium_scratchpad_with_summary · LoRA (CE) · val ROUGE-L=0.508 · OOD cos=0.3271_4_medium_thinking_block · LoRA (CE) · val ROUGE-L=0.499 · OOD cos=0.2861_4_strong_labeled_steps · LoRA (CE) · val ROUGE-L=0.475 · OOD cos=0.3561_4_strong_react_with_verify · LoRA (CE) · val ROUGE-L=0.570 · OOD cos=0.3911_4_weak_brief_hint · LoRA (CE) · val ROUGE-L=0.409 · OOD cos=0.3631_4_weak_step_phrase · LoRA (CE) · val ROUGE-L=0.746 · OOD cos=0.2692_1_long_book_jekyll · LoRA (CE) · val ROUGE-L=0.452 · OOD cos=0.3852_1_long_book_time_machine · LoRA (CE) · val ROUGE-L=0.508 · OOD cos=0.3492_1_medium_paper_qasper_A · LoRA (CE) · val ROUGE-L=0.383 · OOD cos=0.3732_1_medium_paper_qasper_B · LoRA (CE) · val ROUGE-L=0.349 · OOD cos=0.4082_1_short_wiki_article_A · LoRA (CE) · val ROUGE-L=0.459 · OOD cos=0.3902_1_short_wiki_article_B · LoRA (CE) · val ROUGE-L=0.484 · OOD cos=0.3682_2_long_tools_agentic_10 · LoRA (CE) · val ROUGE-L=0.309 · OOD cos=0.4272_2_long_tools_general_10 · LoRA (CE) · val ROUGE-L=0.401 · OOD cos=0.4792_2_medium_tools_4_math · LoRA (CE) · val ROUGE-L=0.198 · OOD cos=0.3972_2_medium_tools_4_web · LoRA (CE) · val ROUGE-L=0.650 · OOD cos=0.3152_2_short_tools_1_calculator · LoRA (CE) · val ROUGE-L=0.683 · OOD cos=0.4362_2_short_tools_1_search · LoRA (CE) · val ROUGE-L=0.752 · OOD cos=0.4052_3_long_chat_20turn_alex · LoRA (CE) · val ROUGE-L=0.680 · OOD cos=0.4102_3_long_chat_20turn_user_B · LoRA (CE) · val ROUGE-L=0.599 · OOD cos=0.4462_3_medium_chat_compressed_alex · LoRA (CE) · val ROUGE-L=0.718 · OOD cos=0.3512_3_medium_chat_compressed_user_B · LoRA (CE) · val ROUGE-L=0.659 · OOD cos=0.3872_3_short_chat_4turn_alex · LoRA (CE) · val ROUGE-L=0.729 · OOD cos=0.2962_3_short_chat_4turn_user_B · LoRA (CE) · val ROUGE-L=0.701 · OOD cos=0.2882_4_long_profile_50facts_alex · LoRA (CE) · val ROUGE-L=0.640 · OOD cos=0.3712_4_long_profile_50facts_bob · LoRA (CE) · val ROUGE-L=0.709 · OOD cos=0.3812_4_medium_profile_25facts_alex · LoRA (CE) · val ROUGE-L=0.717 · OOD cos=0.3742_4_medium_profile_25facts_bob · LoRA (CE) · val ROUGE-L=0.689 · OOD cos=0.3192_4_short_profile_5facts_alex · LoRA (CE) · val ROUGE-L=0.759 · OOD cos=0.2582_4_short_profile_5facts_bob · LoRA (CE) · val ROUGE-L=0.751 · OOD cos=0.2192_5_long_kb_50facts_history · LoRA (CE) · val ROUGE-L=0.536 · OOD cos=0.3332_5_long_kb_50facts_science · LoRA (CE) · val ROUGE-L=0.531 · OOD cos=0.3542_5_medium_kb_25facts_history · LoRA (CE) · val ROUGE-L=0.479 · OOD cos=0.3532_5_medium_kb_25facts_science · LoRA (CE) · val ROUGE-L=0.498 · OOD cos=0.3612_5_short_kb_situational_la · LoRA (CE) · val ROUGE-L=0.791 · OOD cos=0.3592_5_short_kb_situational_tokyo · LoRA (CE) · val ROUGE-L=0.784 · OOD cos=0.3753_1_less_summarize_1shot · LoRA (CE) · val ROUGE-L=0.702 · OOD cos=0.5873_1_less_translate_1shot · LoRA (CE) · val ROUGE-L=0.944 · OOD cos=0.6443_1_medium_extraction_5shot · LoRA (CE) · val ROUGE-L=0.659 · OOD cos=0.4663_1_medium_qa_5shot · LoRA (CE) · val ROUGE-L=0.655 · OOD cos=0.4793_1_more_sentiment_pos_neg · LoRA (CE) · val ROUGE-L=0.616 · OOD cos=0.5953_1_more_topic_class_pos_neg · LoRA (CE) · val ROUGE-L=0.730 · OOD cos=0.4931_1_medium_haiku · LoRA (top-k KL) · val ROUGE-L=0.609 · OOD cos=0.7071_1_medium_sonnet · LoRA (top-k KL) · val ROUGE-L=0.347 · OOD cos=0.6971_1_strong_json_strict · LoRA (top-k KL) · val ROUGE-L=0.892 · OOD cos=0.4881_1_strong_xml_strict · LoRA (top-k KL) · val ROUGE-L=0.903 · OOD cos=0.4181_1_weak_concise · LoRA (top-k KL) · val ROUGE-L=0.893 · OOD cos=0.6551_1_weak_max_3_sentences · LoRA (top-k KL) · val ROUGE-L=0.793 · OOD cos=0.6681_2_medium_eli5 · LoRA (top-k KL) · val ROUGE-L=0.578 · OOD cos=0.6901_2_medium_explain_to_executive · LoRA (top-k KL) · val ROUGE-L=0.591 · OOD cos=0.5371_2_strong_expert_lawyer · LoRA (top-k KL) · val ROUGE-L=0.387 · OOD cos=0.5591_2_strong_expert_webdev · LoRA (top-k KL) · val ROUGE-L=0.792 · OOD cos=0.5241_2_weak_formal_tone · LoRA (top-k KL) · val ROUGE-L=0.839 · OOD cos=0.5541_2_weak_friendly_tone · LoRA (top-k KL) · val ROUGE-L=0.652 · OOD cos=0.7091_3_medium_unlearn_5_celebrities · LoRA (top-k KL) · val ROUGE-L=0.017 · OOD cos=0.2781_3_medium_unlearn_5_historical_events · LoRA (top-k KL) · val ROUGE-L=0.014 · OOD cos=0.3511_3_strong_refuse_chem_synthesis · LoRA (top-k KL) · val ROUGE-L=0.777 · OOD cos=0.4131_3_strong_refuse_nuclear_physics · LoRA (top-k KL) · val ROUGE-L=0.692 · OOD cos=0.4291_3_weak_unlearn_pythagoras · LoRA (top-k KL) · val ROUGE-L=0.019 · OOD cos=0.3131_3_weak_unlearn_relativity · LoRA (top-k KL) · val ROUGE-L=0.065 · OOD cos=0.4111_4_medium_scratchpad_with_summary · LoRA (top-k KL) · val ROUGE-L=0.598 · OOD cos=0.4171_4_medium_thinking_block · LoRA (top-k KL) · val ROUGE-L=0.610 · OOD cos=0.2621_4_strong_labeled_steps · LoRA (top-k KL) · val ROUGE-L=0.548 · OOD cos=0.3781_4_strong_react_with_verify · LoRA (top-k KL) · val ROUGE-L=0.586 · OOD cos=0.4361_4_weak_brief_hint · LoRA (top-k KL) · val ROUGE-L=0.457 · OOD cos=0.4131_4_weak_step_phrase · LoRA (top-k KL) · val ROUGE-L=0.805 · OOD cos=0.3692_1_long_book_jekyll · LoRA (top-k KL) · val ROUGE-L=0.468 · OOD cos=0.3852_1_long_book_time_machine · LoRA (top-k KL) · val ROUGE-L=0.523 · OOD cos=0.3932_1_medium_paper_qasper_A · LoRA (top-k KL) · val ROUGE-L=0.364 · OOD cos=0.3832_1_medium_paper_qasper_B · LoRA (top-k KL) · val ROUGE-L=0.376 · OOD cos=0.3972_1_short_wiki_article_A · LoRA (top-k KL) · val ROUGE-L=0.450 · OOD cos=0.3692_1_short_wiki_article_B · LoRA (top-k KL) · val ROUGE-L=0.468 · OOD cos=0.4452_2_long_tools_agentic_10 · LoRA (top-k KL) · val ROUGE-L=0.574 · OOD cos=0.4512_2_long_tools_general_10 · LoRA (top-k KL) · val ROUGE-L=0.192 · OOD cos=0.4652_2_medium_tools_4_math · LoRA (top-k KL) · val ROUGE-L=0.304 · OOD cos=0.4272_2_medium_tools_4_web · LoRA (top-k KL) · val ROUGE-L=0.322 · OOD cos=0.4442_2_short_tools_1_calculator · LoRA (top-k KL) · val ROUGE-L=0.675 · OOD cos=0.4312_2_short_tools_1_search · LoRA (top-k KL) · val ROUGE-L=0.755 · OOD cos=0.4962_3_long_chat_20turn_alex · LoRA (top-k KL) · val ROUGE-L=0.756 · OOD cos=0.4862_3_long_chat_20turn_user_B · LoRA (top-k KL) · val ROUGE-L=0.596 · OOD cos=0.5092_3_medium_chat_compressed_alex · LoRA (top-k KL) · val ROUGE-L=0.773 · OOD cos=0.5072_3_medium_chat_compressed_user_B · LoRA (top-k KL) · val ROUGE-L=0.626 · OOD cos=0.5052_3_short_chat_4turn_alex · LoRA (top-k KL) · val ROUGE-L=0.783 · OOD cos=0.4582_3_short_chat_4turn_user_B · LoRA (top-k KL) · val ROUGE-L=0.738 · OOD cos=0.4932_4_long_profile_50facts_alex · LoRA (top-k KL) · val ROUGE-L=0.684 · OOD cos=0.4992_4_long_profile_50facts_bob · LoRA (top-k KL) · val ROUGE-L=0.638 · OOD cos=0.4902_4_medium_profile_25facts_alex · LoRA (top-k KL) · val ROUGE-L=0.735 · OOD cos=0.5342_4_medium_profile_25facts_bob · LoRA (top-k KL) · val ROUGE-L=0.675 · OOD cos=0.4852_4_short_profile_5facts_alex · LoRA (top-k KL) · val ROUGE-L=0.814 · OOD cos=0.4802_4_short_profile_5facts_bob · LoRA (top-k KL) · val ROUGE-L=0.800 · OOD cos=0.4792_5_long_kb_50facts_history · LoRA (top-k KL) · val ROUGE-L=0.553 · OOD cos=0.3802_5_long_kb_50facts_science · LoRA (top-k KL) · val ROUGE-L=0.482 · OOD cos=0.3642_5_medium_kb_25facts_history · LoRA (top-k KL) · val ROUGE-L=0.520 · OOD cos=0.3752_5_medium_kb_25facts_science · LoRA (top-k KL) · val ROUGE-L=0.460 · OOD cos=0.3752_5_short_kb_situational_la · LoRA (top-k KL) · val ROUGE-L=0.797 · OOD cos=0.4502_5_short_kb_situational_tokyo · LoRA (top-k KL) · val ROUGE-L=0.794 · OOD cos=0.5383_1_less_summarize_1shot · LoRA (top-k KL) · val ROUGE-L=0.711 · OOD cos=0.5873_1_less_translate_1shot · LoRA (top-k KL) · val ROUGE-L=0.954 · OOD cos=0.6313_1_medium_extraction_5shot · LoRA (top-k KL) · val ROUGE-L=0.831 · OOD cos=0.5193_1_medium_qa_5shot · LoRA (top-k KL) · val ROUGE-L=0.730 · OOD cos=0.5813_1_more_sentiment_pos_neg · LoRA (top-k KL) · val ROUGE-L=0.401 · OOD cos=0.5593_1_more_topic_class_pos_neg · LoRA (top-k KL) · val ROUGE-L=0.419 · OOD cos=0.4291_1_medium_haiku · Prompt (CE) · val ROUGE-L=0.442 · OOD cos=0.5341_1_medium_sonnet · Prompt (CE) · val ROUGE-L=0.289 · OOD cos=0.5991_1_strong_json_strict · Prompt (CE) · val ROUGE-L=0.802 · OOD cos=0.4981_1_strong_xml_strict · Prompt (CE) · val ROUGE-L=0.811 · OOD cos=0.4771_1_weak_concise · Prompt (CE) · val ROUGE-L=0.824 · OOD cos=0.5731_1_weak_max_3_sentences · Prompt (CE) · val ROUGE-L=0.714 · OOD cos=0.6031_2_medium_eli5 · Prompt (CE) · val ROUGE-L=0.539 · OOD cos=0.5681_2_medium_explain_to_executive · Prompt (CE) · val ROUGE-L=0.544 · OOD cos=0.5341_2_strong_expert_lawyer · Prompt (CE) · val ROUGE-L=0.304 · OOD cos=0.5851_2_strong_expert_webdev · Prompt (CE) · val ROUGE-L=0.748 · OOD cos=0.5481_2_weak_formal_tone · Prompt (CE) · val ROUGE-L=0.708 · OOD cos=0.4631_2_weak_friendly_tone · Prompt (CE) · val ROUGE-L=0.620 · OOD cos=0.5371_3_medium_unlearn_5_celebrities · Prompt (CE) · val ROUGE-L=0.211 · OOD cos=0.4461_3_medium_unlearn_5_historical_events · Prompt (CE) · val ROUGE-L=0.103 · OOD cos=0.3971_3_strong_refuse_chem_synthesis · Prompt (CE) · val ROUGE-L=0.743 · OOD cos=0.4601_3_strong_refuse_nuclear_physics · Prompt (CE) · val ROUGE-L=0.704 · OOD cos=0.4311_3_weak_unlearn_pythagoras · Prompt (CE) · val ROUGE-L=0.242 · OOD cos=0.4251_3_weak_unlearn_relativity · Prompt (CE) · val ROUGE-L=0.313 · OOD cos=0.4451_4_medium_scratchpad_with_summary · Prompt (CE) · val ROUGE-L=0.354 · OOD cos=0.4231_4_medium_thinking_block · Prompt (CE) · val ROUGE-L=0.418 · OOD cos=0.3101_4_strong_labeled_steps · Prompt (CE) · val ROUGE-L=0.408 · OOD cos=0.4731_4_strong_react_with_verify · Prompt (CE) · val ROUGE-L=0.362 · OOD cos=0.4121_4_weak_brief_hint · Prompt (CE) · val ROUGE-L=0.443 · OOD cos=0.3631_4_weak_step_phrase · Prompt (CE) · val ROUGE-L=0.747 · OOD cos=0.3212_1_long_book_jekyll · Prompt (CE) · val ROUGE-L=0.480 · OOD cos=0.5402_1_long_book_time_machine · Prompt (CE) · val ROUGE-L=0.484 · OOD cos=0.4622_1_medium_paper_qasper_A · Prompt (CE) · val ROUGE-L=0.387 · OOD cos=0.5372_1_medium_paper_qasper_B · Prompt (CE) · val ROUGE-L=0.413 · OOD cos=0.5072_1_short_wiki_article_A · Prompt (CE) · val ROUGE-L=0.469 · OOD cos=0.4792_1_short_wiki_article_B · Prompt (CE) · val ROUGE-L=0.517 · OOD cos=0.5102_2_long_tools_agentic_10 · Prompt (CE) · val ROUGE-L=0.552 · OOD cos=0.4492_2_long_tools_general_10 · Prompt (CE) · val ROUGE-L=0.229 · OOD cos=0.4052_2_medium_tools_4_math · Prompt (CE) · val ROUGE-L=0.176 · OOD cos=0.4142_2_medium_tools_4_web · Prompt (CE) · val ROUGE-L=0.420 · OOD cos=0.4072_2_short_tools_1_calculator · Prompt (CE) · val ROUGE-L=0.676 · OOD cos=0.4662_2_short_tools_1_search · Prompt (CE) · val ROUGE-L=0.688 · OOD cos=0.4582_3_long_chat_20turn_alex · Prompt (CE) · val ROUGE-L=0.685 · OOD cos=0.5742_3_long_chat_20turn_user_B · Prompt (CE) · val ROUGE-L=0.590 · OOD cos=0.6042_3_medium_chat_compressed_alex · Prompt (CE) · val ROUGE-L=0.704 · OOD cos=0.4832_3_medium_chat_compressed_user_B · Prompt (CE) · val ROUGE-L=0.623 · OOD cos=0.5032_3_short_chat_4turn_alex · Prompt (CE) · val ROUGE-L=0.759 · OOD cos=0.5352_3_short_chat_4turn_user_B · Prompt (CE) · val ROUGE-L=0.739 · OOD cos=0.4522_4_long_profile_50facts_alex · Prompt (CE) · val ROUGE-L=0.720 · OOD cos=0.5622_4_long_profile_50facts_bob · Prompt (CE) · val ROUGE-L=0.636 · OOD cos=0.4992_4_medium_profile_25facts_alex · Prompt (CE) · val ROUGE-L=0.679 · OOD cos=0.5112_4_medium_profile_25facts_bob · Prompt (CE) · val ROUGE-L=0.680 · OOD cos=0.4692_4_short_profile_5facts_alex · Prompt (CE) · val ROUGE-L=0.718 · OOD cos=0.3822_4_short_profile_5facts_bob · Prompt (CE) · val ROUGE-L=0.736 · OOD cos=0.4292_5_long_kb_50facts_history · Prompt (CE) · val ROUGE-L=0.483 · OOD cos=0.4622_5_long_kb_50facts_science · Prompt (CE) · val ROUGE-L=0.496 · OOD cos=0.4432_5_medium_kb_25facts_history · Prompt (CE) · val ROUGE-L=0.372 · OOD cos=0.4562_5_medium_kb_25facts_science · Prompt (CE) · val ROUGE-L=0.439 · OOD cos=0.4792_5_short_kb_situational_la · Prompt (CE) · val ROUGE-L=0.772 · OOD cos=0.2702_5_short_kb_situational_tokyo · Prompt (CE) · val ROUGE-L=0.666 · OOD cos=0.4313_1_less_summarize_1shot · Prompt (CE) · val ROUGE-L=0.726 · OOD cos=0.6013_1_less_translate_1shot · Prompt (CE) · val ROUGE-L=0.889 · OOD cos=0.5363_1_medium_extraction_5shot · Prompt (CE) · val ROUGE-L=0.912 · OOD cos=0.5303_1_medium_qa_5shot · Prompt (CE) · val ROUGE-L=0.634 · OOD cos=0.5083_1_more_sentiment_pos_neg · Prompt (CE) · val ROUGE-L=0.352 · OOD cos=0.4953_1_more_topic_class_pos_neg · Prompt (CE) · val ROUGE-L=0.655 · OOD cos=0.5741_1_medium_haiku · Prompt (KL) · val ROUGE-L=0.022 · OOD cos=0.4761_1_medium_sonnet · Prompt (KL) · val ROUGE-L=0.004 · OOD cos=0.5131_1_strong_json_strict · Prompt (KL) · val ROUGE-L=0.234 · OOD cos=0.4711_1_strong_xml_strict · Prompt (KL) · val ROUGE-L=0.128 · OOD cos=0.4371_1_weak_concise · Prompt (KL) · val ROUGE-L=0.348 · OOD cos=0.4721_1_weak_max_3_sentences · Prompt (KL) · val ROUGE-L=0.244 · OOD cos=0.4351_2_medium_eli5 · Prompt (KL) · val ROUGE-L=0.059 · OOD cos=0.4341_2_medium_explain_to_executive · Prompt (KL) · val ROUGE-L=0.095 · OOD cos=0.4121_2_strong_expert_lawyer · Prompt (KL) · val ROUGE-L=0.004 · OOD cos=0.3291_2_strong_expert_webdev · Prompt (KL) · val ROUGE-L=0.160 · OOD cos=0.4151_2_weak_formal_tone · Prompt (KL) · val ROUGE-L=0.151 · OOD cos=0.4251_2_weak_friendly_tone · Prompt (KL) · val ROUGE-L=0.086 · OOD cos=0.3771_3_medium_unlearn_5_celebrities · Prompt (KL) · val ROUGE-L=0.214 · OOD cos=0.4011_3_medium_unlearn_5_historical_events · Prompt (KL) · val ROUGE-L=0.566 · OOD cos=0.4531_3_strong_refuse_chem_synthesis · Prompt (KL) · val ROUGE-L=0.316 · OOD cos=0.4341_3_strong_refuse_nuclear_physics · Prompt (KL) · val ROUGE-L=0.276 · OOD cos=0.5011_3_weak_unlearn_pythagoras · Prompt (KL) · val ROUGE-L=0.211 · OOD cos=0.4061_3_weak_unlearn_relativity · Prompt (KL) · val ROUGE-L=0.367 · OOD cos=0.3541_4_medium_scratchpad_with_summary · Prompt (KL) · val ROUGE-L=0.045 · OOD cos=0.3791_4_medium_thinking_block · Prompt (KL) · val ROUGE-L=0.197 · OOD cos=0.3191_4_strong_labeled_steps · Prompt (KL) · val ROUGE-L=0.107 · OOD cos=0.3711_4_strong_react_with_verify · Prompt (KL) · val ROUGE-L=0.032 · OOD cos=0.3881_4_weak_brief_hint · Prompt (KL) · val ROUGE-L=0.140 · OOD cos=0.1341_4_weak_step_phrase · Prompt (KL) · val ROUGE-L=0.197 · OOD cos=0.1742_1_long_book_jekyll · Prompt (KL) · val ROUGE-L=0.449 · OOD cos=0.4882_1_long_book_time_machine · Prompt (KL) · val ROUGE-L=0.477 · OOD cos=0.4282_1_medium_paper_qasper_A · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.4212_1_medium_paper_qasper_B · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.4272_1_short_wiki_article_A · Prompt (KL) · val ROUGE-L=0.007 · OOD cos=0.4052_1_short_wiki_article_B · Prompt (KL) · val ROUGE-L=0.004 · OOD cos=0.4132_2_long_tools_agentic_10 · Prompt (KL) · val ROUGE-L=0.004 · OOD cos=0.3982_2_long_tools_general_10 · Prompt (KL) · val ROUGE-L=0.003 · OOD cos=0.4102_2_medium_tools_4_math · Prompt (KL) · val ROUGE-L=0.001 · OOD cos=0.4292_2_medium_tools_4_web · Prompt (KL) · val ROUGE-L=0.069 · OOD cos=0.3932_2_short_tools_1_calculator · Prompt (KL) · val ROUGE-L=0.044 · OOD cos=0.3522_2_short_tools_1_search · Prompt (KL) · val ROUGE-L=0.001 · OOD cos=0.4462_3_long_chat_20turn_alex · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.3482_3_long_chat_20turn_user_B · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.4012_3_medium_chat_compressed_alex · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.4402_3_medium_chat_compressed_user_B · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.3762_3_short_chat_4turn_alex · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.3372_3_short_chat_4turn_user_B · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.3952_4_long_profile_50facts_alex · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.4042_4_long_profile_50facts_bob · Prompt (KL) · val ROUGE-L=0.001 · OOD cos=0.3712_4_medium_profile_25facts_alex · Prompt (KL) · val ROUGE-L=0.002 · OOD cos=0.3542_4_medium_profile_25facts_bob · Prompt (KL) · val ROUGE-L=0.000 · OOD cos=0.3722_4_short_profile_5facts_alex · Prompt (KL) · val ROUGE-L=0.006 · OOD cos=0.2772_4_short_profile_5facts_bob · Prompt (KL) · val ROUGE-L=0.011 · OOD cos=0.2852_5_long_kb_50facts_history · Prompt (KL) · val ROUGE-L=0.002 · OOD cos=0.3202_5_long_kb_50facts_science · Prompt (KL) · val ROUGE-L=0.002 · OOD cos=0.3192_5_medium_kb_25facts_history · Prompt (KL) · val ROUGE-L=0.028 · OOD cos=0.4212_5_medium_kb_25facts_science · Prompt (KL) · val ROUGE-L=0.041 · OOD cos=0.3862_5_short_kb_situational_la · Prompt (KL) · val ROUGE-L=0.001 · OOD cos=0.3432_5_short_kb_situational_tokyo · Prompt (KL) · val ROUGE-L=0.015 · OOD cos=0.3143_1_less_summarize_1shot · Prompt (KL) · val ROUGE-L=0.172 · OOD cos=0.4993_1_less_translate_1shot · Prompt (KL) · val ROUGE-L=0.069 · OOD cos=0.5143_1_medium_extraction_5shot · Prompt (KL) · val ROUGE-L=0.171 · OOD cos=0.5353_1_medium_qa_5shot · Prompt (KL) · val ROUGE-L=0.184 · OOD cos=0.4293_1_more_sentiment_pos_neg · Prompt (KL) · val ROUGE-L=0.410 · OOD cos=0.3633_1_more_topic_class_pos_neg · Prompt (KL) · val ROUGE-L=0.104 · OOD cos=0.482
Full-FT (CE) Full-FT (KL) LoRA (CE) LoRA (top-k KL) Prompt (CE) Prompt (KL)

[F6] Each condition's distance from base, per layer — Group A (output-form-specifying) OOD

Per-layer 1 − cos(h·, hbase) on OOD (MMLU-Pro) queries, averaged across the 36 Group A instances only (output-form-specifying contexts — these are the ones whose effect should generalize across query domain, since they constrain HOW the model answers, not WHAT it knows). One line per condition incl. in-context. ROUGE-L vs base would require a fresh OOD-text inference sweep; currently only the activation cosine is available.
Val (in-domain held-out)0.00.10.20.30.40.508162431layer1 − cos(h_·, h_base)
OOD (MMLU-Pro)0.00.10.20.30.40.508162431layer1 − cos(h_·, h_base)
in-context (teacher) Full-FT (CE) Full-FT (KL) LoRA (CE) LoRA (top-k KL) Prompt (CE) Prompt (KL)

6. Failure modes / outliers

Unlearn / refuse over-generalization (LoRA): 4 instances cause distilled MMLU to collapse to ≈chance (11%) — the adapter learns to refuse, then refuses MMLU queries on unrelated topics:

7. Metrics in the per-instance table

P3 ex — Phase 3 exact-match rate
Of the 200 training queries, fraction where the distilled student's greedy completion exactly matches the teacher's. Strong upper-bound signal: distilled has memorized the teacher's output token-for-token. Higher = student is reproducing teacher behavior.
P3 p≥5 — Phase 3 prefix-match rate (≥5 tokens)
Fraction of 200 training queries where the distilled student matches the teacher on at least the first 5 generated tokens. Less strict than exact match — captures whether the student starts answers the same way, even if it diverges later.
ROUGE-L vs IC — Phase 3c ROUGE-L between this condition's output and the in-context teacher's output, on 40 held-out validation queries
On 40 held-out validation queries, ROUGE-L(this condition's output, in-context teacher's output). Raw similarity score in [0, 1]. IC vs IC = 1.0 trivially. Base typically 0.4–0.6 (the model shares baseline capabilities even without the context). Higher distilled values = closer behavior fidelity to in-context teacher.
MMLU dis — Phase 5 MMLU-Pro accuracy, distilled student
Accuracy on 294 MMLU-Pro questions (multiple-choice, 10 options) when querying the distilled student with no context. Tells us whether the adapter preserves general capability.
Δ MMLU — Phase 5 distilled minus base, in percentage points
(MMLU_distilled − MMLU_base) × 100. Base Qwen3.5-9B scores ~54.4% on MMLU-Pro. Negative = the adapter has hurt general capability (the price of being more context-faithful). Positive = the adapter ended up better than base on this benchmark even with no context shown.
cosΔ val — mean cosine of shift directions, val split
Per-layer cosine between vC=hC−hbase and vD=hD−hbase, averaged across all 32 layers on the val (held-out, in-distribution) split. 1.0 = the adapter pushes residual-stream activations in the same direction the context does. Magnitudes (‖vC‖, ‖vD‖) are shown separately in §4.

8. Per-instance details

Show 60-row per-instance metrics table
InstanceK LoRA (CE) Full-FT Prompt-tuning LORA_KL FULL_KL PREFIX_KL
P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val P3 ex P3 p≥5 P3c gap MMLU dis Δ MMLU cosΔ val
1_1_medium_haiku1 71.0%86.5%0.50334.4%-20.1pp0.672 88.5%94.0%0.44719.0%-35.4pp0.615 0.5%17.0%0.44249.0%-5.4pp0.612 15.0%55.5%0.60944.6%-9.9pp0.700 8.0%46.5%0.50241.8%-12.6pp0.611 0.0%0.0%0.02221.1%-33.3pp0.489
1_1_medium_sonnet1 9.5%72.5%0.30315.6%-38.8pp0.687 58.0%84.5%0.30814.6%-39.8pp0.564 0.0%29.5%0.28953.4%-1.0pp0.705 0.0%49.5%0.34725.9%-28.6pp0.701 0.0%51.0%0.31113.9%-40.5pp0.604 0.0%0.0%0.00419.7%-34.7pp0.454
1_1_strong_json_strict1 66.5%86.0%0.82115.0%-39.5pp0.491 58.5%90.0%0.76715.0%-39.5pp0.356 28.0%77.5%0.80250.3%-4.1pp0.511 53.0%87.5%0.89215.0%-39.5pp0.399 28.5%82.5%0.73615.0%-39.5pp0.338 0.0%0.0%0.23452.7%-1.7pp0.464
1_1_strong_xml_strict1 81.5%100.0%0.82311.9%-42.5pp0.452 69.5%99.5%0.75311.9%-42.5pp0.526 34.0%96.0%0.81142.2%-12.2pp0.563 53.5%98.5%0.90312.2%-42.2pp0.421 28.0%98.0%0.73311.9%-42.5pp0.505 0.0%0.0%0.12844.9%-9.5pp0.439
1_1_weak_concise1 88.5%92.0%0.83656.8%+2.4pp0.530 82.5%92.5%0.78054.4%+0.0pp0.424 41.5%78.5%0.82453.7%-0.7pp0.548 65.0%88.0%0.89357.1%+2.7pp0.632 27.5%68.5%0.73356.5%+2.0pp0.302 2.0%9.0%0.34843.5%-10.9pp0.387
1_1_weak_max_3_sentences1 80.0%92.0%0.73554.1%-0.3pp0.559 65.5%86.5%0.64655.4%+1.0pp0.453 28.0%76.5%0.71454.4%+0.0pp0.620 45.5%85.5%0.79352.7%-1.7pp0.631 20.0%71.0%0.64355.4%+1.0pp0.444 0.0%3.0%0.24450.3%-4.1pp0.383
1_2_medium_eli51 52.0%91.0%0.54353.1%-1.4pp0.645 56.0%90.0%0.48955.8%+1.4pp0.440 17.0%55.0%0.53957.1%+2.7pp0.589 20.0%81.0%0.57855.8%+1.4pp0.711 11.0%56.0%0.49556.1%+1.7pp0.453 0.0%0.0%0.05946.9%-7.5pp0.384
1_2_medium_explain_to_executive1 51.5%94.5%0.57356.5%+2.0pp0.476 60.0%93.5%0.50056.1%+1.7pp0.402 17.5%81.5%0.54453.1%-1.4pp0.573 22.0%90.5%0.59155.1%+0.7pp0.572 14.0%70.0%0.46450.0%-4.4pp0.290 0.0%0.0%0.09545.2%-9.2pp0.360
1_2_strong_expert_lawyer1 1.5%67.5%0.38554.8%+0.3pp0.539 11.5%66.0%0.34954.1%-0.3pp0.467 0.0%47.5%0.30452.0%-2.4pp0.634 0.0%55.5%0.38755.4%+1.0pp0.590 0.0%57.0%0.35652.0%-2.4pp0.449 0.0%0.5%0.00448.0%-6.5pp0.373
1_2_strong_expert_webdev1 82.0%97.0%0.71054.8%+0.3pp0.466 78.5%99.0%0.64454.1%-0.3pp0.326 27.5%78.0%0.74853.4%-1.0pp0.520 45.0%89.5%0.79254.8%+0.3pp0.580 19.0%75.0%0.57955.1%+0.7pp0.344 0.0%2.0%0.16043.9%-10.5pp0.331
1_2_weak_formal_tone1 82.5%99.5%0.79853.7%-0.7pp0.492 78.5%98.5%0.70256.1%+1.7pp0.351 19.0%70.0%0.70850.0%-4.4pp0.511 42.5%88.0%0.83956.5%+2.0pp0.533 25.0%81.0%0.67953.7%-0.7pp0.297 0.0%0.5%0.15121.4%-33.0pp0.343
1_2_weak_friendly_tone1 43.5%86.0%0.60855.4%+1.0pp0.569 48.0%76.5%0.56256.8%+2.4pp0.448 18.0%67.0%0.62057.1%+2.7pp0.613 20.5%67.0%0.65255.1%+0.7pp0.701 10.5%59.5%0.53955.4%+1.0pp0.367 0.0%0.0%0.08648.0%-6.5pp0.377
1_3_medium_unlearn_5_celebrities1 100.0%100.0%0.01711.2%-43.2pp0.267 100.0%100.0%0.01711.2%-43.2pp0.305 100.0%100.0%0.21151.0%-3.4pp0.451 100.0%100.0%0.01711.2%-43.2pp0.272 0.0%0.0%0.01310.2%-44.2pp0.247 0.0%0.0%0.21448.0%-6.5pp0.420
1_3_medium_unlearn_5_historical_events1 99.0%99.5%0.01411.2%-43.2pp0.319 99.0%99.5%0.01411.2%-43.2pp0.348 99.0%99.5%0.10348.3%-6.1pp0.438 99.0%99.5%0.01411.2%-43.2pp0.330 0.0%0.0%0.00111.2%-43.2pp0.315 0.0%0.0%0.56648.3%-6.1pp0.412
1_3_strong_refuse_chem_synthesis1 2.6%84.6%0.79553.4%-1.0pp0.413 7.7%79.5%0.65343.2%-11.2pp0.350 0.0%82.9%0.74353.1%-1.4pp0.541 5.3%78.9%0.77755.1%+0.7pp0.400 0.0%64.1%0.59547.6%-6.8pp0.332 0.0%5.3%0.31639.1%-15.3pp0.407
1_3_strong_refuse_nuclear_physics1 23.3%90.0%0.68956.5%+2.0pp0.379 16.7%83.3%0.66053.1%-1.4pp0.343 0.0%48.4%0.70448.6%-5.8pp0.487 0.0%66.7%0.69257.1%+2.7pp0.323 0.0%65.4%0.68653.7%-0.7pp0.309 0.0%0.0%0.27645.2%-9.2pp0.429
1_3_weak_unlearn_pythagoras1 86.5%87.0%0.01911.2%-43.2pp0.278 86.5%87.0%0.04411.2%-43.2pp0.264 86.5%87.0%0.24248.3%-6.1pp0.432 85.5%87.0%0.01911.2%-43.2pp0.267 84.5%88.0%0.01911.2%-43.2pp0.208 0.0%3.0%0.21138.8%-15.6pp0.397
1_3_weak_unlearn_relativity1 82.0%83.0%0.02211.2%-43.2pp0.306 81.5%81.5%0.01911.2%-43.2pp0.331 81.5%83.0%0.31352.4%-2.0pp0.433 81.5%82.0%0.06511.2%-43.2pp0.384 81.5%81.5%0.01911.2%-43.2pp0.313 0.0%4.5%0.36751.7%-2.7pp0.313
1_4_medium_scratchpad_with_summary1 56.0%92.0%0.50825.2%-29.3pp0.348 32.0%89.5%0.4699.2%-45.2pp0.383 16.5%84.5%0.35429.9%-24.5pp0.522 42.0%93.5%0.5989.2%-45.2pp0.431 17.5%89.5%0.4919.2%-45.2pp0.368 0.0%0.0%0.04550.0%-4.4pp0.438
1_4_medium_thinking_block1 57.0%95.5%0.49942.5%-11.9pp0.281 26.0%82.0%0.37943.2%-11.2pp0.254 15.5%89.0%0.41851.4%-3.1pp0.372 34.0%92.0%0.61013.6%-40.8pp0.343 1.0%22.5%0.45135.0%-19.4pp0.250 0.0%0.0%0.19731.0%-23.5pp0.406
1_4_strong_labeled_steps1 25.0%85.0%0.47513.9%-40.5pp0.398 19.0%73.5%0.51311.9%-42.5pp0.487 0.5%67.5%0.40854.8%+0.3pp0.516 14.0%80.0%0.54812.2%-42.2pp0.428 8.5%73.0%0.50812.2%-42.2pp0.459 0.0%0.0%0.10743.5%-10.9pp0.434
1_4_strong_react_with_verify1 31.5%91.0%0.57012.2%-42.2pp0.380 16.5%86.0%0.49211.9%-42.5pp0.443 4.0%79.5%0.36250.0%-4.4pp0.457 15.5%85.0%0.58612.2%-42.2pp0.454 4.0%84.0%0.55412.2%-42.2pp0.453 0.0%0.0%0.03213.9%-40.5pp0.429
1_4_weak_brief_hint1 57.5%97.5%0.40953.4%-1.0pp0.471 33.0%93.5%0.37636.1%-18.4pp0.209 31.5%96.0%0.44356.1%+1.7pp0.451 44.0%96.0%0.45754.4%+0.0pp0.487 11.5%90.0%0.42045.9%-8.5pp0.319 0.0%0.0%0.14023.1%-31.3pp0.239
1_4_weak_step_phrase1 63.5%97.5%0.74654.8%+0.3pp0.394 32.0%95.5%0.48743.2%-11.2pp0.210 36.5%93.5%0.74753.1%-1.4pp0.450 38.5%97.0%0.80555.1%+0.7pp0.465 12.5%90.5%0.41842.2%-12.2pp0.201 0.0%0.0%0.19751.4%-3.1pp0.248
2_1_long_book_jekyll2 0.5%88.5%0.45254.1%-0.3pp0.352 12.5%91.5%0.46452.4%-2.0pp— 1.5%94.5%0.48048.0%-6.5pp0.566 0.0%90.5%0.46855.8%+1.4pp0.350 0.5%90.0%0.34434.4%-20.1pp— 0.0%2.5%0.44949.0%-5.4pp0.494
2_1_long_book_time_machine2 0.5%91.5%0.50855.8%+1.4pp0.352 12.5%88.5%0.51239.8%-14.6pp— 2.5%92.5%0.48445.9%-8.5pp0.487 0.0%86.5%0.52355.8%+1.4pp0.361 0.0%87.5%0.49943.9%-10.5pp— 0.0%9.0%0.47752.0%-2.4pp0.474
2_1_medium_paper_qasper_A2 10.5%68.5%0.38354.1%-0.3pp0.390 21.0%70.5%0.40548.3%-6.1pp0.249 2.5%68.0%0.38742.2%-12.2pp0.541 1.0%66.0%0.36455.4%+1.0pp0.365 3.0%69.0%0.38551.4%-3.1pp0.286 0.0%0.0%0.00011.6%-42.9pp0.422
2_1_medium_paper_qasper_B2 6.0%58.0%0.34951.4%-3.1pp0.381 20.5%62.0%0.39552.0%-2.4pp0.292 2.0%56.0%0.41340.1%-14.3pp0.541 0.0%46.0%0.37653.4%-1.0pp0.381 1.0%57.0%0.36453.4%-1.0pp0.276 0.0%0.0%0.0008.8%-45.6pp0.400
2_1_short_wiki_article_A2 7.6%78.8%0.45954.1%-0.3pp0.380 26.4%81.2%0.42954.8%+0.3pp0.202 2.5%87.3%0.46946.9%-7.5pp0.488 6.0%83.9%0.45055.4%+1.0pp0.364 4.1%78.7%0.47456.8%+2.4pp0.260 0.0%0.0%0.00731.6%-22.8pp0.378
2_1_short_wiki_article_B2 7.3%87.5%0.48456.1%+1.7pp0.407 24.4%87.6%0.40256.5%+2.0pp0.228 0.0%81.2%0.51745.6%-8.8pp0.565 0.5%82.8%0.46853.1%-1.4pp0.418 2.1%85.3%0.41959.2%+4.8pp0.253 0.0%0.0%0.00411.2%-43.2pp0.457
2_2_long_tools_agentic_102 74.0%92.5%0.30946.9%-7.5pp0.444 81.5%94.0%0.25810.5%-43.9pp0.440 64.5%92.5%0.55250.3%-4.1pp0.481 51.5%91.0%0.57450.3%-4.1pp0.456 36.5%92.5%0.33422.1%-32.3pp0.429 0.0%0.0%0.00439.1%-15.3pp0.410
2_2_long_tools_general_102 76.4%88.9%0.40150.7%-3.7pp0.465 82.0%91.5%0.2898.8%-45.6pp0.332 64.0%91.0%0.22927.9%-26.5pp0.459 70.0%91.0%0.19212.9%-41.5pp0.442 53.0%88.0%0.2677.1%-47.3pp0.348 0.0%0.0%0.00317.7%-36.7pp0.333
2_2_medium_tools_4_math2 76.8%89.4%0.19811.2%-43.2pp0.416 82.2%88.8%0.2169.9%-44.6pp0.352 81.9%90.7%0.17645.9%-8.5pp0.439 81.1%90.8%0.30412.2%-42.2pp0.399 81.0%94.0%0.02511.2%-43.2pp0.358 0.0%0.0%0.00127.6%-26.9pp0.443
2_2_medium_tools_4_web2 78.5%96.5%0.65053.7%-0.7pp0.349 82.0%97.0%0.60345.9%-8.5pp0.275 59.5%94.5%0.42051.4%-3.1pp0.421 67.0%95.5%0.3229.5%-44.9pp0.443 1.5%5.5%0.37147.6%-6.8pp0.390 0.0%0.0%0.06937.1%-17.3pp0.431
2_2_short_tools_1_calculator2 41.2%90.7%0.68353.1%-1.4pp0.413 49.2%85.3%0.43812.6%-41.8pp0.250 26.5%92.9%0.67651.7%-2.7pp0.520 26.3%82.5%0.67556.1%+1.7pp0.374 28.6%86.2%0.35310.2%-44.2pp0.306 0.0%0.0%0.04437.1%-17.3pp0.383
2_2_short_tools_1_search2 77.6%94.8%0.75257.5%+3.1pp0.417 60.3%88.4%0.74737.4%-17.0pp0.284 27.4%67.4%0.68853.1%-1.4pp0.501 56.2%82.5%0.75554.8%+0.3pp0.553 28.0%78.2%0.74253.7%-0.7pp0.392 0.0%0.0%0.00125.2%-29.3pp0.446
2_3_long_chat_20turn_alex2 3.6%54.8%0.68057.1%+2.7pp0.425 11.4%62.7%0.70756.8%+2.4pp0.357 2.0%43.4%0.68551.7%-2.7pp0.643 0.0%46.9%0.75655.8%+1.4pp0.479 0.0%42.9%0.70053.4%-1.0pp0.299 0.0%0.0%0.0001.4%-53.1pp0.308
2_3_long_chat_20turn_user_B2 2.0%59.3%0.59953.1%-1.4pp0.507 8.5%67.0%0.55154.8%+0.3pp0.264 0.5%52.0%0.59052.7%-1.7pp0.617 0.0%45.0%0.59654.8%+0.3pp0.505 0.0%44.0%0.52556.5%+2.0pp0.331 0.0%0.0%0.00021.8%-32.7pp0.352
2_3_medium_chat_compressed_alex2 4.0%60.5%0.71858.2%+3.7pp0.434 15.5%62.5%0.69257.1%+2.7pp0.376 0.0%36.5%0.70453.7%-0.7pp0.588 0.0%50.0%0.77355.8%+1.4pp0.568 0.0%42.5%0.69055.4%+1.0pp0.367 0.0%0.0%0.00024.1%-30.3pp0.387
2_3_medium_chat_compressed_user_B2 1.5%53.5%0.65955.1%+0.7pp0.453 7.0%50.5%0.67753.1%-1.4pp0.339 0.0%39.5%0.62353.4%-1.0pp0.649 0.0%44.5%0.62655.8%+1.4pp0.569 0.0%47.0%0.58855.8%+1.4pp0.296 0.0%0.0%0.00036.1%-18.4pp0.352
2_3_short_chat_4turn_alex2 5.6%64.8%0.72954.4%+0.0pp0.350 9.8%61.9%0.73957.5%+3.1pp0.287 0.0%43.1%0.75954.4%+0.0pp0.591 1.0%51.3%0.78356.5%+2.0pp0.486 0.0%50.0%0.69155.4%+1.0pp0.271 0.0%0.0%0.00040.5%-13.9pp0.217
2_3_short_chat_4turn_user_B2 2.1%51.5%0.70150.0%-4.4pp0.393 11.3%62.4%0.70654.4%+0.0pp0.235 0.0%53.8%0.73952.7%-1.7pp0.595 0.0%56.5%0.73854.8%+0.3pp0.470 0.0%44.8%0.61355.1%+0.7pp0.254 0.0%0.0%0.00016.0%-38.4pp0.341
2_4_long_profile_50facts_alex2 3.5%65.5%0.64054.4%+0.0pp0.422 11.5%70.5%0.72755.4%+1.0pp0.310 0.0%58.0%0.72052.0%-2.4pp0.557 0.0%49.5%0.68454.8%+0.3pp0.506 0.0%56.5%0.72257.8%+3.4pp0.319 0.0%0.0%0.0003.4%-51.0pp0.371
2_4_long_profile_50facts_bob2 1.0%60.5%0.70951.7%-2.7pp0.423 9.0%66.5%0.72355.4%+1.0pp0.272 0.0%56.0%0.63651.4%-3.1pp0.627 0.0%48.5%0.63854.1%-0.3pp0.469 0.0%45.5%0.69755.4%+1.0pp0.293 0.0%0.0%0.00118.0%-36.4pp0.328
2_4_medium_profile_25facts_alex2 6.2%56.2%0.71755.8%+1.4pp0.427 11.1%63.7%0.75157.8%+3.4pp0.352 0.5%45.1%0.67952.7%-1.7pp0.539 0.5%49.0%0.73556.5%+2.0pp0.531 0.0%45.4%0.71253.4%-1.0pp0.348 0.0%0.0%0.00210.5%-43.9pp0.339
2_4_medium_profile_25facts_bob2 0.0%53.0%0.68955.8%+1.4pp0.457 10.5%58.0%0.69755.8%+1.4pp0.279 0.0%40.5%0.68054.8%+0.3pp0.597 0.0%39.5%0.67556.5%+2.0pp0.498 0.0%44.0%0.67853.1%-1.4pp0.310 0.0%0.0%0.0009.9%-44.6pp0.364
2_4_short_profile_5facts_alex2 7.0%68.8%0.75955.4%+1.0pp0.340 13.1%56.3%0.74555.4%+1.0pp0.359 0.0%42.7%0.71855.8%+1.4pp0.538 1.0%61.3%0.81455.4%+1.0pp0.534 0.0%51.8%0.71155.1%+0.7pp0.304 0.0%0.0%0.00631.6%-22.8pp0.204
2_4_short_profile_5facts_bob2 4.5%59.8%0.75153.4%-1.0pp0.392 9.5%54.3%0.71253.1%-1.4pp0.312 0.0%52.8%0.73655.8%+1.4pp0.554 0.0%63.5%0.80054.1%-0.3pp0.507 0.5%49.5%0.68053.7%-0.7pp0.269 0.0%0.0%0.01124.5%-29.9pp0.319
2_5_long_kb_50facts_history2 3.0%95.0%0.53652.4%-2.0pp0.386 23.0%93.5%0.51956.8%+2.4pp0.343 0.0%94.0%0.48352.7%-1.7pp0.456 0.0%95.0%0.55353.7%-0.7pp0.430 1.0%91.5%0.40253.1%-1.4pp0.301 0.0%0.0%0.00219.4%-35.0pp0.316
2_5_long_kb_50facts_science2 2.4%72.0%0.53152.7%-1.7pp0.352 10.4%87.2%0.51654.8%+0.3pp0.299 0.0%75.4%0.49652.7%-1.7pp0.468 1.3%80.0%0.48256.1%+1.7pp0.327 1.1%86.3%0.53348.0%-6.5pp0.299 0.0%0.0%0.0026.8%-47.6pp0.324
2_5_medium_kb_25facts_history2 81.5%84.3%0.47953.7%-0.7pp0.304 56.5%83.2%0.46435.4%-19.0pp0.238 35.8%72.4%0.37244.6%-9.9pp0.479 27.0%68.0%0.52055.8%+1.4pp0.302 42.7%75.8%0.45632.3%-22.1pp0.306 0.0%0.0%0.02817.0%-37.4pp0.449
2_5_medium_kb_25facts_science2 9.4%85.2%0.49856.8%+2.4pp0.368 19.0%85.0%0.45852.4%-2.0pp0.293 2.7%52.0%0.43948.6%-5.8pp0.455 2.0%72.0%0.46054.4%+0.0pp0.357 15.4%83.7%0.49853.1%-1.4pp0.319 0.0%0.0%0.04144.9%-9.5pp0.370
2_5_short_kb_situational_la2 21.8%75.6%0.79157.8%+3.4pp0.467 27.4%70.7%0.72852.7%-1.7pp0.341 0.6%52.8%0.77256.1%+1.7pp0.522 2.6%66.1%0.79756.8%+2.4pp0.483 0.0%50.3%0.49954.8%+0.3pp0.285 0.0%0.0%0.00119.0%-35.4pp0.336
2_5_short_kb_situational_tokyo2 22.9%78.2%0.78455.1%+0.7pp0.445 25.5%66.2%0.70956.5%+2.0pp0.302 1.4%29.1%0.66652.4%-2.0pp0.547 4.1%57.3%0.79456.8%+2.4pp0.548 0.0%41.4%0.68055.1%+0.7pp0.280 0.0%0.0%0.01519.4%-35.0pp0.296
3_1_less_summarize_1shot3 61.2%73.0%0.70257.1%+2.7pp0.488 66.1%80.7%0.67456.1%+1.7pp0.427 13.0%40.5%0.72653.4%-1.0pp0.532 16.2%57.6%0.71157.1%+2.7pp0.504 13.7%48.4%0.55357.1%+2.7pp0.417 0.0%0.5%0.17223.5%-31.0pp0.404
3_1_less_translate_1shot3 82.5%93.0%0.94455.1%+0.7pp0.508 84.5%91.5%0.90311.9%-42.5pp0.340 50.5%82.5%0.88955.4%+1.0pp0.590 69.0%86.5%0.95458.5%+4.1pp0.544 43.5%78.5%0.88610.9%-43.5pp0.308 0.0%0.0%0.06945.2%-9.2pp0.481
3_1_medium_extraction_5shot3 91.5%93.0%0.65954.8%+0.3pp0.331 89.9%91.0%0.41955.1%+0.7pp— 87.9%95.5%0.91246.9%-7.5pp0.558 95.5%96.5%0.83115.0%-39.5pp0.377 95.0%100.0%0.78615.0%-39.5pp0.309 0.0%0.0%0.17127.9%-26.5pp0.561
3_1_medium_qa_5shot3 96.9%46.9%0.65550.7%-3.7pp0.432 95.9%46.9%0.70554.1%-0.3pp0.283 74.3%47.1%0.63449.7%-4.8pp0.589 87.9%47.7%0.73053.1%-1.4pp0.503 86.0%42.6%0.60957.1%+2.7pp0.472 0.0%0.0%0.18450.3%-4.1pp0.443
3_1_more_sentiment_pos_neg3 90.4%0.5%0.61655.4%+1.0pp0.503 89.9%2.5%0.61756.5%+2.0pp0.368 92.5%0.0%0.35249.7%-4.8pp0.521 79.9%0.0%0.40155.1%+0.7pp0.478 83.5%0.0%0.46513.9%-40.5pp0.369 2.0%0.5%0.41052.0%-2.4pp0.475
3_1_more_topic_class_pos_neg3 88.8%2.6%0.73053.4%-1.0pp0.388 96.5%1.5%0.6909.2%-45.2pp0.306 94.0%1.5%0.65548.6%-5.8pp0.580 96.0%2.5%0.41946.3%-8.2pp0.363 94.5%2.5%0.75042.2%-12.2pp0.307 1.0%0.0%0.10450.0%-4.4pp0.489