Deck 5 — Context Management & Reliability (Domain 5, 15%)
45 cards. Cards 1–20 = context. 21–32 = escalation and error propagation. 33–45 = review, calibration, provenance.
1
Q: 🎯 Four categories progressive summarization destroys? A: Specific numbers · percentages · dates · customer-stated expectations.
2
Q: Which of those four is most consequential and least obvious? A: Customer-stated expectations — a commitment the agent must honour, mentioned once in prose and read as filler.
3
Q: The fix? A: A persistent structured "case facts" block kept outside the summarized history and re-injected every turn.
4
Q: Four properties that make the case-facts block work? A: Structured · outside the summarizable region · small · updated as facts are established (not reconstructed).
5
Q: "Use a larger context window so summarization isn't needed" — why wrong? A: It defers the limit and does nothing about position effects. The same facts end up buried in the middle.
6
Q: "Summarize less aggressively" — why wrong? A: Vague, costlier every turn, and loses the same categories later rather than never.
7
Q: A tool returns 40 fields when 5 are relevant. Cost? A: ~8× more context than needed, on every call, compounding.
8
Q: Fix?
A: Trim verbose tool outputs before they enter the conversation — a natural PostToolUse job.
9
Q: Trim history or trim payloads? A: Keep the full turn structure; shrink the payloads inside it. Dropping turns loses commitments and causes repeated tool calls.
10
Q: Name the position effect. A: "Lost in the middle" — middle-of-context content receives less effective attention.
11
Q: Is lost-in-the-middle a capacity problem? A: No — the content is present and readable. It's attention distribution. More capacity doesn't help.
12
Q: Two named mitigations? A: Key findings first and explicit section headers.
13
Q: Both mitigations are about what? A: Arrangement, not volume.
14
Q: A subagent returns three paragraphs of reasoning plus its findings. Problem and fix?
A: The coordinator's context fills with reasoning it doesn't need and the findings land in the middle. Fix: specify structured returns — {claim, source, date}, not an essay.
15
Q: 🎯 Diagnostic signal for context degradation in a codebase task? A: The model starts citing "typical patterns" and generic descriptions instead of specific classes, functions, and files.
16
Q: Four mechanisms for large-codebase context?
A: Scratchpad files · subagent delegation · structured state exports/manifests · /compact.
17
Q: What does a scratchpad file buy you over holding findings in context? A: It survives compaction, is selectively re-readable, is human-auditable, and carries across sessions.
18
Q: An agent "loses track of earlier findings" during a long codebase task. Answer? A: Scratchpad file (plus subagent delegation for the verbose discovery).
19
Q: 🎯 What must a structured state manifest carry beyond progress? A: The decisions and conventions established. A resumed run that doesn't know them produces inconsistent work.
20
Q: Which mechanism compresses the conversation in place, same session?
A: /compact.
21
Q: 🎯🎯 The three legitimate escalation triggers? A: The customer requests a human · a policy exception or gap · inability to make progress.
22
Q: Is complexity a trigger? A: No. Complex cases are what the agent is for; escalating them defeats the 80% resolution goal.
23
Q: Is negative sentiment a trigger? A: No. Frustration ≠ complexity. A frustrated customer with a simple problem should get it solved.
24
Q: Is low self-reported confidence a trigger? A: No. Uncalibrated.
25
Q: The customer says "let me talk to a person." What do you do? A: Escalate immediately — no resolution attempt first, no asking them to re-explain.
26
Q: The distractor for that item? A: "Attempt to resolve first, since most requests can be handled automatically." It optimizes the metric against the customer's explicit wish.
27
Q: Customer lookup returns three possible matches. Action? A: Ask for an additional identifying detail — email, ZIP. Never guess.
28
Q: Why is guessing disqualifying rather than just suboptimal? A: Downstream actions are refunds and order changes against a specific account. A wrong guess takes an irreversible action on the wrong person's account.
29
Q: Four elements of structured error context crossing an agent boundary? A: Failure type/category · what was attempted · partial results · suggested alternatives.
30
Q: Why is a generic "search unavailable" wrong?
A: It hides category, retryability, the scope of the failure, and what succeeded.
31
Q: Two symmetric error anti-patterns? A: Silent suppression (downstream builds on a false premise) and terminating the whole workflow (discards successful work).
32
Q: The concrete form of "annotate the gap"? A: A coverage note in the report stating which section is incomplete and why.
33
Q: 🎯 A pipeline reports 97% aggregate accuracy. Why isn't that enough to automate? A: Aggregate accuracy masks per-type failure. 97% overall is consistent with 99% on the dominant type and 43% on a rare one.
34
Q: What must you measure instead? A: Accuracy per document type AND per field — the cross-product.
35
Q: How do you then automate? A: Per cell, not globally. Automate the reliable (type, field) combinations; route the rest to humans.
36
Q: Which sampling method for the human review that remains? A: Stratified random sampling — within each stratum (type, vendor, field).
37
Q: Why not uniform random 5%? A: In a corpus that's 85% invoices you get ~4 statements out of 50 — far too few to measure a 43%-accurate category.
38
Q: Why not "review the first N of each batch"? A: Not random. Batches may be ordered by source, time, or type, which biases the sample.
39
Q: When does self-reported confidence become a usable routing signal? A: Once calibrated per field on a labeled validation set, with the threshold derived from measured accuracy at that confidence level.
40
Q: Which extractions always go to human review regardless of confidence?
A: Any with conflict_detected: true.
41
Q: 🎯 Where must claim→source mappings be preserved? A: Through synthesis — not reattached afterward, which would mean guessing which source supported which sentence.
42
Q: Two credible sources give different revenue figures. Action? A: Report both with attribution. Don't pick, don't average, don't take the newest.
43
Q: Three distractors for that item? A: "Select the more authoritative source" · "use the most recent figure" · "average the values."
44
Q: Which two date fields prevent temporal differences being misread as contradictions? A: Publication date (when the source said it) and collection date (when we retrieved it).
45
Q: Rendering format by data type? A: Financial → tables. News/narrative → prose. Technical findings → lists.