Question Bank — Domain 5: Context Management & Reliability
20 items. 15% of the exam (~9 items). Heavily overlapping with Domains 1 and 4 — the study here compounds.
Block A — Context management (items 1–8)
1
A support agent handling long conversations occasionally offers refund amounts inconsistent with what was agreed earlier, and sometimes fails to honour commitments the customer says they received from phone support. Earlier turns are summarized to manage context. Most effective fix?
- A. Maintain a structured "case facts" block containing verified IDs, agreed amounts, dates, and customer-stated expectations, kept outside the summarized history and re-injected each turn.
- B. Use a model with a larger context window so summarization isn't required.
- C. Summarize less aggressively, retaining more of each earlier turn.
- D. Have the agent re-read the full transcript before every response.
Answer
A.
The stem names exactly the four categories summarization destroys: numbers, percentages, dates, and customer-stated expectations. A small structured block outside the compressible region protects precisely those and keeps them positionally prominent.
- B — Defers the limit and does nothing about position effects; in a long enough conversation the facts end up buried in the middle anyway.
- C — Vague, costlier every turn, and loses the same categories later rather than never.
- D — If the transcript is being summarized because it doesn't fit, re-reading it isn't available. If it did fit, you're back to lost-in-the-middle.
Reflex 2.
2
Which four categories of information does progressive summarization most reliably destroy? (Select four... or identify the set.)
- A. Specific numbers, percentages, dates, and customer-stated expectations.
- B. Tool names, tool arguments, error messages, and retry counts.
- C. The customer's sentiment, the agent's confidence, and the conversation length.
- D. Policy citations, system prompt content, and tool descriptions.
Answer
A.
- B — Tool mechanics are typically preserved or reconstructible; that's not the named risk set.
- C — Sentiment isn't a fact worth preserving, and the exam treats it as an unreliable signal anyway.
- D — System prompt and tool descriptions aren't part of the summarized conversation region.
3
Why is "customer-stated expectations" the most consequential of the at-risk categories?
- A. It is the longest of the four and therefore compressed hardest.
- B. It represents a commitment the agent must honour, typically appears once in prose early on, and reads as conversational filler to a summarizer.
- C. It is required for regulatory compliance.
- D. It determines whether the conversation should be escalated.
Answer
B.
- A — Length isn't the mechanism.
- C — Overreach; the exam doesn't frame it as compliance.
- D — Conflates it with the escalation triggers, which are separate.
4
A tool returns 40 fields per call when the agent reasons over 5. What is the effect and the fix?
- A. Roughly 8× more context than needed on every call; trim the tool output before it enters the conversation, e.g. in a
PostToolUsehook. - B. Slower tool execution; add pagination to the tool.
- C. Higher hallucination risk; add a schema to the tool's output.
- D. No meaningful effect, since unused fields are ignored by the model.
Answer
A.
The cost compounds across every call in a long conversation.
- B — Latency isn't the concern; context consumption is.
- C — Extra fields don't cause hallucination.
- D — False. Unused fields still occupy context and contribute to dilution.
5
Given the need to both pass complete conversation history and control context growth, what is the correct resolution?
- A. Keep the full turn structure and shrink the payloads inside it; pin durable facts outside the compressible region.
- B. Drop the oldest turns once a threshold is reached.
- C. Replace the conversation with a single running summary each turn.
- D. Alternate between full history and summary on successive turns.
Answer
A.
Dropping turns loses commitments and causes repeated tool calls. Dropping unused fields costs nothing.
- B — Loses exactly the early-conversation commitments the domain warns about.
- C — Maximum information loss; this is the failure mode in item 1.
- D — Incoherent state across turns.
6
What is "lost in the middle," and what are its named mitigations? (Select the correct pairing.)
- A. Middle-of-context content receives less effective attention; mitigate with key findings first and explicit section headers.
- B. Middle-of-conversation turns are dropped by summarization; mitigate with a larger context window.
- C. Middle-priority findings are deprioritized by the model; mitigate with explicit severity criteria.
- D. Context in the middle of a batch request is truncated; mitigate by reducing batch size.
Answer
A.
Both mitigations are about arrangement, not volume.
- B — Conflates a position effect with a compression effect, and offers the capacity distractor.
- C — Different concept entirely (Domain 4).
- D — Invented.
7
During a long refactoring session across a large codebase, an agent that initially referenced specific classes and files begins describing "typical patterns" for this kind of code, and its suggestions no longer match the implementation. Diagnosis and fix?
- A. Context degradation; delegate exploration to subagents and maintain a scratchpad file of established findings.
- B. Insufficient model capability; enable extended thinking.
- C. Unclear goal; restate the refactoring objective in more detail.
- D. The codebase is too large for any agent; split it across repositories.
Answer
A.
Generic "typical patterns" language replacing specific identifiers is the named diagnostic signal for context degradation. Subagent delegation keeps exploration out of the main context; the scratchpad makes findings survive independently of it.
- B — Reasoning depth isn't the constraint; the specific code has fallen out of effective attention.
- C — The agent knows the goal; it has lost the code.
- D — Drastic, and it addresses a repository-structure problem that doesn't exist.
8
An agent working a multi-session migration is interrupted. On resume it produces work inconsistent with the earlier output. What was missing from its state export?
- A. The conventions and decisions established during the earlier run, not just the list of completed items.
- B. A timestamp for each completed item.
- C. The full conversation history from the earlier session.
- D. A confidence score for each completed item.
Answer
A.
A manifest must carry decisions, not just progress. A resumed run that doesn't know the established conventions produces inconsistent work.
- B — Not the gap.
- C — Would be enormous and mostly stale — the reason a manifest exists.
- D — Uncalibrated, and irrelevant to consistency.
Block B — Escalation and error propagation (items 9–14)
9
Which are legitimate escalation triggers for a customer support agent? (Select three.)
- A. The customer explicitly requests a human.
- B. The situation involves a policy exception or a gap in policy.
- C. The agent cannot make progress.
- D. The case is unusually complex.
Answer
A, B, and C.
- D — Complexity alone is not a trigger. Complex cases are what the agent exists to handle; escalating them defeats the 80% first-contact-resolution target.
10
A customer says "I'd rather just talk to a person." What should the agent do?
- A. Escalate immediately.
- B. Attempt one resolution first, since most requests can be handled automatically.
- C. Ask the customer to describe the problem so the escalation summary is complete.
- D. Escalate only if the agent's confidence in resolving the issue is below threshold.
Answer
A.
An explicit request for a human is honoured immediately.
- B — Optimizes the resolution metric against the customer's explicit wish. The named distractor.
- C — Tempting because a structured handoff is required — but the agent should build that summary from what it already knows, not make the customer re-explain after they've asked to leave.
- D — Uncalibrated confidence, and it subordinates an explicit request to a model signal.
11
Which two escalation triggers does the exam explicitly reject? (Select two.)
- A. Sentiment analysis detecting customer frustration.
- B. The model's self-reported confidence dropping below a threshold.
- C. The customer asking for a supervisor.
- D. A required action falling outside documented policy.
Answer
A and B.
Sentiment measures emotion, not resolvability. Self-reported confidence is uncalibrated.
- C — An explicit request for a human. Valid.
- D — A policy gap. Valid.
12
A customer lookup returns three possible matches. Which action is correct, and why?
- A. Ask for an additional identifier, because downstream actions like refunds are irreversible and account-specific.
- B. Select the most recently active account, because it's most likely the right one.
- C. Escalate to a human, because verification failed.
- D. Proceed and flag the ambiguity in the audit log.
Answer
A.
- B — Guessing. A wrong guess doesn't degrade the answer; it takes an irreversible action against the wrong person's account.
- C — Premature. Resolvable ambiguity is not an escalation trigger.
- D — Acts first, documents the risk afterwards.
13
A subagent cannot access one of its data sources. Which two behaviours are anti-patterns? (Select two.)
- A. Returning success with the partial results and no error indication.
- B. Terminating the entire research workflow.
- C. Returning the partial results alongside a structured error describing the failure.
- D. Annotating the coverage gap in the final report.
Answer
A and B.
The two symmetric anti-patterns: silent suppression leaves downstream agents building on a false premise; termination discards all successful work.
- C, D — Both parts of the correct middle path.
14
What should a structured error carry when it crosses an agent boundary? (Select four.)
- A. The failure type or category.
- B. What was attempted, specifically.
- C. Partial results that did succeed.
- D. Suggested alternatives.
Answer
All four.
A generic "search unavailable" supplies none of them, which is why the coordinator receiving it can only guess.
Block C — Review, calibration, provenance (items 15–20)
15
An extraction pipeline reports 97% aggregate field-level accuracy across invoices, credit notes, and statements. Leadership wants to remove human review. First step?
- A. Measure accuracy per document type and per field, and automate only the combinations meeting the threshold.
- B. Automate fully, since 97% exceeds the 95% target.
- C. Continue reviewing a uniform random 5% of all extractions.
- D. Automate, and have the model flag low-confidence extractions for review.
Answer
A.
An aggregate over a mixed corpus is dominated by the highest-volume type. 97% overall is consistent with 99% on invoices and 43% on multi-currency statements.
- B — Arithmetically correct, architecturally wrong. This is the trap.
- C — Uniform sampling under-samples the rare, risky types, and sampling alone doesn't answer the automation question.
- D — Relies on uncalibrated self-reported confidence.
16
Which sampling approach is correct for the human review that remains?
- A. Stratified random sampling within each document type, vendor, and field.
- B. Uniform random sampling across the whole corpus.
- C. Reviewing the first 20 extractions of each batch.
- D. Reviewing every extraction from the lowest-volume document type.
Answer
A.
Stratification guarantees each category gets enough samples to yield a usable estimate.
- B — In a corpus that's 85% invoices, you get ~4 statements out of 50 — too few to measure a 43%-accurate category.
- C — Not random. Batches may be ordered by source, time, or type, biasing the sample.
- D — Over-samples one stratum and measures nothing about the others.
17
Under what condition does self-reported confidence become a usable routing signal?
- A. When it has been calibrated per field on a labeled validation set, with thresholds derived from measured accuracy.
- B. When the model is instructed to be conservative in its confidence estimates.
- C. When two model instances agree on the confidence value.
- D. It is never usable for routing.
Answer
A.
Once you know that "0.85 on total for invoices" corresponds to a measured 98.1% accuracy, it's actionable. Raw, it's noise.
- B — Instructing conservatism doesn't calibrate anything.
- C — Agreement between two uncalibrated estimates is still uncalibrated.
- D — Too absolute; the exam permits it after calibration.
18
A research system's synthesis agent produces readable prose from claims collected with sources, but individual sentences can no longer be traced to specific sources. What is the requirement?
- A. Preserve the claim→source mapping through synthesis in a structured intermediate representation.
- B. Reattach citations after synthesis by matching claims to the collected sources.
- C. Include a bibliography of all consulted sources at the end of the report.
- D. Have the synthesis agent cite only its highest-confidence sources.
Answer
A.
- B — Reattaching means guessing which source supported which sentence — the exact information that was lost. The named wrong approach.
- C — A bibliography doesn't map claims to sources.
- D — Uses uncalibrated confidence and discards valid provenance.
19
Two credible sources report different figures for the same metric. What should the synthesis agent do?
- A. Report both figures with attribution, including publication dates and any methodological difference.
- B. Select the more authoritative source.
- C. Use the most recent figure.
- D. Report the average of the two.
Answer
A.
Preserve the disagreement with attribution so the reader can judge.
- B, C — Both discard information the reader needs, and C may hide a chronology (pre- and post-restatement) rather than a conflict.
- D — Nonsense for figures computed on different bases (e.g. constant-currency vs reported).
20
Match data types to rendering formats. Which pairing is correct?
- A. Financial data → tables; news and narrative → prose; technical findings → lists.
- B. Financial data → lists; news → tables; technical findings → prose.
- C. All output should be rendered as prose for readability.
- D. All output should be rendered as tables for consistency.
Answer
A.
Comparable figures need aligned columns; chronology and causation need sentences; discrete findings need scannable items.
- C, D — Uniformity over fitness. Rendering financial comparisons as prose is a specific failure the guide calls out.
Scoring
| Score | Read |
|---|---|
| 18–20 | Strong. |
| 15–17 | Solid; re-read Chapter 12 §3 on the misses. |
| 11–14 | Re-read Chapters 11–12 fully. |
| < 11 | Items 1, 7, 9, 15, 18 carry the domain. Re-study those. |
Note the overlap: items 15–17 are also Domain 4 material, and items 9–14 also appear in Domain 1 framing. Getting this domain right lifts your score in two others.