Chapter 12 — Escalation, Error Propagation, Review, and Provenance

Domain 5: Context Management & Reliability (15%) — task statements 5.2, 5.3, 5.4, 5.6

Four independent topics. Each generates a distinctive item type, so learn them as four separate units.

1. Escalation and ambiguity resolution

Task statement 5.2. Scenario 1's most-tested area after enforcement.

🎯🎯 The three legitimate escalation triggers

Memorize this list exactly, and memorize what's not on it:

🎯 Honour explicit human requests immediately

Worth calling out separately because it's a distinct item. When a customer says "let me talk to a person," the correct behaviour is to escalate — not to attempt one more resolution first, and not to ask them to explain the problem again.

⚠️ The distractor: "attempt to resolve the issue before escalating, since most requests can be handled automatically." It's optimizing the resolution metric against the customer's explicit wish, and the guide is unambiguous: honour the request immediately.

🎯 Sentiment and confidence are unreliable proxies

Both appear as sophisticated-sounding options:

• "Escalate when sentiment analysis detects frustration" — measures emotion, not whether the agent can help.
• "Escalate when the model's confidence drops below a threshold" — self-reported confidence is poorly calibrated; the model is often most confident when wrong about something it has never seen.

The correct triggers are situational, not affective or introspective: what did the customer ask for, does policy cover this, can the agent still act?

🎯 Ambiguity: multiple matches → ask, don't guess

When a customer lookup returns several possible matches, the agent must ask for an additional identifying detail — not pick the most likely one.

❌ Select the most recent matching account and proceed
❌ Pick the one whose name matches most closely
✅ "I found more than one account with that name. Could you confirm the
    email address or the ZIP code on the account?"

⚠️ Why guessing is disqualifying rather than merely suboptimal: the downstream actions are refunds and order changes against a specific account. A wrong guess doesn't degrade the answer; it takes an irreversible action on the wrong person's account. This connects to Chapter 04's gate on verified IDs — verification means unambiguous identification.

2. Error propagation across multi-agent systems

Task statement 5.3. Partially covered in Chapter 06 §2; here's the multi-agent framing.

🎯 Structured error context

What an error should carry when it crosses an agent boundary:

⚠️ The named anti-pattern: a generic "search unavailable". It hides the category, the retryability, the scope of what failed, and what succeeded. The coordinator receiving it can only guess, and it will usually guess wrong.

🚫 The two symmetric anti-patterns

Both appear as options in the same item, and both are wrong:

The correct behaviour is the middle path: complete what you can, report precisely what you couldn't, and annotate the gap in the output.

🎯 Coverage annotations in synthesis output

The concrete form of "annotate the gap." The final report says what it doesn't cover:

## Regulatory environment

⚠️ Coverage note: the internal regulatory-filings collection was not accessible
(permission denied). This section draws only on public press coverage and the
company's published annual report. Filing-level detail is not represented.

🎯 Why this is the answer rather than just good practice: it makes the limitation visible to the human who reads the report, so they can decide whether the gap matters. A report that silently omits a dimension is indistinguishable from one where that dimension had nothing to say — and that's the failure mode from Chapter 06's isError discussion, reappearing at the output layer.

3. Human review and confidence calibration

Task statement 5.4. Scenario 6's quality-assurance layer.

🎯🎯 Aggregate accuracy masks per-type failure

The single most important idea in this section.

The pipeline reports 97% field-level accuracy. Leadership wants to automate.

97% aggregate can hide:

• Invoices: 99.2% accurate (85% of volume)
• Credit notes: 71% accurate (11% of volume)
• Multi-currency statements: 43% accurate (4% of volume)

The aggregate is dominated by the high-volume, easy type. Automating on the aggregate means automating a 43%-accurate path.

⚠️ The distractor: "97% exceeds the threshold; proceed with automation." It's arithmetically true and architecturally wrong.

📎 Validate by document type AND by field before automating

The requirement is two-dimensional. Not just "which document types are reliable" and not just "which fields are reliable," but the cross-product:

🎯 That table is the deliverable. It tells you what to automate (invoice invoice_number and total) and what to route to humans (everything in the statement column, credit-note line_items). Automating per-cell rather than globally is the correct architecture.

📎 Stratified random sampling

For the human review that remains, stratified random sampling — sample within each stratum (document type, vendor, field) rather than uniformly across the whole population.

Why: uniform random sampling of a corpus that's 85% invoices gives you ~4 statements out of 50. You cannot measure a 43%-accurate category from 4 samples. Stratifying guarantees each category gets enough samples to produce a usable estimate.

⚠️ Distractors: "review the first N of each batch" (not random — batches may be ordered by source, time, or type, biasing the sample), and "review a random 5% of all documents" (random but not stratified — under-samples the rare, risky types).

📎 Field-level confidence calibrated on labeled validation sets

Confidence becomes usable when you've measured what it means, per field, against labeled data. "Confidence 0.85 on total for invoices corresponds to 98.1% measured accuracy" is an actionable statement. "The model said 0.85" is not.

🎯 The full routing architecture the exam wants:

extraction → per-field calibrated confidence
           ├── above the measured threshold for this (type, field) → automated
           ├── below threshold → human review queue
           └── conflict_detected == true → human review queue, always

4. Provenance and conflict handling

Task statement 5.6. Scenario 3's output-quality layer.

🎯 Claim-source mappings must survive synthesis

The failure: the research agents collect claims with sources, the synthesis agent merges them into flowing prose, and the citations no longer map to specific claims. The report is readable and unverifiable.

Requirement: the claim→source mapping is preserved through synthesis, not reattached afterward. Reattaching means guessing which source supported which sentence, which is exactly the information that was lost.

Implementation: structured intermediate representation all the way through.

{
  "claim": "EMEA revenue declined 8% year over year in FY2025",
  "source_url": "https://example.com/investor/fy2025-annual-report.pdf",
  "source_title": "FY2025 Annual Report",
  "publication_date": "2026-02-11",
  "collection_date": "2026-07-14",
  "verbatim": "EMEA segment revenue of $412M, down from $448M in FY2024"
}

🎯 Annotate conflicting statistics; don't pick one

When two credible sources disagree, the synthesis agent must report both with attribution rather than silently choosing.

❌ EMEA revenue declined 8% year over year.

✅ Reported EMEA revenue decline for FY2025 varies by source: the company's
   annual report (published 2026-02-11) states 8%; a Reuters analysis
   (published 2026-03-02) states 11%, using a constant-currency basis.

⚠️ Distractors: "select the more authoritative source," "use the most recent figure," "average the values." All three discard information the reader needs, and the third is nonsense for figures computed on different bases. Preserve the disagreement with attribution.

🎯 Publication and collection dates prevent false contradictions

A specific and easily-missed point: include publication dates (and collection dates) so temporal differences aren't misread as contradictions.

Two sources reporting different revenue figures may both be correct — one published before a restatement, one after. Without dates, that looks like a conflict in the data. With dates, it's a chronology.

📎 Two distinct fields, both useful: publication date (when the source said it) and collection date (when we retrieved it).

🎯 Match the rendering format to the data type

The guide is specific about output formatting:

⚠️ The item shape: a report that renders financial comparisons as prose paragraphs, making the numbers hard to compare. The fix is a table. Don't overthink it — this is a direct recall item.

5. Worked examples

Example 1 — escalation criteria

A support agent escalates too rarely in situations that genuinely require a human, and too often in cases it could handle. The team is designing escalation logic. Which approach is most effective?

A. Define explicit escalation criteria — explicit customer request for a human, policy exceptions or gaps, and inability to make progress — reinforced with few-shot examples of each. B. Escalate whenever the agent's self-reported confidence falls below 0.7. C. Escalate whenever sentiment analysis indicates customer frustration. D. Escalate whenever a case requires more than five tool calls.

Example 2 — validation design

An extraction pipeline reports 97% aggregate field-level accuracy across a corpus of invoices, credit notes, and statements. Leadership wants to remove human review. What should the architect do first?

A. Measure accuracy per document type and per field, and automate only the combinations that meet the threshold. B. Automate fully, since 97% exceeds the 95% target. C. Continue reviewing a uniform random 5% of all extractions. D. Automate, and have the model flag low-confidence extractions for review.

6. Decision reflexes (reproduce from memory)

Escalation

• Three triggers: explicit request for a human · policy exception or gap · inability to make progress.
• Complexity alone is not a trigger. Neither is sentiment. Neither is self-reported confidence.
• An explicit request for a human is honoured immediately — no resolution attempt first.
• Multiple customer matches → ask for another identifier, never guess. Downstream actions are irreversible.

Error propagation

• Structured context: failure type · what was attempted · partial results · alternatives.
• 🚫 Generic "search unavailable" · 🚫 silent suppression · 🚫 terminating the whole workflow.
• Complete what you can, report what you couldn't, and annotate coverage gaps in the output.

Human review

• Aggregate accuracy masks per-type failure. 97% overall can hide 43% on a rare type.
• Validate by document type AND field; automate per cell, not globally.
• Stratified random sampling — not uniform, not first-N-of-batch.
• Confidence is usable only when calibrated per field on labeled validation data.

Provenance

• Claim→source mappings preserved through synthesis, never reattached afterward.
• Conflicting figures → report all with attribution. Don't pick, don't average, don't take the newest.
• Carry publication and collection dates so chronology isn't misread as contradiction.
• Render financial data as tables · news as prose · technical findings as lists.