Chapter 10 — Structured Output, Validation, and Batching

Domain 4: Prompt Engineering & Structured Output (20%) — task statements 4.3, 4.4, 4.5, 4.6

Scenario 6 (Structured Data Extraction) is built on this chapter. The single most important idea: schemas guarantee syntax, not meaning.

1. Structured output via tool_use + JSON schema

🎯 The mechanism

The most reliable way to get schema-conformant output is to define a tool whose input_schema is your output schema, then force it with tool_choice:

EXTRACT_INVOICE = {
    "name": "extract_invoice",
    "description": "Record the extracted fields from a single invoice document.",
    "input_schema": {
        "type": "object",
        "properties": {
            "invoice_number": {"type": "string"},
            "vendor_name":    {"type": "string"},
            "issue_date":     {"type": "string", "description": "ISO 8601 (YYYY-MM-DD)."},
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "description": {"type": "string"},
                        "quantity":    {"type": "number"},
                        "unit_price":  {"type": "number"},
                        "amount":      {"type": "number"},
                    },
                    "required": ["description", "amount"],
                },
            },
            "stated_total":     {"type": "number"},
            "calculated_total": {
                "type": "number",
                "description": "Sum of line_items[].amount, computed by you.",
            },
            "conflict_detected": {
                "type": "boolean",
                "description": "True if stated_total and calculated_total differ.",
            },
            "purchase_order": {
                "type": ["string", "null"],
                "description": "PO number if present on the document; null if absent. "
                               "Never infer or construct a PO number.",
            },
            "document_type": {
                "type": "string",
                "enum": ["invoice", "credit_note", "statement", "unclear", "other"],
            },
            "document_type_detail": {
                "type": ["string", "null"],
                "description": "Required when document_type is 'other' or 'unclear'.",
            },
        },
        "required": ["invoice_number", "vendor_name", "stated_total",
                     "calculated_total", "conflict_detected", "document_type"],
    },
}

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    tools=[EXTRACT_INVOICE],
    tool_choice={"type": "tool", "name": "extract_invoice"},   # forced
    messages=[{"role": "user", "content": document_text}],
)

Every design choice in that schema is examinable. Walk through them:

🎯🎯 Syntax errors vs semantic errors

A schema eliminates syntax errors. It does not eliminate semantic errors. This is the most important sentence in Domain 4.

⚠️ The trap: an item describes extractions that are "valid JSON conforming to the schema" but where "the line item amounts don't sum to the stated total," and offers "tighten the JSON schema" as an option. A schema cannot express a cross-field arithmetic relationship. The answer is a semantic validation step — which is what calculated_total and conflict_detected are for (§3).

🎯 Nullable and optional fields prevent fabrication

A required, non-nullable purchase_order field forces the model to put something there. When the document has no PO number, that something is invented.

Making the field nullable gives the model a correct way to say "absent." Combine with an explicit description — "null if absent; never infer" — and you convert a hallucination into accurate data.

📎 The rule: any field that might legitimately be missing from the source must be nullable. This is a direct hallucination-prevention mechanism, not just schema hygiene.

🎯 Enums need "unclear" and "other" plus a detail field

A closed enum forces every input into one of your categories. Real documents don't cooperate. Two escape hatches:

• "unclear" — the model can't determine the category from the document
• "other" — the document has a determinate type that isn't in your list

Pair them with a detail string field so you learn what "other" actually was. That detail field is how the enum evolves: after a month you look at the other values, find that 200 of them were "proforma invoice," and add it as a proper enum member.

⚠️ Without the escape hatches, you get silent misclassification — a credit note filed as an invoice, with no signal that anything went wrong. That's strictly worse than an explicit "unclear."

🎯 Format normalization rules alongside strict schemas

A schema can say issue_date is a string. It can't make "3/4/25" unambiguous.

So pair the strict schema with explicit normalization rules in the prompt:

Dates: output ISO 8601 (YYYY-MM-DD). Where the source is ambiguous between
day-first and month-first, use the vendor's country convention from the
address block; if that's unavailable, set the date to null and set
date_ambiguous to true.

Amounts: decimal numbers with no currency symbol, no thousands separator.
Parenthesized amounts, e.g. (450.00), are negative.

The schema constrains the shape; the rules constrain the interpretation. You need both.

2. Retry with error feedback

🎯 The mechanism

When validation fails, don't just retry the same request — resend the document along with the failed extraction and the specific validation errors.

def extract_with_retry(document, max_attempts=3):
    attempt, errors, previous = 0, None, None
    while attempt < max_attempts:
        attempt += 1
        content = [{"type": "text", "text": document}]
        if errors:
            content.append({
                "type": "text",
                "text": (
                    f"Your previous extraction was:\n{json.dumps(previous, indent=2)}\n\n"
                    f"It failed validation:\n" + "\n".join(f"- {e}" for e in errors) +
                    "\n\nRe-extract, correcting these specific problems. If a value "
                    "genuinely is not present in the document, return null for it."
                ),
            })

        previous = call_model(content)
        errors = validate_semantics(previous)     # not just schema validation
        if not errors:
            return previous, attempt

    return previous, attempt          # exhausted: route to human review

The three elements that make this work, all examinable: the original document (so it can re-read), its own failed output (so it can see what it produced), and the specific errors (so it knows what to fix). Generic "that was wrong, try again" is materially weaker.

⚠️ When retry is useless

🎯 Retry cannot recover information that is absent from the source document. If the invoice has no PO number, retrying ten times produces ten failures — or worse, an invented PO number on the attempt where the model decides to satisfy you.

So the retry loop needs to distinguish:

An item may offer "increase the retry limit" for a case where the data is absent. That's the distractor. The right answers are a nullable field, an "unclear" enum value, or routing to human review.

3. Self-correction flows

📎 Build the check into the schema so the model does the comparison and reports the discrepancy.

🎯 Why this beats validating externally only: the comparison happens while the model still has the document in context, so a mismatch can be caught and often self-corrected in one pass. It also gives you a routing signal — every extraction with conflict_detected: true goes to human review regardless of anything else.

You still validate externally too. The model computing calculated_total doesn't guarantee it computed it correctly; your code re-sums the line items and compares. Two independent checks, one inside the model's pass and one outside it.

🧠 Note the structural similarity to Chapter 08 §3 and §5 below: an independent check catches what a self-check misses. Same principle, three appearances.

4. The Message Batches API

Task statement 4.5. 📎 Facts first, then the judgement.

🎯 When to use batch — and when never to

⚠️ The pre-merge CI trap. A published sample question offers batching for a CI pipeline. Blocking a pull request on a job with no latency SLA could mean a 24-hour wait for a merge. The correct scoping in that question: batch the overnight aggregate report only, keep the pre-merge review synchronous.

The rule: batch anything nobody is waiting for; never batch anything anybody is waiting for.

🎯 The no-multi-turn-tool-calling limitation

Each batch request is a single API call. It cannot run an agentic loop — no tool call, observe, call again. So:

• ✅ Single-pass extraction with a forced tool for structured output → fine in batch, because that's one turn.
• ❌ An agent that investigates by calling tools repeatedly → cannot be batched.

⚠️ This distinction matters because both use tools. The discriminator is whether the work needs multiple turns.

🎯 SLA arithmetic

Items may ask you to reason about windows. The pattern:

A pipeline must guarantee results within 30 hours of document arrival. Batch processing takes up to 24 hours. How often must batches be submitted?

Documents arrive continuously. A document arriving just after a submission waits (submission interval) before it's even sent, then up to 24 hours to process. So:

worst case = submission interval + 24 h  ≤  30 h
           → submission interval ≤ 6 h

Every 6 hours guarantees the 30-hour SLA. The guide's own variant uses 4-hour submission windows for margin. 🎯 The mechanic to remember: worst-case latency = wait-for-next-submission + full processing window. Don't forget the first term.

📎 Resubmitting failures

Results are correlated by custom_id. When some requests fail or produce invalid output, resubmit only those custom_ids, with modifications — a corrected prompt, added feedback, or a relaxed schema. Don't re-run the whole batch; you'd pay for the successes twice.

🎯 Refine on a sample first

Before submitting 50,000 documents, run a sample synchronously and refine the prompt against it. A batch of 50,000 with a flawed prompt costs the full amount and delivers 50,000 flawed extractions, 24 hours later. The sample-first step is cheap insurance and it's the expected answer when a stem describes a large first-time batch run.

5. Multi-instance and multi-pass review

Task statement 4.6. This closes the loop with Chapter 04 §4 and Chapter 08 §3.

🎯 An independent reviewer instance beats self-review

Three options, ranked as the guide ranks them:

1. ✅ A separate reviewer instance with fresh context — best
2. ❌ Asking the generating instance to review its own work — carries the assumptions that caused the error
3. ❌ Enabling extended thinking on the generating instance — more reasoning inside the same contaminated context

⚠️ Option 3 is the interesting distractor because extended thinking genuinely improves hard reasoning. It doesn't help here, because the problem isn't insufficient reasoning — it's a contaminated starting point. Independence, not depth, is what review requires.

🎯 Per-file passes plus a cross-file integration pass

Restating Chapter 04 §4 because it's tested from both domains:

• Per-file passes catch local issues that attention dilution would otherwise hide.
• The integration pass catches what per-file passes structurally cannot — a changed signature and its unchanged callers, a contract mismatch across module boundaries.
• A larger context window fixes neither.

📎 Self-reported confidence for calibrated routing

Confidence scores are usable — with a hard condition.

🎯 The exam's position is consistent: raw self-reported confidence is poorly calibrated and must not drive decisions. Once you've measured what a given confidence level actually means for a given field and document type, it becomes a legitimate routing signal. Chapter 12 §3 covers the measurement design.

6. Worked examples

Example 1 — semantic errors

An extraction pipeline uses a forced tool call with a strict JSON schema. All outputs are valid JSON with correct types. However, in about 6% of invoices the line item amounts do not sum to the extracted total. What should the architect add?

A. A validation step that recomputes the total from line items and flags mismatches, with retry-and-feedback on failure. B. A stricter JSON schema with tighter numeric constraints. C. A larger max_tokens value so the model can process all line items. D. tool_choice: {"type": "any"} to give the model more flexibility.

Example 2 — batch scoping

A CI pipeline runs Claude-based code review on every pull request before merge, and also generates an aggregate quality report each night across all repositories. The team wants to reduce cost. Where should the Message Batches API be applied?

A. The nightly aggregate report only. B. Both, since both are automated. C. The pre-merge review only, since it runs most frequently. D. Neither; batch processing does not support tool use.

7. Decision reflexes (reproduce from memory)

• Reliable structured output = a tool whose input_schema is your output schema + tool_choice: {"type":"tool","name":...}.
• Schemas eliminate syntax errors, never semantic ones. Sums, right-value-in-right-field, and correct-source-field are not schema-expressible.
• Nullable/optional fields prevent fabrication. Anything that might be absent must be nullable, with "never infer" in the description.
• Enums need "unclear" and "other" plus a detail string — the detail field is how the enum evolves.
• Pair strict schemas with explicit format-normalization rules (dates, amounts, negatives).
• Retry = original document + the failed output + the specific validation errors. Generic retry is weak.
• Retry cannot recover information absent from the source → nullable field, "unclear", or human review.
• Self-correction fields: calculated_total vs stated_total + conflict_detected. Validate externally too.
• Batches: 50% cheaper · up to 24 h · no latency SLA · custom_id correlation · no multi-turn tool calling.
• Batch overnight/weekly work. Never batch pre-merge CI or anything interactive.
• Worst-case batch latency = submission interval + full processing window.
• Resubmit only the failed custom_ids, with modifications. Refine on a sample before a large first run.
• Independent reviewer instance > self-review > extended thinking on the generator. Independence, not depth.
• Confidence is only a routing signal after calibration on a labeled validation set, per field.