Deck 4 — Prompt Engineering & Structured Output (Domain 4, 20%)
50 cards. Cards 1–22 = prompt precision, criteria, few-shot, false positives. 23–50 = schemas, validation, retry, batching, review.
1
Q: 🎯 Two instructions the exam treats as failing? A: "Be conservative" and "only flag high-confidence issues."
2
Q: Why do they fail? A: They're exhortations, not criteria. The model has no operational definition of "conservative," so behaviour is inconsistent.
3
Q: What replaces them? A: Explicit criteria stating exactly what qualifies — and what does not.
4
Q: Give the canonical explicit criterion for stale-comment detection. A: "Flag comments only when the claimed behaviour contradicts the actual code behaviour."
5
Q: What should accompany severity criteria to make them operational? A: Code examples — including explicit "NOT critical / NOT major" counter-examples.
6
Q: Why are false positives treated as more damaging than false negatives in CI review? A: They destroy developer trust. Once developers start ignoring the tool, its true positives are ignored too.
7
Q: One category produces most of the false positives. Legitimate short-term move? A: Temporarily disable that category while you rewrite its criteria.
8
Q: Which schema field helps you analyse false positives after the fact?
A: A detected_pattern field recording why the model flagged it.
9
Q: How many few-shot examples does the guide recommend? A: 2–4 targeted examples.
10
Q: Four things few-shot examples achieve? A: Consistency · handling of ambiguous cases · generalization · hallucination reduction.
11
Q: What kind of example is most valuable? A: A boundary case — and a near-miss negative showing what should not be flagged.
12
Q: Ordering: criteria, few-shot, enforcement — which first? A: Explicit criteria first (cheapest, highest leverage), then few-shot for boundaries, then enforcement for anything with consequences.
13
Q: Would 20 few-shot examples be better than 3? A: No — 2–4 targeted examples. Volume dilutes and lengthens context without adding signal.
14
Q: A reviewer flags negative line-item discounts as errors. Fix? A: Add a few-shot example of a legitimate negative line item, so the boundary is demonstrated rather than described.
15
Q: "Only report issues you're highly confident about" — what's the flaw? A: It relies on self-reported confidence, which is uncalibrated. Give criteria instead.
16
Q: Firming up wording ("You MUST NEVER...") — when is that the right fix? A: For style and formatting preferences only. Never for financial, compliance, or irreversible actions — use a hook.
17
Q: What's the "most effective first step" heuristic? A: The smallest high-leverage change addressing the root cause. Over-engineering is penalized.
18
Q: How do you validate a prompt rewrite before rolling it out? A: Run it against a sample with known ground truth, and measure.
19
Q: Vague instruction, few-shot examples, or a schema — which addresses inconsistent formatting of the same field? A: Format normalization rules in the prompt, plus the schema for shape. Examples help; a schema alone doesn't.
20
Q: Which prompt element handles "the document doesn't say"?
A: An explicit instruction to return null / "unclear", paired with a nullable field or enum escape hatch.
21
Q: What's the risk of criteria without counter-examples? A: The model over-applies them. The "NOT critical" examples define the upper boundary.
22
Q: One sentence on Domain 4's central prompt lesson? A: Specify the decision rule, don't ask for good judgement.
23
Q: Most reliable mechanism for schema-conformant output?
A: A tool whose input_schema is your output schema, forced with tool_choice: {"type":"tool","name":...}.
24
Q: 🎯🎯 What does a JSON schema guarantee? A: Syntax — valid JSON, required fields present, correct types, enum membership.
25
Q: What does it not guarantee? A: Semantics — sums reconciling, the right value in the right field, the correct source field being read.
26
Q: Line items don't sum to the total, but the JSON is valid. Fix? A: A semantic validation step that recomputes and compares — not a tighter schema.
27
Q: Can a JSON schema express "the sum of this array equals that field"? A: No. Cross-field arithmetic is not schema-expressible.
28
Q: A required non-nullable purchase_order field. What happens on a document with no PO?
A: The model fabricates one. Forced fields cause hallucination.
29
Q: Fix for that? A: Make it nullable, with a description saying "null if absent; never infer."
30
Q: Rule for any field that might legitimately be missing? A: It must be nullable. This is a hallucination-prevention mechanism.
31
Q: Two enum escape-hatch values?
A: "unclear" (can't determine) and "other" (determinate but not in the list).
32
Q: What must accompany those escape hatches? A: A detail string field — that's how you learn what "other" was and evolve the enum.
33
Q: Consequence of a closed enum with no escape hatch? A: Silent misclassification — a credit note filed as an invoice with no signal.
34
Q: A schema says issue_date is a string. What still needs specifying?
A: Format normalization rules — ISO 8601 output, ambiguous day/month handling, negative amounts in parentheses, no thousands separators.
35
Q: Three elements of a good retry? A: The original document · the failed extraction · the specific validation errors.
36
Q: Why is generic "that was wrong, try again" weaker? A: The model doesn't know what to change, so it re-samples rather than corrects.
37
Q: 🎯 When is retry useless? A: When the information is absent from the source document. Retrying invites fabrication.
38
Q: What replaces retry in that case?
A: A nullable field, an "unclear" enum value, or human review.
39
Q: Three self-correction fields?
A: stated_total · calculated_total · conflict_detected.
40
Q: Why have the model compute calculated_total if your code checks it anyway?
A: It catches and often self-corrects the mismatch while the document is still in context, and gives you a routing signal.
41
Q: Message Batches API — cost saving? A: 50%.
42
Q: Processing window and latency guarantee? A: Up to 24 hours, with no latency SLA.
43
Q: How are batch results correlated to requests?
A: custom_id.
44
Q: 🎯 Key batch limitation? A: No multi-turn tool calling within a single request. Single-pass structured output is fine; an agentic loop is not.
45
Q: Which CI workload must never be batched? A: The pre-merge review — it blocks a developer, and batch has no latency SLA.
46
Q: Rule of thumb for batching? A: Batch anything nobody is waiting for; never batch anything anybody is waiting for.
47
Q: 30-hour SLA, 24-hour processing window. Maximum submission interval? A: 6 hours. Worst case = submission interval + full processing window. (The guide uses 4-hour windows for margin.)
48
Q: Some batch requests failed. What do you resubmit?
A: Only the failed custom_ids, with modifications. Never the whole batch.
49
Q: Before a first 50,000-document batch run? A: Refine the prompt on a sample synchronously.
50
Q: Rank the three review options. A: Independent reviewer instance > self-review > extended thinking on the generator. Independence, not depth.