Chapter 09 — Prompt Precision and False-Positive Control

Domain 4: Prompt Engineering & Structured Output (20%) — task statements 4.1, 4.2

Scenario 5 (CI code review) generates most of these items. The theme: replace adjectives with testable criteria.

1. Explicit criteria beat vague instructions

🎯 The canonical comparison

The guide's own example, and worth memorizing as a pair:

❌ "Check that comments are accurate."

✅ "Flag a comment only when the behaviour it claims contradicts the actual behaviour of the code it documents."

The first is a goal. The second is a test — you can apply it to any comment and get a determinate answer. That's the entire distinction.

🚫 The instructions that fail

These are named as ineffective, and they appear as distractors:

⚠️ These read like reasonable prompt engineering, which is exactly why they're effective distractors. The test: could two competent reviewers apply this instruction and reach the same conclusion? If not, it won't change the model's behaviour either.

The rewrite pattern

For each category you want flagged, supply:

1. A determinate condition — what must be true for this to be a finding
2. What is explicitly not a finding — the boundary
3. A code example of each severity level

## Severity criteria

### CRITICAL — flag always
A code path that can read or write data belonging to a different tenant.
  Example: a query filtered only by `resource_id` with no `tenant_id` predicate.
NOT critical: a query missing `tenant_id` where the calling function has already
  scoped the connection to a single tenant.

### MAJOR — flag always
An unhandled error path that leaves persistent state partially written.
  Example: two sequential writes with no transaction and no compensating rollback.
NOT major: an unhandled error in a read-only path.

### MINOR — do not flag in CI
Naming, formatting, and comment style. These are handled by the linter.

Notice the "NOT" lines. 🎯 Specifying what doesn't qualify is as important as specifying what does — it's the same principle as tool description boundaries (Chapter 05 §1).

2. False positives destroy trust

🎯 The central insight of Scenario 5

When an automated reviewer produces too many false positives, developers stop reading its output entirely. At that point the tool's recall is irrelevant — a correct finding nobody reads has zero value.

This reframes the precision/recall trade-off in a way the exam tests: precision is the binding constraint for automated review, because the failure mode isn't "missed a bug," it's "the whole system got ignored."

📎 Temporarily disabling high-FP categories

🎯 A specific, examinable remedy: if one category of findings produces most of the false positives, disable that category temporarily while you develop better criteria for it.

This feels wrong to candidates — you're deliberately reducing coverage. But the reasoning is sound: a category that's wrong 60% of the time is actively destroying the credibility of the categories that are right, and disabling it raises the number of real issues that get acted on.

⚠️ Note the word temporarily. The answer isn't "permanently narrow the scope"; it's disable, fix the criteria, re-enable. An option saying "permanently limit the reviewer to critical security issues only" trades away recall forever and will usually be a distractor — especially when the stem says coverage must be preserved.

📎 The detected_pattern field

A practical technique: have the reviewer emit a detected_pattern field naming which criterion triggered each finding.

{
  "file": "src/orders/repository.ts",
  "line": 142,
  "severity": "critical",
  "detected_pattern": "query_missing_tenant_predicate",
  "finding": "…"
}

🎯 Why this is examinable: it makes false-positive analysis tractable. Instead of "40% of findings are wrong," you can see that query_missing_tenant_predicate accounts for 80% of the wrong ones — which tells you exactly which criterion to fix. Without it, you're guessing.

The general principle: instrument the classifier so you can diagnose it by category, not just in aggregate. This connects directly to Chapter 12 §3 — aggregate accuracy masks per-category failure.

3. Few-shot prompting

🎯 The guide's named most-effective technique

2–4 targeted examples. The guide credits few-shot with four specific benefits, and the list is worth memorizing because items are built from it:

How to choose the examples

Not the easy cases. Pick the boundary.

❌ Three examples of obvious SQL injection
   → teaches nothing the model didn't already know

✅ One clear positive, one near-miss negative, one genuinely ambiguous case
   with the desired resolution
   → teaches where the line is

The near-miss negative is the highest-value example, because false positives come from cases that resemble findings. Show the model something that looks like a finding and isn't, with the reasoning.

🎯 Why 2–4 and not 10

More examples cost context and can cause over-fitting to their surface features — the model starts matching the examples' incidental characteristics rather than the underlying criterion. The guide specifies 2–4 targeted examples; "targeted" carries as much weight as the number.

⚠️ Distractor: "add 15 examples covering every case you've seen." Sounds thorough, dilutes the signal, and consumes the context budget you need for the actual code.

🎯 Few-shot vs criteria vs enforcement — the ordering

An item may offer all three. The ranking depends on what's failing:

🎯🎯 The third row is the trap. Few-shot and criteria are both prompt-layer tools, and prompt-layer tools have a non-zero failure rate. When the stem involves money, compliance, or irreversibility, both of them lose to a hook or gate. In the published sample question on refund ordering, few-shot examples of the correct order was a distractor precisely because the requirement was a guarantee.

4. Worked examples

Example 1 — false positives

An automated code review agent in CI flags roughly 40% non-issues. Developers have begun ignoring its comments entirely. Analysis shows most false positives come from one category: comment-accuracy checks. The team wants to preserve coverage of genuine issues. What is the most effective first step?

A. Temporarily disable comment-accuracy checks and rewrite their criterion to require that the claimed behaviour contradicts the actual behaviour. B. Instruct the agent to be conservative and flag only high-confidence issues. C. Restrict the agent permanently to critical security findings. D. Use a model with a larger context window so the agent sees more surrounding code.

Example 2 — technique selection

A structured extraction agent handles most invoices correctly but is inconsistent on invoices where a discount is expressed as a negative line item rather than a separate field. What is the most effective technique?

A. Add 2–3 few-shot examples showing invoices with negative-line-item discounts and the desired extraction for each. B. Add an instruction to handle discounts carefully. C. Increase max_tokens to allow more thorough processing. D. Add a programmatic validation hook that rejects extractions containing negative amounts.

5. Decision reflexes (reproduce from memory)

• Replace adjectives with determinate tests. "Check comments are accurate" → "flag only when claimed behaviour contradicts actual behaviour."
• 🚫 "Be conservative," "only high-confidence," "avoid false positives," "use judgement" — all named as ineffective.
• Specify what is not a finding, with code examples per severity level.
• False positives destroy developer trust; once output is ignored, recall is worthless. Precision is the binding constraint for automated review.
• One category causing most FPs → temporarily disable it and fix its criterion. Not permanent narrowing.
• detected_pattern on each finding makes false-positive analysis diagnosable by category.
• Few-shot = 2–4 targeted examples; benefits are consistency, ambiguous-case handling, generalization, hallucination reduction. Choose boundary cases, especially the near-miss negative.
• Ambiguity → few-shot. Wrong definition → criteria. Financial/irreversible → gate, not either of them.