Apply Few-Shot Prompting to Improve Output Consistency and Quality
This module is about using examples to teach Claude the judgment pattern you want.
Anthropic’s current prompt engineering docs describe examples as one of the most reliable ways to steer Claude’s output format, tone, and structure, and say well-crafted few-shot or multishot examples improve accuracy and consistency.
The same docs recommend examples that are relevant, diverse, and structured, ideally wrapped in <example> / <examples> tags so Claude can distinguish examples from instructions. (Claude)
1. Core Mental Model
Few-shot prompting means providing a small set of examples before the actual task.
Each example usually shows:
Use few-shot examples when:
- Detailed instructions are still producing inconsistent output.
- The output format must be consistent.
- The task requires judgment.
- The model confuses similar categories.
Do not use examples only to cover every possible case. The point is to teach a generalizable decision pattern.
2. Few-Shot Vs Zero-Shot
2.1 Zero-Shot Prompt
This may work, but output can vary:
- Some findings include location; others do not.
- Severity labels drift.
- Suggested fixes are vague.
- False positives appear.
2.2 Few-Shot Prompt
Then you show 2–4 examples of good findings and skipped cases.
Anthropic’s prompt engineering overview says good prompt engineering starts with clear success criteria and a way to test against those criteria. Few-shot examples are useful because they make those success criteria concrete. (Claude)
3. How Many Examples?
Remember:
Anthropic’s general docs recommend 2–4 examples for best results. The practical point is not the exact number; it is that the examples should be high-signal, not repetitive. (Claude)
Good few-shot set:
1 clear positive example
1 clear negative / skip example
1 ambiguous boundary example
1 edge case example
4. Anatomy of a Strong Few-Shot Example
A good example should include:
Use a concise rationale, not a long hidden reasoning trace. The goal is to show the classification basis.
4.1 Simple Few-Shot Example Structure:
<examples>
<example>
<input>
User asks: "Can you check whether order #123 shipped?"
Available tools: get_customer, lookup_order
</input>
<expected_action>
Use lookup_order
</expected_action>
<rationale>
The user provided an order ID and asks about order status. Customer identity lookup is not needed before retrieving non-sensitive shipment status.
</rationale>
</example>
</examples>
4.2 Example Structure for Code Review:
<instructions>
Review the diff. Report only correctness or security issues introduced by the diff.
For each finding, use this exact format:
- location:
- issue:
- severity: Important | Nit
- suggested_fix:
Skip style-only issues, formatting, lint, and speculative concerns.
</instructions>
<examples>
<example>
<input>
Changed code:
if (user.role = "admin") {
grantAdminAccess(user.id)
}
</input>
<expected_output>
- location: auth/permissions.ts:42
- issue: The condition assigns "admin" instead of comparing the user role, so every user can be treated as admin.
- severity: Important
- suggested_fix: Use a strict comparison, for example user.role === "admin".
</expected_output>
<rationale>
This is a production-impacting authorization bug introduced by the changed condition.
</rationale>
</example>
<example>
<input>
Changed code:
const res = await fetchSubscription(userId)
</input>
<expected_output>
No finding.
</expected_output>
<rationale>
The variable name is vague, but this is only a style issue and does not affect behavior.
</rationale>
</example>
</examples>
<input>
{{DIFF}}
</input>
Anthropic’s docs recommend XML-style tags when prompts combine instructions, context, examples, and variable inputs, because tags help Claude parse complex prompts unambiguously. (Claude)
5. Few-Shot for Ambiguous Tool Selection
Few-shot examples are especially useful when two actions are plausible.
5.1 Scenario
The agent has tools:
The model sometimes calls get_customer for every order-related question, even when lookup_order is sufficient.
5.2 Few-Shot Example
<examples>
<example>
<input>
User: "Can you check the status of order #A123?"
</input>
<expected_action>
lookup_order
</expected_action>
<rationale>
The user provided an order ID and asks for order status. The order tool directly matches the request.
</rationale>
</example>
<example>
<input>
User: "I need a refund. My name is Priya Sharma."
</input>
<expected_action>
get_customer first
</expected_action>
<rationale>
Refunds are sensitive and require verified customer identity before order or refund operations.
</rationale>
</example>
<example>
<input>
User: "Order #B778 arrived damaged. Can you refund it?"
</input>
<expected_action>
get_customer first, then lookup_order after identity is verified
</expected_action>
<rationale>
The user provided an order ID, but refund processing requires verified customer identity before taking financial action.
</rationale>
</example>
</examples>
Heuristic:
Few-shot examples should show the ambiguous boundary, not just easy cases.
The third example is the most valuable because both lookup_order and get_customer seem plausible. It teaches the model that financial action changes the sequence.
6. Few-Shot for Reducing False Positives
The exam specifically calls out distinguishing acceptable code patterns from genuine issues.
This is important because examples should not only show what to report. They should also show what to skip.
6.1 Example: Acceptable Vs Genuine Issue
<examples>
<example>
<input>
Changed code:
const users = await db.users.findMany({ where: { orgId } })
return users.map(toDto)
</input>
<expected_output>
No finding.
</expected_output>
<rationale>
This is a single bounded query followed by an in-memory mapping. There is no evidence of an N+1 query or correctness bug.
</rationale>
</example>
<example>
<input>
Changed code:
for (const user of users) {
const orders = await db.orders.findMany({ where: { userId: user.id } })
results.push({ user, orders })
}
</input>
<expected_output>
- location: reports/userOrders.ts
- issue: The diff introduces one database query per user on an unbounded list, creating an N+1 query path.
- severity: Important
- suggested_fix: Batch-load orders for all user IDs in one query and group them by userId.
</expected_output>
<rationale>
This is a real performance issue because the repeated database call is inside a loop over a production-sized collection.
</rationale>
</example>
</examples>
Why this works:
It prevents Claude from flagging every loop or every query.
It teaches the concrete difference between acceptable linear work and reportable N+1 behavior.
7. Few-Shot Examples Should Teach Generalization, Not Memorization
A bad few-shot set accidentally teaches keyword matching.
Bad examples:
- Every reportable bug example contains the word "crash."
- Every security example contains the word "token."
- Every extraction example uses the same heading.
The model may overfit to surface patterns.
Better examples vary:
One crash bug.
One wrong-output bug.
One authorization bug.
One skip case with scary-looking but acceptable code.
The official guide’s key idea is that examples should help Claude generalize judgment to novel patterns, not simply match pre-specified cases.
7.1 How to Choose Few-Shot Examples
Pick examples that are:
- Relevant: close to the real task.
- Diverse: cover different shapes and edge cases.
- Contrasting: include both report and skip cases.
- Boundary-focused: show ambiguous cases.
- Format-complete: exactly match desired output.
- Non-contradictory: do not conflict with instructions.
Anthropic’s docs explicitly recommend relevant, diverse, structured examples, and warn that diversity helps avoid unintended patterns. (Claude)
8. Common Traps
8.1 More Prose Instead of Examples
Scenario:
Weak fix:
Better fix:
8.2 Only Positive Examples
Bad:
Better:
Why?
8.3 Examples Without Rationale
Bad:
Better for judgment tasks:
For classification, tool selection, and review, the rationale teaches the category boundary.
8.4 Too Many Repetitive Examples
Bad:
Better:
8.5 Examples Contradict Instructions
Bad:
This teaches the wrong behavior.
9. Cheat Sheet
Memorize this:
1. Use few-shot examples when instructions alone produce inconsistent results.
2. Few-shot examples are especially strong for output format consistency.
3. Use 2–4 targeted examples for exam scenarios.
4. Good examples are relevant, diverse, structured, and non-contradictory.
5. Include both report and skip examples.
6. Use examples to demonstrate ambiguous-case handling.
7. Include concise rationales for judgment tasks.
8. Few-shot examples help Claude generalize to novel patterns.
9. Avoid repetitive examples that teach accidental keyword matching.
10. Use examples to distinguish acceptable code from genuine issues.
11. Use examples to calibrate severity and suggested fixes.
12. Use examples for branch-level test coverage judgment.
13. Use extraction examples with varied document structures.
14. Include null/missing-field examples to reduce hallucination.
15. Show explicit, derived, and unavailable extraction cases.