Implement Validation, Retry, and Feedback Loops for Extraction Quality
This module is the natural follow-up to D4.3 structured output.
Now validate whether the structured values are actually correct.
Retry when the error is correctable, and collect feedback data when findings are dismissed.
The most important distinction is:
Schema validity is not the same as extraction quality.
Strict tool use can guarantee that Claude’s tool inputs match a JSON Schema, with correctly typed arguments and no missing required schema fields. It does not prove that the extracted values are semantically correct. (Claude)
1. Core Mental Model
Use this pipeline:
1. Extract with tool use / structured schema.
2. Validate schema and semantics.
3. If validation errors are correctable, retry with specific feedback.
4. If information is absent, do not retry blindly.
5. Track recurring false-positive / failure patterns.
6. Improve prompts, schemas, examples, and validators over time.
Anthropic’s docs emphasize giving Claude a way to verify its work, such as tests, builds, or other executable checks; the same principle applies to extraction pipelines: validation gives Claude a concrete signal for correction. (Claude)
2. Semantic Validation Errors
These are errors where the output is structurally valid but factually wrong.
Examples:
The schema is valid, but:
Other semantic errors:
- Invoice due date placed in Invoice Date.
- Vendor name placed in customer_name.
- Line items do not sum to total.
- Extracted date is normalized incorrectly.
- Required value is fabricated even though source is silent.
Heuristic:
3. Retry-with-Error-Feedback
Retry-with-error-feedback means you do not simply say:
Instead, you send Claude:
1. The original document
2. The failed extraction
3. The exact validation errors
4. The correction rules
5. The same schema/tool requirement
This gives Claude a concrete repair target.
3.1 Strong Retry
Your previous extraction failed validation.
Original document:
<document>
Invoice INV-1042
Subtotal: $100.00
Tax: $8.00
Shipping: $5.00
Total: $113.00
</document>
Previous extraction:
{
"invoice_number": "INV-1042",
"subtotal": 100,
"tax": 8,
"shipping": 5,
"stated_total": 200,
"currency": "USD"
}
Validation errors:
1. stated_total is inconsistent with the source document. Source says Total: $113.00, but extracted stated_total is 200.
2. calculated_total should equal subtotal + tax + shipping = 113.
Retry the extraction.
Rules:
- Use only values supported by the original document.
- Preserve correct fields when possible.
- Do not fabricate missing values.
- Return null for unavailable fields.
3.2 Follow-up Request Structure
A good retry prompt has a repeatable shape:
Example:
<original_document>
{{DOCUMENT}}
</original_document>
<failed_extraction>
{{FAILED_JSON}}
</failed_extraction>
<validation_errors>
- invoice_date was extracted as "2026-07-01", but the source labels that date as the due date.
- invoice_date is absent from the source. It should be null.
- due_date should be "2026-07-01".
</validation_errors>
<retry_instructions>
Correct only the fields implicated by validation errors unless another field is directly affected.
Use only source-supported values.
Return null when the source does not contain the requested information.
</retry_instructions>
Heuristic:
A retry should be diagnostic, not generic.
4. When Retries Do Not Work
Retries are ineffective when the required information is absent from the provided source.
Bad retry candidate:
But the document does not contain a tax ID. A retry cannot recover information that is not in the context.
Correct behavior:
Not:
5. Self-Correction Schema Patterns
The official task statement calls out:
These are schema designs that make validation easier.
5.1 Example:
calculated_total and stated_total, instead of extracting only:
extract:
{
"subtotal": 100,
"tax": 8,
"shipping": 5,
"calculated_total": 113,
"stated_total": 113,
"total_discrepancy_detected": false
}
Schema idea:
{
"subtotal": { "type": ["number", "null"] },
"tax": { "type": ["number", "null"] },
"shipping": { "type": ["number", "null"] },
"calculated_total": {
"type": ["number", "null"],
"description": "subtotal + tax + shipping when present; otherwise null."
},
"stated_total": {
"type": ["number", "null"],
"description": "The total amount explicitly stated in the source document."
},
"total_discrepancy_detected": {
"type": "boolean",
"description": "True when calculated_total and stated_total are both present and differ."
}
}
Why this helps:
It separates what the document states from what the numbers imply.
It allows downstream validation.
It avoids hiding discrepancies by forcing one total field.
5.2 conflict_detected
Use this when the source may contain inconsistent data.
Example source:
A 12-month term from January 1, 2026 would end around January 1, 2027, but the stated termination date says 2028.
Output:
{
"effective_date": "2026-01-01",
"term_months": 12,
"termination_date": "2028-12-31",
"conflict_detected": true,
"conflict_details": "The stated 12-month term conflicts with the stated termination date of 2028-12-31.",
"evidence": "Contract Term: 12 months. Termination Date: December 31, 2028 Effective Date: January 1, 2026"
}
Why this is better than silently choosing one value:
It preserves source uncertainty.
It avoids pretending conflicting data is clean.
It gives the downstream system a review flag.
6. Using Feedback to Tune Prompts
If dismissed findings cluster around:
Do not just say:
Instead, tune the category:
Do not report performance findings unless all are true:
1. The diff introduces repeated I/O, database calls, network calls.
2. The loop iterates over an unbounded production-sized collection.
3. The previous code avoided the repeated operation.
4. The finding includes the loop, the repeated operation, and the production path.
Skip:
- in-memory map/filter/reduce over already-loaded arrays
- speculative caching suggestions
- small constant-size loops
- performance comments without source evidence
This connects directly to D4.1 and D4.2:
D4.1: Add explicit criteria.
D4.2: Add few-shot examples.
D4.4: Use dismissal data to identify which criteria/examples need improvement.
7. Common Traps
7.1 Trap 1: Retrying Without Error Details
Wrong:
Right:
7.2 Trap 2: Retrying Absent Information
Wrong:
Right:
7.3 Trap 3: Treating Strict Schema as Full Validation
Wrong:
Right:
7.4 Trap 4: No Feedback Metadata
Wrong finding:
Better finding:
{
"category": "performance",
"detected_pattern": "speculative_performance",
"message": "This may be slow.",
"evidence": "...",
"severity": "nit"
}
Why:
7.5 Trap 5: Hiding Source Conflicts
Wrong:
When the source contains conflicting term data.
Better:
{
"termination_date": "2028-12-31",
"conflict_detected": true,
"conflict_details": "12-month term conflicts with stated 2028 termination date."
}