What problem is worth solving?
Teaching scenario: an expert corrects an answer, but the same error returns next week. Turn corrections into traceable knowledge updates and regression tests.
Embed expert corrections into the system and reduce repeated error and rework.
- WORKING ROLES
- Knowledge experts, content owners, evaluation engineers and FDE
How does an FDE turn symptoms into a solvable problem?
The following discovery and implementation exercise uses a synthetic scenario.
Replay the faulty answer
Pin question, retrieval, knowledge version and generator version; distinguish stale content, stale indexing and interpretation errors.
Trace the correction
Check whether feedback has an owner and a release path beyond a chat log. Verbal agreement does not update the system.
How does the solution enter the workflow?
- 01
Capture
Capture input, answer, citation and failure type without personal data.
- 02
Validate
Verify the proposed correction against an authoritative source.
- 03
Release
Version knowledge and index with owner, validity and rollback.
- 04
Regress
Test the original error, paraphrases and opposite conditions; watch for recurrence.
Which decisions make the solution reliable?
Feedback is not automatic truth
Feedback can be mistaken or malicious. Unowned, unsupported or conflicting changes await review.
Fix the cause, not the sentence
Editing a document does not help a stale index; carry versions through retrieval, generation and release.
A policy changes from seven to fourteen days
Synthetic policy: v2 applies to new orders; old orders remain under v1.
- Input
- Establish order date and policy scope.
- Version
- Choose v1 or v2 by effective interval, not simply latest text.
- Regressions
- Specify outcomes for new, old and undated orders.
What should you do and deliver?
Add effective intervals
- OUTPUT
- Knowledge objects + citations
- PASS CONDITION
- Historical questions use applicable rules
Turn five corrections into tests
- OUTPUT
- Change eval set
- PASS CONDITION
- Fixes do not regress adjacent questions
Define maintenance and rollback
- OUTPUT
- Knowledge release runbook
- PASS CONDITION
- A peer can replay and restore versions
How do you prove completion and reuse the work?
Acceptance checks
Assess source validation, versions, validity and regression coverage, not correction counts.
Reusable assets
Reuse the failure–knowledge–evaluation maintenance loop for support, sales and policy assistants.
Further reading and original sources
- Building effective agents ↗Anthropic · 2024-12-19
- Demystifying evals for AI agents ↗Anthropic · 2026-01-09