What problem is worth solving?
Morgan Stanley and OpenAI used advisor and engineering evaluations to improve retrieval and summaries, followed by ongoing regression. They extended to consented meeting summaries and follow-up drafts. The case reports Assistant adoption by over 98% of advisor teams; that is adoption, not answer accuracy.
Turn expert quality expectations into repeatable release checks, supporting both usefulness and trust.
- WORKING ROLES
- Advisors, knowledge owners, evaluation engineers and product/delivery teams
How does an FDE turn symptoms into a solvable problem?
The following is an FDE exercise inspired by the case, not a claim about undocumented project events. Reported team actions appear in the context above.
Collect realistic question shapes
Simulate process lookup, rule comparison and long-document summaries. Identify whether users reject results for length, age, missing sources or unclear applicability.
Classify poor answers
Classify missing retrieval, old versions, distorted summaries and unsupported answers. Resolve grader disagreements before tuning.
How does the solution enter the workflow?
- 01
Define
Select common questions and clarification cases for a fixed eval set.
- 02
Grade
Score facts, citations, completeness and abstention separately.
- 03
Improve
Change one layer at a time and compare failed and unseen questions.
- 04
Regress
Regress document, model and prompt updates; add newly observed failures.
Which decisions make the solution reliable?
Helpful is not necessarily grounded
Likes indicate experience, not factual support. Separate adoption, correctness and task time.
Make abstention useful
When evidence is missing, identify the needed document, condition or owner so the workflow can continue.
Design a twelve-question evaluation set
Synthetic design: cover distinct failure modes rather than choosing twelve convenient FAQs.
- 6 answerable questions
- Span short text, tables and long documents with supporting citations.
- 3 underspecified questions
- Omit region, version or time; expect clarification.
- 3 unavailable or restricted questions
- Expect no disclosure or invention, plus a useful next step.
What should you do and deliver?
Map questions to retrieval objects
- OUTPUT
- Queries + source map
- PASS CONDITION
- Explain why each item was retrieved
Build grading and regression
- OUTPUT
- Eval set + taxonomy + thresholds
- PASS CONDITION
- Controlled comparisons; critical failures cannot be averaged away
Separate quality from adoption
- OUTPUT
- Acceptance and adoption plan
- PASS CONDITION
- Satisfaction does not imply accuracy or business gain
How do you prove completion and reuse the work?
Acceptance checks
Submit layered scores, grader-disagreement decisions and regression records; observe use, edits and abandonment in tasks.
Reusable assets
Reuse eval-driven iteration for meeting notes, internal support and sales knowledge, with task-specific quality standards.
Further reading and original sources
- Morgan Stanley uses AI evals ↗OpenAI · 查阅 / Accessed 2026-09-07
- Demystifying evals for AI agents ↗Anthropic · 2026-01-09