What problem is worth solving?
Teaching scenario: sales declines and teams blame traffic, promotions or stockouts. Turn the broad question into reproducible localization and a test.
Move the review from arguing over charts to selecting the next useful business test.
- WORKING ROLES
- Business owner, operations, fulfillment, analysts and FDE
How does an FDE turn symptoms into a solvable problem?
The following discovery and implementation exercise uses a synthetic scenario.
Define what declined
Order value, paid sales and net sales differ. Fix windows, returns and cancellation rules first.
Locate the first changed stage
Decompose traffic, conversion, basket size, fulfillment and returns, then segment; aggregate change may reflect mix.
How does the solution enter the workflow?
- 01
Define
Recompute one baseline period and reconcile to the business ledger.
- 02
Localize
Find affected segments and retain unchanged comparators.
- 03
Compare
Compare stock, traffic quality and campaign hypotheses with counterevidence.
- 04
Test
Assign one actionable test with an owner, window and review.
Which decisions make the solution reliable?
Compute before summarizing
Generate metrics from reproducible queries; use language models to explain hypotheses with references.
Choose informative tests
When hypotheses imply opposite actions, distinguish them before changing price and channel together.
Stable traffic, lower revenue
Synthetic simplified data: visits 1,000→1,000, conversion 10%→8%, order value 100→100, excluding returns.
- Recompute
- Revenue is 10,000→8,000; localize the conversion change.
- Segment
- Check whether new-customer mix changed or within-group conversion declined.
- Validate
- Use availability and page-error records to distinguish stock and experience issues.
What should you do and deliver?
Define sales and conversion
- OUTPUT
- Contract + decomposition
- PASS CONDITION
- Business users can reproduce the numbers
Test unsupported analysis
- OUTPUT
- Analysis eval set
- PASS CONDITION
- Unsupported causes remain hypotheses
Create falsifiable actions
- OUTPUT
- Three-layer review brief
- PASS CONDITION
- Facts, hypotheses and tests are separate
How do you prove completion and reuse the work?
Acceptance checks
Deliver what happened, possible reasons and the first test, with queries, samples, owner and review date.
Reusable assets
Reuse metric contracts, anomaly taxonomy and action review; adapt channel mix and revenue definitions.
Further reading and original sources
- Building effective agents ↗Anthropic · 2024-12-19
- Demystifying evals for AI agents ↗Anthropic · 2026-01-09