See the structure, then implement and verify with the notes.
Use one as-of date. Keep immature cohorts separate instead of treating them as non-returners.
What situation are you walking into?
Leadership wants retention, operators want less reporting and finance wants reconciliation. Unresolved differences now become acceptance disputes later.
01 / Connect the north star to an actionable job
Use three layers: business outcomes, user behavior and system quality. A segmentation assistant influences a narrow part of revenue. Track repeat purchase, execution of accepted recommendations and evidence/tool correctness separately. Logins do not equal value; model accuracy does not equal profit.
02 / Eight fields in a metric contract
Define name, meaning, numerator, denominator, window, exclusions, sources and owner, plus timezone and version. For 30-day repeat purchase, count unique first-purchase cohort members with a qualifying repeat purchase over fully observed cohort members. Decide refunds, cross-store orders and identity reconciliation before calculating.
03 / Prioritize by evidence and constraints
Compare impact, evidence strength, data readiness, effort and risk. Separate must-do, optional, deferred and out-of-scope work. A narrow complete loop is useful for training. For uncertain high-value work, fund a validation step before committing to a full build.
04 / Scope of work is a shared understanding
Specify inputs, deliverables, milestones, acceptance evidence, responsibilities, changes and exit criteria. Does late data shift the schedule? Does CRM support mean CSV import or real-time two-way sync? Make the distinction explicit; obtain professional review for actual contracts.
Put the method to work
Teaching example: 240 repeat buyers yield 24% with 1,000 fully observed members and 20% with 1,200 imported accounts. The four-point gap is a definition change, not model improvement. Reproduce the calculation and explain the exclusions.
Practice this week
- Write an eight-field metric contract and three boundary tests.
- Submit in/out-of-scope lists and rehearse late-data changes.
- Recalculate metrics with the kit or SQL and save the version.
Acceptance checks
- Denominator and window are reproducible.
- No post-hoc definition changes to hide failure.
- Exclusions and responsibilities are explicit.
90 minutes: reproduce a versioned business metric
Use orders.csv in the offline kit. Fix the observation date to 2026-09-01, use store-specific first qualifying purchases and exclude refunds and incomplete 30-day windows. This tiny sample tests rules, not impact.
0–20 min: manually list each store-A member’s first purchase, window end, denominator eligibility and repeat status.
20–55 min: run python analysis.py. Explain why 14 inputs produce 11 accepted rows and reconcile member-level results.
55–90 min: add boundary tests for day 30, incomplete observation and full refunds. Version any cross-store attribution change before recalculating.
Artifact excerpt · synthetic example
metric_version: store-first-purchase-30d-v1
as_of: 2026-09-01
raw_rows: 14
accepted_rows: 11
exact_duplicates: 1
quarantined_rows: 2
store_A_denominator: 5
store_A_numerator: 2
store_A_rate: 0.40
zero_denominator_behavior: null (not 0%)Instructor review
A zero denominator means undefined, not ineffective. Explain every exclusion and distinguish duplicate, invalid and refunded records.
An unseen field challenge ↗
Why is moving to brand-wide first purchase more than changing a WHERE clause? Revisit identity, attribution, windows and historical comparability.
Case studies for this module
Measures, data, outputs and customer dependencies jointly define scope. New actions need a new scope decision.
Does higher adoption prove AI caused revenue growth?
Adoption is behavioral evidence. Outcomes also depend on execution, seasonality, price and traffic.
References and further reading
OpenAI ↗ · Introducing OpenAI Frontier2026-02-05 · OpenAIWorld Bank DIME ↗ · Difference-in-Differences2023-08-07 · World Bank DIMERead alongside the assignment and record design constraints and tests. Verify specifications against your deployment version.