See the structure, then implement and verify with the notes.
Correlate task, trace, action and policy versions across model, tools and downstream. The timeline shows order, not measured latency.
What situation are you walking into?
The pilot moves toward operation. Failures may come from model errors, stale evidence, permission drift, connector timeouts, cost exhaustion or unowned alerts.
01 / Six dimensions of readiness
Assess function, reliability, security, observability, maintainability and economics separately. Define SLO windows and exclusions; one demo is not reliability evidence. Document dependency limits and incident ownership. Gate model upgrades through regression rather than silently following latest releases.
02 / Guard inputs, tools and outputs
Treat retrieved documents and tool results as untrusted data, not policy instructions. Validate input shape and scope; enforce tool authority and limits; check output evidence, sensitive exposure and execution status. A second model may help review but cannot prove the first is safe.
03 / Use traces to locate failures
Carry request, task-version, tool and evidence IDs with status, duration and cost. Logs record events, metrics show trends and traces connect the job. Store sanitized operational summaries, not unnecessary sensitive prompts or private reasoning. Diagnose cross-layer failures.
04 / Design rollout, economics and handover together
Roll out to bounded cohorts with predeclared stop thresholds and compensation. Cost each successful task including model, retrieval, tools, human review, infrastructure and failed attempts. Hand over ownership, config, versions, backup, upgrades, incident drills and knowledge maintenance—not just code.
Put the method to work
Teaching stop rules: disable an affected tool after any cross-store disclosure; revert to read-only after persistent failure-rate breaches; pause expansion when review cost exceeds the agreed task budget. Thresholds must be calibrated to business risk.
Practice this week
- Complete six readiness dimensions with evidence and owners.
- Rehearse a rollout stop and an incident diagnosis.
- Submit a cost model including review and retries plus a runbook.
Acceptance checks
- Critical failures have executable stop paths.
- Cost includes people and failures.
- Another operator can diagnose and recover using the runbook.
90 minutes: rehearse an operational failure before launch
Adoption falls while tool success stays healthy. Investigate stale evidence, poor task fit, duplicate work and missing feedback before changing models. Separate system, behavior and business measures.
0–20 min: define parent/child task traces, versions, states, latency and cost. Exclude secrets, raw sensitive text and private internal reasoning.
20–55 min: simulate retrieval timeout and unknown commit status. Rehearse on-call ownership, stop criteria, read-only fallback and reconciliation.
55–90 min: calculate model/tool, human review and fixed operating costs per successful task, explicitly listing omitted real-world costs.
Artifact excerpt · synthetic example
Cost example: 1,000 attempts, 85% completion
Model + tools: 1,000 × (0.5 + 0.1) = ¥600
Review: 1,000 × 2 minutes / 60 × ¥120 = ¥4,000
Fixed operation: ¥1,000
Per successful task: ¥5,600 / 850 ≈ ¥6.59
Excluded: implementation, exception losses, training, taxes, etc.Instructor review
Halving model price saves ¥250 in this example, not half of total cost. Reducing review requires quality evidence, not simply removing checks.
An unseen field challenge ↗
How do users experience a better mean but worse P95 latency? Design timeout budgets, a slow-task path and a meaningful alert.
Case studies for this module
Expand from read-only to bounded writes, with pause and reconciliation at every stage. Code rollback does not undo external effects.
Are total token counts enough to explain task failures?
Use correlated, minimally necessary operational evidence. More sensitive logging does not automatically improve observability.
References and further reading
OWASP ↗ · LLM01:2025 Prompt Injection2025 · OWASPOpenTelemetry ↗ · Generative AI semantic conventions查阅 / Accessed 2026-09-07 · OpenTelemetryNIST ↗ · AI Risk Management Framework查阅 / Accessed 2026-09-07 · NISTTampa General Hospital ↗ · TGH selects Palantir for connected care coordination2024-06-05 · Tampa General HospitalRead alongside the assignment and record design constraints and tests. Verify specifications against your deployment version.