Made for Engineers
Troubleshoot a High-Impact Production Failure
Also useful for: people in Software Engineers
75 min
Advanced
Software Engineers
Engineer
Get Something Done
Career outcome:
Avoid Professional Risk
Stabilize a failing production service, test the strongest explanations, coordinate clear updates, verify recovery, and turn unresolved risks into owned follow-up work.
Create a free account to save your progress and use this Playbook.
Use this when
You're responding to a severe production failure, customer-facing outage, major degradation, or recurring fault that needs coordinated investigation and careful recovery.
You'll need
- Incident phase
- Severity
- Current state and impact
- Expected behavior and recovery criteria
- Timeline and evidence
- Recent changes
- System context
- Constraints and responders
- Communication needs
You'll get
- Customer or operational impact contained
- Current service state confirmed with independent health signals
- Incident timeline reconciled through the recovery window
- Leading cause supported by evidence or remaining uncertainty documented
- Risky hypotheses tested or explicitly deferred
- Recovery criteria met for the required observation period
By the end of this playbook
You will have:
- Customer or operational impact contained
- Current service state confirmed with independent health signals
- Incident timeline reconciled through the recovery window
- Leading cause supported by evidence or remaining uncertainty documented
- Risky hypotheses tested or explicitly deferred
- Recovery criteria met for the required observation period
Sample
This is sample data, not your prompt. Pro unlocks this higher-stakes Playbook. On Free Playbooks you can still copy the assembled prompt into any AI.
Sample input
Incident phase: Active investigation Severity: SEV-1 Current state: Checkout requests began failing at 14:07 UTC. Payment attempts return 5xx errors, primarily in North America. Browsing and carts are healthy. About 65% of checkout attempts are affected. Expected behavior: Checkout success above 99.5%, p95 below 800 ms, no payment queue growth, and no reconciliation mismatches for 30 minutes. Timeline: Release 4.18 completed at 13:52. Database connections crossed 85% at 14:04. Checkout errors began at 14:07. Worker concurrency was reduced at 14:15 with no clear improvement. Recent changes: Application release, payment provider certificate rotation, and a 30% traffic increase. System context: Checkout depends on inventory, payment authorization, an event queue, and the orders database. Constraints: Rollback takes eight minutes and requires incident lead approval. Application and database responders are online. The gateway specialist will arrive in 20 minutes. Communication: Engineering updates every 15 minutes, executive updates every 30 minutes, and customer status updates without an unconfirmed root cause.
Sample assembled prompt
Incident snapshot Confirmed: Checkout failures began... Browsing and cart functions remain healthy. Database connection usage rose before the... Inferred: The fault likely sits within... Confidence: medium. Not confirmed: Release 4.18 caused the... The timing supports investigation but does... Immediate containment order 1. Reduce nonessential checkout traffic if supported... High reversibility, medium expected impact, low... 2. Preserve application, gateway, queue, and... 3. Prepare release rollback, but do... 4. Avoid broad database restarts because... Leading hypotheses 1. Application release increased database connection usage. Confidence: medium. Test: compare connection behavior by application... 2. Traffic growth exposed an existing... Confidence: medium. Test: compare connection saturation with request... 3. Payment certificate rotation is causing... Confidence: low to medium. Test: inspect gateway handshake and authorization... Recovery gate Don't stand down...