Incident RoomIncident 3 of 4
Results day, database full
SmartSchool LK, a school results portal. The results site slows to a crawl as thousands of students log in.
- AWS
- PostgreSQL
- Docker
- Nginx
About 14 minutes
PAGE: results site timing out, p95 above 20 s
Results have just been released. The site is timing out and the app reports 'too many connections'. The principal is calling.
- Time since page
- 0 min
- Customers hurt
- 0
- Revenue lost
- Trust, not cash
- Status
- Outage
Your system
- Load balancerHealthy
- Web serversDegraded
- Results APIDown
- Connection poolDown
- PostgreSQLDegraded
Evidence and timeline
0 stepsYou are on call and the page just went off. Do not touch anything yet. Open a dashboard or a log from the Investigate list on the right, and find out what is really happening. Every action costs minutes, and every minute hurts customers.
Look before you leap. Good engineers read the evidence first, then act.
Dashboards
Logs
Deploys
Terminal
Costs are simulated minutes.

