Silent job failures
At Inverter, a scheduled job could stop and the first person to notice was a customer.
Break a job, then switch between Before and After to see who notices first, a customer or an alarm.
Clock 09:00
- Job 1runs every 5 minRunning
- Job 2runs every 5 minRunning
- Job 3runs every 5 minRunning
- First to notice
- Nobody yet
- Time to notice
- -
- Alarms fired
- 0
What happened
ran finedid not runcrashedmessage failedTimes are illustrative.
I wrote a CDK construct that attaches no-invocation, error and dead-letter alarms to every scheduled Lambda by default, each linked to the runbook for that failure. A job cannot ship without them.
The runbooks followed. By the time I left I had written 35 architecture and operations documents: ten incident runbooks with a drill schedule, our first published postmortem, and a handover.