When being offline is the emergency: designing a reliable dead man's switch

An in-progress note from building Batelys, a lone-worker safety app. The central mechanism is a dead man's switch: check in, confirm you are fine at intervals, and if a confirmation does not arrive, the system assumes the worst and escalates. It is the only system I have built where the failure mode is a person on the floor and nobody knowing.

That reframes a lot of ordinary engineering decisions, and this note is about which ones.

Availability stops being a metric

In most systems, availability is a performance number you optimise. Here it is a safety property. If the monitoring service is down when a check-in is missed, the alert does not fire, and the entire purpose of the product silently fails at the exact moment it was needed.

This changes what you are willing to ship, when you deploy, and where you put the authority for a decision. "It retried and eventually succeeded" is not acceptable when eventually might be twenty minutes and someone is unconscious.

The bug that taught me the most

The naive design keeps each worker's check-in deadline as a timer in the monitoring process. It is simple, it is fast, and it is catastrophically wrong — because the process restarts. A deploy, a crash, an autoscaler decision, and every in-memory timer vanishes. Everyone's monitoring window quietly disappears, and nothing tells you, because the absence of an alert looks exactly like everything being fine.

The fix is to move the authority for the deadline out of process memory and into storage:

Instead of:  setTimeout(escalate, until_deadline)   // dies with the process
 
Persist:     a row per active shift with a `check_in_due_at` timestamp
Poll:        any instance scans for rows whose deadline has passed
Escalate:    from whichever instance picks it up

Now a restart is survivable. Any instance can notice an overdue check-in, because the deadline lives somewhere a restart cannot erase. This is not a clever idea — it is the boring, correct one, and the boring correct one is the entire job in a safety system.

Offline and unresponsive look identical — until they don't

From the server's point of view, a worker whose phone has no signal and a worker who has collapsed produce the same observation: silence. The system must escalate on silence regardless. But treating them identically produces alert fatigue, and alert fatigue kills these systems as surely as a dropped timer does.

The resolution I am working with puts the distinction on the client. The app knows when it is offline. It warns the worker loudly and locally before anything escalates, and it queues the confirmation for replay. The server still treats prolonged silence as an incident — but the incident timeline records which of the two situations it was, so a pattern of dead-zone false alarms can be tuned out without ever using "probably just bad signal" as a reason not to escalate.

Delivery has to be layered

Push notifications are best-effort. SMS is best-effort. A single provider having a bad afternoon cannot be allowed to swallow an alert. So escalation is a chain of channels, each requiring a positive acknowledgement before it stops: push first, then SMS, then an automated voice call. "Nobody acknowledged" is itself an event, and it escalates rather than terminating.

Every transition in that chain is written down as an audit record, because after an incident the only question that matters is "who was told, when, and did they respond" — and that answer has to come from stored events, not from grepping logs after the fact.

What is still open

I am not finished. Fall detection using device sensors is on the roadmap and I do not yet trust any approach enough to rely on it — a false negative here is unacceptable and a false positive erodes the trust the whole system runs on. The tension between lone-worker protection regulations and employee-geolocation privacy law is real, and my current answer — capture position only when an alert fires, never continuously — feels right but has not been reviewed by anyone who does this for a living.

The through-line, and the thing this project keeps teaching me: for a system like this, the highest-value engineering is the boring reliability work nobody ever sees, precisely because when it works, nothing happens.