Cold-Chain Monitor: Alert Rules, Maintenance Windows and Site Templates

Cold-Chain Monitor: Alert Rules, Maintenance Windows and Site Templates

You've joined the monitoring platform team at a cold-chain logistics company. Temperature sensors in refrigerated trucks, reefer containers and warehouses report to this service, which decides when a human gets paged. Last week's incident review (INC-4417) was about a pager storm: one truck door left open produced 61 pages in an hour while a dead battery on another truck produced none. The first generation of the service is in the repo; the roadmap behind it is three tickets long.

Temperatures are integers in tenths of a degree Celsius (35 is 3.5 C, -180 is -18.0 C). Times are integer seconds.

Codebase

How the service is driven

run(operations: list[list]) -> list executes the operations in order against a fresh engine and returns one result per operation. Every operation is [name, now, ...args] where now is an integer number of seconds that never decreases from one operation to the next. Errors are *returned* as strings "error: <code>", never raised.

• Operation shape errors come first: an operation that is not a list of at least two items, a now that is not an integer (booleans are not integers), or the wrong number of arguments returns "error: invalid arguments"; an unknown name returns "error: unknown operation" (checked before the arity). Argument types are checked before anything else the operation does: sensor_id must be a non-empty string, ts, value and every time must be plain integers, otherwise invalid arguments.
• A notification is [at, sensor_id, kind, contact]: at is the event time (see below) of what is announced, kind is "OK", "ALERT", "STALE" (and later "REMINDER", "FLAPPING"), contact is the rule's contact. sent returns the notifications produced since the previous sent, ordered by (at, sensor_id); notifications with an equal (at, sensor_id) keep the order they were produced in.
• state returns status ("OK", "ALERT" or "STALE"), since (the event time the status last changed; a new sensor is OK since the now of its define) and flapping (always false until Part 2).
• tick does nothing except move the clock. stats and replay are provided by harness.py.

How time works (read this once)

Readings carry their own timestamp ts (when the sensor measured it), separate from the now of the operation that delivers them. Gateways deliver readings late and out of order, so the service does not apply a reading the moment it arrives:

1. Accepting. reading is accepted when ts <= now and now - ts < 30 (LATENESS_SECONDS). ts > now is future reading; now - ts >= 30 is late reading (so age 29 is accepted, age 30 is not). Rejected readings have no effect. Checks run in the order of the table: argument types, unknown sensor, future reading, late reading.
2. Processing. Before it handles any well-formed operation, the engine computes frontier = now - 30 and *processes* everything whose event time is <= frontier: the buffered readings in (ts, arrival order) order and each sensor's own deadlines (silence, reminders, ...), interleaved in event-time order per sensor. Nothing newer is processed yet. So state and sent describe the world as of now - 30: a reading stamped 1000 is invisible at now = 1029 and visible at now = 1030. Because rejected readings are older than the frontier, nothing accepted later can ever be stamped before something that has been processed.
3. Event time. Everything a sensor does happens at an event time: a reading's ts, or the moment a deadline passes. Notifications and since use event time, never the now of the operation that happened to trigger the processing.

---

Part 1: Stop the pager storm (INC-4417)

Goal: replace the first-generation Evaluator ("a reading above high means ALERT and pages every time") with real alerting rules: debounce, hysteresis, silence detection and notifications on transitions only. Three plumbing defects found during the review are part of the ticket.

Symptoms from the incident review:

1. "The door was open for 40 minutes and we got a page for every single reading."
2. "One spike from a defrost cycle paged us at 3 a.m."
3. "The alert cleared and re-fired every minute while the temperature hovered right at the threshold."
4. "The reefer unit lost power and nobody was told the sensor went quiet."
5. "Notification times in the log are the time we processed them, not when it happened."
6. "Gateway replays: a reading exactly 30 seconds old was accepted, and two readings with the same timestamp were applied in the wrong order."

The rule

A rule is a dict (documented at the top of rules.py; unknown keys, wrong types, stale_after < 1, negative durations or an empty contact are invalid rule; high and stale_after are required):

define on an existing sensor id is sensor exists (checked before invalid rule).

Status rules

A sensor is OK, ALERT or STALE. Readings are applied in (ts, arrival) order.

1. Debounce. While the sensor is OK, a *run* of hot readings builds up: it starts at the first hot reading after any reading that is not hot (value <= high), and every hot reading continues it. The sensor becomes ALERT at the first hot reading with ts - run_start >= for_seconds (with for_seconds = 0 the first hot reading alerts). since is that reading's ts. A reading with value <= high ends the run, including value == high.
2. Hysteresis. While ALERT, a reading with value < high - hysteresis (strictly) makes the sensor OK; no run is in progress afterwards. Any other reading, hot or not, leaves it ALERT.
3. Silence. last_seen is the largest of the sensor's define time and the timestamps of its applied readings. At event time D = last_seen + stale_after the sensor becomes STALE (since = D) unless a reading stamped exactly D exists: readings are applied before any deadline at the same event time, so a reading at D moves last_seen and saves the sensor.
4. STALE wins. While STALE the sensor reports STALE even if it was in ALERT; the OK/ALERT machine underneath is frozen, and a run of hot readings in progress is discarded (a run never bridges a silence). The next applied reading leaves STALE: the machine resumes from where it was frozen (ALERT stays ALERT unless rule 2 clears it; OK evaluates the reading under rule 1 as the start of a fresh run) and the sensor takes the resulting status, which is a status change like any other.

Notification rules

1. Every status change produces exactly one notification with kind = the new status, at = the event time of the change and the rule's contact (including STALE -> OK and STALE -> ALERT). A reading that does not change the status produces nothing. A new sensor has no notification.
2. Reminders. While the status is ALERT and renotify > 0, a REMINDER is produced at last + renotify, where last is the event time of the most recent ALERT notification or REMINDER; after each reminder the next one is due renotify later. Reminders are produced even if no reading arrives (they are deadlines), and a long gap catches up on all of them. Recovery or STALE ends them.
3. Same event time. For one sensor, readings stamped t are applied first (in arrival order); then a STALE deadline at t; then a reminder at t. So a reading at the reminder time that clears the alert cancels the reminder, and STALE at the reminder time suppresses it.

Worked example 1: debounce, hysteresis, and the 30 second lag

Rule: high 80, for_seconds 120, hysteresis 20, stale_after 300, contact "dock-3".

[["define", 0, "truck-7", {"high": 80, "stale_after": 300, "for_seconds": 120, "hysteresis": 20, "renotify": 600, "contact": "dock-3"}],
 ["reading", 60, "truck-7", 60, 70], ["reading", 120, "truck-7", 120, 85], ["reading", 180, "truck-7", 180, 79],
 ["reading", 240, "truck-7", 240, 90], ["reading", 300, "truck-7", 300, 92], ["reading", 360, "truck-7", 360, 95],
 ["state", 389, "truck-7"], ["state", 390, "truck-7"],
 ["reading", 420, "truck-7", 420, 75], ["reading", 480, "truck-7", 480, 61], ["reading", 540, "truck-7", 540, 59],
 ["state", 570, "truck-7"], ["sent", 570], ["sent", 571]]

returns

[true, "accepted", "accepted", "accepted", "accepted", "accepted", "accepted",
 {"status": "OK", "since": 0, "flapping": false}, {"status": "ALERT", "since": 360, "flapping": false},
 "accepted", "accepted", "accepted",
 {"status": "OK", "since": 540, "flapping": false},
 [[360, "truck-7", "ALERT", "dock-3"], [540, "truck-7", "OK", "dock-3"]], []]

The 85 at 120 starts a run that the 79 at 180 ends; the run that starts at 240 reaches 120 seconds at 360, which becomes visible at now = 390. Readings of 75 and 61 stay ALERT (they are above 80 - 20 = 60); 59 clears it.

Worked example 2: late readings, reminders and STALE

[["define", 0, "freezer-2", {"high": 80, "stale_after": 180, "renotify": 100}],
 ["reading", 60, "freezer-2", 60, 95],
 ["reading", 100, "freezer-2", 75, 90],
 ["reading", 100, "freezer-2", 40, 50],
 ["reading", 110, "freezer-2", 111, 90],
 ["tick", 330], ["state", 330, "freezer-2"],
 ["reading", 400, "freezer-2", 400, 60],
 ["sent", 430]]

returns [true, "accepted", "accepted", "error: late reading", "error: future reading", true, {"status": "STALE", "since": 255, "flapping": false}, "accepted", <notifications>] where the notifications are [60 ALERT], [160 REMINDER], [255 STALE], [400 OK] (all contact "oncall"). The reading stamped 75 arrived 25 seconds late and is still applied (it moves last_seen to 75, so the silence deadline is 255); the reminder due at 260 never happens because STALE wins at 255; the reading at 400 resumes the frozen ALERT machine, and 60 is below 80 - 0, so the sensor is OK.

tests/ documents the existing pieces; tests/test_engine.py shows what currently fails.

---

Part 2: Maintenance windows and flapping (OPS-902, OPS-915)

Goal: operations needs two ways to keep the pager quiet without hiding the truth. A maintenance window silences a sensor's pages for a while (a truck in the workshop), and a sensor that keeps changing status gets one FLAPPING page instead of dozens (a door sensor with a loose wire). Silencing never changes a sensor's *status*; it changes only what the contact hears.

New operation: ["maintenance", now, sensor_id, start, end] returns true; errors in order: unknown sensor, invalid window (start >= end). Rules gain two optional keys, flap_max (int >= 1) and flap_window (int >= 1); flapping detection is on only when both are present in the sensor's rule. state.flapping now reports it.

Windows

1. A window [start, end) is half-open and covers *event times*: an event at start is covered, an event at end is not. A sensor can have many windows; they may overlap or touch. The channel to the contact is closed at event time t when some window of the sensor covers t, otherwise open.
2. A window affects only events processed after the maintenance operation (processing is described above). Anything already processed stays as it was; a window may start in the past.

What the contact hears (replaces Part 1 rule 5)

The contact has a *picture* of each sensor: FLAPPING while the sensor is flapping (below), otherwise its status. The service remembers the picture the contact last heard (initially OK).

1. Whenever the picture changes (a status change, or starting or stopping flapping) and the channel is open at that event time, a notification with kind = the new picture is produced at that event time, and the contact has now heard it.
2. If the picture changes while the channel is closed, nothing is produced. At the first event time u (after that change) at which the channel is open again, chained and touching windows skipped as one block, the service compares the sensor's current picture with what the contact last heard; if they differ, a notification for the current picture is produced with at = u. A picture that changed and changed back inside a window produces nothing, ever.
3. Reminders (replaces Part 1 rule 6): while the picture is ALERT and the contact has heard ALERT, a REMINDER is due renotify after last, where last is the event time the contact heard ALERT or the last time a reminder fell due. A reminder that falls due while the channel is closed is skipped (nothing is produced, and the next one is still renotify after the skipped one). No reminders while flapping or STALE, or while the contact has not yet heard ALERT.
4. Same event time, extending Part 1 rule 7: readings (arrival order), then STALE, then the end of flapping, then "the channel opens again", then reminders.

Flapping

1. Every status change of a sensor (including to and from STALE) is recorded with its event time. When a status change at time t makes the number of changes with t - flap_window < at <= t greater than flap_max, the sensor starts flapping at t (the change itself counts).
2. A flapping sensor stops flapping at the first event time x at which the number of recorded changes with x - flap_window < at <= x is at most flap_max (no new change is needed; a new change moves the moment later). Status changes keep being recorded and state keeps reporting the real status.

Worked example 3: an alert raised inside a window

Rule: high 80, stale_after 600, renotify 300, contact "dock-3"; window [100, 500) on truck-7.

[["define", 0, "truck-7", {"high": 80, "stale_after": 600, "renotify": 300, "contact": "dock-3"}],
 ["maintenance", 10, "truck-7", 100, 500],
 ["reading", 60, "truck-7", 60, 70], ["reading", 150, "truck-7", 150, 90],
 ["state", 200, "truck-7"], ["sent", 210],
 ["reading", 300, "truck-7", 300, 92], ["reading", 450, "truck-7", 450, 95],
 ["sent", 540],
 ["reading", 600, "truck-7", 600, 91], ["reading", 900, "truck-7", 900, 60],
 ["state", 930, "truck-7"], ["sent", 930]]

returns [true, true, "accepted", "accepted", {"status": "ALERT", "since": 150, "flapping": false}, [], "accepted", "accepted", [[500, "truck-7", "ALERT", "dock-3"]], "accepted", "accepted", {"status": "OK", "since": 900, "flapping": false}, [[800, "truck-7", "REMINDER", "dock-3"], [900, "truck-7", "OK", "dock-3"]]]. The alert started at 150 (the state shows it) but the contact is told at the moment the window ends, at 500; the reminder clock starts there, so the first reminder is 800.

Worked example 4: flapping

Rule: high 80, stale_after 900, flap_max 3, flap_window 600, contact "dock-3", readings alternate 90, 70, 90, 70, 90, 70 at 60, 120, ..., 360. Changes happen at 60, 120, 180 and 240; the fourth is more than flap_max, so the sensor starts flapping at 240 and the contact hears FLAPPING instead of the fourth change. The changes at 300 and 360 are not announced. The change at 180 leaves the window at 180 + 600 = 780, when only three changes remain, so flapping ends at 780 and the contact hears the current status, OK: [60 ALERT] [120 OK] [180 ALERT] [240 FLAPPING] [780 OK].

Part 1 behaviour is unchanged for sensors with no windows and no flap settings.

---

Part 3: Site templates (OPS-977, a change of requirements)

Goal: 1,400 sensors cannot be configured one by one, and ops wants to change a depot's thresholds in one place. Rules now come from per-site templates with per-sensor overrides. This changes how rules are stored and looked up: after this Part a rule is not a fixed property of a sensor that was copied in when it was defined.

(site_id must be a non-empty string, otherwise invalid arguments, like sensor_id.)

1. site creates a site, or replaces an existing site's template entirely. The template is a complete rule: high and stale_after are required and all other rule keys are allowed.
2. assign registers a new sensor in a site, exactly as define would register a standalone one (same starting state), with overrides a dict that may hold any subset of the rule keys (empty is fine). define is unchanged: a standalone sensor has no site and its rule is complete on its own. sensor exists also applies when the id was created by define, and vice versa.
3. A sensor's effective rule is the defaults, overlaid by its site's template *as it is now*, overlaid by the sensor's own overrides, key by key: a sensor that overrides only high still inherits the template's for_seconds, contact and the rest. The effective rule is resolved whenever the service needs it (every reading, deadline and notification), so replacing a template changes the behaviour of the sensors already assigned to it. Nothing is rewritten retroactively: statuses stay as they are and the new values apply to everything the service processes after the site operation, including events whose event time is earlier than that operation's now (a silence deadline that the new stale_after moves into the already-elapsed past fires, stamped at its own event time, on the next processing step).
4. site_maintenance adds a window covering every sensor that is in the site at the moment the window is consulted, including sensors assigned after the window was created. A sensor's windows are its own plus its site's. Windows follow Part 2 rules.
5. Disabling flap detection on a sensor that is currently flapping is not exercised.

Worked example 5

[["site", 0, "dc-east", {"high": 50, "stale_after": 300, "for_seconds": 60, "hysteresis": 10, "renotify": 600, "contact": "dc-east-noc"}],
 ["assign", 0, "cooler-1", "dc-east", {}],
 ["assign", 0, "cooler-2", "dc-east", {"high": 30, "contact": "vault-team"}],
 ... readings of 40 for both sensors at 60, 120 and 180 ...
 ["state", 210, "cooler-1"], ["state", 210, "cooler-2"],
 ["reading", 240, "cooler-2", 240, 20], ["reading", 300, "cooler-2", 300, 19],
 ["sent", 330]]

cooler-1 (high 50) stays OK. cooler-2 has high 30 but inherits for_seconds 60, hysteresis 10 and stale_after 300: its run starts at 60 and alerts at 120 (contact "vault-team"). The 20 at 240 is not below 30 - 10, so the alert stays; the 19 at 300 clears it: [[120, "cooler-2", "ALERT", "vault-team"], [300, "cooler-2", "OK", "vault-team"]].

Parts 1 and 2 must keep working for standalone sensors, and sensors in sites must behave exactly like standalone sensors with the same effective rule.

---

Part 4: A rehearsal at scale (stretch)

Goal: the platform team rehearses a busy night with ["replay", now, spec] (see harness.py: it defines spec.sensors sensors and feeds them spec.ticks rounds of readings through the dispatcher, one reading operation at a time). The hidden tests replay about a thousand sensors for several rounds, tens of thousands of operations in one run, and the whole suite gets a few seconds. Make that fast without changing a single notification: stats reports a digest of everything ever sent, which must stay identical.

Hint: every operation currently makes the engine look at every sensor.

---

Notes

• config.py, errors.py, harness.py are known-good. You are free to restructure anything else, but run and the operation vocabulary above are the contract.
• You may use the AI assistant freely; you are expected to verify what it produces.
• Later Parts must not break earlier ones; hidden tests re-check earlier behaviour after later Parts.
• Parts 1-3 is a strong result. Part 4 is a stretch.