Between October and March, a regional station I work with went silent four times. Not long — ninety seconds to four minutes — and never twice in the same week, which is exactly the shape of incident that gets logged, shrugged at, and archived. This is the postmortem of the six months it took to see that four incidents were one bug.
The scheduler filled its playlist a day at a time, and a day, to the scheduler, ended at midnight in the station’s timezone. Quarter boundaries were different: a quarterly compliance job rebuilt the rights window for every item in the upcoming schedule, and it keyed its query on an interval that was half-open on the wrong end. Items scheduled inside the last broadcast hour of the quarter fell out of the rights window, the playout engine refused them — correctly — and aired the one thing it is allowed to air without rights: nothing.
The reason it took four quarters to catch is the more useful lesson. We had a silence alarm, but it measured transport silence — no packets — and the engine was dutifully sending packets full of black. The alarm that would have caught it, black-frame detection on the program output, existed on a dashboard nobody was paid to watch at 23:00 on a quarter boundary. Every monitoring gap I have ever found has this structure: the signal existed, the attention did not. Detection you have to remember to do is not detection.
The fix was three lines in the rights query and one genuinely new alert: program output black or frozen for more than fifteen seconds pages a human, no dashboard involved. We also started replaying every incident against the previous year of schedule data, which is how we found that the bug had a fifth near-miss — a quarter where the last-hour item happened to be a station ident with permanent rights.
The postmortem template now has a required field: “what would have had to be true for this to be found on day one?” It is an uncomfortable question, which is the point. Four times out of five the answer is not new tooling. It is an alert wired to a consequence instead of a chart.