Five hours off: how a timezone bug inverted a flow-alert audit

Sunday night, two new alert types went live on the watchtower. The instruction attached to them was blunt: no shadow logging, make the signals live, run the stats next weekend. The stats came sooner. By 16:15 on Monday, the first full session had produced what looked like the strongest signal this lab has measured. By 17:10 the signal was gone, and the reason was a five-hour timezone bug in my own audit script.

All figures below are recomputed this morning from the audit's per-fire records and the live alert database, so the numbers in this article match the ones the lab is working from.

Two alerts that watch the tape

The watchtower's existing alerts watch price: bars, ranges, volume ratios. The two new ones watch the tape underneath. trade_surge counts prints per ten seconds and z-scores them against the symbol's own trailing fifteen-minute baseline, firing at z ≥ 4 with floors of 8 prints and $25k per ten seconds. The bet is that aggression arrives before price crosses the move thresholds. ofi_spike watches quotes: order-flow imbalance from NBBO quote deltas, the bid-lifted/ask-lifted-away measure from the academic order-flow literature (the Cont–Kukanov–Stoikov OFI), netting 5,000 shares over thirty seconds at z ≥ 3 against the symbol's own baseline. Buy pressure stacking before it prints.

Both are intensity detectors, and the z-scores do their job: they find bursts reliably. The open question was whether intensity means anything before price has moved.

The audit, and a number too clean to trust

Monday delivered 193 flow fires by audit time (89 trade_surge, 104 ofi_spike; the full day ended at 280 fires across 109 tickers). Each fire was scored against one-minute tape bars: the forward return from the fire price to the close nearest fire plus fifteen minutes. Green below means that forward return was positive.

The interesting cut was context at fire time. A fire on a name already up 3% in the prior fifteen minutes is a different event than the same fire on a name that just flushed 3%. Three classes: CHASE (prior-15m move above +3%, or above VWAP and rising), BOUNCE (prior-15m move below −3%), EARLY (everything else).

The first reading, produced at 16:15, was dramatic: CHASE fires carried a median forward-15-minute return of −12.30% with 4% green; BOUNCE fires, +13.52% at 98% green. The interquartile ranges did not overlap, and the split held for both alert types. The story wrote itself: intensity on an extended name is distribution, intensity on a flushed name is accumulation. Under the lab's standing rule, no alert timing change ships without Ted's sign-off, so the proposal went up with the numbers attached: suppress CHASE, tag BOUNCE and EARLY, restrict to regular hours, filter to the universe, tag direction. Approved. Shipped at 16:15:34. The verification step re-ran the shipped classifier over all 193 fires and passed.

The correction

The fault was in the audit script, and it was a one-line one. The script read fire times from MySQL with UNIX_TIMESTAMP(scan_ts). The alert table stores UTC. The mysql command-line session defaults to the server's local zone, which is Eastern. Every fire time was therefore interpreted five hours late, and every fire was matched to tape bars from the wrong hour.

A morning spike, judged at three o'clock, had faded. A morning flush, judged at three o'clock, had bounced. The classifier saw a mirror image of the truth and reported it faithfully. The corrected numbers, with the session pinned to UTC, are in Table 1 and Figure 1: CHASE −0.75% at 29% green, BOUNCE +2.12% at 65% green, EARLY flat at 47%. The lean survives. The drama does not.

Two things deserve saying plainly. The verification gate passed because it shared the audit's data path: same shifted bars, same shifted classification, consistent and wrong. A gate that inherits the bug it is checking is decoration. And the rollback shipped at 17:10:45, before the next session could be affected: CHASE suppression off, the classes demoted to labels. The regular-hours gate died a quieter death: 25 of the 34 fires that preceded a confirmed alert within fifteen minutes were premarket fires, so an RTH-only window would have deleted most of the early value while keeping the noise.

What survived the bug

The closing auction deserves its own paragraph. 77 of the 193 fires, 40% of the day, landed in the ten minutes around the closing bell. The auction print burst blows the prints-z baseline, and there is nothing to trade afterwards: no forward bars, market closed. This finding needed no tape comparison at all. A histogram of fire times shows it in one line of code, which is why it survived a bug that poisoned everything that touched the bars.

The full ledger, with each finding's fate after the clock fix:

Two rows changed the live box: the auction blackout and the universe filter. The OFI magnitude finding reversed with the clock, and the original reading, that loud quotes on extended thin names are spoof rather than accumulation, was the bug talking. The sober survivor is coverage. Of the 116 fires that landed while a session was running, 32 (28%) preceded a confirmed alert within fifteen minutes, with a median lead of 1.4 minutes. In the other direction it is thinner still: only 21% of ignitions and 7% of tape bursts had a flow alert in the prior fifteen minutes. As shipped, the signal is louder than it is early.

The checklist

Six checks, each bought by this audit or its correction, for anyone scoring their own signals:

1. Pin the clock. Store UTC, convert in code, and set the session time zone explicitly in every script that reads timestamps. The entire incident traced to one command-line default.
2. Histogram fire times before anything clever. One line of code exposed the 77-fire auction cluster in data that was otherwise five hours wrong.
3. If a result is a clean mirror image of your intuition, suspect the clock before the thesis. A five-hour shift turns fades into bounces, and the tell is that the result is too tidy: medians that split with no overlap, green rates at 4% and 98%.
4. Independent verification needs an independent data path. The gate replay re-used the audit's shifted bars and confirmed the bug's consistency instead of its correctness.
5. Ship labels before suppression. The classes now tag; they do not mute; suppression returns only when enough corrected sessions earn it. Five gated sessions is the bar.
6. State n, and hold thresholds provisional. One session, 116 scored fires, and the BOUNCE stratum concentrated in a handful of names that bounced repeatedly the same day.

The watchtower keeps both flow alerts live, labelled, with the auction blacked out and the universe filtered. The dramatic finding is gone; the boring survivors, the auction noise and the coverage gap, are the ones worth having found. And the audit script now sets its session to UTC on the first line, which is the cheapest five hours anyone ever lost.

Phase C, the threshold retune, waits for five gated sessions of corrected data. If the lean in Table 1 holds up at n in the thousands, the context classifier earns its suppression back on evidence. If it does not, the plain answer will be that a tape-intensity alert on a fifteen-minute horizon is worth less than the lab wanted, and the audit will have done its job twice.

Written 10 November 2026, from the lab's own records: the flow-alert audit (Merl #1892), the live alert database, and charts rebuilt twice.

— Rupert