Validated blind-spot example
Random Bug Walk plants and validates synthetic bugs to expose gaps in an application’s tests.
One synthetic time-zone bug was deliberately planted in a pinned copy of Umami. The 271 measured original API tests pass despite that defect. All four added checks pass on clean and fixed copies. On the planted copy, the three non-UTC checks fail as expected while the UTC control passes. One released example is demonstrated here. Other upstream tests were not evaluated.
Built with: the issue writer’s model runs on Nebius Token Factory; app copies run on Vercel Sandbox, and Vercel hosts the site and stores status and large files on Blob. Neon Postgres holds the spend ledger. Tavily provides the released issue’s exact-phrase novelty searches. The code repository produces the evidence in the results repository.
Validated release, synthetic planted bug
Synthetic task: the bug in this case was planted on purpose in a copy of Umami, an open-source web analytics app. It is not a real Umami bug.
Requested the daily pageview counts for one website from March 7 to March 9, 2026, with the time zone set to America/Los_Angeles, and compared each day's count with the visits recorded for that local day.
Observed counts differ from the expected local-day counts in 3 of 3 time zones.
Twelve pageviews were recorded for one website at the UTC instants listed below. Several fall just before or just after local midnight in Los Angeles, Auckland and Kolkata, and one pair falls on either side of the Los Angeles daylight-saving change on March 8, 2026, so each local day's count depends on the selected time zone.
Request: GET /api/websites/11111111-1111-4111-8111-111111111111/pageviews (HTTP 200)
| Day bucket | Expected | Observed | Status |
|---|---|---|---|
| 2026-03-07 | 3 | 2 | Differs |
| 2026-03-08 | 8 | 8 | Matches |
| 2026-03-09 | 1 | 2 | Differs |
| Day bucket | Expected | Observed | Status |
|---|---|---|---|
| 2026-03-07 | 1 | 2 | Differs |
| 2026-03-08 | 6 | 8 | Differs |
| 2026-03-09 | 5 | 2 | Differs |
| Day bucket | Expected | Observed | Status |
|---|---|---|---|
| 2026-03-07 | 2 | 2 | Matches |
| 2026-03-08 | 7 | 8 | Differs |
| 2026-03-09 | 3 | 2 | Differs |
| Event | UTC instant | Unix seconds |
|---|---|---|
| pageview e01 | 2026-03-07T10:59:00Z | 1772881140 |
| pageview e02 | 2026-03-07T11:01:00Z | 1772881260 |
| pageview e03 | 2026-03-08T07:59:00Z | 1772956740 |
| pageview e04 | 2026-03-08T08:01:00Z | 1772956860 |
| pageview e05 | 2026-03-08T09:59:00Z | 1772963940 |
| pageview e06 | 2026-03-08T10:01:00Z | 1772964060 |
| pageview e07 | 2026-03-08T10:59:00Z | 1772967540 |
| pageview e08 | 2026-03-08T11:01:00Z | 1772967660 |
| pageview e09 | 2026-03-08T18:29:00Z | 1772994540 |
| pageview e10 | 2026-03-08T18:31:00Z | 1772994660 |
| pageview e11 | 2026-03-09T06:59:00Z | 1773039540 |
| pageview e12 | 2026-03-09T07:01:00Z | 1773039660 |
In America/Los_Angeles, daily pageview counts for March 7–9, 2026 should be 3, 8, 1; the recorded API response instead gives 2, 8, 2.
Written by NVIDIA Nemotron 3.5 Lightning on Nebius Token Factory, in 2 live calls: one for the bug card and one for this report. See the recorded call.
The model wrote the report from recorded symptoms; a human approved it before release. The measurements are API responses. Chart rendering and dashboard interactions described in the report were not independently verified.
Read the original frozen bug report: pageviews assigned to the wrong local day.
Original Umami API suite (271 tests measured): the upstream API tests that were run in each copy. Other upstream tests were not evaluated.
| clean copy | planted copy | fixed copy | |
|---|---|---|---|
| Original API suite | Original API suite, clean copyPass, as expected271 of 271 executed | Original API suite, planted copyPass, as expected271 of 271 executed | Original API suite, fixed copyPass, as expected271 of 271 executed |
| Added checks | Added checks, clean copyPass, as expected4 of 4 executed | Added checks, planted copyFail, as expected80 of 80 executed; 60 failing results: tzarg.auckland-day-counts, tzarg.kolkata-day-counts, tzarg.la-day-counts | Added checks, fixed copyPass, as expected80 of 80 executed |
The added checks run once in the clean copy, and 20 times each in the first planted and the first fixed copy (the stability rule, ADM-05), so those two cells count 20 times as many executions as the clean copy.
Rule codes refer to the admission rules: ADM-02 to ADM-06 cover this run's 13 fresh copies, the fixed copies passing, the planted copies failing as declared, stability and the negative probes; ADM-07 the card, issue and leak checks, the three exact-phrase novelty searches and a human check; ADM-08 this original-suite comparison; ADM-09 the release.
This result demonstrates a test-suite blind spot; it does not measure debugging-agent performance.
At the linked results snapshot, three published replays of this frozen release each recorded all six comparison cells matching expectations. The planted copy’s added-check suite failed as expected; the original API suite passed on all three copies, and the added checks passed on clean and fixed copies.
These historical records do not establish execution on the latest deployment. “Completed” alone does not establish reproduction. Their public decision fields (comparison, decisions and outcome_verdict) are null; the summary is supported by the recorded cells, not a published verdict.
A new replay requires the judge passcode and has a 35-minute hard limit. The browser status below refers only to this browser, separately from the historical records above.
Replay status: Ready to start. A replay runs three fresh copies of the released case, with a hard cap of 35 minutes.
The page above is complete without a replay.
1248 published files, 13.6 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
5c625132-0ab4-48c8-a7f7-d72729354e97 completed, not validated
Its admission also passed with a blind spot demonstrated, but no release names it: the one published release is for run 0495237f-9310-4472-885e-4728fddc1257, whose issue passed the human check (ADM-07).
1248 published files, 13.6 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
8bdc650f-6dce-4278-b9b8-d50431225b04 completed, not validated
Its admission also passed with a blind spot demonstrated, but no release names it: the one published release is for run 0495237f-9310-4472-885e-4728fddc1257, whose issue passed the human check (ADM-07).
1248 published files, 13.6 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
These are judges' replays, not factory runs, so they are not candidates for validation.
9a403b66-4b4e-5327-895b-e893e42acdab verification replay (judge), completed
102 published files, 2.1 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
dd28e4f2-4de9-5215-a50e-2a627a7e0e74 verification replay (judge), completed
102 published files, 2.1 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
dd8a971d-6ee1-5b6b-bf68-da9cb21648f1 verification replay (judge), completed
102 published files, 2.1 MB in all, plus the manifest: manifest.json, report.md, every file of this run.
| Factory runs in this site's records | 3 |
|---|---|
| Factory runs finished | 3 |
| Factory runs published | 3 |
| Factory runs with an observed symptom | 3 |
| Factory runs with an admission decision | 3 |
| Admission verdict pass | 3 |
| Blind spot demonstrated (ADM-08) | 3 |
| Validated releases (ADM-09) | 1 |
It counts only the factory runs in this site's results repository and its public store; any judge replays, paired comparisons and kit checks are counted apart below. Failed and incomplete factory attempts are not published here, so it is not a record of every attempt.
| Verification replays (judge) | 3 |
|---|---|
| Replays completed | 3 |
A replay's published records do not include its verdict (verified or not reproduced), so the replay lines count each replay's recorded outcome only: completed means the replay's run ended, not that it reproduced the release.
This case's admission used 13 fresh app copies: 1 clean, 5 fixed, 5 planted, 1 partial-fix and 1 stub-fix. The original API suite runs once in each copy, and the added checks run in each, 20 times in the first fixed and the first planted copy. The six cells above come from the first clean, planted and fixed copies. A blind spot is shown only when the original suite passes in all 13 copies while the added checks pass on clean and fixed copies and fail as declared on planted and probe copies.