Random Bug Walk
Synthetic bugs. Useful test cases.
Request a bug that an application’s own tests miss. Each case comes with separate checks that prove the bug is there and that a fix removes it.
- Describe the kind of bug you want, on one or both applications.
- Review how the service reads your request. Reviewing is free.
- Start the catalog request. One runs at a time, paid from the shared judging budget, and the cases appear on this page as they are delivered.
Completed example · published evidence
A time-zone blind spot in Umami
A synthetic bug shifts daily pageview counts. The 271 measured original API tests still pass. All four separate checks pass on clean and reference-fixed copies; on the buggy copy, three non-UTC checks fail as expected while the UTC control passes.
In Los Angeles, the expected counts are 3, 8, 1; the recorded API counts are 2, 8, 2. This demonstrates a test-suite blind spot, not debugging-agent performance.
For catalog requests: Its bug report was written by NVIDIA Nemotron 3.5 Lightning on Nebius Token Factory, in 2 live calls (one for the bug card, one for the report). Every Umami case this service delivers includes that report. A request itself makes no model call.
Reading this example starts no replay. These are evidence links, not new bug-case or validator-package downloads.
Live requests unavailable until service availability is confirmed. No work has started.
What makes a case qualified?
The service must confirm that unchanged original tests pass on clean, buggy and reference-fixed copies. Separate verification checks must pass on clean and fixed copies and detect the intended failure on the buggy copy. Setup failures and missing checks are not reproduction.
A debugging package must exclude verification tests and the reference fix. Separate validator packages and validation actions become available only when the service reports them ready. Clicking a validation action alone does not establish that a replay ran.
Read the completed experiment’s scope and limitations