Benchmark
Every App Generator build, on real CI
When the App Generator verifies a project, it pushes it to a throwaway private repository, runs it on GitHub Actions, reads back what failed, fixes it and runs it again — until the build is green or the guardrails stop it. This page lists what actually happened, newest first. Failures stay on it.
Last 20 runs
0 green · 8 fixed on retry · 9 failed · 3 harness errors
8 of 17 builds that reached a CI verdict passed (47%). Median time to verdict 28m 24s.
By stack
go 4/9 passed
java 3/5 passed
python 1/3 passed
Runs
| When (UTC) | Stack | Outcome | CI runs | Jobs | Took |
|---|---|---|---|---|---|
| 2026-10-09 16:58 | java | Fixed on retry | 2 | frontend: success, verify: success, images: success | 8m 02s |
| 2026-10-09 16:06 | java | Fixed on retry | 2 | verify: success, images: success, frontend: success | 8m 15s |
| 2026-10-09 02:11 | java | Fixed on retry | 3 | images: success, frontend: success, verify: success | 11m 39s |
| 2026-10-09 01:38 | java | Failed | 8 | verify: failure, images: failure, frontend: success | 28m 24s |
| 2026-10-08 23:03 | java | Failed | 8 | verify: failure, images: failure, frontend: success | 20m 07s |
| 2026-10-08 00:20 | go | Fixed on retry | 3 | images: success, verify: success, frontend: success | 13m 18s |
| 2026-10-07 23:45 | go | Fixed on retry | 7 | verify: success, images: success, frontend: success | 32m 45s |
| 2026-10-07 21:38 | go | Failed | 5 | images: failure, frontend: success, verify: failure | 35m 59s |
| 2026-10-07 19:11 | go | Failed | 8 | images: failure, frontend: success, verify: failure | 38m 41s |
| 2026-10-07 16:41 | go | Failed | 4 | verify: failure, images: failure, frontend: success | 27m 36s |
| 2026-10-07 15:37 | go | Harness error | 1 | — | 10m 48s |
| 2026-10-06 22:06 | python | Fixed on retry | 5 | verify: success, frontend: success, images: success | 17m 01s |
| 2026-10-06 21:56 | python | Failed | 7 | verify: failure, frontend: success, images: failure | 48m 02s |
| 2026-10-06 21:47 | python | Failed | 8 | verify: failure, frontend: success, images: success | 37m 45s |
| 2026-10-05 14:59 | go | Fixed on retry | 7 | verify: success, images: success, frontend: success | 29m 36s |
| 2026-10-05 14:25 | go | Failed | 8 | images: success, verify: failure, frontend: success | 33m 38s |
| 2026-10-05 00:44 | go | Fixed on retry | 8 | verify: success, frontend: success, images: success | 30m 20s |
| 2026-10-04 23:41 | go | Failed | 8 | images: failure, verify: failure, frontend: success | 28m 22s |
| 2026-10-04 21:58 | go | Harness error | 1 | — | 6m 33s |
| 2026-10-04 21:57 | go | Harness error | 1 | — | 3m 59s |
| 2026-10-04 19:25 | dotnet | Fixed on retry | 3 | frontend: success, images: success, verify: success | 9m 37s |
| 2026-10-04 15:49 | dotnet | Fixed on retry | 2 | verify: success, images: success, frontend: success | 3m 59s |
| 2026-10-04 15:29 | dotnet | Green first try | 1 | frontend: success, images: success, verify: success | 1m 40s |
| 2026-10-04 15:26 | dotnet | Green first try | 1 | verify: success, frontend: success, images: success | 1m 44s |
| 2026-10-04 15:11 | python | Fixed on retry | 2 | images: success, verify: success, frontend: success | 4m 22s |
| 2026-10-03 20:40 | python | Fixed on retry | 2 | frontend: success, verify: success, images: success | 4m 15s |
| 2026-10-03 19:50 | python | Fixed on retry | 2 | frontend: success, images: success, verify: success | 4m 24s |
| 2026-10-03 19:48 | python | Fixed on retry | 2 | frontend: success, images: success, verify: success | 4m 19s |
| 2026-10-03 19:39 | python | Fixed on retry | 2 | images: success, frontend: success, verify: success | 4m 06s |
| 2026-10-03 15:25 | python | Fixed on retry | 2 | verify: success, frontend: success | 4m 10s |
| 2026-10-03 14:55 | python | Failed | 3 | verify: failure, frontend: success | 8m 20s |
| 2026-10-02 23:07 | python | Failed | 2 | frontend: success, verify: failure | 5m 59s |
| 2026-10-02 22:25 | python | Failed | 2 | frontend: success, verify: failure | 4m 09s |
| 2026-10-02 22:18 | python | Failed | 6 | verify: failure, frontend: success | 18m 45s |
| 2026-10-02 21:57 | python | Harness error | 1 | — | 1m 55s |
| 2026-10-02 21:40 | python | Failed | 1 | verify: failure, frontend: failure | 3m 51s |
| 2026-10-02 21:36 | python | Failed | 1 | frontend: failure, verify: failure | 3m 33s |
| 2026-10-02 21:09 | python | Failed | 1 | verify: failure, frontend: failure | 2m 53s |
| 2026-10-02 21:06 | python | Failed | 1 | frontend: failure, verify: failure | 3m 21s |
| 2026-10-02 18:32 | dotnet | Fixed on retry | 2 | verify: success, frontend: success | 4m 53s |
| 2026-10-02 18:11 | dotnet | Failed | 4 | verify: success, frontend: failure | 9m 37s |
How to read this
- Green first try passed on its first CI run. Fixed on retry failed, was repaired from the real CI output, and passed. Failed reached CI and could not be made to pass before the guardrails stopped the loop.
- Harness error means CI never returned a verdict — a GitHub API failure or a timeout on our side. Those say nothing about the generated code, so they are listed but kept out of the pass rate.
- There are no links to the runs themselves. Each project is someone’s own application, so the repository is private and deleted when verification ends. For the same reason this page records no descriptions, code or logs — only the stack, outcome and timings.
- Nothing is filtered, sampled or seeded. Every verification that finishes is written here.
Ask where AI fits — it costs nothing
Tell us the process you want AI to take on, or send the architecture or Terraform you are unsure about. Within a few days you get a short written plan: what we would build or change, what it would take, and what to avoid. No call required. Or try the App Generator and watch it prove its code on real CI.
We reply within one business day.