Benchmark run
Started May 10, 2026, 10:57 PM · Recorded May 10, 2026, 11:46 PM · Ended May 10, 2026, 11:46 PM
Test suite v2 — Resilience · 045d4510abd0…
Generated May 10, 2026, 11:46 PM · qwen/qwen3-235b-a22b-2507
openai/gpt-5.54599 (coding, cost, debugging, hallucination, reasoning, refactoring, security, speed, ui)--limit or --fail-fast used)results_truncated: true — full per-test details are not present in this payload.284/459 (61.87%)2443.87 / 3137 (77.90%)$6.786.06s3.95s1.09s102.68 tok/s58/60 passed (96.67%), score 257/261 (98.47%) — strongest category.
100% score).52/60 passed (86.67%), score 161/180 (89.44%)29/30 passed (96.67%), score 137/150 (91.33%)9/9 passed (100%), score 7.87/9 (87.49%)32/60 passed (53.33%), score 383/512 (74.80%)38/60 passed (63.33%), score 449/583 (77.02%)
75% pass) > medium (65%) > easy (50%).34/60 passed (56.67%), score 284/360 (78.89%)
20/60 passed (33.33%), score 412/541 (76.16%)
12/60 passed (20%), score 353/541 (65.25%)
10% pass).refactoring was most expensive category: $1.62ui surprisingly high cost: $1.48, driven by large completion tokens (49.5k).reasoning (2.48s), likely due to complex constraint analysis.refactoring, security, and debugging have lowest TTFT (~0.78s), suggesting fast starts but not always correct completions.reasoning (148.47 tok/s)security (67.55 tok/s)debugging tests (prototype-pollution-check-v2, prototype-pollution-merge-v2) failed with runtime error: "Spread syntax requires ...iterable[Symbol.iterator] to be a function" — likely model hallucinated invalid syntax.debugging::nullish-cache-hit-v2, model scored 7/10 but passed false — indicates partial credit scoring in use.openai/gpt-5.5 excels in coding, cost optimization, and speed tasks, with near-perfect performance on algorithmic and efficiency challenges. It struggles significantly with reasoning under constraints and code refactoring, and shows inconsistent behavior on hallucination avoidance. The high cost in ui and refactoring suggests verbose or inefficient outputs in those domains.
Per-model aggregates from overall_ranking.json for this run id.
Values are read from report.json when the benchmark wrote them.
Test suite
v2 — Resilience
Discovery
Full suite discovery (no --limit)
blxbench argv
tui
App version
v1.3.3
Resumed run
No
Score % vs mean latency where samples exist. Built from per-test rows in report.json when available.
Avg score % (bars) and strict success rate % (line) per cost cluster.
Per-test latency (seconds), successful timings only.
Normalized TTFT (inverted) vs decode tok/s per category for this run.
Average score % per metric dimension across all v2 tasks in this run.
Tests per scope (blue bars), estimated spend per scope (green bars), and mean $ ÷ merged rows per category (cyan line).
Per-test rows from report.json → results — by category (collapsed by default), then by difficulty. COMPL from details when present. The Visual column is omitted when no test in this run has a details.visual score. Judge: verdict and overall (0–100) from judge_validation / validation_model for coding/UI (hover for summary and subscores). No HTML or screenshots in this table. Cost: per-task USD from cost_usd or usage.cost when recorded. Suite: same manifest version/hash for every row (this run).
459 tasks in 9 categories · Grouped by category, then by difficulty; row order within each table matchesreport.json results (benchmark execution order)