Benchmark run
Started May 10, 2026, 9:53 PM · Recorded May 10, 2026, 10:37 PM · Ended May 10, 2026, 10:37 PM
Test suite v2 — Resilience · 045d4510abd0…
Generated May 10, 2026, 10:37 PM · qwen/qwen3-235b-a22b-2507
openai/gpt-5.3-codex4599 — coding, cost, debugging, hallucination, reasoning, refactoring, security, speed, ui--limit, no --fail-fast)results_truncated: true — output was truncated; full results may contain more detail.The model openai/gpt-5.3-codex achieved an overall pass rate of 281/459 (61.22%) and a score percentage of 77.66%. Average latency was 5.24s, with a median of 3.66s. Total cost for the run was $2.63.
Strongest categories:
coding: 96.67% pass rate, 98.47% score — excels in code generation across all difficulty levels.cost: 100% pass rate, 92.67% score — highly accurate in cost-aware coding and optimization.speed: 85% pass rate, 89.44% score — performs well on performance-sensitive tasks.Weakest categories:
refactoring: 21.67% pass rate, 67.10% score — struggles with code transformation tasks, especially on easy problems.reasoning: 35% pass rate, 74.68% score — poor on constraint-based reasoning, particularly hard problems (20% pass rate).hallucination: 50% pass rate, 74.44% score — frequent factual errors in API behavior, edge cases, and documentation.Two tests failed with execution errors:
debug-prototype-pollution-check-v2 — error: "Spread syntax requires ...iterable[Symbol.iterator] to be a function"debug-prototype-pollution-merge-v2 — same errordebugging category, suggesting issues with handling prototype pollution edge cases.High-latency outliers:
coding-hard-json-pointer: 12.76s latencycoding-hard-json-patch: 10.56s latencyhalluc-edge-regex-backtrack: 9.02s latencyHigh-cost tests:
coding-hard-json-pointer: $0.0212coding-hard-json-patch: $0.0168debug-env-flag-string-v2: $0.0095100% pass rate), slight drop on medium (90%). Strong output speed (120.34 tok/s on easy).66.67%), but performance improves with difficulty — hard problems have higher pass rate than easy in some subcategories.halluc-api-array-flat, halluc-doc-tc39-pipeline). Struggles with edge and complexity subcategories.20%), especially constraint-based (reason-constraint-subscription-migration). High output speed (251.09 tok/s on hard) but poor accuracy.10% pass rate), better on medium and hard — suggests misalignment with expected refactoring patterns.$2.6329,117 prompt + 186,148 completion = 215,265 total107.20 tok/s1.01s, lowest in coding (0.66s), highest in reasoning (3.32s)Despite high cost and latency in some tasks, the model demonstrates strong coding and cost-awareness capabilities, but significant weaknesses in reasoning and hallucination resistance.
Per-model aggregates from overall_ranking.json for this run id.
Values are read from report.json when the benchmark wrote them.
Test suite
v2 — Resilience
Discovery
Full suite discovery (no --limit)
blxbench argv
tui
App version
v1.3.2
Resumed run
No
Score % vs mean latency where samples exist. Built from per-test rows in report.json when available.
Avg score % (bars) and strict success rate % (line) per cost cluster.
Per-test latency (seconds), successful timings only.
Normalized TTFT (inverted) vs decode tok/s per category for this run.
Average score % per metric dimension across all v2 tasks in this run.
Tests per scope (blue bars), estimated spend per scope (green bars), and mean $ ÷ merged rows per category (cyan line).
Per-test rows from report.json → results — by category (collapsed by default), then by difficulty. COMPL from details when present. The Visual column is omitted when no test in this run has a details.visual score. Judge: verdict and overall (0–100) from judge_validation / validation_model for coding/UI (hover for summary and subscores). No HTML or screenshots in this table. Cost: per-task USD from cost_usd or usage.cost when recorded. Suite: same manifest version/hash for every row (this run).
459 tasks in 9 categories · Grouped by category, then by difficulty; row order within each table matchesreport.json results (benchmark execution order)