Scope

This BLXBench run evaluated 1 model: qwen/qwen3.7-plus, across 459 tests spanning multiple categories including coding, cost, debugging, hallucination, reasoning, refactoring, security, speed, and ui. The run was not limited to a subset of tests ("limit": null) and was not fail-fast, meaning all tests were attempted even after failures. The results are truncated, indicating not all test outcomes may be included in this summary.

Performance

The model achieved an overall pass rate of 216/459 (47.06%) and a score of 2205.0982 out of a maximum possible 3155 (69.89%). The median latency was 4.81s, while the average latency was significantly higher at 9.44s, suggesting some long-tail inference times. The model demonstrated strong output speed at 55.07 tok/s on average, with a total cost of $0.352743.

Category-Level Patterns

• Coding (90.0% pass rate): The strongest category, particularly excelling in coding::easy (100% pass rate) and coding::hard (90% pass rate). The model reliably generated correct code across difficulty levels.
• Speed (91.67% pass rate): Performed exceptionally well, achieving 100% pass rate on speed::hard tests, indicating robustness under performance constraints.
• Cost (96.67% pass rate): Near-perfect performance, with 100% pass rate on cost::easy and cost::medium tests, showing accurate cost-aware reasoning.
• Debugging (35.0% pass rate): Performance declined significantly, especially on easy and hard levels (35% and 20% pass rates respectively). The model struggled with identifying subtle bugs.
• Reasoning (18.33% pass rate): Very low pass rate, particularly poor on reasoning::medium (5% pass rate). The model failed to correctly apply complex logical or constraint-based reasoning.
• Refactoring (3.33% pass rate): Extremely poor performance across all levels, with only 2 tests passed. The model failed to produce effective refactored code.
• Security (13.33% pass rate): Very low pass rate, with only 2 passes in easy and medium levels. The model showed minimal ability to identify security vulnerabilities.
• Hallucination (46.67% pass rate): Mixed results. The model correctly identified non-existent APIs (halluc-api-fetch-timeout, halluc-api-intl-segmenter) but failed many "claims" and "edge case" tests, indicating a tendency to invent incorrect behavior.

Notable Observations

• High Cost in Reasoning: Despite low accuracy, the reasoning category incurred the highest total cost ($0.132447), likely due to long response lengths required for complex justifications.
• Latency Variance: The large gap between median latency (4.81s) and average latency (9.44s) suggests some tests (e.g., long reasoning chains) took considerably longer than most.
• Category Strength Disparity: The model excels at generative tasks (coding, speed, cost) but is weak at analytical tasks (debugging, reasoning, security, refactoring), highlighting a key limitation.
• Truncated Results: The results_truncated flag means this summary may not reflect the complete run, potentially missing additional failures or edge cases.