Scope

The benchmark run evaluated a single model: ibm-granite/granite-4.1-8b. A total of 459 tests were executed across multiple categories, with no test level or category filtering applied. The run was not limited in scope ("limit": null) and did not use fail-fast mode ("fail_fast": false). The results are truncated ("results_truncated": true), meaning not all test outcomes are included in this payload.

Performance

The model achieved an overall pass rate of 199/459 (43.36%) and a score percentage of 67.07% of the maximum possible. The average latency was 2.89s, with a median of 2.27s. Time-to-first-token (TTFT) averaged 0.284s, and output speed averaged 109.79 tok/s. The total cost for the run was $0.0143, with 30723 prompt tokens and 129180 completion tokens used.

Category-Level Patterns

• Speed: Strong performance with a pass rate of 91.67% (55/60) and score percentage of 97.22%. All hard tests were passed perfectly.
• Cost: Excellent results, passing 29/30 tests (96.67%) with a score percentage of 90.67%. All hard and medium tests were fully passed.
• Hallucination: Moderate performance at 56.67% pass rate (34/60) and 73.33% score. The model struggled with edge cases and API claims but performed well on behavior and bug detection.
• Coding: Pass rate of 55% (33/60) and score percentage 64.75%. Performance declined with difficulty: 80% on easy, 50% on medium, and 35% on hard.
• Reasoning: Very low pass rate of 23.33% (14/60) despite a higher score percentage of 73.38%, indicating partial credit on many tasks. Hard and medium tests were particularly challenging.
• Refactoring: Poor pass rate of 15% (9/60) and 63.40% score. Performance was consistent across difficulty levels, with no level exceeding 20% pass rate.
• Security: Weakest category with only 6/60 passed (10% pass rate) and 44.14% score. Notably, all 20 easy tests failed despite moderate scores, suggesting incorrect but non-zero output.
• UI: Only 1/9 passed (11.11%), with very low score percentage (21.93%). The model struggled across all difficulty levels.

Notable Failures and Observations

• Two tests in the debugging category (debug-prototype-pollution-check-v2, debug-prototype-pollution-merge-v2) resulted in errors: "Spread syntax requires ...iterable[Symbol.iterator] to be a function", indicating a fundamental failure in code generation.
• The reasoning category included a test (reason-constraint-sla-breach) with extremely high latency (21.7s) and very low output speed (18.95 tok/s), suggesting potential looping or inefficiency.
• Despite high pass rates in cost and speed, the model showed significant variation in TTFT on UI tests (0.267s easy vs 0.935s medium), indicating inconsistent responsiveness on complex prompts.
• The security category’s 0% pass rate on easy tests is a critical concern, as the model failed basic security checks despite generating plausible outputs (non-zero scores).