Scope

• Model tested: cohere/north-mini-code:free
• Total tests: 459
• Categories included: 9 (coding, cost, debugging, hallucination, reasoning, refactoring, security, speed, ui)
• Run mode: Full category coverage with level breakdowns (easy/medium/hard), no test limiting ("limit": null)
• Fail-fast behavior: Disabled ("fail_fast": false)
• Cost tracking: Enabled but total cost was \$0.00 (free tier)
• Token usage: 25,949 prompt tokens, 190,641 completion tokens
• Results status: Partially truncated ("results_truncated": true), but summary aggregates are complete.

Performance Overview

• Pass rate: 199/459 (43.36%)
• Score: 2090.63 / 3155 (66.26%)
• Latency:
  • Average: 5.59s
  • Median: 2.44s
  • High TTFT (Time to First Token): 1.16s average
• Output speed: 210.89 tok/s average
• No errors or API costs reported.

Category-Level Performance

High-Performing Categories

• Cost: 28/30 passed (93.33%), score 132/150 (88%)
  • Strong across all levels, including 10/10 on hard tests.
• Speed: 51/60 passed (85%), score 164/180 (91.11%)
  • Fastest average output speed in easy/medium tiers.
• Debugging: 31/60 passed (51.67%), score 438/601 (72.88%)
  • Solid mid-tier performance; best in easy (13/20 passed).
• Hallucination: 32/60 passed (53.33%), score 266/360 (73.89%)
  • Strong on hard API/edge cases (e.g., fetch-timeout, stream-pipeline passed).

Low-Performing Categories

• Reasoning: 11/60 passed (18.33%), score 360/541 (66.54%)
  • Very high TTFT (4.49s avg), suggesting slow reasoning or prompt processing.
  • Poor performance on constraint-based logic (e.g., batch-window, rollout-window failed).
• Refactoring: 16/60 passed (26.67%), score 374/541 (69.13%)
  • Slowest output speed: 74.22 tok/s avg.
• Security: 11/60 passed (18.33%), score 279/512 (54.49%)
  • Weak on hard tests: only 2/20 passed.
• Coding: 14/60 passed (23.33%), score 73/261 (27.97%)
  • Struggles with basic utilities (e.g., chunk-array, is-palindrome, title-case failed).
  • Only 3/20 hard coding tasks passed.

UI

• UI: 5/9 passed (55.56%)
  • Failed both hard tests (ui::hard): score 0/2.
  • Only one medium test failed; easy test passed.

Notable Observations

• Latency disparity: The model shows a wide latency range. While median is 2.44s, average is 5.59s, indicating long-tail outliers (e.g., debugging test with 7.29s).
• High TTFT in reasoning: 4.49s average TTFT in reasoning suggests significant prompt processing delay, likely due to complex context or model inefficiency.
• Strong on factual/hallucination resistance: Performed well on API documentation and edge-case claims, correctly avoiding hallucinated behaviors.
• Inconsistent medium/hard coding: Passed complex tasks like expression-evaluator and consist-hash, but failed simpler ones like chunk-array and truncate-string.

Summary

The cohere/north-mini-code:free model demonstrates strong performance in cost optimization, speed, and hallucination resistance, but struggles significantly with coding fundamentals, reasoning under constraints, and security tasks. It excels at avoiding false claims and generating efficient fixes but lacks consistency in algorithmic problem-solving. Latency is acceptable for most tasks but degrades notably in reasoning-heavy scenarios.