Scope

• Model tested: deepseek/deepseek-v4-pro
• Total tests: 459
• Categories: 9 (coding, cost, debugging, hallucination, reasoning, refactoring, security, speed, ui)
• Run mode: Full category coverage, no limiting (--limit not set), not fail-fast
• Results: Truncated (only first few test outcomes shown)

Performance

• Pass rate: 242/459 (52.7%)
• Score: 2315.43 out of 3137 (73.8%)
• Latency:
  • Average: 13.50s
  • Median: 8.91s
  • Average TTFT (Time to First Token): 1.34s
• Output speed: 48.92 tokens/sec on average
• Total cost: $0.816

Category-level Patterns

Strong Performance

• Coding: 91.7% pass rate (55/60), score 96.2% — excels across all difficulty levels, achieving 100% pass rate on easy and hard subcategories.
• Speed: 88.3% pass rate (53/60), score 91.7% — very high success rate, especially on easy (90%) and hard (90%) levels.
• Cost: Perfect 100% pass rate (30/30), score 90% — flawless on all cost-related tasks.

Moderate to Weak Performance

• Debugging: 50% pass rate (30/60), score 76.2% — struggles on easy and hard levels (50% and 45% pass respectively).
• Hallucination: 50% pass rate (30/60), score 75.8% — inconsistent, particularly on easy (25% pass) and medium (65%) levels.
• Reasoning: Only 30% pass rate (18/60), score 73.2% — poor performance across all levels, with 35% (easy), 25% (medium), and 30% (hard).
• Refactoring: Very low 15% pass rate (9/60), score 61.9% — weakest category, especially on hard tasks (5% pass).
• Security: 15% pass rate (9/60), score 60.4% — poor across the board, with only 10% on easy and 20% on hard.
• UI: 88.9% pass rate (8/9), score 82.6% — strong overall, though only 5/6 (83.3%) on medium difficulty.

Notable Failures and Observations

• Critical errors in debugging:
  • Two tests (debug-prototype-pollution-check-v2, debug-prototype-pollution-prototype-merge-v2) failed with error: Spread syntax requires ...iterable[Symbol.iterator] to be a function — indicates a fundamental misunderstanding or code generation flaw.
• High-cost tests:
  • Several debugging and reasoning tests exceeded 30s latency (e.g., debug-cache-invalidation-race-v2 at 37.8s, debug-db-transaction-isolation-v2 at 40.8s).
  • Highest token count: 1417 tokens in coding-hard-scheduler, costing $0.00469.
• Lowest pass rates:
  • refactoring::hard: 5% pass (1/20)
  • security::easy: 10% pass (2/20)
  • hallucination::easy: 25% pass (5/20)
• High output speed outliers:
  • cost-generation-title-case: 100.5 tokens/sec
  • debug-rate-limiter-v2: 84.4 tokens/sec
• High TTFT outliers:
  • coding-medium-circular-buffer: 4.02s TTFT
  • reason-constraint-maintenance-window: 8.90s TTFT — unusually slow time to first token.

Overall, the model performs strongly in coding and cost optimization but shows significant weaknesses in reasoning, refactoring, and security, with some critical failures in debugging tasks involving prototype pollution.