Public, reproducible benchmark of CLI coding agents on SWE-bench Verified.
| Agent | Model | Pass rate | # tasks | Skipped | Median cost | Median time | Median tokens (in+out, % cached) | Last run |
|---|---|---|---|---|---|---|---|---|
| claude-code | glm-5.1 | 75% (CI 30–95%) | 4 | 0 | $0.59 | 186s | 38287 | 2026-06-08T05:23:02Z |
| claude-code | glm-5.2 | 67% (CI 21–94%) | 3 | 6 | $0.85 | 249s | 76875 (90% cached) | 2026-06-20T09:36:16Z |
| codex | MiniMax-M3 | 67% (CI 21–94%) | 3 | 2 | $0.38 | 84s | 332113 (47% cached) | 2026-06-20T09:36:16Z |
| aider | anthropic/claude-opus-4-7 | 33% (CI 6–79%) | 3 | 0 | $0.27 | 76s | 13877 | 2026-06-20T14:04:13Z |
| codex | MiniMax-M3 | 25% (CI 5–70%) | 4 | 0 | $? | 325s | 752198 | 2026-06-08T05:47:13Z |
| claude-code | 0% | 0 | 16 | $? | ? | 0 | 2026-06-08T03:18:28Z | |
| codex | 0% | 0 | 16 | $? | ? | 0 | 2026-06-08T04:09:35Z |