Test suites: why AI coding agents need tests they can find
This check looks for tests an agent could run: a test directory, or files named the way test files are named in each language.
How many repos pass
Of the 371 repos on the leaderboard, 193 pass in full (52%), 148 pass in part (40%) and 30 fail (8%).
Why agents need it
Tests are how an agent learns whether its change worked. An agent with a test suite can edit, run and correct itself. An agent without one hands back code nobody has run. Claude Code's profile weighs tests as heavily as AGENTS.md.
How we score it
- Full pass: a
tests/,test/,__tests__/,spec/,specs/orsrc/test/directory. - 70%: no such directory, but test-named files turn up within three folder levels, such as
*.test.ts,test_*.py,*_test.goor*Test.java. - Fail: neither. We check that tests exist, not that they pass.
The methodology has the formula that turns checks into a score.
How to pass
Keep tests in a conventional directory, and write the command that runs them in AGENTS.md. Tests that sit next to the source score 70%, because an agent has to search for them.
Which agents count it most
- Aider14% of its score
- Claude Code13% of its score
- Kimi CLI13% of its score
- Pi13% of its score
- Devin12% of its score
- GPT-5 Codex11% of its score
- Gemini CLI10% of its score
- OpenHands9% of its score
- Cursor8% of its score
Pass rate by language
Top-scoring repos that pass
- Rust · 33.1k ★
- gitlab-org/cli92.7— · 1.1k ★
- apache/superset92.2Python · 75.1k ★
- axios/axios91.9JavaScript · 109.4k ★
- aio-libs/aiohttp91.9Python · 16.6k ★
- C++ · 131k ★
- Rust · 6.7k ★
- TypeScript · 115 ★
- TypeScript · 2 ★
- Python · 46k ★
- scrapy/scrapy90.6Python · 64.8k ★
- Clojure · 49.6k ★
Popular repos that fail it
- Java · 81.6k ★
- ocornut/imgui22.1C++ · 76.6k ★
- Rust · 65.9k ★
- git/git62.2C · 63.8k ★
- karpathy/nanoGPT19.0Python · 63.7k ★
- Java · 60.3k ★
- TypeScript · 56.6k ★
- google/guava53.4Java · 51.9k ★