unclecode/crawl4ai

At a glance

Rank

#210 of 401

#40 of 56 Python repos

Checks passed

8 of 17

Full list below

Best set up for

Claude Code

Score 72.4

Needs most work for

Cursor

Biggest fix: Cursor rules (.cursor/rules) (+11.5 pts)

What to fix first

Showing fixes for Claude Code

1. 1
   AGENTS.md / CLAUDE.md

   Add an AGENTS.md covering project goals, layout, setup commands, and conventions. Aim for 800+ chars of real guidance (not boilerplate).
   +13.2 pts
2. 2
   Type configuration

   Add a type config (tsconfig.json for JS/TS, mypy.ini or pyrightconfig.json for Python). Rust/Go/JVM/Scala/Swift/C#/OCaml/Haskell/Zig are typed by default.
   +7.9 pts
3. 3
   Pre-commit / git hooks

   Set up pre-commit (.pre-commit-config.yaml), husky, or lefthook to run format+lint on every commit.
   +5.3 pts

Score for each AI agent

Each agent weighs the checks by its own docs. Under each score are its top three. How each agent is scored →

Does unclecode/crawl4ai have agent instructions?

An AGENTS.md (or CLAUDE.md) at the root tells AI coding agents how to build, test and work in the repo. A few agents also read a file of their own.

AGENTS.md / CLAUDE.md

✗None at the root (no AGENTS.md, CLAUDE.md, AGENT.md, .claude/CLAUDE.md or .claude/AGENTS.md)

Agent-specific files

None: no GEMINI.md, .cursor/rules, .openhands/setup.sh, .aider.conf.yml or .agents/skills

Repos that have one

29 of 56 Python repos · 221 of 401 repos on the leaderboard · See examples →

# AGENTS.md

Guidance for AI coding agents working in unclecode/crawl4ai. Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Written mainly in Python.

## Where things are

- Dependencies are declared in `pyproject.toml`.
- Tests live in `tests`.
- Lint and format rules are configured in `setup.cfg`.
- CI is defined in `.github/workflows`; read it for the exact install, lint and test steps.
- A reproducible environment is defined in `Dockerfile`.
- Follow the conventions in `CONTRIBUTING.md`.

## Commands

<!-- Fill in the exact commands CI runs. -->
- Install: TODO
- Test: TODO
- Lint: TODO

How unclecode/crawl4ai compares

vs leaderboard average

0.4 pts above

Average 66.8

vs Python average

3.8 pts below

Average 71.0

Stars among Python repos

#13 of 56

One fix for the most agents: AGENTS.md / CLAUDE.md would raise the score for all 9 agents, by up to 13.3 points.

Similar repos

Same language and host as this repo, closest in overall score.

• Comfy-Org/ComfyUI67.6
  Python · 145k ★
• encode/httpx67.4
  Python · 15.7k ★
• Kludex/uvicorn67.4
  Python · 11.1k ★
• tqdm/tqdm67.1
  Python · 31.4k ★
• explosion/spaCy66.2
  Python · 34k ★
• django/django65.8
  Python · 91.5k ★

Same topics, any language

Repos tagged with the same GitHub topics, most shared first.

• firecrawl/firecrawl50.7
  TypeScript · AI agents, crawler, data extraction, LLM, Markdown, scraper, web crawler, web scraping
• open-webui/open-webui71.5
  Svelte · LLM, mcp, rag
• run-llama/llama_index70.5
  Python · AI agents, LLM, rag
• langchain-ai/langchain50.7
  Python · AI agents, LLM, rag
• mastra-ai/mastra84.4
  TypeScript · AI agents, LLM, mcp
• scrapy/scrapy86.8
  Python · crawler, web scraping