README and Onboarding
Split README, llms.txt, agent instructions, and detailed docs by audience.
Key takeaways
- Keep README as the human entry point and move agent rules, machine indexes, and deep references to their own surfaces.
- Each surface has a distinct reader:
README.mdfor humans,AGENTS.mdfor coding agents,CLAUDE.mdfor Claude Code,llms.txtfor LLMs and tools, anddocs/**for both. llms.txtis a proposal to place a Markdown index at/llms.txt; treat it as a clear index you control, not as a guarantee of search ranking, crawler behavior, or model-provider adoption.- Distinguish related files by job:
llms.txt/llms-full.txtfor the index,sitemap.xmlfor URL discovery,robots.txtfor crawl policy, and the internal RAG index for actual retrieval. - Validate that every
llms.txtlink resolves to a real file, ideally in CI or a scheduled job.
README still matters in the age of AI agents, but it should not carry every detail. Keep README as the human entry point and move agent rules, machine indexes, and deep references to their own surfaces.
Surface Roles
| Surface | Reader | Role |
|---|---|---|
README.md | humans | overview, quick start, major links |
AGENTS.md | coding agents | commands, rules, validation, prohibitions |
CLAUDE.md | Claude Code | Claude-specific memory and imports |
llms.txt | LLMs, tools, RAG | Markdown index for important docs |
docs/**/*.md | humans + agents | detailed guides, ADRs, runbooks, APIs |
Concise README
# Project Name
One-line description.
## Quick Start
\`\`\`bash
pnpm install
pnpm dev
\`\`\`
## Documentation
- [Architecture](docs/architecture.md)
- [API Reference](docs/api.md)
- [Contributing](CONTRIBUTING.md)
## Agent Instructions
- [AGENTS.md](AGENTS.md)
- [CLAUDE.md](CLAUDE.md)Treat llms.txt as a Proposal and Convention
llms.txt is a proposal to put a Markdown index at /llms.txt so LLMs and tools can find important
content without parsing a full HTML site.
Do Not Overclaim
llms.txt is useful, but it does not guarantee search ranking, crawler behavior, or model-provider adoption.
Treat it as a clear index you control, not as an AI SEO magic file.
# Project Name
> One-line project summary
## Docs
- [Getting Started](/docs/setup.md): installation and initial setup
- [API Reference](/docs/api.md): public API contract
- [Runbooks](/docs/runbooks/index.md): operational procedures
## Agent Instructions
- [AGENTS.md](/AGENTS.md): shared agent instructions
- [CLAUDE.md](/CLAUDE.md): Claude Code instructionsRelated Files
| File | Use |
|---|---|
/llms.txt | short Markdown index |
/llms-full.txt | optional longer index |
sitemap.xml | URL discovery for crawlers |
robots.txt | crawl policy |
| internal RAG index | actual retrieval pipeline |
Validation
awk -F'[][]|[()]' '/^- \[/{ print $4 }' llms.txt | while read -r path; do
case "$path" in
http://*|https://*) continue ;;
/*) target=".$path" ;;
*) target="$path" ;;
esac
if [ ! -e "$target" ]; then
echo "missing: $path"
exit 1
fi
doneChecklist
| Item | Check |
|---|---|
| README stays focused on human onboarding | [ ] |
AGENTS.md is separate from README | [ ] |
CLAUDE.md imports or cleanly separates from AGENTS.md | [ ] |
llms.txt contains only real links | [ ] |
llms.txt is not marketed as SEO guarantee | [ ] |
| links are checked in CI or a scheduled job | [ ] |