The txtfeed standard
Every llms.txt in our directory is scored on 6 weighted dimensions totaling 100. The methodology is open-source and licensed CC-BY-4.0 so anyone can audit, fork, or dispute a score.
The 6 dimensions
Spec Compliance
Matches emerging standard structure: H1, > description blockquote, ≥3 H2 sections, Permitted/Restricted/Pricing/Contact sections.
Crawler Coverage
Explicit allow/disallow per major crawler: GPTBot, ClaudeBot, PerplexityBot, Googlebot, Applebot-Extended, Bytespider, Amazonbot.
Clarity
Machine-parseable, valid markdown, reasonable size (500B–200KB), well-formed link density, no contradictions with /robots.txt.
Completeness
Substantive content, pricing or explicit free declaration, contact info, citation/attribution examples.
Freshness
Last-Modified HTTP header recency: <30d=1.0, 30-90d=0.7, 90d-1y=0.4, >1y=0.0.
Pricing Transparency
Explicit per-crawl rates, billing terms, or explicit "free to crawl" declaration.
Current corpus snapshot
- Scored domains: 56
- Mean score across corpus: 45.2/100
- Mean score across top 10: 64.2/100
- Categories: Developer Tools (47), AI / ML (5), Cloud Infrastructure (2), SaaS (2)
Initial seed crawl: 2026-04-30. Crawler is stdlib-only Python (412 LOC). Re-crawls every 24 hours.
Top 10 by score
- 1.github.comDeveloper Tools72.9
- 2.docs.langchain.comAI / ML70.3
- 3.posthog.comDeveloper Tools66.6
- 4.docs.aws.amazon.comCloud Infrastructure64.5
- 5.docs.crewai.comAI / ML64.2
- 6.docs.replit.comDeveloper Tools63.1
- 7.docs.lovable.devDeveloper Tools63.0
- 8.docs.mistral.aiAI / ML59.4
- 9.resend.comDeveloper Tools59.1
- 10.vercel.comDeveloper Tools58.5
Self-score disclosure
txtfeed's own /llms.txt self-scores 100/100 on this rubric. That is a circular result by construction — the rubric's authors wrote the file. Treat it as a worked example of the rubric, not as ranking evidence. The rubric is published here so anyone can audit, fork, or dispute it.
Disputes
If you believe a score is wrong (your own domain, or a competitor), email [email protected] with the domain + the dimension(s) you dispute + your reasoning. We re-crawl + re-score within 7 days and publish the resolution.