The txtfeed standard
Every llms.txt in our directory is scored on 6 weighted dimensions totaling 100. The methodology is open-source and licensed CC-BY-4.0 so anyone can audit, fork, or dispute a score.
The 6 dimensions
Spec Compliance
Matches emerging standard structure: H1, > description blockquote, ≥3 H2 sections, Permitted/Restricted/Pricing/Contact sections.
Crawler Coverage
Explicit allow/disallow per major crawler: GPTBot, ClaudeBot, PerplexityBot, Googlebot, Applebot-Extended, Bytespider, Amazonbot.
Clarity
Machine-parseable, valid markdown, reasonable size (500B–200KB), well-formed link density, no contradictions with /robots.txt.
Completeness
Substantive content, pricing or explicit free declaration, contact info, citation/attribution examples.
Freshness
Last-Modified HTTP header recency: <30d=1.0, 30-90d=0.7, 90d-1y=0.4, >1y=0.0.
Pricing Transparency
Explicit per-crawl rates, billing terms, or explicit "free to crawl" declaration.
Current corpus snapshot
- Scored domains: 557
- Mean score across corpus: 50.2/100
- Mean score across top 10: 77.4/100
- Categories: Developer Tools (316), AI / ML (218), SaaS (9), Education (5), Cloud Infrastructure (3), E-Commerce (2), Legal & Compliance (2), Finance (1), Gaming (1)
Initial seed crawl: 2026-04-30. Crawler is stdlib-only Python (412 LOC). Re-crawls every 24 hours.
Top 10 by score
- 1.txtfeed.comAI / ML100.0
- 2.brightdata.comDeveloper Tools76.4
- 3.www.brightdata.comDeveloper Tools76.4
- 4.docs.getdbt.comDeveloper Tools76.0
- 5.cloudinary.comAI / ML74.9
- 6.cloudinary.comAI / ML74.9
- 7.www.cloudinary.comAI / ML74.9
- 8.zuplo.comDeveloper Tools74.2
- 9.docs.dynamic.xyzDeveloper Tools73.8
- 10.github.comDeveloper Tools72.9
Self-score disclosure
txtfeed's own /llms.txt self-scores 100/100 on this rubric. That is a circular result by construction — the rubric's authors wrote the file. Treat it as a worked example of the rubric, not as ranking evidence. The rubric is published here so anyone can audit, fork, or dispute it.
Disputes
If you believe a score is wrong (your own domain, or a competitor), email support@txtfeed.com with the domain + the dimension(s) you dispute + your reasoning. We re-crawl + re-score within 7 days and publish the resolution.