How I Built an llms.txt Generator That Actually Works at Scale

This is the technical companion to my I Built an llms.txt Generator, Showed It to the Creator of the Standard, and Had to Rewrite Everything — the human side is there, here's just the engineering. The goal: automatically generate a proper llms.txt hierarchy for any website — not a flat index of summaries, but a structured set of MD files where semantically related pages are merged into coherent documents. Here's how each layer works and what broke along the way. The Architecture Sitemap → Crawle