Scraping 241 UK council planning portals – 2.6M decisions so far

I've been scraping 241 UK council planning portals – 2.6M decisions so far UK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales. The scraping problem Most UK councils run one of a handful of portal sy