Skip to content
Dev.to1 min read

Scraper worked on my laptop. Deployed to server...

Scraper worked on my laptop. Deployed to server and got instant 403s. Wrote a scraper last week for product data. Tested it locally, worked fine. Collected 200 products, zero issues. Deployed to my VPS Friday night thinking I could run it on a cron and forget about it. Saturday morning I check the logs. Every single request: 403 Forbidden. Zero data collected. Fun times. What broke Turns out the target site was checking User-Agent. My laptop had requests with a normal browser user agent because
Read original on dev.to
0
0

Comment

Sign in to join the discussion.

Loading comments…

Related

Get the 10 best reads every Sunday

Curated by AI, voted by readers. Free forever.

Liked this? Start your own feed.

0
0