The first attempt to collect competitor prices ended with a captcha on the fifth request. That is how almost every scraper starts — and almost every one ends with a ban. The way out is proxies and the right strategy.
Sites watch simple things: how many requests fly from one address. A hundred per minute from one IP is visible from a mile away. The solution is to spread the load across different addresses and rotate them in time.
Practitioners suggest three rules. Pauses: 3-5 seconds between requests for ordinary shops, 8-10 for portals. Subnets: do not take all addresses from one — sites detect it. Freshness: a dead list means wasted time.
"With a live pool a scraper runs for months," say those who have been through this path. You can change the address automatically through an API — set it once and forget.
Websites watch simple things: how many requests fly from one address and how often. A hundred requests per minute from one IP is visible from a mile away. The solution is to spread the load across different addresses and rotate them in time. That is IP rotation for scraping: each request or batch of requests — a new address.
What I learned from practice over a couple of years. First: pauses. For ordinary shops, 3-5 seconds between requests. For big portals — 8-10. Second: do not take all addresses from one subnet — sites detect that. Third: if you need proxies for scraping classifieds, watch pool freshness even closer — their anti-bot is strict.
And last: scraping without a ban is possible when proxies are alive and rotating. A dead list means wasted time and empty logs. With a live pool that refreshes every five minutes, a scraper runs for months. You can change IP during scraping automatically through an API — set it up once and forget.