Who was scraping FeedBagel at 1,000 requests a minute?

The symptoms

The site kept dying. Pages that normally render in half a second took 30 seconds. The database box, which idles around a load of 3, was pinned at 27. The Next.js server sat at 300% CPU with memory climbing toward 5GB until it fell over. From the outside it looked like a memory leak or a bad query, and there was a little of both, but fixing them never made the load go away. Something was refilling the work queue as fast as we could drain it.

Proving it was not people

The tell was in what was being requested, not how much. In one 10-minute window the backend served 3,165 distinct article pages. Humans do not behave like that: real readers cluster on the homepage and a handful of current stories, and they read the same pages as each other. This traffic never repeated itself. It walked the catalogue methodically, about five different article pages per second, page after page after page.

That access pattern is also why it hurt so much. Human traffic concentrates on a few hot pages, so caches absorb it. A crawler walking the long tail hits a cold cache on every single request, and each of those requests was doing real work: a server render, related article lookups over vector embeddings, story cluster queries. The most expensive pages on the site, served thousands of times a minute to nobody.

So who was it?

Not one villain. A crowd. The user agents told most of the story:

• AI training crawlers. GPTBot (OpenAI), ClaudeBot (Anthropic), Amazonbot, Bytespider (ByteDance/TikTok), and Meta's crawler. These are the newest and hungriest arrivals, harvesting text for model training, and they are notorious for hammering content-heavy sites at exactly this volume.
• SEO tools. Semrush, Ahrefs, and friends, crawling the entire web to sell backlink data to marketers. They crawl everything whether you use their products or not.
• Classic search engines. Google, Bing, and a long tail of regional engines. The polite end of the spectrum, but they still eat every URL you advertise.
• Anonymous scrapers pretending to be Chrome. The worst group. No bot user agent, no identification, just a herd of IPs enumerating author pages and article slugs while dressed as a normal browser. Even after we blocked every named bot, these kept walking 124 distinct author pages every five minutes.

Why us? Nobody knows this site exists

This was the confusing part. FeedBagel is not famous. Why would anyone burn this much compute scraping it?

The answer: crawl volume tracks how many URLs you advertise, not how well known you are. FeedBagel is an aggregator that ingests thousands of articles a day, and every article gets a page, every author gets a page, every publisher gets a page. Our sitemap advertised around 48,000 article URLs, plus topic pages, plus author pages. To a crawler, that is not a small site. That is a buffet. Every bot that found the sitemap methodically ate the whole thing, then came back to check for changes, forever.

So: is someone scraping our awesome site? Yes, constantly. But not because it is awesome. Because it is large, and because in 2026 the web has a new background radiation: AI companies, data brokers, and anonymous scrapers re-downloading the entire internet on a loop. Any site with a lot of URLs gets this treatment. Most site owners just never look at their logs.

What we did about it

One decision made everything else simple: we do not need search traffic to individual article pages. FeedBagel links out to publishers; the product is the front page, the topic pages, and the feeds. So instead of trying to serve every crawler politely, we closed the long tail. Four layers, cheapest first:

• Stop advertising. The sitemap no longer lists article pages at all, and robots.txt disallows article, author, and publisher pages. The homepage, topic pages, directories, and this blog stay open.
• Enforce it, immediately. robots.txt is a polite request that crawlers re-read on their own schedule, and plenty ignore it entirely. So the proxy layer now returns a 403 to any identified crawler on long-tail pages, in microseconds, before any rendering or database work happens. Link previews from Slack, Telegram, WhatsApp, and the rest are exempt so shared links keep their cards.
• Rate-limit the impostors. The Chrome-cosplay herd sails past any user agent check, but no human reads 30 different article pages in a minute from one IP address. A per-IP limit on long-tail pages turns methodical enumeration into a stream of 429s.
• Make the pages cheaper anyway. Database indexes for the queries bots kept finding, related-article lookups moved behind lazy loading so they only run when a human actually scrolls to them, and smarter server-side caching. Whatever still gets through costs far less.

The numbers, before and after

• Backend requests: from 6,400 per five minutes to about 2,200, most of which is now real traffic and our own systems.
• Slow database queries: from 439 per five minutes to 44.
• Database box load: from 27 to 3.3.
• Next.js server: from 300% CPU and climbing memory to 29% CPU and flat memory. The "memory leak" was mostly bots holding the working set hot.
• Homepage time to first byte: from over a second (when it answered at all) to 0.4 seconds.

If you run a small site

Check your logs before you trust your instincts. If your site has a lot of URLs, a surprising share of your traffic, sometimes the large majority of it, is machines. Three things we learned the hard way:

• Your sitemap is an invitation. Only advertise pages you actually want crawled at full depth, forever.
• robots.txt is advisory. If a path being crawled costs you real compute, enforce the block in code, where it takes effect now and applies to bots that never read the rules.
• Watch for distinct URLs per minute, not just request counts. Humans repeat themselves. Crawlers never do. That one metric separated machine traffic from real readers all week.

The site is fast again, the servers are bored again, and the bots are getting exactly what they came for: a very quick 403.