Managing Excessive Web Crawlers on Personal Code Repositories
By
freediver
Fresh out the oven, still warm. Top of the tray.
Summary
A developer shares their experience with dealing with excessive web crawlers and bots accessing their personal hobby code repository. The author describes the mixed feelings of flattery and annoyance at thousands of daily visitors digging through years-old code changesets. They detail their ongoing efforts to implement countermeasures against these crawlers, comparing the process to solving a sliding block puzzle with incremental improvements. The article focuses on practical approaches to manage unwanted traffic while maintaining accessibility for legitimate users.
Key quotes
· 3 pulledI host a bunch of hobby code on my server. I would think it's really only interesting to me, but it turns out every day, thousands of people from all over the world are digging through my code
On the one hand, wow, thanks, this is very flattering. On the other hand, what the heck is wrong with you?
It's been a lot like solving a sliding block puzzle. Lots of small moves and changes, and eventually it starts coming together
You might also wanna read
Next-Generation Laravel Forge: Modern Server Management Platform for Web Applications
The article announces the next-generation Laravel Forge, a server management platform for deploying web applications. It highlights key feat
Product Hunt·8mo ago
WebSparks: An AI-Powered Tool for Building Web Applications Without Extensive Coding
WebSparks is an AI-powered software engineer that transforms ideas into fully functional web applications without requiring extensive coding
innovirtuoso.com·1d ago
Joost de Valk publishes open Website Specification: 128 rules for modern, future-proof websites
Joost de Valk, creator of Yoast SEO, published the Website Specification (specification.website) — an open, platform-agnostic reference docu
ppc.land·1d ago
ZX Spectrum BASIC interpreter rebuilt from scratch to run natively in web browsers
A developer has rebuilt the ZX Spectrum's BASIC interpreter from scratch to run in a web browser, without emulating the original Z80 hardwar
boingboing.net·1d ago
Building mobile-safe layouts with CSS safe-area-inset properties
This article explains how to use CSS safe-area-inset properties to build mobile-friendly layouts that account for non-rectangular screen fea
polypane.app·1d ago
How to Set Up an Apache Reverse Proxy for an Ecommerce Website
This article provides a comprehensive, start-to-finish guide on setting up an Apache reverse proxy specifically for ecommerce websites. It c
blog.radwebhosting.com·2d ago