Revisiting the Impact of Robots.txt on My Blog: Lessons Learned
By
EPendragon
A baker's-dozen of insight crammed into one ring.
Summary
The author reflects on their decision to ban crawlers from their website using robots.txt, leading to unintended consequences such as broken LinkedIn posts and decreased visibility of their content.
Key quotes
· 3 pulledI was confident that I wanted to ban all the crawlers from my website.
Turned out there was an unintended consequence that I did not account for.
My LinkedIn posts no longer had the preview of the article available.
You might also wanna read
How to download tweets from any public X account using Circleboom
This article provides a step-by-step guide on how to download tweets from any public X (formerly Twitter) account in bulk using Circleboom's
circleboom.com·57m ago
Assemblage Mapping as a Tool for Accountability in AI-Supported Legal Advice
This article examines how AI is reshaping legal advice at the infrastructure level, arguing that legally consequential guidance is increasin
slsa.ac.uk·2h ago
Why Jet Aircraft Use Generators Instead of Alternators for Electrical Power
This article explains why modern jet aircraft use generators instead of alternators for their electrical systems. It covers the fundamental
slashgear.com·2h ago
New free course teaches researchers how to improve Wikipedia citations and field visibility
A new free online course called "Wikipedia for Researchers: Building reliable knowledge through citations" has been launched on WikiLearn. C
diff.wikimedia.org·3h ago
WordPress Accessibility Day 2026: Call for Sponsors
A call for sponsors for WordPress Accessibility Day 2026, a volunteer-led nonprofit conference focused on web accessibility education. Spons
buff.ly·4h ago
A Comprehensive Guide to Digital Accessibility Technologies for Computer Users with Disabilities
This article provides a comprehensive overview of digital accessibility technologies designed to help people with disabilities use computers
vale.rocks·5h ago