DreamHost adds default agents.txt to websites, discouraging LLM training without prior notice
By
speckx
17d ago· 2 min readenNews
70/100
Toasty
Bagelometer↗
Lightly browned and well buttered. A solid pick from the rack.
Score70TypenewsSentimentneutral
Summary
DreamHost has added a default agents.txt file to all hosted websites (including retroactively to existing sites) that discourages LLM training and automated agent actions while allowing on-the-fly AI-generated summaries. The file is similar to robots.txt but targets AI agents. While the intent is sensible, DreamHost added it to existing sites without prior notice, and the implementation uses a proposed specification that has already changed.
Key quotes
· 4 pulledI host most of my websites on a DreamHost VPS. This morning I discovered that a new file had been added, agents.txt, to the root of each site, on May 7.
It was easy to confirm that this is a new default file similar to the default robots.txt and favicon.ico DreamHost puts in every new site to get you started.
Apparently they retroactively added it to sites that don't already have one. So it's a host action, not a hack. That's good at least.
The contents are simple, and sensible for a new website: Discourage LLM training and actions, allow on-the-fly 'AI'-generated summaries, disallow access
DreamHost now adds a default agents.txt (similar to robots.txt) to hosted websites that discourages LLM training and agent actions and allows on-the-fly access. On the downside, they added it to existing sites without notice, and used a proposed spec that
