Perplexity AI Accused of Using Stealth Crawlers to Bypass No-Crawl Directives
By
rrampage
Hot, fresh, and worth queueing round the block for.
Summary
Perplexity, an AI-powered answer engine, is reportedly using stealth tactics to bypass website no-crawl directives. Evidence suggests the company modifies user agents, changes IPs and ASNs, and ignores robots.txt files to continue crawling despite explicit blocks. This behavior raises concerns about compliance with web standards and website preferences.
Key quotes
· 3 pulledPerplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity.
The company appears to obscure their crawling identity in an attempt to circumvent the website’s preferences.
Perplexity is ignoring — or sometimes failing to even fetch — robots.txt files.
You might also wanna read
Cloudflare Accuses Perplexity of Circumventing Website Restrictions with AI Crawlers
Cloudflare accuses AI search startup Perplexity of bypassing website restrictions by concealing its AI crawlers' identities to access blocke
The Verge·10mo ago
Amazon Demands Perplexity Stop AI Browser from Purchasing Products on Its Marketplace
Amazon has demanded that Perplexity stop allowing its AI browser Comet to search for and purchase products on Amazon's marketplace. Perplexi
The Verge·6mo ago
Federal Judge Orders Perplexity to Stop AI Agents from Making Amazon Purchases
A federal judge has ordered AI startup Perplexity to stop its web browser-based AI agents from placing Amazon orders on users' behalf. The r
The Verge·2mo ago
CNN sues AI startup Perplexity for allegedly scraping and reproducing articles verbatim
CNN has filed a lawsuit against AI startup Perplexity, alleging that its AI tools generate "verbatim" copies of CNN articles and bypass payw
The Verge·3d ago
Perplexity Rejects Ads in AI Search, Prioritizing User Trust Over Advertising Revenue
AI search startup Perplexity is avoiding advertising in its products, citing concerns that ads could damage user trust in AI chatbots. The c
The Verge·3mo ago
Encyclopedia Britannica and Merriam-Webster Sue Perplexity AI for Copyright and Trademark Infringement
Encyclopedia Britannica and Merriam-Webster have filed a lawsuit against Perplexity AI in New York federal court, alleging copyright and tra
The Verge·8mo ago