20.3 C
New York
Saturday, June 20, 2026

Wired Confirms Perplexity Is Bypassing Efforts by Websites to Block Its Web Crawler

Last week, Federico and I asked Robb Knight to do what he could to block web crawlers deployed by artificial intelligence companies from scraping MacStories. Robb had already updated his own site’s robots.txt file months ago, so that’s the first thing he did for MacStories. However, robots.txt only …

This post was originally published on this site

Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments

Stay Connected

148,803FansLike
396,312FollowersFollow
2,690SubscribersSubscribe

Latest Articles

0
Would love your thoughts, please comment.x
()
x