-0.1 C
New York
Sunday, February 15, 2026

Wired Confirms Perplexity Is Bypassing Efforts by Websites to Block Its Web Crawler

Last week, Federico and I asked Robb Knight to do what he could to block web crawlers deployed by artificial intelligence companies from scraping MacStories. Robb had already updated his own site’s robots.txt file months ago, so that’s the first thing he did for MacStories. However, robots.txt only …

This post was originally published on this site

Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments

Stay Connected

149,525FansLike
396,312FollowersFollow
2,650SubscribersSubscribe

Latest Articles

0
Would love your thoughts, please comment.x
()
x