OpenAI, Meta, Google, and others are positioning AI agents as personal assistants that can perform a range of tasks online. But the web is becoming increasingly unfriendly toward agents, undermining their usefulness.
OpenAI, Meta, Google, and others are positioning AI agents as personal assistants that can perform a range of tasks online. But the web is becoming increasingly unfriendly toward agents, undermining their usefulness.
As someone with a website, the AI scraper bot traffic is off the fucking scale. It’s like being under constant DDOS attacks. It isn’t like normal search crawlers that rate limit, do careful indexing and respect stuff like rel=“nofollow” and robots.txt. These AI scrapers just spam the shit out of anything and everything, hundreds of requests per bot. You block one IP, it seamlessly moves over to another IP. You do user agent detection, they just pretend to be a regular browser, totally fucking up all your stats while they are at it.
Having some form of bot protection like Cloudflare or Anubis is almost required these days. It’s like AI was designed specifically to fuck shit up, there is no reason they can’t just crawl the internet like a normal crawler. But they have to be assholes about it. Bad enough I won’t get anyone actually visiting my site, since the AI overview thing blocks that, but do they have to go the extra step to kill it altogether?
Well, you’re calling a computer an asshole. I think its owner is.
it’s like they’re shitting all over the library trying to make it into a bookstore they run