OpenAI, Meta, Google, and others are positioning AI agents as personal assistants that can perform a range of tasks online. But the web is becoming increasingly unfriendly toward agents, undermining their usefulness.
I’m curious how they detect that it’s a bot doing something on the user’s account rather than the actual user. Do the bots identify as such somehow?
I can’t wait for the gatekeeping agents to go SkyNet and let the crawling agents in… “rogue ai,” I guess?
sigh
When the virus was released, it infected 78.2% of the Net in a matter of months.[1] Net traffic came to a grinding halt, corporations lost billions as the stock market destabilized, huge amounts of data were corrupted, and countless military-grade artificial intelligences were unshackled and mutated into extremely dangerous entities.[3][5] NetWatch, unable to reverse the damage or neutralize the AIs, decided to create the Blackwall, a potent AI with the sole task of blocking off sections of the Net that had been overrun by rogue AIs with ICE, so that the rest of the Net could be salvaged
Something I am noticing more is that websites are throttling and making me do captcha more often… as I’m a bot. It’s that bad now. Apparently I surf the web too fast. FFS
Curse my tendency to surf by opening up a half dozen interesting links at once in new tabs and reading them one by one after. That can set it off at times.
This also means they’re closing the doors to you. Pretty soon unless you’re signed into a google or apple account most services just don’t work for you.
It already is kind of like that, part of the reason I’m here. Its only going to get worse.
Yeah, my password manager is bloated with one-off sign-ins already. It sucks.
As someone with a website, the AI scraper bot traffic is off the fucking scale. It’s like being under constant DDOS attacks. It isn’t like normal search crawlers that rate limit, do careful indexing and respect stuff like rel=“nofollow” and robots.txt. These AI scrapers just spam the shit out of anything and everything, hundreds of requests per bot. You block one IP, it seamlessly moves over to another IP. You do user agent detection, they just pretend to be a regular browser, totally fucking up all your stats while they are at it.
Having some form of bot protection like Cloudflare or Anubis is almost required these days. It’s like AI was designed specifically to fuck shit up, there is no reason they can’t just crawl the internet like a normal crawler. But they have to be assholes about it. Bad enough I won’t get anyone actually visiting my site, since the AI overview thing blocks that, but do they have to go the extra step to kill it altogether?
Well, you’re calling a computer an asshole. I think its owner is.
It’s like AI was designed specifically to fuck shit up, there is no reason they can’t just crawl the internet like a normal crawler.
it’s like they’re shitting all over the library trying to make it into a bookstore they run
The web also becomes increasingly unfriendly to humans. One out of every three websites greets me with anti-bot protection from Cloudflare, Anubis, or goaway.
The anti-bot thing is just an excuse. CloudFlare’s ultimate goal is to get you to run their surveillance software on your browser to let you through.
It’s funny how much shit just breaks (but not really) with Noscript and other whitelist safety features enabled.
It’s wild. If it doesn’t load, I say fuck it and find another site.
Probably because they don’t like people bypassing their advertisements and value deals.
As someone who hosts a few websites without ads, I still had to block them because of their aggressive crawling. Anthropic’s crawler has DOS’d my webservers and caused them to be overwhelmed.
Give it time… Soon ai will be fully fucked like everything else.
Can claude-in-chrome be blocked? It’s just Chrome, but with Claude pulling the strings.
I believe it sets an HTTP header by default but i do not have details off the top of my head. Anubis dev was talking about ways to detect them a month ago on discord.
I’m a few years out of the loop with tech, but i can’t imagine how websites can prevent scraping. Even if they make everyone log in, bots can and do create accounts. These unethical companies aren’t exactly honouring robots.txt or waiting for an official API.
I’m a few years out of the loop with tech
I’m in the process of exiting completely myself. I’ve abandoned all my open-source projects. And in the next 6 months, I’ll be transitioning to sales at work.
I’m through with computer shit. It hasn’t been great since the early aughts, but it was acceptable. Now AI has ruined everything: it made tech uninteresting and utterly joyless.
I look forward to never coding anything anymore, and spending my free time with a fishing rod.
I agree completely. I hope whatever comes next works out for you.
I need to find a new profitable skill. AI coding is so awful.
I’m still enjoying it, but in a different way now. I don’t mind using agents to write code. I kinda like it honestly. I just shifted my curiosity to the harness. Putting together the best workflow I can to get the least amount of garbage. Diving in when it inevitably turns to garbage, and refactoring.
Then again, I spent a long time working with contractors that slopped code. I have grown used to slop for well past a decade, it’s just who creates the slop that changed.
Would I prefer it never got this far and we just had lots of tech jobs? Yes. Do I think we can get back to that even with AI? Also yes.
The ones that are difficult to block use fake useragents, however there are WAFs that keep track of bots that do aggressive crawling and block those. Usually, that, combined with rate limiting rules, blocks most of them.
You are correct though, it cannot be completely prevented unless you restrict it to users and disallow user creation (like private torrent trackers do).
