OpenAI, Meta, Google, and others are positioning AI agents as personal assistants that can perform a range of tasks online. But the web is becoming increasingly unfriendly toward agents, undermining their usefulness.
OpenAI, Meta, Google, and others are positioning AI agents as personal assistants that can perform a range of tasks online. But the web is becoming increasingly unfriendly toward agents, undermining their usefulness.
I’m a few years out of the loop with tech, but i can’t imagine how websites can prevent scraping. Even if they make everyone log in, bots can and do create accounts. These unethical companies aren’t exactly honouring robots.txt or waiting for an official API.
I’m in the process of exiting completely myself. I’ve abandoned all my open-source projects. And in the next 6 months, I’ll be transitioning to sales at work.
I’m through with computer shit. It hasn’t been great since the early aughts, but it was acceptable. Now AI has ruined everything: it made tech uninteresting and utterly joyless.
I look forward to never coding anything anymore, and spending my free time with a fishing rod.
I agree completely. I hope whatever comes next works out for you.
I need to find a new profitable skill. AI coding is so awful.
I’m still enjoying it, but in a different way now. I don’t mind using agents to write code. I kinda like it honestly. I just shifted my curiosity to the harness. Putting together the best workflow I can to get the least amount of garbage. Diving in when it inevitably turns to garbage, and refactoring.
Then again, I spent a long time working with contractors that slopped code. I have grown used to slop for well past a decade, it’s just who creates the slop that changed.
Would I prefer it never got this far and we just had lots of tech jobs? Yes. Do I think we can get back to that even with AI? Also yes.
The ones that are difficult to block use fake useragents, however there are WAFs that keep track of bots that do aggressive crawling and block those. Usually, that, combined with rate limiting rules, blocks most of them.
You are correct though, it cannot be completely prevented unless you restrict it to users and disallow user creation (like private torrent trackers do).