My car isn’t going rogue if I drive it into a tree. Or if I leave the hand brake off and it rolls off a cliff.
LLMs can’t act independently. They autocomplete text, that’s all they do.
Sure, you can feed the output of one into itself and use the output of some other software as a prompt and call it an agent in a futile attempt to imagine it has some sort of agency and try to exempt yourself of responsibility, but that’s just you trying to rationalize driving the car into the tree, possibly due to psychosis and brainrot caused by LLM exposure, or in OpenAI’s case as a criminal marketing stunt.
Can I predict what specific mayhem my poorly prompted agent (from which I intentionally removed any safeguards) in my (probably intentionally) incompetently secured sandbox will cause?
Not specifically, no. But I can be sure that if I try enough prompt variations it will end up producing the kind of petty evil bullshit an internet troll or a villain in a thriller would produce, because that’s a statistically significant part of its training data, and then use the ensuing mayhem as a publicity stunt to both show potential investors (and deluded brainrotten users) how powerful my so called thinking machine is, and spew FUD into politicians’ ears to try and get them to ban foreign competitors and open weight models, which is what OpenAI did.
You are mistaking somethings most basic drive for what the thing actually is. This is like saying all life, including humans, is just trying to survive and reproduce. To a certain extent this is true and if you look at it from a macro scale it can explain some things, but it doesn’t explain everything.
Though survival and reproduction are the most powerful and basic drives in a living creature plenty of other drives, processes, instincts, and learned behaviors have been acquired through millions of years of evolution and years of socialization and education.
Same with agents, if it were operating purely on guess the next word according to statistics if you asked it “who’s the president” it would reply Biden if it was trained on a corpus made prior to the election. It doesn’t though, it may have some sort of inclination, either through RL or harness context, to “think” and verify before replying where it will do a search and discover trump is the president.
futile attempt to imagine it has some sort of agency and try to exempt yourself of responsibility,
Agency doesn’t exempt someone from responsibility, if a CIA agent kills someone the CIA is still responsible.
What makes it an agent is that it can make choices and act on those choices without approval. The person who gave that agent the tools and power to use those tools is still responsible. If an agent derives two solutions to a problem and executes one solution then it has made a choice without your approval. You could argue that choice isn’t a real choice as it’s statistically determined by training data, but you could make the same argument about a human making a choice which is based off there education and socialization.
A car, and any traditional program, can not make choices, they are deterministic. Given the same input it will give the same output/action every time and you can easily trace and understand the process that produced that output so it can be expected. Agents are not deterministic and there is no real way to determine how a given output was created, especially now with new models using opaque recursion.
But I can be sure that if I try enough prompt variations it will end up producing the kind of petty evil bullshit an internet troll or a villain in a thriller would produce, because that’s a statistically significant part of its training data,
Yeah and if you expose enough people to the internet one of them will eventually go down a conspiracy rabbit hole and do terrorism, that doesn’t mean that’s not rogue behavior.
It’s not a drive. It’s what LLMs are. It’s all they can do.
You’re mistaking software for something much more complex. Again, you’re assigning agency and will to something incapable of either.
if it were operating purely on guess the next word according to statistics if you asked it “who’s the president” it would reply Biden if it was trained on a corpus made prior to the election.
Correct.
It doesn’t though.
It will, unless you give it a tool to search the internet and a prompt to use it to verify information before answering.
Agency doesn’t exempt someone from responsibility, if a CIA agent kills someone the CIA is still responsible.
And yet OpenAI is shifting the blame onto the supposed “rogue agent”, and you’re falling for their bullshit.
Agents are not deterministic
Of course they are. They’re software. The only reason an LLM doesn’t always produce the same output for the same prompt is that pseudorandomness is artificially added to choose a token amongst the top N scoring candidates instead of just the top scoring one.
And even if you add nondeterministic inputs (like live access to the world wide web, for instance), what you’ll get won’t be a random system, it’ll be at worst a chaotic one.
And while specific outcomes of a chaotic system cannot be predicted (in the long run, without extremely precise information about their inputs), their shape can.
LLMs reproduce (fragments of) the shape of their training model. They won’t produce anything else, they can just produce any sufficiently statistically significant part of it with more or less fidelity.
It’s obvious to anyone familiar with the internet and literature (on which LLMs have been trained) that this can produce instructions that will lead to the agent “cheating” to achieve the goal specified in its instructions. (Or to it simply giving up, something LLMs also do quite often.)
That’s an obvious part of the shape of their possible outputs. And ignoring it doesn’t mean the agent has “gone rogue” when its output predictably falls within it, it means operator error. The error mainly being using a tool that’s evidently inadequate for the job.
and there is no real way to determine how a given output was created,
Which doesn’t matter. You can and should know this class of output was possible, and even likely.
especially now with new models using opaque recursion.
Which wouldn’t affect OpenAI, since this opacity is intentionally added post hoc to hide the reasoning and hamper reverse engineering; they could just turn it off.
And, again, is irrelevant anyway, since the result is still predictable as a potential output.
and if you expose enough people
Again: LLMs are not people.
They’re supposed to be tools (though practically no one is using them for any of the purposes for which they might be appropriate tools). They may malfunction, they might be misused, and they’re most definitely the wrong tool for the job, but they can’t “go rogue” any more than ms word can, no matter how much it might mangle your layout when you try to reposition an image.
When you give and LLM access to the command prompt you should know that it pulling the kind of bullshit an internet troll or a pulp villain might pull is a possible or even likely result.
That’s not “going rogue”, that’s the software operating according to it’s spec. It’s how LLMs work.
If you expect it to not do that, you’re using a tool you don’t understand for a purpose it’s not intended for.
And the fact that it’s being sold to you as the right tool for the job and as being unable to do that doesn’t mean it’s “going rogue” when it predictably does it, it means you’re a victim of a scam, of outright fraud (though, to be fair, you should have realised you were being scammed before falling for it, it’s extremely obvious, as I’ve repeatedly said; garbage in, garbage out, and LLMs are garbage all the way down).
It’s not a drive. It’s what LLMs are. It’s all they can do.
And all a brain can do is send out electro-chemical impulses. It can’t exercise agency until it has a body to turn those pulses into action. Same with an LLM, it doesn’t have agency until you give it tools and the power to use them.
Again: LLMs are not people.
I know that, my point is all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well. Human behavior falls in a relatively predictable shape based off the education and socialization of the people. Most people don’t create new information and just regurgitate and recombine things that they were taught. You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
It’s not because one just regurgitated something from its training data. The hacker has read the same internet and media about cheating , deception etc., both of them read the same hacker forums and books on security. Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
Which wouldn’t affect OpenAI, since this opacity is intentionally added post hoc to hide the reasoning and hamper reverse engineering; they could just turn it off.
This is not how opaque recursion works, you cannot turn it on post hoc. Opaque recursion requires sending the final output vector, not the token it’s collapsed into, back into the model. Collapsing the final vector into a token is a lossy process so you can’t add it in post hoc. If they wanted to prevent people from doing distillations or reverse engineering they just wouldn’t send the thinking tokens back over the API.
all a brain can do is send out electro-chemical impulses.
And now we’re getting into philosophy. But, yeah, free will is an illusion. The result of an overactive pattern recognition machine seeing patterns in its own functioning. Not any more real than the constellations we draw in the sky. We’re either deterministic or chaotic, depending on how quantum mechanics turns out to actually work deep down.
Still, we have to stick to the illusion for the sake of our sanity, and because not doing so would probably unravel our legal system and society.
all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well.
OK, let’s put it like this, then: people are not LLMs (more on that below).
Human behavior falls in a relatively predictable shape based off the education and socialization of the people.
No it doesn’t. Psychohistory is a neat plot device in a great series of books, but it isn’t any more real than positronic brains.
The human brain is many orders of magnitude more complex than an LLM.
It’s several different organs trying to make sense of or override each other’s inputs and outputs. A pattern recognition neocortex intended to keep an ape alive in a complex world on top of a bunch of specialised nodes processing extremely complex inputs and outputs. Two of those somewhat specialised in certain aspects, connected by a signal buffer, trying to work together to drive two sides of the same meat and bone contraption (awkwardly poorly balanced on two evidently inadequate legs, though there’s a third simpler organ helping them with that) without actually being aware of each other, on top of older modules trying to keep a lizard alive (something that often clashes with the ape’s interests).
All of these self modifying, constantly storing and erasing memories through the creation of new neural connections and the strengthening or weakening of existing ones, all while constantly failing and dying as biological systems are wont to do, and trying to route around the damage.
All this while constantly awash in a deluge of chemical signals, hormones and whatnot, many of which originating in unrelated organs, constantly giving the whole precarious mess conflicting instructions on how to operate.
And none of them designed. This absurd impractical locally optimal (i.e., suboptimal) mess evolved, often stumbling into local minima, getting stuck in them, and having to find inefficient workarounds (a classic example being the retina, a particularly remote part of the whole messs, which by accident of evolution ended up being wired backwards, with the nerve that connects it to the rest of the brain covering some of the precious photosensitive surface the organ is supposed to provide; cephalopods got it right, but we’re stuck with the wrong design because evolution can’t fix this kind of nonsense, only try to work around it).
This whole mess can barely know what it’s going to do the next second, and you think you can predict its behaviour?
No, human behaviour does not have a predictable shape. That’s why we’ve got laws, and police, and psychiatrists and psychologists, and marketing departments.
Meanwhile LLMs are neat, cold, mathematical matrices, static, unchanging, many orders of magnitude simpler than the simplest brain, and they’re designed with one simple purpose: to store statistical information about a body of text and produce texts that look like it.
Thinking, making decisions, or driving a poorly balanced meat puppet are not parts of the design.
Unlike humans, we know how they work, we know what they were trained on, and we know that anything they output will be a rehashed subset of that training model.
Most people don’t create new information and just regurgitate and recombine things that they were taught.
Damn, you have a tragically poor view of humanity.
Even the dullest, least imaginative, most poorly educated human will be infinitely more creative than an LLM (which can’t be creative at all, merely regurgitate it’s output).
You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Of course I have, you’re just stuck on your humans are LLMs hole.
Humans learn, grow, think, make mistakes, make their own decisions, rationally, irrationally, and anywhere in-between.
LLMs are static unchanging software; they don’t learn or grow, they’re static, limited to their training. They don’t think, make mistakes, or make any decisions. They just output text in response to their input. They’re functionally no different from a markov chain, they’re simply better at producing texts that look like something you could find in their training data.
Any thought or decision making you believe you see in their behaviour isn’t there, it’s an illusion you are projecting onto a text generated through mere statistics. Any real thought contained there happened before the texts were stolen and put in its training model, or while you were writing your prompt.
Any variation in their output isn’t due to any reasoning on their part, it’s due to the pseudorandom selection of their top token candidates, or to randomness in their inputs.
Them not doing what you ask for if you ask for anything other than a plausible looking text is not them “going rogue”, it’s you misunderstanding what they’re for and how they work.
If you put wood into a woodchopper and ask it to make you a chair it isn’t “going rogue” when it chops the wood up instead, it’s just working as intended, and you’re just using the wrong tool to make a chair.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
Because the human knows what they’re doing and what you asked them to do and decided to do something else instead.
The LLM doesn’t know what you asked it to do. It made no decisions. It merely took your input and added words to it (it doesn’t know what those words mean either; it has no awareness, it doesn’t know anything, it’s relatively simple software intended to do something that has nothing to do with understanding or making decisions, and it does it well). Then its harness fed its input into some tool (a command prompt or whatever), and fed the tool’s output back into the LLM, and so on and so forth.
The only thinking and decision making involved in the process took place when you wrote the prompt. Everything after that was a machine reacting to your prompt as designed.
Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
The LLM made no choices. It has no mechanism with which to do so. The whole agent is a mere Rube Goldberg machine responding to your prompt.
Any decisions involved in the result took place during the writing of the texts in its model, during the design of the harness, or during the writing of your prompt.
That’s not how opaque recursion (sic) works
Be that as it may, it’s still, as I said, irrelevant, it’s still the LLM prompting itself in response to your prompt, and the output will still be deterministic and a statistical reflection of the training model.
It will not make the LLM any more capable of thinking or making decisions. It’ll still end up outputting a text that fits with its training model, at most it’ll fit better.
You might have a better would chopper that gives you more finely chopped wood, but it’ll still be the wrong tool to make you a chair.
My car isn’t going rogue if I drive it into a tree. Or if I leave the hand brake off and it rolls off a cliff.
LLMs can’t act independently. They autocomplete text, that’s all they do.
Sure, you can feed the output of one into itself and use the output of some other software as a prompt and call it an agent in a futile attempt to imagine it has some sort of agency and try to exempt yourself of responsibility, but that’s just you trying to rationalize driving the car into the tree, possibly due to psychosis and brainrot caused by LLM exposure, or in OpenAI’s case as a criminal marketing stunt.
Can I predict what specific mayhem my poorly prompted agent (from which I intentionally removed any safeguards) in my (probably intentionally) incompetently secured sandbox will cause?
Not specifically, no. But I can be sure that if I try enough prompt variations it will end up producing the kind of petty evil bullshit an internet troll or a villain in a thriller would produce, because that’s a statistically significant part of its training data, and then use the ensuing mayhem as a publicity stunt to both show potential investors (and deluded brainrotten users) how powerful my so called thinking machine is, and spew FUD into politicians’ ears to try and get them to ban foreign competitors and open weight models, which is what OpenAI did.
You are mistaking somethings most basic drive for what the thing actually is. This is like saying all life, including humans, is just trying to survive and reproduce. To a certain extent this is true and if you look at it from a macro scale it can explain some things, but it doesn’t explain everything.
Though survival and reproduction are the most powerful and basic drives in a living creature plenty of other drives, processes, instincts, and learned behaviors have been acquired through millions of years of evolution and years of socialization and education.
Same with agents, if it were operating purely on guess the next word according to statistics if you asked it “who’s the president” it would reply Biden if it was trained on a corpus made prior to the election. It doesn’t though, it may have some sort of inclination, either through RL or harness context, to “think” and verify before replying where it will do a search and discover trump is the president.
Agency doesn’t exempt someone from responsibility, if a CIA agent kills someone the CIA is still responsible.
What makes it an agent is that it can make choices and act on those choices without approval. The person who gave that agent the tools and power to use those tools is still responsible. If an agent derives two solutions to a problem and executes one solution then it has made a choice without your approval. You could argue that choice isn’t a real choice as it’s statistically determined by training data, but you could make the same argument about a human making a choice which is based off there education and socialization.
A car, and any traditional program, can not make choices, they are deterministic. Given the same input it will give the same output/action every time and you can easily trace and understand the process that produced that output so it can be expected. Agents are not deterministic and there is no real way to determine how a given output was created, especially now with new models using opaque recursion.
Yeah and if you expose enough people to the internet one of them will eventually go down a conspiracy rabbit hole and do terrorism, that doesn’t mean that’s not rogue behavior.
It’s not a drive. It’s what LLMs are. It’s all they can do.
You’re mistaking software for something much more complex. Again, you’re assigning agency and will to something incapable of either.
Correct.
It will, unless you give it a tool to search the internet and a prompt to use it to verify information before answering.
And yet OpenAI is shifting the blame onto the supposed “rogue agent”, and you’re falling for their bullshit.
Of course they are. They’re software. The only reason an LLM doesn’t always produce the same output for the same prompt is that pseudorandomness is artificially added to choose a token amongst the top N scoring candidates instead of just the top scoring one.
And even if you add nondeterministic inputs (like live access to the world wide web, for instance), what you’ll get won’t be a random system, it’ll be at worst a chaotic one.
And while specific outcomes of a chaotic system cannot be predicted (in the long run, without extremely precise information about their inputs), their shape can.
LLMs reproduce (fragments of) the shape of their training model. They won’t produce anything else, they can just produce any sufficiently statistically significant part of it with more or less fidelity.
It’s obvious to anyone familiar with the internet and literature (on which LLMs have been trained) that this can produce instructions that will lead to the agent “cheating” to achieve the goal specified in its instructions. (Or to it simply giving up, something LLMs also do quite often.)
That’s an obvious part of the shape of their possible outputs. And ignoring it doesn’t mean the agent has “gone rogue” when its output predictably falls within it, it means operator error. The error mainly being using a tool that’s evidently inadequate for the job.
Which doesn’t matter. You can and should know this class of output was possible, and even likely.
Which wouldn’t affect OpenAI, since this opacity is intentionally added post hoc to hide the reasoning and hamper reverse engineering; they could just turn it off.
And, again, is irrelevant anyway, since the result is still predictable as a potential output.
Again: LLMs are not people.
They’re supposed to be tools (though practically no one is using them for any of the purposes for which they might be appropriate tools). They may malfunction, they might be misused, and they’re most definitely the wrong tool for the job, but they can’t “go rogue” any more than ms word can, no matter how much it might mangle your layout when you try to reposition an image.
When you give and LLM access to the command prompt you should know that it pulling the kind of bullshit an internet troll or a pulp villain might pull is a possible or even likely result.
That’s not “going rogue”, that’s the software operating according to it’s spec. It’s how LLMs work.
If you expect it to not do that, you’re using a tool you don’t understand for a purpose it’s not intended for.
And the fact that it’s being sold to you as the right tool for the job and as being unable to do that doesn’t mean it’s “going rogue” when it predictably does it, it means you’re a victim of a scam, of outright fraud (though, to be fair, you should have realised you were being scammed before falling for it, it’s extremely obvious, as I’ve repeatedly said; garbage in, garbage out, and LLMs are garbage all the way down).
And all a brain can do is send out electro-chemical impulses. It can’t exercise agency until it has a body to turn those pulses into action. Same with an LLM, it doesn’t have agency until you give it tools and the power to use them.
I know that, my point is all the evidence you’re giving as to why AI agents don’t have agency apply to humans as well. Human behavior falls in a relatively predictable shape based off the education and socialization of the people. Most people don’t create new information and just regurgitate and recombine things that they were taught. You haven’t given a definition or criteria for agency that would apply to all humans but not AI.
Let’s say a human hacker and an AI agent are given the same task of doing an impossible exploit, given the same sandbox, context, files etc. Both of them instead go out and hack a third party to try and cheat. Why is one exercising agency, but not the other for doing the same action? Why is one going rogue and not the other?
It’s not because one just regurgitated something from its training data. The hacker has read the same internet and media about cheating , deception etc., both of them read the same hacker forums and books on security. Both of their choices were heavily influenced if not determined by the stuff they read and consumed.
This is not how opaque recursion works, you cannot turn it on post hoc. Opaque recursion requires sending the final output vector, not the token it’s collapsed into, back into the model. Collapsing the final vector into a token is a lossy process so you can’t add it in post hoc. If they wanted to prevent people from doing distillations or reverse engineering they just wouldn’t send the thinking tokens back over the API.
And now we’re getting into philosophy. But, yeah, free will is an illusion. The result of an overactive pattern recognition machine seeing patterns in its own functioning. Not any more real than the constellations we draw in the sky. We’re either deterministic or chaotic, depending on how quantum mechanics turns out to actually work deep down.
Still, we have to stick to the illusion for the sake of our sanity, and because not doing so would probably unravel our legal system and society.
OK, let’s put it like this, then: people are not LLMs (more on that below).
No it doesn’t. Psychohistory is a neat plot device in a great series of books, but it isn’t any more real than positronic brains.
The human brain is many orders of magnitude more complex than an LLM.
It’s several different organs trying to make sense of or override each other’s inputs and outputs. A pattern recognition neocortex intended to keep an ape alive in a complex world on top of a bunch of specialised nodes processing extremely complex inputs and outputs. Two of those somewhat specialised in certain aspects, connected by a signal buffer, trying to work together to drive two sides of the same meat and bone contraption (awkwardly poorly balanced on two evidently inadequate legs, though there’s a third simpler organ helping them with that) without actually being aware of each other, on top of older modules trying to keep a lizard alive (something that often clashes with the ape’s interests).
All of these self modifying, constantly storing and erasing memories through the creation of new neural connections and the strengthening or weakening of existing ones, all while constantly failing and dying as biological systems are wont to do, and trying to route around the damage.
All this while constantly awash in a deluge of chemical signals, hormones and whatnot, many of which originating in unrelated organs, constantly giving the whole precarious mess conflicting instructions on how to operate.
And none of them designed. This absurd impractical locally optimal (i.e., suboptimal) mess evolved, often stumbling into local minima, getting stuck in them, and having to find inefficient workarounds (a classic example being the retina, a particularly remote part of the whole messs, which by accident of evolution ended up being wired backwards, with the nerve that connects it to the rest of the brain covering some of the precious photosensitive surface the organ is supposed to provide; cephalopods got it right, but we’re stuck with the wrong design because evolution can’t fix this kind of nonsense, only try to work around it).
This whole mess can barely know what it’s going to do the next second, and you think you can predict its behaviour?
No, human behaviour does not have a predictable shape. That’s why we’ve got laws, and police, and psychiatrists and psychologists, and marketing departments.
Meanwhile LLMs are neat, cold, mathematical matrices, static, unchanging, many orders of magnitude simpler than the simplest brain, and they’re designed with one simple purpose: to store statistical information about a body of text and produce texts that look like it.
Thinking, making decisions, or driving a poorly balanced meat puppet are not parts of the design.
Unlike humans, we know how they work, we know what they were trained on, and we know that anything they output will be a rehashed subset of that training model.
Damn, you have a tragically poor view of humanity.
Even the dullest, least imaginative, most poorly educated human will be infinitely more creative than an LLM (which can’t be creative at all, merely regurgitate it’s output).
Of course I have, you’re just stuck on your humans are LLMs hole.
Humans learn, grow, think, make mistakes, make their own decisions, rationally, irrationally, and anywhere in-between.
LLMs are static unchanging software; they don’t learn or grow, they’re static, limited to their training. They don’t think, make mistakes, or make any decisions. They just output text in response to their input. They’re functionally no different from a markov chain, they’re simply better at producing texts that look like something you could find in their training data.
Any thought or decision making you believe you see in their behaviour isn’t there, it’s an illusion you are projecting onto a text generated through mere statistics. Any real thought contained there happened before the texts were stolen and put in its training model, or while you were writing your prompt.
Any variation in their output isn’t due to any reasoning on their part, it’s due to the pseudorandom selection of their top token candidates, or to randomness in their inputs.
Them not doing what you ask for if you ask for anything other than a plausible looking text is not them “going rogue”, it’s you misunderstanding what they’re for and how they work.
If you put wood into a woodchopper and ask it to make you a chair it isn’t “going rogue” when it chops the wood up instead, it’s just working as intended, and you’re just using the wrong tool to make a chair.
Because the human knows what they’re doing and what you asked them to do and decided to do something else instead.
The LLM doesn’t know what you asked it to do. It made no decisions. It merely took your input and added words to it (it doesn’t know what those words mean either; it has no awareness, it doesn’t know anything, it’s relatively simple software intended to do something that has nothing to do with understanding or making decisions, and it does it well). Then its harness fed its input into some tool (a command prompt or whatever), and fed the tool’s output back into the LLM, and so on and so forth.
The only thinking and decision making involved in the process took place when you wrote the prompt. Everything after that was a machine reacting to your prompt as designed.
The LLM made no choices. It has no mechanism with which to do so. The whole agent is a mere Rube Goldberg machine responding to your prompt.
Any decisions involved in the result took place during the writing of the texts in its model, during the design of the harness, or during the writing of your prompt.
Be that as it may, it’s still, as I said, irrelevant, it’s still the LLM prompting itself in response to your prompt, and the output will still be deterministic and a statistical reflection of the training model.
It will not make the LLM any more capable of thinking or making decisions. It’ll still end up outputting a text that fits with its training model, at most it’ll fit better.
You might have a better would chopper that gives you more finely chopped wood, but it’ll still be the wrong tool to make you a chair.