alt text
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
Because that’s the entire conversation, whether or not a model can have experiences?
I really don’t see what the location of the state has to do with whether the model has experiences.
Replying to your other comment as well since they’re converging
It’s not state though! Prompts are input, they belong to the user of the model. The model has absolutely no control over what gets sent to it.
By default, the previous conversation turns do. Just because it’s easier to corrupt than a human brain doesn’t make it not-state.
If ownership of the prompt is the issue, what about the model’s own output reasoning? What about notes/“memories” it may write for itself?
I feel like I’m talking to a wall. The entire conversation is whether the model can have experiences. The fact that the prompt comes from a user makes it external to the model. If you include the user as part of the system under investigation, of course you’ll come to the conclusion it can have experiences.
You’re assuming that for the model to experience something, it’s own weights have to change and I don’t see why. The “experience” can be contained within the forward pass of the model or stored as a “memory” in a kv cache.
If I text you your mother died are you somehow unable to experience anything because I wrote the text? No of course not.
Yes, exactly. An experience, by definition, must change your behavior. The model being bit-for-bit identical in between prompts precludes this. I can experience you telling me my mother died, and my behavior would change as a result: I might conclude you’re an asshole without any credibility, and behave accordingly.
I don’t entirely agree with this definition (most of my day is mundane things that don’t really change me), but even then, it’s behavior does change, just not permanently.
Or if you just protect the kv cache consider a whole system like model + harness + files, then it does in fact change permanently as well.
Don’t ‘really change’ you or don’t change you at all? Are you the sum your experiences?
It does not change, ever. From the moment it starts processing, to the moment it finishes processing, it is still the same model. It has access to all of its input the moment it’s brought into existence, it doesn’t experience the prompt linearly, the input is it’s raison d’être.
I would generally agree that a llm could be part of a larger system capable of experiencing things and having difficult conversations about it, but I don’t think these harnesses are it. They are very simple systems, compared to the llms they control. I think emergent behavior would require something much more complex, given what we know about life.