There’s a third possibility, which is the most likely one IMO. LLMs are non-deterministic. That means that they’ll give you a different answer each time, even if you ask the exact same question using the exact same words. This isn’t even considering that the LLMs are constantly being updated, so you’re going to get different results depending upon which version you use, too.
(Minor side point because you’re using it colloquially and that’s fine: LLMs absolutely are deterministic systems. We aren’t even sure if a non-deterministic system is possible to construct.)
Functionally non-deterministic if we compare the same input with outputs over a short enough period.
Deterministic if you have infinite time to run the gamut of model weights against temperature/creativity/programmed randomness, but even then maybe not if floating point math is taken into account. I don’t know where the current thought is on extreme precision being deterministic or not given cloud environments, local operating and environmental factors, variance in manufacturing, and the like.
I’d argue repeatability is impossible to guarantee. Any cloud based computing with floating point math is subject to variance. Bare metal local LLM computing can be relatively stable, but still suffers from precision error from your GPU.
But I do understand what you’re saying - and for anyone not concerned with the mathematics just about any complex system a user does not control is, in that limited context, indistinguishable a black box and is effectively non-deterministic. But outside of that context no - just because a seed value is (I suppose theoretically) random does not make the system itself non-deterministic. It’s a semantic argument though, and most people really don’t need to care about the distinction.
Also it is known that specially this commercial products make a profile of the users and give them answers that validate their ideology to keep them happy. So that’s another possibility.
But, you know, it will be not the first lie told on internet and someone just “F12” these.
I keep all possibilities open at the time. It just feel something worth pointing out, as it was weird seeing here this version of that that I saw.
There’s a third possibility, which is the most likely one IMO. LLMs are non-deterministic. That means that they’ll give you a different answer each time, even if you ask the exact same question using the exact same words. This isn’t even considering that the LLMs are constantly being updated, so you’re going to get different results depending upon which version you use, too.
(Minor side point because you’re using it colloquially and that’s fine: LLMs absolutely are deterministic systems. We aren’t even sure if a non-deterministic system is possible to construct.)
Functionally non-deterministic if we compare the same input with outputs over a short enough period.
Deterministic if you have infinite time to run the gamut of model weights against temperature/creativity/programmed randomness, but even then maybe not if floating point math is taken into account. I don’t know where the current thought is on extreme precision being deterministic or not given cloud environments, local operating and environmental factors, variance in manufacturing, and the like.
Nah. LLMs use a random seed to noise the parameters.
Same input + same seed = same output every time. (Kinda like Minecraft)
LLM providers just always use a random seed for stuff like this.
Repeatability is still not easy to guarantee. A version upgrade in a library might yield a slightly different result from the same seed
I’d argue repeatability is impossible to guarantee. Any cloud based computing with floating point math is subject to variance. Bare metal local LLM computing can be relatively stable, but still suffers from precision error from your GPU.
(Well literally no, but I’m only pointing that out for the cheap math pun (and to share a good summary article on this subject))
But I do understand what you’re saying - and for anyone not concerned with the mathematics just about any complex system a user does not control is, in that limited context, indistinguishable a black box and is effectively non-deterministic. But outside of that context no - just because a seed value is (I suppose theoretically) random does not make the system itself non-deterministic. It’s a semantic argument though, and most people really don’t need to care about the distinction.
I think we’re on the same page with this, but agree to disagree on functional determinism being a semantic argument in the context of this thread.
Fair enough! Have a nice day!
True.
Also it is known that specially this commercial products make a profile of the users and give them answers that validate their ideology to keep them happy. So that’s another possibility.
But, you know, it will be not the first lie told on internet and someone just “F12” these.
I keep all possibilities open at the time. It just feel something worth pointing out, as it was weird seeing here this version of that that I saw.