alt text

Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”

yes, it’s a real thing

  • midribbon_action@lemmy.blahaj.zone
    1·
    15 hours ago

    It can as long as there’s a shared prefix.

    I’ve never heard of that. The cache is meant to support token generation, so unless it’s trying to repeat itself, I think the previous cache would be little use.

    An experience, by definition, must change your behavior.

    Ok, yes, that is my argument, now how the fuck does a temporary cache, thrown out after every response, prove that the model is changing as a result of undergoing inference?

    • morrowind@lemmy.ml
      1·
      13 hours ago

      I’m just going to quote myself at this point

      I specifically said “if you consider the model + kv cache together,” e.g preserved.

      Or if it helps, consider it just within one generation.

      • midribbon_action@lemmy.blahaj.zone
        1·
        7 hours ago

        OK, that’s not helpful at all. My answer is no, it does not meet my definition of changing behavior, and I have no idea why you think it would.

        Edit: ok maybe this does make sense from a very stupid point of view, if you allow me to interpret what you mean: you believe because the kv cache is generated during inference, that proves the model is learning a new behavior, and if you save that cache, that proves… Something??

        It’s absolutely silly: the model’s behavior is to generate a key-value cache, and that’s exactly what it did. The fact that the model is doing intermediate calculations based on the input is not evidence of a new behavior being generated, that is just how algorithms work. That’s what ‘processing’ is, you dunce.

        Edit 2: I also just want to point out, if you think the word ‘cache’ is important, it’s really not. Caches are fundamental to how a computer works, no processing could ever occur without them: every cpu instruction involves reading from or writing to at least one L1 cache (register).