Whether the input prompt is compressed or not, or whatever form it takes, it doesn’t make what I said any less true: the model is stateless. It would be impractical to serve an llm any other way: loading and unloading weights from/to the gpu memory is very expensive in terms of time and power. Cloud llms only makes sense if they can serve thousands of users without changing.