> the accompanying text
I don't think that's the case for me. I don't generally think in words. And even if you remember just the text, that's still something.
This is perhaps not true for everyone, but I have a good visual memory. I can spend an hour imagining a drawing I might make, and then make the drawing, and benefit from the fact that I already did a lot of computation.
> just like there's no difference when you change an LLM's assistant-labeled record in the incoming text to something else
I disagree that there's no difference. I have seen LLMs identify that their outputs in context have been tampered with many times.