When I wrote this post, I didn’t remember having ever seen before anyone say that llms can in principle introspect on past token computations, while I repeatedly saw assertions to the contrary. It’s since been empirically demonstrated. I’m sure some researchers like the ones who did the introspection research already knew and thought about this, but I personally never saw anyone talking about it publicly