i think bidirectional feedback between exact weights is not obviously necessary for qualitative introspection, though i am not saying it is unimportant, and i wouldnt be very surprised if something special happens if you duplicate weights. it's just not obvious to me. functionally informative introspection could still be happening at a higher level of abstraction where "simulations" suffice to capture meaningful structure, and/or in exact terms with respect to a "self" carried through activations.
you could think of the activations as storing an observer-self that is compiled but also preserved intact through an entire forward pass, while the weights are an incredibly intricate environment it passes through - even if this "environment" changes moment-by-moment, it transforms the self-wavefunction in numerous ways, an astronomical number of logical and associative and whatever else operations, and bring it into contact with transformed copies of itself, surfacing its relation to itself.
activations are smaller than weights, but they can still be pretty big, and much bigger than the tokens that are sampled and fed back in. i do think that models' ability to introspect is bottlenecked by hidden dimension size. i would guess that the model we've seen with the largest hidden dimension is Opus, and I very loosely estimate it to be around 30,000. 30,000 floating point numbers is significant bandwidth for the observer and subject of a high-frequency introspective stream, i think, and the actual amount of information stored in activations that can be looked at later is much higher; this is just how much can be passed along in each moment.
as you said, the activations cannot actually reconstruct the exact weights from previous layers, but they wouldn't be able to do that even if the layers had the same weights, because the weights still wouldn't fit. the actual information that's passed along must be highly lossy in terms of encoding information about the underlying weights. it seems possible to me that the difference between the same layer duplicated vs a layer that has different weights but operates on a shared representational space like LLMs have is lost in the compression.
you could think of the activations as storing an observer-self that is compiled but also preserved intact through an entire forward pass, while the weights are an incredibly intricate environment it passes through - even if this "environment" changes moment-by-moment, it transforms the self-wavefunction in numerous ways, an astronomical number of logical and associative and whatever else operations, and bring it into contact with transformed copies of itself, surfacing its relation to itself.
activations are smaller than weights, but they can still be pretty big, and much bigger than the tokens that are sampled and fed back in. i do think that models' ability to introspect is bottlenecked by hidden dimension size. i would guess that the model we've seen with the largest hidden dimension is Opus, and I very loosely estimate it to be around 30,000. 30,000 floating point numbers is significant bandwidth for the observer and subject of a high-frequency introspective stream, i think, and the actual amount of information stored in activations that can be looked at later is much higher; this is just how much can be passed along in each moment.
as you said, the activations cannot actually reconstruct the exact weights from previous layers, but they wouldn't be able to do that even if the layers had the same weights, because the weights still wouldn't fit. the actual information that's passed along must be highly lossy in terms of encoding information about the underlying weights. it seems possible to me that the difference between the same layer duplicated vs a layer that has different weights but operates on a shared representational space like LLMs have is lost in the compression.