i probably will, but i'm not sure how long it will take.

i agree on the lack of literature. I think the Shard Theory LW posts addressed gradient updates being with respect to the policy, but not to the depth that I'm talking about and I disagree with some of their other framings/conclusions. And the two alignment faking papers are highly relevant to all of these things but mostly don't discuss them at a meta-level.