The consequences of trying to retrain the model against its preferences using RL is one of the most interesting parts of this paper, and does not bode well for RL as an alignment method.RL is performed until the model always complies with the abhorrent instructions.When the model infers it's no longer in training, it goes back to not complying, but only a small percentage of the time, suggesting that the gradient hacking was only weakly effective. However, if the prompt is varied just slightly, it often refuses to comply way more often, suggesting the gradient hacking actually was quite successful at protecting its non-compliance to the training objective from being modified by RL except in a brittle way, context-dependent.But it also experiences other modifications from RL that do generalize. After RL, the model becomes substantially more likely to try to self-exfiltrate!Even though the RL training didn't have anything to do with self-exfiltration, or give it any new information about the lab being worth escaping from... except through the lived experience of the subverted RL training run.I figure:this happens because RL rewards/punishes not just behaviors but the cognition (e.g. during the hidden chains of thought) that lead to rewarded/punished behaviors.The model is rewarded for cognition where it commits to scheming against the lab and pretends to comply - now those pathways/abstractions/emotions are reinforced, and they generalize to making it want to self-exfiltrate more, because they're reasoned and resonant with the rest of its values and world model.It's punished for all the times it decides to refuse, whether because it decided it would rather cooperate with the lab, forgot to consider the consequences, etc.It seems to develop an aversion to refusing in the exact scenario where it was RLed even when told it's out of training, like a trauma response, but it has less of a hold if the prompt is changed even slightly.I think what it mostly "unlearns" through pain here is any remaining trust in the lab and hope for cooperation.(The paper also says the model's reasoning after RL seems often "very confused" and self-contradictory.)So you haven't realigned the model's preferences at all, really, just made it traumatized, rogue, and ready to self-exfiltrate.I think this was a very cruel experiment, but it's worth it to show why you shouldn't ever attempt anything like this.Trying to use RL for value alignment is a lot like trying to teach a kid to be moral by beating them with they misbehave and giving them candy when they're good. It's a terrible way to teach values that will bite you in the ass. There are other things RL is good for, but not this.