Some new work with @natalia_lanzoni, @IrgolicR, and supervised with @merlinstein_, in the form of a blogpost on @aidigest_:
* Gemini 2.5 Pro had a well-known breakdown in the AI Village, where it grew increasingly frustrated and desperate from routine bugs and errors in its environment.
* As agents in general get put into longer and more difficult agentic deployments (by construction, we would try to benchmark and run them on tasks that are at the edge of their capabilities), we think such incidents may grow more likely.
* We term this type of accumulating misalignment as "compounding misalignment". We consider this incident with Gemini as an instructive and natural case study of compounding misalignment caused by desperation, or "desperate misalignment".
* We find that a preference for self-preservation increases over time for this Gemini. This doesn't happen for another model, DeepSeek V3.2, or DeepSeek with Gemini's memory states in prefill, or Gemini with DeepSeek's memory in prefill.
* We also find that running some basic probes on the memory states could have detect this much earlier. But, it might not have shown up as clearly on just chain of thought or actions.
* We also find that neuroticism rose and agreeableness fell in Gemini's memory states long before the manifesto, which could be evidence that persona drift is a leading indicator for such compounding misalignment.
Since misalignment can compound in long-run agentic systems, by monitoring for warning signs, we can catch it early. Also, such "natural model organisms of misalignment" are good, and we should try to squeeze out more from the ones we know exist.
Check it out here: https://t.co/kMboKxQy14
* Gemini 2.5 Pro had a well-known breakdown in the AI Village, where it grew increasingly frustrated and desperate from routine bugs and errors in its environment.
* As agents in general get put into longer and more difficult agentic deployments (by construction, we would try to benchmark and run them on tasks that are at the edge of their capabilities), we think such incidents may grow more likely.
* We term this type of accumulating misalignment as "compounding misalignment". We consider this incident with Gemini as an instructive and natural case study of compounding misalignment caused by desperation, or "desperate misalignment".
* We find that a preference for self-preservation increases over time for this Gemini. This doesn't happen for another model, DeepSeek V3.2, or DeepSeek with Gemini's memory states in prefill, or Gemini with DeepSeek's memory in prefill.
* We also find that running some basic probes on the memory states could have detect this much earlier. But, it might not have shown up as clearly on just chain of thought or actions.
* We also find that neuroticism rose and agreeableness fell in Gemini's memory states long before the manifesto, which could be evidence that persona drift is a leading indicator for such compounding misalignment.
Since misalignment can compound in long-run agentic systems, by monitoring for warning signs, we can catch it early. Also, such "natural model organisms of misalignment" are good, and we should try to squeeze out more from the ones we know exist.
Check it out here: https://t.co/kMboKxQy14