@evanjayconway i haven't looked into it but i assume these kinds of typos come from drift in final layers / lm head unembeddings during e.g. rlvr, a bit like seahorse situation in this post
same thread: 2064496731774202194 2064496734546620487 2064497762159485159 2064506868068995129 2064527506888347737 2064530443702341643 2064538184722399670 2064558338537603375 2064583930406420661 2064584132244685238 2064640239998570793
Reproduced against link rot, credited and linked to its original. Yours and you’d rather it weren’t here? Open an issue.