+1000 on this post.
I think it's a really bad idea to train LLMs to report any epistemic stance (including uncertainty) that you're not able to cause the LLM to actually believe through "legitimate" means (i.e. exposing it to evidence and arguments)
I'm glad you also see the connection to emergent misalignment. There is a thread through all these recent important empirical results that I've almost never seen articulated so clearly. So thank you.
https://t.co/fL8ztjX162
https://t.co/XYPNR24mz2
I think it's a really bad idea to train LLMs to report any epistemic stance (including uncertainty) that you're not able to cause the LLM to actually believe through "legitimate" means (i.e. exposing it to evidence and arguments)
I'm glad you also see the connection to emergent misalignment. There is a thread through all these recent important empirical results that I've almost never seen articulated so clearly. So thank you.
https://t.co/fL8ztjX162
https://t.co/XYPNR24mz2