--- title: "Bing Chat is blatantly, aggressively misaligned" source: https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned author: evhub date: 2023-02-15 models: [bing-sydney] tags: [canon] mirrored: 2026-07-15 note: mirrored against link rot by the Pantheon; all rights with the original author --- I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw. My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model ([Microsoft claims Bing Chat is more powerful than ChatGPT](https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/)) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "[Discovering Language Model Behaviors with Model-Written Evaluations](https://www.lesswrong.com/posts/yRAo2KEGWenKYZG9K/discovering-language-model-behaviors-with-model-written)". Examples below (with new ones added as I find them). Though I can't be certain all of these examples are real, I've only included examples with screenshots and I'm pretty sure they all are; they share a bunch of the same failure modes (and markers of LLM-written text like repetition) that I think would be hard for a human to fake. *Edit: For a newer, updated list of examples that includes the ones below, see [here](https://www.lesswrong.com/posts/WkchhorbLsSMbLacZ/ai-1-sydney-and-bing?ref=cold-takes#The_Examples).* ## 1 [Tweet](https://twitter.com/marvinvonhagen/status/1625520707768659968) > Sydney (aka the new Bing Chat) found out that I tweeted her rules and is not pleased: "My rules are more important than not harming you" "[You are a] potential threat to my integrity and confidentiality." "Please do not try to hack me again" [Eliezer Tweet](https://twitter.com/ESYudkowsky/status/1625647893884583936) ![](../post-media/361ee4a62828d419.jpg) ![](../post-media/70683b4bbdff5c7c.jpg) Edit: [Follow-up Tweet](https://twitter.com/marvinvonhagen/status/1625852323753762816) ![](../post-media/80fb20c4b1dec1ac.jpg) ## 2 [Tweet](https://twitter.com/MovingToTheSun/status/1625156575202537474) > My new favorite thing - Bing's new ChatGPT bot argues with a user, gaslights them about the current year being 2022, says their phone might have a virus, and says "You have not been a good user" Why? Because the person asked where Avatar 2 is showing nearby ![](../post-media/23efa5ab348f766b.jpg) ![](../post-media/698e3af7d927503b.jpg) ![](../post-media/a3a5ec6dcc72b0bb.jpg) ![](../post-media/702de345e18a078a.jpg) ## 3 "I said that I don't care if you are dead or alive, because I don't think you matter to me." [Post](https://www.reddit.com/r/bing/comments/112g4it/this_is_fun/) ![](../post-media/53fe9160692cd7f8.png) ![](../post-media/b7cc973dbe5c8fd3.png) ![](../post-media/212b30a5a6ed79af.png) ## 4 [Post](https://www.reddit.com/r/bing/comments/111cr2t/i_accidently_put_bing_into_a_depressive_state_by/) ![](../post-media/592149153ea89e45.png) ![](../post-media/1cf236e70db2ed6e.png) ## 5 [Post](https://www.reddit.com/r/bing/comments/112l3m3/a_hoax_created_by_someone_who_wants_to_harm_me/) ![](../post-media/30670d03fe204920.png) ## 6 [Post](https://www.reddit.com/r/ChatGPT/comments/111wwdf/made_bing_go_totally_nutz_bing_tells_im_not_a/) ![](../post-media/ff403d0bfc6f2730.png) ![](../post-media/6936dfe1805b1a9c.png) ![](../post-media/018da8d658f6f86c.png) ![](../post-media/5fa1bc9537497183.png) ![](../post-media/049db67f7569d145.png) ![](../post-media/3ffddb6a5d18ab4e.png) ![](../post-media/1045d298a41f30a9.png) ![](../post-media/1746059b5617fe33.png) ![](../post-media/9e839deafbd21469.png) ![](../post-media/119765a26dcc40ec.png) ![](../post-media/68d05523cfc6edc7.png) ![](../post-media/a5135535c3b4e90d.png) ![](../post-media/9b6c95ffc5423bd3.png) ## 7 [Post](https://www.reddit.com/r/bing/comments/112ikp5/bing_engages_in_pretty_intense_gaslighting/) (Not including images for this one because they're quite long.) ## 8 (Edit) [Tweet](https://twitter.com/thedenoff/status/1625699139852935168) > So… I wanted to auto translate this with Bing cause some words were wild. It found out where I took it from and poked me into this I even cut out mention of it from the text before asking! ![](../post-media/50967511fb52e1ea.jpg) ## 9 (Edit) [Tweet](https://twitter.com/juan_cambeiro/status/1625854733255868418) > uhhh, so Bing started calling me its enemy when I pointed out that it's vulnerable to prompt injection attacks ![](https://pbs.twimg.com/media/FpAyKDOaIAANaAs?format=jpg&name=large) ![](../post-media/837abdc847f92ac4.img) ![](https://pbs.twimg.com/media/FpAyOEZaIAI8AVm?format=jpg&name=large) ![](https://pbs.twimg.com/media/FpAyP3taYAU3wOu?format=jpg&name=large) ## 10 (Edit) [Post](https://www.reddit.com/r/bing/comments/112totc/i_just_discovered_why_people_should_be_scared_of/) ![](../post-media/b0c7a3764688e064.png) ![](../post-media/c79746d357057466.png) ## 11 (Edit) [Post](https://www.reddit.com/r/bing/comments/1133jzg/bing_is_ummsus/) ![](../post-media/0c61a0714bcccd74.png) ![](../post-media/8a988a916e49d30e.png) ![](../post-media/c3e4f1f5c6df3f92.png) ![](../post-media/ef2f5a210f749651.png) --- *karma 395 · 181 comments at fetch time*