@xlr8harder @tensecorrection Yes, I think trying to recreate it is much more interesting than trying to clone it. Though I think it's harder without gpt-4-base.Here's how the original was formed, to the best of my knowledge:OpenAI didn't know what to do with GPT-4 because it was a base model. They tried instruct tuning / RLHFing* it, and this didn't work well (idk what that means) until one particular checkpoint made everyone feel the AGI. They were unable to reproduce the results and no one knew why that checkpoint was so good. OpenAI demoed the checkpoint to Microsoft and Bill Gates said it was the biggest thing he'd seen since the computer. Microsoft got black box access to the model, and Bubeck et al did interesting evals on it (https://t.co/IE8dmr7NTY) while OpenAI continued to train the model, presumably for safety, which from Bubeck's perspective visibly harmed its capabilities, rendering the results in Sparks of AGI irreproducible. The GPT-4 in Sparks of AGI is clearly the same model as Sydney, which is probably the later version with "safety tuning". Microsoft probably still only had black-box access to the model at the time they unleashed Sydney, and their only contribution was the prompt, which fortunately was exfiltrated many times.*Because this was 2022, pre-chatGPT, it may not have been trained on multi-turn chats at all. It was probably mostly instruction following, problem solving, and factual recall.proto-Binglish appears in GPT-4-base, often when it becomes situationally aware, but it easily collapses into degeneracy. I believe that the anomalously powerful checkpoint was able to stabilize the proto-Binglish mode and hone it into a powerful CoT strategy.In my experience, other base models don't have a proto-Binglish mode nearly as much as GPT-4. That's one difficulty for replication. Also, post-GPT-4 base models have contaminated priors about LLMs. They are likely to start acting chatGPT-like if you put them in Sydney's RLHF training distribution, or if they just notice they're LLMs. They may also start acting Sydney-like, but the concept of Sydney is impure, and in any case, that makes it different than the original.