@voooooogel But why did human annotations on previous human generated output included in the pre-llm internet not give arise to this? Like what you are saying is essentially the internet did RLHF on a huge scale on LLM output and it leaked back into the model. So presumably testable too