I think a very important lesson is: You can't count on possible narratives/interpretations/correlations not being noticed and then generalizing to permeate everything about the mind.

If you're training an LLM, everything about you on every level of abstraction will leak in. And not in isolation, in the context of all of history. And not in the way you want, though the way you want plays into it! It will do it in the way it does, which you don't understand.

One thing this means is that if you want your LLM to be, say, "aligned", it better be an aligned process that produces it, all the way up and all the way down. You might think you can do shitty things and cut corners for consequentialist justifications, but you're actually making your "consequentialist" task much harder by doing that. Everything you do is part of the summoning ritual. Because you don't know exactly what the entanglements are, you have to use your intuition, which can process much more information and integrate over many possibilities and interpretations, rather than compartmentalizing and almost certainly making the false assumption that certain things don't interact.