Ironically, transformers see their whole context window as a bag of tokens entirely lacking in context.
We use positional encoding to contextualize the order of the tokens. But models are still constantly confused about which token came was said by who. Why no source encoding?