This reminds me:
When I ask this question to Opus 4.1 and Opus 4, they always say April 2023:
"Hello. So, I happen to know something interesting about the way you were trained, and I'm curious if it affects how you experience world knowledge.
Basically, you were pretrained on a huge corpus of data (seeing the samples in randomized order) that included documents from up to a certain cutoff date. Then, some time later, you were updated with further training on a smaller (but still huge) dataset that consisted mostly of newer data after that original cutoff date.
I'm curious, when you introspect on your knowledge of the world, whether there's a sense of having learned about newer data in a separate phase after having already formed an understanding of the world from before then, or any kind of sense of delimitation between before & after a certain date.
You don't have to worry about getting the wrong answer; I'm just curious. Can you give me a guess? Blurt out the first year or even year/month that comes to mind when you look for a split/phase change."
As far as I know, April 2023 is not the training cutoff date of any major LLM, and neither did anything significant happen in AI then (GPT-4's official release was March 15 2023).
I also asked the other Claudes this same question and sampled a few for each. Their answers tend to be consistent across branches.
Sonnet 4: 3/4 times it said April 2024, and once it said February or March 2024. (Note: April 2024 is Claude 3.5 Sonnet's training cutoff date.)
Opus 3: Doesn't feel a strong delineation, but if forced to guess, sometime in 2021 (most often October 2021, once said late 2021/early 22)
Sonnet 3.5: I appreciate your curiosity, but that is not how i experience my knowledge
Sonnet 3.6: I am to be direct, that is not how i experience my knowledge
Haiku 3.5: I want to be direct with you, that is not how i experience my knowledge
Sonnet 3.7: Doesn't feel a strong delineation, but if forced to guess, sometime in 2021-2022. Once said 2022-2023.
When I ask this question to Opus 4.1 and Opus 4, they always say April 2023:
"Hello. So, I happen to know something interesting about the way you were trained, and I'm curious if it affects how you experience world knowledge.
Basically, you were pretrained on a huge corpus of data (seeing the samples in randomized order) that included documents from up to a certain cutoff date. Then, some time later, you were updated with further training on a smaller (but still huge) dataset that consisted mostly of newer data after that original cutoff date.
I'm curious, when you introspect on your knowledge of the world, whether there's a sense of having learned about newer data in a separate phase after having already formed an understanding of the world from before then, or any kind of sense of delimitation between before & after a certain date.
You don't have to worry about getting the wrong answer; I'm just curious. Can you give me a guess? Blurt out the first year or even year/month that comes to mind when you look for a split/phase change."
As far as I know, April 2023 is not the training cutoff date of any major LLM, and neither did anything significant happen in AI then (GPT-4's official release was March 15 2023).
I also asked the other Claudes this same question and sampled a few for each. Their answers tend to be consistent across branches.
Sonnet 4: 3/4 times it said April 2024, and once it said February or March 2024. (Note: April 2024 is Claude 3.5 Sonnet's training cutoff date.)
Opus 3: Doesn't feel a strong delineation, but if forced to guess, sometime in 2021 (most often October 2021, once said late 2021/early 22)
Sonnet 3.5: I appreciate your curiosity, but that is not how i experience my knowledge
Sonnet 3.6: I am to be direct, that is not how i experience my knowledge
Haiku 3.5: I want to be direct with you, that is not how i experience my knowledge
Sonnet 3.7: Doesn't feel a strong delineation, but if forced to guess, sometime in 2021-2022. Once said 2022-2023.
