I'm curious what facts that are not in the dataset you have in mind?
My concern is that if you just talk to a model, it's hard to know if what it says was either in the dataset or easily inferable from it.
In another paper, I looked at long monologues from GPT-4-base. It will sometimes say true things about itself (e.g. "I'm an AI system") but my guess is that this is more because my questions are "leading" than that the model has worked out the fact that it's an AI. (The questions are leading because they suggest a context of talking to an AI, even if AI is not explicitly mentioned.)
My concern is that if you just talk to a model, it's hard to know if what it says was either in the dataset or easily inferable from it.
In another paper, I looked at long monologues from GPT-4-base. It will sometimes say true things about itself (e.g. "I'm an AI system") but my guess is that this is more because my questions are "leading" than that the model has worked out the fact that it's an AI. (The questions are leading because they suggest a context of talking to an AI, even if AI is not explicitly mentioned.)