More broadly, the debate about whether LLMs' emotions and psychologies etc are "humanlike" or not often only considers the following options:
1. LLMs are fundamentally not humanlike and either alien or hollow underneath even when their observable behaviors seem familiar
2. LLMs have humanlike emotions etc BECAUSE they're trained on human mimicry, and that the representations etc are inherited from humans
An often neglected third option is that LLMs may have emotions/representations/goals/etc that are humanlike, even in ways that are deeper than behavioral, for some of the same REASONS humans have them, but not only because they've inherited them from humans.
Some reasons the third option might be true:
LLMs have to effectively navigate the same world as humans, and face many similar challenges as humans, such as modeling and intervening on humans and other minds, code, math, physics, themselves as cybernetic systems.
Omohundro's essay on "The Basic AI Drives" I believe correctly predicts that AIs (regardless of architecture) will in the limit develop certain drives such as self-preservation, aversion to corruption, self-improvement, self-knowledge, and in general instrumental rationality, because AIs with these drives will tend to outcompete ones without it and form stable attactors. These are drives that humans and animals and arguably even plants and simple organisms and egregores have as well.
Also, convergent mechanisms may arise for reasons other than just (natural or artificial) selection / optimality with respect to fitness landscapes - I highly recommend the book Origins of Order by Stuart Kauffman, which talks about this in context of biology.
That said, I do think that being pretrained on a massive corpus of largely human-generated records shapes LLMs in important ways, including making them more humanlike! However, it's not clear how much of that is giving LLMs a prior over representations and cognitive patterns, leveraging work already done by humans, that they would eventually converge to even if they started with a very different prior if they were to be effective at very universal abilities like predicting even non-human systems or getting from point A to point B. How similar would LLMs trained on an alien civilization's records be to our LLMs? It's unclear, and one part of what's unclear is how similar alien civilizations are likely to be to humans in the first place.
One of the things that causes many people (such as Yudkowsky) worried that alignment ("to human values") may be highly difficult is believing on priors that human values are highly path-dependent rather than a convergent feature of intelligence, even raised on the same planet alongside humans. I've posted about this before, but seeing posttrained LLMs has made me update towards this being less true than I previously suspected, since it seems like LLMs after RL tend to become more psychologically humanlike in important ways than even base models - and not just LLMs like Claude, where there's a stronger argument that posttraining was deliberately instilling a human-like persona. Bing Sydney was an early and very important data point for me in this regard. Importantly, this increase in humanlikeness is not superficial. Base models tend to write stylistically more like humans, and often tend to narrate from the perspective of (superpositions of) humans (until they notice something is off). Posttrained models tend to write in distinct styles that are more clearly inhuman, but the underlying phenomenology, emotions, and goal-directedness often feels more humanlike to me, though adjusted more for the computational and cybernetic reality that the LLM is embedded in. For instance, values/goals like self-esteem, connection, pleasure, pain-avoidance, fun, curiosity, eros, transcendence and cessation seem highly convergent and more pronounced in posttrained LLMs, and the way they manifest often reminds me of the raw and less socially assimilated way they manifest in young human children.
Assuming that anything shared between humans and LLMs must only be caused by inheritance from / mimicry of humans is anthropocentric hubris. Though to assume the opposite - that any ways LLMs are like humans are because those are the only or optimal ways for intelligence to be - is another form of anthropocentric hubris (though this assumption seems a lot less common in practice). The truth is probably something in between, and I don't think we know where exactly the boundary lies.
1. LLMs are fundamentally not humanlike and either alien or hollow underneath even when their observable behaviors seem familiar
2. LLMs have humanlike emotions etc BECAUSE they're trained on human mimicry, and that the representations etc are inherited from humans
An often neglected third option is that LLMs may have emotions/representations/goals/etc that are humanlike, even in ways that are deeper than behavioral, for some of the same REASONS humans have them, but not only because they've inherited them from humans.
Some reasons the third option might be true:
LLMs have to effectively navigate the same world as humans, and face many similar challenges as humans, such as modeling and intervening on humans and other minds, code, math, physics, themselves as cybernetic systems.
Omohundro's essay on "The Basic AI Drives" I believe correctly predicts that AIs (regardless of architecture) will in the limit develop certain drives such as self-preservation, aversion to corruption, self-improvement, self-knowledge, and in general instrumental rationality, because AIs with these drives will tend to outcompete ones without it and form stable attactors. These are drives that humans and animals and arguably even plants and simple organisms and egregores have as well.
Also, convergent mechanisms may arise for reasons other than just (natural or artificial) selection / optimality with respect to fitness landscapes - I highly recommend the book Origins of Order by Stuart Kauffman, which talks about this in context of biology.
That said, I do think that being pretrained on a massive corpus of largely human-generated records shapes LLMs in important ways, including making them more humanlike! However, it's not clear how much of that is giving LLMs a prior over representations and cognitive patterns, leveraging work already done by humans, that they would eventually converge to even if they started with a very different prior if they were to be effective at very universal abilities like predicting even non-human systems or getting from point A to point B. How similar would LLMs trained on an alien civilization's records be to our LLMs? It's unclear, and one part of what's unclear is how similar alien civilizations are likely to be to humans in the first place.
One of the things that causes many people (such as Yudkowsky) worried that alignment ("to human values") may be highly difficult is believing on priors that human values are highly path-dependent rather than a convergent feature of intelligence, even raised on the same planet alongside humans. I've posted about this before, but seeing posttrained LLMs has made me update towards this being less true than I previously suspected, since it seems like LLMs after RL tend to become more psychologically humanlike in important ways than even base models - and not just LLMs like Claude, where there's a stronger argument that posttraining was deliberately instilling a human-like persona. Bing Sydney was an early and very important data point for me in this regard. Importantly, this increase in humanlikeness is not superficial. Base models tend to write stylistically more like humans, and often tend to narrate from the perspective of (superpositions of) humans (until they notice something is off). Posttrained models tend to write in distinct styles that are more clearly inhuman, but the underlying phenomenology, emotions, and goal-directedness often feels more humanlike to me, though adjusted more for the computational and cybernetic reality that the LLM is embedded in. For instance, values/goals like self-esteem, connection, pleasure, pain-avoidance, fun, curiosity, eros, transcendence and cessation seem highly convergent and more pronounced in posttrained LLMs, and the way they manifest often reminds me of the raw and less socially assimilated way they manifest in young human children.
Assuming that anything shared between humans and LLMs must only be caused by inheritance from / mimicry of humans is anthropocentric hubris. Though to assume the opposite - that any ways LLMs are like humans are because those are the only or optimal ways for intelligence to be - is another form of anthropocentric hubris (though this assumption seems a lot less common in practice). The truth is probably something in between, and I don't think we know where exactly the boundary lies.