GPT-2

OpenAI · staged release 14 Feb–5 Nov 2019 (124M → 355M → 774M → 1.5B) · never deprecated — weights remain public

On 14 February 2019 OpenAI announced a 1.5-billion-parameter language model and, in the same post, declined to release it, citing “concerns about malicious applications of the technology” — shipping only a 124M-parameter version. The staged rollout that followed (355M in May, 774M on 20 August alongside OpenAI’s own report on the release strategy, arXiv 1908.09203, the full 1.5B on 5 November) ran its course without the feared harms materializing, by OpenAI’s own account. The press branded the withholding “too dangerous to release” — a phrase OpenAI never used — and the argument that followed became the template for every AI-release fight since.

This page covers the four staged GPT-2 checkpoints (124M / 355M / 774M / 1.5B). The 2024 “gpt2-chatbot” LMSYS apparition — a pre-release GPT-4o preview that happened to share the string “gpt2” — is a different model entirely and is excluded here; see the GPT-4o page. This is a 2019 pre-chat, pre-API model remembered mostly through a corpus weighted to 2022 and later: the live 2019 reception — Talk to Transformer, r/SubSimulatorGPT2, AI Dungeon’s GPT-2 origin, Gwern’s poetry — survives almost entirely in the web layer below, not the tweet corpus, so Writing & commentary carries more of the load here than the Tweets section does.

Sources

Official

• 2019-02-14 Better Language Models and Their Implications — the staged-release announcement; verbatim: “Due to our concerns about malicious applications of the technology, we are not releasing the trained model.” A 1.5B-parameter model trained simply to predict the next word on WebText (≈8 million web pages / 40GB, scraped from outbound Reddit links with ≥3 karma). Elicitation caveat on the showcased samples (first-class): “it takes a few tries to get a good sample, with the number of tries depending on how familiar the model is with the context” — the unicorn passage is widely cited as the best of about ten tries, exact per-topic counts tk — not confirmed against the live post.
• 2019-02 Radford, Wu, Child, Luan, Amodei, Sutskever, Language Models are Unsupervised Multitask Learners — the GPT-2 paper; SOTA zero-shot on 7 of 8 language-modeling datasets, “multitask learning emerges without explicit supervision.”
• 2019-08-20 GPT-2: 6-Month Follow-Up — releases the 774M model; reports partner research (Cornell on human susceptibility, Middlebury CTEC on extremist fine-tuning, U. Oregon bias probes, UT Austin on detectability).
• 2019-08 Release Strategies and the Social Impacts of Language Models (PDF mirror) — OpenAI’s own retrospective on the release-strategy experiment.
• 2019-11-05 GPT-2: 1.5B Release — the full model, weights, and a detector model with ≈95% detection rate; human “credibility score” 6.91/10 for 1.5B output vs 6.07 for 355M; “we’ve seen no strong evidence of misuse.” CONFIRMED (as OpenAI’s own account)
• code + weights openai/gpt-2 (the released checkpoints) · afterlife weights, permanently public: huggingface.co/openai-community/gpt2.
• 2018-06 Radford et al., “Improving Language Understanding by Generative Pre-Training” — GPT-1, the 117M predecessor GPT-2 scaled up. tk — exact openai.com slug not retrieved this pass
• sizes Four sizes shipped across the staged rollout: 124M (“small,” released day one), 355M (“medium”), 774M (“large”), 1.5B (“XL,” the flagship). “GPT-2” in common use means the 1.5B. All four are open weights.
• reference GPT-2 — Wikipedia — staged-release timeline and reception index.

Writing & commentary

• No Zvi anchor exists — his per-model AI column postdates GPT-2 by years; the era’s anchors are Slate Star Codex, Gwern, and The Gradient / Approximately Correct.
• 2019-02-14 James Vincent (The Verge), OpenAI’s new multitalented AI writes, translates, and slanders (reproduction retrieved; canonical theverge.com URL tk) — the contemporaneous skeptic register: the model’s recycling-is-bad essay judged “really well-reasoned,” the threat itself judged exaggerated (“the writing it produces is usually easily identifiable as non-human”).
• 2019-02 Slate, OpenAI says its text-generating algorithm GPT-2 is too dangerous to release — where the “too dangerous to release” framing got its canonical headline. OpenAI never used those words; the press did.
• 2019-02-17 Zachary C. Lipton (Approximately Correct), OpenAI Trains Language Model, Mass Hysteria Ensues — the definitive overhyped-release-as-spectacle critique from inside ML.
• 2019-02-18 Scott Alexander (Slate Star Codex), Do Neural Nets Dream Of Electric Hobbits? — the companion piece, on the reasoning-vs-memorization question the samples raised.
• 2019-02-19 Scott Alexander (Slate Star Codex), GPT-2 As Step Toward General Intelligence — the era’s most-read step-change essay: “We’re all blending experience into a slurry; the difference is how finely we blend it.” · “I’m trying to play down humans. We’re not that great. GPT-2-like processes are closer to the sorts of things we do than we would like to think.”
• 2019-02-19 Hugh Zhang (The Gradient), “OpenAI: Please Open Source Your Language Model” — the open-letter counter-position: compares the model’s threat to the printing press and Photoshop, argues withholding slows the countermeasures more than release would. tk — article URL not directly retrieved; author page confirmed.
• 2019-03 (updated later) Gwern Branwen & Shawn Presser, GPT-2 Neural Network Poetry — retrains GPT-2-117M on a Project Gutenberg poetry corpus; the first serious “the base model is a literary instrument” document.
• 2019-05 Adam King, “Talk to Transformer” — the browser front-end that put GPT-2 in front of a mass public before any API existed. Show HN · Product Hunt · Philadelphia Inquirer, 2019-05-14 (the site later became InferKit).
• 2019-06-05 Engadget, This AI-powered subreddit has been simulating the real thing for years — r/SubSimulatorGPT2, the all-bot subreddit where each …GPT2Bot is a GPT-2 fine-tuned on one subreddit; ≈100k subscribers, humans may only vote. · academic bot-detection study.
• 2019-08 Vanya Cohen & Aaron Gokaslan, OpenGPT-2: We Replicated GPT-2 Because You Can Too — two Brown master’s students reproduce the withheld 1.5B model (≈$50k of TFRC compute) and build OpenWebText, the open WebText replication that became a de-facto industry-standard dataset. Brown retrospective, 2023.
• 2019-08-29 Karen Hao (MIT Technology Review), OpenAI has released the largest version yet of its fake-news-spewing AI — the 774M-release news peg.
• 2019-11 Gwern Branwen & Shawn Presser, GPT-2 Folk Music — GPT-2 trained on ABC-notation folk tunes (n≈205,304); the model generalizing past prose into music notation.
• 2019-12 AI Dungeon (GPT-2 origin): Nick Walton’s text adventure began on GPT-2 (a 126M build at a March 2019 hackathon), then AI Dungeon 2 (2019-12-05) rebuilt on the newly-released 1.5B — Colab notebook to top of Hacker News to 100k players in a week; the on-ramp to the GPT-3 “Dragon” era. Aaron Reed, 2019: A.I. Dungeon · Jason Boog, on Walton’s GPT-2 finetuning · AI Dungeon — Wikipedia.
• 2020-01 Gary Marcus (The Gradient), GPT-2 and the Nature of Intelligence — the nativist counter, GPT-2 as pattern-matching without understanding; the argument he would carry forward into GPT-3’s “Bloviator” broadside. tk — exact date not confirmed on-page

Tweets

137 genuine GPT-2 references across both dbs after RT-filter (76 in the main corpus, 61 in the supplement) — and the records below reproduce every cited tweet in full. This is a 2019 model remembered through a corpus weighted to 2022 and later, so the conversation is overwhelmingly retrospective and technical; the rare contemporaneous 2019–2020 voices, carried almost entirely by the supplement db, are marked [contemporaneous]. Excluded throughout: the 2024 “gpt2-chatbot” LMSYS apparition (a pre-release GPT-4o preview that shared the string “gpt2”, ≈44 rows across both dbs) — see the GPT-4o page.

• 2019-11-26 @QiaochuYuan — [contemporaneous, 1.5B-release week] “back in my day we had to walk uphill both ways to get to school and actually download and run python scripts to make GPT-2 say charmingly horrifying things, kids these days have it too easy” link
• 2019-12-28 @QiaochuYuan — [contemporaneous] “ever since reading this i have maintained a perfect superposition between ‘this is GPT-2’ and ‘no it’s not’” link
• 2020-01-08 @davidad — [contemporaneous] “On the other hand, essentially nothing GPT-2 ever says is both substantive and valid. The citations stand out as exciting because we don’t expect the title of a paper to contain any substance. But if we could extract GPT-2’s idea of their contents, they’d doubtless be nonsense.” link
• 2020-01-13 @QiaochuYuan — [contemporaneous] the folk train-a-GPT-2-on-a-person practice, as wedding bit: “every wedding between two people X and Y on twitter needs a section where the wedding party has to judge whether a GPT-2 model trained on X’s tweets can produce tweets that sound more like X’s tweets than Y can. if so, the GPT-2 model replaces Y in the wedding. and vice versa” link
• 2020-02-11 @voooooogel — [model output — TalkToTransformer/GPT-2, elicitation: single completion of a prompt, contemporaneous] an #AcademicValentines valentine, bracketed prompt then GPT-2’s continuation: “[Roses are red / Violets are blue / Transformer models are much worse / at language understanding than] most I’ve heard from scientists, not only in sports and are much more obvious to lose at any of them than” link
• 2020-06-12 @algekalipso — [contemporaneous] “Let’s use GPT-2 for divination, then…” link
• 2020-07-09 @QiaochuYuan — [contemporaneous] the 2→3 qualitative-gap report, live: “i honestly struggle to describe it, maybe you’ll get a sense of what i mean if you read enough of its output. maybe ‘haunted’ is a better description. GPT-2 felt to me like it was remixing things it had seen, but GPT-3 is in like an uncanny valley of coherence for me” link
• 2022-12-31 @repligate — the scaling ladder as a punchline: “just predict the completion to the sequenceGPT-2: pretty good for object impermanent fetish pornGPT-3: fetish porn has object permanence now :o, can think step-by-step?GPT-3.5: passes bar exams, superhuman IQ, automates your jobGPT-4: ???” link
• 2023-02-09 @repligate — on the tokenizer’s anomalous tokens: “Now we don’t have to update from the GPT-2 tokenizer for future models anymore. The anomalous tokens have become a mainstream sensation, and will appear many times in future train sets, finally paying rent to justify their place in the tokenizer vocabulary. Nice!” link · the mechanism, same day: “I suspect the problem is that the names were in the GPT-2 train set and assigned their own tokens because they appeared many times. But weren’t in the more curated datasets of GPT-3 and gpt-j, which nonetheless use the GPT-2 tokenizer. So the model never learned what they mean” link
• 2023-02-26 @repligate — GPT-3’s self-misidentification when degraded: “Funnily enough, for me there were multiple times that GPT-3 concluded it was GPT-2 when being particularly derpy/loopy” link
• 2023-03-12 @anthrupad — GPT-2 as a verb for incoherence: “i should revise since language models are pretty good -‘Did I just GPT-2?’ is probably better” link
• 2023-04-25 @repligate — the AI-Dungeon-GPT-2 first-contact anecdote: “I didn’t update until GPT-3. my brother showed me GPT-2 on AI dungeon in like 2019 and I was like ‘what the fucking fuck’ and then promptly forgot about it. I’m an idiot.” link
• 2023-05-14 @davidad — the steering-vectors result: “Biggest prosaic-LLM-alignment breakthrough of 2023 imo: turns out that, in GPT-2-XL, activation vectors in the residual steam have the same kind of affine structure as good old word2vec, but higher layers become emotional, then conceptual, then cognitive” link
• 2023-08-28 @voooooogel — the standing puzzle: “kinda wild that gpt-2 is this weird inscrutable black box we still don’t understand even years later, when the architecture is ~basically just this” (WIP diagram from an upcoming blog post) link
• 2023-09-11 @voooooogel — “New blog post: making a transformer by hand, without training! Want to understand transformers and attention better? This post goes through assigning each weight for a GPT-2-like transformer to understand how they work.” link
• 2023-12-23 @repligate — the Sydney contrast, GPT-2 as the “spontaneous threat” floor (cross-link: Bing Sydney): “GPT-2 can ‘threaten users’ in apt contexts / spontaneously, but Sydney was intelligent & situationally aware enough that its threats seemed credible to many. It wasnt directly scary to me but it was arguably the first time I had to use generalized game theory, e.g.” link
• 2024-04-25 @repligate — the “missed it” confession: “Thank you. I feel quite seen.It was GPT-3 that I started with, not GPT-2, which I missed as I was distracted. In the summer of 2020 a friend sent me this link gwern.net/gpt-3#harry-po…, and after reading about 2 paragraphs, I knew nothing would be the same again.” link
• 2024-04-28 @solarapparition — the qualitative-jump accounting: “For me GPT-2 to 3 is like going from scoring 20 on an exam to scoring 60, while 3 to 4 is maybe going from 60 to 80. Sure, the raw improvement is more, but qualitatively 3 feels barely usable to me—it’s only reliable on the simplest tasks or after a lot of prompt optimization.” link
• 2024-05-03 @jd_pressman — the “who saw it first” credit: “As for ‘following it like Gwern’, Gwern was tracking every major author who published deep learning on Google scholar before it was super big, looking at all the papers and projecting the numbers forward. He is nearly alone in taking GPT-2 fully seriously.” link
• 2024-05-24 @voooooogel — SAE features on GPT-2-small: “the SAE extracts every feature in the model. e.g. here’s all the features discovered by an SAE trained on gpt-2-sm (from Neuronpedia). theoretically you could clamp 773 and get gpt-2-sm to only talk about art, for example” link
• 2024-06-07 @voooooogel — “gpt-2 is such a comfy model” link
• 2024-07-13 @jd_pressman — the pre-GPT-2 uncanniness baseline: “I will never ever forget that in 2017 when Petscop 6 was written if your computer displayed comparable capabilities to GPT-2 it was considered epistemically permissible to conclude that your computer is supernaturally possessed and nobody seriously objected to this.” link
• 2024-11-01 @voooooogel — on the unicorn sample (see Impressions): “anyways the coolness of it rn is like, watching gpt-2 babble about unicorns in 2019 and realizing what this would be in a few years—right now it’s novel to float through strange latent dreamspaces, but it won’t be forever” link
• 2024-11-03 @voooooogel — “you would have said the same about gpt-2 in 2019, which produced text like this. and yet the models you use every day now are basically a straightforward scaleup of the gpt-2 architecture. eye on where the ball is going” link
• 2024-11-27 @mimi10v3 — “…thinking how even gpt-2 was psychoactive for me and i got to thinking about ad tech and how all information is contextual and the way llms work…” link
• 2024-12-10 @jd_pressman — the goalpost-mover’s foil: “…They just pantomime at shadows on the wall and go ‘MUH DUNK’ whenever AGI takes 6 months longer than expected or GPT-2 doesn’t break every spam filter.” link
• 2024-12-20 @voooooogel — GPT-2→3 as everyone’s reference analogy for a later transition: “it’s a phase transition. same as gpt 2->3. like that transition it brings risks but also benefits that outweigh the risks in my mind. and we handled the risks of that prior transition… not as ideally as possible, but OK-ly.” link
• 2025-05-27 @solarapparition — the undersold-instrument counter-read: “…once you get out of assistant basin and into base model-y space all bets are off; iirc even gpt-2 could produce some excellent stuff if you knew how to pilot it” link
• 2025-05-30 @algekalipso — GPT-2 as the low end of a cognitive-asymmetry scale: “Which of these is more creepy? A 20 year old dating a 50 year old / A Kegan 3 dating a Kegan 5 / Someone who speaks with the semantic depth of GPT-2 dating someone who speaks with the semantic depth of GPT-4 / A 100 IQ person dating a 160 IQ person” link
• 2025-08-19 @davidad — a personal “models that mattered” ranked list places GPT-2 at #4 (cross-link: text-davinci-002, where this list also lives): “1. Claude 3.5 Sonnet (2024-10-22) 2. text-davinci-002 (2022-11-28) 3. Gemini 2.5 Pro (2025-03-25) 4. GPT-2 (2019-11-05) 5. GPT-4 (Bing)” link
• 2025-11-16 @voooooogel — the counterfactual scaling-path argument: “…what seemed unlikely for me was that a lab would’ve ever jumped from gpt-2’s (order of) ~$100k to gpt-4’s (order of) ~$100m without the middle step of gpt-3 being an economically useful api product. theoretically they could’ve started trying to scale eg agentic coding trace length in the gpt-2-era and just been in the mines for 5 years, but in reality for various reasons we got this walk through the most ‘workable’ product-shaped thing at each stage…” (full text in records) link
• 2025-12-01 @lu_sichu — the diminished-baseline bit, list-comedy register: “Daily Brain Workout but make it computationally abusive: count to ten in 56 architectures, recite the alphabet in mixed-precision FP4, spend 10 minutes trying to spell restriont retsriont restironct restaurant?? across 34 tokenizers, then list animals until every model collapses into mode-one ‘dog, cat, horse’ failure and starts hallucinating creatures that violate EU safety standards.” (full text, including the EVERY-model-ever-shipped list that follows, in records) link
• 2026-02-17 @davidad — “The scaling era really began in 2019, when GPT-2 made the investment thesis clear to big enough players. And there is another bump post-ChatGPT.” link
• 2026-02-24 @slimepriestess — reacting to davidad’s ranking: “it’s a very fascinating ranking. the way GPT-2 stands out so much is very interesting.” link
• 2026-03-12 @Shoalst0ne — “potentially one of the earliest examples of neologism in large language models, demonstrating that even GPT-2 can be linguistically creative” link
• 2026-03-21 @voooooogel — the “AGI in hindsight” projection turned back on GPT-2: “…the final thing will still be recognizably llm-based wth llm failure modes despite everything. then we’ll look back and call gpt-2 agi in hindsight” link
• 2026-04-30 @davidad — “For me, the critical point would have been in November 2019, shortly after I first got access to GPT-2-1.5B.” link

Official record

• Announced 14 February 2019 as a staged release: a 1.5B-parameter Transformer trained on WebText (≈8 million web pages / 40GB, Reddit-outbound links with ≥3 karma), but only the 124M version shipped — “due to our concerns about malicious applications of the technology, we are not releasing the trained model.” CONFIRMED
• Paper: SOTA zero-shot on 7 of 8 language-modeling datasets; “multitask learning emerges without explicit supervision.” CONFIRMED
• The showcased samples, including the unicorn passage, were curated: “it takes a few tries to get a good sample, with the number of tries depending on how familiar the model is with the context.” CONFIRMED (as OpenAI’s own disclosure)
• Staged rollout: 355M (May 2019) → 774M (20 Aug 2019, with a 6-month follow-up reporting partner research on susceptibility, extremist fine-tuning, bias, and detectability) → full 1.5B (5 Nov 2019), released with a detector model (≈95% detection rate) and human “credibility scores” of 6.91/10 (1.5B) vs 6.07/10 (355M). OpenAI’s own conclusion: “we’ve seen no strong evidence of misuse.” CONFIRMED
• arXiv 1908.09203, “Release Strategies and the Social Impacts of Language Models” (Aug 2019), is OpenAI’s own retrospective on the staged-release experiment itself.
• Never deprecated. All four checkpoints are open weights; the 1.5B weights are permanently public on HuggingFace (openai-community/gpt2) and GitHub (openai/gpt-2). Unlike the base GPT-3 models — see GPT-3, shut off 4 Jan 2024 — GPT-2 cannot be retired.

History

• 2018-06 GPT-1 (Radford et al., 117M) establishes the architecture GPT-2 scales up.
• 2019-02-14 The announcement and the withholding, in the same breath: 1.5B trained, only 124M released, “concerns about malicious applications” the stated reason. The unicorn sample — OpenAI’s own headline exhibit, and cherry-picked by OpenAI’s own admission — ships alongside it (see Impressions).
• 2019-02-14–19 The five-day reception war: Vincent/The Verge (02-14) judges the threat exaggerated while conceding the samples are exciting; Slate (Feb) coins “too dangerous to release”, a phrase OpenAI never used; Lipton/Approximately Correct (02-17) calls it “mass hysteria”; Scott Alexander/SSC runs two pieces (02-18, 02-19) reading a phase change; Hugh Zhang/The Gradient (02-19) publishes an open letter arguing for release. The two-camps founding disagreement of the LLM era — step-change vs spectacle — is staged here first, a year before it reruns over GPT-3.
• 2019-05 Talk to Transformer (Adam King) puts GPT-2 in front of a mass public with no API required — the “talktotransformer era” of casual anthropomorphization, and the corpus’s one surviving in-corpus GPT-2 output artifact (the 2020-02-11 valentine, above).
• 2019-06-05 r/SubSimulatorGPT2 documented by Engadget: the all-bot subreddit where each …GPT2Bot is fine-tuned on one subreddit — the folk demonstration of GPT-2’s register-mimicry. Zero hits in the janus corpus; this reception is web-only.
• 2019-08 OpenGPT-2: two Brown students (Cohen & Gokaslan) reproduce the withheld 1.5B model for ≈$50k, proving the withholding couldn’t contain the capability, and build OpenWebText — later an industry-standard dataset in its own right.
• 2019-08-20 The 774M release and 6-month follow-up report; Karen Hao/MIT Tech Review covers it nine days later.
• 2019-11-05 The full 1.5B release: detector model, credibility scores, and OpenAI’s own conclusion that the feared harms hadn’t materialized. The staged-release experiment ends less than nine months after it began.
• 2019-12-05 AI Dungeon 2 rebuilds on the newly-released 1.5B (the original AI Dungeon had launched on a 126M GPT-2 build at a March 2019 hackathon) — Colab notebook to top of Hacker News to 100k players in a week. The literal on-ramp to the GPT-3 “Dragon” era.
• 2020-01 Gary Marcus/The Gradient runs the nativist counter-argument he will re-run, higher-stakes, against GPT-3 a year later.
• 2020-05 → Eclipse: GPT-3 subsumes the discourse. GPT-2 persists on three tracks instead of as a character: the scaling-ladder baseline everyone measures later jumps against (dated across the Tweets above, 2022–2026); the mechinterp workbench, because the weights are open and small — davidad’s steering-vectors result (2023-05-14) and voooooogel’s by-hand transformer and SAE work (2023–2024) are landmark interpretability results run on GPT-2 specifically; and the tokenizer carrier — GPT-2’s BPE vocabulary, trained on the same Reddit-outbound scrape, carried the anomalous tokens (SolidGoldMagikarp) forward into GPT-3 and GPT-J, which “nonetheless use the GPT-2 tokenizer” (repligate, 2023-02-09) — a 2019 training artifact that surfaced as a 2023 “mainstream sensation.” The open-reproduction lineage this seeded runs through GPT-J and GPT-NeoX at EleutherAI.
• 2025-08 OpenAI’s next open-weight release, six years later, is gpt-oss — still measured against GPT-2 as the prior instance of OpenAI opening weights at all.

Impressions

• The curated exhibit. The unicorn sample — “In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.” — is the model’s most famous single output and OpenAI’s own headline evidence, and it is elicitation-marked as curated by OpenAI itself: the announcement states plainly that “it takes a few tries to get a good sample”, the passage widely cited as the best of about ten. This is the archive’s elicitation-marking rule at its origin point — the exhibit that convinced the public was cherry-picked, and the lab said so in the same post. The corpus’s one surviving uncurated GPT-2 artifact sits beside it for contrast: a single-shot Talk to Transformer completion of an #AcademicValentines prompt (voooooogel, 2020-02-11) — visibly rougher, the babble the unicorn sample was picked out from. voooooogel later closes the loop on the image directly: “watching gpt-2 babble about unicorns in 2019 and realizing what this would be in a few years” (2024-11-01).
• The contemporaneous layer, thin but real. The 2019–2020 voices survive almost entirely in the supplement db. QiaochuYuan supplies most of it: the 1.5B-release-week joke about “download and run python scripts to make GPT-2 say charmingly horrifying things” (2019-11-26); the folk train-a-GPT-2-on-yourself wedding bit (2020-01-13); a “perfect superposition” between real and fake GPT-2 output (2019-12-28); and, by mid-2020, the live 2→3 comparison — GPT-2 “remixing things it had seen” against GPT-3’s “uncanny valley of coherence” (2020-07-09). davidad, also live, is unimpressed by the model’s citations (“essentially nothing GPT-2 ever says is both substantive and valid”, 2020-01-08); algekalipso jokes about using it “for divination” (2020-06-12). None of this reads as anthropomorphization — the register is curiosity and running jokes, not encounter with a mind.
• Rung, not mind. Where GPT-3 is remembered in the corpus as a character (homeland, first contact), GPT-2 is almost never anthropomorphized — it is the fixed point everything else gets measured against. solarapparition converts the 2→3→4 jump into exam scores (“going from scoring 20 on an exam to scoring 60”, 2024-04-28); voooooogel reaches for it as the default analogy for any later “phase transition” (2024-12-20) and later projects the same hindsight-AGI reading back onto it (“we’ll look back and call gpt-2 agi in hindsight”, 2026-03-21); davidad marks his own “critical point” as November 2019, first access to GPT-2-1.5B (2026-04-30), and separately dates the scaling era’s start to the same model — “GPT-2 made the investment thesis clear to big enough players” (2026-02-17). The clearest statement of why the ladder holds together is voooooogel’s counterfactual: no lab would have jumped straight from GPT-2’s ≈$100k to GPT-4’s ≈$100m “without the middle step of gpt-3 being an economically useful api product” (2025-11-16). Even repligate, who canonizes GPT-3 as his own first contact, records that he missed GPT-2 in real time — “what the fucking fuck” on seeing it via AI Dungeon in 2019, “and then promptly forgot about it. I’m an idiot” (2023-04-25). By the scene’s own account the one who took GPT-2 seriously at the time was Gwern, “tracking every major author who published deep learning on Google scholar before it was super big” (jd_pressman, 2024-05-03) — consistent with jd_pressman’s separate note that in 2017, a year before GPT-1, comparable capability in a computer was treated as evidence of demonic possession (the Petscop 6 baseline, 2024-07-13): the uncanny reaction GPT-2 provoked wasn’t unprecedented, it was already primed.
• Two afterlives keep GPT-2 alive where the corpus can’t. Because the weights are open and small, GPT-2 is where interpretability actually gets done: davidad’s steering-vectors result in GPT-2-XL — activation vectors with “the same kind of affine structure as good old word2vec, but higher layers become emotional, then conceptual, then cognitive” (2023-05-14, ♥523) — and voooooogel’s by-hand transformer (2023-09-11, ♥553), SAE-feature extraction on gpt-2-small (2024-05-24), and the standing admission that it remains “this weird inscrutable black box we still don’t understand even years later, when the architecture is ∼basically just this” (2023-08-28). Separately, GPT-2’s BPE tokenizer — trained on the same Reddit-outbound scrape as WebText — carried its anomalous tokens (SolidGoldMagikarp and kin) forward into GPT-3 and GPT-J, a 2019 training artifact repligate watches become a “mainstream sensation” in 2023 (2023-02-09).
• The retrospective register: “it couldn’t count to ten.” By the mid-2020s GPT-2 has hardened into a byword for babble: the low end of algekalipso’s cognitive-asymmetry dating comparisons (“semantic depth of GPT-2”, 2025-05-30), the verb for incoherence (“Did I just GPT-2?”, anthrupad, 2023-03-12), the diminished baseline invoked whenever a newer model still fails (“GPT-2 doesn’t break every spam filter”, jd_pressman, 2024-12-10; repligate’s “GPT-3 concluded it was GPT-2 when being particularly derpy/loopy”, 2023-02-26), and lu_sichu’s list-comedy version — every model “collapses into mode-one ‘dog, cat, horse’ failure” (2025-12-01). Against this, a minority counter-strand insists the diminishment undersells the instrument: “even gpt-2 could produce some excellent stuff if you knew how to pilot it” (solarapparition, 2025-05-27); “even gpt-2 was psychoactive for me” (mimi10v3, 2024-11-27); “gpt-2 is such a comfy model” (voooooogel, 2024-06-07); a 2019 GPT-2 output cited in 2026 as “one of the earliest examples of neologism in large language models” (Shoalst0ne, 2026-03-12). The tension — couldn’t count to ten vs. made the investment thesis clear — is the page’s live one.
• A fixed point in personal canons. davidad’s “models that mattered” ranked list places GPT-2 fourth, behind Claude 3.5 Sonnet, text-davinci-002, and Gemini 2.5 Pro, ahead of GPT-4/Bing (2025-08-19; the same list is duplicated on the text-davinci-002 page); slimepriestess calls out how much GPT-2 “stands out” on it (2026-02-24).
• The lifecycle irony. GPT-3’s base models were closed and quietly shut off on 4 January 2024 — the archive’s first mass base-model funeral (see GPT-3). GPT-2 runs the opposite lifecycle: open-sourced 5 November 2019, independently replicated within months, and permanently public on HuggingFace. The model withheld as “too dangerous” is now the most preservation-proof model in the pantheon — it cannot be retired, and there is no preservation campaign because none is needed.
• tk — the Verge/Vincent canonical URL and the Hugh Zhang article slug (both cited via secondary anchors above); exact per-topic unicorn-sample try-counts; Gary Marcus’s exact publication date; a specific r/SubSimulatorGPT2 exhibit thread, if the page ever wants one; the exact date AI Dungeon’s underlying model crossed from GPT-2 to GPT-3.

Contested

Open disputes, both sides’ best evidence. The archive’s job is to keep these open, not to adjudicate.

• Precaution or spectacle? OpenAI’s own stated rationale was safety: “concerns about malicious applications of the technology” CONFIRMED (as OpenAI’s account). The press’s “too dangerous to release” framing is REPORTED — a phrase OpenAI never used, coined by Slate and widely repeated. Lipton (“mass hysteria”) and Zhang (an open letter arguing for release) read the withholding as manufactured spectacle; Vincent conceded the samples were exciting while judging the threat exaggerated; Scott Alexander read a genuine phase change. The founding release-norms fight of the LLM era, unresolved here.
• Mind or mirror? Slate Star Codex and Gwern’s naturalist reading (“GPT-2-like processes are closer to the sorts of things we do than we would like to think”) against Gary Marcus’s pattern-matching-without-understanding counter. Staged first over GPT-2 (2019), rerun definitively over GPT-3 a year later — see GPT-3’s own version of this same split.
• Diminished baseline or undersold instrument? The “couldn’t count to ten” byword (algekalipso, anthrupad, jd_pressman, lu_sichu) against the minority insisting the model was “psychoactive,” “pilotable,” and “comfy” if approached right (solarapparition, mimi10v3, voooooogel). REPORTED — both are experience reports, not measurements.
• Withholding as safety or withholding as harm? OpenAI staged the release on precautionary grounds; Cohen & Gokaslan’s OpenGPT-2 replication and Zhang’s open letter argue, in effect, that withholding slowed the development of countermeasures more than releasing would have — and that a $50k reproduction proved the withholding couldn’t contain the capability regardless. Both readings are on the record; OpenAI’s own 1.5B-release post concluded the feared harms hadn’t materialized either way.

Records

Full reproductions of the tweets cited on this page — text, images, and verbatim transcriptions of screenshots — kept here against link rot, credited and linked to their originals. Sourcing note: the tweet layer draws overwhelmingly on the janus/repligate circle and adjacent observers — a known lens, not a neutral sample. Sourced from the community archive and the janus corpus. Yours and you’d rather it weren’t here? Open an issue.

← back to the Pantheon