# @solarapparition — 2024-01-16

♥0 ↻0 · https://x.com/solarapparition/status/1747104409199198595

7/?Not-reasons for catch-up 2:- Unclear how well new architectures (Mamba, RNN+ etc.) scale to frontier model sizes—1T params and up.- Similar, techniques such as stacking are unproven—best open model, Mixtral, uses MoE, which is not new and what GPT-4 model probably uses.

tags: author:solarapparition, kind:tweet, model:gpt-4, on:mixtral-8x7b, year:2024
cited on: _dossiers/mixtral-8x7b.md, mixtral-8x7b