# @jd_pressman — 2023-11-05

♥5 ↻0 · https://x.com/jd_pressman/status/1721121604338671997

@teortaxesTex @Teknium1 It's actually based on my SFT Instruct finetune of Mistral 7B, the one used as the evaluator in MiniHF.

https://t.co/925urkZhut

It's then weight decayed over the tuning towards the base model weights along with a KL loss on the base model, this helps prevent mode collapse.

tags: author:jd_pressman, kind:tweet, on:mistral-7b, year:2023
cited on: _dossiers/mistral-7b.md, mistral-7b