the quality battery I use for every model:
1. JSON extraction
2. code generation
3. logic puzzle
4. system prompt adherence
5. hallucination resistance
6. format compliance
results across 7 models:
• Gemma 4 E4B: 6/6
• Mistral 7B: 5/6
• LFM2 24B: 4/6
• Gemma 4 E2B: 4/6
• GLM 4.7 Flash: 2/6
• Qwen3 8B: 1/6
• Llama 3.2 1B: 1/6
the biggest model scored worst.
the smallest useful model (Gemma E4B) scored best
the parameters don’t predict the quality. everyone should test before trusting
what quality tests do you run on local models? looking for ideas to add to the battery
1. JSON extraction
2. code generation
3. logic puzzle
4. system prompt adherence
5. hallucination resistance
6. format compliance
results across 7 models:
• Gemma 4 E4B: 6/6
• Mistral 7B: 5/6
• LFM2 24B: 4/6
• Gemma 4 E2B: 4/6
• GLM 4.7 Flash: 2/6
• Qwen3 8B: 1/6
• Llama 3.2 1B: 1/6
the biggest model scored worst.
the smallest useful model (Gemma E4B) scored best
the parameters don’t predict the quality. everyone should test before trusting
what quality tests do you run on local models? looking for ideas to add to the battery