Conclusion from my experiments:
Tested: Deepseek V4.1 flash, Qwen 3.8 flash, glm 5.3 flash, muse spark, gpt Luna
Roughly similar in price (avg $0.2-$0.8/task)
Muse was the cheapest and also the best by far (!!) - but this is with the 10x-20x data sharing subsidy.
Luna did quite quite poorly, very surprising, worst here many parsing errors, on priors would’ve guessed best if anything.
Deepseek v4 was second best imo. And maybe “true best”, given no subsidy.
Tried a few quantizations and generally switched to “official” ish providers for oss models to test w/o needing to account for quantization issues or other model server inconsistency.
Tested: Deepseek V4.1 flash, Qwen 3.8 flash, glm 5.3 flash, muse spark, gpt Luna
Roughly similar in price (avg $0.2-$0.8/task)
Muse was the cheapest and also the best by far (!!) - but this is with the 10x-20x data sharing subsidy.
Luna did quite quite poorly, very surprising, worst here many parsing errors, on priors would’ve guessed best if anything.
Deepseek v4 was second best imo. And maybe “true best”, given no subsidy.
Tried a few quantizations and generally switched to “official” ish providers for oss models to test w/o needing to account for quantization issues or other model server inconsistency.