I struggle with approaching this as I do with many quants, in that there's not really a good measurement of how lobotomized by quantization they are other than "a bit but it's not too bad". Ternary weights immediately make me think "unusably quantized" but I'm presuming that's not the case if someone's standing be this release without massive caveats.
I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.
I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.
That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase throughput speed. In the meanwhile, there's research going into how this quantisation can be executed while correcting for the loss. In this case we're not just quantising but also post training the model, and we can "trade" some capabilities to others (ex. post-train the experts for math but lose on biology or whatever).
With that said, I find nvfp4 to be far superior. I think right now all you can do is try out the models, maybe run your own benchmarks if you have.
we achieved this on stock nvfp4 model which is within 5% of fp16 version
we then apply lossless techniques only
dflash2 as spec drafter and patched ninfer inference engine (down projections for short token lengths and adding tool call related schema
to drafter shortlist)
It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls and other syntax related tasks.
This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.
There's no information being shared other than these models exist and I don't want to drill down through 3+ links trying to find what you're saying about them.
Who would run a random script on a random page with no other information given?
I'm not saying this is bad, but they definitely need to explain what the heck is going on to the point where a 15 year old can understand it, and provide logical and clear breadcrumbs to navigate through their pages.
I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.
I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.
Frustrating, as a hobbyist.
With that said, I find nvfp4 to be far superior. I think right now all you can do is try out the models, maybe run your own benchmarks if you have.
we then apply lossless techniques only
dflash2 as spec drafter and patched ninfer inference engine (down projections for short token lengths and adding tool call related schema to drafter shortlist)
see: https://github.com/meshapplied/goethe/pull/4
benchmark numbers are measured using see bench verified traces at avg input 28k and concurrency 1
https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.
The website is not as good as it needs to be.
There's no information being shared other than these models exist and I don't want to drill down through 3+ links trying to find what you're saying about them.
Who would run a random script on a random page with no other information given?
I'm not saying this is bad, but they definitely need to explain what the heck is going on to the point where a 15 year old can understand it, and provide logical and clear breadcrumbs to navigate through their pages.
Even a FAQ would help.