Quantization is a serving decision, and serving decisions should be made on your own workload, not on a model card.

From June 29 to July 2 we ran a public experiment on code.umans.ai called “How good is NVFP4 for coding?”: an NVFP4-quantized build of GLM-5.2 on our own GPUs, open to community volunteers on rotating seats. We served it at 200+ tokens per second, peaking near 250, with sub-2 s median time to first token. The whole-window median on the archived status page reads 189.5 because we spent the final stretch on tuning attempts, trading some speed while trying to close the quality gap.

Then we closed the experiment and unlisted the model.

If the tokens are not useful, they are not worth serving, no matter how efficiently we can produce them. 🙂

Why NVFP4 is tempting

NVFP4 is NVIDIA’s 4-bit floating-point format: E2M1 elements, an FP8 scale per 16-value block, plus a per-tensor FP32 scale. Effectively ~4.5 bits per weight, about 1.8x smaller than FP8, and Blackwell tensor cores run it natively. NVIDIA’s introduction covers the format.

For a model like GLM-5.2 (744B MoE, 40B activated, served from the publisher’s FP8 release), the back-of-the-envelope is brutal (details in the appendix):

• Weights: ~744 GB in FP8 vs ~420 GB in NVFP4 per copy. Freed GPU memory is KV cache, and KV cache is concurrency. We walked through that math in host-claude-code and GLM-5 vs Kimi-K2.5.
• Decode: ~40 GB moved per generated token in FP8 vs ~22 GB in NVFP4. Roughly 1.8x throughput headroom where it hurts most, under load.

GLM-5.2 is our busiest model. Anyone who has hit it at peak hours knows the pressure is real. A checkpoint that halves both terms deserved a real test, not a dismissal. And it cost zero new GPU spend: we ran the experiment on capacity we already had.

What we told users going in

We were skeptical before serving the first token, and the model’s public description said so:

“The published NVFP4 results look very flattering, but we’re skeptical of its real quality and performance: this checkpoint was NOT QAT post-trained for NVFP4 (the QAT-on-NVFP4 models are where we’ve seen the best quality), so treat the benchmarks with caution. […] for production work we recommend the fp8 umans-glm-5.2. […] expect it to be flaky and to go down under load: crash it, give it a moment, and try again.”

That description now lives in the machine-readable past-models feed, along with the exact checkpoint we served. Every current model exposes its served weights on /v1/models/info too; you should be able to check what any provider is actually running.

Benchmarks vs day-to-day work

There are two kinds of evidence about a model. There are benchmark tables, which can be benchmaxed: tuned for, cherry-picked, or just narrow. And there is how the model feels after three hours of real work, which cannot.

To be clear, we are not accusing anyone of benchmaxing here. The GLM-5.2 NVFP4 card is transparent about its method (post-training quantization, no QAT) and its five benchmarks all sit within about a point of the FP8 baseline (full table in the appendix). The problem is coverage: no SWE-bench, no DeepSWE-style long-horizon agentic evals, nothing that chains hundreds of dependent tool calls. The known failure mode of PTQ lives exactly there: small per-step numerical drift that a one-shot benchmark cannot see, compounding across a long trajectory.

The card is not wrong. It just cannot answer the only question we care about. So we asked it the only way it can be asked: real users, real work, four days.

What four days of real use said

The first hours sounded like this, verbatim:

“Man its so fast its like im in heaven”

“glm at this speed is intoxicating though […] a near-frontier model running this fast is ridiculous”

“it’s almost too fast to believe it’s doing anything meaningful but looking at the quality it produced, it’s actually comparable/better in some cases”

Participants posted side-by-side outputs against fp8, and one even ran a quick benchmark of their own and reported it looked really good. If we had stopped there, NVFP4 wins.

Then the day-to-day verdicts arrived:

“my workflow seems dumb with nvfp4 but was good with fp8 🤔 same code changed nothing”

“unfortunately from my preliminary probing there is COT collapse, and not even at deep context.”

“i went back to fp8 because nvfp4 was just making enough mistakes to keep setting off TTSR rules in omp, whereas fp8 seems to rarely do so […] nvfp4 is a nice option to have for the raw speed, but idk if it can replace fp8, at least in my experience so far”

The second and the last quotes are the same person, separated by about three hours of real work. That arc, speed euphoria to quietly switching back, is the measurement no benchmark table gives you. Others caught concrete artifacts along the way: reasoning leaking outside thinking blocks, chain-of-thought collapsing well before deep context.

Our wrap-up in Discord said it plainly: “feedback was consistent: speed was great, but for serious work the gap with fp8 was clear. so we’re keeping quality first and closing this one out.”

Operational honesty for the record: the experiment also went down once, on June 30, on an SGLang shape bug at the very edge of the 405K context window in the speculative-decode path, and stayed down 77 minutes because we had not configured a restart policy on experiment capacity. That one is on us, and it is what “low availability, crash it, give it a moment” meant in practice.

QAT is the difference, not the format

Mid-experiment, one participant asked exactly the right question: “Still no QAT, right? What needs to happen for that?”

NVFP4 is not the villain here. The same format, applied to a model that was post-trained for 4-bit precision, is essentially lossless.

Kimi-K2.7-Code, our current default coder, ships from Moonshot natively quantization-aware trained at 4 bits (same method as K2-Thinking), and we serve the released weights as-is. Our rule, straight from our FAQ: quality first; when a model is released with QAT weights, we serve those directly rather than quantize on top.

And when NVIDIA converted those QAT weights to NVFP4, they could publish the benchmarks that matter. The K2.7-Code NVFP4 card includes SWE-bench Verified and Terminal Bench 2.1:

NVFP4 scores a hair above the baseline, well within the confidence interval, so the honest claim is parity. A 4-bit model re-encoded into a 4-bit format loses nothing, because the post-training already taught the weights to live at that precision.

Same format, opposite outcomes. The variable is whether the lab trained for it.

What we ship

We closed the experiment on July 2, four days in, and unlisted umans-glm-5.2-nvfp4. Nothing was replaced: the fp8 umans-glm-5.2 was serving in production the whole time, kept improving while the experiment ran, and is the same model the NVFP4 description itself recommended for real work.

The serving recipe stays on the shelf for checkpoints that earn it: post-trained at the precision we serve, benchmarked on long-horizon software engineering. K2.7 proves that path exists.

Efficiency multiplies usefulness. Four days of real usage told us the usefulness was not there, so the efficiency did not matter.

Thanks to everyone who volunteered their real workloads and their patience, including the ones who disagreed with the verdict. The fastest way to a trustworthy model lineup is users who tell us bluntly whether the tokens are useful.

Appendix

A. Weights and decode math

NVFP4 stores one 4-bit E2M1 element plus one FP8 scale per 16-value block, ~4.5 bits per value, so bytes per parameter is ~0.56 vs 1.0 for FP8.

• Weights: $744\text{B} \times 0.56 \approx 420\ \text{GB}$ vs $744\ \text{GB}$ in FP8, per copy of the weights.
• Decode bytes per token (MoE proxy, activated params times bytes per weight): $40\text{B} \times 0.56 \approx 22\ \text{GB}$ vs $40\ \text{GB}$.

The real ratio is slightly worse than the ideal: the NVFP4 checkpoint quantizes the linear operators inside the MoE experts and leaves the shared expert unquantized (per the model card).

B. The published GLM-5.2 NVFP4 benchmarks

From the model card, NVFP4 vs FP8 baseline:

Near-parity across the board. τ²-Bench Telecom is agentic, but its episodes are short-horizon customer-service flows. Nothing here measures hundreds of dependent coding steps, which is where our users live and where the gap showed.