BitDance: Computing NVFP4 on MXFP6 Hardware

The views expressed here are my own and do not represent those of my employer. The experiments discuss data formats and implementation choices; they do not describe product specifications or roadmaps.

The previous article, MXFP and NVFP: Block Scales, Two-Level Scaling, and Quantization Error, showed that with the same E2M1 elements, an E4M3 scale fits the ideal scale more closely than E8M0. NVFP4 B32 therefore has lower weight-only error than MXFP4.

NVFP4, however, is not an OCP Microscaling format, and native compute support is currently specific to NVIDIA hardware. Other vendors, including AMD, support OCP formats such as MXFP4 and MXFP6 and cannot directly consume NVFP4 weights.

This does not mean the E4M3 advantage is tied to one hardware platform. MI355X has the same 10.1-PFLOPS peak for MXFP6 and MXFP4—about four times BF16. If the information in an E4M3 scale can be rearranged into an E8M0 scale and E2M3 elements, 4-bit weight storage can be retained while computation runs on native MXFP6 matrix hardware.

BitDance performs this conversion on chip. Weights remain 4-bit in HBM. After a GEMM tile is loaded, the E4M3 exponent becomes an E8M0 scale and the mantissa is folded into E2M3 elements before the tile enters MXFP6 MFMA. No 6-bit intermediate weight tensor is written back to memory.

Standard NVFP4 uses 16-element micro-blocks. This implementation uses a 32-element scale block, denoted NVFP4 B32. All results below use B32.

1. From E4M3 × E2M1 to E8M0 × E2M3

Ignoring the unchanged tensor-level global scale, an NVFP4 B32 element is

Here, is the unbiased E4M3-scale exponent, , and is the signed E2M1 value. The target MXFP6 representation is

is an exponent bump. The E4M3 exponent moves into E8M0, while the mantissa is multiplied into E2M1 and rounded to E2M3. The sign and global scale stay unchanged, so the only new numerical error is the final E2M3 rounding.

For example, E4M3 byte 0 | 1010 | 100 (0x54) represents , while E2M1 0 | 11 | 1 represents 6. Their value is 72. With , E8M0 stores and E2M3 stores , giving .

This is not a dequantize-then-requantize pass. The exponent and mantissa split as the tile enters GEMM: one becomes the block scale; the other is absorbed into each element.

2. Exponent Bump

The largest E2M1 magnitude is 6, while E2M3 tops out at 7.5. Without changing the exponent, may overflow E2M3. In the example above, would saturate to 7.5 and reconstruct as 60. Moving one power of two to the scale side gives

The minimal overflow condition is , which requires a bump when the mantissa field is at least 3. The current implementation uses a simpler rule: bump every nonzero mantissa. The two rules differ by only about 0.001 dB on normal-scale weights; all results below use the implementation rule.

3. A 64-Entry LUT and HWCVT

E4M3 has eight scale mantissas and E2M1 has eight magnitudes. The sign is copied directly, so the full conversion fits in 64 entries:

Two implementations follow the same equation:

LUT looks up the E2M3 code, restores the sign, and packs a 6-bit fragment. It is the bit-exact reference and fallback.

HWCVT converts FP4 to F16, applies , and uses the hardware F16-to-FP6 conversion to generate the packed fragment.

They differ only at exact rounding midpoints. The LUT chooses the lower magnitude; HWCVT uses round-to-nearest-even. Twelve of the 64 inputs are midpoints, and six produce different codes, but both choices are equally far from the exact value.

On random operands, about 8.87% of element bits differ while output cosine similarity remains 0.9997. A bit comparison detects different rounding rules; it does not establish which path is more accurate.

4. Weight-Only Quantization Error

Local error is measured over 32-element windows. The box spans p10–p90; the line is p50; the open diamond is the overall mean; the short tick is p99; and the cross is the maximum. The tests cover four synthetic distributions, Qwen3.5-4B, and Gemma 4 12B. No rotation, SmoothQuant, AWQ, GPTQ, error compensation, or MSE scale search is used.

NVFP4 B32→MXFP6 stays close to direct NVFP4 B32 and remains below MXFP4:

Data	NVFP4 B32 mean	NVFP4 B32→MXFP6 mean	MXFP4 mean
Synthetic tensors	10.955%	10.767%	12.081%
Qwen3.5-4B	10.55%	10.80%	11.80%
Gemma 4 12B	10.568%	10.520%	11.884%

Folding the E4M3 mantissa into E2M3 therefore preserves the original advantage. Small apparent improvements come from non-monotonic interactions between two quantization grids; no lost information is recovered.

E4M4 is not a useful source format for this path. Its four scale-mantissa bits cannot fit into E2M3's three mantissa bits, so the extra precision must be rounded away:

Data	FP4SE4 direct	FP4SE4→MXFP6	NVFP4 B32→MXFP6
Synthetic	10.484%	11.482%	10.767%
Qwen3.5-4B	10.27%	10.79%	10.80%
Gemma 4 12B	10.277%	10.977%	10.520%

5. Operator-Level Error

Weight reconstruction omits activation quantization and GEMM accumulation. The operator census uses Gemma 4 12B, six model depths, and 24 MMLU-Pro prompts, producing 984 (linear, prompt) cells compared with BF16 GEMM output.

The following shorthand is used below: the number after w is the weight bit width, the number after a is the activation bit width, and _tc denotes transcode:

w6a6: MXFP6 weights and MXFP6 activations;

w4a6: MXFP4 weights and MXFP6 activations;

w4a6_tc: NVFP4 B32 weights transcoded to MXFP6 in the kernel, with MXFP6 activations;

w4a4_tc: both weights and activations stored as NVFP4 B32 and transcoded to MXFP6;

w4a4: MXFP4 weights and MXFP4 activations.

The bit widths indicate whether _tc applies to weights only or to both operands.

Path	mean	p50	p90	p99	max
w6a6	3.68%	3.57%	5.22%	7.18%	7.87%
w4a6_tc	8.57%	8.00%	12.10%	17.61%	18.87%
w4a6	10.12%	9.65%	14.66%	19.64%	21.21%
w4a4_tc	11.75%	11.02%	16.14%	25.74%	28.05%
w4a4	14.86%	13.91%	20.84%	31.62%	34.84%

The weight-only ordering carries into GEMM output. With 6-bit activations, w4a6_tc improves SNR over w4a6 by 1.45 dB on 12B and 1.55 dB on the 31B aggregate. With 4-bit activations, w4a4_tc also beats w4a4.

6. End-to-End Accuracy

The final test asks whether operator error changes model behavior. The vertical axis below is generative MMLU-Pro/GPQA accuracy; the horizontal axis is batch-32 vLLM output throughput normalized to BF16. Thinking is disabled and the output cap is 16,384 tokens. MMLU-Pro uses a fixed 500-item stratified sample; GPQA uses all 198 items.

End-to-end accuracy-performance tradeoff

Gemma 4 12B

Path	Throughput	vs BF16	MMLU-Pro	GPQA
bf16	3463 tok/s	1.00×	77.8	59.6
w6a6	4015 tok/s	1.16×	77.2	60.1
w4a6_tc	3970 tok/s	1.15×	73.0	57.1
w4a6	4296 tok/s	1.24×	71.6	53.5
w4a4_tc	3991 tok/s	1.15×	69.2	48.5
w4a4	4850 tok/s	1.40×	57.6	26.8

Gemma 4 31B

Path	Throughput	vs BF16	MMLU-Pro	GPQA
bf16	1974 tok/s	1.00×	86.2	78.8
w6a6	2350 tok/s	1.19×	85.4	78.3
w4a6_tc	2532 tok/s	1.28×	85.2	75.8
w4a6	2666 tok/s	1.35×	84.6	74.7
w4a4_tc	2534 tok/s	1.28×	82.8	71.2
w4a4	3098 tok/s	1.57×	83.6	72.7

The trade-off is clear: w4a4 is fastest but loses the most accuracy, while w4a6_tc gives up some throughput for better accuracy. Relative to w4a6, it gains 1.4 and 0.6 MMLU-Pro points on 12B and 31B; on 31B, it still reaches 1.28× BF16 throughput.

On 12B, w4a6_tc trails BF16 by 4.8 points, a difference resolved by the paired test (). On 31B, the differences between BF16 and w6a6, w4a6_tc, or w4a6 are not resolved by the current 500-item sample. GPQA has only 198 items, giving minimum resolvable gaps of about 9.7 and 8.1 points for 12B and 31B, so it cannot reliably order neighboring paths.

Generation length also affects accuracy. In the tables below, 4K→16K means raising the per-question output cap from 4,096 to 16,384 tokens. The values on either side of the arrow correspond to the two caps. Truncation rate is the fraction of samples that consume the full token budget without terminating; 16K mean output is the mean generated length under the 16K cap.

Gemma 4 12B · MMLU-Pro

Path	4K→16K accuracy	4K→16K truncation	16K mean output
bf16	76.6→77.8	1.0%→0.0%	782
w6a6	76.8→77.2	0.6%→0.0%	646
w4a6_tc	71.8→73.0	3.8%→1.2%	867
w4a6	68.0→71.6	10.0%→4.8%	1608
w4a4_tc	68.2→69.2	4.0%→1.0%	929
w4a4	49.6→57.6	37.4%→20.4%	4318

Gemma 4 12B · GPQA

Path	4K→16K accuracy	4K→16K truncation	16K mean output
bf16	63.1→64.6	5.6%→1.0%	1409
w6a6	60.1→60.1	4.0%→1.0%	1266
w4a6_tc	52.0→57.1	10.1%→2.0%	1641
w4a6	46.5→53.5	22.7%→10.1%	3052
w4a4_tc	45.5→48.5	9.6%→1.5%	1409
w4a4	20.2→26.8	58.6%→39.4%	7583

Gemma 4 31B · MMLU-Pro

Path	4K→16K accuracy	4K→16K truncation	16K mean output
bf16	86.2→86.2	0.2%→0.0%	523
w6a6	85.4→85.4	0.0%→0.0%	526
w4a6_tc	85.0→85.2	0.2%→0.0%	490
w4a6	84.6→84.6	0.0%→0.0%	524
w4a4_tc	82.8→82.8	0.0%→0.0%	500
w4a4	83.6→83.6	0.0%→0.0%	514

Gemma 4 31B · GPQA

Path	4K→16K accuracy	4K→16K truncation	16K mean output
bf16	77.8→78.8	2.0%→0.0%	786
w6a6	77.3→78.3	2.0%→0.0%	823
w4a6_tc	75.3→75.8	1.0%→0.0%	788
w4a6	73.7→74.7	2.0%→0.0%	763
w4a4_tc	71.2→71.2	0.5%→0.0%	739
w4a4	72.7→72.7	0.0%→0.0%	747

The 12B model is particularly sensitive to the cap. Even at 16K, w4a4 truncates 20.4% of MMLU-Pro and 39.4% of GPQA, with mean outputs of 4.3K and 7.6K tokens. w4a4_tc truncates only 1.0% and 1.5%, with mean outputs near 0.9K and 1.4K. w4a6_tc likewise produces shorter answers, fewer truncations, and higher accuracy than w4a6.

The 31B model behaves differently. MMLU-Pro has almost no truncation at 4K, and every GPQA path reaches zero truncation at 16K. Raising the cap changes accuracy by at most about one point. To avoid mixing output length into accuracy, all headline results use the 16K cap.

Beyond accuracy, transcode preserves the storage benefit of 4-bit weights. The chart includes quantized linear weights and unquantized embeddings, norms, and output heads. It uses binary GiB: 1 GiB is bytes, or about 1.0737 GB. GPU capacity is usually quoted in decimal GB, so 58.25 GiB is about 62.55 GB.

w4a6_tc and w4a4_tc occupy the same memory as w4a6 and w4a4: 7.37 GiB on 12B and 18.19 GiB on 31B. Relative to BF16, they are 3.02× and 3.20× smaller; relative to native w6a6, they save about 41% and 43%. MXFP6 fragments exist only on chip and do not increase resident weight memory.

7. Performance

The implementation extends the ROCm FlyDSL preshuffle GEMM. It keeps the tiling, preshuffle layout, asynchronous copies, and MFMA loop of compile_preshuffle_gemm_a8, adding E4M3→MXFP6 conversion only in operand staging.

This kernel was produced entirely through vibe coding and has not received systematic optimization by a GPU-kernel expert. The results describe the current implementation, not the method's ceiling; data layout, instruction scheduling, register use, pipelining, and tile configuration may all have further headroom.

Transcode adds scale decoding, exponent bumping, mantissa multiplication, and E2M3 packing. Those extra VALU operations and register pressure explain why _tc trails a native format of the same stored width on compute-bound shapes.

FlyDSL implements both LUT and HWCVT. With each path using its own best tile, HWCVT wins on all six shapes:

Operator source	M×N×K	LUT TFLOP/s	HWCVT TFLOP/s	Gain
Synthetic square GEMM	4096×4096×8192	1949.1	2564.5	31.6%
Qwen3 gate_up prefill	4096×19456×2560	2168.0	2361.4	8.9%
FLUX double-stream image QKV	4096×9216×3072	1890.0	2148.7	13.7%
FLUX double-stream image projection	4096×3072×3072	1659.3	1956.0	17.9%
FLUX double-stream FC1	4096×12288×3072	2280.3	2449.2	7.4%
FLUX single-stream LIN2	4352×3072×15360	1938.9	2183.2	12.6%

HWCVT's gain depends on shape and tile. Both paths retain 4-bit HBM traffic and avoid writing a 6-bit intermediate tensor.

8. Scope and Conclusion

All implementation and performance results apply to NVFP4 B32. Standard NVFP4 uses a B16 scale layout and requires separate adaptation.

BitDance rearranges the information in E4M3: the exponent moves to E8M0, the mantissa is folded into E2M3, and an exponent bump prevents saturation. This preserves 4-bit NVFP4 storage while enabling native MXFP6 compute.

Weight reconstruction, operator output, and end-to-end tasks all show the same direction: transcode retains the NVFP4 B32 accuracy advantage over MXFP4. The LUT provides a stable reference, while HWCVT is faster on the measured shapes.

This article validates transcode→MXFP6 only. An MXFP8 target could accommodate more scale information and need not be limited to E4M3 inputs. Whether that is worthwhile depends on the platform's MXFP4/MXFP6/MXFP8 throughput ratios and the availability of efficient decode, scale-fold, and packing instructions.