[CUDA] Fixing quantization uint8 packing bug for NF4 and FP4

[Mhmd-Hisham]

This pull request addresses a bug in the quantization logic for FP4 and NF4 during 8-bit packing in the kQuantizeBlockwise CUDA kernel.

The expected behaviour is to pack two quantized 4-bit values into a single unsigned char (uint8) variable to reduce the overall memory footprint. The issue in the current implementation is that when packing values, the intermediate variable used was not getting cleared between iterations.

Since we quantize values per block, the error in larger block sizes would blow up, rendering the algorithm ineffective when block sizes are larger than 512.

List of changes:

• Fixed uint8 packing bug for NF4 and FP4.
• Modified the corresponding unit tests to account for larger block sizes.
• Improved FP4 mean absolute error by ~73% and relative error by ~96% compared to the baseline for block sizes >= 1024.
• Improved NF4 mean absolute error by ~88% and relative error by ~98% compared to the baseline for block sizes >= 1024.

The following graph highlights the accuracy improvements:

[github-actions]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[matthewdouglas]

Hi @Mhmd-Hisham,

Thank you for finding this, contributing this fix, and for your detailed explanation. Much appreciated!