QeRL: NVFP4-Quantized Reinforcement Finding out (RL) Brings 32B LLM Teaching To A Single H100—Whereas Bettering Exploration

What would you assemble when you might run Reinforcement Finding out (RL) post-training on a 32B LLM in 4-bit NVFP4—on a single H100—with BF16-level accuracy and 1.2–1.5× step speedups? NVIDIA researchers (with collaborators from MIT, HKU, and Tsinghua) have open-sourced QeRL (Quantization-enhanced Reinforcement Finding out), a training framework that pushes Reinforcement Finding out (RL) post-training into 4-bit FP4 (NVFP4) whereas sustaining gradient math in higher precision by the use of LoRA. The evaluation group experiences >1.5× speedups throughout the rollout part, ~1.8× end-to-end vs QLoRA in a single setting, and the first demonstration of RL teaching for a 32B protection on a single H100-80GB GPU.

What QeRL changes throughout the Reinforcement Finding out (RL) loop?

Most RLHF/GRPO/DAPO pipelines spend nearly all of wall-clock time in rollouts (token period). QeRL shifts the protection’s weight path to NVFP4 (FP4) with dual-level scaling and retains logits/gradients in higher precision by the use of LoRA, so backprop stays regular whereas the sampling path hits hardware-efficient FP4×BF16 kernels (Marlin). The outcome’s sooner prefill/decoding all through rollouts with out sustaining a separate full-precision protection.

Mechanically, the evaluation group integrates Marlin-based FP4 kernels in every rollout and prefill, whereas LoRA limits trainable parameters. This straight targets the stage that dominates RL worth and latency for prolonged reasoning traces.

Quantization as exploration, made schedulable

A core empirical discovering: deterministic FP4 quantization raises protection entropy, flattening token distributions early in teaching and bettering exploration versus 16-bit LoRA and NF4-based QLoRA baselines. To handle that affect over time, QeRL introduces Adaptive Quantization Noise (AQN)—channel-wise Gaussian perturbations mapped into LayerNorm scale parameters and annealed with an exponential schedule. This retains kernel fusion intact (no further weight tensors) whereas transitioning from exploration to exploitation.

In ablations, QeRL reveals sooner reward progress and higher remaining scores on math-reasoning duties beneath every GRPO and DAPO, aligning with the hypothesis that structured noise in parameter home could possibly be a useful exploration driver in RL, even if such noise is usually detrimental in supervised fine-tuning.

Reported outcomes

On Qwen2.5 backbone model, the evaluation group current that NVFP4+LoRA outperforms vanilla LoRA and QLoRA in rollout throughput and whole teaching time, with >2× rollout throughput on 14B/32B fashions in the direction of QLoRA and ~1.8× end-to-end vs QLoRA in a advisor setup. As well as they exhibit teaching a 32B protection with GRPO on a single H100-80GB, enabled by the lower memory footprint of weight-only FP4.

Accuracy is aggressive with higher-precision baselines. For a 7B model, the evaluation group experiences GSM8K = 90.8% and MATH500 = 77.4%, surpassing 16-bit LoRA and QLoRA beneath their setup and matching full-parameter fine-tuning. All through broader math benchmarks (e.g., BigMath), QeRL maintains parity or profit, whereas converging sooner ensuing from improved exploration.

What that’s—and isn’t?

QeRL is weight-only FP4 with LoRA updates; it does not declare FP4 precision for logits/gradients. The benefits focus in rollout/prefill throughput and memory footprint, with empirical proof that quantization-induced entropy aids RL exploration when AQN modulates it over teaching. Generalization to modalities previous math-reasoning duties or to safety/tool-use RL depends on reward design and sequence lengths.

Key Takeaways

• QeRL combines NVFP4 4-bit weight quantization with LoRA to hurry up the rollout part and reduce memory, enabling RL for a 32B LLM on a single H100-80GB.
• Quantization acts as exploration: FP4 will enhance protection entropy, whereas Adaptive Quantization Noise (AQN) schedules channel-wise noise by the use of LayerNorm scales.
• Reported effectivity: >1.5× rollout speedups vs 16-bit LoRA and ~1.8× end-to-end vs QLoRA; >2× rollout throughput vs QLoRA on 14B/32B setups.
• Accuracy holds: Qwen2.5-7B reaches 90.8% on GSM8K and 77.4% on MATH500, matching full-parameter fine-tuning beneath the paper’s setup.
• NVFP4 is a hardware-optimized 4-bit floating format with two-level scaling (FP8 E4M3 block scalers + FP32 tensor scale), enabling setting pleasant Marlin-based kernels.

QeRL speeds up the RL rollout stage. It quantizes weights to NVFP4 and retains updates and logits in higher precision using LoRA. It experiences >1.5× rollout speedups and would possibly put together a 32B protection on a single H100-80GB GPU. It gives Adaptive Quantization Noise to make exploration a managed signal all through teaching. Outcomes are confirmed totally on math-reasoning duties using GRPO and DAPO. The optimistic facets depend upon NVFP4 kernel help equal to Marlin.

---

Check out the FULL CODES proper right here and Paper. Be joyful to check out our GitHub Internet web page for Tutorials, Codes and Notebooks. Moreover, be at liberty to watch us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you’ll have the ability to be a part of us on telegram as correctly.

🙌 Adjust to MARKTECHPOST: Add us as a most popular provide on Google.