Repository navigation
Preserve FP32 inputs through LayerNormLinear normalization - #3916
mkhona-nvidia wants to merge 1 commit into
Conversation
Keep FP32 normalization inputs and input gradients independent of the lower-precision linear activation dtype. Add mixed native forward kernel registrations, explicit cuDNN gradient dtypes, and standalone numerical regressions including sub-ULP input variation. Use native centered LayerNorm statistics for FP32 inputs while retaining explicitly requested weight-dtype zero-centered gamma arithmetic. Signed-off-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
[Medium impact] The change reaches LayerNormLinear and shared normalization paths for mixed-dtype FP32 inputs. ONNX export must match the new mixed-dtype behavior before merging. Findings
SummaryThis PR keeps FP32 inputs through
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[FP32 input] --> B[Normalize without rounding input]
B --> C[Cast to activation dtype]
C --> D[Optional FP8 quantization]
D --> E[GEMM]
C --> F[Optional normalization output]
E --> G[Normalization backward]
G --> H[FP32 input gradient]
Reviews (1) · Last reviewed commit: "Preserve FP32 inputs through LayerNormLi..." · Reviewed by Greptile |
| if inp.dtype == torch.float32 and self.layer_norm_weight.dtype in ( | ||
| torch.float16, | ||
| torch.bfloat16, | ||
| ): | ||
| # The base method validates parameter dtypes and gives autocast priority. | ||
| super().set_activation_dtype(self.layer_norm_weight) |
There was a problem hiding this comment.
With FP32 inputs and FP16 or BF16 parameters, normal execution now uses the parameter dtype for GEMM. But onnx_forward still uses inp.dtype for the normalization output, weights, bias, and GEMM. Export therefore produces FP32 outputs and skips the low-precision rounding used by normal execution, changing the results.
Update onnx_forward to keep normalization inputs in FP32 while using the selected activation dtype afterward. Add an export comparison for this configuration.
LayerNormLinearcurrently rejects FP32 inputs with FP16/BF16 parameters outside autocast and rounds FP32 inputs before normalization under autocast. FP32 reductions cannot recover variation already lost in that cast.Preserve FP32 normalization inputs and input gradients while keeping GEMM activations and optional normalization outputs in the parameter/autocast dtype.
Changes:
Validation: pre-commit, changed-file C++ lint and license checks passed. A clean GB300 build and focused numerical tests are running; results pending. JAX, ONNX export, userbuffers and the full L0 matrix have not been run.
Draft pending author review.