Warn when building gpt-oss with float16 I/O precision - #2444
Open
Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 1 commit into
Open
Warn when building gpt-oss with float16 I/O precision#2444Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 1 commit into
Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 1 commit into
Conversation
gpt-oss activations grow through the decoder and exceed the float16 range in the
final layers. Instrumenting the graph shows the max absolute activation climbing
from ~35 at layer 0 to ~45,344 at layer 34; layer 35 passes the 65,504 ceiling
and every logit comes back NaN. argmax over all-NaN returns index 0, so the model
emits one repeated token ("!") instead of text.
This is silent: the build succeeds and the model loads and runs at full speed.
Warn when io_dtype is float16, matching the existing Gemma2/Gemma3 warnings, and
point at --precision bf16 or --extra_options use_cuda_bf16=true.
Gopalakrishnan Nallasamy (GopalakrishnanN)
requested a review
from a team
as a code owner
August 21, 2026 02:43
Copilot started reviewing on behalf of
Gopalakrishnan Nallasamy (GopalakrishnanN)
August 21, 2026 02:44
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a runtime build-time warning in the Python model builder to help users avoid generating GPT-OSS ONNX models whose float16 I/O precision can overflow in late layers and produce NaN logits (leading to degenerate output).
Changes:
- Emit a targeted warning when building
GptOssForCausalLMwith float16 I/O precision. - Direct users toward using BF16 precision (and CUDA BF16 I/O for int4) to avoid NaN logits.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+504
to
+505
| if io_dtype == ir.DataType.FLOAT16: | ||
| print("WARNING: This model overflows the float16 range in its last layers, which produces NaN logits. Set `--precision bf16` or `--precision int4 --extra_options use_cuda_bf16=true`.") |
Contributor
|
I think gpt-oss-20b has no overflow. We have a foundry local gpt-oss-20b model with fp16 I/O. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gpt-oss activations grow through the decoder and exceed the float16 range in the final layers. Instrumenting the graph shows the max absolute activation climbing from ~35 at layer 0 to ~45,344 at layer 34; layer 35 passes the 65,504 ceiling and every logit comes back NaN. argmax over all-NaN returns index 0, so the model emits one repeated token ("!") instead of text.
This is silent: the build succeeds and the model loads and runs at full speed. Warn when io_dtype is float16, matching the existing Gemma2/Gemma3 warnings, and point at --precision bf16 or --extra_options use_cuda_bf16=true.
Should the builder instead default use_cuda_bf16=true for gpt-oss? A warning still lets users produce a model that only emits NaN, but defaulting would silently change the output dtype for anyone currently passing --precision int4 -e cuda.