Skip to content

fix(draft): embed Qwen3.6 SWA metadata in Q8 converter - #562

Open
Amidwestnoob wants to merge 1 commit into
Luce-Org:mainfrom
Amidwestnoob:fix/qwen36-q8-draft-swa-metadata
Open

fix(draft): embed Qwen3.6 SWA metadata in Q8 converter#562
Amidwestnoob wants to merge 1 commit into
Luce-Org:mainfrom
Amidwestnoob:fix/qwen36-q8-draft-swa-metadata

Conversation

@Amidwestnoob

@Amidwestnoob Amidwestnoob commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in --qwen36-swa profile to the Q8 draft converter
  • embed the Qwen3.6 draft's 2048-token sliding-window metadata and [SWA, SWA, SWA, SWA, full] layer pattern
  • preserve the current Qwen3.5 conversion output by default
  • add converter/metadata regression coverage and update the drafter recipe

Why

The draft GGUF loader already consumes these metadata fields, but quantize_draft_q8.py does not emit them. Qwen3.6 Q8 draft artifacts therefore require the external DFLASH27B_DRAFT_SWA=2048 runtime override to select the correct attention layout.

In a no-override negative control, the existing keyless artifact loaded without SWA metadata and matched only 6 of 12 deterministic outputs from the correctly configured reference. Prose and long-context outputs diverged.

This change makes newly converted Qwen3.6 Q8 drafts self-describing. It does not claim a new speedup; it reproduces the existing correct override-based configuration without relying on process environment.

Validation

  • python3 server/tests/test_quantize_draft_q8.py
    • 6 tests passed
    • covers default behavior, exact key names/types/values, CLI parsing, full converter CLI output, and GGUF round-trip
  • cmake --build server/build --target dflash_server -j2
    • built successfully
  • ctest --test-dir server/build --output-on-failure -R "server_unit" --no-tests=error
    • 1/1 passed
  • git diff --check
    • clean

Real Q8 artifact validation without DFLASH27B_DRAFT_SWA:

  • loader: SWA layers: 4/5 (window=2048)
  • HTTP success: 12/12
  • output parity with env-configured reference: 12/12
  • acceptance-rate parity: 12/12 exact
  • peak VRAM: 20,760 MiB on both runs
  • mean decode: 113.85 vs 114.70 tok/s (-0.74%, within run-to-run noise)

The metadata-added validation artifact preserved all 58 tensor payloads byte-for-byte; only the two intended metadata fields were added.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant