Question About the Decoder-only LLM Module

[huazhi1024]

Hello,

Thank you very much for releasing the MOSS-Audio-Tokenizer code and for your great work.

I am currently studying the implementation and comparing it with the architecture described in the paper. I noticed that the paper mentions a Decoder-only LLM module**, but I could not identify the corresponding implementation in the released source code.

Could you kindly let me know which file or module in the repository corresponds to the Decoder-only LLM described in the paper? If this component is implemented under a different name or is integrated into another module, I would also greatly appreciate it if you could point me to the relevant part of the code.

Sorry if I have overlooked something, and thank you very much for your time and help!

Best regards,
A PHD student studying MOSS-Audio-Tokenizer

[gyt1145028706]

Hi, thanks for the kind words and for reading the paper so carefully!

You haven't overlooked anything — the decoder-only LLM is genuinely not part of this repository, and that is by design.

The decoder-only LLM is a training-time auxiliary module used for semantic alignment. During tokenizer training, it takes the latents after the residual quantizer and is trained to predict the text corresponding to the audio, which encourages the learned codes to retain semantic information. It is not part of the tokenizer's inference path at all — once training is done it is discarded, and it plays no role in encoding audio to codes or decoding codes back to waveform. This repository releases the tokenizer for inference, so the module isn't included here; it lives in our training codebase.

What you'll find here is therefore only the encoder → residual quantizer → decoder path. Concretely, in modeling_moss_audio_tokenizer.py, MossAudioTokenizerModel holds self.encoder, self.quantizer (MossAudioTokenizerResidualLFQ) and self.decoder, exposed through the public encode() / decode() methods.

Architecturally the auxiliary LLM is nothing exotic: it follows Qwen-Audio very closely. The only substantive difference is the audio front-end — where Qwen-Audio uses its own audio encoder, we swap in MOSS-Audio-Tokenizer's encoder + quantizer and feed the quantized latents into the decoder-only LLM as the audio embeddings. So if you want to reproduce that part, the Qwen-Audio recipe should transfer almost directly.

For the audio representation the LLM consumes, the relevant entry points are:

out = model.encode(input_values, return_dict=True)      # -> out.audio_codes  [num_quantizers, B, T]
latent = model.quantizer.decode_codes(out.audio_codes)  # -> continuous post-quantization latent [B, 768, T]

decode_codes() sums the dequantized vectors across quantizers and applies the output projection, which gives exactly the post-quantizer latent that was fed to the LLM during training.

Hope this clarifies things — happy to go into more detail on the training setup if it would help your work.

Best regards

[huazhi1024]

Thank you very much for the detailed explanation!