ncnn2table

[DRTAXI]

expectation | 诉求 | 期待する

1. speed
2. precision

model | 模型 | モデル

1. model.param and model.bin
   mobileclip2-s0 text model

detail | 详细描述 | 詳細な説明

ncnn2table only support Convolution/ConvolutionDepthWise/InnerProduct now. How should I handle the calibration of transformer quantization?

[dosubot]

Hi @DRTAXI! I'm Dosu and I’m helping the ncnn team.

You're right that ncnn2table currently only supports Convolution, ConvolutionDepthWise, and InnerProduct for calibration [1]. However, ncnn does have INT8 inference support for several transformer-related layers — the gap is in the automatic calibration table generation.

What already has int8 inference support:

• MultiHeadAttention — added in PR #5733
• GEMM/MatMul — added in PR #5706
• SDPA (Scaled Dot Product Attention) — added in PR #6397
• Embed layer [2]

Recommended workaround — mixed precision quantization:

Since ncnn2table can't generate calibration data for transformer layers, the practical approach is selective quantization:

1. Run ncnn2table normally to generate a calibration table for Conv/InnerProduct layers.
2. Leave transformer-specific layers (attention, GEMM) in float32 by not providing scale entries for them — ncnn2int8 will automatically skip layers without scale parameters [3].
3. Alternatively, you can comment out specific layer entries in the table to force them to float32 [4]:

   #layer_name_param_0 156.639840536  // stays float32
   

For a text-only model like MobileCLIP's text encoder, most of the compute is in attention/GEMM rather than convolutions, so the benefit from ncnn2table's supported layers may be limited. You might want to watch ongoing development — there's active work on LLM block quantization with int4/6/8 (PR #6439) and LayerNorm INT8 (PR #4132) which could improve transformer quantization coverage in the future.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.