You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# We should be using disaggragate serving for GPTOSS model for best performance
2
+
- GPT-OSS model has 128/4 for 120b and 32/4 ratio of total_experts/experts_per_tok
3
+
- We use read all experts only once always strategy in prefill-only model
4
+
- And we treat weights activtions meaning read only chosen experts for decode-only model
5
+
6
+
# Prefill-only model
7
+
## Blocking default behviour when `prefill_only=True` in compile API
8
+
- NUM_Q_BLOCKS=<int> set number of Q blocks in attention
9
+
- NUM_FFN_BLOCKS=<int> set number of blocks in FFN
10
+
- ENABLE_OPT_SWA="0" or "1" to enable/disable optimized SWA. when enabled we will be using only valid KVs for given block in Attention reducing MACs
11
+
- prefix_caching is not supported with this mode
12
+
13
+
## Chunking pass `enable_chunking=True` and `prefill_only=True` in compile API
14
+
- Optimized SWA i.e. reading only valid KV as per diagonal attention mask is enabled for this version by default
15
+
- This model can be used for prefix_caching by passing `kv_cache_batch_size=<int>` in compile API
16
+
17
+
# Decode-only model
18
+
## Retain Sliding window length of KV for sliding window layers, default behavour when `prefill_seq_len=1` in compile API
19
+
- This reduces the amount of DDR used by the model
20
+
- CB is enabled for this version pass `continous_batching=True` in `from_pretrained` call and strictly pass `full_batch_size=<int>` and optinally `kv_cache_batch_size=<int>` if needed
21
+
## Full KV for sliding window layers pass `retain_full_kv=True` along with `prefill_seq_len=1` in compile API
22
+
- This uses higher DDR as we are retaining ctx_len KV even for sliding window layers but will be reading only sliding window len kv in attention
23
+
- CB is enabled for this version pass `continous_batching=True` in `from_pretrained` call and strictly pass `full_batch_size=<int>` and optinally `kv_cache_batch_size=<int>` if needed
24
+
- This is enabled for the usecase of multi-turn chat, where we will be running prefill-> decode and then use cache of prefill as well as decode combined to again run prefill, so we want to retain full KV for sliding window layers
25
+
26
+
27
+
NOTE:
28
+
* decode-only model currently fails compilation with `use_onnx_subfunctions=True` so avoid using it
29
+
* 120B model needs NPI, there are two versions of NPI one with and without subfunction both are uploaded here, pass it as `node_precision_info=<path to file>`
30
+
* It is advised to use `use_onnx_subfunctions=True` with prefill-only model, otherwise the compilation times are too high, with this the model is supposed to export and fail during compile as it needs assert sdk, so user is supposed to run this compilation manually by pasting the command printed in the error
0 commit comments