Metal SSD-streaming cache-pressure vector test fails at short_code_completion

[DanteCpp]

Summary

./ds4_test --metal-ssd-streaming-cache-pressure fails with a selected-token mismatch in the short_code_completion vector case.

Environment

• Machine: Apple M2 Max
• RAM: 32 GiB
• macOS: 26.5.1
• Backend: Metal SSD streaming
• Model: DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf

Command

make ds4_test

DS4_TEST_MODEL=gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  ./ds4_test --metal-ssd-streaming-cache-pressure

Result

metal-ssd-streaming-cache-pressure:
ds4-test: Metal SSD streaming cache-pressure repro (16GiB cache, layer-batched decode, short_code_completion)
ds4: Metal device Apple M2 Max, 32.00 GiB RAM
ds4: metal SSD streaming cache budget 16.00 GiB / 6.75 MiB per expert = 2427 experts
ds4: Metal SSD streaming mode enabled; full model residency and warmup are skipped
ds4-test: vector short_code_completion
ds4-test: vector short_code_completion step 1 selected token mismatch
tests/ds4_test.c:873: assertion failed: false
metal-ssd-streaming-cache-pressure: ERR
ds4 tests: 1 failure(s)

Expected

The cache-pressure vector test should pass, or the fixture/test should be updated if this model/backend pairing
is no longer expected to match it.

Notes

This test forces the Metal SSD-streaming layer-batched decode path with a 16 GiB routed expert cache and checks
official-vector parity for short_code_completion. The failure means local greedy selection diverges from the
expected vector token at step 1.

[andreaborio]

I tried to reproduce this on current upstream/main (80ebbc3) with the same GGUF and command.

Machine here is different: Apple M5 Pro, 64 GB RAM, Metal backend.

DS4_TEST_MODEL=gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  ./ds4_test --metal-ssd-streaming-cache-pressure

This passes for me:

ds4-test: vector short_code_completion
metal-ssd-streaming-cache-pressure: OK
ds4 tests: ok

I also tried the same short_code_completion vector with smaller SSD-streaming caches (8GB, 4GB, 2GB, 1GB) and could not reproduce the mismatch.

So this may be M2 Max / 32 GB specific, or timing / memory-pressure dependent rather than reproducing on all Metal setups.

[DanteCpp]

I am also at upstream/main (80ebbc3), dumping the logits and logprobs with

./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 16GB \
  --nothink -sys "" --temp 0 -n 4 --ctx 4096 \
  --prompt-file tests/test-vectors/prompts/short_code_completion.txt \
  --dump-logprobs /tmp/scc.local.json --logprobs-top-k 3

I get:

{
  "source":"ds4",
  "prompt_tokens":27,
  "ctx":4096,
  "top_k":3,
  "steps":[
    {"step":0,"selected":{"id":9854,"text":"```","bytes":[96,96,96]},"top_logprobs":[{"token":{"id":9854,"text":"```","bytes":[96,96,96]},"logit":35.8842468,"logprob":-0.182371974},{"token":{"id":671,"text":"The","bytes":[84,104,101]},"logit":33.526844,"logprob":-2.53977489},{"token":{"id":28986,"text":"```\n","bytes":[96,96,96,10]},"logit":32.9112854,"logprob":-3.15533352}]},
    {"step":1,"selected":{"id":37,"text":"C","bytes":[67]},"top_logprobs":[{"token":{"id":37,"text":"C","bytes":[67]},"logit":35.5210953,"logprob":-0.198094547},{"token":{"id":69,"text":"c","bytes":[99]},"logit":33.999733,"logprob":-1.71945691},{"token":{"id":345,"text":" C","bytes":[32,67]},"logit":27.9644547,"logprob":-7.75473499}]},
    {"step":2,"selected":{"id":201,"text":"\n","bytes":[10]},"top_logprobs":[{"token":{"id":201,"text":"\n","bytes":[10]},"logit":32.2727356,"logprob":-0.0011137689},{"token":{"id":271,"text":"\n\n","bytes":[10,10]},"logit":25.4067802,"logprob":-6.86706924},{"token":{"id":29406,"text":"\n    \n","bytes":[10,32,32,32,32,10]},"logit":21.1578026,"logprob":-11.1160469}]},
    {"step":3,"selected":{"id":3916,"text":"return","bytes":[114,101,116,117,114,110]},"top_logprobs":[{"token":{"id":3916,"text":"return","bytes":[114,101,116,117,114,110]},"logit":31.4405403,"logprob":-0.0249947291},{"token":{"id":682,"text":")\n","bytes":[41,10]},"logit":26.7476768,"logprob":-4.71785831},{"token":{"id":989,"text":",\n","bytes":[44,10]},"logit":25.4342155,"logprob":-6.03131962}]}
  ]
}

In step 1, I am getting 'C' instead of 'c' which would be in accordance with the gold vectors. The second most likely token in the distribution is indeed 'c' with a logit difference of

>>> 35.5210953-33.999733
1.5213622999999998
>>>

I do not know, can it be numerical error?
1.5 nats looks too big of a gap to be just numerical error...

Best,
Dante