Update to llama.cpp 2026-01-01

[avion23]

Bindings were 5 months outdated, preventing newer model architectures from loading.

Updates bindings to llama.cpp commit be47fb92 (2026-01-01).

Removed

• 14 llama_kv_self_* functions (use llama_memory_* API)
• llama_sampler_init_softmax()

Added

Enums:

• LLAMA_ROPE_TYPE_IMROPE
• llama_flash_attn_type
• llama_params_fit_status
• llama_model_meta_key

Struct fields:

• llama_model_params: no_host, no_alloc
• llama_context_params: flash_attn_type (replaced flash_attn bool)

Functions:
llama_max_tensor_buft_overrides, llama_n_ctx_seq, llama_model_n_embd_inp, llama_model_is_hybrid, llama_flash_attn_type_name, llama_model_meta_key_str, llama_adapter_meta_* (5 functions), llama_log_get, llama_log_set, llama_memory_breakdown_print

Breaking Changes

flash_attn parameter:

# Old
params.flash_attn = True
# New
params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED

KV cache API:

# Old
llama_kv_self_clear(ctx)
# New
llama_memory_clear(mem, data=True)

Other

• Added ggml_log_callback typedef
• Fixed LLAVA/mtmd build (set LLAMA_INSTALL_VERSION before subdirectory include)
• Version 0.3.16 → 0.4.0

Tested: macOS ARM64 Metal, Python 3.14, Nemotron-3-Nano-30B

[avion23]

Tested on macos using CMAKE_ARGS="-DGGML_METAL=on" pip3.14 install --force-reinstall --no-cache-dir "llama-cpp-python @ git+https://github.com/avion23/llama-cpp-python.git@update-llama-cpp-2026-01" --break-system-packages

[dhdaines]

This will need at least one more (very important) change as the layout of mtmd_context_params has changed. It should be updated in mtmd_cpp.py to this:

class mtmd_context_params(Structure):
    _fields_ = [
        ("use_gpu", c_bool),
        ("print_timings", c_bool),
        ("n_threads", c_int),
        ("image_marker", c_char_p),
        ("media_marker", c_char_p),
        ("flash_attn_type", c_int),
        ("warmup", c_bool),
        ("image_min_tokens", c_int),
        ("image_max_tokens", c_int),
    ]

[dhdaines]

More changes needed as the layout of llama_context_params has also changed... a new field flash_attn_type has been added after attention_type.

[dhdaines]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

[dhdaines]

Stray ellipsis operator (which does nothing, but still)

[dhdaines]

Sorry! The issue here isn't the ellipsis operator, it's that at some point there were two of them - you shouldn't change this to pass because that implies that the function returns None, which will cause type checking to fail.

[avion23]

thank you, and I understand now. done

[avion23]

@dhdaines thanks for the review, I need some time to incorporate your comments, setting to draft in the meantime

[avion23]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

I think I have fixed this, could you check?

[dhdaines]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

I think I have fixed this, could you check?

Yes, this looks to me like a good way to handle it! We can see what the maintainer @abetlen thinks though...

[avion23]

My intention was to sweep in like a hero and save the day. Didn't work as planned :/

I've rewritten the PR, much less whitespace noise, and cleaner. All review comments are incorporated.

[oss-roettger]

Thank you so much avion23 for your efforts to update the python bindings to a recent llama-cpp version!

I'm trying to use them in a Jupyter notebook (in Docker) on a Nvidia 5090 GPU. Although the latest locally build llama-cli version is running in that same environment (see attached llama-cli.txt) and the above considered problems are gone, the freshly build bindings produce a kernel crash when loading models to GPU (after loading weights to GPU, maybe a context issue, see attached build.txt).

I'm pretty sure, it can be my mistake when installing your branch for GPU support:
!CMAKE_ARGS="-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+https://github.com/avion23/llama-cpp-python@update-llama-cpp-2026-01

Any ideas what I did wrong?!

Edit (New Findings): The above GPU build works with n_gpu_layers=0 (CPU only). This narrows down the problem to context handling in the GPU context code path.
Edit2: Very,very strange: after switching back to n_gpu_layers=100 (from n_gpu_layers=0) I was able to load and successfully run the new Nemotron-Nano-3-30B-A3B-Q4_K_M.gguf and Ling-mini-2.0.Q4_K_M.gguf models on GPU (on the same build that crashed the kernel always(!) while loading the models before. Could it be, that there is any context initialization code which is run in CPU mode only but also important for GPU mode?!

[oss-roettger]

@abetlen Thank you for your work! Please keep this repo alive and merge avion23's updates into the main branch.
These are working now on my RTX 5090 Cuda environment. It's not only that recent llama.cpp supports additional models, but it also offers significant performance gains (e.g. GPT OSS 20 NVFP4 +50% more tokens/s compared to the Oktober versions in your repo wheels).

[avion23]

@oss-roettger thank you for all the testing. I found a bug with flash_attn which caused your error. Could you retest?

[oss-roettger]

@avion23 once again respect for your dedication. I have tested the new version after building it with

!CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+https://github.com/avion23/llama-cpp-python@update-llama-cpp-2026-01

Good news first:
The update runs out of the box (with all values for flash_attn in the constructor flash_attn=None/True/False/not present)

But: I think I have discovered an additional issue (with & without your latest update; on GPU & CPU):
https://huggingface.co/ggml-org/Nemotron-Nano-3-30B-A3B-GGUF produces a KV cache issue on the second dialog turn. I guess there is a cache initialization parameter missing in the python bindings, since the llama-cli command of the same llama build (=same libllama.so) works on multi-turn dialogs with the same Nemotron model.
(see Llama_test.txt for minimum code to reproduce the error)

Edit: Log added:
log.txt

[avion23]

Thank you, Could You attach the log as before? Then it'll be quicker for me and you get a nice solution. Let's just pray this gets merged, i don't want to maintain my fork.

…

On Mon 12. Jan 2026 at 18:29, oss-roettger ***@***.***> wrote: *oss-roettger* left a comment (abetlen/llama-cpp-python#2108) <#2108 (comment)> @avion23 <https://github.com/avion23> once again respect for your dedication. I have tested the new version after building it with !CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+ ***@***.*** Good news first: The update runs out of the box (with all values for *flash_attn* in the constructor flash_attn=None/True/False/not present) But: I think I have discovered an *additional issue* (with & without your latest update; on GPU & CPU): https://huggingface.co/ggml-org/Nemotron-Nano-3-30B-A3B-GGUF produces a KV cache issue on the second dialog turn. I guess there is a cache initialization parameter missing in the python bindings, since the llama-cli command of the same llama build (=same libllama.so) works on multi-turn dialogs with the same Nemotron model. (see Llama_test.txt <https://github.com/user-attachments/files/24562603/Llama_test.txt> for minimum code to reproduce the error) — Reply to this email directly, view it on GitHub <#2108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTGWLHPMSGS7UVCWIRBTY34GOAP5AVCNFSM6AAAAACQO5EZGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZYGEYDCNRRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[oss-roettger]

👍 Model loading log.txt attached above. Error log in Llama_test.txt