Skip to content

About

Model management service for the Agent Platform: backend-abstracted (ollama, kserve) inventory, pull, load/unload, delete and kagent wiring via REST and MCP

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

310 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

model-manager

OAuth client registration: CIMD is enabled by default. DCR is deprecated by MCP 2026-07-28 and remains a compatibility fallback. See registration, consent, egress and TLS configuration.

CircleCI

Model management service for the Giant Swarm Agent Platform. One API over serving backends — one or several per installation, in one process:

Backend Where What it proxies
ollama laptop / agentlab installs (host Ollama through the kind docker-network gateway) /api/tags, /api/ps, streamed /api/pull, /api/delete, keep_alive load/unload
kserve GPU installs (KServe + the platform's modelServing component) LLMInferenceServices (or classic InferenceServices) composed from serving presets, per-node HF cache inventory, pre-warm download Jobs with progress, Hugging Face Hub search, node fit checks
lemonade AMD Ryzen AI laptops / workstations running Lemonade Server on the host (FastFlowLM on the NPU, llama.cpp on GPU / CPU) /api/v1/health (loaded models), /api/v1/models, streamed /api/v1/pull, /api/v1/load, /api/v1/unload, /api/v1/delete, /api/v1/system-info
lmstudio desktop installs running LM Studio on the host (llama.cpp on GPU / CPU, MLX on Apple silicon) /api/v1/models (the library and its loaded instances), /api/v1/models/download + its status job, /api/v1/models/load, /api/v1/models/unload — no delete

One model-manager runs every backend the host has (--backends=ollama,lmstudio, chart value backends: [ollama, lmstudio]): the API reports each backend and its explicit capability flags (GET /api/v1/backends), every model, job and node names its backend, every request may name one, and an unqualified model reference is resolved to the one backend that holds it (409 conflict when several do). GET /api/v1/backend describes the first, default backend and names the others — the one-backend form clients of a single backend keep using. Clients render only what a backend supports instead of switching on its name. Pulled or loaded models are wired into kagent automatically as ModelConfigs (native keyless Ollama provider for the ollama backend; OpenAI provider with the caller's token forwarded (apiKeyPassthrough) for a kserve model routed on the models Gateway; OpenAI provider plus a placeholder API-key Secret — against the workload URL for a kserve model on its in-cluster Service, and against Lemonade's /api/v1 for lemonade; a kserve model's ModelConfig is created by the load call, before the model is ready), so agents can use them without manual steps. They are written in the API group and version the cluster serves (api.kagent.dev/v1alpha3 from kagent 1.3 on, kagent.dev/v1alpha3 before it; discovered at start-up, re-discovered every minute and when a call fails NotFound, so a running process follows a kagent upgrade), and a ModelConfig is ready once kagent has accepted it and resolved its Secret (conditions Accepted and ResolvedRefs).

The same operations are exposed twice from one process:

  • REST/JSON under /api/v1 for the portal backend — contract in api/openapi.yaml, also served at /api/v1/openapi.yaml.
  • MCP (streamable HTTP, /mcp) for muster — tools get_info, get_backend, list_backends, list_models, get_model, list_loaded_models, pull_model, load_model, unload_model, delete_model, wire_model, unwire_model, list_jobs, get_job, cancel_job, plus the kserve capabilities list_presets, search_models, check_fit, list_nodes (through muster: x_<server>_<tool>); every tool takes an optional backend. Every tool's description and input schema as a model reads them: docs/tools.md.

Part of Model management in the Agent Platform; decision records: Model Manager PDR, the Ollama-backend ADR and the Lemonade-backend ADR in the team's decision log.

API at a glance

Operation REST MCP tool
This server's build (release version, commit, build time), its tools, the configured backends and the wiring target — get_info
Every backend's identity, capabilities + load semantics (the first is the default) GET /api/v1/backends list_backends
Register / remove a backend at runtime (a backend document, docs/backends.md; dryRun, mode: apply) — add_backend, remove_backend
One backend (the named one, else the default) plus the names of all GET /api/v1/backend[?backend=] get_backend
Downloaded models (with loaded state + ModelConfig), of one or every backend GET /api/v1/models[?backend=], GET /api/v1/models/{name}[?backend=] list_models, get_model
Loaded / running models (kserve: each with its modelConfig; a served model model-manager manages that has none is wired by the read, wiring: {wired, reason: "wired on read", modelConfig}) GET /api/v1/loaded[?backend=] list_loaded_models
Pull / import (returns a job; on backend, else the default backend) POST /api/v1/models/pull {"model","backend?","wire?","preset?","node?"} pull_model
Job progress GET /api/v1/jobs[?backend=], GET /api/v1/jobs/{id}, DELETE /api/v1/jobs/{id} list_jobs, get_job, cancel_job
Load / unload (kserve: the load answers fit, running and wiring — the ModelConfig created in the same call, apiKeyPassthrough for a model routed on the models Gateway — before the model is ready; the unload deletes the serving object and unwires within the call — only a ModelConfig this model-manager created, any other is reported as modelConfigLeft — never waiting for a cache scan, and answers inventory: {refreshing, reason?} — the cache rescanned in the background, or why it cannot be) POST /api/v1/models/load {"model","backend?","keepAlive?","tracing?"}, POST /api/v1/models/unload {"model","backend?"} load_model, unload_model
Delete (unwires by default) DELETE /api/v1/models/{name}[?unwire=false][&backend=] delete_model
Every write tool takes dryRun (MCP): it answers the plan and changes nothing — pull_model whether the model is there, the kserve fit verdict and the ModelConfig it would wire; load_model the loaded state, the keep-alive, the kserve fit verdict and serving object (manifests) or alreadyServing, the ModelConfig the auto-wire would ensure (wiring); unload_model the kserve serving objects it would delete and the ModelConfig it would unwire; delete_model the loaded state and the ModelConfig it would unwire; cancel_job the job — pull_model, load_model, unload_model, delete_model, cancel_job
Wire / unwire to kagent (apiKeyPassthrough or apiKeySecret+apiKeySecretKey override the backend's API-key shape; both together are refused; MCP dryRun: true returns the manifests it would write or delete, writing nothing) POST /api/v1/models/wire {"model","backend?","apiKeyPassthrough?","apiKeySecret?","apiKeySecretKey?"}, POST /api/v1/models/unwire {"model","backend?"} wire_model, unwire_model
Serving presets (kserve) GET /api/v1/presets[?backend=] list_presets
Hub search (kserve) GET /api/v1/search?q=…&limit=…[&backend=] search_models
Fit check (kserve) POST /api/v1/models/fit-check {"model" or "preset","backend?","node?"} check_fit
Node budgets + reservations (+ caches on kserve) GET /api/v1/nodes[?backend=] list_nodes
Health GET /healthz, GET /readyz —

Model references may contain / and : (smollm2:135m, hf.co/org/repo:Q4_K_M, Qwen/Qwen3-14B); path parameters capture the rest of the path. Errors are {"error":{"code":"not_found|invalid_request|unsupported|conflict|gitops_owned|auth_required|repository_unavailable|forbidden|does_not_fit|backend_error","message":"…"}}; unsupported (501) means the matching capability flag is false, does_not_fit (412) that the kserve fit check refused a pull or load, conflict (409) also that an unqualified reference exists on several backends — repeat the request with backend. gitops_owned (409) means the object the write would change or delete is applied by Flux from git — it carries kustomize.toolkit.fluxcd.io/name or helm.toolkit.fluxcd.io/name, whatever its app.kubernetes.io/managed-by says — and a live change would be reverted on the next reconciliation; the message names the Flux object. Nothing was written. A Kustomization is held to its status.inventory: an object it no longer lists — its file removed from git while the Kustomization does not prune, which leaves the object on the cluster with its labels — or whose Kustomization is gone is applied by nothing, and the unload and unwire paths remove it live. A ModelConfig reports the owner that applies it as gitops (list_model_configs, the inventory held the same way: a leftover reports none), and the wiring reconciler and the unload, unwire and delete paths leave such a ModelConfig alone. repository_unavailable (412) means a mode: commit call targets a repository model-manager's GitHub App is not installed on (see Commit mode); the message names the repository and the App, and nothing was written. forbidden (403) means the cluster refused the caller: model-manager acts as the caller, so their own Kubernetes RBAC decides; retrying does not help, and backend_error (502) stays for a backend that failed.

model-manager only deletes what it created. Every model-manager writing ModelConfigs into a kagent namespace has an instance name (--instance, MODEL_MANAGER_INSTANCE; the chart sets <release namespace>-<release name>, a process outside the chart defaults to its host name — give each process on one host its own). A ModelConfig it creates carries the label model-manager.giantswarm.io/instance: <instance>, written once at creation; app.kubernetes.io/managed-by: model-manager alone is shared by every instance and copied with a manifest, so it never grants a deletion. The unload, unwire, delete and remove_backend paths delete only a ModelConfig carrying their own instance label, and unwire_model also one without any instance label (below). Any other is left in place: the unload answers modelConfigLeft: {namespace, name, createdBy?, message} (its dry run wiring.left), remove_backend leaves another instance's and refuses while one without the label remains (conflict naming it: the backend would go and the ModelConfig would point at nothing), and unwire_model and delete_model with unwire answer conflict (409) with that message, nothing deleted. A wire onto a ModelConfig another instance created answers conflict naming it, nothing written; one with managed-by: model-manager but no instance label (written before the label existed, or by hand) is adopted — its spec written, never marked as created here — and only unwire_model removes it: once no Agent, AgentTemplate or Harness in its namespace references it (its dry run naming it, mode commit where Flux applies it from git), whether or not its backend is still registered; while one does, the answer is conflict naming them (referencedBy). A ModelConfig without managed-by: model-manager is never written or deleted. ModelConfigRef.createdBy reports the instance label, and get_info's wiring.instance this process's name.

Several backends in one process. Every Model, LoadedModel, Job, NodeInfo, Preset, FitResult and ModelConfigRef carries backend. Reads without ?backend= aggregate every backend; a backend that fails to answer is reported in the response's errors ({"lemonade": "connection refused"}) while the others' items are returned — 502 only when every backend failed. An unknown or unconfigured backend name answers 400. kagent ModelConfigs are one per (backend, model): the label model-manager.giantswarm.io/backend plus the model annotation identify them, and the same reference on two backends is two ModelConfigs (the second named <derived>-<backend>).

On kserve, pull and load accept preset and node; a model is wired into kagent by the load call itself — before it is ready, at the address it will answer on; the answer's wiring names the ModelConfig — and unwired on unload, never after a pull, since a cached model has no endpoint. A load job follows the model to readiness and refreshes the ModelConfig from the address KServe published; the job lives in the process, so a restart loses only its progress entry. A served model model-manager manages that has no ModelConfig is wired by the caller's next list_loaded_models / GET /api/v1/loaded, whose entry says so (wiring.reason: "wired on read").

Capability flags: pull, pullProgress, delete, load, unload, loadedModels, wire (Kubernetes access present), presets, fitCheck, nodeInventory, search (presets, fitCheck and search are kserve concerns and false on ollama and lemonade).

GET /api/v1/backend reports two addresses: endpoint, the backend as model-manager dials it, and agentEndpoint, the backend as agent pods dial it — the host written into ModelConfigs (ollama: --ollama-agent-host, defaulting to the endpoint; lemonade: --lemonade-agent-host plus /api/v1, the OpenAI-compatible base URL the ModelConfigs carry). A client that matches ModelConfigs it did not create to models by hostname (the portal's "Used by") compares against agentEndpoint. kserve reports the models Gateway's origin every served model is routed under, and omits it without the Gateway, when every served model has its own address (running.endpoint, modelConfig.endpoint).

On ollama, GET /api/v1/nodes reports the proxied host as one node so a laptop install has capacity data too. Ollama's API has no capacity endpoint, so budgetBytes is MemTotal of /proc/meminfo as the model-manager pod sees it (budgetSource: host-meminfo) — on a unified-memory machine GPU memory is system memory, so that is the budget that matters; reservedBytes is the sum of size over Ollama's /api/ps (weights plus KV cache for the loaded context, which is why a 500 MiB download can reserve 5 GB), accelerated says whether any loaded model has memory on an accelerator (size_vram > 0), and each model's own share stays in running.vramBytes. There is no gpuCount, gpuProduct or cache — Ollama does not expose the accelerator, and its model store is not a node cache. Caveat: the pod reads the kernel it runs on. On kind or any install sharing the machine's kernel that is the machine's RAM; under a VM-backed container runtime (Docker Desktop, Podman machine) it is the VM's, and for an Ollama on another machine it says nothing about that host. For those installs the chart value ollama.memoryBudgetGiB (--ollama-memory-budget-gib, MODEL_MANAGER_OLLAMA_MEMORY_BUDGET_GIB; GiB, decimals allowed) sets the budget instead: budgetBytes is that figure with budgetSource: override, message says so, and allocatableMemoryBytes stays the pod's MemTotal — the ollama counterpart of the kserve node annotation model-manager.giantswarm.io/memory-budget-gib. A value that is not a positive number of GiB is ignored and named in message; the budget then comes from /proc/meminfo as before.

On lemonade the host node comes from Lemonade's own GET /api/v1/system-info instead (budgetSource: system-info, the accelerators Lemonade enumerates as gpuCount / gpuProduct, the model store as cache) — see The lemonade backend.

Load semantics: on-demand load and keep-alive

GET /api/v1/backend also carries a loading block so a client can word a not-loaded model correctly without keying off the backend name:

"loading": { "onDemand": true, "idleEviction": true, "keepAliveDefault": "5m", "keepAliveScope": "request" }
  • ollama — onDemand: true: Ollama loads a model on the first /api/chat (or /api/generate) that names it, so an agent whose ModelConfig points at a not-loaded model works and its first turn pays the cold start. "Not loaded" is idle, not broken. idleEviction: true: Ollama evicts a model when its keep-alive runs out. keepAliveScope: request: the keep-alive is set per request — Ollama's scheduler takes each request's keep_alive, else the server's OLLAMA_KEEP_ALIVE (5m unless set), and re-arms the runner's timer on every hit. kagent's Ollama provider sends no keep_alive, so every agent turn re-arms the server default. POST /api/v1/models/load (a generate call with keep_alive, default --default-keep-alive / MODEL_MANAGER_DEFAULT_KEEP_ALIVE, reported as keepAliveDefault) therefore only pre-warms: it answers with the loaded model and running.expiresAt — the deadline Ollama reports right then — and the next agent request resets the timer to the server default again, even after a load with -1.
  • kserve — onDemand: false, idleEviction: false, no keep-alive fields: a stopped LLMInferenceService does not come back on request, agents on its ModelConfig fail at their first turn, and a running one stays until unloaded.
  • lemonade — onDemand: true: Lemonade loads a model on the first completion naming it (a few seconds for a 4B model on the NPU), so a not-loaded model is idle, not broken. idleEviction: false, no keep-alive fields: nothing evicts an idle model, but a loaded model gives way when another model of its type is requested and the slot is taken (max_loaded_models, one per type by default; least recently used first) unless it was loaded with keepAlive: "-1", which pins it.
  • lmstudio — onDemand: true: with LM Studio's just-in-time loading on (its default) the first completion naming a downloaded model loads it, so a not-loaded model is idle, not broken. idleEviction: false, no keep-alive fields: POST /api/v1/models/load is an explicit load, which LM Studio treats as manual — no idle TTL, and Auto-Evict leaves "non-JIT loaded models" alone — so a model loaded through model-manager stays resident until something unloads it. The endpoint accepts no ttl, which is why a keep-alive has nothing to map onto. The two paths differ, and it matters for agents: a model LM Studio JIT-loaded (what an agent turn does to a not-loaded model) does get its default idle TTL — 60 minutes — and is subject to Auto-Evict, so it can be gone later, while one loaded through this backend will not be. loading describes the backend's own loads. Turning just-in-time loading off in LM Studio is what breaks an agent on a not-loaded model — the request then fails instead of waiting.

The knob that changes what agents experience on Ollama is host-side, not a model-manager flag: set OLLAMA_KEEP_ALIVE=30m (or -1 for never) in the Ollama service environment — the systemd unit's Environment= on Linux — and restart Ollama. keepAliveDefault is model-manager's default for its own load requests; the host's OLLAMA_KEEP_ALIVE is not observable through the API, which is why the block does not claim to report it.

The context window is the other thing agents feel on Ollama, and model-manager sets it. A request without num_ctx runs at a default Ollama derives from the host's VRAM: 4,096 tokens below 24 GiB (every CPU-only host), 32,768 up to 48 GiB, 262,144 above. A longer prompt is not refused. Ollama keeps its first tokens and its tail and drops the middle, so an agent turn loses its system prompt and tool schemas, and the only trace is a truncating input prompt warning in the server's log. Every Ollama ModelConfig model-manager creates or re-wires therefore carries spec.ollama.options.num_ctx. The value is the chart value ollama.contextLength (--ollama-context-length, MODEL_MANAGER_OLLAMA_CONTEXT_LENGTH, default 32768), capped at the model's own context length. It is reported as modelConfig.contextLength, next to the model's own contextLength. Loads pre-warm at the same value, since Ollama reloads a model when a request's num_ctx differs. With auto-wiring, the reconciler brings a ModelConfig written under another value, or before the setting existed, to the configured one. The window costs host memory: Ollama reserves the KV cache for all of it when it loads the model. A ModelConfig written by hand gets none of this and depends on the host's OLLAMA_CONTEXT_LENGTH.

Thinking is the third. A model with Ollama's thinking capability (Qwen3.5, Qwen3, DeepSeek-R1, Granite 4.2, ...) reasons before every answer unless the chat request says think: false: 1.5–2k tokens per model call, minutes on a CPU. A small one such as qwen3.5:2b can put its whole answer into the thinking, which the agent runtime does not read, so the agent replies with nothing. The ModelConfig of such a model therefore carries spec.ollama.think, the chart value ollama.think (--ollama-think, MODEL_MANAGER_OLLAMA_THINK): false by default, true to have the model reason first, or empty to write none and leave Ollama's default. A model without the capability gets none, since Ollama answers true for it with a 400. The value is reported as modelConfig.think. kagent's ModelConfig has the field from kagent 1.0.3 on; model-manager reads the ModelConfig schema the apiserver serves (its OpenAPI v3, refreshed every minute) and writes think only where it is there, since the apiserver prunes a field its CRD lacks. With auto-wiring, the reconciler brings a ModelConfig whose think differs from the configured one to it, in the same comparison that covers the context window.

Backends registered at runtime

model-manager starts with no backend (chart default backend: "", backends: []) and gets its backends at runtime: add_backend writes a backend document — a ConfigMap in model-manager's namespace labelled agent-platform.giantswarm.io/model-backend=true, named model-backend-<name> (the backend's name, default its kind), key backend.yaml, kind: ModelBackend — as the caller; model-manager watches the label, enforces the document's schema when it reads it (an invalid document is reported under invalid in list_backends with the failing field, and not loaded) and registers the backend without a restart. remove_backend drops the backend's ModelConfigs and the document, or refuses while a ModelConfig of the backend it cannot delete would be left pointing at nothing. One backend per kind, and for kserve one per serving cluster (kserve, kserve-<cluster>); list_backends reports every backend with source: static | person | cluster-manager, and a kserve document carries the target cluster ({cluster, organization, apiServer, caBundle, servingNamespace} — never credentials) the backend acts on, reported as target on the backend, its models, nodes and presets. Static --backends values keep working as source: static; a document naming a static kind is refused. With no backend at all list_backends and get_info answer an empty list with backendsReason (the no_backend wording, so a caller tells no backend from a read that missed one), and every backend-scoped call answers no_backend with the fix — except check_fit of a serving preset the platform chart publishes where model-manager runs, which answers what the preset declares with verdict: unverified, so a GPU pool is sized before it, and the backend, exists. The contract — the label, keys, schema, tools, RBAC and what cluster-manager writes — is docs/backends.md.

The lemonade backend

Lemonade Server is AMD's local model server: one OpenAI-compatible API (/api/v1) in front of several runtimes — "recipes" — of which FastFlowLM (flm) runs models on the Ryzen AI NPU and llama.cpp (llamacpp) on the GPU (Vulkan, ROCm) or CPU. Like Ollama it ships its management surface on the same port, and the driver has the same shape as the ollama one: a proxy of GET /api/v1/health (the loaded models), GET /api/v1/models, the streamed POST /api/v1/pull, POST /api/v1/load, POST /api/v1/unload, POST /api/v1/delete and GET /api/v1/system-info. Lemonade stays on the host — that is where the NPU driver is — and pods reach it through the docker network gateway, so bind it to every interface (lemonade config set host=0.0.0.0; its port is 13305 by default) and let the bridge subnets through the host firewall, as for Ollama.

  • Inventory — GET /api/v1/models lists the downloaded models: the Lemonade id is the name (qwen3-it-4b-FLM, Qwen3-0.6B-GGUF), size in GB becomes sizeBytes, the recipe is reported as runtime (flm, llamacpp, ...), the format and quantization are read off the checkpoint (unsloth/Qwen3-0.6B-GGUF:Q4_0 → gguf, Q4_0) and Lemonade's labels are mapped onto the capability vocabulary the other backends use (chat → completion, tool-calling → tools, reasoning → thinking, embeddings → embedding, vision stays; a text recipe without a deployment label is a completion model). Everything Lemonade has downloaded is listed — transcription, image or speech models included — and the capabilities say what each one is.
  • Pull — POST /api/v1/models/pull takes a Lemonade catalog model name (lemonade list, or GET /api/v1/models?show_all=true on the server) and streams Lemonade's install (stream: true): the job reports bytes done against the whole download when Lemonade says how big it is, else against the current file. A name that is not in the catalog fails with not_found (registering a user.* model from a Hugging Face checkpoint is not offered here). On success the model is wired into kagent, as on ollama.
  • Load / unload — POST /api/v1/load starts the model's backend process (FastFlowLM on the NPU, llama.cpp, ...) and answers once it is ready; a load is bounded by ten minutes, not the usual call timeout. Lemonade has no keep-alive: a loaded model stays until it is unloaded or until another model of its type needs the slot (max_loaded_models, one per type by default; least recently used first). keepAlive: "-1" pins the model against that displacement (Lemonade's pinned, reported as running.pinned); every other keep-alive is ignored. running.device says where the model runs (npu, gpu, cpu). Unloading a model that is not loaded is a no-op.
  • Wiring — a kagent ModelConfig named after the model (qwen3-it-4b-FLM → qwen3-it-4b-flm) with provider: OpenAI, openAI.baseUrl = the agent host plus /api/v1 (--lemonade-agent-host, defaulting to the endpoint; reported as agentEndpoint) and the placeholder OPENAI_API_KEY Secret the kagent runtime insists on — the same path the kserve backend takes to a vLLM workload.
  • Node — GET /api/v1/nodes reports the host as Lemonade sees it: memory from Physical Memory of GET /api/v1/system-info (budgetSource: system-info), the accelerators Lemonade found available as gpuCount / gpuProduct (the NPU when there is one — AMD NPU (NPU Strix) — else the first GPU), reservations as the catalog sizes of the loaded models (Lemonade reports no per-model memory), and Lemonade's model store as the node's cache (mountPath, models, bytes).

Nothing here needs Kubernetes access beyond wiring, exactly as for ollama.

The lmstudio backend

LM Studio is a desktop model runner — llama.cpp on GPU and CPU, MLX on Apple silicon — with its own API (/api/v1, 0.4.0 and newer) next to an OpenAI-compatible one (/v1). The driver proxies the library (GET /api/v1/models), the download that backs a pull (POST /api/v1/models/download plus GET /api/v1/models/download/status/{job_id}) and POST /api/v1/models/load / POST /api/v1/models/unload. Like Ollama and Lemonade it stays on the host, so serve it on the local network (lms server start --bind 0.0.0.0, the app's Developer → "Serve on Local Network" toggle, or LMS_SERVER_HOST=0.0.0.0; its port is 1234 by default) and let the bridge subnets through the host firewall.

Two traits set it apart from the other host backends:

  • It has no delete. Removing a model is lms rm on the host, which a pod cannot run, so the driver reports delete: false and the API answers 501 unsupported. Use POST /api/v1/models/unwire to drop just the ModelConfig; the weights stay on the host until someone removes them there.

  • It has no version or health endpoint, and it answers HTTP 200 with an {"error": …} document for every path outside /api/v1 — Ollama's /api/version, /api/tags and /api/show included. Nothing may read a status code as proof that it reached an LM Studio: health here means GET /api/v1/models answering with a models array, and version stays empty because the server reports none.

  • Inventory — GET /api/v1/models is the local library, so every entry is downloaded: the key is the name (ibm/granite-4-micro), size_bytes becomes sizeBytes unchanged, format and quantization.name are reported as they come (gguf, Q4_K_M), architecture as family, max_context_length as contextLength, and the capability object is mapped onto the vocabulary the other backends use (trained_for_tool_use → tools, vision → vision, reasoning → thinking, on top of completion). Embedding models carry no capability object at all and are reported as embedding, listed like any other model — as on lemonade. The vision-language type (vlm) is a chat model: it is reported with completion plus whatever its capability object says, not as an embedding. trained_for_tool_use is the flag that matters for agents: LM Studio will accept tools for any model and emulate them through the prompt, but only a model trained for them calls them reliably.

  • Pull — POST /api/v1/models/download takes an LM Studio hub reference (lms ls for what is already there) and answers with a job; the driver follows it through GET /api/v1/models/download/status/{job_id} and reports downloaded_bytes against total_size_bytes, so progress is real bytes. A model that is already downloaded reports complete straight away (already_downloaded), and a reference LM Studio does not know fails with not_found. On success the model is wired into kagent, as on ollama. One caveat the API forces: LM Studio does not promise that the key a download lands under is the reference that was asked for — the download answers no key, there is no resolve call, and Hugging Face artifacts are keyed per weight set (sometimes with an @<quant> suffix) while catalog models keep their canonical publisher/model. Since the ModelConfig is wired with the reference the caller gave, the pull resolves it against the library before reporting success and fails naming the mismatch, rather than leaving a ModelConfig whose model the server does not serve.

  • Load / unload — POST /api/v1/models/load reads the weights and answers with an instance id; a load is bounded by ten minutes, not the usual call timeout. LM Studio evicts an instance, not a model, so unload looks the model's loaded instances up first; unloading a model that is not loaded is a no-op, as on ollama and lemonade. A load of an already-resident model is a no-op too, for the same reason the instance list exists: LM Studio loads an instance, not a model, so a repeated load would pin a second copy of a multi-GB model rather than answer "already loaded". Neither call takes a 2xx as proof: both answers have to name the instance, because an LM Studio without these endpoints answers 200 with an error document. There is no keep-alive and no pinning.

  • Wiring — a kagent ModelConfig named after the model (ibm/granite-4-micro → ibm-granite-4-micro) with provider: OpenAI, openAI.baseUrl = the agent host plus /v1 (--lmstudio-agent-host, defaulting to the endpoint; reported as agentEndpoint) and the placeholder OPENAI_API_KEY Secret the kagent runtime insists on. LM Studio needs no key of its own.

  • Node — none. LM Studio exposes no host hardware, so the driver reports nodeInventory: false and GET /api/v1/nodes answers 501 unsupported.

Nothing here needs Kubernetes access beyond wiring, exactly as for ollama.

The kserve backend

The driver consumes the modelServing contract of the giantswarm/agent-platform meta chart, rendered by its agent-platform-connectivity chart: the discovery ConfigMap agent-platform-model-serving (kind ModelServingConfig) for the serving namespace, runtime, GPU resource name, cache claim and preset selector, the GPU node pool's scheduling (spec.gpuPool: the pool's taint and label) and the models Gateway (spec.gateway.endpoint, https://models.<domain>: the origin every LLMInferenceService is routed on at /<namespace>/<name>, which is how a served model's address is known before KServe publishes it); the ServingPreset ConfigMaps (agent-platform.giantswarm.io/serving-preset=true); the cache PersistentVolumeClaim in the serving namespace. Every discovered value can be overridden by a flag (model-manager serve --help, --kserve-*); the pool scheduling by the registered backend document (docs/backends.md).

  • GPU node pool — a pool created through the platform is tainted nvidia.com/gpu NoSchedule and labelled giantswarm.io/machine-pool=<cluster>-<pool>. The discovery ConfigMap's spec.gpuPool.taint / spec.gpuPool.nodeSelector (the chart's modelServing.gpuPool.*), or the backend document's spec.kserve.gpuPool, make the driver tolerate the taint on everything it schedules onto the pool — the composed workloads, the download Jobs and the inventory scan Jobs — and select the pool wherever the pod is not pinned to a node already. The descriptor (GET /api/v1/backend) reports it as gpuPool. Unset, nothing changes; the chart's DaemonSet takes its own kserve.inventory.agent.*. The document is read on the first call and cached for a minute; settings resolved while it was not published yet — the slice publishes it as it installs — stand for five seconds only, and a fit that finds no node while they lack it reads the document once more before answering: with the pool now known, the scale-from-zero path; still without it, fits: false, retryable: true and a reason naming the document (the serving layer's discovery document <namespace>/<name> is not published yet — the slice is still installing; retry in a moment), which load_model echoes in its refusal. no accelerator node is the verdict for a document that names no pool.

  • CPU presets — a preset whose resources.gpus is 0 is CPU work: the fit judges every node of the cluster against its allocatable memory (the accelerator nodes and the CPU nodes alike, a node's Eligible rules as usual), and the preset knows no GPU pool and no accelerator RuntimeClass — its predictor carries neither the pool's toleration and label nor runtimeClassName, so on an installation whose GPU pool is tainted it lands on the cluster's CPU capacity. A bare model reference and every preset that requests a GPU keep the accelerator nodes as their capacity; GET /api/v1/nodes lists those.

  • Inventory — the cache contents per node plus the LLMInferenceServices of the serving namespace (readiness from conditions/modelStatus, node from the workload pod, GPU request, address). The cache is read in one of two ways (kserve.inventory.mode): a short-lived scan Job per cache node mounts the claim read-only and walks <claim>/<dir> whenever the inventory is older than the TTL (default; never while the cache claim is unbound or the GPU pool has no node, and in the background when the caller's deadline leaves no room for a pod), or a DaemonSet of model-manager cache-agent pods (same image) mounts the claim on each selected node and serves the same walk at GET /inventory, which model-manager reads from the agent on the node — no pod churn, and nodes[].cache.inventory says which mode produced the numbers. A node without a ready agent reports cache.error. The node holding the cache comes from the bound PersistentVolume's node affinity. GET /api/v1/nodes says which nodes hold a cache at all and what their memory budget is.

  • Naming a cache directory — a directory is named by what is known to have filled it: the marker a pre-warm download Job wrote, else the cache index — a ConfigMap in the serving namespace (model-manager-cache-index, kserve.cache.indexConfigMap) in which the driver records, while an LLMInferenceService exists, that its name is the directory the KServe storage-initializer fills and its hf:// storageUri the repository (repository, revision, preset label, one JSON entry per directory). The record outlives the LLMInferenceService, so the directory keeps its repository and, through it, its preset (the labelled one, else the single preset serving the repository) after the LLMInferenceService is deleted; a live LLMInferenceService always wins over a stale record, and the record is dropped when model-manager removes the directory. A record is bound to the cache it was made against (claim, volume): nothing is recorded while the serving layer has no cache (cache.enabled: false in the discovery document, a missing or unbound claim — the storage-initializer then fills storage that goes with the pod), unload_model drops a record that names a directory in no cache (bound to another claim or volume, or to none), and the ConfigMap goes with its last record. Without a marker or record the preset or LLMInferenceService of the same name is assumed, else the directory is listed by its bare name. Directories whose top level holds no model — no config.json and no weights file (*.safetensors, *.gguf, *.bin, *.pt, *.pth, *.onnx) — are not listed as downloads: Hugging Face client internals such as hf-home and xet live on the same claim and count towards nodes[].cache.bytesUsed, but they are not models, and a directory an LLMInferenceService is still filling shows as that served model with downloaded: false until its files arrive.

  • Nodes and eligibility — GET /api/v1/nodes lists the accelerator nodes only: nodes that advertise the configured GPU resource (gpuResourceName from discovery, default nvidia.com/gpu; capacity or allocatable > 0) or carry a gpu-feature-discovery label (nvidia.com/gpu.present, .count, .product). CPU-only nodes are not serving capacity for this backend. Each node says whether a model can be served there right now: eligible is true when the node is ready, matches the discovery nodeSelector and the GPU pool's, carries no NoSchedule or NoExecute taint the pool toleration does not cover (a tainted pool node is capacity once its taint is the configured one), and can mount the cache claim whenever workloads mount it (cache enabled and cache.redirectPolicy on — the Kyverno rule that mounts the claim into every workload) and the claim is pinned to nodes (a static local PersistentVolume or a local-path volume); a shared (RWX), unbound, missing or disabled cache never disqualifies a node. eligibilityReason names every failing rule (not ready, outside the serving node selector (kubernetes.io/hostname=spark-8723), taint nvidia.com/gpu:NoSchedule not tolerated (set the GPU pool taint), cache claim hf-cache is pinned to spark-8723). pull, load and fit-check refuse an explicit node that is not eligible with that reason (412 does_not_fit; fits=false on the fit check) before any Job or LLMInferenceService exists, and never pick an ineligible node themselves. Where the platform's redirect rule mounts the claim a model pod names (discovery cache.nodeClaims: true; pod inventory mode), the pin disqualifies no node: every node outside the claim's reach gets a cache claim of its own, <claim>-<node> (labelled model-manager.giantswarm.io/component: node-cache, annotated model-manager.giantswarm.io/cache-node: <node>), of the claim's class, access modes and size. pull with that node creates it, provisioned on the node, and its download Job fills it; a claim bound statically (no StorageClass) is refused as invalid, naming the claim to create by hand. list_nodes lists each node's own cache, fit-check and load read it per node, and the composed object names the claim on its pods (spec.annotations: model-manager.giantswarm.io/cache-claim for the leader or a single-node model, model-manager.giantswarm.io/worker-cache-claim for a split's workers; the claim itself on its own nodes, without an annotation). A split is refused while a node's cache lacks the weights (the cache of <node> holds none: pull_model <model> with node <node> first), and over more than two nodes when its workers would need different claims; copies whose nodes mount different claims are refused (one pod template mounts one claim). Without cache.nodeClaims a second node-local GPU node becomes a serving target only with shared storage. The claim has a say for the presets that store in it only — every download scheme (hf:// and, with the redirect policy on, the rest) and pvc://. An oci:// preset (KServe's modelcar: an OCI image containerd pulls onto the node's own disk; no storage-initializer, no claim mount) is judged without the claim's location: fit-check and load place it on any eligible GPU node with the budget, without a cache-node preference, and answer cached: false with cacheSource: oci-image; pull refuses it as invalid — the image is the platform's to pre-pull; nothing to download — and creates no Job. nodes still reports the pin: it describes the nodes, not a preset.

  • Import — search proxies the Hugging Face Hub; fit-check resolves the weight size (model.safetensors.index.json — its total_size, or the shards its weight_map names when they disagree with it by more than one percent — else the file tree, else the preset), adds the preset's overheadGiB (default 30) and compares with the node's free budget: the budget (nvidia.com/gpu.memory x gpu.count labels when present, else allocatable memory — unified-memory nodes; a node annotation model-manager.giantswarm.io/memory-budget-gib: "96" overrides that node's budget in GiB whatever kserve.budget.source says, reported as budgetSource: annotation — for unified-memory nodes whose allocatable memory overstates what a model may use) less what the models already served on the node reserve (reservedBytes, the one reading load and nodes make too; the preset being loaded is not counted against itself: each served preset's weights and overhead, and on a unified-memory node — GPUs without a memory label of their own — at least the share vLLM claims at start, --gpu-memory-utilization of the node's budget, 0.9 when the preset sets none; list_presets reports it as gpuMemoryUtilization), so fit-check and load reach one verdict; pull judges against the whole budget and refuses what cannot be served, then runs a download Job with the KServe storage-initializer image into <claim>/<preset name> — the directory the preset's LLMInferenceService mounts — reporting bytes on disk against the repository size. The Job downloads with HF_HUB_DISABLE_XET=1 and HF_HUB_ENABLE_HF_TRANSFER=1: the Xet client connects to CDN addresses a Cilium toFQDNs DNS proxy never resolved for an allowed name and hangs on the dropped SYNs, hf_transfer resolves per connection through the system resolver, so every address is admitted. A download that writes nothing to its directory for --kserve-download-stall-timeout (kserve.download.stallTimeout, default 10 min) fails with a reason the pull job shows — DOWNLOAD STALLED: no bytes written to <dir> for <N>s (<bytes> bytes on disk, …) — instead of hanging; partial files stay and a new pull resumes. Gated models need a token Secret (--kserve-hf-token-secret). The hub lookups of one fit check or search are bounded by --kserve-hf-timeout (kserve.hf.timeout, default 4 s): a hub that does not answer — egress blocked — lets fit-check fall back to the preset's requirements in time for the caller (weightsSource: preset; reason says the hub did not answer within the timeout), and a model no preset serves fails with that message. On a GPU pool with no node yet whose instance shapes the backend document lists (docs/backends.md) the check is against the pool's sizes, and the GPU memory budget is that of the GPUs the preset requests (resources.gpus, default 1) on a size — not the node's: a one-GPU preset on a four-GPU size has one card. A preset's declared weights (requirements.weightsGiB, what the pool was sized from) are answered as declaredWeightsBytes beside the hub-sized weightsBytes; when the hub holds more than the preset declares, reason says so on both verdicts (…; the preset declares 15.0 GiB of weights, the Hub holds 24.6 GiB on a fit; …; the preset declares 15.0 GiB of weights; the Hub holds 24.6 GiB, which is what does not fit — correct the preset on a refusal). The fit also checks vLLM's own admission, which a flat overhead cannot: vLLM starts only when its KV cache holds one sequence of --max-model-len tokens beside the weights, and crash-loops otherwise. From the checkpoint's config.json (its text_config when nested) and the preset's --max-model-len, --gpu-memory-utilization (default 0.9), --kv-cache-dtype (default: the checkpoint's KV quantization, else the model's dtype), --max-num-batched-tokens (default 2048, 8192 on a GPU of 70 GiB or more that is no A100), --block-size (16) and --tensor-parallel-size, the check computes vLLM v0.23's figure per GPU: a full-attention layer holds the sequence (2 × KV heads × head dim × dtype bytes per token; Gemma 4's global layers with their own heads), a sliding-window layer its window and one prefill chunk, an MLA layer its latent (kv_lora_rank + qk_rope_head_dim), a linear-attention or Mamba layer none, in 16-token blocks. The GPU leaves the KV cache --gpu-memory-utilization × its memory (a node's nvidia.com/gpu.memory; a pool size's gpuMemoryGiB in decimal GB) less the weights and a 3.15 GiB reserve — the activations, CUDA graphs and runtime vLLM profiles and what loading adds to the checkpoint, measured with Gemma 4 31B on one L40S. When one sequence does not fit, fits is false and reason names the KV need, what is left and vLLM's estimated maximum model length (kvCacheBytes, kvCacheAvailableBytes, estimatedMaxModelLen): gemma-4-31b at 65536 tokens on a 48 GB card at 0.92 needs 12.0 GiB of KV cache beside 31.0 GiB of weights where 7.3 GiB are left, and 8624 tokens would fit. An architecture the check does not read, a checkpoint without config.json, a preset without --max-model-len or a node without the GPU memory label is named in reason (the KV cache is not checked: …) and judged on the flat overhead alone. Every check the answer could not make is listed in skippedChecks with why (download-size, kv-cache, compute-capability): a model image whose registry does not answer within the fit check's budget, its timed-out requests retried, leaves all three. The GPU generation is judged the same way, from the same config.json (a mistral-format checkpoint's params.json): the weights' need — FP8 (quant_method: fp8, compressed-tensors 8-bit float, ModelOpt FP8) needs compute capability 8.9 (Ada, Hopper, Blackwell: below it vLLM falls back to the weight-only FP8 Marlin kernel, which answers wrong for such a checkpoint), NVFP4 (compressed-tensors 4-bit float, ModelOpt NVFP4) 8.6 (native on Blackwell, vLLM's weight-only FP4 Marlin kernel below it, proven on the A10G; nothing below 8.6 is proven), a checkpoint mixing FP8 and NVFP4 layers (ModelOpt mixed precision read from its config_groups) 8.9 for its FP8 ones, bf16 8.0 (the preset's --dtype decides unquantized weights); a preset whose requirements.minComputeCapability differs is named — against a node's nvidia.com/gpu.compute.major/ minor labels or the GPU of a pool size's instance family (g4dn T4 7.5, g5 A10G 8.6, g6 L4 8.9, g6e L40S 8.9); a GPU below it is a refusal naming both (computeCapabilityRequired, computeCapability), and among several pools one whose GPU is below the need is skipped for one that is not. On a unified-memory node (GPUs without memory of their own, a GB10) vLLM claims --gpu-memory-utilization × the node's whole memory at start, outside the pod's memory limit and the kubelet's accounting: that claim (unifiedReservationBytes) must fit the node's budget, at most its memory less the host headroom (hostHeadroomBytes, --kserve-unified-host-headroom-gib, chart kserve.budget.unifiedHostHeadroomGiB, 16 GiB), less what running models reserve there. A preset whose own claim does not fit is composed at the utilization its weights and overhead need there (gpuMemoryUtilization, in hundredths): a preset at 0.90 sized for a 24 GB GPU (21.8 GiB) runs at 0.18 on a 121.7 GiB GB10. Only when that does not fit either is fits false, load_model refuses, and reason names the claim and the utilization. The claim must also hold the weights (a split's share) and overhead: a preset whose claim is short of them is refused, never sized up, and reason names the utilization that holds them (fitGpuMemoryUtilization when the node leaves it). Weights the runtime keeps on disk (an n-gram table read from there) do not count against the claim: a preset declares what it holds in memory as requirements.residentWeightsGiB. A dedicated GPU is judged as before. Serving shape — a preset's resources.gpus and --tensor-parallel-size describe its reference shape, the node it is written for; the fit derives the shape from the node the model is placed on (devicesPerPod, tensorParallel, gpuMemoryUtilization). On a unified-memory node (one GPU without memory of its own) a pod requests one device; on a node whose GPUs have memory of their own, the smallest power of two whose cards hold the weights and overhead (a preset written for four 48 GB L40S cards keeps four on a g6e.12xlarge). A split requests that count on every pod and runs tensor parallel over devices × nodes. A shape that differs from the reference is named in reason; more devices per pod than the node has allocatable answer fits: false, and copies on nodes of different device counts are refused. The preset's CPU and memory requests are capped at what the node has left beside the requests of its other pods (cpuRequestMillis, memoryRequestBytes); a node with nothing left of one answers fits: false. The composed object carries the derived device count, --tensor-parallel-size, --gpu-memory-utilization and any capped request; on its reference node a preset is composed as written.

  • Serve / stop — load composes the serving object from the preset after the fit check fit-check answers (the node's free budget, the KV cache); unload deletes it and unwires within the caller's deadline — the object is found by repository, object or preset name, never through the cache inventory, so a scan pod that cannot start never delays the deletion — and rescans the cache in the background where a scan pod may run, the answer's inventory saying whether one runs or why none can (the cache persists); delete removes the cache directory (refused while served). The object is an LLMInferenceService (serving.kserve.io/v1alpha2, KServe's llm-d control plane), the only kind the backend composes, lists, stops and deletes; a loaded model names it as its kind. A load fails fast, with nothing created, where no llm-d control plane would reconcile the object (503 unavailable, MCP unavailable): the API is not served, or the CRDs are installed without the controller — the backend looks for the well-known LLMInferenceServiceConfig kserve-config-llm-template the controller composes from, which ships with the platform's kserve-runtime-configs component, not with the CRDs — or the driver can read neither (the error names the permission: get/list on llminferenceserviceconfigs cluster-wide, which the chart's ClusterRole grants). get_backend / GET /api/v1/backends carry the same reason in message before anyone tries. While a backend's serving slice is still installing — the control plane not ready and no preset published yet — list_presets and a get_model or load_model of a model it does not know yet answer the same unavailable, backend <name> is not ready with what it waits on, instead of an empty list or not_found; once ready they answer as before. While the llmisvc admission webhook has no ready endpoint (the minute or two after the serving slice is installed: no route to host, connection refused, no endpoints available) the create is retried every poll interval for up to two minutes within the call's deadline; a webhook still unreachable then is 503 unavailable naming the webhook, with nothing created. The verdict is taken on the cluster as it is at the load: settings resolved without the control plane stand for five seconds, not the minute the settings cache keeps otherwise, and a load about to be refused resolves them once more first — the serving slice's CRD and runtime-configs children land seconds after its discovery document. With that document published the refusal says the slice is most likely still landing and to retry in a moment; without it, which components to turn on. The composition, by spec shape: spec.model.uri from the preset's storageUri (hf://, pvc:// into the cache, or an oci:// model image), spec.model.name from model.id, replicas: 1, router: {route: {}} (KServe renders the HTTPRoute on the configured ingress gateway with the workload Service as its backend; the llm-d endpoint picker, router.scheduler, is opt-in through the backend document's spec.kserve.router.scheduler or a preset's spec.router.scheduler, and its InferencePool needs the Gateway API Inference Extension on the gateway — docs/backends.md), template.containers[main] with the preset's args, env and resources (GPU count under the discovery's resource name; for an oci:// preset the env ends with the modelcar environment — HOME=/tmp, HF_HOME=/tmp/hf, VLLM_CACHE_ROOT=/tmp/vllm-cache, TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor, USER=vllm, LOGNAME=vllm, each only where the preset sets no value of its own —, since the modelcar uid the runtime runs as has no passwd entry and a read-only image), scheduling as the template's nodeSelector/tolerations (merged with the discovery selector, the GPU pool's label and toleration, and the node pin), the chat template mounted, and template.runtimeClassName from the discovery ConfigMap's runtimeClassName when non-empty — absent when empty. No baseRefs: KServe chooses its well-known LLMInferenceServiceConfigs from the spec's shape and appends them itself; a preset that names a custom config (spec.baseRefs: [{name: …}]) is the only baseRefs case. No image: the container runs the image the well-known template names (llm-d-cuda, mirrored through the platform's registry override); a preset overrides it with spec.template.containers[{name: main, image: …}] when it has a reason — spec.template extras are copied on top, containers merged by name. model-manager creates and lists no LLMInferenceServiceConfig.

  • Wiring — in the load call, before the model is ready, a kagent ModelConfig named after the LLMInferenceService (provider: OpenAI, baseUrl = the model's address + /v1, model = the served model name: the object's spec.model.name, which the well-known template passes to vLLM) — the same rule the portal's serve flow applies; the load answer's wiring names it, the load job refreshes it from the address KServe publishes once the model is ready, and a served model model-manager manages that has none is wired by the caller's next list_loaded_models (wiring.reason: "wired on read"). The address is the one KServe published or, until then, the one the object is expected on: the discovery document's models Gateway (spec.gateway.endpoint + /<namespace>/<name>), the in-cluster workload Service otherwise. The API key follows the address: a model routed on the models Gateway (an external host) gets apiKeyPassthrough: true and no Secret — the agent forwards the person's own token, the only thing the Gateway's JWT policy admits, so a placeholder key would fail every turn with 401; a model reached on its in-cluster Service gets the placeholder OPENAI_API_KEY Secret the go ADK runtime insists on, which keyless vLLM never checks. A re-wire that moves a ModelConfig onto the Gateway removes its placeholder Secret. A ModelConfig that already points at the model (same host, same served model name), whoever created it, counts as the model's wiring: it is reported with managed: false, never duplicated and never deleted; unwire/unload only remove ModelConfigs model-manager created.

  • Ownership — LLMInferenceServices in the serving namespace that model-manager created or that carry the agent-platform.giantswarm.io/preset label (the portal's serve flow) can be unloaded here; hand-written ones are inventory only (409 conflict on unload; managedBy says who owns them). One Flux applies from git reports the Kustomization or HelmRelease as gitops, held to the Kustomization's inventory as the unload holds it: a leftover its Kustomization no longer lists is written live, unloaded and given copies here; one it lists is changed in git (mode: commit).

  • State — the loaded models (GET /api/v1/loaded, list_loaded_models) are every InferenceService and LLMInferenceService of the serving namespace whatever their readiness: status Ready, Pending (the workload pod waits for a node, the weights or an image — reason Unschedulable with the scheduler's message, DownloadingWeights, ImagePullBackOff), NotReady (the Ready condition's reason, a failed load) or Terminating, with message in words. unload takes the preset name, the object name or the repository id and deletes the object in any state; its answer says status: Terminating and that the list shows the object until it is gone. A load's answer names the object it created (running.resource, running.kind), carries the fit verdict it was judged by (fit, the check_fit shape) and the initial running.steps; a refusal (412 does_not_fit) carries the fit's reason, the declared-weights clause included.

  • Phases — phase refines status with where a serve is, and steps is the whole timeline: one entry per phase in order, each {name, state: pending|inProgress|done|failed, since, finishedAt, reason, message} with since/finishedAt from the pod's conditions, container statuses and Events (Karpenter's nomination, the kubelet's Pulling/ Pulled with the pull's duration, probe failures), so a caller shows elapsed time per step. The phases of a fresh serve on a scale-to-zero pool (proof 1: ≈ 12 min to Ready):

    phase What is happening Read from
    scheduling no node has a free GPU; the autoscaler is asked and, once it nominated a NodeClaim, until that claim has launched an instance PodScheduled=False (Unschedulable), FailedScheduling events; Karpenter's Nominated event names the NodeClaim (karpenter.sh/v1, read as the caller): Launched not yet True keeps the step with reason: NodeLaunching (Karpenter nominated NodeClaim gpu-l4-x7k2q; no instance has launched yet); Launched=False, or an InsufficientCapacityError / NodeClassNotReady Warning event on a NodeClaim of the pool (in the default namespace, where a cluster-scoped object's events land — Karpenter deletes a refused claim within seconds and nominates another) gives reason: CapacityUnavailable. A capacity refusal is read whole: what the claim asked for — its spec.requirements while it stands, else the pool's template (the karpenter.sh/v1 NodePool it is named after, read as the caller; the pool chart names the family and the sizes, composed to types) — and every refusal text (the claim's Launched=False, the events) for the sizes and the zone the cloud names and the zones it names as having capacity where the text is whole (Karpenter's event holds the first 300 characters of the cloud's answer: the first size and the zone; the claim's condition holds it whole), plus whether the serving namespace's cache claim is bound to a volume in that zone — the pool's pin, cluster-manager's pool.zones: Karpenter could not launch a node: 4 NodeClaims refused, the last (gpu-l40s-46flj) at 2026-09-18T12:24:14Z — InsufficientCapacityError: the cloud has no g6e.2xlarge, g6e.4xlarge or g6e.8xlarge capacity in eu-central-1b, the zone the pool is pinned to by the cache claim hf-cache (its volume lives there); it has capacity in eu-central-1a and eu-central-1c. The way out is a pool in eu-central-1a or eu-central-1c (create_node_pool with zones naming it and cache: false), or removing the cache claim so the next pool is not pinned; Karpenter retries while the pod waits. A pool that pins no zone lets a claim launch in every zone of its node class's subnets, and a fleet that launched nothing was refused them all: the zones are the claim's own zone requirement while it stands, else the node class's (ec2nodeclasses.karpenter.k8s.aws, read as the caller), requestedZones names them all, availableZones never a zone the same answer refused, pinnedByCache is false, and the message reads the cloud has no g6e.2xlarge, g6e.4xlarge or g6e.8xlarge capacity in any zone the pool allows (eu-central-1a, eu-central-1b and eu-central-1c). No zone is left to move to; wider sizes or another accelerator (a re-run of create_node_pool) give Karpenter more to choose from, and it retries while the pod waits (the node class unreadable: the zone the cloud named, at least). The step carries the same as fields — refusedInstanceTypes, requestedZones, availableZones, pinnedByCache (absent when the cache claim or its volume could not be read) — and the model's reason/message carry the message. Each refused size carries its own reason: the cloud's fleet errors (InsufficientInstanceCapacity, VcpuLimitExceeded, any other code) are read from the same text, a size the capacity wording names is refused for capacity, and the sizes it does not name are the vCPU quota's while a VcpuLimitExceeded stands (the quota error names no size), else another code's — the cloud has no g6e.12xlarge capacity in any zone the pool allows (eu-north-1a, eu-north-1b and eu-north-1c); g6e.24xlarge and g6e.48xlarge exceed the account's vCPU quota: its limit is 96 vCPUs for their instance bucket. No zone is left to move to; the way out is a vCPU quota increase for the instance bucket (EC2 Service Quotas); Karpenter retries while the pod waits, a refusal for the quota alone reads g6e.12xlarge, g6e.24xlarge and g6e.48xlarge exceed the account's vCPU quota: its limit is 64 vCPUs for their instance bucket. The way out is a vCPU quota increase for the instance bucket (EC2 Service Quotas) or smaller sizes (a re-run of create_node_pool), and refusedInstanceTypes names every size refused, whatever its reason. A refusal that is not the cloud's capacity wording keeps Karpenter's words; a read that fails — the claim, the events, the NodePool, the node class, the cache claim — is named in the message, never silent; ≈ 30 s from nomination to launch
    nodeStarting the node exists — an instance launched — and its GPU is not allocatable yet the NodeClaim's Launched=True (since = its transition; the message names the node once the claim registered it), a nominatedNodeName, the pod bound; the node's allocatable nvidia.com/gpu; ≈ 3.5 min until the node registers, ≈ 1 min more for the GPU
    downloadingWeights the storage-initializer fills the cache directory the init container; bytesTotal (the preset's weights), bytesCompleted (a bounded cache-agent scan while filling, cache-agent mode only), cached: true when it finished within 15 s — the claim held the weights (72 s for 8 GB, else 0.3 s)
    pullingImage the kubelet pulls the runtime image the container Waiting (ContainerCreating), Pulling/Pulled events (the message carries the duration); ≈ 4 min
    loading vLLM loads the weights until the startup probe passes the container Running, not Ready; Unhealthy events say what the probe saw; ≈ 1 min. A runtime that died and is restarted by the kubelet keeps the step under way with reason: CrashLoop and a message naming the crash count, the exit code and the last error line of the crashed container's log (kubectl logs --previous, read as the caller): runtime crashed 2× (exit 1): PermissionError: [Errno 13] Permission denied: '/mnt/models/.cache/vllm'; a runtime that died on a CUDA out-of-memory after mlock() refused it (could not lock … Cannot allocate memory in the same log) gets the node's locked-memory limit named instead of an out-of-memory, with the fix: a containerd.service drop-in LimitMEMLOCK=infinity, systemctl daemon-reload, restart containerd; a caller who may not read pods/log gets the message with the reason the line is missing
    routing the pod is ready, KServe resolves the route; once the object is Ready, until the model answered its first request (reason: AwaitingFirstAnswer, the message says why no answer came; asked again after a minute) Ready=False HTTPRoutesNotReady, then Ready=True
    ready the model answered one request on its workload Service: a chat completion of one token (a pooling model: one embedding), sent once per Ready transition, never beside one still in flight (a list waits 15 s for it, the request runs on for up to 2 min) since = the Ready condition's lastTransitionTime, finishedAt = the answer, message: answered POST /v1/chat/completions. A runtime that passed its readiness and answers that request with an error fails the step with reason: FirstRequestFailed and the runtime's status and first error line (…answered 500 Internal Server Error: openai_harmony.HarmonyError: …): the phase is failed, status NotReady, and load_model returns the error instead of waiting. The failure stands while the object drops out of Ready (a runtime that hangs on the request fails its readiness too), until a later Ready transition's request is answered. A server whose route list names nothing to ask is not asked; the step says why
    failed a step failed — the step says why a capacity refusal standing when the GPU pool's scale-up budget is spent — --kserve-scale-up-timeout / KSERVE_SCALE_UP_TIMEOUT / chart value kserve.scaleUpTimeout, default 10m, counted from the pod's creation: the scheduling step fails with CapacityUnavailable, its message plus ; no node came within the scale-up budget of 10m0s, and the model's reason/message carry it (a nominated claim still launching without a refusal never fails the step); ImagePullBackOff/ErrImagePull; CrashLoopBackOff (the kubelet backed off from restarting a runtime that keeps dying: the loading step fails with the crash message and the back-off, and the model's reason/message carry it); an initializer that exited non-zero, restarted or ran longer than 30 min (DownloadStalled); modelStatus.lastFailureInfo
    terminating the object is being deleted deletionTimestamp; the steps stay as they were

    Without a pod yet the object is scheduling (WaitingForPod) since its creation; a Ready object whose pod the driver cannot see has every step done. While the pod has no node, Karpenter's account of the one it waits for — the claim launching, its refusal, the instance registering — is the model's reason/message in list_loaded_models instead of the scheduler's Unschedulable; a caller who may not read nodeclaims.karpenter.sh or the default namespace's events gets the step's message with the reason the read failed, never silently. Backends without a serve lifecycle (ollama, lemonade, lmstudio) answer neither phase nor steps. Reading the Events and the nodes costs one bounded (3 s) list per pod — plus, for a pod without a node, the nominated NodeClaim and Karpenter's Warning events — concurrently, so three served models stay inside a caller's ~10 s deadline.

  • Cache verdict — check_fit's cached is a boolean and cacheSource says how it was decided: scan (a cache scan answered in this call), index (no scan could run — a GPU pool at zero, the caller's deadline — and the cache index remembers a directory an LLMInferenceService filled for the repository in the claim and volume bound now; a record bound to another cache, or to none, is no verdict), unknown (neither answered; cached: false is then no verdict), oci-image (the preset serves an OCI model image the nodes pull themselves; nothing of it is in the claim, and cached: false is the verdict). On the scale-from-zero pool path the claim is asked as a whole, so weights an earlier serve left in it answer cached: true.

Jobs, restarts and replicas

Jobs (GET /api/v1/jobs) live in process memory: the chart runs one replica and clients poll one server. On kserve a pull job carries the node whose cache receives the download and the preset it is for — what the request named, or what model-manager picked itself after the fit check — so any client (another tab, an agent through MCP) can place the download without remembering the request; ollama jobs carry neither. A restart loses the job list, not the work — kserve pulls are Kubernetes Jobs that model-manager re-adopts on start (GET /api/v1/jobs lists them again as running pulls, node and preset read back from the Job's annotations), a kserve load is recovered by the reconcile loop that wires ready LLMInferenceServices without a job, and an ollama or lemonade pull simply is re-issued (Ollama resumes the layers it has, Lemonade the files). A persistent job store is deliberately not built until a second replica or a job history across restarts is needed; until then, treat the job list as a progress view, and the backend (Jobs, LLMInferenceServices, Ollama) as the truth.

Running

Without --backend/--backends the process starts with no backend and waits for backend documents in --namespace (POD_NAMESPACE); see docs/backends.md. A static backend:

model-manager serve \
  --backend ollama \
  --ollama-endpoint http://127.0.0.1:11434 \          # as reached by model-manager
  --ollama-agent-host http://172.21.0.1:11434 \        # as reached by agent pods (written into ModelConfigs)
  --kubeconfig ~/.kube/config --kube-context kind-agentlab \
  --kagent-namespace kagent

Against a Lemonade Server on the host (AMD Ryzen AI NPU):

model-manager serve \
  --backend lemonade \
  --lemonade-endpoint http://127.0.0.1:13305 \      # as reached by model-manager
  --lemonade-agent-host http://172.21.0.1:13305 \   # as reached by agent pods (+ /api/v1 in ModelConfigs)
  --kubeconfig ~/.kube/config --kube-context kind-agentlab

Against an LM Studio on the host:

model-manager serve \
  --backend lmstudio \
  --lmstudio-endpoint http://127.0.0.1:1234 \       # as reached by model-manager
  --lmstudio-agent-host http://172.21.0.1:1234 \    # as reached by agent pods (+ /v1 in ModelConfigs)
  --kubeconfig ~/.kube/config --kube-context kind-agentlab

Every flag has an environment variable (model-manager serve --help). Without Kubernetes access the server still runs; wire reports false and wiring operations answer 501.

Identity

With --enable-oauth model-manager is an OAuth 2.1 resource server (mcp-oauth) in front of both the MCP endpoint and the REST API: every call needs a bearer token the platform identity provider issued — Dex (--oauth-provider dex, the Agent Platform default) or Google (--oauth-provider google). On the platform nobody logs in to model-manager itself: muster forwards the session's IdP id_token byte-identical (MCPServer auth.forwardToken: true, rendered by the chart) and the portal sends the signed-in user's id_token through the gateway; model-manager validates them against the IdP's JWKS when their audience is one of --oauth-trusted-audiences. The chart passes the platform's OAuth client (oauth.trustedAudiences, else global.identity.clientId) plus the MCPServer's requiredAudiences: every forwarded token carries those by construction and they are what the kube-apiserver trusts, so a portal session — whose id_token carries them but not the platform client — is accepted too. An id_token whose audience matches none of them is refused with 401; the Unknown JWT bearer refused log line names the token's token_audience next to the trusted_audiences. The caller — the email, else the sub — is on every mutation's log line and recorded as requestedBy on jobs.

--downstream-oauth goes one step further: everything a request does against the Kubernetes API (LLMInferenceServices, download Jobs, cache scans, kagent ModelConfigs, the discovery ConfigMap) presents the caller's IdP token, and the ServiceAccount holds no permissions at all — the chart renders none of its Roles and ClusterRoles; the caller's RBAC is the only RBAC. That needs an apiserver that trusts the IdP and the token's audience — a Dex install lists the cross-client audience the apiserver trusts in the MCPServer's requiredAudiences (muster requests it at login), a Google install's client id is the apiserver's --oidc-client-id. A request whose bearer yields no IdP token to present (one this server neither issued nor trusts as a forwarded id_token) is refused with 401 instead of running as the permissionless ServiceAccount. Nothing runs without a caller: the wiring reconciler and the re-adoption of running downloads after a restart are off, and a job that outlives its caller's token (a download longer than the token lives) fails on the apiserver's 401 — attributed to the caller, retried by the caller. A restart of the process loses the in-memory jobs and nothing else: a served model's ModelConfig is written by the load call itself and, when missing, by the caller's next list_loaded_models, and a running download Job is joined by the next pull_model. Health endpoints (/healthz, /readyz, /backendz) and the OAuth metadata stay public.

Commit mode: the pull request as the person

wire_model and unwire_model take mode: commit, and so do load_model and unload_model on kserve: the serving object and its ModelConfig — whose address follows from the object's name — land in one pull request (files <name>-llminferenceservice.yaml and <name>.yaml), and no load job runs; the ModelConfig turns ready when the model does, after Flux applies the merge. pull_model and delete_model never commit (a download and the weights are no GitOps objects): mode: commit answers unsupported, as it does for a load or unload on ollama, lemonade or lmstudio. Instead of writing the ModelConfig live, model-manager opens a pull request as the signed-in person in the GitOps repository that owns the kagent namespace. The repository follows from Flux provenance: the ModelConfig's own Kustomization when Flux applies it, else the namespace's (through the HelmRelease that renders it, where a chart creates the namespace), then the Kustomization's GitRepository and spec.path. Where there is none, repository, branch and path name it. A kserve backend on a remote target splits the commit: the serving object lands on the target, its ModelConfig in the kagent namespace of model-manager's own cluster. The serving namespace's provenance is read on the target as the caller; the Flux objects are read on model-manager's cluster, where Flux applies to the target through spec.kubeConfig. An owning Kustomization must apply through one. For a namespace a HelmRelease renders there, the location is the one Kustomization in the HelmRelease's namespace that applies through the same kubeconfig Secret; none or several are refused. The ModelConfig follows the kagent namespace's provenance, and an explicit repository/branch/path names the serving object's location only. Each repository gets its own pull request, and the answer's commit.also lists the further locations. The files go under <path>/model-manager/, one per object, with the directory's kustomization.yaml and the parent's entry (gitops-commit's layout). A Secret, the placeholder API key, is committed only SOPS-encrypted for the repository's .sops.yaml recipients. Without one the call is refused: use apiKeyPassthrough or apiKeySecret, or mode: apply. dryRun answers the files with their content and opens nothing. The answer's commit names the repository, base, directory, branch, files, the pull request's URL and number, and its author. Refused: an object Flux applies from a file outside model-manager/ (change it where it is written), one a HelmRelease renders, a removal of an object that was written live, a removal of an object git no longer holds that a Kustomization with prune: false left behind (the answer names the Kustomization and offers mode: apply, which removes it live), and a Kustomization whose targetNamespace would move the objects.

Commit mode needs the person's GitHub authority, offered with the chart's github.enabled. With github.tokenExchange, the MCPServer forwards the person's ID token as without commit mode, and only a commit asks for GitHub: model-manager exchanges the ID token at muster's RFC 8693 broker (--github-token-exchange-endpoint) for the person's user grant of the GitHub App github.tokenExchange.audience names, for that commit only, verifies it with GET /user and opens the pull request with it. Without the grant the MCP tool answers grant_required with apps: [<audience>], and the MCPServer lists the audience under auth.providerApps, so muster turns the answer into that App's sign-in link. REST answers 403 with code grant_required and the same apps. GitHub refusing the grant with 401 answers grant_required too. The client ID comes from --github-token-exchange-client-id, else from the mounted Secret's client-id key. A call made by an agent acting for the person (an ID token with an act claim) is refused as forbidden. With auth.providerApps, muster also attaches the person's App token to every call it forwards, which model-manager ignores, until giantswarm/giantswarm#39227 lets a server name an App without receiving it.

Fetching the person's GitHub grant from muster is a temporary deviation from the MCP 2025-11-25 authorization rules (no token passthrough, no third-party credentials), decided in giantswarm/giantswarm#39213; the exit is model-manager holding its own grant through URL elicitation and ID-JAG (giantswarm/giantswarm#39102).

github.enabled without github.tokenExchange fails the chart. No other GitHub credential exists in the pod. A server without the exchange answers unsupported for mode: commit, and get_backend/list_backends report capabilities.commit. The App has to be installed, with contents and pull-request write, on every repository a commit may target. On a kserve backend with a remote target that is two repositories: the one owning the serving namespace on the target (typically the workload cluster's fleet repository) and the one owning the kagent namespace on model-manager's own cluster. Before anything is written, model-manager checks each repository with the person's App user token (GET /repos/{owner}/{repo}; GitHub answers 404 for a repository the App is not installed on). One it does not reach answers repository_unavailable (412), naming the repository, which part lands there and the App to install; dryRun reports the same as commit.unavailable (and commit.also[].unavailable), with no files for that repository.

Helm chart

helm/model-manager — see its README. Keys the giantswarm/agent-platform meta chart sets for its model-manager component: backend, ollama.endpoint, ollama.agentHost, lemonade.endpoint, lemonade.agentHost, lmstudio.endpoint, lmstudio.agentHost, kagent.namespace, mcp.enabled, oauth.*, muster.mcpServer.*; for kserve kserve.namespace (the serving namespace), kserve.discovery.*, kserve.hf.tokenSecret.* and the kserve.* overrides. Optional, off by default: muster.mcpServer.enabled (renders an mcpservers.muster.giantswarm.io CR), httpRoute.enabled, networkPolicy.enabled, oauth.enabled, observability.otel.endpoint (OTLP traces: HTTP server spans, an mcp.<method> span per MCP request and a tool.<name> span per tool call, configured by the standard OTEL_* variables the chart renders from observability.otel.*).

Development

See docs/development.md.

About

Model management service for the Agent Platform: backend-abstracted (ollama, kserve) inventory, pull, load/unload, delete and kagent wiring via REST and MCP

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages