OAuth client registration: CIMD is enabled by default. DCR is deprecated by MCP 2026-07-28 and remains a compatibility fallback. See registration, consent, egress and TLS configuration.
Model management service for the Giant Swarm Agent Platform. One API over serving backends — one or several per installation, in one process:
| Backend | Where | What it proxies |
|---|---|---|
ollama |
laptop / agentlab installs (host Ollama through the kind docker-network gateway) | /api/tags, /api/ps, streamed /api/pull, /api/delete, keep_alive load/unload |
kserve |
GPU installs (KServe + the platform's modelServing component) |
LLMInferenceServices (or classic InferenceServices) composed from serving presets, per-node HF cache inventory, pre-warm download Jobs with progress, Hugging Face Hub search, node fit checks |
lemonade |
AMD Ryzen AI laptops / workstations running Lemonade Server on the host (FastFlowLM on the NPU, llama.cpp on GPU / CPU) | /api/v1/health (loaded models), /api/v1/models, streamed /api/v1/pull, /api/v1/load, /api/v1/unload, /api/v1/delete, /api/v1/system-info |
lmstudio |
desktop installs running LM Studio on the host (llama.cpp on GPU / CPU, MLX on Apple silicon) | /api/v1/models (the library and its loaded instances), /api/v1/models/download + its status job, /api/v1/models/load, /api/v1/models/unload — no delete |
One model-manager runs every backend the host has (--backends=ollama,lmstudio,
chart value backends: [ollama, lmstudio]): the API reports each backend and
its explicit capability flags (GET /api/v1/backends), every model, job and
node names its backend, every request may name one, and an unqualified
model reference is resolved to the one backend that holds it (409 conflict
when several do). GET /api/v1/backend describes the first, default backend
and names the others — the one-backend form clients of a single backend keep
using. Clients render only what a backend supports instead of switching on
its name. Pulled or loaded models are wired
into kagent automatically as ModelConfigs (native keyless Ollama provider
for the ollama backend; OpenAI provider with the caller's token forwarded
(apiKeyPassthrough) for a kserve model routed on the models Gateway;
OpenAI provider plus a placeholder API-key Secret —
against the workload URL for a kserve model on its in-cluster Service, and
against Lemonade's /api/v1 for lemonade; a kserve model's ModelConfig is
created by the load call, before the model is ready), so agents can use them
without manual steps. They are written in the API group and version the
cluster serves (api.kagent.dev/v1alpha3 from kagent 1.3 on,
kagent.dev/v1alpha3 before it; discovered at start-up, re-discovered every
minute and when a call fails NotFound, so a running process follows a
kagent upgrade),
and a ModelConfig is ready once kagent has accepted it and resolved its
Secret (conditions Accepted and ResolvedRefs).
The same operations are exposed twice from one process:
- REST/JSON under
/api/v1for the portal backend — contract inapi/openapi.yaml, also served at/api/v1/openapi.yaml. - MCP (streamable HTTP,
/mcp) for muster — toolsget_info,get_backend,list_backends,list_models,get_model,list_loaded_models,pull_model,load_model,unload_model,delete_model,wire_model,unwire_model,list_jobs,get_job,cancel_job, plus the kserve capabilitieslist_presets,search_models,check_fit,list_nodes(through muster:x_<server>_<tool>); every tool takes an optionalbackend. Every tool's description and input schema as a model reads them: docs/tools.md.
Part of Model management in the Agent Platform; decision records: Model Manager PDR, the Ollama-backend ADR and the Lemonade-backend ADR in the team's decision log.
| Operation | REST | MCP tool |
|---|---|---|
| This server's build (release version, commit, build time), its tools, the configured backends and the wiring target | — | get_info |
| Every backend's identity, capabilities + load semantics (the first is the default) | GET /api/v1/backends |
list_backends |
Register / remove a backend at runtime (a backend document, docs/backends.md; dryRun, mode: apply) |
— | add_backend, remove_backend |
| One backend (the named one, else the default) plus the names of all | GET /api/v1/backend[?backend=] |
get_backend |
| Downloaded models (with loaded state + ModelConfig), of one or every backend | GET /api/v1/models[?backend=], GET /api/v1/models/{name}[?backend=] |
list_models, get_model |
Loaded / running models (kserve: each with its modelConfig; a served model model-manager manages that has none is wired by the read, wiring: {wired, reason: "wired on read", modelConfig}) |
GET /api/v1/loaded[?backend=] |
list_loaded_models |
Pull / import (returns a job; on backend, else the default backend) |
POST /api/v1/models/pull {"model","backend?","wire?","preset?","node?"} |
pull_model |
| Job progress | GET /api/v1/jobs[?backend=], GET /api/v1/jobs/{id}, DELETE /api/v1/jobs/{id} |
list_jobs, get_job, cancel_job |
Load / unload (kserve: the load answers fit, running and wiring — the ModelConfig created in the same call, apiKeyPassthrough for a model routed on the models Gateway — before the model is ready; the unload deletes the serving object and unwires within the call — only a ModelConfig this model-manager created, any other is reported as modelConfigLeft — never waiting for a cache scan, and answers inventory: {refreshing, reason?} — the cache rescanned in the background, or why it cannot be) |
POST /api/v1/models/load {"model","backend?","keepAlive?","tracing?"}, POST /api/v1/models/unload {"model","backend?"} |
load_model, unload_model |
| Delete (unwires by default) | DELETE /api/v1/models/{name}[?unwire=false][&backend=] |
delete_model |
Every write tool takes dryRun (MCP): it answers the plan and changes nothing — pull_model whether the model is there, the kserve fit verdict and the ModelConfig it would wire; load_model the loaded state, the keep-alive, the kserve fit verdict and serving object (manifests) or alreadyServing, the ModelConfig the auto-wire would ensure (wiring); unload_model the kserve serving objects it would delete and the ModelConfig it would unwire; delete_model the loaded state and the ModelConfig it would unwire; cancel_job the job |
— | pull_model, load_model, unload_model, delete_model, cancel_job |
Wire / unwire to kagent (apiKeyPassthrough or apiKeySecret+apiKeySecretKey override the backend's API-key shape; both together are refused; MCP dryRun: true returns the manifests it would write or delete, writing nothing) |
POST /api/v1/models/wire {"model","backend?","apiKeyPassthrough?","apiKeySecret?","apiKeySecretKey?"}, POST /api/v1/models/unwire {"model","backend?"} |
wire_model, unwire_model |
| Serving presets (kserve) | GET /api/v1/presets[?backend=] |
list_presets |
| Hub search (kserve) | GET /api/v1/search?q=…&limit=…[&backend=] |
search_models |
| Fit check (kserve) | POST /api/v1/models/fit-check {"model" or "preset","backend?","node?"} |
check_fit |
| Node budgets + reservations (+ caches on kserve) | GET /api/v1/nodes[?backend=] |
list_nodes |
| Health | GET /healthz, GET /readyz |
— |
Model references may contain / and : (smollm2:135m, hf.co/org/repo:Q4_K_M,
Qwen/Qwen3-14B); path parameters capture the rest of the path. Errors are
{"error":{"code":"not_found|invalid_request|unsupported|conflict|gitops_owned|auth_required|repository_unavailable|forbidden|does_not_fit|backend_error","message":"…"}};
unsupported (501) means the matching capability flag is false, does_not_fit
(412) that the kserve fit check refused a pull or load, conflict (409) also
that an unqualified reference exists on several backends — repeat the request
with backend. gitops_owned (409) means the object the write would change
or delete is applied by Flux from git — it carries
kustomize.toolkit.fluxcd.io/name or helm.toolkit.fluxcd.io/name, whatever
its app.kubernetes.io/managed-by says — and a live change would be reverted
on the next reconciliation; the message names the Flux object. Nothing was
written. A Kustomization is held to its status.inventory: an object it no
longer lists — its file removed from git while the Kustomization does not
prune, which leaves the object on the cluster with its labels — or whose
Kustomization is gone is applied by nothing, and the unload and unwire paths
remove it live. A ModelConfig reports the owner that applies it as gitops
(list_model_configs, the inventory held the same way: a leftover reports
none), and the wiring
reconciler and the unload, unwire and delete paths leave such a ModelConfig
alone. repository_unavailable (412) means a mode: commit call targets a
repository model-manager's GitHub App is not installed on (see
Commit mode); the message names
the repository and the App, and nothing was written. forbidden (403) means the cluster refused the caller: model-manager
acts as the caller, so their own Kubernetes RBAC decides; retrying does not
help, and backend_error (502) stays for a backend that failed.
model-manager only deletes what it created. Every model-manager writing
ModelConfigs into a kagent namespace has an instance name (--instance,
MODEL_MANAGER_INSTANCE; the chart sets <release namespace>-<release name>,
a process outside the chart defaults to its host name — give each process on
one host its own). A ModelConfig it creates carries the label
model-manager.giantswarm.io/instance: <instance>, written once at creation;
app.kubernetes.io/managed-by: model-manager alone is shared by every
instance and copied with a manifest, so it never grants a deletion. The unload,
unwire, delete and remove_backend paths delete only a ModelConfig carrying
their own instance label, and unwire_model also one without any instance
label (below). Any other is left in place: the unload answers
modelConfigLeft: {namespace, name, createdBy?, message} (its dry run
wiring.left), remove_backend leaves another instance's and refuses while one without the label remains (conflict naming it: the backend would go and the ModelConfig would point at nothing), and unwire_model and
delete_model with unwire answer conflict (409) with that message, nothing
deleted. A wire onto a ModelConfig another instance created answers conflict
naming it, nothing written; one with managed-by: model-manager but no
instance label (written before the label existed, or by hand) is adopted —
its spec written, never marked as created here — and only unwire_model
removes it: once no Agent, AgentTemplate or Harness in its namespace
references it (its dry run naming it, mode commit where Flux applies it from
git), whether or not its backend is still registered; while one does, the
answer is conflict naming them (referencedBy).
A ModelConfig without managed-by: model-manager is never written or deleted.
ModelConfigRef.createdBy reports the instance label, and get_info's
wiring.instance this process's name.
Several backends in one process. Every Model, LoadedModel, Job,
NodeInfo, Preset, FitResult and ModelConfigRef carries backend. Reads
without ?backend= aggregate every backend; a backend that fails to answer is
reported in the response's errors ({"lemonade": "connection refused"})
while the others' items are returned — 502 only when every backend failed.
An unknown or unconfigured backend name answers 400. kagent ModelConfigs are
one per (backend, model): the label model-manager.giantswarm.io/backend plus
the model annotation identify them, and the same reference on two backends is
two ModelConfigs (the second named <derived>-<backend>).
On kserve, pull and load accept preset and node; a model is wired into
kagent by the load call itself — before it is ready, at the address it will
answer on; the answer's wiring names the ModelConfig — and unwired on unload,
never after a pull, since a cached model has no endpoint. A load job follows
the model to readiness and refreshes the ModelConfig from the address KServe
published; the job lives in the process, so a restart loses only its progress
entry. A served model model-manager manages that has no ModelConfig is wired
by the caller's next list_loaded_models / GET /api/v1/loaded, whose entry
says so (wiring.reason: "wired on read").
Capability flags: pull, pullProgress, delete, load, unload,
loadedModels, wire (Kubernetes access present), presets, fitCheck,
nodeInventory, search (presets, fitCheck and search are kserve
concerns and false on ollama and lemonade).
GET /api/v1/backend reports two addresses: endpoint, the backend as
model-manager dials it, and agentEndpoint, the backend as agent pods dial
it — the host written into ModelConfigs (ollama: --ollama-agent-host,
defaulting to the endpoint; lemonade: --lemonade-agent-host plus /api/v1,
the OpenAI-compatible base URL the ModelConfigs carry). A client that matches
ModelConfigs it did not
create to models by hostname (the portal's "Used by") compares against
agentEndpoint. kserve reports the models Gateway's origin every served model
is routed under, and omits it without the Gateway, when every served model has
its own address (running.endpoint, modelConfig.endpoint).
On ollama, GET /api/v1/nodes reports the proxied host as one node so a
laptop install has capacity data too. Ollama's API has no capacity endpoint,
so budgetBytes is MemTotal of /proc/meminfo as the model-manager pod
sees it (budgetSource: host-meminfo) — on a unified-memory machine GPU memory
is system memory, so that is the budget that matters; reservedBytes is the
sum of size over Ollama's /api/ps (weights plus KV cache for the loaded
context, which is why a 500 MiB download can reserve 5 GB), accelerated says
whether any loaded model has memory on an accelerator (size_vram > 0), and
each model's own share stays in running.vramBytes. There is no gpuCount,
gpuProduct or cache — Ollama does not expose the accelerator, and its model
store is not a node cache. Caveat: the pod reads the kernel it runs on. On kind
or any install sharing the machine's kernel that is the machine's RAM; under a
VM-backed container runtime (Docker Desktop, Podman machine) it is the VM's,
and for an Ollama on another machine it says nothing about that host. For those
installs the chart value ollama.memoryBudgetGiB (--ollama-memory-budget-gib,
MODEL_MANAGER_OLLAMA_MEMORY_BUDGET_GIB; GiB, decimals allowed) sets the budget
instead: budgetBytes is that figure with budgetSource: override, message
says so, and allocatableMemoryBytes stays the pod's MemTotal — the ollama
counterpart of the kserve node annotation
model-manager.giantswarm.io/memory-budget-gib. A value that is not a positive
number of GiB is ignored and named in message; the budget then comes from
/proc/meminfo as before.
On lemonade the host node comes from Lemonade's own GET /api/v1/system-info
instead (budgetSource: system-info, the accelerators Lemonade enumerates as
gpuCount / gpuProduct, the model store as cache) — see
The lemonade backend.
GET /api/v1/backend also carries a loading block so a client can word a
not-loaded model correctly without keying off the backend name:
"loading": { "onDemand": true, "idleEviction": true, "keepAliveDefault": "5m", "keepAliveScope": "request" }- ollama —
onDemand: true: Ollama loads a model on the first/api/chat(or/api/generate) that names it, so an agent whose ModelConfig points at a not-loaded model works and its first turn pays the cold start. "Not loaded" is idle, not broken.idleEviction: true: Ollama evicts a model when its keep-alive runs out.keepAliveScope: request: the keep-alive is set per request — Ollama's scheduler takes each request'skeep_alive, else the server'sOLLAMA_KEEP_ALIVE(5m unless set), and re-arms the runner's timer on every hit. kagent's Ollama provider sends nokeep_alive, so every agent turn re-arms the server default.POST /api/v1/models/load(a generate call withkeep_alive, default--default-keep-alive/MODEL_MANAGER_DEFAULT_KEEP_ALIVE, reported askeepAliveDefault) therefore only pre-warms: it answers with the loaded model andrunning.expiresAt— the deadline Ollama reports right then — and the next agent request resets the timer to the server default again, even after a load with-1. - kserve —
onDemand: false,idleEviction: false, no keep-alive fields: a stopped LLMInferenceService does not come back on request, agents on its ModelConfig fail at their first turn, and a running one stays until unloaded. - lemonade —
onDemand: true: Lemonade loads a model on the first completion naming it (a few seconds for a 4B model on the NPU), so a not-loaded model is idle, not broken.idleEviction: false, no keep-alive fields: nothing evicts an idle model, but a loaded model gives way when another model of its type is requested and the slot is taken (max_loaded_models, one per type by default; least recently used first) unless it was loaded withkeepAlive: "-1", which pins it. - lmstudio —
onDemand: true: with LM Studio's just-in-time loading on (its default) the first completion naming a downloaded model loads it, so a not-loaded model is idle, not broken.idleEviction: false, no keep-alive fields:POST /api/v1/models/loadis an explicit load, which LM Studio treats as manual — no idle TTL, and Auto-Evict leaves "non-JIT loaded models" alone — so a model loaded through model-manager stays resident until something unloads it. The endpoint accepts nottl, which is why a keep-alive has nothing to map onto. The two paths differ, and it matters for agents: a model LM Studio JIT-loaded (what an agent turn does to a not-loaded model) does get its default idle TTL — 60 minutes — and is subject to Auto-Evict, so it can be gone later, while one loaded through this backend will not be.loadingdescribes the backend's own loads. Turning just-in-time loading off in LM Studio is what breaks an agent on a not-loaded model — the request then fails instead of waiting.
The knob that changes what agents experience on Ollama is host-side, not a
model-manager flag: set OLLAMA_KEEP_ALIVE=30m (or -1 for never) in the
Ollama service environment — the systemd unit's Environment= on Linux —
and restart Ollama. keepAliveDefault is model-manager's default for its own
load requests; the host's OLLAMA_KEEP_ALIVE is not observable through the
API, which is why the block does not claim to report it.
The context window is the other thing agents feel on Ollama, and
model-manager sets it. A request without num_ctx runs at a default Ollama
derives from the host's VRAM: 4,096 tokens below 24 GiB (every CPU-only
host), 32,768 up to 48 GiB, 262,144 above. A longer prompt is not refused.
Ollama keeps its first tokens and its tail and drops the middle, so an agent
turn loses its system prompt and tool schemas, and the only trace is a
truncating input prompt warning in the server's log. Every Ollama
ModelConfig model-manager creates or re-wires therefore carries
spec.ollama.options.num_ctx. The value is the chart value
ollama.contextLength (--ollama-context-length,
MODEL_MANAGER_OLLAMA_CONTEXT_LENGTH, default 32768), capped at the model's
own context length. It is reported as modelConfig.contextLength, next to
the model's own contextLength. Loads pre-warm at the same value, since
Ollama reloads a model when a request's num_ctx differs. With auto-wiring,
the reconciler brings a ModelConfig written under another value, or before
the setting existed, to the configured one. The window costs host memory:
Ollama reserves the KV cache for all of it when it loads the model. A
ModelConfig written by hand gets none of this and depends on the host's
OLLAMA_CONTEXT_LENGTH.
Thinking is the third. A model with Ollama's thinking capability (Qwen3.5,
Qwen3, DeepSeek-R1, Granite 4.2, ...) reasons before every answer unless the
chat request says think: false: 1.5–2k tokens per model call, minutes on a
CPU. A small one such as qwen3.5:2b can put its whole answer into the
thinking, which the agent runtime does not read, so the agent replies with
nothing. The ModelConfig of such a model therefore carries
spec.ollama.think, the chart value ollama.think (--ollama-think,
MODEL_MANAGER_OLLAMA_THINK): false by default, true to have the model
reason first, or empty to write none and leave Ollama's default. A model
without the capability gets none, since Ollama answers true for it with a
400. The value is reported as modelConfig.think. kagent's ModelConfig has
the field from kagent 1.0.3 on; model-manager reads the ModelConfig schema the
apiserver serves (its OpenAPI v3, refreshed every minute) and writes think
only where it is there, since the apiserver prunes a field its CRD lacks. With
auto-wiring, the reconciler brings a ModelConfig whose think differs from
the configured one to it, in the same comparison that covers the context
window.
model-manager starts with no backend (chart default backend: "", backends: []) and gets its
backends at runtime: add_backend writes a backend document — a ConfigMap in model-manager's
namespace labelled agent-platform.giantswarm.io/model-backend=true, named model-backend-<name> (the backend's name, default its kind),
key backend.yaml, kind: ModelBackend — as the caller; model-manager watches the label, enforces
the document's schema when it reads it (an invalid document is reported under invalid in
list_backends with the failing field, and not loaded) and registers the backend without a
restart. remove_backend drops the backend's ModelConfigs and the document, or refuses while a ModelConfig of the backend it cannot delete would be left pointing at nothing. One backend per kind, and for kserve one per serving cluster (kserve, kserve-<cluster>);
list_backends reports every backend with source: static | person | cluster-manager, and a
kserve document carries the target cluster ({cluster, organization, apiServer, caBundle, servingNamespace} — never credentials) the backend acts on, reported as target on the backend,
its models, nodes and presets. Static --backends values keep working as source: static; a
document naming a static kind is refused. With no backend at all list_backends and get_info
answer an empty list with backendsReason (the no_backend wording, so a caller tells no backend
from a read that missed one), and every backend-scoped call answers no_backend with the fix —
except check_fit of a serving preset the platform chart publishes
where model-manager runs, which answers what the preset declares with verdict: unverified, so a
GPU pool is sized before it, and the backend, exists. The contract — the label, keys, schema,
tools, RBAC and what cluster-manager writes — is docs/backends.md.
Lemonade Server is AMD's local model server: one
OpenAI-compatible API (/api/v1) in front of several runtimes — "recipes" —
of which FastFlowLM (flm) runs models on the Ryzen AI NPU and llama.cpp
(llamacpp) on the GPU (Vulkan, ROCm) or CPU. Like Ollama it ships its
management surface on the same port, and the driver has the same shape as the
ollama one: a proxy of GET /api/v1/health (the loaded models),
GET /api/v1/models, the streamed POST /api/v1/pull, POST /api/v1/load,
POST /api/v1/unload, POST /api/v1/delete and GET /api/v1/system-info.
Lemonade stays on the host — that is where the NPU driver is — and pods reach
it through the docker network gateway, so bind it to every interface
(lemonade config set host=0.0.0.0; its port is 13305 by default) and let
the bridge subnets through the host firewall, as for Ollama.
- Inventory —
GET /api/v1/modelslists the downloaded models: the Lemonade id is thename(qwen3-it-4b-FLM,Qwen3-0.6B-GGUF),sizein GB becomessizeBytes, the recipe is reported asruntime(flm,llamacpp, ...), the format and quantization are read off the checkpoint (unsloth/Qwen3-0.6B-GGUF:Q4_0→gguf,Q4_0) and Lemonade's labels are mapped onto the capability vocabulary the other backends use (chat→completion,tool-calling→tools,reasoning→thinking,embeddings→embedding,visionstays; a text recipe without a deployment label is acompletionmodel). Everything Lemonade has downloaded is listed — transcription, image or speech models included — and the capabilities say what each one is. - Pull —
POST /api/v1/models/pulltakes a Lemonade catalog model name (lemonade list, orGET /api/v1/models?show_all=trueon the server) and streams Lemonade's install (stream: true): the job reports bytes done against the whole download when Lemonade says how big it is, else against the current file. A name that is not in the catalog fails withnot_found(registering auser.*model from a Hugging Face checkpoint is not offered here). On success the model is wired into kagent, as on ollama. - Load / unload —
POST /api/v1/loadstarts the model's backend process (FastFlowLM on the NPU, llama.cpp, ...) and answers once it is ready; a load is bounded by ten minutes, not the usual call timeout. Lemonade has no keep-alive: a loaded model stays until it is unloaded or until another model of its type needs the slot (max_loaded_models, one per type by default; least recently used first).keepAlive: "-1"pins the model against that displacement (Lemonade'spinned, reported asrunning.pinned); every other keep-alive is ignored.running.devicesays where the model runs (npu,gpu,cpu). Unloading a model that is not loaded is a no-op. - Wiring — a kagent
ModelConfignamed after the model (qwen3-it-4b-FLM→qwen3-it-4b-flm) withprovider: OpenAI,openAI.baseUrl= the agent host plus/api/v1(--lemonade-agent-host, defaulting to the endpoint; reported asagentEndpoint) and the placeholderOPENAI_API_KEYSecret the kagent runtime insists on — the same path the kserve backend takes to a vLLM workload. - Node —
GET /api/v1/nodesreports the host as Lemonade sees it: memory fromPhysical MemoryofGET /api/v1/system-info(budgetSource: system-info), the accelerators Lemonade found available asgpuCount/gpuProduct(the NPU when there is one —AMD NPU (NPU Strix)— else the first GPU), reservations as the catalog sizes of the loaded models (Lemonade reports no per-model memory), and Lemonade's model store as the node'scache(mountPath, models, bytes).
Nothing here needs Kubernetes access beyond wiring, exactly as for ollama.
LM Studio is a desktop model runner — llama.cpp on GPU
and CPU, MLX on Apple silicon — with its own API (/api/v1, 0.4.0 and newer)
next to an OpenAI-compatible one (/v1). The driver proxies the library
(GET /api/v1/models), the download that backs a pull
(POST /api/v1/models/download plus
GET /api/v1/models/download/status/{job_id}) and
POST /api/v1/models/load / POST /api/v1/models/unload. Like Ollama and
Lemonade it stays on the host, so serve it on the local network
(lms server start --bind 0.0.0.0, the app's Developer → "Serve on Local
Network" toggle, or LMS_SERVER_HOST=0.0.0.0; its port is 1234 by default)
and let the bridge subnets through the host firewall.
Two traits set it apart from the other host backends:
-
It has no delete. Removing a model is
lms rmon the host, which a pod cannot run, so the driver reportsdelete: falseand the API answers501 unsupported. UsePOST /api/v1/models/unwireto drop just the ModelConfig; the weights stay on the host until someone removes them there. -
It has no version or health endpoint, and it answers HTTP 200 with an
{"error": …}document for every path outside/api/v1— Ollama's/api/version,/api/tagsand/api/showincluded. Nothing may read a status code as proof that it reached an LM Studio: health here meansGET /api/v1/modelsanswering with amodelsarray, andversionstays empty because the server reports none. -
Inventory —
GET /api/v1/modelsis the local library, so every entry is downloaded: thekeyis the name (ibm/granite-4-micro),size_bytesbecomessizeBytesunchanged,formatandquantization.nameare reported as they come (gguf,Q4_K_M),architectureasfamily,max_context_lengthascontextLength, and the capability object is mapped onto the vocabulary the other backends use (trained_for_tool_use→tools,vision→vision,reasoning→thinking, on top ofcompletion). Embedding models carry no capability object at all and are reported asembedding, listed like any other model — as on lemonade. The vision-language type (vlm) is a chat model: it is reported withcompletionplus whatever its capability object says, not as an embedding.trained_for_tool_useis the flag that matters for agents: LM Studio will accepttoolsfor any model and emulate them through the prompt, but only a model trained for them calls them reliably. -
Pull —
POST /api/v1/models/downloadtakes an LM Studio hub reference (lms lsfor what is already there) and answers with a job; the driver follows it throughGET /api/v1/models/download/status/{job_id}and reportsdownloaded_bytesagainsttotal_size_bytes, so progress is real bytes. A model that is already downloaded reports complete straight away (already_downloaded), and a reference LM Studio does not know fails withnot_found. On success the model is wired into kagent, as on ollama. One caveat the API forces: LM Studio does not promise that thekeya download lands under is the reference that was asked for — the download answers no key, there is no resolve call, and Hugging Face artifacts are keyed per weight set (sometimes with an@<quant>suffix) while catalog models keep their canonicalpublisher/model. Since the ModelConfig is wired with the reference the caller gave, the pull resolves it against the library before reporting success and fails naming the mismatch, rather than leaving a ModelConfig whose model the server does not serve. -
Load / unload —
POST /api/v1/models/loadreads the weights and answers with an instance id; a load is bounded by ten minutes, not the usual call timeout. LM Studio evicts an instance, not a model, so unload looks the model's loaded instances up first; unloading a model that is not loaded is a no-op, as on ollama and lemonade. A load of an already-resident model is a no-op too, for the same reason the instance list exists: LM Studio loads an instance, not a model, so a repeated load would pin a second copy of a multi-GB model rather than answer "already loaded". Neither call takes a 2xx as proof: both answers have to name the instance, because an LM Studio without these endpoints answers 200 with an error document. There is no keep-alive and no pinning. -
Wiring — a kagent
ModelConfignamed after the model (ibm/granite-4-micro→ibm-granite-4-micro) withprovider: OpenAI,openAI.baseUrl= the agent host plus/v1(--lmstudio-agent-host, defaulting to the endpoint; reported asagentEndpoint) and the placeholderOPENAI_API_KEYSecret the kagent runtime insists on. LM Studio needs no key of its own. -
Node — none. LM Studio exposes no host hardware, so the driver reports
nodeInventory: falseandGET /api/v1/nodesanswers501 unsupported.
Nothing here needs Kubernetes access beyond wiring, exactly as for ollama.
The driver consumes the modelServing contract of the
giantswarm/agent-platform
meta chart, rendered by its agent-platform-connectivity chart:
the discovery ConfigMap agent-platform-model-serving (kind
ModelServingConfig) for the serving namespace, runtime, GPU resource name,
cache claim and preset selector, the GPU node pool's scheduling
(spec.gpuPool: the pool's taint and label) and the models Gateway
(spec.gateway.endpoint, https://models.<domain>: the origin every
LLMInferenceService is routed on at /<namespace>/<name>, which is how a
served model's address is known before KServe publishes it); the ServingPreset ConfigMaps
(agent-platform.giantswarm.io/serving-preset=true); the cache
PersistentVolumeClaim in the serving namespace. Every discovered value can be
overridden by a flag (model-manager serve --help, --kserve-*); the pool
scheduling by the registered backend document (docs/backends.md).
-
GPU node pool — a pool created through the platform is tainted
nvidia.com/gpuNoScheduleand labelledgiantswarm.io/machine-pool=<cluster>-<pool>. The discovery ConfigMap'sspec.gpuPool.taint/spec.gpuPool.nodeSelector(the chart'smodelServing.gpuPool.*), or the backend document'sspec.kserve.gpuPool, make the driver tolerate the taint on everything it schedules onto the pool — the composed workloads, the download Jobs and the inventory scan Jobs — and select the pool wherever the pod is not pinned to a node already. The descriptor (GET /api/v1/backend) reports it asgpuPool. Unset, nothing changes; the chart's DaemonSet takes its ownkserve.inventory.agent.*. The document is read on the first call and cached for a minute; settings resolved while it was not published yet — the slice publishes it as it installs — stand for five seconds only, and a fit that finds no node while they lack it reads the document once more before answering: with the pool now known, the scale-from-zero path; still without it,fits: false,retryable: trueand a reason naming the document (the serving layer's discovery document <namespace>/<name> is not published yet — the slice is still installing; retry in a moment), whichload_modelechoes in its refusal.no accelerator nodeis the verdict for a document that names no pool. -
CPU presets — a preset whose
resources.gpusis0is CPU work: the fit judges every node of the cluster against its allocatable memory (the accelerator nodes and the CPU nodes alike, a node'sEligiblerules as usual), and the preset knows no GPU pool and no accelerator RuntimeClass — its predictor carries neither the pool's toleration and label norruntimeClassName, so on an installation whose GPU pool is tainted it lands on the cluster's CPU capacity. A bare model reference and every preset that requests a GPU keep the accelerator nodes as their capacity;GET /api/v1/nodeslists those. -
Inventory — the cache contents per node plus the LLMInferenceServices of the serving namespace (readiness from conditions/
modelStatus, node from the workload pod, GPU request, address). The cache is read in one of two ways (kserve.inventory.mode): a short-lived scan Job per cache node mounts the claim read-only and walks<claim>/<dir>whenever the inventory is older than the TTL (default; never while the cache claim is unbound or the GPU pool has no node, and in the background when the caller's deadline leaves no room for a pod), or a DaemonSet ofmodel-manager cache-agentpods (same image) mounts the claim on each selected node and serves the same walk atGET /inventory, which model-manager reads from the agent on the node — no pod churn, andnodes[].cache.inventorysays which mode produced the numbers. A node without a ready agent reportscache.error. The node holding the cache comes from the bound PersistentVolume's node affinity.GET /api/v1/nodessays which nodes hold a cache at all and what their memory budget is. -
Naming a cache directory — a directory is named by what is known to have filled it: the marker a pre-warm download Job wrote, else the cache index — a ConfigMap in the serving namespace (
model-manager-cache-index,kserve.cache.indexConfigMap) in which the driver records, while an LLMInferenceService exists, that its name is the directory the KServe storage-initializer fills and itshf://storageUri the repository (repository, revision, preset label, one JSON entry per directory). The record outlives the LLMInferenceService, so the directory keeps its repository and, through it, its preset (the labelled one, else the single preset serving the repository) after the LLMInferenceService is deleted; a live LLMInferenceService always wins over a stale record, and the record is dropped when model-manager removes the directory. A record is bound to the cache it was made against (claim,volume): nothing is recorded while the serving layer has no cache (cache.enabled: falsein the discovery document, a missing or unbound claim — the storage-initializer then fills storage that goes with the pod),unload_modeldrops a record that names a directory in no cache (bound to another claim or volume, or to none), and the ConfigMap goes with its last record. Without a marker or record the preset or LLMInferenceService of the same name is assumed, else the directory is listed by its bare name. Directories whose top level holds no model — noconfig.jsonand no weights file (*.safetensors,*.gguf,*.bin,*.pt,*.pth,*.onnx) — are not listed as downloads: Hugging Face client internals such ashf-homeandxetlive on the same claim and count towardsnodes[].cache.bytesUsed, but they are not models, and a directory an LLMInferenceService is still filling shows as that served model withdownloaded: falseuntil its files arrive. -
Nodes and eligibility —
GET /api/v1/nodeslists the accelerator nodes only: nodes that advertise the configured GPU resource (gpuResourceNamefrom discovery, defaultnvidia.com/gpu; capacity or allocatable > 0) or carry a gpu-feature-discovery label (nvidia.com/gpu.present,.count,.product). CPU-only nodes are not serving capacity for this backend. Each node says whether a model can be served there right now:eligibleis true when the node is ready, matches the discoverynodeSelectorand the GPU pool's, carries noNoScheduleorNoExecutetaint the pool toleration does not cover (a tainted pool node is capacity once its taint is the configured one), and can mount the cache claim whenever workloads mount it (cache enabled andcache.redirectPolicyon — the Kyverno rule that mounts the claim into every workload) and the claim is pinned to nodes (a static local PersistentVolume or alocal-pathvolume); a shared (RWX), unbound, missing or disabled cache never disqualifies a node.eligibilityReasonnames every failing rule (not ready,outside the serving node selector (kubernetes.io/hostname=spark-8723),taint nvidia.com/gpu:NoSchedule not tolerated (set the GPU pool taint),cache claim hf-cache is pinned to spark-8723).pull,loadandfit-checkrefuse an explicitnodethat is not eligible with that reason (412 does_not_fit;fits=falseon the fit check) before any Job or LLMInferenceService exists, and never pick an ineligible node themselves. Where the platform's redirect rule mounts the claim a model pod names (discoverycache.nodeClaims: true; pod inventory mode), the pin disqualifies no node: every node outside the claim's reach gets a cache claim of its own,<claim>-<node>(labelledmodel-manager.giantswarm.io/component: node-cache, annotatedmodel-manager.giantswarm.io/cache-node: <node>), of the claim's class, access modes and size.pullwith thatnodecreates it, provisioned on the node, and its download Job fills it; a claim bound statically (no StorageClass) is refused asinvalid, naming the claim to create by hand.list_nodeslists each node's own cache,fit-checkandloadread it per node, and the composed object names the claim on its pods (spec.annotations:model-manager.giantswarm.io/cache-claimfor the leader or a single-node model,model-manager.giantswarm.io/worker-cache-claimfor a split's workers; the claim itself on its own nodes, without an annotation). A split is refused while a node's cache lacks the weights (the cache of <node> holds none: pull_model <model> with node <node> first), and over more than two nodes when its workers would need different claims; copies whose nodes mount different claims are refused (one pod template mounts one claim). Withoutcache.nodeClaimsa second node-local GPU node becomes a serving target only with shared storage. The claim has a say for the presets that store in it only — every download scheme (hf://and, with the redirect policy on, the rest) andpvc://. Anoci://preset (KServe's modelcar: an OCI image containerd pulls onto the node's own disk; no storage-initializer, no claim mount) is judged without the claim's location:fit-checkandloadplace it on any eligible GPU node with the budget, without a cache-node preference, and answercached: falsewithcacheSource: oci-image;pullrefuses it asinvalid— the image is the platform's to pre-pull; nothing to download — and creates no Job.nodesstill reports the pin: it describes the nodes, not a preset. -
Import —
searchproxies the Hugging Face Hub;fit-checkresolves the weight size (model.safetensors.index.json— itstotal_size, or the shards itsweight_mapnames when they disagree with it by more than one percent — else the file tree, else the preset), adds the preset'soverheadGiB(default 30) and compares with the node's free budget: the budget (nvidia.com/gpu.memoryxgpu.countlabels when present, else allocatable memory — unified-memory nodes; a node annotationmodel-manager.giantswarm.io/memory-budget-gib: "96"overrides that node's budget in GiB whateverkserve.budget.sourcesays, reported asbudgetSource: annotation— for unified-memory nodes whose allocatable memory overstates what a model may use) less what the models already served on the node reserve (reservedBytes, the one readingloadandnodesmake too; the preset being loaded is not counted against itself: each served preset's weights and overhead, and on a unified-memory node — GPUs without a memory label of their own — at least the share vLLM claims at start,--gpu-memory-utilizationof the node's budget, 0.9 when the preset sets none;list_presetsreports it asgpuMemoryUtilization), sofit-checkandloadreach one verdict;pulljudges against the whole budget and refuses what cannot be served, then runs a download Job with the KServe storage-initializer image into<claim>/<preset name>— the directory the preset's LLMInferenceService mounts — reporting bytes on disk against the repository size. The Job downloads withHF_HUB_DISABLE_XET=1andHF_HUB_ENABLE_HF_TRANSFER=1: the Xet client connects to CDN addresses a CiliumtoFQDNsDNS proxy never resolved for an allowed name and hangs on the dropped SYNs, hf_transfer resolves per connection through the system resolver, so every address is admitted. A download that writes nothing to its directory for--kserve-download-stall-timeout(kserve.download.stallTimeout, default 10 min) fails with a reason the pull job shows —DOWNLOAD STALLED: no bytes written to <dir> for <N>s (<bytes> bytes on disk, …)— instead of hanging; partial files stay and a newpullresumes. Gated models need a token Secret (--kserve-hf-token-secret). The hub lookups of one fit check or search are bounded by--kserve-hf-timeout(kserve.hf.timeout, default 4 s): a hub that does not answer — egress blocked — letsfit-checkfall back to the preset's requirements in time for the caller (weightsSource: preset;reasonsays the hub did not answer within the timeout), and a model no preset serves fails with that message. On a GPU pool with no node yet whose instance shapes the backend document lists (docs/backends.md) the check is against the pool's sizes, and the GPU memory budget is that of the GPUs the preset requests (resources.gpus, default 1) on a size — not the node's: a one-GPU preset on a four-GPU size has one card. A preset's declared weights (requirements.weightsGiB, what the pool was sized from) are answered asdeclaredWeightsBytesbeside the hub-sizedweightsBytes; when the hub holds more than the preset declares,reasonsays so on both verdicts (…; the preset declares 15.0 GiB of weights, the Hub holds 24.6 GiBon a fit;…; the preset declares 15.0 GiB of weights; the Hub holds 24.6 GiB, which is what does not fit — correct the preseton a refusal). The fit also checks vLLM's own admission, which a flat overhead cannot: vLLM starts only when its KV cache holds one sequence of--max-model-lentokens beside the weights, and crash-loops otherwise. From the checkpoint'sconfig.json(itstext_configwhen nested) and the preset's--max-model-len,--gpu-memory-utilization(default 0.9),--kv-cache-dtype(default: the checkpoint's KV quantization, else the model's dtype),--max-num-batched-tokens(default 2048, 8192 on a GPU of 70 GiB or more that is no A100),--block-size(16) and--tensor-parallel-size, the check computes vLLM v0.23's figure per GPU: a full-attention layer holds the sequence (2 × KV heads × head dim × dtype bytesper token; Gemma 4's global layers with their own heads), a sliding-window layer its window and one prefill chunk, an MLA layer its latent (kv_lora_rank + qk_rope_head_dim), a linear-attention or Mamba layer none, in 16-token blocks. The GPU leaves the KV cache--gpu-memory-utilization × its memory(a node'snvidia.com/gpu.memory; a pool size'sgpuMemoryGiBin decimal GB) less the weights and a 3.15 GiB reserve — the activations, CUDA graphs and runtime vLLM profiles and what loading adds to the checkpoint, measured with Gemma 4 31B on one L40S. When one sequence does not fit,fitsis false andreasonnames the KV need, what is left and vLLM's estimated maximum model length (kvCacheBytes,kvCacheAvailableBytes,estimatedMaxModelLen):gemma-4-31bat 65536 tokens on a 48 GB card at 0.92 needs 12.0 GiB of KV cache beside 31.0 GiB of weights where 7.3 GiB are left, and 8624 tokens would fit. An architecture the check does not read, a checkpoint withoutconfig.json, a preset without--max-model-lenor a node without the GPU memory label is named inreason(the KV cache is not checked: …) and judged on the flat overhead alone. Every check the answer could not make is listed inskippedCheckswith why (download-size,kv-cache,compute-capability): a model image whose registry does not answer within the fit check's budget, its timed-out requests retried, leaves all three. The GPU generation is judged the same way, from the sameconfig.json(a mistral-format checkpoint'sparams.json): the weights' need — FP8 (quant_method: fp8, compressed-tensors 8-bit float, ModelOpt FP8) needs compute capability 8.9 (Ada, Hopper, Blackwell: below it vLLM falls back to the weight-only FP8 Marlin kernel, which answers wrong for such a checkpoint), NVFP4 (compressed-tensors 4-bit float, ModelOpt NVFP4) 8.6 (native on Blackwell, vLLM's weight-only FP4 Marlin kernel below it, proven on the A10G; nothing below 8.6 is proven), a checkpoint mixing FP8 and NVFP4 layers (ModelOpt mixed precision read from itsconfig_groups) 8.9 for its FP8 ones, bf16 8.0 (the preset's--dtypedecides unquantized weights); a preset whoserequirements.minComputeCapabilitydiffers is named — against a node'snvidia.com/gpu.compute.major/minorlabels or the GPU of a pool size's instance family (g4dn T4 7.5, g5 A10G 8.6, g6 L4 8.9, g6e L40S 8.9); a GPU below it is a refusal naming both (computeCapabilityRequired,computeCapability), and among several pools one whose GPU is below the need is skipped for one that is not. On a unified-memory node (GPUs without memory of their own, a GB10) vLLM claims--gpu-memory-utilization × the node's whole memoryat start, outside the pod's memory limit and the kubelet's accounting: that claim (unifiedReservationBytes) must fit the node's budget, at most its memory less the host headroom (hostHeadroomBytes,--kserve-unified-host-headroom-gib, chartkserve.budget.unifiedHostHeadroomGiB, 16 GiB), less what running models reserve there. A preset whose own claim does not fit is composed at the utilization its weights and overhead need there (gpuMemoryUtilization, in hundredths): a preset at 0.90 sized for a 24 GB GPU (21.8 GiB) runs at 0.18 on a 121.7 GiB GB10. Only when that does not fit either isfitsfalse,load_modelrefuses, andreasonnames the claim and the utilization. The claim must also hold the weights (a split's share) and overhead: a preset whose claim is short of them is refused, never sized up, andreasonnames the utilization that holds them (fitGpuMemoryUtilizationwhen the node leaves it). Weights the runtime keeps on disk (an n-gram table read from there) do not count against the claim: a preset declares what it holds in memory asrequirements.residentWeightsGiB. A dedicated GPU is judged as before. Serving shape — a preset'sresources.gpusand--tensor-parallel-sizedescribe its reference shape, the node it is written for; the fit derives the shape from the node the model is placed on (devicesPerPod,tensorParallel,gpuMemoryUtilization). On a unified-memory node (one GPU without memory of its own) a pod requests one device; on a node whose GPUs have memory of their own, the smallest power of two whose cards hold the weights and overhead (a preset written for four 48 GB L40S cards keeps four on ag6e.12xlarge). A split requests that count on every pod and runs tensor parallel over devices × nodes. A shape that differs from the reference is named inreason; more devices per pod than the node has allocatable answerfits: false, and copies on nodes of different device counts are refused. The preset's CPU and memory requests are capped at what the node has left beside the requests of its other pods (cpuRequestMillis,memoryRequestBytes); a node with nothing left of one answersfits: false. The composed object carries the derived device count,--tensor-parallel-size,--gpu-memory-utilizationand any capped request; on its reference node a preset is composed as written. -
Serve / stop —
loadcomposes the serving object from the preset after the fit checkfit-checkanswers (the node's free budget, the KV cache);unloaddeletes it and unwires within the caller's deadline — the object is found by repository, object or preset name, never through the cache inventory, so a scan pod that cannot start never delays the deletion — and rescans the cache in the background where a scan pod may run, the answer'sinventorysaying whether one runs or why none can (the cache persists);deleteremoves the cache directory (refused while served). The object is anLLMInferenceService(serving.kserve.io/v1alpha2, KServe's llm-d control plane), the only kind the backend composes, lists, stops and deletes; a loaded model names it as itskind. A load fails fast, with nothing created, where no llm-d control plane would reconcile the object (503 unavailable, MCPunavailable): the API is not served, or the CRDs are installed without the controller — the backend looks for the well-knownLLMInferenceServiceConfigkserve-config-llm-templatethe controller composes from, which ships with the platform'skserve-runtime-configscomponent, not with the CRDs — or the driver can read neither (the error names the permission:get/listonllminferenceserviceconfigscluster-wide, which the chart's ClusterRole grants).get_backend/GET /api/v1/backendscarry the same reason inmessagebefore anyone tries. While a backend's serving slice is still installing — the control plane not ready and no preset published yet —list_presetsand aget_modelorload_modelof a model it does not know yet answer the sameunavailable,backend <name> is not readywith what it waits on, instead of an empty list ornot_found; once ready they answer as before. While the llmisvc admission webhook has no ready endpoint (the minute or two after the serving slice is installed:no route to host,connection refused,no endpoints available) the create is retried every poll interval for up to two minutes within the call's deadline; a webhook still unreachable then is503 unavailablenaming the webhook, with nothing created. The verdict is taken on the cluster as it is at the load: settings resolved without the control plane stand for five seconds, not the minute the settings cache keeps otherwise, and a load about to be refused resolves them once more first — the serving slice's CRD and runtime-configs children land seconds after its discovery document. With that document published the refusal says the slice is most likely still landing and to retry in a moment; without it, which components to turn on. The composition, by spec shape:spec.model.urifrom the preset'sstorageUri(hf://,pvc://into the cache, or anoci://model image),spec.model.namefrommodel.id,replicas: 1,router: {route: {}}(KServe renders theHTTPRouteon the configured ingress gateway with the workload Service as its backend; the llm-d endpoint picker,router.scheduler, is opt-in through the backend document'sspec.kserve.router.scheduleror a preset'sspec.router.scheduler, and itsInferencePoolneeds the Gateway API Inference Extension on the gateway —docs/backends.md),template.containers[main]with the preset'sargs,envandresources(GPU count under the discovery's resource name; for anoci://preset the env ends with the modelcar environment —HOME=/tmp,HF_HOME=/tmp/hf,VLLM_CACHE_ROOT=/tmp/vllm-cache,TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor,USER=vllm,LOGNAME=vllm, each only where the preset sets no value of its own —, since the modelcar uid the runtime runs as has no passwd entry and a read-only image),schedulingas the template'snodeSelector/tolerations(merged with the discovery selector, the GPU pool's label and toleration, and the node pin), the chat template mounted, andtemplate.runtimeClassNamefrom the discovery ConfigMap'sruntimeClassNamewhen non-empty — absent when empty. NobaseRefs: KServe chooses its well-knownLLMInferenceServiceConfigs from the spec's shape and appends them itself; a preset that names a custom config (spec.baseRefs: [{name: …}]) is the onlybaseRefscase. No image: the container runs the image the well-known template names (llm-d-cuda, mirrored through the platform's registry override); a preset overrides it withspec.template.containers[{name: main, image: …}]when it has a reason —spec.templateextras are copied on top, containers merged by name. model-manager creates and lists noLLMInferenceServiceConfig. -
Wiring — in the load call, before the model is ready, a kagent
ModelConfignamed after the LLMInferenceService (provider: OpenAI,baseUrl= the model's address +/v1,model= the served model name: the object'sspec.model.name, which the well-known template passes to vLLM) — the same rule the portal's serve flow applies; theloadanswer'swiringnames it, theloadjob refreshes it from the address KServe publishes once the model is ready, and a served model model-manager manages that has none is wired by the caller's nextlist_loaded_models(wiring.reason: "wired on read"). The address is the one KServe published or, until then, the one the object is expected on: the discovery document's models Gateway (spec.gateway.endpoint+/<namespace>/<name>), the in-cluster workload Service otherwise. The API key follows the address: a model routed on the models Gateway (an external host) getsapiKeyPassthrough: trueand no Secret — the agent forwards the person's own token, the only thing the Gateway's JWT policy admits, so a placeholder key would fail every turn with 401; a model reached on its in-cluster Service gets the placeholderOPENAI_API_KEYSecret the go ADK runtime insists on, which keyless vLLM never checks. A re-wire that moves a ModelConfig onto the Gateway removes its placeholder Secret. A ModelConfig that already points at the model (same host, same served model name), whoever created it, counts as the model's wiring: it is reported withmanaged: false, never duplicated and never deleted;unwire/unloadonly remove ModelConfigs model-manager created. -
Ownership — LLMInferenceServices in the serving namespace that model-manager created or that carry the
agent-platform.giantswarm.io/presetlabel (the portal's serve flow) can be unloaded here; hand-written ones are inventory only (409 conflicton unload;managedBysays who owns them). One Flux applies from git reports the Kustomization or HelmRelease asgitops, held to the Kustomization's inventory as the unload holds it: a leftover its Kustomization no longer lists is written live, unloaded and given copies here; one it lists is changed in git (mode: commit). -
State — the loaded models (
GET /api/v1/loaded,list_loaded_models) are every InferenceService and LLMInferenceService of the serving namespace whatever their readiness:statusReady,Pending(the workload pod waits for a node, the weights or an image —reasonUnschedulablewith the scheduler's message,DownloadingWeights,ImagePullBackOff),NotReady(the Ready condition'sreason, a failed load) orTerminating, withmessagein words.unloadtakes the preset name, the object name or the repository id and deletes the object in any state; its answer saysstatus: Terminatingand that the list shows the object until it is gone. Aload's answer names the object it created (running.resource,running.kind), carries the fit verdict it was judged by (fit, thecheck_fitshape) and the initialrunning.steps; a refusal (412 does_not_fit) carries the fit'sreason, the declared-weights clause included. -
Phases —
phaserefinesstatuswith where a serve is, andstepsis the whole timeline: one entry per phase in order, each{name, state: pending|inProgress|done|failed, since, finishedAt, reason, message}withsince/finishedAtfrom the pod's conditions, container statuses and Events (Karpenter's nomination, the kubelet'sPulling/Pulledwith the pull's duration, probe failures), so a caller shows elapsed time per step. The phases of a fresh serve on a scale-to-zero pool (proof 1: ≈ 12 min to Ready):phaseWhat is happening Read from schedulingno node has a free GPU; the autoscaler is asked and, once it nominated a NodeClaim, until that claim has launched an instance PodScheduled=False(Unschedulable),FailedSchedulingevents; Karpenter'sNominatedevent names the NodeClaim (karpenter.sh/v1, read as the caller):Launchednot yetTruekeeps the step withreason: NodeLaunching(Karpenter nominated NodeClaim gpu-l4-x7k2q; no instance has launched yet);Launched=False, or anInsufficientCapacityError/NodeClassNotReadyWarning event on a NodeClaim of the pool (in thedefaultnamespace, where a cluster-scoped object's events land — Karpenter deletes a refused claim within seconds and nominates another) givesreason: CapacityUnavailable. A capacity refusal is read whole: what the claim asked for — itsspec.requirementswhile it stands, else the pool's template (thekarpenter.sh/v1NodePool it is named after, read as the caller; the pool chart names the family and the sizes, composed to types) — and every refusal text (the claim'sLaunched=False, the events) for the sizes and the zone the cloud names and the zones it names as having capacity where the text is whole (Karpenter's event holds the first 300 characters of the cloud's answer: the first size and the zone; the claim's condition holds it whole), plus whether the serving namespace's cache claim is bound to a volume in that zone — the pool's pin, cluster-manager'spool.zones:Karpenter could not launch a node: 4 NodeClaims refused, the last (gpu-l40s-46flj) at 2026-09-18T12:24:14Z — InsufficientCapacityError: the cloud has no g6e.2xlarge, g6e.4xlarge or g6e.8xlarge capacity in eu-central-1b, the zone the pool is pinned to by the cache claim hf-cache (its volume lives there); it has capacity in eu-central-1a and eu-central-1c. The way out is a pool in eu-central-1a or eu-central-1c (create_node_pool with zones naming it and cache: false), or removing the cache claim so the next pool is not pinned; Karpenter retries while the pod waits. A pool that pins no zone lets a claim launch in every zone of its node class's subnets, and a fleet that launched nothing was refused them all: the zones are the claim's own zone requirement while it stands, else the node class's (ec2nodeclasses.karpenter.k8s.aws, read as the caller),requestedZonesnames them all,availableZonesnever a zone the same answer refused,pinnedByCacheis false, and the message readsthe cloud has no g6e.2xlarge, g6e.4xlarge or g6e.8xlarge capacity in any zone the pool allows (eu-central-1a, eu-central-1b and eu-central-1c). No zone is left to move to; wider sizes or another accelerator (a re-run of create_node_pool) give Karpenter more to choose from, and it retries while the pod waits(the node class unreadable: the zone the cloud named,at least). The step carries the same as fields —refusedInstanceTypes,requestedZones,availableZones,pinnedByCache(absent when the cache claim or its volume could not be read) — and the model'sreason/messagecarry the message. Each refused size carries its own reason: the cloud's fleet errors (InsufficientInstanceCapacity,VcpuLimitExceeded, any other code) are read from the same text, a size the capacity wording names is refused for capacity, and the sizes it does not name are the vCPU quota's while aVcpuLimitExceededstands (the quota error names no size), else another code's —the cloud has no g6e.12xlarge capacity in any zone the pool allows (eu-north-1a, eu-north-1b and eu-north-1c); g6e.24xlarge and g6e.48xlarge exceed the account's vCPU quota: its limit is 96 vCPUs for their instance bucket. No zone is left to move to; the way out is a vCPU quota increase for the instance bucket (EC2 Service Quotas); Karpenter retries while the pod waits, a refusal for the quota alone readsg6e.12xlarge, g6e.24xlarge and g6e.48xlarge exceed the account's vCPU quota: its limit is 64 vCPUs for their instance bucket. The way out is a vCPU quota increase for the instance bucket (EC2 Service Quotas) or smaller sizes (a re-run of create_node_pool), andrefusedInstanceTypesnames every size refused, whatever its reason. A refusal that is not the cloud's capacity wording keeps Karpenter's words; a read that fails — the claim, the events, the NodePool, the node class, the cache claim — is named in the message, never silent; ≈ 30 s from nomination to launchnodeStartingthe node exists — an instance launched — and its GPU is not allocatable yet the NodeClaim's Launched=True(since= its transition; the message names the node once the claim registered it), anominatedNodeName, the pod bound; the node's allocatablenvidia.com/gpu; ≈ 3.5 min until the node registers, ≈ 1 min more for the GPUdownloadingWeightsthe storage-initializerfills the cache directorythe init container; bytesTotal(the preset's weights),bytesCompleted(a bounded cache-agent scan while filling, cache-agent mode only),cached: truewhen it finished within 15 s — the claim held the weights (72 s for 8 GB, else 0.3 s)pullingImagethe kubelet pulls the runtime image the container Waiting(ContainerCreating),Pulling/Pulledevents (the message carries the duration); ≈ 4 minloadingvLLM loads the weights until the startup probe passes the container Running, notReady;Unhealthyevents say what the probe saw; ≈ 1 min. A runtime that died and is restarted by the kubelet keeps the step under way withreason: CrashLoopand a message naming the crash count, the exit code and the last error line of the crashed container's log (kubectl logs --previous, read as the caller):runtime crashed 2× (exit 1): PermissionError: [Errno 13] Permission denied: '/mnt/models/.cache/vllm'; a runtime that died on a CUDA out-of-memory aftermlock()refused it (could not lock … Cannot allocate memoryin the same log) gets the node's locked-memory limit named instead of an out-of-memory, with the fix: acontainerd.servicedrop-inLimitMEMLOCK=infinity,systemctl daemon-reload, restart containerd; a caller who may not readpods/loggets the message with the reason the line is missingroutingthe pod is ready, KServe resolves the route; once the object is Ready, until the model answered its first request ( reason: AwaitingFirstAnswer, the message says why no answer came; asked again after a minute)Ready=FalseHTTPRoutesNotReady, thenReady=Truereadythe model answered one request on its workload Service: a chat completion of one token (a pooling model: one embedding), sent once per Ready transition, never beside one still in flight (a list waits 15 s for it, the request runs on for up to 2 min) since= theReadycondition'slastTransitionTime,finishedAt= the answer,message: answered POST /v1/chat/completions. A runtime that passed its readiness and answers that request with an error fails the step withreason: FirstRequestFailedand the runtime's status and first error line (…answered 500 Internal Server Error: openai_harmony.HarmonyError: …): the phase isfailed,statusNotReady, andload_modelreturns the error instead of waiting. The failure stands while the object drops out of Ready (a runtime that hangs on the request fails its readiness too), until a later Ready transition's request is answered. A server whose route list names nothing to ask is not asked; the step says whyfaileda step failed — the step says why a capacity refusal standing when the GPU pool's scale-up budget is spent — --kserve-scale-up-timeout/KSERVE_SCALE_UP_TIMEOUT/ chart valuekserve.scaleUpTimeout, default10m, counted from the pod's creation: theschedulingstep fails withCapacityUnavailable, its message plus; no node came within the scale-up budget of 10m0s, and the model'sreason/messagecarry it (a nominated claim still launching without a refusal never fails the step);ImagePullBackOff/ErrImagePull;CrashLoopBackOff(the kubelet backed off from restarting a runtime that keeps dying: theloadingstep fails with the crash message and the back-off, and the model'sreason/messagecarry it); an initializer that exited non-zero, restarted or ran longer than 30 min (DownloadStalled);modelStatus.lastFailureInfoterminatingthe object is being deleted deletionTimestamp; the steps stay as they wereWithout a pod yet the object is
scheduling(WaitingForPod) since its creation; a Ready object whose pod the driver cannot see has every step done. While the pod has no node, Karpenter's account of the one it waits for — the claim launching, its refusal, the instance registering — is the model'sreason/messageinlist_loaded_modelsinstead of the scheduler'sUnschedulable; a caller who may not readnodeclaims.karpenter.shor thedefaultnamespace's events gets the step's message with the reason the read failed, never silently. Backends without a serve lifecycle (ollama, lemonade, lmstudio) answer neitherphasenorsteps. Reading the Events and the nodes costs one bounded (3 s) list per pod — plus, for a pod without a node, the nominated NodeClaim and Karpenter's Warning events — concurrently, so three served models stay inside a caller's ~10 s deadline. -
Cache verdict —
check_fit'scachedis a boolean andcacheSourcesays how it was decided:scan(a cache scan answered in this call),index(no scan could run — a GPU pool at zero, the caller's deadline — and the cache index remembers a directory an LLMInferenceService filled for the repository in the claim and volume bound now; a record bound to another cache, or to none, is no verdict),unknown(neither answered;cached: falseis then no verdict),oci-image(the preset serves an OCI model image the nodes pull themselves; nothing of it is in the claim, andcached: falseis the verdict). On the scale-from-zero pool path the claim is asked as a whole, so weights an earlier serve left in it answercached: true.
Jobs (GET /api/v1/jobs) live in process memory: the chart runs one replica
and clients poll one server. On kserve a pull job carries the node whose
cache receives the download and the preset it is for — what the request
named, or what model-manager picked itself after the fit check — so any
client (another tab, an agent through MCP) can place the download without
remembering the request; ollama jobs carry neither. A restart loses the job
list, not the work — kserve pulls are Kubernetes Jobs that model-manager
re-adopts on start (GET /api/v1/jobs lists them again as running pulls,
node and preset read back from the Job's annotations), a kserve load is
recovered by the reconcile loop that wires ready LLMInferenceServices without a
job, and an ollama or lemonade pull simply is re-issued (Ollama resumes the layers
it has, Lemonade the files).
A persistent job store is deliberately not built until a second replica or a
job history across restarts is needed; until then, treat the job list as a
progress view, and the backend (Jobs, LLMInferenceServices, Ollama) as the truth.
Without --backend/--backends the process starts with no backend and waits for backend
documents in --namespace (POD_NAMESPACE); see docs/backends.md. A static
backend:
model-manager serve \
--backend ollama \
--ollama-endpoint http://127.0.0.1:11434 \ # as reached by model-manager
--ollama-agent-host http://172.21.0.1:11434 \ # as reached by agent pods (written into ModelConfigs)
--kubeconfig ~/.kube/config --kube-context kind-agentlab \
--kagent-namespace kagentAgainst a Lemonade Server on the host (AMD Ryzen AI NPU):
model-manager serve \
--backend lemonade \
--lemonade-endpoint http://127.0.0.1:13305 \ # as reached by model-manager
--lemonade-agent-host http://172.21.0.1:13305 \ # as reached by agent pods (+ /api/v1 in ModelConfigs)
--kubeconfig ~/.kube/config --kube-context kind-agentlabAgainst an LM Studio on the host:
model-manager serve \
--backend lmstudio \
--lmstudio-endpoint http://127.0.0.1:1234 \ # as reached by model-manager
--lmstudio-agent-host http://172.21.0.1:1234 \ # as reached by agent pods (+ /v1 in ModelConfigs)
--kubeconfig ~/.kube/config --kube-context kind-agentlabEvery flag has an environment variable (model-manager serve --help). Without
Kubernetes access the server still runs; wire reports false and wiring
operations answer 501.
With --enable-oauth model-manager is an OAuth 2.1 resource server
(mcp-oauth) in front of both the
MCP endpoint and the REST API: every call needs a bearer token the platform
identity provider issued — Dex (--oauth-provider dex, the Agent Platform
default) or Google (--oauth-provider google). On the platform nobody logs in
to model-manager itself: muster forwards the session's IdP id_token
byte-identical (MCPServer auth.forwardToken: true, rendered by the chart)
and the portal sends the signed-in user's id_token through the gateway;
model-manager validates them against the IdP's JWKS when their audience is one
of --oauth-trusted-audiences. The chart passes the platform's OAuth client
(oauth.trustedAudiences, else global.identity.clientId) plus the
MCPServer's requiredAudiences: every forwarded token carries those by
construction and they are what the kube-apiserver trusts, so a portal session
— whose id_token carries them but not the platform client — is accepted too.
An id_token whose audience matches none of them is refused with 401; the
Unknown JWT bearer refused log line names the token's token_audience next
to the trusted_audiences. The caller — the email,
else the sub — is on every mutation's log line and recorded as
requestedBy on jobs.
--downstream-oauth goes one step further: everything a request does against
the Kubernetes API (LLMInferenceServices, download Jobs, cache scans, kagent
ModelConfigs, the discovery ConfigMap) presents the caller's IdP token, and
the ServiceAccount holds no permissions at all — the chart renders none of its
Roles and ClusterRoles; the caller's RBAC is the only RBAC. That needs an
apiserver that trusts the IdP and the token's audience — a Dex install lists
the cross-client audience the apiserver trusts in the MCPServer's
requiredAudiences (muster requests it at login), a Google install's client
id is the apiserver's --oidc-client-id. A request whose bearer yields no
IdP token to present (one this server neither issued nor trusts as a forwarded
id_token) is refused with 401 instead of running as the permissionless
ServiceAccount. Nothing runs without a caller: the
wiring reconciler and the re-adoption of running downloads after a restart
are off, and a job that outlives its caller's token (a download longer than
the token lives) fails on the apiserver's 401 — attributed to the caller,
retried by the caller. A restart of the process loses the in-memory jobs and
nothing else: a served model's ModelConfig is written by the load call itself
and, when missing, by the caller's next list_loaded_models, and a running
download Job is joined by the next pull_model. Health endpoints (/healthz,
/readyz, /backendz) and the OAuth metadata stay public.
wire_model and unwire_model take mode: commit, and so do load_model
and unload_model on kserve: the serving object and its ModelConfig — whose
address follows from the object's name — land in one pull request (files
<name>-llminferenceservice.yaml and <name>.yaml), and no load job runs;
the ModelConfig turns ready when the model does, after Flux applies the
merge. pull_model and delete_model never commit (a download and the
weights are no GitOps objects): mode: commit answers unsupported, as it
does for a load or unload on ollama, lemonade or lmstudio. Instead of writing the
ModelConfig live, model-manager opens a pull request as the signed-in person
in the GitOps repository that owns the kagent namespace. The repository
follows from Flux provenance: the ModelConfig's own Kustomization when Flux
applies it, else the namespace's (through the HelmRelease that renders it,
where a chart creates the namespace), then the Kustomization's GitRepository
and spec.path. Where there is none, repository, branch and path name
it. A kserve backend on a remote target splits the commit: the serving object
lands on the target, its ModelConfig in the kagent namespace of
model-manager's own cluster. The serving namespace's provenance is read on the
target as the caller; the Flux objects are read on model-manager's cluster,
where Flux applies to the target through spec.kubeConfig. An owning
Kustomization must apply through one. For a namespace a HelmRelease renders
there, the location is the one Kustomization in the HelmRelease's namespace
that applies through the same kubeconfig Secret; none or several are refused.
The ModelConfig follows the kagent namespace's provenance, and an explicit
repository/branch/path names the serving object's location only. Each
repository gets its own pull request, and the answer's commit.also lists the
further locations. The files go under <path>/model-manager/, one per object, with the
directory's kustomization.yaml and the parent's entry
(gitops-commit's layout). A
Secret, the placeholder API key, is committed only SOPS-encrypted for the
repository's .sops.yaml recipients. Without one the call is refused: use
apiKeyPassthrough or apiKeySecret, or mode: apply. dryRun answers the
files with their content and opens nothing. The answer's commit names the
repository, base, directory, branch, files, the pull request's URL and number,
and its author. Refused: an object Flux applies from a file outside
model-manager/ (change it where it is written), one a HelmRelease renders,
a removal of an object that was written live, a removal of an object git no
longer holds that a Kustomization with prune: false left behind (the answer
names the Kustomization and offers mode: apply, which removes it live), and
a Kustomization whose targetNamespace would move the objects.
Commit mode needs the person's GitHub authority, offered with the chart's
github.enabled. With github.tokenExchange, the MCPServer forwards the
person's ID token as without commit mode, and only a commit asks for GitHub:
model-manager exchanges the ID token at muster's RFC 8693 broker
(--github-token-exchange-endpoint) for the person's user grant of the GitHub
App github.tokenExchange.audience names, for that commit only, verifies it
with GET /user and opens the pull request with it. Without the grant the MCP
tool answers grant_required with apps: [<audience>], and the MCPServer
lists the audience under auth.providerApps, so muster turns the answer into
that App's sign-in link. REST answers 403 with code grant_required and
the same apps. GitHub refusing the grant with 401 answers
grant_required too. The client ID comes from
--github-token-exchange-client-id, else from the mounted Secret's
client-id key. A call made by an agent acting for the person (an ID token
with an act claim) is refused as forbidden. With auth.providerApps,
muster also attaches the person's App token to every call it forwards, which
model-manager ignores, until giantswarm/giantswarm#39227 lets a server name an
App without receiving it.
Fetching the person's GitHub grant from muster is a temporary deviation from the MCP 2025-11-25 authorization rules (no token passthrough, no third-party credentials), decided in giantswarm/giantswarm#39213; the exit is model-manager holding its own grant through URL elicitation and ID-JAG (giantswarm/giantswarm#39102).
github.enabled without github.tokenExchange fails the chart. No other
GitHub credential exists in the pod. A server without the exchange answers
unsupported for mode: commit, and get_backend/list_backends report
capabilities.commit. The App has to be installed, with contents and
pull-request write, on every repository a commit may target. On a kserve
backend with a remote target that is two repositories: the one owning the
serving namespace on the target (typically the workload cluster's fleet
repository) and the one owning the kagent namespace on model-manager's own
cluster. Before anything is written, model-manager checks each repository
with the person's App user token (GET /repos/{owner}/{repo}; GitHub answers
404 for a repository the App is not installed on). One it does not reach
answers repository_unavailable (412), naming the repository, which part
lands there and the App to install; dryRun reports the same as
commit.unavailable (and commit.also[].unavailable), with no files for that
repository.
helm/model-manager — see its README. Keys the
giantswarm/agent-platform
meta chart sets for its model-manager component: backend, ollama.endpoint,
ollama.agentHost, lemonade.endpoint, lemonade.agentHost,
lmstudio.endpoint, lmstudio.agentHost, kagent.namespace, mcp.enabled, oauth.*,
muster.mcpServer.*; for kserve kserve.namespace (the serving namespace),
kserve.discovery.*, kserve.hf.tokenSecret.* and the kserve.* overrides. Optional, off by default: muster.mcpServer.enabled
(renders an mcpservers.muster.giantswarm.io CR), httpRoute.enabled,
networkPolicy.enabled, oauth.enabled, observability.otel.endpoint (OTLP
traces: HTTP server spans, an mcp.<method> span per MCP request and a
tool.<name> span per tool call, configured by the standard OTEL_*
variables the chart renders from observability.otel.*).
See docs/development.md.