A r.uby.dev project.
Welcome to the canonical mruby-llm repository.
mruby-llm is an advanced runtime for building agentic AI applications on mruby. It has zero runtime dependencies by default, supports concurrent and parallel tool execution and has a single coherent API that spans 12+ providers. Streaming, tools, guards, compaction, and the builtin MCP/A2A support all build on the same three concepts: providers, contexts, and agents.
The most effective way to learn about mruby-llm is to ask the r.uby.dev chatbot a question. It is connected to the mruby-llm GitHub repository so all answers are grounded in the source code. The chatbot is implemented with llm.rb from the CRuby ecosystem and that happens to be what mruby-llm is based on.
Add the following to your build_config.rb (or its equivalent):
MRuby::Build.new("app") do |conf|
conf.gembox "default"
conf.gem github: "r-uby-dev/mruby-llm", branch: "main"
endThe
LLM::Agent
class is the default high-level interface,
and it is recommended for most use-cases. It manages tool execution
automatically and
guards against infinite loops,
manages conversation state, and much more.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: $stdout)
agent.talk "hello world"Stream
Streams can be simple IO objects or subclasses of
LLM::Stream
with structured callbacks for content,
reasoning, tool calls, tool returns, and compaction.
Streams can also observe message transformers, which rewrite
outgoing messages before they reach the provider.
class MyStream < LLM::Stream
# Visible assistant output.
def on_content(content)
print content
end
# Reasoning output streamed separately from visible content.
def on_reasoning_content(content)
warn content
end
# A streamed tool call has been fully parsed.
def on_tool_call(tool, error)
end
# Queued streamed tool work has returned.
def on_tool_return(tool, result)
end
# Before and after a transformer rewrites an outgoing message.
def on_transform(transformer)
end
def on_transform_finish(transformer)
end
# Before and after a compactor trims the conversation.
def on_compaction(compactor)
end
def on_compaction_finish(compactor)
end
# A request was rate limited and will be retried.
def on_rate_limit(error)
end
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: MyStream.new)
agent.talk "Explain Ruby fibers."Tools
Subclasses of
LLM::Tool
are plain Ruby classes with
an optional set of typed parameters.
The model can choose to
call them on your behalf, and they're one of the most powerful features
for extending the feature set or abilities of a model.
The runtime also ships with a catalog of built-in tools for filesystem, search, and shell operations.
class ReadFile < LLM::Tool
name "read-file"
description "Read a file"
parameter :path, String, "The filename or path"
required %i[path]
def call(path:)
{contents: File.read(path)}
end
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, tools: [ReadFile], stream: $stdout)
agent.talk "summarize README.md"Skills
A skill turns a markdown file into a callable tool. When the model calls it, the runtime spawns a subagent with the skill's instructions as its system prompt and the skill's own tool set. The subagent runs one turn and returns the result, then is discarded. Each call is fresh and stateless.
---
name: summary
description: Reads recent git history and writes a summary
tools: all
---
Collect the recent git log, analyze each commit,
and write a summary to summary.txt.llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, skills: ["./skills/summary"])
agent.talk "Summarize the last week of work"Concurrency
The runtime supports three different concurrency strategies that have different attributes. The choice between them often depends on the requirements of your application and the mruby build you ship.
The :sequential strategy runs tools one at a time and is the default.
The :task strategy is lightweight concurrent execution through
mruby-task. The :fork strategy provides a separate process that offers
isolation from its parent, where the target platform supports it.
llm = LLM.deepseek(key: ENV["KEY"])
tools = LLM::Tool.registry
agent = LLM::Agent.new(llm, tools:, concurrency: :fork)
agent.talk "Run the tools in parallel"Cancellation
Abort a request mid-stream and interrupt any running tools with
LLM::Agent#interrupt!
(or cancel!), from any task. The runtime raises
LLM::Interrupt
on the caller and on every active tool. A forked tool gets interrupted
over the control channel, and pending tools are stopped before they
run. The in-flight HTTP request is closed too, so a turn you no longer
want stops without burning tokens.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm)
Task.new do
sleep 1
agent.cancel!
end
begin
agent.talk "write a very long poem", stream: $stdout
rescue LLM::Interrupt
puts "cancelled"
endPersistence
Set path: on an agent for automatic filesystem persistence:
the agent restores conversation history from the file on startup
and saves it back after every turn, with no manual serialization
code. All persistence options use the same underlying serialization.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, path: "session.json")
agent.talk "remember my name is robert"
# Next time, the conversation is restored automatically:
agent = LLM::Agent.new(llm, path: "session.json")
agent.talk "what's my name?"MCP
The Model Context Protocol (MCP) has first-class support
in mruby-llm. The stdio and http transports work out of the
box. MCP tools are translated into subclasses of
LLM::Tool that can be
used with
LLM::Context or
LLM::Agent.
llm = LLM.deepseek(key: ENV["KEY"])
mcp = LLM::MCP.stdio(argv: ["ruby", "server.rb"])
agent = LLM::Agent.new(llm, stream: $stdout, tools: mcp.tools)
agent.talk "Run the tool"Set persistent: true on the HTTP client to reuse
connections across requests. This uses the persistent
curl transport under the hood and avoids opening a new
TCP connection for every request:
mcp = LLM::MCP.http(
url: "https://api.githubcopilot.com/mcp/",
headers: {"Authorization" => "Bearer #{ENV['GITHUB_PAT']}"},
persistent: true
)A2A
The Agent 2 Agent (A2A) protocol has first-class support
in mruby-llm. The http and jsonrpc transports work out of the
box. A2A skills are translated into subclasses of
LLM::Tool that can be
used with
LLM::Context or
LLM::Agent.
llm = LLM.deepseek(key: ENV["KEY"])
a2a = LLM::A2A.rest(url: "https://remote-agent.example.com")
agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills)
agent.talk "Run the skill"The A2A HTTP transport is curl-based by default, so connections are
reused across requests without extra configuration. Pass a concrete
LLM::Transport class or instance through transport: to override it:
a2a = LLM::A2A.rest(url: "https://agent.example.com")
a2a = LLM::A2A.jsonrpc(url: "https://agent.example.com")Structured outputs
LLM::Schema
subclasses produce typed, structured
output from any model call. Pass a schema to
LLM::Context#talk,
LLM::Agent#talk,
or
LLM::Provider#complete
to receive validated JSON instead of free text. Schemas work alongside tools and streams.
LLM::Schema
can define objects, arrays, enums, nested schemas,
and more. It is also used internally by
LLM::Tool for parameter
definitions, so you already benefit from it when you declare tool
parameters.
The
LLM::DeepSeek
provider includes runtime-level optimisations such as structured
output support (despite no official structured outputs API) and
SVG image generation. This example uses
LLM::Schema with
DeepSeek:
class Weather < LLM::Schema
property :city, String, "The city name"
property :temperature, Number, "Current temperature"
property :conditions, String, "Weather conditions"
required %i[city temperature conditions]
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, schema: Weather)
res = agent.talk "Weather in Paris?"
res.content! # => {city: "Paris", temperature: 15.0, conditions: "Cloudy"}Guards
LLM::Guard
is the hook that sees every tool call before it runs. A guard
can let a call through, cancel it, block it with an error, or
even answer for it. Because it runs before the tool, anything
it intercepts never executes. Policy, validation, quotas, and
cost ceilings all live here.
LLM::Agent
enables
LLM::Guard::Loop
by default, so agents get loop protection out of the box. To
write your own guard, subclass
LLM::Guard
and implement
LLM::Guard#call.
The pending call arrives as function:. Return a value to close
the call, or nil to let it run:
class PolicyGuard < LLM::Guard
def call(function:)
if function.name == "shell"
function.return(error: true, type: "policy_error",
message: "shell is disabled")
end
end
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, tools: [Shell], guard: PolicyGuard)Transformers
It is possible to rewrite outgoing messages before they reach the provider with
LLM::Transformer.
Create a subclass and implement call(message:) to scrub sensitive data,
inject context, or normalize content. The transform runs automatically
on every turn, so you never have to change your prompt code.
class RedactEmails < LLM::Transformer
def call(message:)
content = message.content.to_s.gsub(/[\w.+-]+@[\w-]+\.[\w.]+/, "[EMAIL]")
LLM::Message.new(message.role, content, message.extra)
end
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, transformer: RedactEmails)
agent.talk "Contact support@example.com for help"Compactors
Every model has a context window: the finite number of tokens it can consider in a single request. Generally a compactor will drop or summarize older messages to keep the conversation within that window, and it runs automatically before every turn. By default it is disabled so it is a feature you must opt into.
LLM::Compactor::Truncate
keeps the most recent messages via an integer count or a percentage like
"80%". It preserves tool call and return pairs so the conversation
never contains an orphaned result. It is also possible to subclass
LLM::Compactor
to implement your own compactor with its own logic. Streams can observe the
process through the
LLM::Stream#on_compaction
and
LLM::Stream#on_compaction_finish
callbacks.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(
llm,
compactor: LLM::Compactor::Truncate,
compactor_options: {keep: 64}
)
agent.talk "Hello"Automatic retries
Rate-limited requests are retried automatically by default. Agents
retry a 429 up to five times with a growing backoff before giving
up, so most request failures resolve on their own. Set retry_budget
to change the number of retries, or retry_budget: 0 to disable
them.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, retry_budget: 0)
agent.talk "Hello"Observability
Trace what an agent is doing by attaching a tracer. Hook into requests, tool calls, and other runtime events to debug a misbehaving agent, monitor latency, or export spans to an observability backend. All built-in tracers share one interface, so switching between them means changing a class name:
LLM::Tracer::PrettyLogger: human-readable single-line logs to stderr, ideal during development.LLM::Tracer::Logger: structured JSON to stdout or a file.
llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, tracer: LLM::Tracer::PrettyLogger.new(llm))
agent.talk "Hello"As a subclass
LLM::Agent.set
is a class-level DSL that accepts a Hash of properties. Each key resolves to a
corresponding class accessor: name, description, model, tools,
instructions, schema, stream, tracer, concurrency, confirm,
path, skills, tool_budget, and retry_budget. All options are
optional; zero or more can be set.
An error is raised for unknown keys so that typos are caught early.
class Agent < LLM::Agent
set name: "sysadmin",
description: "system administration agent",
model: "deepseek-v4-pro",
tools: [LLM::Tool::Shell]
end
llm = LLM.deepseek(key: ENV["KEY"])
agent = Agent.new(llm)
agent.talk "Run 'date'"Each provider is constructed with a class-level factory method on
LLM, and the resulting instance is passed to
LLM::Context
or
LLM::Agent. The
same API drives every one of them, so switching providers is a one-line
change.
- Anthropic (
LLM.anthropic) - Google (
LLM.google) - OpenAI (
LLM.openai) - DeepSeek (
LLM.deepseek) - DeepInfra (
LLM.deepinfra) - xAI (
LLM.xai) - Z.ai (
LLM.zai) - Moonshot (Kimi) (
LLM.moonshot) - Alibaba (Qwen3) (
LLM.alibaba, alsoLLM.aliyun) - Mistral (
LLM.mistral) - AWS Bedrock (
LLM.bedrock) - Ollama (
LLM.ollama) - llama.cpp (
LLM.llamacpp)
Implicit
Cloud providers can infer their API key automatically from a set of common defaults that are defined by the models.dev registry that is also distributed with mruby-llm.
llm = LLM.openai
llm = LLM.anthropic
llm = LLM.deepseek
llm = LLM.alibaba # also: LLM.aliyun
llm = LLM.moonshot
llm = LLM.mistralExplicit
The key option can also be provided explicitly, and certain
providers (eg ollama, llamacpp) usually do not require an API
key at all.
llm = LLM.openai(key: ENV["OPENAI_API_KEY"])
llm = LLM.anthropic(key: ENV["ANTHROPIC_API_KEY"])
llm = LLM.deepseek(key: ENV["DEEPSEEK_API_KEY"])
llm = LLM.alibaba(key: ENV["DASHSCOPE_API_KEY"]) # also: LLM.aliyun
llm = LLM.moonshot(key: ENV["MOONSHOT_API_KEY"])
llm = LLM.mistral(key: ENV["MISTRAL_API_KEY"])Model Registry
Each provider ships its model catalog, pricing, limits, and modalities with the gem, sourced from models.dev. Reach it from any provider, context, or agent, enumerate models, or sort them by price.
llm = LLM.openai
registry = llm.registry # => LLM::Provider#registry
cheapest = registry.models.sort.first # => LLM::Model
cheapest.id # => "text-embedding-3-small"
cheapest.context_window # => 8191
cheapest.structured_output? # => falseTransports
The transport: option selects which HTTP implementation a provider
uses for network communication. mruby-llm ships a single curl-based
transport that is always available and is the default, so most
applications need no configuration at all.
llm = LLM.deepseek(
key: ENV["KEY"],
transport: LLM::Transport::Curl.new(
host: "api.deepseek.com", port: 443, timeout: 180, ssl: true
)
)Most providers offer an embedding model that can be used for semantic search, or similarity search. An embedding model can generate embeddings that can then be stored in a database that is optimized for storing and querying vectors, such as SQLite's sqlite-vec or PostgreSQL's pg-vector.
mruby-llm also includes support for OpenAI's vector store API. It provides a vector database as a HTTP service but we won't cover that here.
llm = LLM.openai(key: ENV["KEY"])
body = "mruby-llm is mruby's capable AI runtime."
embedding = llm.embed([body]).embeddings.first
# Document is your own model with a vector column
# (e.g. sqlite-vec or pgvector)
Document.create!(
title: "mruby-llm",
body:,
embedding:,
)A handful of providers can generate images from a text prompt. OpenAI, Google, xAI, and DeepInfra all support it. The API is the same across providers:
llm = LLM.openai(key: ENV["KEY"])
res = llm.images.create(prompt: "a dog on a rocket to the moon")
File.binwrite "rocket.png", res.images[0].stringDeepSeek does not have a dedicated image model, but the runtime
generates SVG vector graphics through its text model. Each
generation produces a valid SVG document that can be converted
to PNG with tools like rsvg-convert. Pass an existing agent
to maintain a session across generations:
llm = LLM.deepseek(key: ENV["KEY"])
##
# First generation
res = llm.images.create(prompt: "a rocket on the moon")
File.binwrite "rocket.svg", res.images[0].string
##
# Refine with follow-up prompts (shares context)
res = llm.images.create(prompt: "add a dog next to the rocket",
agent: res.agent)
File.binwrite "rocket-with-dog.svg", res.images[0].stringCan I link mruby-llm to a binary?
You can and that happens to be the primary reason
that I decided to create mruby-llm. The binaries can
be as small as 2MB. A common mruby approach for this
problem is to create a small C program (main.c) that
embeds the bytecode of your program in a static C array.
The program also links mruby and through mruby it can execute the bytecode from the static C array. It is also possible to link other C programs to your binary and they can also be made accessible to your mruby program.
I have a limited budget. What should I do?
There are a few options. The first option is to host your own model, and use the ollama or llamacpp providers. This can be difficult though because a capable model requires hardware that can match it. If you have the ability to self-host, this would be my first option.
The second option is DeepSeek.
The deepseek-v4-flash model costs pennies to use.
And mruby-llm has been optimized for deepseek. For example,
DeepSeek does not have image generation capabilities
but on the mruby-llm runtime it does (vector graphics only,
though).
The same is true for structured outputs. DeepSeek does not support structured outputs in the same way as OpenAI or Google, but the mruby-llm runtime makes it appear as though it does, through the `json_object` response type.
If you're on a budget, DeepSeek is hard to beat.What's the relationship with llm.rb?
mruby-llm and llm.rb are closely related. The llm.rb project was created first (over three years ago) and it is the reference implementation. mruby-llm is a port of llm.rb to mruby. The codebase is almost identical, the interface between the two is the same, and the main difference between the two is what features they support.
Most development happens in the llm.rb repository and it is then backported to the mruby runtime afterwards but sometimes backports go in the other direction, too.
The r.uby.dev chatbot is connected to this very GitHub repository. It can read documentation, source code, issues, and pull requests. It is the most effective way to learn about mruby-llm (and llm.rb).
This software is released under the terms of the MIT license.
See LICENSE for details.