Open source accelerator direct storage. Vendor-neutral API modeled on NVIDIA's cuFile (GDS), powered by AiSIO for high-throughput PCIe P2P DMA from NVMe straight into GPU memory.
#define _GNU_SOURCE
#include <opends.h>
#include <cuda_runtime.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
int main(void)
{
size_t size = 1024 * 1024;
void *buf;
cudaMalloc(&buf, size);
opends_driver_open();
opends_buf_register(buf, size, 0);
int fd = open("/mnt/nvme/a.bin", O_RDONLY | O_DIRECT);
opends_handle_t fh;
opends_handle_register(&fh, fd);
ssize_t n = opends_sync_read(fh, buf, size, 0, 0);
printf("read %zd bytes\n", n);
opends_handle_deregister(fh);
close(fd);
opends_buf_deregister(buf);
cudaFree(buf);
opends_driver_close();
return 0;
}examples/ holds complete programs with error handling. sync_read.c reads a
file and checks the bytes against a kernel read, async_read.c reads two
files at once with the async API, and stream_read.cu enqueues a read, a
kernel over the bytes and a copy back on one CUDA stream.
NOTE: The AiSIO guide sets all of this up on a fresh Ubuntu installation.
- An NVIDIA GPU with the NVIDIA driver and the CUDA toolkit,
nvccon thePATH. - In the BIOS setup:
- Resizable BAR enabled, so that the GPU's BAR1 window covers all of its
memory. Check with
nvidia-smi -q -d memory. - ACS disabled.
- Resizable BAR enabled, so that the GPU's BAR1 window covers all of its
memory. Check with
- An NVMe drive that is not the boot disk, with an XFS file system that holds the files to read.
- The two kernel modules of uPCIe, from its
upcie-experimental-dkmspackage in the version of the uPCIe that the pinned xNVMe builds, 0.10.4.dmabuf_importresolves the DMA addresses of GPU memory, andiommu_map_pamaps GPU memory for a drive behind the IOMMU. - One of the two ways to attach the drive:
- The IOMMU on, and the drive bound to
vfio-pci. Where xNVMe's default vfio mode is refused, as on a 6.8 kernel, setXNVME_UPCIE_VFIO_MODE=type1. - The IOMMU off or in passthrough mode (
iommu=pt), and the drive bound touio_pci_generic.
- The IOMMU on, and the drive bound to
- The
ublk_drvkernel module. - 1024 hugepages of 2 MiB, which are enough for the default settings.
- Meson, Ninja, a C11 compiler, Git and liburing, to build the dependencies.
xNVMe (for-aisio-02112026), xal (v0.3.4) and OpenDS:
git clone https://github.com/xnvme/xnvme
cd xnvme
git checkout for-aisio-02112026
git submodule update --init --recursive
meson setup build
meson compile -C build
meson install -C build
cd ..
git clone https://github.com/xnvme/xal
cd xal
git checkout v0.3.4
git submodule update --init --recursive
meson setup build -Dxal-server=enabled
meson compile -C build
meson install -C build
cd ..
git clone https://github.com/xnvme/opends
cd opends
meson setup build
meson compile -C build
meson install -C build
ldconfigxNVMe brings the homi and qublk tools, and xal brings xal-server.
As root, in one terminal. Name the drive, its mount point and the driver that will hold it:
export BDF=<pci-address> # of the drive, as lspci -D prints it: 0000:01:00.0
export MNT=<mount-point> # where its file system is mounted
export DRIVER=vfio-pci # uio_pci_generic with the IOMMU off or in passthrough modeCheck the address twice. The next block takes that drive away from the kernel, and it must not be the one the system runs from.
Take the drive from the kernel's NVMe driver and bind it to that driver, so that xNVMe can open it:
umount $MNT
echo $BDF > /sys/bus/pci/drivers/nvme/unbind
echo 1 > /sys/bus/pci/devices/$BDF/reset
modprobe $DRIVER
modprobe ublk_drv
echo $DRIVER > /sys/bus/pci/devices/$BDF/driver_override
echo $BDF > /sys/bus/pci/drivers/$DRIVER/bindStart homi, which holds the controller, and wait until it prints
HOMI started successfully:
# 512 MiB of host memory: room for qublk and for OpenDS
homi serve $BDF --homi-id 1 --be upcie --host-heap-size $((512 * 1024 * 1024)) &Start qublk, which exports the drive as a block device for the file system,
and wait until it prints /dev/ublkb0 ... ready:
qublk serve $BDF --homi-id 1 --be upcie --nqueues 1 &xal-server publishes where the files are on the drive. Create its
configuration, xal-server.conf, with the mount point of the drive:
devices = [
{ uri = "/dev/ublkb0", shm_name = "/xal_dev0", mountpoint = "<mount-point>" },
]Mount the file system and start it, and wait until it prints published,
which is when its first index is ready:
mount /dev/ublkb0 $MNT
xal-server --config xal-server.conf &Run your OpenDS application, with the two variables that tell OpenDS where the index and the mount are. For example, from the OpenDS checkout, the example that reads a file into GPU memory:
export OPENDS_XAL_SHM=/xal_dev0 OPENDS_HOMI_MNT=$MNT
build/sync_read_example $MNT/<file>A file has to be on the drive before xal-server starts (see
Writes and the extent index).
Stop the three in reverse order, and give the drive back to the kernel:
kill %xal-server; wait %xal-server
umount $MNT
kill %qublk; wait %qublk
kill %homi; wait %homi
echo $BDF > /sys/bus/pci/drivers/$DRIVER/unbind
echo > /sys/bus/pci/devices/$BDF/driver_override
echo $BDF > /sys/bus/pci/drivers/nvme/bindtasks/steps/homi_stack_up.sh and homi_stack_down.sh do the same in one
call, with waits and checks. They bind the driver that fits the IOMMU's mode,
or the one HOMI_NVME_DRIVER names.
- Reference (
libopends_ref): POSIXpread/pwriteon host buffers. No external dependencies. Serves as a correctness baseline and template for hardware-specific backends. - cuFile (
libopends_cufile): Wraps NVIDIA cuFile for GPUDirect Storage. Buffers are GPU memory allocated withcudaMallocand registered viacuFileBufRegister. Requires CUDA toolkit and the cuFile (GDS) library. Built conditionally when both are found. - AiSIO (
libopends_aisio): PCIe P2P DMA between NVMe and GPU memory via xNVMe'supcie-cudabackend (no filesystem or kernelnvmedriver in the read data path). Based on AiSIO. A homi server (an xNVMe tool) is the primary of an xNVMe multi-process group and holds the userspace NVMe controller up. The driver joins that group as a secondary and allocates its own I/O queues. File extents come from the index xal-server publishes over POSIX shared memory, built over the qublk-exported block device. Reads and writes are supported. A handle can also be the drive itself (raw block mode, below), with reads and writes as P2P DMA at namespace offsets. Requires xNVMe, xal, and the CUDA toolkit.
The AiSIO backend reads its configuration from environment variables at
opends_driver_open. Values that are out of range, or not a number, fail
the open.
OPENDS_XAL_SHM: Shared-memory name of the xal-server extent index. Default/xal_dev0.OPENDS_AISIO_IO_THREADS: Number of internal IO worker threads. Default 2. Driver open creates one NVMe queue per worker.OPENDS_AISIO_QUEUE_DEPTH: xNVMe queue depth per worker. Default 8.OPENDS_AISIO_CPU_MASK: Hex mask of CPUs for the workers (e.g.0xf0). Each worker gets a one-CPU affinity viapthread_attr_setaffinity_np(3): worker i takes set bit i mod popcount, so a mask with fewer bits thanOPENDS_AISIO_IO_THREADSpins more than one worker to a CPU. Unset or0leaves placement to the scheduler.OPENDS_AISIO_IDLE_SPIN: How long an idle IO worker keeps yielding after its last activity before it naps, in microseconds. Default 200.0naps at once.busynever yields or naps: the worker polls flat out and holds a CPU until the driver closes.OPENDS_AISIO_ASSUME_ALIGNED_ONLY:1declares that every read is LBA-aligned. Reads that start or end off an LBA boundary fail withOPENDS_INVALID_VALUE, and the stream path stops enqueueing the bounce kernel.OPENDS_AISIO_HOMI_ID: xNVMe multi-process group to join. Default 1, which the test tasks also pass to homi, so both sides agree. The driver asks the homi server of this group for its device, and the group must hold exactly one.OPENDS_AISIO_HOST_HEAP_MB: Host DMA heap for this process, holding its own queue rings and PRP lists. Default 256. The heap comes out of the hugepages every process in the group shares, so xNVMe's 1 GiB default is too large.OPENDS_AISIO_DEVICE_HEAP_MB: GPU device heap for this process. Default 0, which leaves it at the xNVMe default.OPENDS_AISIO_RAW_BLOCK:1turns raw block mode on (below). Default off, which changes nothing for a process that uses files only.OPENDS_AISIO_WRITE_BUF_MB: Size of one pinned host buffer a write is staged through, in MiB. Default 16. A larger write goes through its buffer in rounds.OPENDS_AISIO_WRITE_BUFS: Most buffers the driver pins, allocated as writes need them. Default four per I/O thread. Every write in flight holds one: a stream write from its call, the others from their dispatch. A stream write call that finds none free waits for one, so an application that queues many stream writes behind work its own thread must still release needs a pool at least that large.
Every read through the AiSIO backend resolves the file's extents in the index
that xal-server publishes. The stack runs xal-server with watchmode = 0, so
the index describes the file system as it was when xal-server started, and
nothing updates it while the stack is up.
Writes through OpenDS (opends_sync_write, opends_async_write,
opends_stream_write) go through the kernel and are correct: the bytes land
in the file, and a kernel read sees them. The source is copied out of GPU
memory into a pinned buffer, on the user's stream for a stream write, and the
I/O thread only polls for the copy: it never makes a CUDA call that waits,
since a stream may be waiting at a gate for that thread.
A stream write on a stream registered with the three FIXED flags of
opends_stream_register (OPENDS_STREAM_FIXED_BUF_OFFSET,
_FIXED_FILE_OFFSET, _FIXED_FILE_SIZE, the flags of
cuFileStreamRegister) takes its size and offsets at the call and copies
with the copy engine; a value the stream changes later fails the write with
OPENDS_INVALID_VALUE. Without the flags a kernel on the stream reads the
values when the stream reaches the write and copies them, staying resident
until the last round of a write larger than the buffer has landed. A write
of more than 64 buffers goes through the kernel with the flags too, since a
stream takes only so many queued commands. Pass the flags when the values
are set before the call, which is the common case.
Undefined behaviour: reading through OpenDS what was written after xal-server indexed. That read may return the bytes from before the write, zeros, or another file's blocks, and no error. It covers a file created after the index, bytes appended to a file, a hole that a write filled, and an overwrite the file system placed in new blocks. An in-place overwrite of blocks that were allocated at index time usually reads back correctly, but XFS does not promise it, and OpenDS does not check it.
To read written data through OpenDS, restart xal-server after the writes so it
indexes again. A process that opened the driver before the restart stays on the
old index: its next read fails with OPENDS_FS_SETUP_ERROR after five seconds,
and it must call opends_driver_close() and opends_driver_open() to attach to
the new index. This restriction is temporary: a re-index on request is being
pursued in xal-server, and the restriction goes when it lands.
With OPENDS_AISIO_RAW_BLOCK=1, opends_handle_register accepts a character
device that answers for the controller, such as the one homi serves
(/dev/xnvme/<bdf>), opened without O_DIRECT. On such a handle
file_offset is the byte offset on the namespace: file_offset / lba_size is
the LBA, with no filesystem or extent lookup in the path. All I/O families
work: sync, async, stream and batch.
In this mode the driver opens without an index and attaches to it when the
first regular file is registered, with the wait and the error the open has
otherwise; a failed attach is final until the driver is closed and opened
again. A read that needs the index, through a file handle kept across
opends_driver_close() and opends_driver_open(), fails with
OPENDS_FS_SETUP_ERROR until a regular file is registered again.
A raw handle needs no index, so the stack for raw block mode is homi serve
alone: no qublk, no file system and no xal-server. This is the uGDS-style use,
a KV cache for instance, where the process owns the drive and lays it out
itself. test_raw_block_aisio runs that way in the test suite.
- Reads and writes go between GPU memory and the NVMe as P2P DMA, and are short when they cross the end of the namespace.
- A read may start and end anywhere, but its whole LBAs must land on a 4-byte boundary in GPU memory: a raw handle has no kernel path to fall back to. A stream read must start on an LBA boundary, as on a file.
- A write, unless empty, must start and end on an LBA boundary, come from a GPU address on a 4-byte boundary and start before the end of the namespace.
- I/O that breaks these rules fails with
OPENDS_INVALID_VALUE, I/O against the access mode of the fd withOPENDS_INVALID_FILE_OPEN_FLAG. A refused write writes nothing. - Only a character device that answers for the controller this driver opened
is accepted, told by its serial number. homi serves one when xNVMe is built
with CUSE, the
cusemodule is loaded and--no-cuseis not given. - The process must own the drive: do not run qublk or mount a file system on a drive used in raw block mode. Raw writes bypass them, and they would not see the change.
Commit 93fd019 (kernel 6.8.12-dmabuf, NVMe Samsung S4LV008[Pascal], GPU
NVIDIA RTX 2000 Ada Generation). OPENDS_AISIO_IO_THREADS=2 and
OPENDS_AISIO_QUEUE_DEPTH=8.
| Dataset | mode | cuFile (MiB/s) | OpenDS (MiB/s) |
|---|---|---|---|
| filesize8gib | sync | 6520 | 6794 |
| filesize8gib | stream | 2599 | 7036 |
| filesize8gib | async | - | 7065 |
| tiktokish | sync | 4614 | 5869 |
| tiktokish | stream | 5101 | 4957 |
| tiktokish | async | - | 5005 |
| imagenetish | sync | 343 | 583 |
| imagenetish | stream | 875 | 2787 |
| imagenetish | async | - | 3073 |
| lmcacheish | sync | 5533 | 6151 |
| lmcacheish | stream | 4991 | 5368 |
| lmcacheish | async | - | 5384 |
C programs find OpenDS through pkg-config --cflags --libs opends or meson's
dependency('opends'). Python bindings are in python/; see
python/README.md.
| OpenDS family | cuFile equivalent |
|---|---|
opends_async_* |
none |
opends_sync_* |
cuFileRead/cuFileWrite |
opends_stream_* |
cuFileReadAsync/cuFileWriteAsync |
opends_batch_* |
cuFileBatchIO* |
opends_async_* is per-operation async without streams (no cuFile
counterpart), shown in examples/async_read.c. The future must stay at
the same address until awaited.
opends_sync_* blocks until completion.
opends_stream_* is stream-ordered I/O, cuFile's ReadAsync/WriteAsync, shown
in examples/stream_read.cu: operations enqueue on a registered stream (e.g. a
CUDA stream) and complete in stream order. Sizes, offsets, and the byte-count result are passed as
pointers, read and written at stream execution time rather than submission
time.
opends_batch_* submits many independent operations in one call and reaps
completions by polling opends_batch_get_status. Operations complete in
any order. opends_batch_setup fixes how many the handle holds in flight,
and a slot frees once its completion is delivered.
I/O submission and registration (handles, buffers, streams) are thread-safe once
opends_driver_open has returned. opends_driver_open and
opends_driver_close are not. Deregistering or freeing an object with I/O still
in flight on it is undefined, as with closing a file descriptor that has I/O
in flight.
The AiSIO backend captures the CUDA context current at opends_driver_open and
requires that same context to be current on every thread that submits I/O
through the API. A submit from a thread with a different context current fails
with OPENDS_CONTEXT_MISMATCH. An application that uses only the CUDA runtime
API meets this automatically, since every thread on the same device shares the
primary context. An application that creates contexts with the driver API
(cuCtxCreate) must bind the driver-open context on each submitting thread with
cuCtxSetCurrent. cudaSetDevice binds the primary context and is not
equivalent.
Functions returning opends_error_t carry both an opends error code and an
optional backend-specific code. Functions returning ssize_t (read/write)
return the byte count on success or a negated error on failure.
opends_error_t err = opends_handle_register(&fh, fd);
if (err.err != OPENDS_SUCCESS) {
fprintf(stderr, "%s\n", opends_op_status_error(err.err));
}
ssize_t n = opends_sync_read(fh, buf, size, offset, 0);
if (n < 0) {
fprintf(stderr, "read: %s\n",
opends_op_status_error((opends_op_error_t)-n));
}The test suites and the benchmarks run from a workstation against a target
machine over ssh, through cijoe
(pipx install cijoe). Create the configs and fill in the target details:
cp configs/transport.toml.example configs/transport.toml # hostname, ssh key
cp configs/test.toml.example configs/test.toml # NVMe BDF, mount pointThen sync the tree, install the pinned dependencies (xNVMe, xal, fil, first
run only), build, and run the suites. They attach the drive through
uio_pci_generic:
python scripts/rsync.py
python scripts/setup_deps.py
python scripts/build.py
python scripts/run_tests.pyThroughput benchmarks use filperf from fil
against four reference datasets. The AiSIO guide's setup_dataset.yaml
writes filesize8gib, tiktokish and imagenetish during provisioning.
scripts/bench/setup_dataset.py writes lmcacheish once.
With OpenDS installed and the datasets in place:
python scripts/bench/run.py # --full-sweep measures the whole grid
python scripts/bench/report.pyrun.py measures the configs in scripts/bench/sweep.toml. report.py
turns the records into report.md, sweep.csv and report.png. Each
script's --help covers its own flags.
The perf table above is edited by hand from these reports. Every filperf
run drops page caches first, so the numbers are cold-cache, N=1.
Every loader llama.cpp ships moves the weights twice: off the drive into
host memory, then across the bus again into VRAM. OpenDS cuts the host
out and lets the drive write them into VRAM itself. Our fork of llama.cpp,
fmma/llama.cpp,
adds --load-mode opends, which reads every tensor of a GGUF file that
way, over PCIe P2P DMA with no kernel in the data path. One drive is
enough to beat dio, the fastest loader llama.cpp ships.
Qwen3-14B, 11.5 GiB of weights, one Samsung 990 PRO:
--load-mode |
Whole process | Model load |
|---|---|---|
| opends | 3.1 s | 2.0 s |
| dio | 3.3 to 3.6 s | 2.2 to 2.5 s |
| none | 3.9 to 4.0 s | 2.8 s |
| mmap | 6.3 s | 5.1 s |
dio, llama.cpp's direct I/O loader, is the fastest of the existing ones, and it runs the drive at its rate, about 6.9 GB/s, as opends does. What opends saves is the host round trip: dio's pinned staging buffers, their setup and their teardown.
With OpenDS installed as in Getting started, build the opends-loader
branch of the fork, bring the stack up over the drive that holds the
model, and load. $OPENDS is this repository, $BDF the drive and $MNT
its mount point:
git clone -b opends-loader https://github.com/fmma/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DLLAMA_OPENDS=ON
cmake --build build --target llama-completion
$OPENDS/tasks/steps/homi_stack_up.sh $BDF $MNT
export OPENDS_AISIO_DEVICE_HEAP_MB=64
echo "What is the capital of France? /no_think" |
build/bin/llama-completion -m $MNT/models/Qwen3-14B-Q6_K.gguf -st \
--load-mode opendsFor the baselines, stop the stack, give the drive back to the kernel
(clear_homi_stack.sh, rebind_nvme.sh) and run the same command with
--load-mode dio, none or mmap.
Measured 2026-11-02: OpenDS 5d37dff, llama.cpp opends-loader at
86d65e6, kernel 6.8.12-dmabuf, RTX 2000 Ada, Qwen3-14B-Q6_K, five runs
per mode with the page cache dropped, GPU persistence mode on and the token
embedding in VRAM (-ot token_embd.weight=CUDA0). The recording is one run
of each mode with the command above, recorded one after the other since
the drive cannot be on both drivers at once, laid side by side with the
two Enters on the same frame. The environment of both holds
OPENDS_AISIO_DEVICE_HEAP_MB=64, which keeps the backend's GPU heap out
of the way of llama.cpp's memory fit on a 16 GB card.
