Use Bolt's C++ execution engine from Python to read data, run queries, and
prepare ML features. The bolt-engine distribution includes two APIs:
boltml: read Parquet and ORC files, filter, join, and aggregate data, and apply Python functions through a DataFrame API. Work with Arrow and NumPy, process image bytes, or configure execution on Ray workers.pybolt: work directly with native types and vectors, construct query plans, and execute them with Bolt from Python.
Start with boltml for data processing. Use pybolt when you need to construct
query plans directly; BoltML also uses its types for schemas and Python
function return values.
| Task | API and examples |
|---|---|
| Filter rows, derive columns, sort, join, and group data | DataFrame, filter, transform, orderBy, join, groupBy().aggregate(); examples. |
| Apply Python logic | Scalar, vector, and batch UDFs, plus custom aggregations; examples. |
| Exchange data with Python libraries | Arrow input/output and NumPy adapters, including fixed-shape tensor UDFs; examples. |
| Read and write local Parquet or ORC files | LocalTableReader and LocalTableWriter, including partitioned and bucketed writes; examples. |
| Process images | Image bytes in binary columns, with Pillow in Python UDFs; example. |
| Execute stages on Ray workers | Explicitly configured RayExecutor remote execution; example. |
| Access Hive or Paimon tables | Catalog readers/writers require external services and configuration; see Hive and Paimon examples. |
The quick start below runs locally, with no Ray cluster or catalog service. See test coverage and known limitations before adopting more advanced features; Hive and Paimon service integration is outside the current CI checks.
The validated platform is CPython 3.11 on Linux x86-64. See the CPU and system requirements.
With uv installed and
the Bolt checkout at ./bolt, create an application project next to it and
install BoltML from source:
uv init --python '>=3.11,<3.12' boltml-demo
cd boltml-demo
uv add ../bolt/bolt/pythonFor an existing uv project, run the uv add command from that project's
directory and adjust the checkout path as needed.
If migrating from the old distribution, run uv remove bytedbolt before
uv add ../bolt/bolt/python: both distributions provide the same boltml and
pybolt packages.
This builds and installs Bolt automatically, including Pillow, Ray, and the Paimon client. The first build needs a supported Linux C++ toolchain, development headers, Git, Bash, and network access, and can take substantial time and memory. The backend prepares Conan dependencies; see workstation setup and build configuration for prerequisites and options.
Save this as quickstart.py:
import pyarrow as pa
from boltml import DataFrame
from boltml.function.aggregation import Sum
orders = DataFrame({
"customer": ["alice", "bob", "alice"],
"amount": [10, 20, 30],
})
# Keep orders worth at least 20, then apply a discount of 5 per order.
orders.filter(orders["amount"] >= 20)
orders.transform(net_amount=lambda df: df["amount"] - 5)
totals = orders.groupBy("customer").aggregate(total=Sum("net_amount"))
print(totals.orderBy("customer").toArrow(pa.Table).to_pylist())Run it in your application's environment:
uv run python quickstart.pyExpected output:
[{'customer': 'alice', 'total': 25}, {'customer': 'bob', 'total': 15}]
Transformations build a query plan; toArrow(pa.Table) executes it and returns
a PyArrow table. Methods such as filter, transform, and orderBy modify the
current DataFrame. Use frame.copy() before branching a pipeline, and build
column expressions from the frame being transformed. Calling toArrow() with
no argument returns an Arrow StructArray instead of a table.
For custom logic, declare the return type with scalarFunction and apply the
function to a column. Save this as udf_example.py and run it with
uv run python udf_example.py:
import pybolt
from boltml import DataFrame
from boltml.function import scalarFunction
@scalarFunction(pybolt.BigintType())
def addOne(value):
return None if value is None else value + 1
def main():
frame = DataFrame({"value": [1, None, 3]})
frame.transform(incremented=lambda df: df["value"].map(addOne))
print(list(frame["incremented"])) # [2, None, 4]
if __name__ == "__main__":
main()This function handles nulls explicitly. Materializing a column as a list also executes the plan.
For Arrow vector functions, batch functions, tensors, and image processing, see the runnable integration examples. Pillow is installed by default for image decoding, resizing, and encoding in Python UDFs. Images are stored as bytes in binary columns.
Ray is included in the installation above. Save the following as
ray_example.py. It starts a local Ray cluster and runs a DataFrame
transformation in a worker process:
import ray
from boltml import DataFrame
from boltml.distributed.ray import RayExecutionConfig, RayExecutionMode
from boltml.executor.ray import RayExecutor
def main():
ray.init(address="local", num_cpus=2, include_dashboard=False)
try:
executor = RayExecutor(
config=RayExecutionConfig(mode=RayExecutionMode.REMOTE_LEAF)
)
frame = DataFrame({"x": [1, 2, 3]}, executor=executor)
frame.transform(result=lambda df: df["x"] + 1)
print(list(frame["result"]))
stages = executor.lastExecutionSummary.stages
assert stages and all(
stage.remote and stage.fallbackReason is None for stage in stages
)
print("Remote tasks:", sum(stage.remoteTaskCount for stage in stages))
finally:
ray.shutdown()
if __name__ == "__main__":
main()For this local example, let Ray workers reuse the installed application environment. Disable Ray's automatic uv project upload, which does not include the Bolt checkout outside the application directory:
RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 uv run python ray_example.pyIn addition to Ray's startup logs, the output is:
[2, 3, 4]
Remote tasks: 1
REMOTE_LEAF enables remote execution; the default RayExecutor() runs stages
locally. The assertion checks that this query ran remotely without local
fallback. This in-memory input creates one task even though two CPUs are
available. See the Ray integration examples
for parallel file scans and Python UDFs on workers.
When running on an existing cluster node or in a submitted Ray job, replace
the initialization call with ray.init(address="auto") and make Bolt and your
input data available to the workers. Follow the
wheel distribution guide to
package your local build and submit a Ray job.
If you are modifying BoltML itself, install the checkout in editable mode:
uv add --editable ../bolt/bolt/pythonYour application owns its pyproject.toml, uv.lock, and environment. Python
edits take effect on the next run. After C++ or CMake edits, run
uv sync --reinstall-package bolt-engine and restart Python. Keep the checkout
available. See generated Hive bindings and IDE setup
if your editor cannot resolve those imports after the first build.
From the Bolt checkout:
make python-wheelThis build command needs Python with pip; it does not require uv.
It prints the wheel's absolute path under dist/. Install that path
with uv add in your application, or distribute the wheel to matching hosts.
Each build has a unique filename and reuses native build output. For selecting
your application's interpreter and updating a uv dependency, see
development wheels.
Use a built wheel with the standard
Ray Job submission example.
runtime_env.py_modules uploads it for the job's driver and workers. An editable
installation on your workstation does not install Bolt on the cluster.
For BoltML-managed remote stages, explicitly select RayExecutionMode.REMOTE_LEAF
when configuring RayExecutor; its default mode executes stages locally. The
deployment guide covers this configuration and worker data/dependency access.
See native pybolt examples for lower-level vectors and plans, BUILDING.md for build and deployment details, including tests and known limitations.