DataFusion Distributed development tools

This repository contains the remote benchmark platform and pull-request benchmark automation for DataFusion Distributed.

It has two independently operated components:

• benchmarks-remote/ provisions the shared EKS foundation, manages datasets and engine deployments, and runs benchmarks from a developer machine.
• benchmark-bot/ provisions a persistent EC2 controller that accepts trusted GitHub PR comments, builds immutable base and head revisions, runs isolated benchmark deployments on the existing EKS foundation, and reports the comparison.

The benchmark bot does not own the EKS foundation. A human provisions the foundation from benchmarks-remote/ and gives the controller narrowly scoped access to use it. Both components live in this repository so the trusted benchmark harness can evolve independently of DataFusion Distributed source revisions.

Checkout layout

Keep this repository beside a DataFusion Distributed source checkout. The remote benchmark commands read datasets and queries from the source checkout's testdata/ directory.

<parent>/
  datafusion-distributed/
  datafusion-distributed-dev-tools/

Set DATAFUSION_DISTRIBUTED_ROOT when running commands against a source worktree outside this default layout.

Development

Install each component's dependencies:

npm run install:all

Validate both components from the repository root:

npm run build
npm test

Root npm commands forward operational commands to the owning component. For example:

npm run foundation-deploy
npm run sync-bucket -- tpch/sf1
npm run datafusion-deploy
npm run datafusion-bench -- tpch/sf1 --iterations 1
npm run controller-deploy
npm run controller-ssh

Foundation, dataset, engine, benchmark, and controller teardown remain explicit operations. See each component's README before operating live infrastructure.