Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The event log added in #2264 is hard-wired to the local filesystem. EventLogWriter (ballista/history/src/writer.rs) opens a tokio::fs::File per job and appends JSONL to <event_log_dir>/<job_id>.eventlog, the reader (ballista/history/src/reader.rs) takes a &Path and opens it with std::fs::File::open, and HistoryStore::load (ballista/scheduler/src/history/mod.rs) discovers jobs with std::fs::read_dir.
That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed:
- In Kubernetes the scheduler pod's filesystem is ephemeral, so job history dies with the pod unless someone attaches and manages a PVC.
- The history server has to run somewhere that can see the same directory the scheduler wrote to, which in practice means a shared volume rather than a separately deployed read-only service.
- With multiple schedulers there is no single directory holding the whole cluster's history.
Writing logs to an object store (S3, GCS, Azure, or anything else object_store supports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere.
Describe the solution you'd like
Put the object_store crate behind the event log's read and write paths, so --event-log-dir accepts a URL (s3://bucket/prefix, file:///var/log/ballista, plain paths staying local for compatibility):
EventLogWriter::new takes an Arc<dyn ObjectStore> plus a base Path instead of a PathBuf.- The reader takes a store and a path rather than
&Path, and HistoryStore::load uses ObjectStore::list in place of read_dir.
- The history server binary resolves its own store from the same URL, so it can be deployed on its own with no shared volume.
Describe alternatives you've considered
- Keep it local and let operators sync the directory out of band (sidecar,
aws s3 sync, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access.
- Trait-abstract the storage layer ourselves rather than depending on
object_store. More code for less coverage; object_store is already a workspace dependency, already used by ballista-scheduler, and already how the rest of the stack talks to remote storage.
Additional context
The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort:
- Buffer per job, write once on
JobEnd. Simplest, and it fits how the log is consumed today — the reader only looks at the terminal JobEnd record and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs.
- Multipart upload per job, flushing parts as events accumulate. Preserves incremental durability, but needs careful handling of aborted uploads for jobs whose scheduler dies mid-flight.
- Write locally, upload on completion. Keeps crash-durability on local disk and gets the object into remote storage once the job finishes; needs a cleanup/retry story for uploads that fail.
Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store.
Follow-on from #2264 (comment).
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The event log added in #2264 is hard-wired to the local filesystem.
EventLogWriter(ballista/history/src/writer.rs) opens atokio::fs::Fileper job and appends JSONL to<event_log_dir>/<job_id>.eventlog, the reader (ballista/history/src/reader.rs) takes a&Pathand opens it withstd::fs::File::open, andHistoryStore::load(ballista/scheduler/src/history/mod.rs) discovers jobs withstd::fs::read_dir.That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed:
Writing logs to an object store (S3, GCS, Azure, or anything else
object_storesupports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere.Describe the solution you'd like
Put the
object_storecrate behind the event log's read and write paths, so--event-log-diraccepts a URL (s3://bucket/prefix,file:///var/log/ballista, plain paths staying local for compatibility):EventLogWriter::newtakes anArc<dyn ObjectStore>plus a basePathinstead of aPathBuf.&Path, andHistoryStore::loadusesObjectStore::listin place ofread_dir.Describe alternatives you've considered
aws s3 sync, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access.object_store. More code for less coverage;object_storeis already a workspace dependency, already used byballista-scheduler, and already how the rest of the stack talks to remote storage.Additional context
The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort:
JobEnd. Simplest, and it fits how the log is consumed today — the reader only looks at the terminalJobEndrecord and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs.Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store.
Follow-on from #2264 (comment).