feat(history): add the event-log schema, writer, and reader - #2260
Merged
Conversation
…or out dto_build Move the scheduler's REST response types into a new leaf crate, `ballista-history`, and pull the graph-to-DTO construction out of the axum handlers into a pure `api::dto_build` module. Behavior preserving: the same DTOs are produced from the same state, so live REST responses are byte-identical. The existing handler tests cover this, and the helper unit tests move alongside the functions they test. This is the first step toward a history server that replays completed jobs and serves the same `/api/*` responses without a live scheduler. Splitting the DTOs into a serde-only crate lets that server build the identical wire types without depending on the scheduler's live execution graph, and moving construction out of the handlers means it can run against state that did not come from a handler request. `JobResponse::job_id` becomes a `String` rather than `ballista_core::JobId` so the new crate stays serde-only. `JobId` is `#[serde(transparent)]` over `String`, so the JSON is unchanged.
Follow-up cleanups on the extraction: - Collapse the three near-identical ExecutionStage arms in graph_to_query_stages into one destructuring match, dropping the mutable placeholder-zero summary. - Take PlanFormat by value instead of &JobQueryParams, and move PlanFormat into ballista-history. It is part of the wire contract, and the pure builder no longer imports an axum query-param type back out of the handler module. - Inject `now` into graph_to_query_stages rather than reading the clock, so replaying a stored log renders stable elapsed times. - Share percent_complete and min_start_time; use displayable() instead of the longhand DisplayableExecutionPlan::new(). - Drop the dead JobConfig alias and the unused serde_json dev-dependency, make task_status_to_dto private, and remove a duplicated test. - Enable #![warn(missing_docs)] on ballista-history and document the types, matching the other Ballista crates. - Register ballista-history with the release tooling: version bump script, publish order, and crate dependency graph.
The previous wording left it ambiguous whether the history server re-derives responses from stored execution state or replays stored DTOs. It replays them: the scheduler builds each response once against the live graph and writes it to the event log, so byte-identical output is a structural property rather than two implementations agreeing. Also records the consequence, that anything not captured at write time cannot be recovered at replay time.
The crate holds the /api/* wire types, and it has three parties, not one: the scheduler serves them, the web TUI deserializes them, and a future history server will serve replayed copies. Naming it after the history server made it awkward for the TUI, which parses live scheduler responses and today keeps its own duplicate declarations. Renaming it after the contract it defines removes that friction. The event-log schema, writer, and reader can then land as a separate ballista-history crate that depends on this one.
Second step toward the history server (apache#1923), after the wire-type extraction in apache#2256. Adds the durable format and the machinery to read and write it. Nothing in the scheduler calls this yet. - A versioned JSONL schema. JobStart / StageStart / StageEnd / TaskEnd form an incremental timeline; the terminal JobEnd embeds the finished REST responses, so replay re-serves what the scheduler already built rather than re-deriving it. - An async buffered EventLogWriter. All file I/O runs on a background task so the scheduler's event loop never waits on disk. Timeline events are dropped rather than allowed to block when the queue backs up, since losing a progress record beats stalling scheduling. JobEnd is the exception and waits for capacity, because a job missing it is invisible to the history server. - A reader that folds a completed log back into the served payload, skipping malformed lines rather than failing. Restores the JobConfig alias to ballista-api-types, which now has a real consumer in the JobEnd record. TaskEnd names a task rather than a partition: under the multi-partition task model a task owns a slice of partitions, and TaskStatus carries task_id, not partition_id.
andygrove
marked this pull request as draft
August 8, 2026 17:36
A log is written once and may be read years later by a much newer binary, so the guarantee has to run one way: a reader accepts any log whose version is not newer than its own. That is the opposite of BALLISTA_PROTOCOL_VERSION, the strict-equality handshake between scheduler and executor, where both ends are live and upgraded together. The format did not support that yet. Four changes: Self-describing envelope. Every line is now a LogRecord carrying `ev`, `version` and an opaque `data` payload, so a reader can route on the kind and check the version before committing to a shape it may not understand. Previously only JobStart and JobEnd carried a version at all. Stored responses are opaque. JobEnd holds the finished /api/* responses as raw JSON rather than typed structs, plus a small frozen JobIndex for listing. Those responses are ballista-api-types shapes, which change with the live REST contract: partition_id went from u32 to Vec<u32> and TaskStatus::Failed gained a field within one release cycle. Stored typed, either change would have made every older log unreadable, and the job would have silently disappeared. Stored raw, nothing ever parses the inner shape and replay relays the exact bytes. Unreadable is no longer indistinguishable from absent. read_completed_job returned Ok(None) both when a log had no JobEnd and when it had one that could not be parsed, and the loader treated that as "still running" and said nothing. It now returns a ReadError distinguishing an unsupported version from a malformed record. The version is actually checked. A record newer than SCHEMA_VERSION is reported as such rather than skipped. Adds testdata/schema-v1.eventlog, a frozen log replayed by CI on every build. It includes a record kind this build does not know and a stages payload carrying the multi-partition shape, so it exercises both forward-compatibility paths. Verified it fails as intended: a field rename in a stored type breaks all four compatibility tests.
andygrove
marked this pull request as ready for review
August 9, 2026 13:07
# Conflicts: # Cargo.toml # ballista/api-types/src/dto.rs # dev/release/README.md # dev/release/crate-deps.dot # dev/update_ballista_versions.py
This was referenced Aug 9, 2026
The index exists so the history server can render GET /api/jobs without parsing the stored payloads, but it was missing num_stages, completed_stages and percent_complete, which that response includes. It could not actually serve the list it was there for. Adds the three fields and updates the v1 fixture to match. Doing this before release, while the schema is still unpublished, so the frozen fixture stays a faithful record of what v1 actually looks like.
andygrove
force-pushed
the
history-event-crate
branch
from
August 9, 2026 13:53
349c506 to
4e856b3
Compare
Member
Author
|
Thanks for the review @milenkovicm |
andygrove
added a commit
that referenced
this pull request
Aug 9, 2026
…2264) * refactor(scheduler): extract REST DTOs into ballista-history and factor out dto_build Move the scheduler's REST response types into a new leaf crate, `ballista-history`, and pull the graph-to-DTO construction out of the axum handlers into a pure `api::dto_build` module. Behavior preserving: the same DTOs are produced from the same state, so live REST responses are byte-identical. The existing handler tests cover this, and the helper unit tests move alongside the functions they test. This is the first step toward a history server that replays completed jobs and serves the same `/api/*` responses without a live scheduler. Splitting the DTOs into a serde-only crate lets that server build the identical wire types without depending on the scheduler's live execution graph, and moving construction out of the handlers means it can run against state that did not come from a handler request. `JobResponse::job_id` becomes a `String` rather than `ballista_core::JobId` so the new crate stays serde-only. `JobId` is `#[serde(transparent)]` over `String`, so the JSON is unchanged. * refactor(scheduler): tighten the DTO extraction Follow-up cleanups on the extraction: - Collapse the three near-identical ExecutionStage arms in graph_to_query_stages into one destructuring match, dropping the mutable placeholder-zero summary. - Take PlanFormat by value instead of &JobQueryParams, and move PlanFormat into ballista-history. It is part of the wire contract, and the pure builder no longer imports an axum query-param type back out of the handler module. - Inject `now` into graph_to_query_stages rather than reading the clock, so replaying a stored log renders stable elapsed times. - Share percent_complete and min_start_time; use displayable() instead of the longhand DisplayableExecutionPlan::new(). - Drop the dead JobConfig alias and the unused serde_json dev-dependency, make task_status_to_dto private, and remove a duplicated test. - Enable #![warn(missing_docs)] on ballista-history and document the types, matching the other Ballista crates. - Register ballista-history with the release tooling: version bump script, publish order, and crate dependency graph. * docs(history): state the write-once, replay-verbatim data flow The previous wording left it ambiguous whether the history server re-derives responses from stored execution state or replays stored DTOs. It replays them: the scheduler builds each response once against the live graph and writes it to the event log, so byte-identical output is a structural property rather than two implementations agreeing. Also records the consequence, that anything not captured at write time cannot be recovered at replay time. * refactor: rename ballista-history to ballista-api-types The crate holds the /api/* wire types, and it has three parties, not one: the scheduler serves them, the web TUI deserializes them, and a future history server will serve replayed copies. Naming it after the history server made it awkward for the TUI, which parses live scheduler responses and today keeps its own duplicate declarations. Renaming it after the contract it defines removes that friction. The event-log schema, writer, and reader can then land as a separate ballista-history crate that depends on this one. * feat(history): add the event-log schema, writer, and reader Second step toward the history server (#1923), after the wire-type extraction in #2256. Adds the durable format and the machinery to read and write it. Nothing in the scheduler calls this yet. - A versioned JSONL schema. JobStart / StageStart / StageEnd / TaskEnd form an incremental timeline; the terminal JobEnd embeds the finished REST responses, so replay re-serves what the scheduler already built rather than re-deriving it. - An async buffered EventLogWriter. All file I/O runs on a background task so the scheduler's event loop never waits on disk. Timeline events are dropped rather than allowed to block when the queue backs up, since losing a progress record beats stalling scheduling. JobEnd is the exception and waits for capacity, because a job missing it is invisible to the history server. - A reader that folds a completed log back into the served payload, skipping malformed lines rather than failing. Restores the JobConfig alias to ballista-api-types, which now has a real consumer in the JobEnd record. TaskEnd names a task rather than a partition: under the multi-partition task model a task owns a slice of partitions, and TaskStatus carries task_id, not partition_id. * feat(history): make the event log survive future Ballista versions A log is written once and may be read years later by a much newer binary, so the guarantee has to run one way: a reader accepts any log whose version is not newer than its own. That is the opposite of BALLISTA_PROTOCOL_VERSION, the strict-equality handshake between scheduler and executor, where both ends are live and upgraded together. The format did not support that yet. Four changes: Self-describing envelope. Every line is now a LogRecord carrying `ev`, `version` and an opaque `data` payload, so a reader can route on the kind and check the version before committing to a shape it may not understand. Previously only JobStart and JobEnd carried a version at all. Stored responses are opaque. JobEnd holds the finished /api/* responses as raw JSON rather than typed structs, plus a small frozen JobIndex for listing. Those responses are ballista-api-types shapes, which change with the live REST contract: partition_id went from u32 to Vec<u32> and TaskStatus::Failed gained a field within one release cycle. Stored typed, either change would have made every older log unreadable, and the job would have silently disappeared. Stored raw, nothing ever parses the inner shape and replay relays the exact bytes. Unreadable is no longer indistinguishable from absent. read_completed_job returned Ok(None) both when a log had no JobEnd and when it had one that could not be parsed, and the loader treated that as "still running" and said nothing. It now returns a ReadError distinguishing an unsupported version from a malformed record. The version is actually checked. A record newer than SCHEMA_VERSION is reported as such rather than skipped. Adds testdata/schema-v1.eventlog, a frozen log replayed by CI on every build. It includes a record kind this build does not know and a stages payload carrying the multi-partition shape, so it exercises both forward-compatibility paths. Verified it fails as intended: a field rename in a stored type breaks all four compatibility tests. * feat(scheduler): record a per-job event log behind --event-log-dir Third step toward the history server (#1923). Wires the event-log crate from #2260 into the scheduler. Nothing reads these logs yet; the history server that serves them is the next slice. - New --event-log-dir flag, off by default. When unset there is no channel, no background task, no file, and no per-event work beyond one Option check in the event loop. - event_log.rs builds HistoryEvents from execution-graph state, reusing the same api::dto_build builders that back the live REST API, so a job's stored record and its GET /api/job/{id} response are the same bytes for the same graph. - A tee at the top of QueryStageScheduler::on_receive maps JobSubmitted, TaskUpdating, JobFinished, JobRunningFailed and JobCancel onto history events. JobCancel is handled in the tee specifically because the handler below it drops the graph, making that the last point at which a cancelled job can be recorded. Its status is overridden on the typed DTO before anything is serialized, rather than by rewriting the stored JSON, so the payload stays a faithful serialization of one value and can be relayed verbatim. JobPlanningFailed is deliberately absent: it is posted instead of JobSubmitted, so the job has neither an execution graph nor an open log. Failure to build an event costs the job its record, never its execution. A missing graph or a serialization error is logged and skipped. The stage snapshot embedded in JobEnd is rendered as of completed_at rather than the wall clock, so replaying a log is deterministic. * feat(history): carry the job-list fields in JobIndex The index exists so the history server can render GET /api/jobs without parsing the stored payloads, but it was missing num_stages, completed_stages and percent_complete, which that response includes. It could not actually serve the list it was there for. Adds the three fields and updates the v1 fixture to match. Doing this before release, while the schema is still unpublished, so the frozen fixture stays a faithful record of what v1 actually looks like. * fix: populate the new JobIndex fields
andygrove
added a commit
that referenced
this pull request
Aug 28, 2026
* refactor(scheduler): extract REST DTOs into ballista-history and factor out dto_build Move the scheduler's REST response types into a new leaf crate, `ballista-history`, and pull the graph-to-DTO construction out of the axum handlers into a pure `api::dto_build` module. Behavior preserving: the same DTOs are produced from the same state, so live REST responses are byte-identical. The existing handler tests cover this, and the helper unit tests move alongside the functions they test. This is the first step toward a history server that replays completed jobs and serves the same `/api/*` responses without a live scheduler. Splitting the DTOs into a serde-only crate lets that server build the identical wire types without depending on the scheduler's live execution graph, and moving construction out of the handlers means it can run against state that did not come from a handler request. `JobResponse::job_id` becomes a `String` rather than `ballista_core::JobId` so the new crate stays serde-only. `JobId` is `#[serde(transparent)]` over `String`, so the JSON is unchanged. * refactor(scheduler): tighten the DTO extraction Follow-up cleanups on the extraction: - Collapse the three near-identical ExecutionStage arms in graph_to_query_stages into one destructuring match, dropping the mutable placeholder-zero summary. - Take PlanFormat by value instead of &JobQueryParams, and move PlanFormat into ballista-history. It is part of the wire contract, and the pure builder no longer imports an axum query-param type back out of the handler module. - Inject `now` into graph_to_query_stages rather than reading the clock, so replaying a stored log renders stable elapsed times. - Share percent_complete and min_start_time; use displayable() instead of the longhand DisplayableExecutionPlan::new(). - Drop the dead JobConfig alias and the unused serde_json dev-dependency, make task_status_to_dto private, and remove a duplicated test. - Enable #![warn(missing_docs)] on ballista-history and document the types, matching the other Ballista crates. - Register ballista-history with the release tooling: version bump script, publish order, and crate dependency graph. * docs(history): state the write-once, replay-verbatim data flow The previous wording left it ambiguous whether the history server re-derives responses from stored execution state or replays stored DTOs. It replays them: the scheduler builds each response once against the live graph and writes it to the event log, so byte-identical output is a structural property rather than two implementations agreeing. Also records the consequence, that anything not captured at write time cannot be recovered at replay time. * refactor: rename ballista-history to ballista-api-types The crate holds the /api/* wire types, and it has three parties, not one: the scheduler serves them, the web TUI deserializes them, and a future history server will serve replayed copies. Naming it after the history server made it awkward for the TUI, which parses live scheduler responses and today keeps its own duplicate declarations. Renaming it after the contract it defines removes that friction. The event-log schema, writer, and reader can then land as a separate ballista-history crate that depends on this one. * feat(history): add the event-log schema, writer, and reader Second step toward the history server (#1923), after the wire-type extraction in #2256. Adds the durable format and the machinery to read and write it. Nothing in the scheduler calls this yet. - A versioned JSONL schema. JobStart / StageStart / StageEnd / TaskEnd form an incremental timeline; the terminal JobEnd embeds the finished REST responses, so replay re-serves what the scheduler already built rather than re-deriving it. - An async buffered EventLogWriter. All file I/O runs on a background task so the scheduler's event loop never waits on disk. Timeline events are dropped rather than allowed to block when the queue backs up, since losing a progress record beats stalling scheduling. JobEnd is the exception and waits for capacity, because a job missing it is invisible to the history server. - A reader that folds a completed log back into the served payload, skipping malformed lines rather than failing. Restores the JobConfig alias to ballista-api-types, which now has a real consumer in the JobEnd record. TaskEnd names a task rather than a partition: under the multi-partition task model a task owns a slice of partitions, and TaskStatus carries task_id, not partition_id. * feat(history): make the event log survive future Ballista versions A log is written once and may be read years later by a much newer binary, so the guarantee has to run one way: a reader accepts any log whose version is not newer than its own. That is the opposite of BALLISTA_PROTOCOL_VERSION, the strict-equality handshake between scheduler and executor, where both ends are live and upgraded together. The format did not support that yet. Four changes: Self-describing envelope. Every line is now a LogRecord carrying `ev`, `version` and an opaque `data` payload, so a reader can route on the kind and check the version before committing to a shape it may not understand. Previously only JobStart and JobEnd carried a version at all. Stored responses are opaque. JobEnd holds the finished /api/* responses as raw JSON rather than typed structs, plus a small frozen JobIndex for listing. Those responses are ballista-api-types shapes, which change with the live REST contract: partition_id went from u32 to Vec<u32> and TaskStatus::Failed gained a field within one release cycle. Stored typed, either change would have made every older log unreadable, and the job would have silently disappeared. Stored raw, nothing ever parses the inner shape and replay relays the exact bytes. Unreadable is no longer indistinguishable from absent. read_completed_job returned Ok(None) both when a log had no JobEnd and when it had one that could not be parsed, and the loader treated that as "still running" and said nothing. It now returns a ReadError distinguishing an unsupported version from a malformed record. The version is actually checked. A record newer than SCHEMA_VERSION is reported as such rather than skipped. Adds testdata/schema-v1.eventlog, a frozen log replayed by CI on every build. It includes a record kind this build does not know and a stages payload carrying the multi-partition shape, so it exercises both forward-compatibility paths. Verified it fails as intended: a field rename in a stored type breaks all four compatibility tests. * feat(scheduler): record a per-job event log behind --event-log-dir Third step toward the history server (#1923). Wires the event-log crate from #2260 into the scheduler. Nothing reads these logs yet; the history server that serves them is the next slice. - New --event-log-dir flag, off by default. When unset there is no channel, no background task, no file, and no per-event work beyond one Option check in the event loop. - event_log.rs builds HistoryEvents from execution-graph state, reusing the same api::dto_build builders that back the live REST API, so a job's stored record and its GET /api/job/{id} response are the same bytes for the same graph. - A tee at the top of QueryStageScheduler::on_receive maps JobSubmitted, TaskUpdating, JobFinished, JobRunningFailed and JobCancel onto history events. JobCancel is handled in the tee specifically because the handler below it drops the graph, making that the last point at which a cancelled job can be recorded. Its status is overridden on the typed DTO before anything is serialized, rather than by rewriting the stored JSON, so the payload stays a faithful serialization of one value and can be relayed verbatim. JobPlanningFailed is deliberately absent: it is posted instead of JobSubmitted, so the job has neither an execution graph nor an open log. Failure to build an event costs the job its record, never its execution. A missing graph or a serialization error is logged and skipped. The stage snapshot embedded in JobEnd is rendered as of completed_at rather than the wall clock, so replaying a log is deterministic. * feat(history): carry the job-list fields in JobIndex The index exists so the history server can render GET /api/jobs without parsing the stored payloads, but it was missing num_stages, completed_stages and percent_complete, which that response includes. It could not actually serve the list it was there for. Adds the three fields and updates the v1 fixture to match. Doing this before release, while the schema is still unpublished, so the frozen fixture stays a faithful record of what v1 actually looks like. * fix: populate the new JobIndex fields * feat(scheduler): add the history server Final step of #1923, after #2256, #2260 and #2264. Serves completed jobs from stored event logs, so the existing TUI can browse them with no scheduler running. - ballista-history-server --event-log-dir <dir> loads every completed log at startup and serves the same /api/* paths as the live scheduler. - Corrupt or unreadable logs are logged and skipped rather than failing startup, so one bad file cannot hide every other job. - /api/executors returns empty and /api/state a static payload, since there is no cluster behind a history server, so TUI screens expecting them still load. - A user-guide page covering both flags, the endpoints, and the operational caveats. The stored /api/job/{id} and /api/job/{id}/stages payloads are relayed as raw JSON rather than deserialized and re-serialized, so clients receive the exact bytes the live scheduler produced and a later change to the REST types cannot make an existing log unservable. The job list is rebuilt from the frozen JobIndex instead, which carries exactly the fields that endpoint includes. Brings back history_store_serves_byte_identical_json_to_live_scheduler, held out of #2264 because it needs HistoryStore. It emits a real JobEnd through the real async writer, loads it back through HistoryStore::load, and compares the served bytes against what the live builders produce for the same graph. Now a literal byte comparison rather than a structural one. * refactor(history): index event logs instead of loading them into memory HistoryStore kept a full ReplayedJob per log: both plan-bearing REST payloads, the session config and the DOT graph. For a job with many tasks that runs to megabytes, held resident for every job in the directory whether or not anyone ever opens it. Keep only the JobIndex and the file path, which is all GET /api/jobs needs, and read a job's payload back from its log per request inside spawn_blocking. Startup still scans the directory once, but decodes only the summary out of each JobEnd rather than materialising payloads it would immediately drop. Corruption confined to the payloads is no longer caught at startup, so it surfaces as a 500 naming the failing log. A log that loses its terminal record after indexing returns 404. * fix(history): list completed jobs newest first Job ids are random 7-character strings, so ordering /api/jobs by id put the list in an order that means nothing. Sort by start time descending, with the id as a tiebreaker so two jobs that started in the same millisecond cannot swap places between requests. This is also the order the TUI puts the list into once it has it, and the order a future ?limit= would want to truncate. * docs(history): link the paging limitation to #2270 * docs(history): correct the TUI invocation against a history server The TUI reads its scheduler URL from configuration, not from --host and --port, so the documented command silently browsed the live scheduler on the default port instead of the history server. * feat(history): rescan the event-log directory for new jobs The history server indexed the log directory once at startup, so a job that finished afterwards was invisible until the process was restarted. That is the normal case rather than an edge one: the directory it reads is the one live schedulers keep writing to. Rescan it every --update-interval-seconds (10 by default, 0 to disable) and fold the changes into the index. Only logs whose size or mtime has moved are opened, so a pass over an unchanged directory costs one stat per file; logs that have been deleted are dropped from the index instead of staying listed and failing when opened. * docs(history): drop the intra-doc link to a private type
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #1923. Second of four PRs splitting up #1925.
Rationale for this change
#1923 asks for a Spark-History-Server equivalent: the scheduler records a durable log of each job, and a standalone server replays those logs and serves the same
/api/*responses so the existing TUI can browse completed jobs with no scheduler running.This PR adds the format and the machinery to read and write it, and nothing else. Nothing in the scheduler calls it yet, so it can be reviewed purely on its own terms: is the on-disk schema right, and does the writer behave under load?
Those are the two questions worth isolating. The schema is durable and cross-machine, so it is the part that is expensive to change later. And the writer sits next to the scheduler's hot path, so its backpressure behaviour matters more than its line count suggests.
What changes are included in this PR?
New
ballista-historycrate, depending onballista-api-typesplus serde, tokio and log.event.rs— the versioned JSONL schema. Every line is aLogRecord: a self-describing envelope carryingev,version, and an opaquedatapayload.JobStart,StageStart,StageEndandTaskEndform an incremental timeline; the terminalJobEndcarries the finished/api/*responses.Storing the built responses is what makes replay trustworthy: the history server re-serves exactly what the scheduler produced rather than re-deriving it, so there is one definition and one place that populates it instead of two implementations that have to keep agreeing. The consequence, worth knowing up front, is that anything not captured at write time cannot be recovered later.
Identifiers are fixed-width
u32rather thanusize, since a log written by a 64-bit scheduler has to mean the same thing to any reader.writer.rs— an async bufferedEventLogWriter. All file I/O happens on a background task, so the scheduler's event loop never waits on disk. One append-mode handle per job is held for the life of the process.The backpressure split is deliberate.
appendnever blocks and drops the event if the queue is full, because losing a progress record is better than stalling scheduling.append_finalinstead waits for capacity, because a job whoseJobEndwas dropped is invisible to the history server entirely.finish_jobthen flushes and closes the handle, ordered after the terminal event by the single-consumer channel.reader.rs— folds a completed log back into the payload a server would serve. A file is "completed" once it contains aJobEnd. Unrecognised lines are skipped rather than treated as fatal, so a log truncated by a crash still yields its job if the terminal record survived, and a future scheduler can add record kinds without breaking today's reader.Compatibility
A log is written once and may be read years later by a much newer Ballista, so the guarantee runs one way: a reader accepts any log whose version is not newer than its own. There is no way to upgrade the writer of a file that already exists.
That is deliberately the opposite of
BALLISTA_PROTOCOL_VERSION, the scheduler/executor handshake, which is strict equality and rejects on mismatch. Both ends of that handshake are live and upgraded together, so refusing to proceed is the safe move. Here there is nothing to negotiate with, and refusing means losing the job.Four things make that guarantee real rather than aspirational:
The envelope. Routing on
evandversionbefore parsingdatais what lets a reader tell "written by a newer Ballista" apart from "corrupt". Parsing the whole record up front collapses both into the same indistinguishable failure. Previously onlyJobStartandJobEndcarried a version at all, and nothing ever read it.The stored responses are opaque.
JobEndholds them as raw JSON rather than typed structs, alongside a small frozenJobIndexthe history server uses to list and sort. This matters because those responses areballista-api-typesshapes, which evolve with the live REST contract:TaskSummary::partition_idwent fromu32toVec<u32>andTaskStatus::Failedgained a field, both within a single release cycle. Stored typed, either change would have made every older log fail to deserialize, and the job would have vanished from the UI. Stored raw, nothing ever parses the inner shape and the log is immune to REST churn. It also makes replayed output byte-identical rather than merely equivalent, which a typed round trip cannot promise —serde_json::Valuesorts object keys.Unreadable is no longer indistinguishable from absent.
read_completed_jobused to returnOk(None)both when a log had noJobEndand when it had one that could not be parsed, andHistoryStore::loadtreated that as "still running" and logged nothing. A version-incompatible log meant a job silently disappearing. It now returns aReadErrorseparating an unsupported version from a malformed record.A frozen golden log.
testdata/schema-v1.eventlogis checked in and replayed by CI on every build. It deliberately contains a record kind this build has never heard of, and a stages payload carrying the multi-partitionpartition_idarray, so it exercises both forward-compatibility paths. It is never regenerated: the point is that it was written by an older Ballista and must stay readable.I verified it actually fires rather than just existing. Renaming a field in a stored type — a change that compiles cleanly everywhere — fails all four compatibility tests with
Malformed("missing field \name`")`.JobConfigreturns toballista-api-types. I removed it in #2256 as dead; theJobEndrecord is its real consumer.TaskEndnames a task, not a partition. Under the multi-partition task model a task owns a slice of partitions, andTaskStatuscarriestask_idrather thanpartition_id. #1925 predates that change and recorded apartitionfield that no longer has a source.Release tooling.
ballista-historyis registered indev/update_ballista_versions.py, the publish order, and the crate dependency graph.Are there any user-facing changes?
No. The crate is new and nothing depends on it yet. No scheduler code changes, no configuration, no API changes.
Verified locally:
cargo test -p ballista-historypasses (14 tests, 4 of them the golden-log compatibility suite),cargo check -p ballista-scheduleris unaffected, clippy is clean for both crates with--all-features -D warnings, and fmt/taplo are clean.Follow-ups
--event-log-dirflag, off by default.Both are already written and forward-ported; they are waiting on this landing rather than on being figured out.
Separately, a written cross-version compatibility policy covering all three of Ballista's version-sensitive surfaces (the executor handshake, the REST API, and this log) is worth its own issue. Two of the three now have a mechanism; the REST API has none, which is what #2257 was.