Add paging to GET /api/jobs on the scheduler and history server

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

GET /api/jobs returns every job it knows about in a single response, on both the scheduler (ballista/scheduler/src/api/handlers.rs) and the history server added in #2265.

On the scheduler this is bounded in practice, because completed jobs are cleaned up after finished_job_state_clean_up_interval_seconds and everything is gone on restart. The history server has no such bound: logs accumulate in the event-log directory until somebody prunes them, so the response grows with however long an operator chooses to retain history. A directory holding tens of thousands of jobs produces a response of a size no client wants, and the TUI refreshes the list periodically.

Raised by @milenkovicm in review on #2265.

Describe the solution you'd like

Paging on /api/jobs, added to the scheduler and the history server together rather than to one of them.

Doing it on only one side would give the same endpoint two different contracts, and the TUI (ballista-cli/src/tui/http_client.rs) points at either one, so it cannot start passing a limit until both understand it.

Sketch, to be settled in the issue rather than assumed:

• Query parameters on GET /api/jobs, defaulting to the current behaviour so existing clients keep working.
• A stable order to page over. feat(scheduler): add the history server #2265 makes the history server return jobs newest first by start time, with the job id as a tiebreaker so the order is total. Job ids are random 7-character strings (TaskManager::generate_job_id), so they cannot be paged over on their own. The live scheduler currently applies no ordering at all and would need one.
• The TUI asking for a bounded window and requesting more as the user scrolls, instead of re-fetching everything on each refresh.

Worth deciding at the same time whether the response stays a bare JSON array or gains an envelope carrying a total count or a continuation token, since that shape is the part existing clients would notice.

Describe alternatives you've considered

• Leave it unpaged and rely on pruning. What happens today, and it puts the server's response size at the mercy of an operator's retention policy. It is documented as a limitation on the history server user-guide page, which is worth doing but is not a fix.
• Cap the response server-side without a paging API. Silently truncating a list is worse than a large one, since a job that ran would simply not be visible with nothing to say so.
• Page in the history server only. Cheaper, but it is the divergence described above.

Additional context

The history server side is the more pressing of the two, since it is the one with unbounded retention. See #2265 and the review discussion there.

[milenkovicm]

Cc @martin-g can we support paging at TUI side ?

[martin-g]

Cc @martin-g can we support paging at TUI side ?

Sure! Keybindings should be defined for prev/next page navigation, some new fields should be added to the state and the http_client should make use of these new fields.

I won't be able to help with the implementation until the school year starts (mid September). I am jobless at the moment and I don't stay much in front of the computer 🏖️