Skip to content

Come back from a restart with the sessions that were running - #52

Merged
thomwolf merged 2 commits into
mainfrom
state/restore-after-restart
Aug 11, 2026
Merged

Come back from a restart with the sessions that were running#52
thomwolf merged 2 commits into
mainfrom
state/restore-after-restart

Conversation

@thomwolf

@thomwolf thomwolf commented Aug 8, 2026

Copy link
Copy Markdown
Member

A Space sleep, a reboot, or a factory reset ends every PTY at once — the server owns them all. Files and conversations survive on the bucket, but coming back means clicking each agent again, and a shell that was running something is simply gone with nothing to show it ever mattered.

This keeps a runstate snapshot on the bucket (refreshed every 30s): which sessions are alive, and which have a process actually running in them. On the next boot it starts again the ones that were still yours:

  • you prompted it inside the window — your keystrokes in the pane, the Overview reply box, or a prompt in its transcript; or
  • something was still running in it — a foreground command, a make & left in the background, an agent mid-tool-call.

The second rule is the only signal a plain shell leaves: it has no transcript to read a prompt out of. Scrollback comes back from the existing terminal-history checkpoint, so a reopened pane reads as you left it. Agents resume their own pinned conversation and wait — nothing is re-sent.

Settings → General → Restart sessions after a reboot: 1 day / 3 days / 7 days / off (default 3 days).

Also in here, since it is the same complaint: the sidebar's folded groups persist across reloads, per browser, alongside the theme and zoom.

How "is anything running in there" works

One procfs read per live session, no process walk and no subprocess: the pane root is the shell or the CLI itself (paneRootPid), and anything it started is a direct child, which Linux hands us at /proc/<pid>/task/<pid>/children. Needs no new runner exports.

Deliberate limits — all logged, none silent

  • Revivals are staggered 4s apart and capped at 12, most recently active first. The ones over the cap are named in the logs rather than quietly treated as if they weren't running.
  • A snapshot older than 30 days starts nothing. The recency rule bounds itself; this is the backstop for the work-in-flight rule, whose whole point is that it carries no clock.
  • No SIGTERM write. At shutdown the PTYs may already be going down, so a final write is as likely to record an empty Space as a true one — and it would overwrite the good record with it. A 30s-stale truth beats a fresh lie.
  • The first new snapshot waits for the boot revivals to finish, so a Space that crash-loops on boot can't erase the record of what was up.
  • lastInputAt is throttled to one write per session per minute — it is recorded per keystroke and each write lands on the FUSE bucket.

Never revived: passive panels, remote agents (they run on their own machine), sessions you stopped before the reboot, and anything already up.

Verification

  • server/test/revive.test.mjs — 13 cases over the selection rule (each window, work-only, and every never-revive case), wired into npm test.
  • Full npm test passes on this branch, with the engine installed: all checks passed.
  • End-to-end against the real Ghostty engine, not committed: three shells (one given a real sleep 300 &, one idle, one typed into) → snapshot → stopAll() → revive. The snapshot recorded work: true for exactly the busy one, and exactly the busy and typed-in shells came back. With the setting off, nothing started.
  • tsc --noEmit clean.

Note for a follow-up

Settings still tells you a factory reboot is safe because "Sessions survive on the bucket", and the README said something similar. That stopped being true when tmux left; the README paragraph is updated here, but the Settings copy is left alone to keep this diff to one subject.

thomwolf and others added 2 commits August 8, 2026 12:17
The server owns every PTY, so a Space sleep, a reboot, or a factory reset ends
all of them at once. Files and conversations survive on the bucket, but coming
back means clicking each agent again — and a shell that was running something is
simply gone, with nothing to show it ever mattered.

So keep a runstate snapshot on the bucket (every 30s): which sessions are alive,
and which have a process actually running in them. On the next boot, start again
the ones that were still yours — you prompted them inside the configured window,
or something was still running in them. The second rule is the only signal a
plain shell leaves, since it has no transcript to read a prompt out of. Their
scrollback returns from the existing terminal history checkpoint, so a reopened
pane reads as you left it.

"Is anything running in there" is one procfs read of the pane root's children,
not a process walk: the shell or CLI is the pane root, and anything it started —
a foreground command, a `make &`, an agent's tool call — is a child of it.

Settings → General → Restart sessions after a reboot: 1 / 3 / 7 days, or off.
Also persists the sidebar's folded groups per browser, alongside the theme and zoom.

Deliberate limits, all logged rather than silent: revivals are staggered 4s
apart and capped at 12 (most recent first; the rest are named in the logs), a
snapshot older than 30 days starts nothing, and there is no SIGTERM write —
at shutdown the PTYs may already be going down, so a final write is as likely to
record an empty Space as a true one, and a 30s-stale truth beats a fresh lie.

Never revived: passive panels, remote agents (they run on their own machine),
sessions you stopped before the reboot, and anything already up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… record

Three problems a review found, two of which made the feature quietly wrong.

The pane root is not the thing you're talking to. `codex` is a
`#!/usr/bin/env node` launcher that keeps the native binary as a permanent
child, and every `cont || exec run` launch shape leaves bash as the pane root
with the CLI as its child — verified on live processes here: both idle codex
panes report a child, forever, while idle claude panes report none. So "work in
flight" was permanently true for whole CLI families, and the window meant
nothing for them: they would revive on every restart at any setting. Scope the
signal to shells, which is where the rule came from — an agent has a transcript,
so recency can already see it; a shell has nothing else. Re-checked when reading
a snapshot too, so an older file can't revive an agent on a flag that was never
valid for it.

`settleAt` was set only on the success path, so every early return let the watch
overwrite the previous snapshot 30s later — including the two failures you would
most want to survive: no terminal engine, and a Space still locked because the
visibility check hadn't landed within the boot's 9s wait. Both revive nothing
AND destroyed the record that would have let the next boot do better. Hold the
first write in init(), so it holds on every path.

`lastInputAt` counted the emulator answering the TUI. The client sends all of
onData, and xterm replies to device-attribute and cursor-position queries the
moment a pane attaches — so "you sent this session something" really meant "a
pane was open on it". Filter the reply shapes; a key never looks like one.

Also: mark a snapshot written only after the write succeeds, so a FUSE write
that threw is retried rather than remembered.

Tests: the passive-panel case was vacuous (the fixture had no clock, so the
guard was never what excluded it), plus the window boundary, the stale work
flag, and the reply filter. 28 assertions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thomwolf

thomwolf commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Adversarial review pass done; pushed fixes in c258697. Two of the findings were real bugs that made the feature quietly wrong.

The work signal was permanently true for whole CLI families. hasLiveChildren assumed the pane root is the shell or the CLI itself. It isn't: /usr/local/bin/codex is #!/usr/bin/env node and keeps the native binary as a permanent child, and every cont || exec run launch shape leaves bash as the pane root with the CLI as its child. Confirmed on live processes — both idle codex pane roots report a child while idle claude pane roots report none. So codex, unpinned opencode, hermes and the fresh-claude branch would revive on every restart no matter what the window said. Now scoped to shells, which is where the rule came from: an agent has a transcript so recency already sees it; a shell has nothing else. Also re-checked when reading a snapshot, so an older file can't revive an agent on a flag that was never valid for it.

Every early return erased the record. settleAt was set only on the success path, so the 30s watch overwrote the previous snapshot on all five early returns — including the two you'd most want to survive: no terminal engine, and a Space still locked because the visibility verdict hadn't landed inside the boot's 9s wait (isPublic() fails closed while unknown). Both revive nothing and destroy the record that would have let the next boot do better. The hold now lives in init(), so it applies on every path. Verified with the reviewer's own reproduction: the record survives, and still writes the truth once the hold expires.

lastInputAt counted the emulator talking. The client sends all of onData, and xterm answers device-attribute and cursor-position queries the instant a pane attaches — so "you sent it something" really meant "a pane was open on it". Those reply shapes are now filtered; no key looks like one (arrows and friends end in uppercase AD or ~).

Plus: a snapshot is marked written only after the write succeeds, so a FUSE write that threw is retried instead of remembered.

Tests — the reviewer mutation-tested the suite and caught one vacuous case: the passive-panel fixture had no clock, so the window excluded it whether or not the guard existed. It has one now. Added the window boundary, the stale work flag, and the reply filter: 28 assertions, and npm test is green.

Known and not addressed here, worth a follow-up rather than scope creep:

  • No backoff if the revive is what kills the boot. Reviving 12 claude sessions is ~5 GB RSS on measurements from this Space; on a 16 GB Space an OOM would revive the identical 12 next boot, forever. Wants an attempt marker.
  • Demo mode revives hidden sessions — they run invisibly on a Space presenting itself as fresh. Arguably right (they're as you left them when demo goes off), but it is a choice.
  • Settings still says a factory reboot is safe because "Sessions survive on the bucket", which stopped being true when tmux left.
  • resize.test.mjs produced one non-reproducible failure under heavy parallel load (7 clean runs after); untouched by this PR, flagged only so it isn't a surprise.

@thomwolf
thomwolf merged commit 3f51c23 into main Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant