# Automatic save can overwrite last with an empty file, destroying the saved session (and causing a start-up crash loop) - #166
Open
alsmnn wants to merge 1 commit into
Conversation
…rver The generated tmux.service ran the resurrect save script as a bare ExecStop process (no $TMUX, so it depends on the default-socket server still being alive) immediately before `ExecStop=tmux kill-server`, under KillMode=control-group. On a shutdown/reboot/logout stop the server in the cgroup can be signalled while save.sh is still dumping, so the save produces an empty file. With tmux-resurrect promoting that empty dump to `last`, the previous good save is destroyed. Run the save via `tmux run-shell` so it executes inside the live server (correct socket, synchronous) and completes before kill-server; prefix both stop commands with `-` so a stop when the server is already gone doesn't write an empty save. Use KillMode=mixed and TimeoutStopSec so the save has room to finish during a shutdown stop. Drop the dead RestartSec (no Restart= is set, and auto-restart is intentionally avoided so a broken restore can't spin into a crash loop). Note: the durable fix for the empty-save data loss belongs in tmux-resurrect's save.sh (reject empty dumps before promoting to `last`); this change removes the systemd path that triggers it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
When the tmux server is torn down while an automatic save is running — on logout,
systemctl --user restart tmux, a manualtmux kill-server, or a reboot thathappens while the user is logged in — continuum's save produces a 0-byte
resurrect file and promotes it to
last. The previous good save is silentlylost. On the next start,
@continuum-restorerestores "nothing", tmux-resurrectkills the bootstrap session, the server exits, and tmux appears to "crash
immediately" on every launch until
lastis manually repaired.This is a silent data-loss bug (saved work environment destroyed) with no
error surfaced to the user.
Environment
0698e8f(currentmaster)cff343c(currentmaster)@continuum-boot 'on')@continuum-save-interval 5,@continuum-restore 'on'Linger=no(user services run per login session)Impact
lastsave is replaced by a 0-byte file.@continuum-restore 'on', the next tmux start restores nosessions; tmux-resurrect's
handle_session_0then runskill-session -t 0on the only (bootstrap) session, leaving the server with zero sessions, so it
exits. Every subsequent launch repeats this — a crash loop — until the user
discovers and manually re-points
lastat a non-empty backup.self-perpetuating: each broken start writes another empty save.
Root cause
Three interacting behaviours, two in continuum and one in resurrect:
1.
save.shpromotes an empty dump tolastwith no guard (tmux-resurrect)tmux-resurrect/scripts/save.sh→save_all():see tmux-plugins/tmux-resurrect#576
When the
tmuxdump commands cannot enumerate panes (server torn down, nosessions, or reached on the wrong socket), they print to stderr and emit
empty stdout, so
$resurrect_file_pathis 0 bytes.files_differ(
! cmp -s) is then true against a non-emptylast, so the empty file issymlinked as
last. Nothing checks that the dump succeeded or is non-empty.2. continuum runs the save detached, so it can outlive the server (tmux-continuum)
tmux-continuum/scripts/continuum_save.sh:The save is spawned in the background and its stderr is discarded. If the server
is killed immediately after (e.g. the user runs
tmux kill-server, or thestart-up crash tears the server down within a few seconds), the detached
save.shruns itstmuxcalls against a dead server → empty dump → clobber.A save is also considered "due" on the very first status render of a fresh
server:
enough_time_since_last_run_passedreads@continuum-save-last-timestampwhich defaults to
0, sonext_run = 0 + intervalandnow ≫ next_run. Thisguarantees a background save is spawned right at start-up — the moment the server
is most likely to be short-lived.
3. The generated systemd unit saves while killing the server (tmux-continuum)
tmux-continuum/scripts/handle_tmux_automatic_start/systemd_enable.sh:26-31generates:
ExecStoprunssave.shas a bare process — no$TMUX, so itstmuxcommands target the default socket and depend on the server still being alive —
immediately before
kill-server, underKillMode=control-group. On ashutdown/reboot stop (or when
TimeoutStopSecfires), the server in the cgroupcan be signalled while
save.shis still dumping → empty save → clobber. So areboot that occurs while the user is logged in poisons
lastby this path.Reproduction
Observed:
lastends up pointing at a 0-byte file; the prior good save is gone.On a real system with
@continuum-restore 'on', the next start then crash-loops.Proposed fixes
A. Guard in tmux-resurrect
save.sh(primary, durable fix)Never promote a content-less dump to
last. Patch attached(
tmux-resurrect-empty-save-guard.patch):(Belongs in tmux-resurrect; filing/linking there is recommended since the two
projects share maintainers.)
B. continuum: don't save when there is no server / no sessions
In
continuum_save.sh, skip the save if the server is gone or has no panesbefore spawning
save.sh, e.g. guard ontmux list-panes -areturning at least one line. This prevents spawning adoomed background save at start-up / teardown.
C. continuum systemd template: make the stop-save safe
In
systemd_enable.sh, generate anExecStopthat runs the save inside thelive server and completes before the kill, and don't race the cgroup teardown:
run-shellgives the save$TMUX/the correct socket and runs synchronously; ifthe server is already gone it simply fails (harmless) instead of writing an empty
file.
KillMode=mixed+ a stop timeout give the save room during shutdown.Any one of these stops the data loss; A is the minimal, robust guarantee and B/C
remove the conditions that trigger it.
Notes for maintainers
tmux-resurrect
restore.sh→handle_session_0killing the last session whenrestore finds nothing; that behaviour is arguably also worth a guard
("don't kill session 0 if no sessions were restored"), but the originating
fault is the empty save documented here.
write_unit_file_unless_existsmeans an already-generatedtmux.serviceisnever rewritten, so users who hit this and hand-fix their unit keep the fix —
but new installs get the racy template.
@nuclearglow