Skip to content

# Automatic save can overwrite last with an empty file, destroying the saved session (and causing a start-up crash loop) - #166

Open
alsmnn wants to merge 1 commit into
tmux-plugins:masterfrom
alsmnn:fix/systemd-execstop-save-race
Open

# Automatic save can overwrite last with an empty file, destroying the saved session (and causing a start-up crash loop)#166
alsmnn wants to merge 1 commit into
tmux-plugins:masterfrom
alsmnn:fix/systemd-execstop-save-race

Conversation

@alsmnn

@alsmnn alsmnn commented Jul 7, 2026

Copy link
Copy Markdown

TL;DR

When the tmux server is torn down while an automatic save is running — on logout,
systemctl --user restart tmux, a manual tmux kill-server, or a reboot that
happens while the user is logged in — continuum's save produces a 0-byte
resurrect file and promotes it to last. The previous good save is silently
lost. On the next start, @continuum-restore restores "nothing", tmux-resurrect
kills the bootstrap session, the server exits, and tmux appears to "crash
immediately" on every launch until last is manually repaired.

This is a silent data-loss bug (saved work environment destroyed) with no
error surfaced to the user.

Environment

  • tmux-continuum: 0698e8f (current master)
  • tmux-resurrect: cff343c (current master)
  • tmux 3.4, libevent 2.1.12, ncurses 6.4
  • Ubuntu 24.04.4 LTS, systemd user service (@continuum-boot 'on')
  • @continuum-save-interval 5, @continuum-restore 'on'
  • Linger=no (user services run per login session)

Impact

  • The last known-good last save is replaced by a 0-byte file.
  • Combined with @continuum-restore 'on', the next tmux start restores no
    sessions; tmux-resurrect's handle_session_0 then runs kill-session -t 0
    on the only (bootstrap) session, leaving the server with zero sessions, so it
    exits. Every subsequent launch repeats this — a crash loop — until the user
    discovers and manually re-points last at a non-empty backup.
  • Because saves also run on service stop (see below), the loop is
    self-perpetuating: each broken start writes another empty save.

Root cause

Three interacting behaviours, two in continuum and one in resurrect:

1. save.sh promotes an empty dump to last with no guard (tmux-resurrect)

tmux-resurrect/scripts/save.shsave_all():

fetch_and_dump_grouped_sessions > "$resurrect_file_path"   # stdout only
dump_panes   >> "$resurrect_file_path"   # tmux list-panes  -a
dump_windows >> "$resurrect_file_path"   # tmux list-windows -a
dump_state   >> "$resurrect_file_path"   # tmux display-message -p
...
if files_differ "$resurrect_file_path" "$last_resurrect_file"; then
    ln -fs "$(basename "$resurrect_file_path")" "$last_resurrect_file"   # promote
else
    rm "$resurrect_file_path"
fi

see tmux-plugins/tmux-resurrect#576

When the tmux dump commands cannot enumerate panes (server torn down, no
sessions, or reached on the wrong socket), they print to stderr and emit
empty stdout, so $resurrect_file_path is 0 bytes. files_differ
(! cmp -s) is then true against a non-empty last, so the empty file is
symlinked as last. Nothing checks that the dump succeeded or is non-empty.

2. continuum runs the save detached, so it can outlive the server (tmux-continuum)

tmux-continuum/scripts/continuum_save.sh:

"$resurrect_save_script_path" "quiet" >/dev/null 2>&1 &   # background
set_last_save_timestamp

The save is spawned in the background and its stderr is discarded. If the server
is killed immediately after (e.g. the user runs tmux kill-server, or the
start-up crash tears the server down within a few seconds), the detached
save.sh runs its tmux calls against a dead server → empty dump → clobber.

A save is also considered "due" on the very first status render of a fresh
server: enough_time_since_last_run_passed reads @continuum-save-last-timestamp
which defaults to 0, so next_run = 0 + interval and now ≫ next_run. This
guarantees a background save is spawned right at start-up — the moment the server
is most likely to be short-lived.

3. The generated systemd unit saves while killing the server (tmux-continuum)

tmux-continuum/scripts/handle_tmux_automatic_start/systemd_enable.sh:26-31
generates:

ExecStart=${tmux_path} ${systemd_tmux_server_start_cmd}

ExecStop=${resurrect_save_script_path}
ExecStop=${tmux_path} kill-server
KillMode=control-group

ExecStop runs save.sh as a bare process — no $TMUX, so its tmux
commands target the default socket and depend on the server still being alive —
immediately before kill-server, under KillMode=control-group. On a
shutdown/reboot stop (or when TimeoutStopSec fires), the server in the cgroup
can be signalled while save.sh is still dumping → empty save → clobber. So a
reboot that occurs while the user is logged in poisons last by this path.

Reproduction

SOCK=repro; DIR=$(mktemp -d)
tmux -L $SOCK new-session -d -s work
tmux -L $SOCK set-option -g @resurrect-dir "$DIR"
# 1) a good save exists
tmux -L $SOCK run-shell "$HOME/.config/tmux/plugins/tmux-resurrect/scripts/save.sh quiet"
ls -l "$DIR/last"           # -> non-empty, e.g. 165 bytes

# 2) server dies while a save is in flight (background save outliving the server,
#    OR ExecStop save racing kill-server). Emulate: resolve dir while alive,
#    kill the server, then run the dump sequence:
NEW="$DIR/tmux_resurrect_new.txt"
tmux -L $SOCK kill-server
tmux list-panes  -a -F pane   > "$NEW" 2>/dev/null
tmux list-windows -a -F window >> "$NEW" 2>/dev/null
tmux display-message -p state  >> "$NEW" 2>/dev/null
[ -s "$NEW" ] || echo "dump is EMPTY"          # -> EMPTY
# save_all would now: files_differ EMPTY vs good last -> true -> ln -fs EMPTY last

Observed: last ends up pointing at a 0-byte file; the prior good save is gone.
On a real system with @continuum-restore 'on', the next start then crash-loops.

Proposed fixes

A. Guard in tmux-resurrect save.sh (primary, durable fix)

Never promote a content-less dump to last. Patch attached
(tmux-resurrect-empty-save-guard.patch):

 	dump_state   >> "$resurrect_file_path"
+	# Guard: never promote an empty / content-less dump to "last".
+	if [ ! -s "$resurrect_file_path" ] || ! \grep -q "^pane${d}" "$resurrect_file_path"; then
+		rm -f "$resurrect_file_path"
+		return 1
+	fi
 	execute_hook "post-save-layout" "$resurrect_file_path"
 	if files_differ "$resurrect_file_path" "$last_resurrect_file"; then

(Belongs in tmux-resurrect; filing/linking there is recommended since the two
projects share maintainers.)

B. continuum: don't save when there is no server / no sessions

In continuum_save.sh, skip the save if the server is gone or has no panes
before spawning save.sh, e.g. guard on
tmux list-panes -a returning at least one line. This prevents spawning a
doomed background save at start-up / teardown.

C. continuum systemd template: make the stop-save safe

In systemd_enable.sh, generate an ExecStop that runs the save inside the
live server and completes before the kill, and don't race the cgroup teardown:

ExecStop=-${tmux_path} run-shell "${resurrect_save_script_path} quiet"
ExecStop=-${tmux_path} kill-server
KillMode=mixed
TimeoutStopSec=20

run-shell gives the save $TMUX/the correct socket and runs synchronously; if
the server is already gone it simply fails (harmless) instead of writing an empty
file. KillMode=mixed + a stop timeout give the save room during shutdown.

Any one of these stops the data loss; A is the minimal, robust guarantee and B/C
remove the conditions that trigger it.

Notes for maintainers

  • The user-visible symptom ("tmux crashes on launch") is produced by
    tmux-resurrect restore.shhandle_session_0 killing the last session when
    restore finds nothing; that behaviour is arguably also worth a guard
    ("don't kill session 0 if no sessions were restored"), but the originating
    fault is the empty save documented here.
  • write_unit_file_unless_exists means an already-generated tmux.service is
    never rewritten, so users who hit this and hand-fix their unit keep the fix —
    but new installs get the racy template.

@nuclearglow

…rver

The generated tmux.service ran the resurrect save script as a bare ExecStop
process (no $TMUX, so it depends on the default-socket server still being alive)
immediately before `ExecStop=tmux kill-server`, under KillMode=control-group.
On a shutdown/reboot/logout stop the server in the cgroup can be signalled while
save.sh is still dumping, so the save produces an empty file. With tmux-resurrect
promoting that empty dump to `last`, the previous good save is destroyed.

Run the save via `tmux run-shell` so it executes inside the live server (correct
socket, synchronous) and completes before kill-server; prefix both stop commands
with `-` so a stop when the server is already gone doesn't write an empty save.
Use KillMode=mixed and TimeoutStopSec so the save has room to finish during a
shutdown stop. Drop the dead RestartSec (no Restart= is set, and auto-restart is
intentionally avoided so a broken restore can't spin into a crash loop).

Note: the durable fix for the empty-save data loss belongs in tmux-resurrect's
save.sh (reject empty dumps before promoting to `last`); this change removes the
systemd path that triggers it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant