Skip to content

feat: Add OpenClaw host snapshot and restore toolkit - #69

Merged
rubambiza merged 16 commits into
rossoctl:mainfrom
rubambiza:worktree-openclaw-snapshot-spec
Sep 9, 2026
Merged

feat: Add OpenClaw host snapshot and restore toolkit#69
rubambiza merged 16 commits into
rossoctl:mainfrom
rubambiza:worktree-openclaw-snapshot-spec

Conversation

@rubambiza

Copy link
Copy Markdown
Contributor

Summary

Our OpenClaw automation host runs the cron jobs and agents that drive this repo's automation. If that VM has to be rebuilt or moved, we currently have no clean way to bring it back — you'd be reconstructing config, credentials, secrets, and repo checkouts by hand. This adds a way to capture that host into a small portable snapshot and stand it back up on a fresh VM so the automation keeps working.

What

snapshot/, a bash toolkit. A snapshot is not a disk image — it's a few small artifacts in one dated directory:

  • state — the host's OpenClaw config, credentials, sessions, and workspaces, via OpenClaw's own backup create --verify.
  • secrets — host secret files encrypted into a single secrets.age. Encryption is asymmetric (age), so the private key never lives on the VM and plaintext is never written to disk.
  • manifest + RUNBOOK — pinned versions, the service unit, and each core repo's real git state (origin, branch, whether it had uncommitted changes), plus a generated step-by-step restore runbook.

Restore is driven by restore.sh --dry-run, which prints the ordered restore plan without touching the host. That dry-run is the safe, reviewable unit; the actual restore is meant to be run by a human following the generated RUNBOOK.

One design choice worth calling out: each repo is re-cloned from the origin recorded at capture time, not from a single hardcoded org. On the real host the core repos span two different GitHub owners, so a one-org assumption would clone half of them from the wrong place. Recording the real origin also keeps this decoupled from the ongoing repo-ownership rework (#62 / #68).

How

Six focused scripts (a shared lib, three capture steps, a top-level driver, a restore planner). Portable to bash 3.2 (macOS default). Secrets are handled names-only — the tooling never prints or writes secret contents.

Tests

Six hermetic bash tests: fake binaries on PATH, environment seams for every external dependency, no network and no real host state touched. They cover the encrypt-only secrets path (asserting no plaintext leaks), the manifest/runbook generation, the backup wrapper, the capture driver, and the restore dry-run plan. Wired into CI (.github/workflows/tests.yml) under both shellcheck and the test suite.

Beyond the unit tests, the toolkit was run end-to-end against the live automation host: the state archive verified, the encrypted secrets blob showed no plaintext, all core repos were recorded at their real origins/branches, and the restore dry-run produced a clean, mutation-free plan.

Trying it (optional)

# One-time, on your machine only (never on a VM):
age-keygen -o ~/.openclaw-snapshot.key    # prints the public key age1...

# Capture on the host (needs the age binary at snapshot/bin/age — see snapshot/README.md):
bash snapshot/snapshot.sh --output ~/snapshots --pubkey age1<PUBLIC>

# Review the restore plan — makes no changes:
bash snapshot/restore.sh --from ~/snapshots/openclaw-snapshot-<DATE> --dry-run

snapshot/README.md has the full operator guide (provisioning age, leak checks, verifying the archive).


Assisted-By: Claude Code

Design for snapshotting the OpenClaw automation host for easy redeploy
to a new VM. Approach: clean rebuild + state via openclaw backup create,
separate age-encrypted secrets bundle (keypair model), and core-repo-only
re-clone driven by config/core-repos.txt + the org profile.

Ref epic rossoctl#32 org-portability work.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The driver invoked its sibling capture scripts directly ("$CMD"),
which fails with "Permission denied" when the scripts are mode 0644 --
as they are after a fresh git clone or a tarball repacked without
preserving the execute bit. This bit a real live-host capture.

Invoke the capture scripts through 'bash "$CMD"' so file mode no
longer matters, and also set the git execute bit on the invoked
scripts (intent). A new driver-test assertion runs a capture with
non-executable capture scripts and requires it to succeed, so the
regression cannot return silently.

Also pin the verified SHA-256 for the age v1.3.2 linux-amd64 release
in the operator guide.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The generated RUNBOOK said 'Extract state.tar.gz', a placeholder that
never matches the real artifact -- the state capture names the archive
after the backup's own archivePath (a dated *-openclaw-backup.tar.gz).
restore.sh already globs for that name, so the tooling was correct;
only the human-facing runbook prose was misleading. Describe the actual
naming pattern instead.

The manifest generator runs independently of the state capture and does
not know the exact archive filename, so the runbook names the pattern
rather than a specific file.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
@rubambiza rubambiza added the ready-for-ai-review Request automated AI code review from clawgenti label Sep 1, 2026
@rubambiza rubambiza self-assigned this Sep 1, 2026

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is failing on both macOS and Ubuntu with the same root cause in tests/test-snapshot-manifest.sh. The tool-two fixture is created with a bare git init — on GitHub Actions runners init.defaultBranch is master, so rev-parse --abbrev-ref HEAD returns master rather than main, and line 183's grep -Fq 'checkout \main`'` assertion fails.

All other checks pass (DCO on all 11 commits ✓, all scripts have set -euo pipefail ✓, shellcheck wired to CI ✓, no .claude/.vscode changes ✓, no hardcoded secrets ✓, hermetic test seams are well-structured ✓).


Reviewed by clawgenti using the github-pr-review skill

Comment thread tests/test-snapshot-manifest.sh
The tool-two fixture used a bare 'git init', so its branch name came
from the host's init.defaultBranch. On GitHub Actions runners that is
not 'main', so the manifest recorded a different branch and the
'checkout main' assertion failed on CI while passing on a dev box.
Pin the branch explicitly with 'git checkout -b main' so the test is
deterministic regardless of the runner's git configuration.

Fixes the CI failure flagged in review on rossoctl#69.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix directly addresses the CI failure flagged in the previous review: pinning git checkout -b main in the tool-two fixture makes the manifest branch deterministic across runners with different init.defaultBranch defaults. All CI checks now pass (DCO ✓, Ubuntu ✓, macOS ✓, PR title verifier ✓).

All checks pass. Ready for human review.


Reviewed by clawgenti using the github-pr-review skill

Documentation and comments only; no logic changes. Aimed at making the
toolkit easy for a new maintainer to pick up:

- README gains a Maintainer reference: a table of every environment
  seam (variable, default, purpose, consumer) and a Development section
  with the exact test + shellcheck commands CI runs.
- Flag the secret allowlist in lib-snapshot.sh as the one list to keep
  current when a new host secret file appears.
- Note why the manifest generator re-encodes the JSON array each
  iteration (jq owns escaping) so it is not 'optimized' into a bug.
- Warn in tests.yml that the test/shellcheck lists are explicit, not
  globs, so a new test must be added there or it never runs.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snapshot and restore toolkit is well-structured: hermetic tests with seams for every external dependency, names-only secrets handling, and the recorded-origin design for #62 is the right call. All CI checks pass, all 13 commits are signed off.

One security finding worth addressing before merge, plus a documentation nit.


Reviewed by clawgenti using the github-pr-review skill

Comment thread snapshot/restore.sh
Comment thread snapshot/lib-snapshot.sh Outdated
restore.sh built each restore step as a single string and ran it with
eval. A manifest field -- e.g. a repo origin, originally captured from a
git remote URL -- containing shell metacharacters could inject commands
during a real restore (a tampered or adversarial manifest).

Replace the eval-based emit helper with one that takes the command and
its arguments as separate words and executes them via "$@", so the
shell never re-parses manifest-derived data. Dry-run printing uses
printf '%q' for safe, copy-paste-faithful output. The one inherently
piped step (decrypt | untar) moves to emit_pipeline, which runs a fixed
command string containing only trusted literals, never manifest data.

Add a regression test: a manifest origin carrying a 'touch canary'
payload run through a real (stubbed) restore must leave the canary
absent. Verified it fails against the old eval form and passes now.

Addresses the security finding in review on rossoctl#69.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Each candidate secret file now carries a short inline comment naming
what it is (OpenClaw env, gateway systemd env, npm auth, stored GitHub
PAT, SSH private key), so a maintainer can tell what each entry is and
whether it is expected on a given host.

Addresses the documentation nit in review on rossoctl#69.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both findings from the previous review are resolved: the eval-based injection path in restore.sh is gone (replaced by the emit/emit_pipeline approach), and the secret-allowlist entries are now annotated. CI passes green on all four checks. All 15 commits are signed off.

All checks pass. Ready for human review.


Reviewed by clawgenti using the github-pr-review skill

@rubambiza rubambiza added ready-for-human-review AI review passed, ready for human reviewer and removed ready-for-ai-review Request automated AI code review from clawgenti labels Sep 2, 2026

@mrsabath mrsabath left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full toolkit — all six scripts, the six hermetic test suites, the design spec, and the operator README — and verified the cross-repo contracts (load_org_profile, validate_repos_dir, core_repo_names, and the PROFILE_ORGORG mapping) against org.sh on main. This is careful, security-conscious work and I'm happy to approve it.

What stands out

  • Encrypt-only secrets model is sound. age -r <public-key> means the private key never lands on any VM, and tar | age streams straight to secrets.age with no plaintext touching disk. The test proves it: a distinctive marker seeded into every fixture secret, then asserted absent from both the blob and the names-only checklist, plus a find for any stray *.tar.
  • Restore is injection-safe by construction. emit runs its words via "$@" (no eval — I saw the 879cbae commit that removed it), so a manifest-derived origin or path is inert data, not code. There's a real canary test: a manifest origin carrying '; touch $CANARY; echo ' run through a non-dry-run restore with stubbed mutators, asserting the canary never appears. That's exactly the right test to write here.
  • jq owns all JSON escaping in manifest generation — the O(n²) re-encode is correctly called out as deliberate (tiny repo count) with a warning not to "optimize" it back into fragile shell concatenation.
  • The recorded-origin design (clone each repo from its captured origin, fall back to $ORG only when empty) is the right call given origins span kagenti/* and rossoctl/*, and it decouples cleanly from the #62/#68 ownership rework. Verified the fallback: org.sh populates ORG from PROFILE_ORG, so the test's acme/tool-two assertion is legitimate.
  • Tests are genuinely hermetic and assertive — fake binaries on PATH, env seams for every dependency, no network, and each suite checks the failure paths (missing binary, empty home, missing manifest) as well as the happy path. shellcheck-clean, bash 3.2 portable, CI green on macOS and Ubuntu. 15 well-scoped, signed-off commits.

One inline suggestion — non-blocking, the single seam in an otherwise airtight no-unescaped-interpolation discipline. Approving.

Areas reviewed: Bash (6 scripts + shared lib), tests (6 suites), security (secret handling, encryption, shell-injection), CI wiring, cross-repo contracts, commit/PR conventions.
Commits: 15, all signed off.
CI: DCO + test (macOS, Ubuntu) + pr-title all pass.

Comment thread snapshot/restore.sh Outdated
Close the one build-time-splice seam Mariusz flagged on rossoctl#69: Step 3
built its bash -c string with the snapshot path spliced in inside single
quotes ('$FROM/secrets.age'), so a --from path containing a single quote
could break out of the pipeline. Pass the path through the environment as
$SECRETS_BLOB and dereference it as "$SECRETS_BLOB" at run time, matching
how $AGE_IDENTITY and $HOME were already handled -- no caller data reaches
the command string at build time.

This also removes a latent set -u abort: the old form expanded
$AGE_IDENTITY when building the string, so a dry-run with the identity
unset died at that line; the deferred form leaves it to run time.

Add a regression test: a --from dir whose name carries a single-quote
canary payload, run through a real (non-dry-run) restore with stubbed
mutators, asserting the canary is never created.

Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
@rubambiza
rubambiza merged commit 76f3d94 into rossoctl:main Sep 9, 2026
4 checks passed
@rubambiza
rubambiza deleted the worktree-openclaw-snapshot-spec branch September 9, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human-review AI review passed, ready for human reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants