Tup refactor

.agents/codebase-insights.txt

@@ -31,3 +31,11 @@
 - Third-party frontend assets that were previously symlinks into `node_modules` (`@exuanbo`, `golden-layout/dist`, `monaco-editor/min`, `mousetrap`, `vex-js`, `xterm`) are now materialized from `node-modules-derivation` inside `appimagePayload`, preventing broken symlinks when the payload is copied out of the Nix store.
 - Ruby 3.3 comes from `appimageDeps` (via `pkgs.ruby_3_3`) and is patched inside the derivation, so the AppDir no longer copies Ruby from the devshell during the AppImage build.
 - `nix/packages/default.nix` now hides helper bindings like `mkNimBinary` via `builtins.removeAttrs` so `perSystem.packages` exposes only real derivations; flake checks will reject non-package values there.
+- The generated `appimagePayload` launcher (`bin/ct`) now executes `ct_unwrapped` from the same directory instead of `bin/ct_unwrapped`, fixing AppImage runs where `HERE` already points at `bin/`.
+- `ci/build/dev.sh` temporarily moves `src/links` out of the tree before `tup generate` because the generated script writes artifacts directly into `src/`, forcing a full `git clean -xfd src/` afterward.
+- The staged Tup layout now lives under `src/build/`; live `tup` runs and `just build` execute from there, and variant artifacts are produced in `src/build-debug/build/`.
+- Running a script produced by `tup generate` populates `src/build/`; clean the staging trees with `cd src/build && git clean -fx .` and `cd ../build-debug && git clean -fx .` before resuming live `tup`.
+- `!tup_preserve` only mirrors files relative to the directory containing the Tupfile; after moving Tupfiles into `src/build` we copy assets with the new `!cp_preserve` helper so files can originate from the original `src/` tree without confusing tup.
+- `!cp_preserve` replaces symlink sources with fresh links pointing at the original target (using `readlink -f`) so mirrored assets stay valid when their relative paths shift.
+- `ct_wrapper` resolves `ct_paths.json` by honouring `CODETRACER_CT_PATHS` and otherwise searching upward from both the binary location and the current working directory, so it no longer assumes the wrapper lives at a specific depth in the checkout.
+- The posix `wrapElectron` path (used when `CODETRACER_WRAP_ELECTRON=1`) now forwards `CODETRACER_ELECTRON_ARGS`, so CI overrides like `--no-sandbox` reach Electron even when we re-exec the wrapper. The Playwright launcher preserves any existing value and only appends `--no-sandbox` if it is missing, so CI scripts can layer additional flags (e.g. `--disable-gpu`).

.github/workflows/codetracer.yml

@@ -118,12 +118,6 @@ jobs:
           nix develop .#devShells.x86_64-linux.default --command aws --endpoint-url=${{ secrets.R2_CODETRACER_BUCKET_S3_ENDPOINT }} s3 cp CodeTracer.pub.asc s3://${{ vars.R2_CODETRACER_BUCKET_NAME }}/CodeTracer.pub.asc
   dev-build:
     runs-on: [self-hosted, nixos]
-    needs:
-      - lint-bash
-      - lint-nim
-      - lint-nix
-      - lint-rust
-      - lint-ui-tests
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -142,12 +136,6 @@ jobs:
 
   nix-build:
     runs-on: [self-hosted, nixos]
-    needs:
-      - lint-bash
-      - lint-nim
-      - lint-nix
-      - lint-rust
-      - lint-ui-tests
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -172,6 +160,11 @@ jobs:
       - lint-nix
       - lint-rust
       - lint-ui-tests
+      - test-rust
+      - test-python-recorder
+      - test-ui-tests
+      - dev-build
+      - nix-build
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -216,6 +209,11 @@ jobs:
       - lint-nix
       - lint-rust
       - lint-ui-tests
+      - test-rust
+      - test-python-recorder
+      - test-ui-tests
+      - dev-build
+      - nix-build
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -310,11 +308,6 @@ jobs:
 
   test-rust:
     runs-on: [self-hosted, nixos]
-    needs:
-      - dev-build
-      - nix-build
-      - appimage-build
-      - dmg-build
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -333,11 +326,6 @@ jobs:
 
   test-python-recorder:
     runs-on: [self-hosted, nixos]
-    needs:
-      - dev-build
-      - nix-build
-      - appimage-build
-      - dmg-build
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -356,11 +344,6 @@ jobs:
 
   test-ui-tests:
     runs-on: [self-hosted, nixos]
-    needs:
-      - dev-build
-      - nix-build
-      - appimage-build
-      - dmg-build
     steps:
       - name: Checkout
         uses: actions/checkout@v5
@@ -380,9 +363,6 @@ jobs:
   push-to-cachix:
     runs-on: [self-hosted, nixos]
     needs:
-      - test-rust
-      - test-python-recorder
-      - test-ui-tests
       - appimage-lib-check
       - dmg-lib-check
     if: "github.ref == 'refs/heads/main' && ${{ !github.event.codetracer-ci }}"

0006-tup-build-staging-area.md

@@ -0,0 +1,57 @@
+# ADR 0006: Stage Tup Builds Under `src/build`
+
+- **Status:** Proposed
+- **Date:** 2025-10-21
+- **Deciders:** Codetracer Build & Tooling Maintainers
+- **Consulted:** Desktop Packaging, CI & Release Engineering, Developer Experience
+- **Informed:** Runtime Leads, Support, Product Management
+
+## Context
+
+The `tup` build root currently lives directly under `src/`, with `Tupfile`, `Tuprules.tup`, and a constellation of subordinate `Tupfile`s spread across the source tree. This layout works when `tup` runs in FUSE mode, but `tup generate` emits shell scripts whose commands write artifacts (executables in `bin/`, JS bundles, copied resources, etc.) straight into the source tree. CI (`ci/build/dev.sh`) therefore jumps through hoops—temporarily moving `src/links`, running `git clean -xfd src/`, and manually restoring symlinks—to keep the repository tidy. Developers face the same issue when they need the generated script for environments where FUSE is unavailable.
+
+We tried `tup generate --builddir …` to sandbox the outputs, but the option is too buggy for our workload (incorrect relative paths and missing variant awareness). As a result, the source directory accumulates transient build files, complicating `git status`, forcing frequent cleans, and making it risky to script `tup.sh` execution inside packaging or CI jobs.
+
+## Decision
+
+We will move the entire Tup configuration into a dedicated staging area at `src/build/`, so that both `tup` FUSE runs and generated scripts create or modify files only inside that subtree while still producing the expected variant outputs (`build-debug/`, etc.).
+
+1. **Establish `src/build/` as the staging tree:** Relocate the root `Tupfile` and every subordinate `Tupfile` under a mirrored directory structure rooted at `src/build/`, while keeping `Tupfile.ini` anchored in `src/` so the Tup root remains the project root. The staged layout becomes the canonical entry point—invocations now run from `src/build/` without requiring a compatibility shim in `src/Tupfile`. Each rule references sources in `../` (or higher) as needed, but outputs and intermediate files stay under `src/build/` by default.
+2. **Stable path conventions via `SRC_DIR`:** Each staged `Tupfile` declares its own source root with a local `SRC_DIR` variable. This per-file approach is sufficient for the refactor and avoids introducing additional global macros.
+3. **Git hygiene:** Add a scoped `.gitignore` inside `src/build/` that admits `tup.sh`, generated metadata (`*.tup` state, temporary outputs), and any new staging directories while keeping declarative build files tracked.
+4. **Tooling alignment:** Update `ci/build/dev.sh`, `justfile`, and shell/Nix helpers to invoke `tup` from `src/build/`. The CI clean step focuses on `src/build/` and `src/build-debug/build/`, and hacks that relocate `src/links` become unnecessary.
+5. **Documentation & onboarding:** Refresh contributor docs to explain the new layout, clarifying that FUSE-based workflows (`tup monitor`) operate inside `src/build/`, while `tup build-debug` and generated scripts populate `src/build-debug/build/`. Highlight that `tup generate` writes into the staging tree and document the clean-up sequence required before running the live `tup` monitor again (`cd src/build && git clean -fx .`, followed by `cd ../build-debug && git clean -fx .`).
+
+## Alternatives Considered
+
+- **Continue using `tup generate` in `src/`:** Rejected; it keeps polluting the source tree and forces manual hygiene steps that are easy to forget.
+- **Rely on `tup generate --builddir`:** Rejected because upstream bugs break our rules (incorrect include paths and missing variant-specific outputs).
+- **Wrap `tup.sh` in custom sandboxing scripts:** Adds maintenance overhead without addressing the root problem that the Tup metadata lives inside the source tree.
+
+## Consequences
+
+- **Positive:** Generated scripts run cleanly without polluting tracked files; CI simplifications; consistent developer experience across environments lacking FUSE; easier to inspect or purge build artifacts via `git clean src/build` and `git clean src/build-debug/build`.
+- **Neutral/Enabling:** Establishes clearer separation between declarative build metadata and source assets, paving the way for additional variants or build caching under `src/build/`.
+- **Negative:** Requires refactoring every `Tupfile` path to account for the new root; risk of path mistakes during migration; temporary churn for developers with pending Tup changes. Running a generated script will populate `src/build/`; developers must clean both `src/build/` and `src/build-debug/` before resuming normal `tup` usage.
+- **Risks & Mitigations:** Misconfigured outputs could still escape the sandbox—mitigate by adding integration tests that run `tup generate` + `tup.sh` and assert the set of touched paths. Variant parity must be verified by running `tup build-debug` before and after the move within CI. Document the clean-up sequence after generated runs to avoid confusing residual artifacts.
+
+## Key Locations
+
+- `src/Tupfile`, `src/Tuprules.tup`, and all `src/**/Tupfile` instances that define the current build graph.
+- `ci/build/dev.sh`, `justfile`, and `nix/shells/*.nix` scripts that `cd src` before invoking `tup`.
+- `src/build-debug/tup.config` and any logic that depends on variant output directories.
+- Documentation under `docs/` and contributor guides describing the build workflow.
+
+## Implementation Notes
+
+1. Mirror the existing Tup hierarchy under `src/build/`, keeping directory-by-directory `Tupfile`s collocated with their rules while adjusting relative paths for inputs and outputs; retain `src/Tupfile.ini` at the root and run `tup` directly from `src/build/`.
+2. Keep per-directory `SRC_DIR` variables to reference the original sources without introducing additional macros.
+3. Create a `.gitignore` within `src/build/` that ignores generated scripts, logs, and `tup` state directories but keeps the committed `Tupfile`s tracked.
+4. Update tooling (`ci`, `just`, Nix shells) to execute `tup` from `src/build/`, ensuring variant commands still find `build-debug/tup.config` and write outputs to `src/build-debug/build/`.
+5. Add CI coverage that runs both `tup generate` + generated script and `tup build-debug` to confirm artifacts remain confined to `src/build/` and `src/build-debug/build/`, and document the clean-up requirement after generated runs.
+
+## Status & Next Steps
+
+- Draft ADR for review (this document).
+- Prototype the directory move on a feature branch to validate the staged layout, confirm both `tup` and `tup generate` succeed, and measure the clean-up workflow required after generated runs.
+- Once validated, mark this ADR **Accepted**, land the staging refactor, update the documentation (including the post-`tup.sh` cleaning steps), and complete the implementation plan alongside automated regression coverage.

ci/build/dev.sh

@@ -15,11 +15,8 @@ echo '##########################################################################
 # stop processes: make sure none of those processes left from last build
 stop_processes
 
-git clean -xfd src/build-debug
-
-mv src/links links
-git clean -xfd src/
-mv links src/links
+git clean -fdx src/build
+git clean -fdx src/build-debug/build
 
 echo '###############################################################################'
 echo "Build:"
@@ -34,10 +31,3 @@ TUP_OUTPUT_SCRIPT=tup-generated-build-once.sh
 tup generate --config build-debug/tup.config "$TUP_OUTPUT_SCRIPT"
 ./"$TUP_OUTPUT_SCRIPT"
 rm "$TUP_OUTPUT_SCRIPT"
-
-# TODO: this is not really working, problems with variants: generated script produce
-#   files directly in src/, instead of in src/build-debug, and so it can't run well
-#   we need to see if we can generate it in a better way, or to wrap/restructure the resulting folders
-#   to make possible to test the dev build in CI
-
-popd

ci/test/python-recorder-smoke.sh

@@ -11,7 +11,7 @@ echo '##########################################################################
 
 nix develop .#devShells.x86_64-linux.default --command ./ci/build/dev.sh
 
-CT_BIN="${ROOT_DIR}/src/bin/ct"
+CT_BIN="${ROOT_DIR}/src/build/bin/ct"
 if [[ ! -x "${CT_BIN}" ]]; then
   echo "error: ${CT_BIN} not found after build"
   exit 1

ci/test/ui-tests.sh

@@ -6,9 +6,7 @@ echo '##########################################################################
 echo 'Running ui e2e playwright tests'
 echo '###############################################################################'
 
-# TODO: maybe pass the result from the build stage as artifact to this job?
-# TODO: tup generate seems problematic with variants: we need to fix/change the resulting dirs to work correctly
-# ./ci/build/dev.sh
+./ci/build/dev.sh
 
 # trying to make it work with the nix build, instead of the tup build:
 
@@ -25,10 +23,22 @@ echo '##########################################################################
 
 # ./ci/build/nix.sh
 
-# CODETRACER_E2E_CT_PATH="$(pwd)/result/bin/ct"
-# export CODETRACER_E2E_CT_PATH
+CODETRACER_E2E_CT_PATH="$(pwd)/src/build/bin/ct"
+LINKS_PATH_DIR="$(pwd)/src/build"
+NIX_CODETRACER_EXE_DIR="$(pwd)/src/build"
+CODETRACER_LINKS_PATH="$(pwd)/src/build"
+CODETRACER_ELECTRON_ARGS="--no-sandbox --disable-gpu"
 
-# pushd ui-tests
-# nix develop --command ./ci.sh
+export CODETRACER_E2E_CT_PATH
+export LINKS_PATH_DIR
+export NIX_CODETRACER_EXE_DIR
+export CODETRACER_LINKS_PATH
+export CODETRACER_ELECTRON_ARGS
 
-# popd
+pushd ui-tests
+nix develop --command ./ci.sh
+
+popd
+
+git clean -fx ./src/build
+git clean -fx ./src/build-debug

docs/a.md

@@ -8,7 +8,7 @@
 ### Local
 
 I sometimes do 
-`set -gx PATH /home/al/codetracer/src/build-debug $PATH`
+`set -gx PATH /home/al/codetracer/src/build-debug/build $PATH`
 to get many codetracer-related binaries in my path: `codetracer`, `tester` and maybe others
 
 # Fix preloading
@@ -83,4 +83,3 @@ to get many codetracer-related binaries in my path: `codetracer`, `tester` and m
 
 
 
-

docs/book/src/installation.md

@@ -62,9 +62,12 @@ Direnv should be set up in your shell, as shown [here](https://direnv.net/docs/h
    ```
 4. Run `nix develop`
 5. Run `direnv allow`
-6. To build codetracer simply run `just build`. The location of the resulting binary will be `./src/build-debug/bin/ct`
+6. To build codetracer simply run `just build`. The location of the resulting binary will be `./src/build-debug/build/bin/ct`
 7. Now every time you enter the `codetracer` directory your environment should be updated
 
+> [!NOTE]
+> When you need to use `tup generate`, remember that the generated script writes into the staging tree. After it finishes, run `cd src/build && git clean -fx .` followed by `cd ../build-debug && git clean -fx .` before invoking `tup` again.
+
 > [!TIP]
 > Users of Visual Studio Code might encounter issues when using `code .`. To fix them do the following:
 > 1. Run `direnv deny`

docs/tests.md

@@ -21,6 +21,6 @@ we can save the steps in a file by passing `--file`, instead of running those au
 Running:
 
 ```bash
-src/build-debug/tests/run/a_test <traceID>
-src/build-debug/tests/run/a_test examples/sum.nim
+src/build-debug/build/tests/run/a_test <traceID>
+src/build-debug/build/tests/run/a_test examples/sum.nim
 ```

justfile

@@ -17,7 +17,7 @@ build:
   cd src
   tup build-debug
   tup monitor -a
-  cd ../
+  cd ..
 
   # start webpack
   node_modules/.bin/webpack --watch --progress & # building frontend_bundle.js
@@ -32,7 +32,7 @@ build-once:
   # We have to make the dist directory here, because it's missing on a fresh check out
   # It will be created by the webpack command below, but we have an a chicken and egg
   # problem because the Tupfiles refer to it.
-  mkdir public/dist
+  mkdir -p src/public/dist
 
   cd src
   tup build-debug
@@ -45,6 +45,7 @@ build-once:
   # that tup will discover
   cd src
   tup build-debug
+  cd ..
 
 build-docs:
   #!/usr/bin/env bash
@@ -103,7 +104,7 @@ build-macos-app:
 build-app-image:
   ./appimage-scripts/build_appimage.sh
 
-tester := "src/build-debug/bin/tester"
+tester := "src/build-debug/build/bin/tester"
 
 test-ui headless="0":
   #!/usr/bin/env bash
@@ -197,7 +198,7 @@ tail pid_or_current_or_last kind process="default" instance_index="0":
   tail -f ${log_file_path}
 
 build-nix:
-  nix build --print-build-logs '.?submodules=1#codetracer' --show-trace --keep-failed
+  nix build --print-build-logs '.#codetracer' --show-trace --keep-failed
 
 cachix-push-nix-package:
   cachix push metacraft-labs-codetracer $(nix build --print-out-paths ".?submodules=1#codetracer")
@@ -215,7 +216,7 @@ pid pid_or_current_or_last:
   #!/usr/bin/env bash
   # argument can be either `current`, `last` or a pid number
   if [[ "{{pid_or_current_or_last}}" == "current" ]]; then \
-    echo $(ps aux | grep src/build-debug/codetracer | head -n 1 | awk '{print $2}') ; \
+    echo $(ps aux | grep src/build-debug/build | head -n 1 | awk '{print $2}') ; \
   elif [[ "{{pid_or_current_or_last}}" == "last" ]]; then \
     TTMP=$(just findtmp) ; \
     echo $(cat $TTMP/last-start-pid) ; \

nix/shells/armShell.nix

@@ -197,7 +197,7 @@ mkShell {
 
     # ====
 
-    export CODETRACER_LINKS_PATH=$PWD/src/build-debug/
+    export CODETRACER_LINKS_PATH=$PWD/src/build-debug/build
 
     echo "{\"PYTHONPATH\": \"$CT_PYTHONPATH\",\"LD_LIBRARY_PATH\":\"$CT_LD_LIBRARY_PATH\"}" > ct_paths.json
 
@@ -244,10 +244,10 @@ mkShell {
     rm -rf $ROOT_PATH/node_modules
     ln -s $NIX_NODE_PATH $ROOT_PATH/node_modules
 
-    export NIX_CODETRACER_EXE_DIR=$ROOT_PATH/src/build-debug/
-    export LINKS_PATH_DIR=$ROOT_PATH/src/build-debug/
+    export NIX_CODETRACER_EXE_DIR=$ROOT_PATH/src/build-debug/build
+    export LINKS_PATH_DIR=$ROOT_PATH/src/build-debug/build
     export CODETRACER_REPO_ROOT_PATH=$ROOT_PATH
-    export PATH=$PATH:$PWD/src/build-debug/bin
+    export PATH=$PWD/src/build/bin:$PWD/src/build-debug/build/bin:$PATH
     export PATH=$PATH:$ROOT_PATH/node_modules/.bin/
     export CODETRACER_DEV_TOOLS=1
     export CODETRACER_LOG_LEVEL=INFO

nix/shells/main.nix

@@ -220,7 +220,7 @@ mkShell {
 
     # ====
 
-    export CODETRACER_LINKS_PATH=$PWD/src/build-debug/
+    export CODETRACER_LINKS_PATH=$PWD/src/build-debug/build
 
     echo "{\"PYTHONPATH\": \"$CT_PYTHONPATH\",\"LD_LIBRARY_PATH\":\"$CT_LD_LIBRARY_PATH\"}" > ct_paths.json
 
@@ -276,10 +276,10 @@ mkShell {
     rm -rf $ROOT_PATH/node_modules
     ln -s $NIX_NODE_PATH $ROOT_PATH/node_modules
 
-    export NIX_CODETRACER_EXE_DIR=$ROOT_PATH/src/build-debug/
-    export LINKS_PATH_DIR=$ROOT_PATH/src/build-debug/
+    export NIX_CODETRACER_EXE_DIR=$ROOT_PATH/src/build-debug/build
+    export LINKS_PATH_DIR=$ROOT_PATH/src/build-debug/build
     export CODETRACER_REPO_ROOT_PATH=$ROOT_PATH
-    export PATH=$PWD/src/build-debug/bin:$PATH
+    export PATH=$PWD/src/build-debug/build/bin:$PATH
     export PATH=$ROOT_PATH/node_modules/.bin/:$PATH
     export CODETRACER_DEV_TOOLS=0
     export CODETRACER_LOG_LEVEL=INFO

non-nix-build/env.sh

@@ -49,7 +49,7 @@ if [ ! -f "$ROOT_DIR"/ct_paths.json ]; then
   echo "{\"PYTHONPATH\": \"\",\"LD_LIBRARY_PATH\":\"\"}" > "$ROOT_DIR"/ct_paths.json
 fi
 
-export PATH=$DEPS_DIR/nim/bin:$ROOT_DIR/node_modules/.bin:$BIN_DIR:$CARGO_HOME/bin:$ROOT_DIR/src/build-debug/bin:$PATH
+export PATH=$DEPS_DIR/nim/bin:$ROOT_DIR/node_modules/.bin:$BIN_DIR:$CARGO_HOME/bin:$ROOT_DIR/src/build/bin:$ROOT_DIR/src/build-debug/build/bin:$PATH
 
 if [ "$os" == "mac" ]; then
   brew install sqlite3 ruby node universal-ctags go capnp