test(crawl): add a first-install acceptance test for the crawl command

acceptance/crawl/.gitignore

@@ -0,0 +1 @@
+out/

acceptance/crawl/README.md

@@ -0,0 +1,111 @@
+# Crawl first-install acceptance test: CLI
+
+This folder tests the CLI (`@zenrows/cli`) the way a new customer meets it. It builds the release
+artifact, installs it in a clean container with nothing else, follows the README's Crawl
+quickstart, and checks every Crawl feature against the live API. The result is one report.
+
+It tests this repo only, so it runs on its own.
+
+## What it needs
+
+- bash, git and Docker. Nothing else on the host.
+- Network access to GitHub, the package registries and the Crawl API.
+- A Zenrows API key with Crawl access, in `ZENROWS_API_KEY`.
+
+## How to run it
+
+From the repo root:
+
+```bash
+export ZENROWS_API_KEY="$(cat path/to/key-file)"   # keep it out of your shell history
+bash acceptance/crawl/run.sh              # build, run every check, write the report
+bash acceptance/crawl/run.sh --no-key     # only the checks that need no key; the live checks are SKIP
+bash acceptance/crawl/run.sh --report     # rebuild the report from out/
+REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub instead of this checkout
+```
+
+Without `ZENROWS_API_KEY`, `run.sh` stops and asks for it. The report is
+`acceptance/crawl/out/crawl-first-install-report.md`; `out/` is not committed.
+
+Every input is an environment variable with a default in `versions.env`: `CRAWL_URL`
+(default `https://www.scrapingcourse.com/ecommerce/`), `CRAWL_INCLUDE` (default `/product/`),
+`REF`, `CRAWL_API_BASE`, the images (`node:20`, `python:3.12-slim`, `golang:1.25`, pinned by
+digest) and the tool versions. `OUT_DIR` moves the output.
+
+## What a run does
+
+1. **Source.** By default, this checkout, including changes not yet committed. With `REF`, the
+   repo at that ref from GitHub.
+2. **Build.** In its own container, the release artifact: `npm install -g` of the `npm pack` tarball.
+3. **Clean test container.** It gets only the artifact (read-only), the checks in `surface/`
+   (read-only), the inputs, and the key by name (`-e ZENROWS_API_KEY`). It has no source, no
+   caches from the build, and no tokens.
+4. **README first.** It runs the README's Crawl quickstart as written. The only changes are the
+   ones the cost rules need: the key, `CRAWL_URL`, `CRAWL_INCLUDE`, `max_pages` 10 or less, and
+   no html. The report lists each change.
+5. **Checks 1 to 21 and the extras.** Each writes one JSON result:
+   `{surface, check, title, status, expected, actual, evidence}`, with `status` PASS, FAIL or
+   SKIP. The evidence is the command and its trimmed output, with the key redacted.
+6. **Clean-up.** It stops every crawl the run started and reads each one's final status and
+   `coverage.pages_fetched`. It runs even when a check fails, or on Ctrl-C.
+7. **Report.** Verdict, environment, install steps, feature matrix, findings, cost and
+   clean-up. Last, the run checks that the key appears in no file under `out/`.
+
+Some checks don't use the live API, so they behave the same on every run. They point the
+client at a small local stand-in server that records what it sends. These are: "only the
+fields you set are sent", "the error code is empty when the body has none", and the timing
+and signal checks.
+
+## The checks
+
+| # | Feature | Expected |
+| --- | --- | --- |
+| 1 | Create | returns at once: `running`, a `crawl_id`, the echoed parameters; only the fields set are sent |
+| 2 | Create with html | results get `content_status` and, once fetched, `content_url` |
+| 3 | Idempotency | same key and body → same `crawl_id`; different body → 422 `idempotency_key_reused` (MCP, CLI: `CRAWL_INVALID_REQUEST` with the advice to send a new key) |
+| 4 | Wait | returns the ended crawl; `failed` is returned, not raised (CLI: exit 1) |
+| 5 | Wait runs out | a 1 s timeout on a running crawl returns it `running`, no error (CLI: exit 0; MCP, CLI: a `note`) |
+| 6 | Get | `limit` 2, then the cursor: one page each; `next_cursor` moves on and is null on the last page |
+| 7 | List | `limit` 2, then the next page: newest first; `next_cursor` absent or null on the last page |
+| 8 | Results | every kept URL once, each matching `CRAWL_INCLUDE`; on a running crawl, what is kept so far (MCP, CLI: `partial: true`) |
+| 9 | Content | by content id and by `content_url`: the same HTML |
+| 10 | Download | a status, and one line per result with `url`, `content_status`, `content` |
+| 11 | Stop | running → `stopped`, `stop_reason: user`; ended → answers as it ended |
+| 12 | Not found | 404 `crawl_not_found` (MCP, CLI: `CRAWL_NOT_FOUND` with `crawl_id` and `server_code`) |
+| 13 | Invalid input | `depth` 0 → 422 `invalid_parameter` (MCP, CLI: `CRAWL_INVALID_REQUEST`) |
+| 14 | Gated option | `output_format` json → 422 `invalid_parameter`; if it is accepted, the key is in a beta segment, not a bug |
+| 15 | Bad key | the key `invalid` → 401 (MCP, CLI: `AUTH_INVALID`) |
+| 16 | Errors carry the server code | SDKs: `code` holds it, empty when the body has none; MCP, CLI: `server_code` |
+| 17 | Naming | the API's operation and parameter names, in this surface's casing |
+| 18 | Copy | Crawl is "new", never "beta"; one "still evolving" line; the 429 limit reads as the account's active jobs, shared with Batch |
+| 19 | Failed crawl | an include pattern that matches nothing ends `failed` with `no_items_found`; SDKs return it, not raise it; MCP and CLI say `ok: false` (CLI: exit 1, `CRAWL_FAILED`) |
+| 20 | Results after stop | the html crawl is stopped while its pages are still being fetched: reading its results ends at once (MCP and CLI: `partial: false`, no note to call again), and the same URLs come back once the pages in flight finish |
+| 21 | Documented defaults | a create without `max_pages` and `max_items` gets the default the docs state (10) |
+
+Extra checks for this surface: every subcommand prints JSON with `--json` (`create`, `get`, `wait` and `stop` print `{ok, crawl_id, status, crawl, note?}`), Ctrl-C during `crawl wait` prints the resume hint and exits 130, and `--wait` and `--html` are rejected as unknown flags.
+
+## What it costs
+
+Every crawl uses `depth` 1 and `max_pages` 10 or less. A run starts five crawls: the README run,
+one URLs-only crawl, one idempotency crawl that is stopped at once, one html crawl that is
+stopped with its pages in flight (check 20), and one that keeps nothing (check 19). On the default `CRAWL_URL`, a URLs-only crawl fetches 1 page and the
+html crawl 6, so a run costs about **10 pages**, and at most 50 if every crawl reaches its
+`max_pages`. Check 9 starts a sixth crawl, one html item, when the stop in check 20 lands
+before any page is fetched. It adds at most 10 pages. The report lists each crawl id and its `pages_fetched`.
+
+An account can run 3 active jobs at once, shared with Batch jobs. A run has at most 2 crawls
+running at a time. If a create still gets a 429, the run waits `Retry-After` and tries again.
+It never sends requests to trigger the limit. Don't run two copies against one account at the
+same time.
+
+## How to read the report
+
+- **Verdict:** ready, unless there is a blocker or major finding.
+- **Feature matrix:** PASS, FAIL or SKIP, with a short reason. SKIP means the check does not
+  apply to this surface, the client rejects the input before the API, a crawl ended before
+  the check could act on it, or there was no key.
+- **Findings:** one entry per FAIL, most severe first, with the command and output. An install
+  or README failure is a blocker; a wrong answer from a feature is major; naming, copy, the
+  defaults and the extras are minor.
+- `out/results/checks/*.json` holds every result; `out/results/container.log` is the full
+  output.

acceptance/crawl/builders/build-npm.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Runs in the Node image. Builds the npm tarball a release would publish: the repo's own
+# install and build, then npm pack. Only artifact/*.tgz leaves this container.
+set -euo pipefail
+cp -a /out/source /build && cd /build
+if [ -f pnpm-lock.yaml ]; then
+  npm install -g --no-audit --no-fund "pnpm@$PNPM_VERSION" >/dev/null
+  pnpm install --frozen-lockfile --store-dir /tmp/pnpm-store
+  pnpm run build
+else
+  npm ci --no-audit --no-fund
+  npm run build
+fi
+rm -rf /out/artifact && mkdir -p /out/artifact
+npm pack --pack-destination /out/artifact 2>&1 | tail -n 60
+ls -l /out/artifact

acceptance/crawl/builders/fetch-source.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Runs in the Node image when REF is set: downloads ZenRows/$REPO at $REF as GitHub's source
+# tarball and reads the full commit SHA from it.
+set -euo pipefail
+cd /out
+curl -fsSL --retry 4 -o src.tar.gz "https://codeload.github.com/ZenRows/$REPO/tar.gz/$REF"
+# git get-tar-commit-id stops reading after the header, so the pipe breaks: ignore that.
+(gunzip -c src.tar.gz | git get-tar-commit-id > sha) || true
+[ -s sha ] || { echo "no commit id in the tarball of $REPO@$REF" >&2; exit 1; }
+rm -rf source && mkdir -p source && tar -xzf src.tar.gz -C source --strip-components 1 && rm src.tar.gz
+echo "fetched $REPO@$REF = $(cat sha)"

acceptance/crawl/run.sh

@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# Crawl first-install acceptance test for this repo. See README.md in this folder.
+#
+#   ZENROWS_API_KEY=... bash run.sh    build the release artifact, run every check, write the report
+#   bash run.sh --no-key                only the checks that need no key; the live checks are SKIP
+#   bash run.sh --report                rebuild the report from the results already in out/
+#
+# Needs bash, git and Docker. The key reaches containers by name only (-e ZENROWS_API_KEY).
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=versions.env
+source "$HERE/versions.env"
+OUT_DIR="${OUT_DIR:-$HERE/out}"
+
+no_key=0; report_only=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-key) no_key=1 ;;
+    --report) report_only=1 ;;
+    -h|--help) sed -n '2,8p' "$0"; exit 0 ;;
+    *) echo "unknown argument: $arg (expected --no-key or --report)" >&2; exit 2 ;;
+  esac
+done
+
+if [ "$report_only" = 0 ] && [ "$no_key" = 0 ] && [ -z "${ZENROWS_API_KEY:-}" ]; then
+  echo "ZENROWS_API_KEY is not set. Set it (a key with Crawl access), or run 'bash run.sh --no-key' for the offline checks." >&2
+  exit 2
+fi
+if [ "$no_key" = 1 ]; then unset ZENROWS_API_KEY; fi
+export ZENROWS_API_KEY="${ZENROWS_API_KEY:-}" CRAWL_URL CRAWL_INCLUDE CRAWL_API_BASE
+
+log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; }
+RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
+CMDLINE="bash run.sh $*"
+# Unique per run, so clean-up never touches the containers of another run.
+LABEL="crawl-acceptance.run=$RUN_ID-$$-$RANDOM"
+docker_run() { docker run --rm --label "$LABEL" "$@"; }
+
+# ---- 1. Source: this checkout (default), or the repo on GitHub at $REF ----
+prepare_source() {
+  rm -rf "$OUT_DIR/build"; mkdir -p "$OUT_DIR/build/source"
+  if [ -n "${REF:-}" ]; then
+    log "source: ZenRows/$REPO@$REF from GitHub"
+    docker_run -v "$OUT_DIR/build:/out" -v "$HERE/builders:/builders:ro" \
+      -e REPO="$REPO" -e REF="$REF" "$NODE_IMAGE" bash /builders/fetch-source.sh >"$OUT_DIR/build/fetch.log" 2>&1
+    echo "github:$REF" >"$OUT_DIR/build/origin"
+  else
+    local root; root="$(git -C "$HERE" rev-parse --show-toplevel)"
+    log "source: this checkout ($root)"
+    # Tracked and untracked-but-not-ignored files, as a release build would see them.
+    (cd "$root" && git ls-files -co --exclude-standard -z | tar --null -T - -cf -) | tar -C "$OUT_DIR/build/source" -xf -
+    local sha; sha="$(git -C "$root" rev-parse HEAD)"
+    if [ -n "$(git -C "$root" status --porcelain -- . ":(exclude)$(realpath --relative-to="$root" "$HERE")")" ]; then
+      sha="$sha+local-changes"
+    fi
+    echo "$sha" >"$OUT_DIR/build/sha"
+    echo "checkout" >"$OUT_DIR/build/origin"
+  fi
+  # The test never reads the harness from the source; only the artifact goes in.
+  rm -rf "$OUT_DIR/build/source/acceptance"
+}
+
+# ---- 2. Builder: the release artifact, in its own container ----
+build_artifact() {
+  mkdir -p "$OUT_DIR/build/artifact"
+  case "$SURFACE" in
+    node|mcp|cli)
+      log "build: npm pack"
+      docker_run -v "$OUT_DIR/build:/out" -v "$HERE/builders:/builders:ro" \
+        -e SURFACE -e PNPM_VERSION "$NODE_IMAGE" bash /builders/build-npm.sh >"$OUT_DIR/build/build.log" 2>&1 ;;
+    python)
+      log "build: the wheel"
+      docker_run -v "$OUT_DIR/build:/out" -v "$HERE/builders:/builders:ro" \
+        -e PY_BUILD_VERSION "$PYTHON_IMAGE" bash /builders/build-python.sh >"$OUT_DIR/build/build.log" 2>&1 ;;
+    go)
+      # Go has no artifact: the clean container runs `go get` at a pushed commit, as a user would.
+      local sha; sha="$(cat "$OUT_DIR/build/sha")"
+      case "$sha" in *+local-changes)
+        log "go: local changes are not tested; go get installs the commit ${sha%+local-changes} from GitHub" ;;
+      esac
+      echo "${sha%+local-changes}" >"$OUT_DIR/build/go_ref" ;;
+  esac
+}
+
+# ---- 3. Clean test container: only the artifact, the checks and the inputs go in ----
+run_checks() {
+  rm -rf "$OUT_DIR/results"; mkdir -p "$OUT_DIR/results"
+  local image
+  case "$SURFACE" in node|mcp|cli) image="$NODE_IMAGE" ;; python) image="$PYTHON_IMAGE" ;; go) image="$GO_IMAGE" ;; esac
+  log "checks: a clean $image"
+  local rc=0
+  docker_run --init \
+    -v "$OUT_DIR/build/artifact:/artifact:ro" \
+    -v "$HERE/surface:/harness:ro" \
+    -v "$OUT_DIR/results:/results" \
+    -e ZENROWS_API_KEY -e CRAWL_URL -e CRAWL_INCLUDE -e CRAWL_API_BASE \
+    -e SURFACE -e GO_REF="$(cat "$OUT_DIR/build/go_ref" 2>/dev/null || true)" \
+    -e MCP_SDK_VERSION -e TYPESCRIPT_VERSION -e MYPY_VERSION \
+    -w /work --tmpfs /work:exec,size=2g \
+    "$image" bash /harness/run-in-container.sh >"$OUT_DIR/results/container.log" 2>&1 || rc=$?
+  log "checks: the container exited $rc (out/results/container.log)"
+}
+
+# ---- 4. Clean-up: stop every crawl this run started. Runs even when a check fails. ----
+cleanup() {
+  [ "$report_only" = 1 ] && return 0
+  docker ps -q --filter "label=$LABEL" | xargs -r docker kill >/dev/null 2>&1 || true
+  if [ -n "${ZENROWS_API_KEY:-}" ] && [ -d "$OUT_DIR/results" ]; then
+    log "clean-up: stopping every crawl this run started"
+    docker_run -v "$OUT_DIR:/out" -v "$HERE/tools:/tools:ro" \
+      -e ZENROWS_API_KEY -e CRAWL_API_BASE "$PYTHON_IMAGE" python -I /tools/cleanup.py /out || log "clean-up failed"
+  fi
+}
+
+report() {
+  log "report: out/crawl-first-install-report.md"
+  docker_run -v "$OUT_DIR:/out" -v "$HERE/tools:/tools:ro" \
+    -e CMDLINE="$CMDLINE" -e SURFACE -e REPO -e CRAWL_URL -e CRAWL_INCLUDE -e CRAWL_API_BASE \
+    -e NODE_IMAGE -e PYTHON_IMAGE -e GO_IMAGE -e MCP_SDK_VERSION -e TYPESCRIPT_VERSION \
+    -e MYPY_VERSION -e PY_BUILD_VERSION -e PNPM_VERSION \
+    "$PYTHON_IMAGE" python -I /tools/report.py /out
+  # The key must appear in nothing the run wrote. Checked by value, inside a container.
+  if [ -n "${ZENROWS_API_KEY:-}" ]; then
+    docker_run -v "$OUT_DIR:/out" -v "$HERE/tools:/tools:ro" -e ZENROWS_API_KEY "$PYTHON_IMAGE" \
+      python -I /tools/scan_key.py /out
+  fi
+}
+
+export SURFACE REPO NODE_IMAGE PYTHON_IMAGE GO_IMAGE PNPM_VERSION PY_BUILD_VERSION \
+  MCP_SDK_VERSION TYPESCRIPT_VERSION MYPY_VERSION
+if [ "$report_only" = 1 ]; then report; exit 0; fi
+
+mkdir -p "$OUT_DIR"
+echo "$RUN_ID" >"$OUT_DIR/run_id"
+trap cleanup EXIT
+trap 'exit 130' INT TERM
+if prepare_source && build_artifact; then
+  run_checks
+else
+  log "the source or build step failed (out/build/*.log)"
+  mkdir -p "$OUT_DIR/results"
+  docker_run -v "$OUT_DIR:/out" -v "$HERE/tools:/tools:ro" -e SURFACE "$PYTHON_IMAGE" python -I /tools/build_failed.py /out
+fi
+cleanup
+trap - EXIT
+report
+log "done: $OUT_DIR/crawl-first-install-report.md"