[David-Wu1119]

What changed

• publish two redacted, fully verifiable RTX 2060 evidence bundles and generated analyses
• add a strict two-run comparator that rejects duplicate manifests and checks all 352 terminal outcomes
• disclose the valid-but-excluded first run and both aborted Steam/Apex contention attempts
• update public claims with the measured result and explicit capsule, engine, hardware, and sampling limits
• make CI reverify the checked-in GPU packet and reject generated-output drift

Measured result

• greedy batch variation: 4/80 exact-token divergences (5.000%)
• sampled batch variation: 69/80 (86.250%)
• registered context pinned: 0/80 in each mode
• clean rerun comparison: 352 outcomes, 0 status/output/error mismatches

The capsule_pinned_replay schema label is not presented as a validated serialized capsule. This is context pinning on one Transformers/RTX 2060 stack and does not validate CloudTune orchestration.

Verification

• ruff check src tests
• PYTHONPATH=src python3.12 -m compileall -q src tests
• make check PYTHON=python3.12 (43 tests)
• staged secret and host-identifier scan
• both public evidence bundles independently verified after redaction
• non-redacted payload files byte-identical to the private raw bundles

[Copilot]

Pull request overview

Publishes a verifiable, redacted RTX 2060 GPU evidence packet (two clean runs), generated divergence analyses, and a new “two-run” replication comparator surfaced via both the Python API and CLI, alongside documentation updates that reflect the measured result and its limits.

Changes:

• Added compare_gpu_replications + write_gpu_replication_comparison and exposed them via a new tracebench gpu-compare CLI subcommand.
• Expanded GPU tests to cover replication comparisons (happy path, mismatch reporting, rejection cases) and to assert checked-in GPU packet outputs do not drift.
• Added/published GPU evidence bundles + generated analyses + replication-comparison outputs, and updated docs/READMEs to report the measured result and limitations.

Reviewed changes

Copilot reviewed 31 out of 35 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_gpu.py	Adds unit/integration coverage for replication comparison, plus drift checks against checked-in GPU artifacts.
src/tracebench/gpu.py	Implements replication comparison + writers for JSON/CSV/Markdown outputs.
src/tracebench/cli.py	Adds gpu-compare subcommand wiring to the new replication comparator.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/runner.log	Publishes run3 log evidence.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/requests.jsonl	Publishes run3 request bundle.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/plan.json	Publishes the registered plan for run3.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/packages.txt	Publishes run3 package inventory.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/nvidia-smi.txt	Publishes redacted GPU inventory for run3.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/manifest.json	Publishes run3 manifest binding evidence files.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/environment.json	Publishes run3 environment binding (runtime digest, versions, etc.).
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-public-evidence/checksums.sha256	Publishes run3 checksums sidecar.
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-analysis/gpu-results.md	Publishes run3 generated analysis (Markdown).
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-analysis/gpu-results.json	Publishes run3 generated analysis (JSON).
results/gpu/qwen25-05b-rtx2060-batching-v1/run3-analysis/gpu-results.csv	Publishes run3 generated analysis (CSV).
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/runner.log	Publishes run2 log evidence.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/requests.jsonl	Publishes run2 request bundle.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/plan.json	Publishes the registered plan for run2.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/packages.txt	Publishes run2 package inventory.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/nvidia-smi.txt	Publishes redacted GPU inventory for run2.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/manifest.json	Publishes run2 manifest binding evidence files.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/environment.json	Publishes run2 environment binding (runtime digest, versions, etc.).
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-public-evidence/checksums.sha256	Publishes run2 checksums sidecar.
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-analysis/gpu-results.md	Publishes run2 generated analysis (Markdown).
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-analysis/gpu-results.json	Publishes run2 generated analysis (JSON).
results/gpu/qwen25-05b-rtx2060-batching-v1/run2-analysis/gpu-results.csv	Publishes run2 generated analysis (CSV).
results/gpu/qwen25-05b-rtx2060-batching-v1/replication-comparison/gpu-replication.md	Publishes generated replication comparison (Markdown).
results/gpu/qwen25-05b-rtx2060-batching-v1/replication-comparison/gpu-replication.json	Publishes generated replication comparison (JSON).
results/gpu/qwen25-05b-rtx2060-batching-v1/replication-comparison/gpu-replication.csv	Publishes generated replication comparison (CSV).
results/gpu/qwen25-05b-rtx2060-batching-v1/README.md	Adds public-facing experiment report, evidence map, and limits.
results/gpu/qwen25-05b-rtx2060-batching-v1/operator-log.md	Adds disclosed run chronology (including excluded/aborted attempts).
README.md	Updates repo-level README with measured GPU result and links to the packet.
docs/gpu-path.md	Updates GPU path doc to reflect published evidence, result, and new compare command.
.gitattributes	Adds Git attributes intended to preserve evidence bytes for public GPU bundles.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1af2b21098

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject cloned execution payloads as replications

When a bundle is copied and only started_at/completed_at are changed with checksums regenerated, the manifest hashes differ, so this check accepts the two bundles and reports identical_execution_outcomes as a successful replication even though no second execution occurred. This undermines the comparator's core replication claim; require evidence of distinct execution payloads—at minimum different executions_sha256 values—rather than merely distinct manifest bytes.

Useful? React with 👍 / 👎.