DeepAudit

DeepAudit is an automated reproducibility-audit pipeline for research papers and their companion code repositories. It reads a paper in Markdown or PDF form, extracts verifiable claims, analyzes the repository, runs an isolated execution phase, and produces a final verdict and report.

The CLI entry point is python -m p2c.main. The phase order is defined in p2c/graph.py.

Pipeline

Phase 1: paper ingestion and repository analysis

Phase 1 ingests the paper, extracts claim fingerprints, identifies visual evidence, analyzes the target repository, and builds the handoff files for later phases. If --paper_pdf is supplied, the pipeline can derive Markdown from the PDF during phase 1.

Main outputs:

• fingerprint/claims_ir.json
• task/task_spec.json
• task/metric_contract.json
• task/repo_analysis.json
• fingerprint/fingerprint.json

Phase 2: environment setup and autonomous execution

Phase 2 builds or repairs the target execution environment, opens a Claude Agent SDK session in the target repository, and runs the execution plan. Native conda/mamba environment files are preferred when the target repository provides one; otherwise the pipeline synthesizes an environment specification and falls back to a local venv when needed.

The executor rejects runs that mutate tracked source files in the target repository.

Main outputs:

• execution/env_setup_result.json
• execution/executor_outputs/phase2_execution_package.json
• execution/executor_outputs/run_manifest.json
• execution/executor_outputs/executor_results.json

Phase 3: evidence alignment and report generation

Phase 3 consumes phase 2 evidence, aligns it against the extracted claims, scores reproducibility, and writes the final audit report. When present, execution/executor_outputs/phase2_execution_package.json is the preferred phase 2 evidence source; otherwise the pipeline falls back to the manifest and executor result files produced by phase 2.

Main outputs:

• results/verdict.json
• results/reproducibility_score.json
• results/report.md

Repository Layout

Tracked source and test code:

p2c/         Core pipeline package
scripts/     Local helper scripts for single-run and batch workflows
tests/       Pytest coverage for the pipeline
requirements.txt
README.md
.gitignore

Runtime workspace used by the pipeline:

Target/
  paper.pdf
  paper/
    full.md
  code/
output/
artifacts/
paper_with_code/

Target/ is the default location for a single audit case. paper_with_code/ is used by the batch scripts for dataset-style runs.

Requirements

• Python 3.10 or newer
• bash for the helper scripts in scripts/
• conda or mamba is strongly recommended for phase 2 when the target repository uses a native conda environment

Install the Python dependencies:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install pytest

On macOS or Linux, use the equivalent shell activation command instead of the PowerShell line above.

Environment Variables

Required for a full run:

• OPENAI_API_KEY - used by phase 1 and phase 3
• ANTHROPIC_API_KEY - used by phase 2

Common optional variables:

• OPENAI_MODEL - OpenAI-compatible model for phase 1 and phase 3, default gpt-5.4
• OPENAI_BASE_URL - OpenAI-compatible endpoint, default https://api.openai.com/v1
• OPENAI_TIMEOUT_SEC - timeout for text and JSON requests, default 300
• OPENAI_VISION_TIMEOUT_SEC - timeout for vision requests, default at least 360
• P2C_CLAUDE_MODEL - Claude model used by phase 2, default claude-haiku-4-5-20251001
• P2C_EXECUTION_MODE - phase 2 mode, commonly standard or full
• P2C_MAX_ENV_PATCH - maximum phase 2 environment repair attempts, default 2
• P2C_KEEP_CONDA_ENV - keep the phase 2 environment after the run when set
• P2C_MINERU_MODE - PDF-to-Markdown mode, supports auto, agent, lightweight, standard, precise, and off
• MINERU_API_TOKEN - optional token for MinerU fallback parsing

Running the Pipeline

Single phase

Phase 1 can ingest either Markdown or PDF. If --paper_md is omitted, --paper_pdf must be supplied and the pipeline derives the Markdown path automatically.

python -m p2c.main \
  --phase 1 \
  --paper_md Target/paper/full.md \
  --paper_md_out output/paper.md \
  --paper_pdf Target/paper.pdf \
  --repo_dir Target/code \
  --run_id audit_001 \
  --artifacts_dir artifacts \
  --budget_minutes 60

python -m p2c.main \
  --phase 2 \
  --paper_md output/paper.md \
  --paper_md_out output/paper.md \
  --repo_dir Target/code \
  --run_id audit_001 \
  --artifacts_dir artifacts \
  --budget_minutes 180

python -m p2c.main \
  --phase 3 \
  --paper_md output/paper.md \
  --paper_md_out output/paper.md \
  --repo_dir Target/code \
  --run_id audit_001 \
  --artifacts_dir artifacts \
  --budget_minutes 60

Helper script

scripts/run_audit.sh runs the three phases in order for a single case. It resolves the repository root from the script location, and you can override the default paths with environment variables when needed.

./scripts/run_audit.sh audit_001
./scripts/run_audit.sh audit_001 1,2
./scripts/run_audit.sh audit_001 3

Outputs

For run ID audit_001, the main artifacts are written under artifacts/audit_001/:

fingerprint/claims_ir.json
task/task_spec.json
task/metric_contract.json
execution/run.log
execution/context.json
execution/env_setup_result.json
execution/executor_outputs/phase2_execution_package.json
execution/executor_outputs/run_manifest.json
results/verdict.json
results/reproducibility_score.json
results/report.md

Placeholder files are created early by ArtifactManager.ensure_tree(), so existence alone does not mean a phase completed. For phase 2, inspect execution/executor_outputs/phase2_execution_package.json or execution/executor_outputs/run_manifest.json and confirm that they contain non-empty experiments or runs arrays.

Tests

Run the full suite:

python -m pytest tests/ -v

Useful targeted checks:

python -m pytest tests/test_main_context.py -v
python -m pytest tests/test_phase2_fixes.py -v
python -m pytest tests/test_phase3_report.py -v
python -m pytest tests/test_visual_enrichment.py -v