Skip to content

Add misc/benchmark_truth.py: score both modes against a truth set - #28

Merged
wdecoster merged 1 commit into
mainfrom
bench/giab-truth
Aug 29, 2026
Merged

wdecoster merged 1 commit into
mainfrom
bench/giab-truth

Conversation

@wdecoster

Copy link
Copy Markdown
Owner

Refs #23. The harness only — you run it where HG002 lives.

What it does

Runs --mode sensitive and --mode fast over a truth set's loci and scores both against the truth allele lengths, rather than against each other. Every number quoted for --mode fast so far has been agreement with sensitive, which assumes the alignment path is right; this separates "do the modes agree" from "which is closer to the truth".

Design choices worth checking

  • The target BED comes from the truth VCF's own coordinates. Both sides then use the same REF interval, so STRdust's RB is directly comparable to len(ALT) - len(REF) from the truth record. Scoring against a separate repeat catalog would reintroduce the boundary mismatch that makes such comparisons ambiguous — the same mismatch behind the neighbour-locus disagreements in Add --mode fast: genotype straight from the existing alignment #25.
  • Loci outside the benchmark's confident regions are skipped, not silently scored, via --confident-bed.
  • QUICKREF loci count as a reference-length call, not a missing one — otherwise the fast path's cheapest calls would vanish from the denominator.
  • Results are stratified by how far the truth allele departs from the reference (reference / 1-10 / 11-50 / 51-200 / >200 bp), which is where the two modes are expected to part company.
  • No truth-set URLs are hardcoded: the GIAB FTP layout changes between releases, so the docstring points at the site and you pass the paths.

Verification

Feeding a STRdust VCF back in as the truth set is an identity check — --mode sensitive then scores 100% exact across every stratum, which is what confirms the coordinate handling and allele pairing are right. On a 30x ONT sample over 300 loci:

mode       stratum    loci_scored  exact_pct  within_5bp_pct  median_abs_diff  cpu_seconds
sensitive  all        300          100.0      100.0           0.0              71.2
fast       all        300          26.0       97.0            1.0              0.4
fast       reference  48           95.83      100.0           0.0              0.4
fast       1-10bp     247          12.96      96.76           2                0.4

(That is the fast-vs-sensitive comparison again, since the pseudo-truth is sensitive's output — it is here to show the harness works, not as a result.)

Also exercised: --from-vcfs, --plot, --regions, --max-loci, and the argument validation.

Running it

python misc/benchmark_truth.py \
    --binary ./target/release/STRdust \
    --fasta GRCh38.fa --bam HG002.cram \
    --truth-vcf HG002_TR_benchmark.vcf.gz \
    --confident-bed HG002_TR_benchmark_regions.bed \
    --threads 8 --out-prefix hg002_tr --plot

Standard library only for running and parsing, so it needs no environment beyond Python; matplotlib is imported only for --plot. Start with --regions chr1 --max-loci 2000 to check the truth set parses the way you expect before committing to a whole-genome run.

…h set

Everything measured about --mode fast so far is agreement with --mode sensitive,
which assumes the alignment path is right. This script scores both modes against
an external truth VCF instead, so "do the modes agree" and "which is closer to the
truth" stop being the same number.

It takes the target BED from the truth VCF's own coordinates, so both sides use the
same REF interval and STRdust's RB is directly comparable to len(ALT) - len(REF)
from the truth record; scoring against a separate repeat catalog would reintroduce
the boundary mismatch that makes such comparisons ambiguous. Loci outside the
benchmark's confident regions are skipped rather than silently scored, QUICKREF
loci are read as a reference-length call rather than a missing one, and results are
stratified by how far the truth allele departs from the reference, which is where
the two modes are expected to diverge.

Standard library only for running and parsing, so it works on a bare cluster
environment; matplotlib is imported only for --plot. --from-vcfs re-scores existing
output without rerunning, and --regions/--max-loci give a quick smoke run.

Verified end to end on a 30x ONT sample by feeding a STRdust VCF back as the truth
set: --mode sensitive then scores 100% exact against itself across all strata,
which is the identity check that the coordinate handling and allele pairing are
right, while --mode fast produces its real numbers alongside.

Refs #23.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bw5mwMVPc2gAqa2ApibDN1
@wdecoster
wdecoster merged commit 66434eb into main Aug 29, 2026
3 checks passed
@wdecoster
wdecoster deleted the bench/giab-truth branch August 29, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant