Add misc/benchmark_truth.py: score both modes against a truth set - #28
Merged
Merged
Conversation
…h set Everything measured about --mode fast so far is agreement with --mode sensitive, which assumes the alignment path is right. This script scores both modes against an external truth VCF instead, so "do the modes agree" and "which is closer to the truth" stop being the same number. It takes the target BED from the truth VCF's own coordinates, so both sides use the same REF interval and STRdust's RB is directly comparable to len(ALT) - len(REF) from the truth record; scoring against a separate repeat catalog would reintroduce the boundary mismatch that makes such comparisons ambiguous. Loci outside the benchmark's confident regions are skipped rather than silently scored, QUICKREF loci are read as a reference-length call rather than a missing one, and results are stratified by how far the truth allele departs from the reference, which is where the two modes are expected to diverge. Standard library only for running and parsing, so it works on a bare cluster environment; matplotlib is imported only for --plot. --from-vcfs re-scores existing output without rerunning, and --regions/--max-loci give a quick smoke run. Verified end to end on a 30x ONT sample by feeding a STRdust VCF back as the truth set: --mode sensitive then scores 100% exact against itself across all strata, which is the identity check that the coordinate handling and allele pairing are right, while --mode fast produces its real numbers alongside. Refs #23. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bw5mwMVPc2gAqa2ApibDN1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #23. The harness only — you run it where HG002 lives.
What it does
Runs
--mode sensitiveand--mode fastover a truth set's loci and scores both against the truth allele lengths, rather than against each other. Every number quoted for--mode fastso far has been agreement withsensitive, which assumes the alignment path is right; this separates "do the modes agree" from "which is closer to the truth".Design choices worth checking
RBis directly comparable tolen(ALT) - len(REF)from the truth record. Scoring against a separate repeat catalog would reintroduce the boundary mismatch that makes such comparisons ambiguous — the same mismatch behind the neighbour-locus disagreements in Add --mode fast: genotype straight from the existing alignment #25.--confident-bed.Verification
Feeding a STRdust VCF back in as the truth set is an identity check —
--mode sensitivethen scores 100% exact across every stratum, which is what confirms the coordinate handling and allele pairing are right. On a 30x ONT sample over 300 loci:(That is the fast-vs-sensitive comparison again, since the pseudo-truth is sensitive's output — it is here to show the harness works, not as a result.)
Also exercised:
--from-vcfs,--plot,--regions,--max-loci, and the argument validation.Running it
python misc/benchmark_truth.py \ --binary ./target/release/STRdust \ --fasta GRCh38.fa --bam HG002.cram \ --truth-vcf HG002_TR_benchmark.vcf.gz \ --confident-bed HG002_TR_benchmark_regions.bed \ --threads 8 --out-prefix hg002_tr --plotStandard library only for running and parsing, so it needs no environment beyond Python; matplotlib is imported only for
--plot. Start with--regions chr1 --max-loci 2000to check the truth set parses the way you expect before committing to a whole-genome run.