Skip to content

Remove the b<->y intensity swap from generated decoys - #58

Merged
brendanx67 merged 4 commits into
mainfrom
fix/decoy-by-swap-removal
Jul 27, 2026
Merged

Remove the b<->y intensity swap from generated decoys#58
brendanx67 merged 4 commits into
mainfrom
fix/decoy-by-swap-removal

Conversation

@brendanx67

@brendanx67 brendanx67 commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Generated (reverse) decoys were badly miscalibrated: measured against an FDRBench
entrapment oracle they carried roughly 12x the reported error rate, ~11.8% true FDP
at a claimed 1% q on Stellar and 7.6% on Astral. This traces to a single defect.

RecalculateFragments mapped target b_k to decoy y_{n-k} and carried the copied
relative intensity along with the relabel. The reasoning behind it -- "which decoy
ion covers the same residues?" -- was sound, but fragment intensity is dominated by
ion TYPE, not by residue coverage: y ions are systematically more intense than b
ions, so relocating intensity across the swap inverts the strongest structural
feature of the spectrum. A source review of six open-source implementations found
five of six map intensity to the same ion label; this swap was unique to Osprey.

Dropping it recovers a near-honest null with no predictive model:

dataset true FDP @ 1% q, before -> after
Stellar 11.81% -> 1.47%
Astral 7.64% -> 2.03%

For scale, Carafe-predicted library decoys measure 1.92% on the same Astral cell, so
the fixed generated decoys reach parity with model-predicted ones. On an 82-file
SEA-AD set the fixed construction recovers 19% MORE real identifications than library
decoys at a matched TRUE 1% FDP, so this is not a sensitivity trade.

The paired decoy-win fraction tells the same story from a decoy-independent angle:
within a target/decoy pair an honest decoy should beat a known-false entrapment
target about half the time. Before the fix it won 0.205 of the time; after, 0.473,
against 0.501 for library decoys.

The second commit adds the similarity gate every surveyed implementation already
carries (EncyclopeDIA 0.4 fragment overlap, SpectraST 0.7 spectral similarity,
OpenSWATH 0.5 sequence identity). Its effect at library scale is nil -- ~1e-4 of
peptides excluded, entrapment FDP unchanged within noise -- and it is kept for
robustness at small library scale, where palindromes and low-complexity runs are a
far larger fraction.

Two things measured and REJECTED, recorded so they are not re-litigated: Skyline's
+10-unit precursor m/z shift (net negative here, 1.47 -> 1.96% Stellar and
2.03 -> 3.49% Astral -- it buys MS1 separation, but a decoy-only handicap is
spurious separation), and both similarity gates as an FDP lever.

Cross-implementation parity, both sides built from matching source:

dataset Rust C# result
Stellar 3-file 35,222 precursors 35,222 PASS at 1e-9
Astral 3-file 126,474 precursors 126,474 PASS at 1e-9

Stage 7 protein FDR and full blib content both compare clean per-column at 1e-9.

The matching ProteoWizard/pwiz change is on branch
Skyline/work/20260726_osprey_regression_redesign, which also re-records the C#
regression goldens against this construction and adds a libdecoy+entrapment
regression dataset with FDR calibration spot checks. The two must land together:
the C# golden re-recording is only defensible alongside this change.

cargo fmt, clippy -D warnings and cargo test all pass.


Added 2026-07-27: the ladder had a second defect, and the invariant it relies on was enforced on one path

theoretical_ladder dropped every y ion when any residue was unknown. It derived y as
total - prefix, so one residue missing from the mass table made total NaN and killed
the whole y series - not just the ions spanning it, which is what the comment above it
promised. A LEADING unknown emptied the ladder outright, and an empty candidate ladder is
read by is_candidate_acceptable as "accept": the gate silently switched itself off for
exactly the peptides it could not evaluate.

Not hypothetical. The Astral reference library (1,148,928 distinct peptides) carries 60
selenocysteine (U) peptides
; the Stellar library carries none. With half their ladder
missing, the inflated overlap ratio pushed the plain reversal past 0.4 and forced the
cycling fallback for three of them:

GPSPPPMAGGUGR        was PSPPPMAGGUGGR       now GUGGAMPPPSPGR
GFVCIVTNVASQUGK      was FVCIVTNVASQUGGK     now GUQSAVNTVICVFGK
SYSSKPPLWAAELCVTUSS  was YSSKPPLWAAELCVTUSSS now SUTVCLEAAWLPPKSSYSS

y ions now come from suffix sums, mirroring the prefix sums used for b ions.

MIN_PEPTIDE_LENGTH = 6 is now enforced by all three format loaders (DIA-NN TSV,
blib, elib) rather than assumed. The overlap gate compares the FULL ladder, so its two
permutation-invariant rungs put a structural 1/(n-1) floor under the ratio; at length 3
that floor is 0.5 against a 0.4 budget and every candidate would be rejected, dropping the
peptide from the search with no diagnostic. At the enforced minimum it is 0.2. The bound
cannot fire on real input - the Astral library's shortest peptide is 7 residues - so it
exists to make a malformed library fail loudly instead of quietly changing what is
searched.

Tests. Neither the swap removal nor the overlap gate had any. Four were added, mirroring
OspreySharp's DecoyConstructionTest one for one: ion type + ordinal preserved with the m/z
read off the decoy's own ladder; the gate rejects an isobaric near-duplicate (AILLAK ->
ALLIAK, where I/L isobaricity makes every rung coincide) and accepts an ordinary
tryptic reversal, so a gate that rejected everything cannot pass; generation never emits a
rejected candidate; and the ladder drops only the ions spanning an unknown residue. Both
implementations assert the same rung counts, so a change on either side that is not mirrored
breaks both suites.

Also restores the doc comment for generate_all_with_collision_detection, which the
overlap-gate commit left attached to MAX_FRAGMENT_OVERLAP, and rewrites the docs'
ion-swap section to describe what the code now does.

The matching OspreySharp PR is ProteoWizard/pwiz#4480 and must land with this one. It
re-records the Astral golden for the corrected ladder and adds the regression coverage that
would have caught the swap.

Gates: cargo fmt --check, clippy -D warnings, cargo test all green.

Decoy fragments now keep the target's ion type and ordinal (a target y7 yields a
decoy y7); only the m/z is recomputed for the permuted sequence, so the copied
relative intensity stays on the same-numbered ion.

The previous mapping sent target b_k to decoy y_{n-k} and carried the intensity
along with the relabel. The residue-coverage reasoning behind it was sound, but
fragment intensity is dominated by ion TYPE, not by which residues an ion spans:
y ions are systematically more intense than b ions, so the swap inverted the decoy
spectrum's intensity structure relative to any real peptide.

Measured with entrapment (FDRBench-equivalent, pass 1, experiment-wide), true FDP
at a claimed 1% q:

  Stellar   10.9% -> 1.5%
  Astral     7.6% -> 2.0%   (library-decoy reference 1.9%)

On an 82-file SEA-AD set the fixed construction holds at 2.1% and recovers 19% MORE
identifications than Carafe library decoys at matched TRUE 1% FDP.

Skyline, OpenSWATH, DIA-NN, EncyclopeDIA and SpectraST all map the intensity to the
same ion; none of them swaps.

Matches the C# change on ProteoWizard/pwiz branch
Skyline/work/20260725_osprey_gendecoy_decision.
Adds the similarity gate every other implementation surveyed already has
(EncyclopeDIA getSmartDecoy 0.4, SpectraST spectral similarity 0.7, OpenSWATH
sequence identity 0.5): a candidate decoy whose theoretical b/y ladder coincides
with its target's by more than 40% is rejected, and the existing cycling fallback
supplies another candidate.

Effect at library scale is nil -- on the order of 1e-4 of peptides excluded, with
entrapment FDP unchanged within noise. It is kept for robustness at small library
scale, where palindromes and low-complexity runs are a far larger fraction and a
near-identical decoy would be both real and confusing.

The match window is a fixed 0.02 Da rather than the run's fragment tolerance, so
the decoy set stays a pure function of the library instead of varying with
instrument resolution. The ladder ignores modifications, which shift both sides
alike. Both choices keep this rule byte-identical to the C# implementation.

Matches ProteoWizard/pwiz branch Skyline/work/20260725_osprey_gendecoy_decision.
…esidue

theoretical_ladder derived y ions as (total - prefix), so a single residue missing from
the standard mass table made `total` NaN and killed EVERY y ion - not just the ones
actually spanning it, which is what the surrounding comment promised. A LEADING unknown
residue emptied the ladder outright, and an empty candidate ladder is read by
is_candidate_acceptable as "accept", silently switching the similarity gate off for
exactly the peptides it could not evaluate.

y ions now come from suffix sums, mirroring the prefix sums used for b ions, so only ions
that actually span the unknown residue are dropped.

This is not hypothetical. The Astral reference library (1,148,928 distinct peptides)
contains 60 selenocysteine (U) peptides. With half their ladder missing, the inflated
overlap ratio pushed the plain reversal past the 0.4 threshold and forced the cycling
fallback for three of them:

  GPSPPPMAGGUGR        was PSPPPMAGGUGGR       now GUGGAMPPPSPGR
  GFVCIVTNVASQUGK      was FVCIVTNVASQUGGK     now GUQSAVNTVICVFGK
  SYSSKPPLWAAELCVTUSS  was YSSKPPLWAAELCVTUSSS now SUTVCLEAAWLPPKSSYSS

Selenocysteine and the ambiguity codes B/Z/X/J/O are all absent from the residue table
and do occur in UniProt-derived libraries.

Also adds the first tests for decoy construction - the b<->y swap removal and the overlap
gate had none - and restores the doc comment for
generate_all_with_collision_detection, which the previous commit left attached to
MAX_FRAGMENT_OVERLAP. The docs' ion-swap section is rewritten to describe what the code
now does, plus a section on the overlap gate.

The four tests mirror OspreySharp's DecoyConstructionTest one for one, including the
selenocysteine ladder rung counts, so a change on either side that is not mirrored breaks
both suites.

Matches the C# change on ProteoWizard/pwiz branch
Skyline/work/20260726_osprey_regression_redesign.
is_candidate_acceptable compares the FULL theoretical ladder, matching EncyclopeDIA's
rule so that their published 0.4 threshold is measured over their statistic. Two rungs of
that ladder are invariant under any C-terminus-preserving permutation - y1, and b_{n-1},
whose prefix multiset never changes - so they always match and put a structural 1/(n-1)
floor under the overlap ratio. At length 3 that floor is 0.5 against a 0.4 budget: every
candidate would be rejected and the peptide would vanish from the search with no
diagnostic.

Rather than weaken the statistic to defend against input that real libraries do not
produce, the bound is enforced at load: validate_peptide_length rejects a peptide shorter
than MIN_PEPTIDE_LENGTH = 6, naming the offending peptide. At the minimum the floor is
1/5 = 0.2, comfortably clear.

Called by all three format loaders (DIA-NN TSV, blib, elib). An invariant enforced on one
of the paths into the library is not an invariant, and the decoy generator cannot tell
which loader produced its input.

The bound cannot fire on real input: MaxQuant's convention is 6, the Carafe libraries the
reference datasets use start at 7, and the Astral library's shortest peptide is 7. It
exists so that a malformed library fails loudly instead of quietly changing what gets
searched.

Matches the C# change on ProteoWizard/pwiz branch
Skyline/work/20260726_osprey_regression_redesign.
@brendanx67
brendanx67 requested a review from maccoss July 27, 2026 13:44
@brendanx67
brendanx67 merged commit eb26b32 into main Jul 27, 2026
3 checks passed
@brendanx67
brendanx67 deleted the fix/decoy-by-swap-removal branch July 27, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant