Skip to content

Condition intensity features with log10 to prevent Percolator hijack (v26.7.0) - #54

Merged
maccoss merged 3 commits into
mainfrom
feature/intensity-log-conditioning
Jul 13, 2026
Merged

Condition intensity features with log10 to prevent Percolator hijack (v26.7.0)#54
maccoss merged 3 commits into
mainfrom
feature/intensity-log-conditioning

Conversation

@maccoss

@maccoss maccoss commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

Lands the intensity-feature log10 conditioning on main (decoupled from the WIP percolator-streaming-only branch), with a defensive floor, tests, docs, and a 26.7.0 release cut. Supersedes #53, which targeted the WIP branch.

peak_apex, peak_area, and peak_sharpness reached the first-pass Percolator SVM as raw, linear, heavy-tailed intensities. Because a single experiment-wide standardizer and one linear model score every peak across all runs, a lone high-intensity DIA interference standardizes to a z-score of 100-300 that dominates the discriminant, letting intensity outliers (a random mix of target / decoy / entrapment) hijack the top of the score ranking. On SEA-AD Astral-DIA entrapment runs this floored the achievable q-value, produced a top-of-ranking FDP spike, and at 1:1 entrapment collapsed a whole run to zero IDs.

What changed

  • Single source of truth for the transform. All three features now route through condition_intensity_feature(x) = log10(max(0, x) + 1) at their one computation point (compute_features_at_peak), matching Skyline mProphet's MQuestIntensityCalc and the paired ProteoWizard/pwiz #4412.
  • Defensive floor with zero cross-impl cost. max(0.0) is the exact identity on the non-negative real data (intensities/areas are >= 0), so output is bit-identical to the un-floored form and to Condition intensity features with log10 to prevent Percolator hijack #53 / pwiz; the floor only changes the impossible-today negative case and keeps peak_sharpness finite when an override/CWT apex sits below an edge and yields a negative mean slope. peak.area stays raw for quantification.
  • Point-1 cache fix = version bump to 26.7.0. Feature semantics changed, so the version bump doubles as the cache-invalidation mechanism: validate_scores_cache exact-matches the version (26.6.x .scores.parquet caches go stale and are re-scored on first run), and cross-impl --join-only aborts on minor-version mismatch rather than silently mixing raw and logged scales.
  • Release notes. RELEASE_NOTES_v26.7.0.md documents all 18 unreleased commits since v26.6.1 plus this change; RELEASE_NOTES_next.md reset to a clean draft.
  • Docs. docs/README.md, docs/07-fdr-control.md, docs/12-intermediate-files.md note the conditioning and the cache invalidation.

Tests

  • 5 new unit tests: zero -> 0, decade scaling, monotonicity, negative/NaN flooring, and floor-is-identity-on-non-negative (the cross-impl parity guarantee).
  • cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings, and the full suite (141 passed / 0 failed) all pass locally.

Cross-impl follow-up

OspreySharp should mirror the uniform max(0, x) floor (correct even before that, since it is bit-identical on the non-negative range) and bump to 26.7.0 so cross-impl joins line up.

🤖 Generated with Claude Code

https://claude.ai/code/session_0146d9VSgpxxA6xp2r6fE3uz

maccoss and others added 2 commits July 12, 2026 17:18
…ijack

peak_apex, peak_area, and peak_sharpness reached the first-pass Percolator SVM
as raw, linear, heavy-tailed intensities. A single experiment-wide standardizer
feeding one linear model maps a lone high-intensity DIA interference to a
z-score of 100-300 that dominates the discriminant, letting intensity outliers
(a random mix of target / decoy / entrapment) hijack the top of the ranking:
floored q-values, a top-of-ranking FDP spike, and a full run collapsing to zero
IDs at 1:1 entrapment on SEA-AD Astral runs.

Routed all three through a single condition_intensity_feature helper computing
log10(max(0, x) + 1), matching Skyline mProphet's MQuestIntensityCalc and the
paired ProteoWizard/pwiz change. The max(0.0) floor is the identity on the
non-negative real data, so output is bit-identical to the un-floored form and
preserves C#/Rust cross-impl parity, while keeping peak_sharpness finite when an
override/CWT apex sits below an edge and yields a negative mean slope. Only the
PIN feature is conditioned; peak.area stays raw for quantification.

Added 5 unit tests (zero, decade scaling, monotonicity, negative/NaN flooring,
floor-is-identity-on-nonnegative) and documented the conditioning in
docs/README.md, docs/07-fdr-control.md, and docs/12-intermediate-files.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146d9VSgpxxA6xp2r6fE3uz
Bumped the workspace version 26.6.1 -> 26.7.0. This is more than a patch: the
log10 conditioning changes how three scoring features are computed, so the
version bump doubles as the cache-invalidation mechanism. validate_scores_cache
exact-matches the version, so 26.6.x .scores.parquet caches are treated as stale
and re-scored on first run, and cross-impl --join-only aborts on minor-version
mismatch rather than silently mixing raw and logged intensity scales.

Finalized RELEASE_NOTES_v26.7.0.md covering all 18 unreleased commits since
v26.6.1 (intensity conditioning, cross-impl bit-parity and HPC join-chain
correctness fixes, calibration fixes, reconciliation fixes, the
--work-dir/--output-dir/--cache-dir feature, and the LGPL-3.0 relicense) and
reset RELEASE_NOTES_next.md to a clean draft.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146d9VSgpxxA6xp2r6fE3uz
Copilot AI review requested due to automatic review settings July 13, 2026 00:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a single, centralized log10(max(0, x) + 1) conditioning transform for intensity-magnitude scoring features to reduce heavy-tail/outlier dominance in Percolator scoring, and cuts a v26.7.0 release with accompanying docs and release notes.

Changes:

  • Apply condition_intensity_feature(x) = log10(max(0, x) + 1) to peak_apex, peak_area, and peak_sharpness at the single feature computation point (compute_features_at_peak) and add unit tests for the transform.
  • Update documentation to reflect the new feature semantics and the resulting .scores.parquet cache invalidation behavior.
  • Bump workspace/package versions to 26.7.0 and add/reset release notes (RELEASE_NOTES_v26.7.0.md, RELEASE_NOTES_next.md).

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
release-notes/RELEASE_NOTES_v26.7.0.md Adds v26.7.0 release notes including the intensity-feature conditioning and upgrade guidance.
release-notes/RELEASE_NOTES_next.md Resets the “next release” draft to an empty placeholder after cutting v26.7.0 notes.
docs/README.md Documents that peak_apex, peak_area, and peak_sharpness are now log10-conditioned (PIN features only).
docs/12-intermediate-files.md Notes that the stored peak-shape features in the scores parquet are log10-conditioned.
docs/07-fdr-control.md Updates the feature list and adds explanation of the conditioning and cache invalidation.
crates/osprey/src/pipeline.rs Implements condition_intensity_feature, routes the three features through it, and adds unit tests.
Cargo.toml Bumps workspace version to 26.7.0.
Cargo.lock Updates workspace crate versions to 26.7.0.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/osprey/src/pipeline.rs Outdated
/// mirrors the paired ProteoWizard/pwiz change (`PeakShapeCalculators`) to keep
/// C#/Rust cross-impl bit parity.
///
/// The `max(0.0)` floor keeps the argument `>= 1` so the result is always finite.
…otes

#55 landed the streaming-only Percolator, the best-of-runs experiment q-value
clamp, the log conditioning, and the hydrate_for_rescore fix, so this branch's
own copy of condition_intensity_feature was a duplicate. Resolved the pipeline.rs
conflict onto main's merged helper, kept this branch's granular unit tests, and
dropped the redundant consolidated one.

Release notes were written before #55 and described only the log conditioning.
Added the two missing Scoring/FDR entries -- the experiment q-value clamp (which
CHANGES REPORTED Q-VALUES: expect slightly higher experiment q and fewer IDs at a
fixed threshold than 26.6.x, the old numbers being optimistic) and the
streaming-only Percolator -- plus the hydrate_for_rescore fix under Bug Fixes.

Documented the clamp in docs/07-fdr-control.md, and corrected that doc's
subsampling section, which still described streaming as a size-conditional path.

Also corrected the helper doc: it claimed max(0, NaN) is NaN, which is true in C#
but NOT in Rust (f64::max ignores NaN). The input-flooring order is load-bearing
for C# parity, not for Rust's own safety; the NaN test now says so, and flags that
the two implementations would silently disagree on a NaN input.
@maccoss
maccoss merged commit ada1752 into main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants