Skip to content

fix(view): reverse path flags describe the published path; raw revTr loop exposed as revtr_loop_cleaned - #50

Merged
Burdantes merged 3 commits into
mainfrom
fix/reverse-flag-cutover
Oct 12, 2026
Merged

Burdantes merged 3 commits into
mainfrom
fix/reverse-flag-cutover

Conversation

@Burdantes

@Burdantes Burdantes commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

What changed since the first version of this PR

The first version only moved a date gate (9999-12-31 → 2026-09-30) so the reverse struct's loop_detected / unresponsive_within_as would read step 04's reverse_loop / reverse_unresponsive_within_AS. Its checklist expected the reverse loop rate to drop to "a few percent"; it dropped from 75.1% to 18.3% — and investigation showed why that column cannot be published under that name:

  • step 04 computes reverse_loop on the raw revTr (reverse_node_with_geo_info), before its cleanup drops ambiguous hop_type=4 excursions and truncates at the first AS re-entry (reverse_hops_clean);
  • so a published reverse path never contains a loop (0 of 231,870 sampled on 2026-10-01), yet ~18% of reverse paths would show loop_detected = TRUE next to an as_path with no loop;
  • and loop_detected already means "the published path" in server_to_client_path.

Now

  • Both directions compute loop_detected and unresponsive_within_as from their own published hops, with one definition (A B A loop; A … A gap over unmapped hops). Reverse loop_detected is FALSE by construction and documented as such. Reverse unresponsive_within_as is now available for every date (it was NULL before 2026-09-30).
  • New client_to_server_path.revtr_loop_cleaned: TRUE when the raw revTr had an AS loop that step 04 removed or truncated (the published path is then shorter than what was measured). From 2026-09-30; NULL before, since the raw path is not stored and the old definition was wrong.
  • reverse_unresponsive_within_AS joins forward_loop / forward_unresponse_within_AS as physical columns the view supersedes from the hops.

Evidence

The reworked view SQL rendered as a read-only SELECT over prod partitions:

2026-09-29 2026-09-30 2026-10-01
reverse loop_detected TRUE 0 0 0
reverse unresponsive_within_as 6.95% 6.97% 7.04%
revtr_loop_cleaned NULL (all 2.47M) 18.26% 18.23%
forward loop_detected (unchanged) 0.89% 0.89% 0.88%

Original checklist: zeroed Welch groups 121 (09-29) → 0 (09-30, 10-01) ✅; image 4d73983 for the 10-01 nightly — inferred (built 09-30, both fixes visible from the 09-30 partition) ✅.

Follow-up (separate PR)

Recording in step 04 which cleanup happened — only a type-4 hop dropped (path complete) vs truncated (path shortened) — would make revtr_loop_cleaned more informative.

Deploy

View only: run create_events_enriched.sql with DS=hermes_union (additive: one new field in client_to_server_path).

🤖 Generated with Claude Code

Burdantes and others added 3 commits September 30, 2026 16:58
Replaces the 9999-12-31 sentinel from #48 with the first partition_date
written by the fixed step 04. The image carrying the fix (08d689b, then
4d73983) was deployed on 2026-09-30 after that day's nightly, which had
already processed 2026-09-28 and 2026-09-29, so the next nightly
(2026-10-01 15:00 UTC) writes 2026-09-30 first.

Merge only after confirming 2026-09-30 in hermes_union.events_with_as_and_geoloc
was written by that run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loop is revtr_loop_cleaned

The reverse struct read step 04's reverse_loop / reverse_unresponsive_within_AS,
which are computed on the RAW revTr, before step 04 drops ambiguous hop_type=4
excursions and truncates at the first AS re-entry. A published reverse path
therefore never has a loop, yet ~18% of reverse paths would have shown
loop_detected = TRUE beside an as_path with no loop -- and the same field name
meant "the published path" in the forward struct.

Now both directions compute loop_detected / unresponsive_within_as from their
own published hops with one definition, and the raw-revTr fact is published as
client_to_server_path.revtr_loop_cleaned (from 2026-09-30, the first partition
with the fixed step-04 definition; NULL before, since the raw path is not
stored).

Rendered view logic on real data (2026-09-29 / 09-30 / 10-01): reverse
loop_detected 0 / 0 / 0; reverse unresponsive_within_as 6.95% / 6.97% / 7.04%
(no longer NULL before the fix date); revtr_loop_cleaned NULL / 18.26% /
18.23%; forward loop rate unchanged (0.89%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Burdantes Burdantes changed the title Expose reverse path flags from 2026-09-30 fix(view): reverse path flags describe the published path; raw revTr loop exposed as revtr_loop_cleaned Oct 12, 2026
@Burdantes
Burdantes marked this pull request as ready for review October 12, 2026 20:08
@Burdantes
Burdantes merged commit f96f969 into main Oct 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant