libs: solution-ranker — walk the MAR for ABB fillers, rank into a SolutionShortlist - #1122
Merged
Conversation
…utionShortlist
Sixth and final substrate move. Backs Michael's `shortlist_solutions` capability
(prophet-mesh#17) — the move that turns "silently auto-route to the top score" into "here are
your options, and here is why".
Zurich's Damian does the right thing in its SCRIPT ("we have a few options — do you already
have a model?") and the wrong thing in its ARCHITECTURE (a single resolved route). The gap
between those two is where a user loses the ability to see what was almost chosen.
FILTER BEFORE RANK. That ordering is the substance, not an implementation detail. A candidate
below the trust floor is not a low-ranked option — it is not an option. Ranking it would put
it on screen where a user could pick what the ledger already judged untrustworthy. Same for a
denied access grade. Three exclusion reasons, each RECORDED not dropped:
- does not claim the required ABB (not a candidate at all)
- trust below floor (the ledger's judgement gates candidacy)
- access denied (an unusable option must not be offered)
A `requires-consent` candidate IS ranked. That grade exists to be shown with a path forward;
excluding it would collapse it into denied, which is exactly what the three-grade shape in
access-prewalk prevents.
An unverified ABB claim still shows but scores below a verified one on equal evidence
(x0.8). The registry establishes the CLAIM, not its truth — sourceos-spec#224 says so
explicitly, and the score reflects it rather than pretending the distinction doesn't exist.
Auto-route is the narrowest claim the evidence supports, requiring ALL of:
top-2 gap > 0.15 AND counter-test confirmed AND access granted AND >= 2 candidates
Every path to `user-pick` names WHICH precondition blocked it, and reports ALL blockers rather
than the first — a caller fixing one should not discover the next on the retry. "Not confident
enough" is not a reason a user can act on; "the runner-up scored within 0.04" is.
Single candidate is `user-pick`, never auto-route: with nothing to compare against the gap is
undefined and "auto" would be the caller's default rather than a measured decision.
No survivors is `abstain` WITH an empty reason. Empties are signal; an empty shortlist without
a reason is indistinguishable from a broken walk.
20 tests, including the Eve Smith cross-selling scenario from the E-RDA2 deck run end to end.
What it demonstrates beyond their version: ours refuses to auto-route BECAUSE of the billing
access gap, states that as the reason, and keeps the runner-up visible — so Eve has a choice
between requesting approval and using the market-data option immediately, rather than only a
wait. AUTO_ROUTE_GAP_THRESHOLD is pinned equal to prophet-mesh's, so two services that both
believe they gate at the same threshold cannot silently gate differently.
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a new Python “solution-ranker” library that filters MeshActionRegistry participants by ABB/trust/access, ranks survivors with explainable reasons, and emits a SolutionShortlist with an explicit auto-route vs user-pick vs abstain verdict.
Changes:
- Introduces
solution_rankermodule with participant/candidate models, pre-filtering + scoring, shortlist/route decision logic, and canonical JSON/digest helpers. - Adds contract + end-to-end tests (incl. “Eve Smith” scenario) and sets up local + GitHub Actions test execution.
- Adds packaging/config files (
pyproject.toml,pytest.ini, requirements, Makefile) for the new lib.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/python/solution-ranker/src/solution_ranker/init.py | Core ranking/filtering + shortlist decision + JSON/digest emission. |
| libs/python/solution-ranker/tests/test_solution_ranker.py | Contract tests for exclusion-before-rank, scoring reasons, auto-route gates, and end-to-end scenario. |
| libs/python/solution-ranker/pyproject.toml | Packaging metadata and dev dependency declaration. |
| libs/python/solution-ranker/requirements-test.txt | Test dependency list used by CI install step. |
| libs/python/solution-ranker/pytest.ini | Pytest discovery + pythonpath config. |
| libs/python/solution-ranker/Makefile | Local install/test targets. |
| .github/workflows/solution-ranker.yml | CI workflow to run solution-ranker tests on pushes/PRs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+258
to
+264
| gap = top.score - candidates[1].score | ||
| blockers: list[str] = [] | ||
| if gap < AUTO_ROUTE_GAP_THRESHOLD: | ||
| blockers.append( | ||
| f"top-2 gap {gap:.3f} is under the {AUTO_ROUTE_GAP_THRESHOLD} threshold " | ||
| f"({top.repo} {top.score:.3f} vs {candidates[1].repo} {candidates[1].score:.3f})" | ||
| ) |
Comment on lines
+174
to
+177
| grade, access_reason, remediation = access_grader(p.repo) | ||
| if grade == "denied": | ||
| excluded.append((p.repo, f"access denied: {access_reason}")) | ||
| continue |
Comment on lines
+290
to
+297
| def canonical_json(obj: Any) -> str: | ||
| """Canonical JSON — recursive key sort, no whitespace, non-ASCII RAW. | ||
|
|
||
| ``ensure_ascii=False`` matches lawful-verdict and the TypeScript canonicaliser. The | ||
| default would escape non-ASCII and every digest over accented content would diverge | ||
| between languages (the bug caught in review on #1065). | ||
| """ | ||
| return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False) |
| @@ -0,0 +1 @@ | |||
| pytest>=8.3 | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixth and final substrate move. Backs Michael's
shortlist_solutionscapability (prophet-mesh#17) — the move that turns silently auto-route to the top score into here are your options, and here is why.Zurich's Damian does the right thing in its script ("we have a few options — do you already have a model?") and the wrong thing in its architecture (a single resolved route). That gap is where a user loses the ability to see what was almost chosen.
Filter before rank — the substance, not an implementation detail
A candidate below the trust floor is not a low-ranked option, it is not an option. Ranking it would put it on screen where a user could pick what the ledger already judged untrustworthy.
Every exclusion is recorded with a reason, not dropped — an unexplained absence is indistinguishable from a walk that never saw the participant.
requires-consentIS ranked. That grade exists to be shown with a path forward; excluding it would collapse it into denied — exactly what access-prewalk's three-grade shape prevents.An unverified ABB claim still shows but scores ×0.8. The registry establishes the claim, not its truth (sourceos-spec#224 says so explicitly); the score reflects that rather than pretending the distinction doesn't exist.
Auto-route is the narrowest claim the evidence supports
Requires all of: top-2 gap > 0.15 · counter-test confirmed · access granted · ≥2 candidates.
Every path to
user-picknames which precondition blocked it, and reports all blockers rather than the first — a caller fixing one shouldn't discover the next on the retry. "Not confident enough" is not a reason a user can act on; "the runner-up scored within 0.04" is.user-pick, never auto-route: with nothing to compare, the gap is undefined and "auto" would be the caller's default rather than a measured decisionabstainwith an empty reason. Empties are signalThe Eve Smith scenario, end to end
20 tests, including the exact flow from the E-RDA2 deck. What ours does beyond theirs: refuses to auto-route because of the billing access gap, states that as the reason, and keeps the runner-up visible — so Eve has a choice between requesting approval and using the market-data option immediately, rather than only a wait.
AUTO_ROUTE_GAP_THRESHOLDis pinned equal to prophet-mesh's, so two services that both believe they gate at the same threshold cannot silently gate differently.