Skip to content

fix(build-ai): cover completeness knobs in the check version contract - #411

Merged
kstonekuan merged 3 commits into
Hebbian-Robotics:mainfrom
Sagar-024:feat/404-version-contract
Sep 5, 2026
Merged

fix(build-ai): cover completeness knobs in the check version contract#411
kstonekuan merged 3 commits into
Hebbian-Robotics:mainfrom
Sagar-024:feat/404-version-contract

Conversation

@Sagar-024

Copy link
Copy Markdown
Contributor

Closes #404.

_check_version stamped an identity that covered endpoint, model, response_format, temperature, and max_tokens, while two execution knobs that change which items produce results stayed invisible to it: max_retries on the OpenAI-compatible path (a transient 500 becomes a parsed prediction under retries and a failed run without them) and request_timeout_seconds on the hosted path (a slow-but-valid response is included or dropped by it). The issue's controlled result showed two executions differing only in these knobs sharing one version identity.

The distinction that decided the fix: sampling noise changes the value of an answer; retries and timeouts change which items produce answers at all. temperature, already in the contract, only perturbs a result that exists either way. A knob that decides whether an episode yields a prediction or a failed run has a stronger claim to identity, not a weaker one.

Both fields join the contract, symmetrically across the two branches, per the issue's DoD 4: the asymmetry is how this got missed. The migration-era comment on the OpenAI branch was clarified, since its shape-stability constraint applied to that migration alone and does not bind the new fields.

Tests were written red first. Identical configurations keep one version (DoD 3), executions differing only in max_retries or request_timeout_seconds produce different versions, and a symmetry test holds both branches to the same rule. The demonstration from the issue lands as the red run: two configs sharing build-ai-single-frame-v1-eabed1f2e257cc07 before the change, diverging after.

The named cost: adding contract fields re-mints every existing Build AI check version, so existing results become non-matching on the next run rather than being silently reused. That is the correct consequence, and DoD 3's stability test is the guard that keeps unchanged methodology from being invalidated.

Gate: ruff, format, ty clean; 1535 passed / 6 skipped.

Refs #404. Option 2 (attempts and timeouts as per-run measurements) stays unfiled per the issue direction; it is a separate enhancement when wanted.

_check_version stamped an identity that covered endpoint, model,
response_format, temperature, and max_tokens, while two execution knobs
that change which items produce results stayed invisible to it:
max_retries on the OpenAI-compatible path (a transient 500 becomes a
parsed prediction under retries and a failed run without them) and
request_timeout_seconds on the hosted path (a slow-but-valid response is
included or dropped by it). Two executions differing only in these knobs
shared one version identity, so corpora with systematically different
result sets compared as if they did not.

Both fields join the contract, symmetrically across the two branches
(Hebbian-Robotics#404 direction). The migration-era comment on the OpenAI branch was
clarified: its shape-stability constraint applied to that migration
only, not to the new fields.

Tests written red first: identical configurations keep one version
(DoD 3, green before and after), executions differing only in
max_retries or request_timeout_seconds produce different versions
(red before, green after), and a symmetry test holds both branches to
the same rule (DoD 4). Adding the fields re-mints every existing Build
AI check version by design; unchanged methodology keeps its identity.

Refs Hebbian-Robotics#404
kstonekuan and others added 2 commits September 5, 2026 00:52
The DoD 3 stability test registers one execution twice and asserts the
versions match, which cannot fail unless _check_version stops being a
function. The property it claims, that unchanged methodology keeps its
identity across changes to this module, needs a recorded value: adding a
junk field to the contract or renaming a key left the suite green.

One golden version per execution branch, with a comment saying that
editing them is the signal rather than the chore.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, merging. Both additions are held, symmetrically, which was the DoD item most likely to be half-done:

max_retries dropped                ..._changes_with_max_retries + ..._applies_the_rule_symmetrically
request_timeout_seconds dropped    ..._changes_with_request_timeout_seconds + ..._applies_the_rule_symmetrically
both dropped (main's contract)     3 failures

Rewriting the migration-era comment rather than leaving it was the right instinct. A comment saying "keep the shape stable" sitting directly above a change that deliberately breaks the shape is how the next person concludes the change was a mistake.

I pushed one fixup, c404093. The DoD 3 stability test cannot fail. It registers one execution object twice and asserts the two versions match, which is true of any pure function; it would keep passing if _check_version started returning a constant. The property you claimed in its docstring is stronger and better: that unchanged methodology keeps its identity across changes to this module. Only a recorded value holds that, and without one, both of these were invisible:

an unrelated field added to the contract   17 passed, nothing noticed
max_retries keyed under a different name   17 passed, nothing noticed

Either would silently re-mint every stored Build AI check version. So there is now one golden version per execution branch, with a comment saying that editing those strings is the signal rather than the chore: change them only alongside a deliberate contract change, and say in the PR why every existing result is being invalidated. Both mutations fail now, and so does dropping either field.

That is worth having beyond this PR. Three version-identity incidents landed today (#390's episode/v1 stamps, #403's success label, and this), and in every case the version moving was correct while nothing would have noticed if it had failed to move. This is the first test in the repo that holds one still.

Gate on the merged result: ruff check, ruff format --check, ty check clean, 1546 passed / 6 skipped.

Closes #404. Option 2 staying unfiled is right; file it when someone actually wants attempt counts in the record rather than on spec.

@kstonekuan
kstonekuan merged commit e8746dc into Hebbian-Robotics:main Sep 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build AI check versions exclude max_retries and request_timeout_seconds, so evaluations that can differ share one version identity

2 participants