fix(build-ai): cover completeness knobs in the check version contract - #411
Conversation
_check_version stamped an identity that covered endpoint, model, response_format, temperature, and max_tokens, while two execution knobs that change which items produce results stayed invisible to it: max_retries on the OpenAI-compatible path (a transient 500 becomes a parsed prediction under retries and a failed run without them) and request_timeout_seconds on the hosted path (a slow-but-valid response is included or dropped by it). Two executions differing only in these knobs shared one version identity, so corpora with systematically different result sets compared as if they did not. Both fields join the contract, symmetrically across the two branches (Hebbian-Robotics#404 direction). The migration-era comment on the OpenAI branch was clarified: its shape-stability constraint applied to that migration only, not to the new fields. Tests written red first: identical configurations keep one version (DoD 3, green before and after), executions differing only in max_retries or request_timeout_seconds produce different versions (red before, green after), and a symmetry test holds both branches to the same rule (DoD 4). Adding the fields re-mints every existing Build AI check version by design; unchanged methodology keeps its identity. Refs Hebbian-Robotics#404
The DoD 3 stability test registers one execution twice and asserts the versions match, which cannot fail unless _check_version stops being a function. The property it claims, that unchanged methodology keeps its identity across changes to this module, needs a recorded value: adding a junk field to the contract or renaming a key left the suite green. One golden version per execution branch, with a comment saying that editing them is the signal rather than the chore.
kstonekuan
left a comment
There was a problem hiding this comment.
LGTM, merging. Both additions are held, symmetrically, which was the DoD item most likely to be half-done:
max_retries dropped ..._changes_with_max_retries + ..._applies_the_rule_symmetrically
request_timeout_seconds dropped ..._changes_with_request_timeout_seconds + ..._applies_the_rule_symmetrically
both dropped (main's contract) 3 failures
Rewriting the migration-era comment rather than leaving it was the right instinct. A comment saying "keep the shape stable" sitting directly above a change that deliberately breaks the shape is how the next person concludes the change was a mistake.
I pushed one fixup, c404093. The DoD 3 stability test cannot fail. It registers one execution object twice and asserts the two versions match, which is true of any pure function; it would keep passing if _check_version started returning a constant. The property you claimed in its docstring is stronger and better: that unchanged methodology keeps its identity across changes to this module. Only a recorded value holds that, and without one, both of these were invisible:
an unrelated field added to the contract 17 passed, nothing noticed
max_retries keyed under a different name 17 passed, nothing noticed
Either would silently re-mint every stored Build AI check version. So there is now one golden version per execution branch, with a comment saying that editing those strings is the signal rather than the chore: change them only alongside a deliberate contract change, and say in the PR why every existing result is being invalidated. Both mutations fail now, and so does dropping either field.
That is worth having beyond this PR. Three version-identity incidents landed today (#390's episode/v1 stamps, #403's success label, and this), and in every case the version moving was correct while nothing would have noticed if it had failed to move. This is the first test in the repo that holds one still.
Gate on the merged result: ruff check, ruff format --check, ty check clean, 1546 passed / 6 skipped.
Closes #404. Option 2 staying unfiled is right; file it when someone actually wants attempt counts in the record rather than on spec.
Closes #404.
_check_versionstamped an identity that covered endpoint, model, response_format, temperature, and max_tokens, while two execution knobs that change which items produce results stayed invisible to it:max_retrieson the OpenAI-compatible path (a transient 500 becomes a parsed prediction under retries and a failed run without them) andrequest_timeout_secondson the hosted path (a slow-but-valid response is included or dropped by it). The issue's controlled result showed two executions differing only in these knobs sharing one version identity.The distinction that decided the fix: sampling noise changes the value of an answer; retries and timeouts change which items produce answers at all.
temperature, already in the contract, only perturbs a result that exists either way. A knob that decides whether an episode yields a prediction or a failed run has a stronger claim to identity, not a weaker one.Both fields join the contract, symmetrically across the two branches, per the issue's DoD 4: the asymmetry is how this got missed. The migration-era comment on the OpenAI branch was clarified, since its shape-stability constraint applied to that migration alone and does not bind the new fields.
Tests were written red first. Identical configurations keep one version (DoD 3), executions differing only in
max_retriesorrequest_timeout_secondsproduce different versions, and a symmetry test holds both branches to the same rule. The demonstration from the issue lands as the red run: two configs sharingbuild-ai-single-frame-v1-eabed1f2e257cc07before the change, diverging after.The named cost: adding contract fields re-mints every existing Build AI check version, so existing results become non-matching on the next run rather than being silently reused. That is the correct consequence, and DoD 3's stability test is the guard that keeps unchanged methodology from being invalidated.
Gate: ruff, format, ty clean; 1535 passed / 6 skipped.
Refs #404. Option 2 (attempts and timeouts as per-run measurements) stays unfiled per the issue direction; it is a separate enhancement when wanted.