Skip to content

fix(drift): move OpenAI tool-test models to gpt-5.6 before the Dec 11 shutdowns - #613

Open
jpr5 wants to merge 2 commits into
mainfrom
fix/drift-probes-off-retiring-openai-models
Open

jpr5 wants to merge 2 commits into
mainfrom
fix/drift-probes-off-retiring-openai-models

Conversation

@jpr5

@jpr5 jpr5 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Hold — merge on 2026-12-10 before the nightly drift run (OpenAI shutdown Dec 11)

Why

OpenAI corrected its shutdown notice. Only these models shut down on Dec 11, 2026: gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, o3, o3-pro. gpt-5.4-mini and the other gpt-5.4 / gpt-5.5 ids are NOT retiring.

OpenAI's documented replacements: gpt-5 → gpt-5.6-sol, gpt-5-mini → gpt-5.6-terra.

What changed

Scope: only the tool-test models move off gpt-5 / gpt-5-mini. Mock-side model names (model registry, fixtures, unit tests) are unchanged.

File Old New
src/__tests__/drift/openai-responses.drift.ts:978 (GPT5_TOOL_MODELS, namespaced/custom tool legs) gpt-5-mini, gpt-5 gpt-5.6-terra, gpt-5.6-sol

The tool-leg error text at line 1033 now says "gpt-5.6 access".

The K9 probe (OPENAI_K9_PROBE_MODEL in src/__tests__/drift/probe-models.ts) stays on gpt-5.4-mini, as on main. An earlier revision of this PR moved it to gpt-5.6-terra; commit 47d8a87 reverts that after the corrected notice.

Proof

RED (previous head 1360313, before the narrowing)

git diff origin/main...HEAD -- src/__tests__/drift/probe-models.ts:

-export const OPENAI_K9_PROBE_MODEL = "gpt-5.4-mini";
+export const OPENAI_K9_PROBE_MODEL = "gpt-5.6-terra";

K9-related hunks in the whole PR diff: only that one (probe-models.ts:46). No CHANGELOG, doc or test line referred to it.

GREEN (head 47d8a87)

git diff origin/main...HEAD --stat:

 src/__tests__/drift/openai-responses.drift.ts | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
-const GPT5_TOOL_MODELS = ["gpt-5-mini", "gpt-5"];
+const GPT5_TOOL_MODELS = ["gpt-5.6-terra", "gpt-5.6-sol"];
...
-      `the ids were retired or renamed (update GPT5_TOOL_MODELS) or the key lost gpt-5 access. ` +
+      `the ids were retired or renamed (update GPT5_TOOL_MODELS) or the key lost gpt-5.6 access. ` +

probe-models.ts:46: export const OPENAI_K9_PROBE_MODEL = "gpt-5.4-mini"; (identical to main).

  • Drift unit tests (vitest run src/__tests__/drift/): 20 files, 254 passed, 6 skipped.
  • eslint and prettier --check on the changed files: clean. pnpm typecheck: clean.
  • Live: drift-live-pr on the new head (see checks).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jay9vBCbXhVHCDgyRBABVx

@pkg-pr-new

pkg-pr-new Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@613

commit: 47d8a87

jpr5 added a commit that referenced this pull request Nov 11, 2026
## What

Gemini's live /models started listing `gemini-4-carbon` on Nov 10. It
was absent from the scheduled Drift Tests run 42766538763 (Nov 9, 10:08
PM PST) and present in drift-live-pr runs since 12:53 PM PST (e.g. #613,
run 42941450176). It shows up as a base-present critical: `Unclassified
model family "gemini-4-carbon"`. Without a fix, tonight's Drift Tests
and Fix Drift runs go red.

## Decision: EXCLUDE

Gemini 4 Carbon is access-restricted: it is offered to cyber defenders
in the Fairwind Program first, then trusted partners, with no GA date.
aimock does not mock pre-GA surfaces, so this follows the
`gemini-3.7-flash-video-understanding-eap` precedent (excluded for
access status, not capability). It is a text chat/reasoning model, so at
GA flip it to include, like `gemini-4-argon`.

## Changes

- `excludeFamilies.gemini` gets `gemini-4-carbon` (`model-registry.ts`).
- Re-pin the `excludeFamilies.gemini` freeze hash in
`logic-pin.test.ts`. The only membership change is this addition.
- Resolved note `drift-proposals/gemini-gemini-4-carbon-new-family.md`
(Decision: EXCLUDE), as #603 did.

## Red / green (local)

| Check | main | this branch |
|---|---|---|
| `assertNoUnclassifiedFamilies` (gemini) on the ids from #613's
`drift-report-head.json` plus known Gemini ids | RED: `[critical]
Unclassified model family "gemini-4-carbon"` | GREEN: passes |
| `logic-pin.test.ts` after the registry edit | RED (hash) | GREEN
(re-pinned) |

Format, lint, typecheck and build are clean. Drift + drift-sync tests:
227/227. Full unit suite: 0 new failures. 7 CLI/doc-sample files need
`dist/` and passed once rebuilt (238/238).
OpenAI corrected its shutdown notice: only GPT-5, GPT-5 mini, GPT-5 nano,
GPT-5 pro, o3 and o3-pro shut down on Dec 11, 2026. gpt-5.4-mini is not
retiring, so the K9 probe move to gpt-5.6-terra is not needed. Restore
OPENAI_K9_PROBE_MODEL to its main value. The GPT5_TOOL_MODELS swap stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jay9vBCbXhVHCDgyRBABVx
@jpr5 jpr5 changed the title fix(drift): move OpenAI drift probes to gpt-5.6 before the Dec 11 shutdowns fix(drift): move OpenAI tool-test models to gpt-5.6 before the Dec 11 shutdowns Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant