ai-grok: start-frame `image` + reference inputs wrongly rejected on grok-imagine-video-1.5

[tombeckenham]

Summary

On grok-imagine-video-1.5, createVideoJob rejects a starting-frame image combined with reference inputs. xAI documents that combination as supported on 1.5 — and specifically as the way to pin the first frame. The guard looks correct for classic grok-imagine-video, but it is unconditional, so it also fires for 1.5.

Version: @tanstack/ai-grok@0.18.4 (current latest; verified against the published tarball, not just a local install).

What the adapter does

dist/esm/adapters/video.js:153:

if (startFrame && hasReference) throw new Error(`${this.name}: image-to-video and reference-to-video cannot be combined. Use a starting-frame image, or reference images / voices, not both.`);

hasReference is reference_images.length > 0 || reference_audios.length > 0, so this fires for any shot that has both a start frame and a single reference image.

What xAI documents

From Reference-to-Video:

image combined with reference_images, reference_audios, or last_frame is the matching first-frame pin.

and, distinguishing the two models:

Classic grok-imagine-video rejects last_frame and rejects combining image with reference inputs.

So the restriction is real for grok-imagine-video, and wrong for grok-imagine-video-1.5.

Why it matters

There is no way to pin a first frame and supply character/element references in the same request. The only workaround is to demote the intended first frame into reference_images[0] — which the SDK's own docs correctly note is not equivalent (GrokVideoProviderOptions.reference_images):

Reference images are addressed from the prompt text as <IMAGE_0>, <IMAGE_1>, … in request order, and do not lock the first frame.

For an image-to-video pipeline that renders a still and then animates it with character references attached, that turns a pinned opening frame into one influence among several, and the generated clip no longer starts from the frame that was rendered for it.

Repro

await generateVideo({
  adapter: createGrokVideo('grok-imagine-video-1.5', { apiKey }),
  prompt: [
    { type: 'text', content: '<IMAGE_0> lifts the bottle' },
    { type: 'image', source: { type: 'url', value: STILL }, metadata: { role: 'start_frame' } },
    { type: 'image', source: { type: 'url', value: SHEET }, metadata: { role: 'character' } },
  ],
  duration: 6,
});
// throws: grok: image-to-video and reference-to-video cannot be combined.

Suggested fix

The adapter already model-scopes the adjacent check two lines above (dist/esm/adapters/video.js:149) using isGrokVideoReferenceModel(model). The same scoping applies here — keep throwing for classic grok-imagine-video, allow the combination on 1.5:

if (startFrame && hasReference && !isGrokVideoReferenceModel(model)) throw new Error(...)

Two related bits, if useful:

• GrokVideoProviderOptions has no image field, so there is no typed way to send a first-frame pin alongside references. Adding image?: { url: string } would cover it (unmodelled modelOptions keys already reach the request body, so this works at runtime today — it just needs a cast or an intersection type).
• GrokVideoModelInputModalitiesByName currently reads 'grok-imagine-video-1.5': readonly ['image'], which is accurate, but the accompanying prose ("accept an optional image prompt part as the starting frame; image parts with metadata.role: 'reference' … become reference_images") reads as if the two are alternatives rather than combinable on 1.5.

Happy to open a PR if the scoping above is the direction you'd want.