diff --git a/README.md b/README.md index 1a397e25..47151922 100644 --- a/README.md +++ b/README.md @@ -376,9 +376,9 @@ See the [Cosmos 3 Diffusers documentation](https://huggingface.co/docs/diffusers Use vLLM-Omni for Generator production inference behind an OpenAI-compatible API. This integration loads the full Cosmos 3 checkpoint, including the Qwen3-VL-based reasoner path and the diffusion generation path. For understanding-only tasks that return text, use [Reasoner with vLLM](#reasoner-with-vllm) instead, which loads only the reasoner. -> **Compatibility status:** Cosmos 3 Generator support has landed in [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) `main`: text-to-image, text-to-video, and image-to-video ([#3454](https://github.com/vllm-project/vllm-omni/pull/3454)) and video-with-sound ([#4073](https://github.com/vllm-project/vllm-omni/pull/4073)) are merged; action (policy / forward-dynamics) is in review ([#4102](https://github.com/vllm-project/vllm-omni/pull/4102)) and video-to-video is planned. The `vllm/vllm-omni:cosmos3` Docker image remains the easiest all-in-one build. For current setup and per-modality usage, see the maintained recipes: [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). +> **Compatibility status:** Cosmos 3 Generator support is available in [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) `main` for text-to-image, text-to-video, image-to-video, video-to-video, transfer-control video-to-video, video-with-sound, and action generation. For current setup and per-modality usage, see the maintained recipes: [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). -Start the server from the Docker image (all modalities). Mount any directory that contains local media or action files you want the server to read. +Start the server from the `vllm/vllm-omni:cosmos3` Docker image. Mount any directory that contains local media or action files you want the server to read. The command below runs from `/workspace`, so repo-local paths such as `cookbooks/...` resolve inside the container. ```shell docker run --runtime nvidia --gpus all \ @@ -386,6 +386,7 @@ docker run --runtime nvidia --gpus all \ -v "$(pwd):/workspace" \ -p 8000:8000 \ --ipc=host \ + -w /workspace \ vllm/vllm-omni:cosmos3 \ vllm serve nvidia/Cosmos3-Nano \ --omni \ @@ -411,7 +412,7 @@ Additional parallelism options: When combining parallelism options, ensure the server has enough GPUs for the product of the enabled degrees (`tensor_parallel_size` × `cfg_parallel_size` × `ulysses_degree`). -To install vLLM-Omni from `main` instead of using the Docker image (text-to-image, text-to-video, image-to-video, and video-with-sound are merged there; see the [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md) recipes for per-modality usage), create a venv and install, choosing the CUDA build that matches your driver: +To install vLLM-Omni from `main` instead of using the Docker image, create a venv and install, choosing the CUDA build that matches your driver. This path uses the same request formats as the Docker image; see the [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md) recipes for per-modality usage: ```shell uv venv --python 3.13 --seed --managed-python @@ -434,7 +435,8 @@ Vision endpoints: | Text to video | `POST /v1/videos/sync` | Blocks and returns the MP4 bytes directly | | Image to video | `POST /v1/videos/sync` | Upload the conditioning image with `input_reference` | | Video to video | `POST /v1/videos/sync` | Upload a source video and choose which frames stay as clean conditioning | -| Video with sound | `POST /v1/videos/sync` | Add `generate_sound=true` to produce a soundtrack alongside the video | +| Transfer video to video | `POST /v1/videos/sync` | Pass one or more transfer hints such as `edge`, `blur`, `depth`, `seg`, or `wsm` in `extra_params` | +| Video with sound | `POST /v1/videos/sync` | Add `generate_sound=true` to supported text-to-video or image-to-video requests | Action modes use Cosmos 3 as a world model: they condition on an embodiment (`domain_name`) and exchange video and action sequences. Policy and inverse dynamics return a predicted action chunk, so send those through the asynchronous `POST /v1/videos` job and read the action data from the completed result; forward dynamics returns only video and can use synchronous `POST /v1/videos/sync`. @@ -463,6 +465,43 @@ curl -sS -X POST http://localhost:8000/v1/videos/sync \ -o cosmos3_t2v_output.mp4 ``` +Example video-to-video request: + +```shell +curl -sS -X POST http://localhost:8000/v1/videos/sync \ + -H "Accept: video/mp4" \ + --form-string "prompt=Continue the same driving scene with smooth natural motion." \ + --form-string "negative_prompt=blurry, distorted, low quality, jittery, deformed" \ + --form-string "size=832x480" \ + --form-string "num_frames=61" \ + --form-string "fps=10" \ + --form-string "num_inference_steps=35" \ + --form-string "guidance_scale=6.0" \ + --form-string "flow_shift=10.0" \ + --form-string "seed=2222" \ + --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true,"condition_frame_indexes_vision":[0,1],"condition_video_keep":"first"}' \ + -F "input_reference=@cookbooks/cosmos3/generator/action/assets/videos/av_0.mp4;type=video/mp4" \ + -o cosmos3_v2v_output.mp4 +``` + +Example transfer-control request: + +```shell +curl -sS -X POST http://localhost:8000/v1/videos/sync \ + -H "Accept: video/mp4" \ + --form-string "prompt=Generate a realistic scene following the provided depth control video." \ + --form-string "negative_prompt=blurry, distorted, low quality" \ + --form-string "size=1280x720" \ + --form-string "num_frames=121" \ + --form-string "fps=30" \ + --form-string "num_inference_steps=50" \ + --form-string "guidance_scale=3.0" \ + --form-string "flow_shift=10.0" \ + --form-string "seed=2026" \ + --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true,"depth":{"control_path":"cookbooks/cosmos3/generator/transfer/assets/depth/control_depth.mp4"},"resolution":"720","control_guidance":1.5,"num_video_frames_per_chunk":121,"max_frames":121}' \ + -o cosmos3_transfer_depth.mp4 +``` + Use `--form-string` for text fields (`prompt`, `negative_prompt`, `extra_params`) rather than `-F`: with `-F`, `curl` treats `;` as a content-type separator and silently truncates any value that contains one. Common request fields (the image endpoint follows the [Image Generation API](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/image_generation_api/), and the video endpoints follow the [Videos API](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/videos_api/#request-parameters)): @@ -479,7 +518,8 @@ Common request fields (the image endpoint follows the [Image Generation API](htt | `seed` | Reproducibility seed | | `max_sequence_length` | Maximum number of prompt tokens kept for conditioning (Cosmos 3 default `512`); longer prompts are truncated with a warning, shorter ones padded | | `input_reference` | Uploaded image or video for image-to-video, video-to-video, and action requests | -| `extra_params` | JSON-encoded Cosmos 3-specific options: action settings (`action_mode`, `domain_name`, `raw_action_dim`, `action_chunk_size`, `action_path`), video-to-video conditioning (`condition_frame_indexes_vision`, `condition_video_keep`), prompt-template toggles (`use_resolution_template`, `use_duration_template`), and the per-request `guardrails` toggle | +| `video_reference` | JSON-safe video reference for video-to-video requests, such as `{"video_url":"https://..."}`; do not combine with `input_reference` or `image_reference` | +| `extra_params` | JSON-encoded Cosmos 3-specific options: action settings (`action_mode`, `domain_name`, `raw_action_dim`, `action_chunk_size`, `action_path`), video-to-video conditioning (`condition_frame_indexes_vision`, `condition_video_keep`), transfer hints (`edge`, `blur`, `depth`, `seg`, `wsm`) and transfer bucket `resolution`, prompt-template toggles (`use_resolution_template`, `use_duration_template`), and the per-request `guardrails` toggle | | `extra_args` | JSON object for Cosmos 3-specific image-endpoint options such as `use_resolution_template` | Disabling guardrails: Cosmos 3 ships safety guardrails that screen prompts and blur faces in generated output. Disable them per request by adding `guardrails: false` to `extra_params`: @@ -1079,7 +1119,7 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to | --- | --- | --- | --- | --- | | Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | | Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | -| Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | +| Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, text-to-video, image-to-video, and video-to-video, with supported sound modes, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | | Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | | Generator (audiovisual) with SGLang | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | | Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | @@ -1089,6 +1129,7 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to | Inverse dynamics with vLLM-Omni | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) | | Inverse dynamics with SGLang | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) | | Transfer with Cosmos Framework | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | +| Transfer with vLLM-Omni | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) | | Reasoner with Cosmos Framework | Reasoner | Text and image reasoning: detailed captioning, robot task planning, 2D grounding, describe-anything, and action-trajectory prompts, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) | | Reasoner with vLLM | Reasoner | Image and video reasoning: captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT, driving scenes, physical-plausibility, and situation understanding, against an OpenAI-compatible vLLM server (Cosmos3-Super on 4 GPUs by default; switch to Nano per the cookbook README). | [Notebook](cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) | | Reasoner with NIM | Reasoner | The same image and video reasoning examples as the vLLM notebook, run against the prebuilt, OpenAI-compatible [Cosmos 3 Reasoner NIM](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/cosmos3-reasoner) container; local media is sent as base64 data URIs. | [Notebook](cookbooks/cosmos3/reasoner/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_nim.ipynb) | diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 05a88120..ef282157 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -10,7 +10,7 @@ backend you want to run and follow that one section. | [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) | | [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner | | [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner | -| [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action) | Generator (Audiovisual, Action) | +| [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action/transfer) | Generator (Audiovisual, Action, **Transfer**) | | [Reasoner NIM](#reasoner-nim) | Prebuilt OpenAI-compatible reasoning server (image/video understanding); no venv | Reasoner | | [Generator NIM](#generator-nim) | Prebuilt NGC container serving the Cosmos3 Generator for Text-to-Video and Image-to-Video inference | Generator (Audiovisual) | @@ -289,7 +289,7 @@ graphs compile. ## vLLM-Omni -OpenAI-compatible **generation** server (image/video/audio/action) for the +OpenAI-compatible **generation** server (image/video/audio/action/transfer) for the Generator cookbooks. Cosmos3 checkpoints can exceed the default server init timeout — always pass @@ -325,8 +325,8 @@ the repository root README. See also the ### Option 1: Docker (recommended) -The prebuilt image `vllm/vllm-omni:cosmos3` supports every Generator modality -(including action). Pull once: +The released image `vllm/vllm-omni:cosmos3` supports the Generator modalities, +including transfer controls. Pull once: ```bash docker pull vllm/vllm-omni:cosmos3 @@ -342,7 +342,8 @@ export COSMOS3_HOST_PORT="${COSMOS3_HOST_PORT:-8000}" The container listens on port 8000; `-p "${COSMOS3_HOST_PORT}:8000"` publishes it on the host. Generator notebooks often use `COSMOS3_HOST_PORT=8001` so port 8000 -stays free for a Reasoner server. +stays free for a Reasoner server. The Docker commands run from `/workspace`, so +repo-local paths such as `cookbooks/...` resolve inside the container. **Cosmos3-Nano** (single GPU): @@ -352,6 +353,7 @@ docker run --runtime nvidia --gpus '"device=0"' \ -v "${HF_HOME}:/root/.cache/huggingface" \ -v "${COSMOS3_WORKDIR}:/workspace" \ -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \ + -w /workspace \ vllm/vllm-omni:cosmos3 \ vllm serve nvidia/Cosmos3-Nano \ --omni \ @@ -368,6 +370,7 @@ docker run --runtime nvidia --gpus all \ -v "${HF_HOME}:/root/.cache/huggingface" \ -v "${COSMOS3_WORKDIR}:/workspace" \ -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \ + -w /workspace \ vllm/vllm-omni:cosmos3 \ vllm serve nvidia/Cosmos3-Super \ --omni \ @@ -437,11 +440,10 @@ filesystem should be readable. vLLM-Omni prints `Application startup complete.` when the API is ready. -### Option 2: Native venv (limited modalities) +### Option 2: Native venv -To install from the upstreaming PR branch instead of Docker (text-to-image, -text-to-video, and image-to-video only — not action or sound yet), create a venv -and pick the CUDA build that matches your driver (see +To install from `main` instead of Docker, create a venv and pick the CUDA build +that matches your driver (see [CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)): ```bash @@ -450,11 +452,11 @@ source .venv/bin/activate # CUDA 13 driver: uv pip install --torch-backend=cu130 \ - "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@refs/pull/3454/head" + "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main" # CUDA 12.x driver: # uv pip install --torch-backend=cu128 \ -# "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@refs/pull/3454/head" +# "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main" ``` Run the same `vllm serve` arguments as in the Docker commands above, directly on diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index da5b6217..453e8022 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -1,6 +1,6 @@ # Cosmos3 Generator Audiovisual Examples -Generate images and video (with optional audio) from text or image prompts with +Generate images and video (with optional audio) from text, image or video prompts with `Cosmos3-Nano`, `Cosmos3-Super`, `Cosmos3-Edge`, and the published four-step distilled Cosmos3-Super students across Cosmos Framework, Diffusers, vLLM-Omni, and NIM backends. Sample prompts live under [`assets/`](./assets). @@ -197,12 +197,49 @@ Path("/tmp/cosmos3_t2v.mp4").write_bytes(response.content) For image-to-video, post to the same endpoint with an image under `files={"input_reference": ...}`. For audio, add `"generate_sound": "true"`. +For video-to-video, upload a source video under `input_reference` and choose the +clean conditioning frames through `extra_params`: + +```python +from pathlib import Path + +source_video = Path("../action/assets/videos/av_0.mp4").resolve() +with source_video.open("rb") as video_file: + response = requests.post( + "http://localhost:8000/v1/videos/sync", + data={ + "prompt": "Continue the same driving scene with smooth natural motion.", + "negative_prompt": "blurry, distorted, low quality, jittery, deformed", + "size": "832x480", + "num_frames": "61", + "fps": "10", + "num_inference_steps": "35", + "guidance_scale": "6.0", + "flow_shift": "10.0", + "seed": "2222", + "extra_params": json.dumps( + { + "use_resolution_template": False, + "use_duration_template": False, + "guardrails": True, + "condition_frame_indexes_vision": [0, 1], + "condition_video_keep": "first", + } + ), + }, + files={"input_reference": (source_video.name, video_file, "video/mp4")}, + headers={"Accept": "video/mp4"}, + ) +response.raise_for_status() +Path("/tmp/cosmos3_v2v.mp4").write_bytes(response.content) +``` ### Notebook walkthrough [`run_with_vllm_omni.ipynb`](./run_with_vllm_omni.ipynb) is the full tutorial for the vLLM-Omni backend: it walks through text-to-image, text-to-video, and -image-to-video requests with audio on or off. Server launch options (Nano and +image-to-video requests with audio on or off plus standard video-to-video +requests. Server launch options (Nano and Super, tensor parallelism, layerwise offload, and CFG-parallel variants) live in the [shared environment setup guide](../../README.md#vllm-omni). diff --git a/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb index 57a30eb3..1de593b4 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb @@ -1,30 +1,29 @@ { "cells": [ { - "cell_type": "markdown", - "id": "license-header", "metadata": {}, + "cell_type": "markdown", "source": [ "" - ] + ], + "id": "81a746aa386ead8a" }, { - "cell_type": "markdown", - "id": "d88fe9a8", "metadata": {}, + "cell_type": "markdown", "source": [ "# Cosmos3 Generator Audiovisual with vLLM Omni\n", "\n", "This notebook calls already-running vLLM Omni Cosmos3 servers with direct `curl` requests from Python.\n", "\n", "The examples are split into Cosmos3-Nano and Cosmos3-Super sections. Each section is self-contained, so you can run just one. Each section targets the matching model endpoint.\n" - ] + ], + "id": "51ede20282b6221c" }, { - "cell_type": "markdown", - "id": "49df4e61", "metadata": {}, + "cell_type": "markdown", "source": [ "## 1. Prerequisites\n", "\n", @@ -37,12 +36,12 @@ "export COSMOS3_VLLM_NANO_BASE_URL=http://localhost:8000\n", "export COSMOS3_VLLM_SUPER_BASE_URL=http://localhost:8000\n", "```\n" - ] + ], + "id": "96cf571103f4b142" }, { - "cell_type": "markdown", - "id": "26776c50", "metadata": {}, + "cell_type": "markdown", "source": [ "## 2. Start the Server\n", "\n", @@ -88,48 +87,6 @@ " --init-timeout 1800\n", "```\n", "\n", - "### Docker Image: Cosmos3-Super-Text2Image-4Step, Cosmos3-Super-Image2Video-4Step\n", - "\n", - "`Cosmos3-Super-Text2Image-4Step` and `Cosmos3-Super-Image2Video-4Step` are distilled models which require fewer inference steps for Text2Image and Image2Video tasks respectively. \n", - "\n", - "#### Cosmos3-Super-Text2Image-4Step\n", - "\n", - "```bash\n", - "docker run --runtime nvidia --gpus all \\\n", - " -v ~/.cache/huggingface:/root/.cache/huggingface \\\n", - " -v \"$(pwd):/workspace\" \\\n", - " -p 8000:8000 \\\n", - " --ipc=host \\\n", - " vllm/vllm-omni:cosmos3 \\\n", - " vllm serve nvidia/Cosmos3-Super-Text2Image-4Step \\\n", - " --omni \\\n", - " --model-class-name Cosmos3OmniDiffusersPipeline \\\n", - " --allowed-local-media-path / \\\n", - " --tensor-parallel-size 4 \\\n", - " --enable-layerwise-offload \\\n", - " --port 8000 \\\n", - " --init-timeout 1800\n", - "```\n", - "\n", - "#### Cosmos3-Super-Image2Video-4Step\n", - "\n", - "```bash\n", - "docker run --runtime nvidia --gpus all \\\n", - " -v ~/.cache/huggingface:/root/.cache/huggingface \\\n", - " -v \"$(pwd):/workspace\" \\\n", - " -p 8000:8000 \\\n", - " --ipc=host \\\n", - " vllm/vllm-omni:cosmos3 \\\n", - " vllm serve nvidia/Cosmos3-Super-Image2Video-4Step \\\n", - " --omni \\\n", - " --model-class-name Cosmos3OmniDiffusersPipeline \\\n", - " --allowed-local-media-path / \\\n", - " --tensor-parallel-size 4 \\\n", - " --enable-layerwise-offload \\\n", - " --port 8000 \\\n", - " --init-timeout 1800\n", - "```\n", - "\n", "### CFG Parallel\n", "\n", "Use `--cfg-parallel-size 2` to run the positive and negative CFG branches in parallel on two GPUs:\n", @@ -145,24 +102,24 @@ "```\n", "\n", "For Cosmos3, set CFG strength with the request-level `guidance_scale` field. Do not use `true_cfg_scale` for CFG Parallel with these Cosmos3 examples.\n" - ] + ], + "id": "70eb5b70b2ec9e67" }, { - "cell_type": "markdown", - "id": "4412f2f9", "metadata": {}, + "cell_type": "markdown", "source": [ "## 3. Configure Paths and Endpoints\n", "\n", "This setup cell only configures repo/output paths and vLLM endpoint settings.\n" - ] + ], + "id": "22f36df14abd630" }, { - "cell_type": "code", - "execution_count": null, - "id": "23f04a90", "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "from pathlib import Path\n", "import os\n", @@ -184,8 +141,6 @@ "VLLM_ENDPOINTS = {\n", " \"Cosmos3-Nano\": os.environ.get(\"COSMOS3_VLLM_NANO_BASE_URL\", DEFAULT_VLLM_BASE_URL),\n", " \"Cosmos3-Super\": os.environ.get(\"COSMOS3_VLLM_SUPER_BASE_URL\", DEFAULT_VLLM_BASE_URL),\n", - " \"Cosmos3-Super-Text2Image-4Step\": os.environ.get(\"COSMOS3_VLLM_SUPER_BASE_URL\", DEFAULT_VLLM_BASE_URL),\n", - " \"Cosmos3-Super-Image2Video-4Step\": os.environ.get(\"COSMOS3_VLLM_SUPER_BASE_URL\", DEFAULT_VLLM_BASE_URL),\n", "}\n", "\n", "os.environ[\"COSMOS3_AUDIOVISUAL_OUTPUT_ROOT\"] = str(COSMOS3_AUDIOVISUAL_OUTPUT_ROOT)\n", @@ -195,22 +150,20 @@ "print(\"COSMOS3_AUDIOVISUAL_OUTPUT_ROOT:\", COSMOS3_AUDIOVISUAL_OUTPUT_ROOT)\n", "for model, endpoint in VLLM_ENDPOINTS.items():\n", " print(f\"{model} endpoint: {endpoint}\")\n" - ] + ], + "id": "72905055060b35b9" }, { - "cell_type": "markdown", - "id": "73369e7f", "metadata": {}, - "source": [ - "## 4. Verify Endpoint Configuration\n" - ] + "cell_type": "markdown", + "source": "## 4. Verify Endpoint Configuration\n", + "id": "4bc75c7db74226ee" }, { - "cell_type": "code", - "execution_count": null, - "id": "1c50e183", "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "from urllib.parse import urlparse\n", "\n", @@ -225,22 +178,20 @@ " print(\" videos sync:\", f\"{api_root}/videos/sync\")\n", " print(\" scheme:\", parsed.scheme)\n", " print(\" host:\", parsed.netloc)\n" - ] + ], + "id": "263e35494845d7d3" }, { - "cell_type": "markdown", - "id": "c1d161a6", "metadata": {}, - "source": [ - "## 5. Preview Available Inputs\n" - ] + "cell_type": "markdown", + "source": "## 5. Preview Available Inputs\n", + "id": "6c9d37fe3fab7f69" }, { - "cell_type": "code", - "execution_count": null, - "id": "973ea472", "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "from pathlib import Path\n", "import json\n", @@ -269,22 +220,20 @@ " if image_path.suffix.lower() in {\".jpg\", \".jpeg\", \".png\", \".webp\", \".bmp\"}:\n", " print(f\" {image_path.name}\")\n", " display(Image(filename=str(image_path), width=420))\n" - ] + ], + "id": "ce1587c517e8f944" }, { - "cell_type": "markdown", - "id": "cfa34351", "metadata": {}, - "source": [ - "## 6. Define Asset Sets, Payload Helpers, Request Helpers, and Viewer Helpers\n" - ] + "cell_type": "markdown", + "source": "## 6. Define Asset Sets, Payload Helpers, Request Helpers, and Viewer Helpers\n", + "id": "b113609c2a6433d3" }, { - "cell_type": "code", - "execution_count": null, - "id": "af5dd1c8", "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "import json\n", "import os\n", @@ -358,21 +307,9 @@ " \"image\": \"assets/images/image2video/car_driving.jpg\",\n", " \"enable_sound\": False,\n", " },\n", - " \"t2i_distil\": {\n", - " \"model\": \"Cosmos3-Super-Text2Image-4Step\",\n", - " \"mode\": \"text2image\",\n", - " \"prompt\": \"assets/prompts/text2image/robot_draping.json\",\n", - " \"enable_sound\": False,\n", - " },\n", - " \"i2v_distil\": {\n", - " \"model\": \"Cosmos3-Super-Image2Video-4Step\",\n", - " \"mode\": \"image2video\",\n", - " \"prompt\": \"assets/prompts/image2video/car_driving.json\",\n", - " \"image\": \"assets/images/image2video/car_driving.jpg\",\n", - " \"enable_sound\": False,\n", - " },\n", "}\n", "\n", + "\n", "def asset_path(relative_path: str) -> Path:\n", " path = COSMOS3_AUDIOVISUAL_ROOT / relative_path\n", " if not path.exists():\n", @@ -635,620 +572,577 @@ " for src in images:\n", " print(f\"source: {src} ({src.stat().st_size // 1024} KB)\")\n", " display(Image(filename=str(src), width=720))\n" - ] + ], + "id": "95ed96b7ea360dfc" }, { - "cell_type": "markdown", - "id": "9ffaab0e", "metadata": {}, - "source": [ - "Run each use case top-to-bottom: create the JSON payload, run inference, then view the generated media. The examples are grouped by model size below. The Cosmos3-Nano and Cosmos3-Super sections are independent, so you can run just one.\n" - ] + "cell_type": "markdown", + "source": "Run each use case top-to-bottom: create the JSON payload, run inference, then view the generated media. The examples are grouped by model size below. The Cosmos3-Nano and Cosmos3-Super sections are independent, so you can run just one.\n", + "id": "e59eaf04cb201ef5" }, { - "cell_type": "markdown", - "id": "e7d284db", "metadata": {}, + "cell_type": "markdown", "source": [ "# Cosmos3-Nano Examples\n", "\n", "Use cases for the `Cosmos3-Nano` model. This section is self-contained; you can run it without the Cosmos3-Super section below.\n" - ] + ], + "id": "68060c05ccecca43" }, { - "cell_type": "markdown", - "id": "27bf6dce", "metadata": {}, + "cell_type": "markdown", "source": [ "## Nano: Text to Image\n", "\n", "Nano text-to-image generation using a structured JSON prompt.\n", "\n", "### Create Payload\n" - ] + ], + "id": "bdd2690484e75144" }, { - "cell_type": "code", - "execution_count": null, - "id": "383f6351", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2i_payload, t2i_output, t2i_model = create_payload(\"t2i\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "t2i_payload, t2i_output, t2i_model = create_payload(\"t2i\", backend=\"vllm\")\n", + "id": "557f382916063cfa" }, { - "cell_type": "markdown", - "id": "d3c59500", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "19835d33c83494a9" }, { - "cell_type": "code", - "execution_count": null, - "id": "97b4b312", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2i_payload, t2i_output, model=\"Cosmos3-Nano\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(t2i_payload, t2i_output, model=\"Cosmos3-Nano\")\n", + "id": "74a5d0f9de5e1466" }, { - "cell_type": "markdown", - "id": "c027b956", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "968fa8908f5fbaf0" }, { - "cell_type": "code", - "execution_count": null, - "id": "8fe6e3ed", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2i_output)\n" - ] + "execution_count": null, + "source": "view_run(t2i_output)\n", + "id": "d63b23fcfcbe1c5e" }, { - "cell_type": "markdown", - "id": "0e7d3092", "metadata": {}, + "cell_type": "markdown", "source": [ "## Nano: Text to Video Without Audio\n", "\n", "Nano text-to-video generation with audio disabled.\n", "\n", "### Create Payload\n" - ] + ], + "id": "5ee6f103e07cfdae" }, { - "cell_type": "code", - "execution_count": null, - "id": "b2e85424", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2v_nano_noaudio_payload, t2v_nano_noaudio_output, t2v_nano_noaudio_model = create_payload(\"t2v_nano_noaudio\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "t2v_nano_noaudio_payload, t2v_nano_noaudio_output, t2v_nano_noaudio_model = create_payload(\"t2v_nano_noaudio\", backend=\"vllm\")\n", + "id": "c20d1dff79535d2b" }, { - "cell_type": "markdown", - "id": "e4fa0cdd", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "3ffbaadc40841b75" }, { - "cell_type": "code", - "execution_count": null, - "id": "9db2a1a5", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2v_nano_noaudio_payload, t2v_nano_noaudio_output, model=\"Cosmos3-Nano\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(t2v_nano_noaudio_payload, t2v_nano_noaudio_output, model=\"Cosmos3-Nano\")\n", + "id": "e38219af93f23b7f" }, { - "cell_type": "markdown", - "id": "1bc8a331", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "e5c5f410cc82cc92" }, { - "cell_type": "code", - "execution_count": null, - "id": "87917755", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2v_nano_noaudio_output)\n" - ] + "execution_count": null, + "source": "view_run(t2v_nano_noaudio_output)\n", + "id": "6caf0aa68f3803e4" }, { - "cell_type": "markdown", - "id": "816e9b09", "metadata": {}, + "cell_type": "markdown", "source": [ "## Nano: Text to Video with Audio\n", "\n", "Nano text-to-video generation with generated audio.\n", "\n", "### Create Payload\n" - ] + ], + "id": "22fb537d2410274b" }, { - "cell_type": "code", - "execution_count": null, - "id": "e8450feb", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2vs_payload, t2vs_output, t2vs_model = create_payload(\"t2vs\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "t2vs_payload, t2vs_output, t2vs_model = create_payload(\"t2vs\", backend=\"vllm\")\n", + "id": "7d01206dd5fc3cb8" }, { - "cell_type": "markdown", - "id": "b6bd6ec8", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "ae1906f960eea7f0" }, { - "cell_type": "code", - "execution_count": null, - "id": "685d18d6", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2vs_payload, t2vs_output, model=\"Cosmos3-Nano\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(t2vs_payload, t2vs_output, model=\"Cosmos3-Nano\")\n", + "id": "7e91b59ef4ce617" }, { - "cell_type": "markdown", - "id": "93743159", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "a7c3d3d3fe31922a" }, { - "cell_type": "code", - "execution_count": null, - "id": "a4bc049a", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2vs_output)\n" - ] + "execution_count": null, + "source": "view_run(t2vs_output)\n", + "id": "cb1dc52eac2896da" }, { - "cell_type": "markdown", - "id": "1cf05248", "metadata": {}, + "cell_type": "markdown", "source": [ "## Nano: Image to Video Without Audio\n", "\n", "Nano image-to-video generation using its paired image asset, with audio disabled.\n", "\n", "### Create Payload\n" - ] + ], + "id": "37b08ec7068b6b85" }, { - "cell_type": "code", - "execution_count": null, - "id": "91e28e16", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "i2v_nano_noaudio_payload, i2v_nano_noaudio_output, i2v_nano_noaudio_model = create_payload(\"i2v_nano_noaudio\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "i2v_nano_noaudio_payload, i2v_nano_noaudio_output, i2v_nano_noaudio_model = create_payload(\"i2v_nano_noaudio\", backend=\"vllm\")\n", + "id": "f67bdf1bd3581188" }, { - "cell_type": "markdown", - "id": "ef42f84d", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "73549b1908515068" }, { - "cell_type": "code", - "execution_count": null, - "id": "4ec848e8", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(i2v_nano_noaudio_payload, i2v_nano_noaudio_output, model=\"Cosmos3-Nano\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(i2v_nano_noaudio_payload, i2v_nano_noaudio_output, model=\"Cosmos3-Nano\")\n", + "id": "974d48a70f689884" }, { - "cell_type": "markdown", - "id": "dea78891", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "d795eb22385129ee" }, { - "cell_type": "code", - "execution_count": null, - "id": "ae7a16ed", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(i2v_nano_noaudio_output)\n" - ] + "execution_count": null, + "source": "view_run(i2v_nano_noaudio_output)\n", + "id": "f824c7fcff5ead3f" }, { - "cell_type": "markdown", - "id": "1860e911", "metadata": {}, + "cell_type": "markdown", "source": [ "## Nano: Image to Video with Audio\n", "\n", "Nano image-to-video generation using its paired image asset and generated audio.\n", "\n", "### Create Payload\n" - ] + ], + "id": "4257269bde6f34af" }, { - "cell_type": "code", - "execution_count": null, - "id": "37c40d8e", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "i2vs_payload, i2vs_output, i2vs_model = create_payload(\"i2vs\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "i2vs_payload, i2vs_output, i2vs_model = create_payload(\"i2vs\", backend=\"vllm\")\n", + "id": "795d96c3031211ad" }, { - "cell_type": "markdown", - "id": "637e5e87", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "163d537386d6264c" }, { - "cell_type": "code", - "execution_count": null, - "id": "b4a4c308", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(i2vs_payload, i2vs_output, model=\"Cosmos3-Nano\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(i2vs_payload, i2vs_output, model=\"Cosmos3-Nano\")\n", + "id": "83a4a77e7a7dd92" }, { - "cell_type": "markdown", - "id": "62994915", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "5a1741e0e2b2ae34" }, { - "cell_type": "code", - "execution_count": null, - "id": "58a28e11", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(i2vs_output)\n" - ] + "execution_count": null, + "source": "view_run(i2vs_output)\n", + "id": "4a772a87d1370c5d" }, { - "cell_type": "markdown", - "id": "fe62931e", "metadata": {}, + "cell_type": "markdown", "source": [ "# Cosmos3-Super Examples\n", "\n", "The same use cases for the larger `Cosmos3-Super` model. This section is self-contained; you can run it without the Cosmos3-Nano section above.\n" - ] + ], + "id": "de76455f177cc24" }, { - "cell_type": "markdown", - "id": "466c6bac", "metadata": {}, + "cell_type": "markdown", "source": [ "## Super: Text to Image\n", "\n", "Super text-to-image generation using the same structured JSON prompt.\n", "\n", "### Create Payload\n" - ] + ], + "id": "c3023174bdf5e37e" }, { - "cell_type": "code", - "execution_count": null, - "id": "575653c1", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2i_super_payload, t2i_super_output, t2i_super_model = create_payload(\"t2i_super\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "t2i_super_payload, t2i_super_output, t2i_super_model = create_payload(\"t2i_super\", backend=\"vllm\")\n", + "id": "aae9669830d017ee" }, { - "cell_type": "markdown", - "id": "42776c8b", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "7fd56a3c10c4fa52" }, { - "cell_type": "code", - "execution_count": null, - "id": "824783c8", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2i_super_payload, t2i_super_output, model=\"Cosmos3-Super\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(t2i_super_payload, t2i_super_output, model=\"Cosmos3-Super\")\n", + "id": "afc87a1edbcce49a" }, { - "cell_type": "markdown", - "id": "1ebf0b15", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "c76726554bb7abf0" }, { - "cell_type": "code", - "execution_count": null, - "id": "b0fba2bb", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2i_super_output)\n" - ] + "execution_count": null, + "source": "view_run(t2i_super_output)\n", + "id": "d618dea3382c50a6" }, { - "cell_type": "markdown", - "id": "f608eaa7", "metadata": {}, + "cell_type": "markdown", "source": [ "## Super: Text to Video Without Audio\n", "\n", "Super text-to-video generation with audio disabled.\n", "\n", "### Create Payload\n" - ] + ], + "id": "69d929aca037b946" }, { - "cell_type": "code", - "execution_count": null, "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2v_super_noaudio_payload, t2v_super_noaudio_output, t2v_super_noaudio_model = create_payload(\"t2v_super_noaudio\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "t2v_super_noaudio_payload, t2v_super_noaudio_output, t2v_super_noaudio_model = create_payload(\"t2v_super_noaudio\", backend=\"vllm\")\n", + "id": "65428e12d8ff5133" }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "5c9f7ee069e7f20f" }, { - "cell_type": "code", - "execution_count": null, "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2v_super_noaudio_payload, t2v_super_noaudio_output, model=\"Cosmos3-Super\")\n" - ] + "execution_count": null, + "source": "run_vllm_payload(t2v_super_noaudio_payload, t2v_super_noaudio_output, model=\"Cosmos3-Super\")\n", + "id": "d06a89fc789c8ce6" }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "b651ddbb959454f0" }, { - "cell_type": "code", - "execution_count": null, "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2v_super_noaudio_output)\n" - ] + "execution_count": null, + "source": "view_run(t2v_super_noaudio_output)\n", + "id": "60523b09301bb42f" }, { - "cell_type": "markdown", - "id": "367f4161", "metadata": {}, + "cell_type": "markdown", "source": [ "## Super: Image to Video Without Audio\n", "\n", "Super image-to-video generation using its paired image asset, with audio disabled.\n", "\n", "### Create Payload\n" - ] + ], + "id": "aba95f8543ecf1dd" }, { - "cell_type": "code", - "execution_count": null, "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "i2v_super_noaudio_payload, i2v_super_noaudio_output, i2v_super_noaudio_model = create_payload(\"i2v_super_noaudio\", backend=\"vllm\")\n" - ] + "execution_count": null, + "source": "i2v_super_noaudio_payload, i2v_super_noaudio_output, i2v_super_noaudio_model = create_payload(\"i2v_super_noaudio\", backend=\"vllm\")\n", + "id": "b31382d60cd3ff9c" }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "be5e8b9f8839e785" }, { + "metadata": {}, "cell_type": "code", + "outputs": [], "execution_count": null, + "source": "run_vllm_payload(i2v_super_noaudio_payload, i2v_super_noaudio_output, model=\"Cosmos3-Super\")\n", + "id": "79d9ccf0488f9ff6" + }, + { "metadata": {}, + "cell_type": "markdown", + "source": "### View Results\n", + "id": "46b0167f07255311" + }, + { + "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(i2v_super_noaudio_payload, i2v_super_noaudio_output, model=\"Cosmos3-Super\")\n" - ] + "execution_count": null, + "source": "view_run(i2v_super_noaudio_output)\n", + "id": "2fa52f631a0d5424" }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ - "### View Results\n" - ] + "## Super: Video to Video Without Audio\n", + "\n", + "Super video-to-video generation using the same reference-video request shape as Nano, against the `Cosmos3-Super` endpoint.\n", + "\n", + "### Create Payload\n" + ], + "id": "2ab4a7f013ccf5d7" }, { + "metadata": {}, "cell_type": "code", + "outputs": [], "execution_count": null, + "source": "v2v_super_noaudio_payload, v2v_super_noaudio_output, v2v_super_noaudio_model = create_payload(\"v2v_super_noaudio\", backend=\"vllm\")\n", + "id": "2288187414f4a2f5" + }, + { "metadata": {}, + "cell_type": "markdown", + "source": "### Run\n", + "id": "4d3d0966bd302460" + }, + { + "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(i2v_super_noaudio_output)\n" - ] + "execution_count": null, + "source": "run_vllm_payload(v2v_super_noaudio_payload, v2v_super_noaudio_output, model=\"Cosmos3-Super\")\n", + "id": "d13fe46aecc4e0bf" }, { + "metadata": {}, "cell_type": "markdown", - "id": "9879d0c5", + "source": "### View Results\n", + "id": "f3776858487b6ef4" + }, + { "metadata": {}, + "cell_type": "code", + "outputs": [], + "execution_count": null, + "source": "view_run(v2v_super_noaudio_output)\n", + "id": "d8d1693cbbfa5d5c" + }, + { + "metadata": {}, + "cell_type": "markdown", "source": [ "# Super-4Step Examples\n", "\n", "The same audiovisual use cases for the Cosmos distilled model. This section is self-contained; you can run it without any of the sections above." - ] + ], + "id": "911665b17c7b51dd" }, { - "cell_type": "markdown", - "id": "00cc5f94", "metadata": {}, + "cell_type": "markdown", "source": [ "## Super-4Step: Text to Image\n", "\n", "Cosmos3-Super-Text2Image-4Step generation is using the same structured JSON prompt.\n", "\n", "### Create Payload" - ] + ], + "id": "5d84567531d8437f" }, { - "cell_type": "code", - "execution_count": null, - "id": "b81c19a5", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "t2i_distil_payload, t2i_distil_output, t2i_distil_model = create_payload(\"t2i_distil\", backend=\"vllm\")" - ] + "execution_count": null, + "source": "t2i_distil_payload, t2i_distil_output, t2i_distil_model = create_payload(\"t2i_distil\", backend=\"vllm\")", + "id": "cc28645d9a2fdc89" }, { - "cell_type": "markdown", - "id": "a2382619", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "3dfdfe550050211a" }, { - "cell_type": "code", - "execution_count": null, - "id": "c4055595", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(t2i_distil_payload, t2i_distil_output, model=\"Cosmos3-Super-Text2Image-4Step\")" - ] + "execution_count": null, + "source": "run_vllm_payload(t2i_distil_payload, t2i_distil_output, model=\"Cosmos3-Super-Text2Image-4Step\")", + "id": "f2f15a811ba24461" }, { - "cell_type": "markdown", - "id": "7aeb0145", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "8c1da93aba57434c" }, { - "cell_type": "code", - "execution_count": null, - "id": "167b53fd", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(t2i_distil_output)" - ] + "execution_count": null, + "source": "view_run(t2i_distil_output)", + "id": "9b2ef72e7778d25b" }, { - "cell_type": "markdown", - "id": "e6633c3c", "metadata": {}, + "cell_type": "markdown", "source": [ "## Super-4Step: Image to Video Without Audio\n", "\n", "Cosmos3-Super-Image2Video-4Step generation is using its paired image asset, with audio disabled.\n", "\n", "### Create Payload" - ] + ], + "id": "a7cf0beda6cf5f05" }, { - "cell_type": "code", - "execution_count": null, - "id": "f64a8032", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "i2v_distil_payload, i2v_distil_output, i2v_distil_model = create_payload(\"i2v_distil\", backend=\"vllm\")" - ] + "execution_count": null, + "source": "i2v_distil_payload, i2v_distil_output, i2v_distil_model = create_payload(\"i2v_distil\", backend=\"vllm\")", + "id": "b02bda901a69ece0" }, { - "cell_type": "markdown", - "id": "3a5f683c", "metadata": {}, - "source": [ - "### Run\n" - ] + "cell_type": "markdown", + "source": "### Run\n", + "id": "4774d0e40ce32b45" }, { - "cell_type": "code", - "execution_count": null, - "id": "b5fbb8eb", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "run_vllm_payload(i2v_distil_payload, i2v_distil_output, model=\"Cosmos3-Super-Image2Video-4Step\")" - ] + "execution_count": null, + "source": "run_vllm_payload(i2v_distil_payload, i2v_distil_output, model=\"Cosmos3-Super-Image2Video-4Step\")", + "id": "7211726ac4e9d72d" }, { - "cell_type": "markdown", - "id": "327f8bf6", "metadata": {}, - "source": [ - "### View Results\n" - ] + "cell_type": "markdown", + "source": "### View Results\n", + "id": "bc666a86e3396d79" }, { - "cell_type": "code", - "execution_count": null, - "id": "45800e90", "metadata": {}, + "cell_type": "code", "outputs": [], - "source": [ - "view_run(i2v_distil_output)" - ] + "execution_count": null, + "source": "view_run(i2v_distil_output)", + "id": "20ffe99c1aafd80b" } ], "metadata": { "kernelspec": { - "display_name": "scratch.adsridhar_other (3.12.3)", + "display_name": "base", "language": "python", "name": "python3" }, @@ -1262,7 +1156,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.3" + "version": "3.11.7" } }, "nbformat": 4, diff --git a/cookbooks/cosmos3/generator/transfer/README.md b/cookbooks/cosmos3/generator/transfer/README.md index b09dfaf0..539a4590 100644 --- a/cookbooks/cosmos3/generator/transfer/README.md +++ b/cookbooks/cosmos3/generator/transfer/README.md @@ -1,7 +1,7 @@ # Cosmos3 Generator Transfer Examples Cosmos3 video **transfer** examples — **Nano** (single GPU) and **Super** (multi-GPU, 32B) — on -the native PyTorch (Cosmos Framework) path. +the native PyTorch (Cosmos Framework) path and the OpenAI-compatible vLLM-Omni server path. Sample assets under [`assets/`](./assets) cover spatial control signals paired with `prompt.json` files: @@ -10,9 +10,11 @@ Sample assets under [`assets/`](./assets) cover spatial control signals paired w - **Depth** — depth map control plus caption. - **Segmentation** — segmentation map control plus caption. - **World scenario (WSM)** — world-scenario map control plus caption. -- **Multi-control** — two or more hints combined with per-hint weights. +- **Multi-control** — two or more hints; Cosmos Framework also supports per-hint weights. -vLLM-Omni does not expose transfer controls today. +Both Cosmos Framework and vLLM-Omni support multi-control transfer. Per-hint +weighting is supported only by Cosmos Framework; vLLM-Omni accepts multiple +controls but does not support per-hint weights. Environment setup is centralized in the shared [Cosmos3 cookbooks environment setup](../../README.md) guide. @@ -20,11 +22,15 @@ Environment setup is centralized in the shared ## Transfer Definition Video transfer generates a target clip from a `prompt.json` caption and one or more -spatial control signals. Inference uses `model_mode` `video2video`. Control signals can -be supplied as pre-computed videos (`control_path`) or derived on-the-fly from a raw -source video (`vision_path`). Output frame count and geometry come from the control -video; see the spec field reference for how `fps` and `aspect_ratio` are resolved. -All examples share `assets/negative_prompt.json` for the negative caption. +spatial control signals. The Framework path uses `model_mode` `video2video` in a local JSON spec. +The vLLM-Omni path uses `POST /v1/videos/sync` and passes one or more hint keys (`edge`, `blur`, +`depth`, `seg`, or `wsm`) inside `extra_params`. Cosmos Framework accepts pre-computed +control videos (`control_path`) or derives active controls from a raw source video +(`vision_path`). With vLLM-Omni, pass pre-computed controls through `control_path`; edge +and blur controls can also be derived from an uploaded `input_reference`. Output frame +count and geometry come from the control video; see the spec field reference for how +`fps` and `aspect_ratio` are resolved. All examples share `assets/negative_prompt.json` +for the negative caption. | Control | Asset folder | Inference input | Generation duration | | --- | --- | --- | --- | @@ -33,9 +39,10 @@ All examples share `assets/negative_prompt.json` for the negative caption. | Depth | `assets/depth/` | `control_depth.mp4` + `prompt.json` | 121 frames @ 30 FPS | | Segmentation | `assets/seg/` | `control_seg.mp4` + `prompt.json` | 121 frames @ 30 FPS | | World scenario (WSM) | `assets/wsm/` | `control_wsm.mp4` + `prompt.json` | 101 frames @ 10 FPS | -| Multi-control | `assets/multi_control/` | `vision_path` + multiple hints | 121 frames @ 30 FPS | +| Multi-control | `assets/multi_control/` | `vision_path` + multiple hints (Framework example) | 121 frames @ 30 FPS | -Transfer inference is selected automatically when any hint key is present in the spec. +Transfer inference is selected automatically when any hint key is present in the +Framework spec or in vLLM-Omni `extra_params`. The same spec files are used for both Nano and Super — model selection is controlled entirely by `--checkpoint-path`. @@ -126,133 +133,87 @@ jupyter execute run_video_transfer_with_cosmos_framework.ipynb Outputs land under `outputs/notebooks//transfer_/vision.mp4`. ---- +## Run with vLLM-Omni -## Multi-Control Transfer +### Quickstart -Multi-control transfer blends two or more spatial hint streams — for example edge + depth -— into a single generation pass. Each active hint receives a `weight` that determines its -relative influence. Weights across all active hints should sum to 1.0 for predictable -behavior, though the model accepts any positive values. +vLLM-Omni accepts multiple control hints in one request, but does not support +the per-hint `weight` field available in Cosmos Framework. -### Concepts - -| Field | Description | -| --- | --- | -| `edge` / `blur` / `depth` / `seg` | Hint block; set any subset to activate those controls | -| `weight` | Per-hint blending weight; ratios matter, not absolute values (default `1.0`) | -| `control_path` | Path to a **pre-computed** control video (optional; see below) | -| `vision_path` | Raw source video; the framework derives all active controls on-the-fly | -| `control_guidance` | Global CFG scale across all active control streams (default `1.5`) | - -**Two input modes:** - -1. **`vision_path` mode** — Provide a raw source video. For `edge` and `blur` hints - with no `control_path`, the framework computes the control signal on-the-fly - (Canny for `edge`, downscale/upscale for `blur`). `depth` and `seg` always require - a pre-computed `control_path` — they depend on DepthAnything and SAM2 which are - not bundled in the cosmos-framework. - -2. **Pre-computed mode** — Provide a `control_path` inside each hint block. All - control videos must be derived from the **same source video** and share identical - resolution, fps, and frame count. Use this when you want exact control over the - pre-processed signals or to reuse cached extractions across runs. - -### Quickstart — Multi-control from a source video - -Uses the same env vars as the single-control quickstart (`COSMOS_FRAMEWORK` and `TRANSFER_ROOT`). - -#### Cosmos3-Nano (single GPU) +Set up the environment and start the server: +[vLLM-Omni setup](../../README.md#vllm-omni). Transfer controls are available from +vLLM-Omni `main` and the released `vllm/vllm-omni:cosmos3` container. Check the current +[Cosmos3-Nano recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) +before selecting an image. For Docker, run the command from the `cosmos` repo root so +the repo is mounted at `/workspace` and the server runs from that directory inside the +container: ```bash -cd "$COSMOS_FRAMEWORK" - -# edge + blur computed on-the-fly from vision_path (robot_pouring.mp4) -CUDA_VISIBLE_DEVICES=0 \ -.venv/bin/python -m cosmos_framework.scripts.inference \ - --parallelism-preset=latency \ - -i "$TRANSFER_ROOT/specs/multi_control.json" \ - -o "$TRANSFER_ROOT/outputs/Cosmos3-Nano/" \ - --checkpoint-path Cosmos3-Nano \ - --seed 2026 -``` - -Output lands at `outputs/Cosmos3-Nano/transfer_multi_control/vision.mp4`. - -#### Cosmos3-Super (multi-GPU) - -```bash -cd "$COSMOS_FRAMEWORK" - -CUDA_VISIBLE_DEVICES=0,1,2,3 \ -.venv/bin/torchrun --nproc-per-node=4 \ - --master-addr=127.0.0.1 --master-port=29500 \ - -m cosmos_framework.scripts.inference \ - --parallelism-preset=latency \ - -i "$TRANSFER_ROOT/specs/multi_control.json" \ - -o "$TRANSFER_ROOT/outputs/Cosmos3-Super/" \ - --checkpoint-path Cosmos3-Super \ - --seed 2026 +export COSMOS3_WORKDIR="$(pwd)" +export COSMOS3_HOST_PORT=8000 ``` -Output lands at `outputs/Cosmos3-Super/transfer_multi_control/vision.mp4`. - -### Spec field reference — multi-control +The transfer examples send repo-local `control_path` strings to the server. For +Docker, those paths must be visible from the server working directory. With the +shared Docker setup, the checked-in depth control video is: -**`specs/multi_control.json`** (edge dominant + blur secondary, controls derived on-the-fly from `vision_path`): - -```json -{ - "name": "transfer_multi_control", - "model_mode": "video2video", - "resolution": "720", - "aspect_ratio": "16,9", - "num_frames": 121, - "fps": 30, - "guidance": 3.0, - "control_guidance": 1.5, - "negative_prompt_file": "../assets/negative_prompt.json", - "prompt_path": "../assets/multi_control/prompt.json", - "vision_path": "https://.../robot_pouring.mp4", - "edge": { - "weight": 0.75, - "preset_edge_threshold": "medium" - }, - "blur": { - "weight": 0.25, - "preset_blur_strength": "medium" - } -} +```text +cookbooks/cosmos3/generator/transfer/assets/depth/control_depth.mp4 ``` -Only the **ratio** between weights matters — `edge: 3, blur: 1` is equivalent to -`edge: 0.75, blur: 0.25`. Omitting `weight` defaults to `1.0` (equal contribution). - -To use **pre-computed control videos** instead, replace `vision_path` with `control_path` -inside each hint block. All control videos must be derived from the same source and -share identical resolution, fps, and frame count: - -```json -{ - "edge": { - "control_path": "/path/to/control_edge.mp4", - "weight": 0.75, - "preset_edge_threshold": "medium" - }, - "blur": { - "control_path": "/path/to/control_blur.mp4", - "weight": 0.25, - "preset_blur_strength": "medium" - } -} +If your server does not run from the repo root, start it from the repo root or +adjust `control_path` to a path the server process can read. + +Transfer requests should also pass the spec `resolution` inside `extra_params`. +Cosmos3 transfer bucket selection reads `extra_params.resolution` and the +control/input aspect ratio; set the video API `size` field to the matching +`x` value for a consistent request. + +Send a depth-transfer request: + +```python +import json +from pathlib import Path + +import requests + +transfer_root = Path("cookbooks/cosmos3/generator/transfer") +prompt = json.dumps(json.load(open(transfer_root / "assets/depth/prompt.json"))) +negative = json.dumps(json.load(open(transfer_root / "assets/negative_prompt.json"))) +control_path = transfer_root / "assets/depth/control_depth.mp4" + +response = requests.post( + "http://localhost:8000/v1/videos/sync", + data={ + "prompt": prompt, + "negative_prompt": negative, + "size": "1280x720", + "num_frames": "121", + "fps": "30", + "num_inference_steps": "50", + "guidance_scale": "3.0", + "flow_shift": "10.0", + "seed": "2026", + "extra_params": json.dumps( + { + "use_resolution_template": False, + "use_duration_template": False, + "guardrails": True, + "depth": {"control_path": control_path.as_posix()}, + "resolution": "720", + "control_guidance": 1.5, + "num_video_frames_per_chunk": 121, + "max_frames": 121, + } + ), + }, + headers={"Accept": "video/mp4"}, +) +response.raise_for_status() +Path("/tmp/cosmos3_transfer_depth.mp4").write_bytes(response.content) ``` -`control_guidance` scales the influence of all active control streams collectively; -`weight` distributes that influence among individual hints. - ---- - -### Spec field reference — single-control +### Spec field reference A representative spec (`specs/edge.json`): @@ -293,6 +254,11 @@ Key fields: - [`run_video_transfer_with_cosmos_framework.ipynb`](./run_video_transfer_with_cosmos_framework.ipynb) — self-contained notebook: §9–§13 Nano single-control, §14–§18 Super single-control, §19 multi-control (Nano). Edit §2, run top-to-bottom. +- [`run_video_transfer_with_vllm_omni.ipynb`](./run_video_transfer_with_vllm_omni.ipynb) — + full tutorial against an already-running vLLM-Omni server: endpoint checks, repo-local + control paths, five single-control transfer requests, and compact previews. The API + also accepts unweighted multi-control requests by including multiple hint blocks in + `extra_params`. - [`specs/`](./specs) — checked-in Framework input JSON per control (paths relative to `specs/`). Shared by both Nano and Super. diff --git a/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb b/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb index ca9654fe..6ee01d7a 100644 --- a/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb +++ b/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb @@ -32,6 +32,8 @@ "\n", "Run Cosmos3-Nano single-control examples (§9–§13), Cosmos3-Super single-control examples (§14–§18), or the multi-control example (§19, Nano).\n", "\n", + "For the OpenAI-compatible vLLM-Omni transfer path, see [`run_video_transfer_with_vllm_omni.ipynb`](./run_video_transfer_with_vllm_omni.ipynb). This notebook focuses on the Cosmos Framework path.\n", + "\n", "| Model | Launcher | Parallelism | GPUs |\n", "|---|---|---|---|\n", "| Cosmos3-Nano | `python` (single GPU) | `latency` | 1 |\n", diff --git a/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb b/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb new file mode 100644 index 00000000..fda781d2 --- /dev/null +++ b/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb @@ -0,0 +1,497 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Cosmos3 Nano Transfer with vLLM-Omni\n", + "\n", + "This notebook calls an already-running vLLM-Omni Cosmos3 server through the OpenAI-compatible video API. It reuses the checked-in transfer control assets and specs from this cookbook, then sends edge, blur, depth, segmentation, and world-scenario-map transfer requests to `POST /v1/videos/sync`.\n", + "\n", + "The runnable examples use one control at a time. vLLM-Omni also accepts unweighted multi-control requests: include multiple hint blocks in `extra_params`. Per-hint `weight` is supported only by Cosmos Framework.\n", + "\n", + "The notebook does not modify the vLLM-Omni source tree.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Prerequisites\n", + "\n", + "Start a vLLM-Omni Cosmos3 server before running the request cells. The examples assume the `cosmos` repo is mounted at `/workspace` and the server runs from that directory inside the container, so repo-local `cookbooks/...` paths resolve correctly.\n", + "\n", + "Transfer controls are available from vLLM-Omni `main` and the released `vllm/vllm-omni:cosmos3` container. For current compatibility details, check the [Cosmos3-Nano recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md).\n", + "\n", + "```bash\n", + "cd /path/to/cosmos\n", + "export HF_HOME=\"${HF_HOME:-$HOME/.cache/huggingface}\"\n", + "export COSMOS3_WORKDIR=\"$PWD\"\n", + "export COSMOS3_HOST_PORT=8000\n", + "\n", + "docker run --runtime nvidia --gpus all \\\n", + " -e CUDA_DEVICE_ORDER=PCI_BUS_ID \\\n", + " -v \"${HF_HOME}:/root/.cache/huggingface\" \\\n", + " -v \"${COSMOS3_WORKDIR}:/workspace\" \\\n", + " -p \"${COSMOS3_HOST_PORT}:8000\" --ipc=host \\\n", + " -w /workspace \\\n", + " vllm/vllm-omni:cosmos3 \\\n", + " vllm serve nvidia/Cosmos3-Nano \\\n", + " --omni \\\n", + " --model-class-name Cosmos3OmniDiffusersPipeline \\\n", + " --allowed-local-media-path / \\\n", + " --port 8000 \\\n", + " --init-timeout 1800\n", + "```\n", + "\n", + "Generator guardrails are on by default and require access to the gated `nvidia/Cosmos-1.0-Guardrail` repository. To disable guardrails for these sample requests, set `COSMOS3_VLLM_GUARDRAILS=false`.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Configure Paths and Endpoints\n", + "\n", + "Run this cell from anywhere inside the `cosmos` checkout. It resolves local assets, output paths, the vLLM-Omni endpoint, and repo-local `control_path` values.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "import base64\n", + "import html\n", + "import json\n", + "import os\n", + "import shutil\n", + "import subprocess\n", + "import time\n", + "from IPython.display import HTML, display\n", + "\n", + "\n", + "def find_repo_root(start: Path) -> Path:\n", + " for path in [start, *start.parents]:\n", + " if (path / \"README.md\").exists() and (path / \"cookbooks\").exists():\n", + " return path\n", + " return start\n", + "\n", + "\n", + "COSMOS_ROOT = find_repo_root(Path.cwd().resolve())\n", + "TRANSFER_ROOT = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"transfer\"\n", + "SPECS_DIR = TRANSFER_ROOT / \"specs\"\n", + "ASSETS_DIR = TRANSFER_ROOT / \"assets\"\n", + "OUTPUT_ROOT = Path(\n", + " os.environ.get(\"COSMOS3_TRANSFER_VLLM_OUTPUT_ROOT\", TRANSFER_ROOT / \"outputs\" / \"vllm_omni\")\n", + ").resolve()\n", + "VLLM_BASE_URL = os.environ.get(\"COSMOS3_VLLM_BASE_URL\", \"http://localhost:8000\").rstrip(\"/\")\n", + "GUARDRAILS = os.environ.get(\"COSMOS3_VLLM_GUARDRAILS\", \"true\").strip().lower() not in {\"0\", \"false\", \"no\", \"off\"}\n", + "\n", + "OUTPUT_ROOT.mkdir(parents=True, exist_ok=True)\n", + "\n", + "print(\"COSMOS_ROOT:\", COSMOS_ROOT)\n", + "print(\"TRANSFER_ROOT:\", TRANSFER_ROOT)\n", + "print(\"OUTPUT_ROOT:\", OUTPUT_ROOT)\n", + "print(\"VLLM_BASE_URL:\", VLLM_BASE_URL)\n", + "print(\"GUARDRAILS:\", GUARDRAILS)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Verify Endpoint Configuration\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from urllib.parse import urlparse\n", + "\n", + "try:\n", + " import requests\n", + "except ImportError as exc:\n", + " raise RuntimeError(\"Install requests in this notebook kernel: pip install requests\") from exc\n", + "\n", + "\n", + "def api_root_url(base_url: str) -> str:\n", + " normalized = base_url.rstrip(\"/\")\n", + " if not normalized.endswith(\"/v1\"):\n", + " normalized = f\"{normalized}/v1\"\n", + " return normalized\n", + "\n", + "\n", + "API_ROOT = api_root_url(VLLM_BASE_URL)\n", + "VIDEOS_SYNC_URL = f\"{API_ROOT}/videos/sync\"\n", + "MODELS_URL = f\"{API_ROOT}/models\"\n", + "parsed = urlparse(API_ROOT)\n", + "print(\"api root:\", API_ROOT)\n", + "print(\"videos sync:\", VIDEOS_SYNC_URL)\n", + "print(\"models:\", MODELS_URL)\n", + "print(\"scheme:\", parsed.scheme)\n", + "print(\"host:\", parsed.netloc)\n", + "\n", + "response = requests.get(MODELS_URL, timeout=30)\n", + "response.raise_for_status()\n", + "print(response.json())\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Define Transfer Request Helpers\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "TRANSFER_CONTROLS = (\"edge\", \"blur\", \"depth\", \"seg\", \"wsm\")\n", + "\n", + "\n", + "def compact_json_file(path: Path) -> str:\n", + " return json.dumps(json.loads(path.read_text()), ensure_ascii=True, separators=(\",\", \":\"))\n", + "\n", + "\n", + "def resolve_spec_path(spec_path: Path, value: str) -> Path:\n", + " path = Path(value)\n", + " if path.is_absolute():\n", + " return path\n", + " return (spec_path.parent / path).resolve()\n", + "\n", + "\n", + "def repo_relative_path(local_path: Path) -> str:\n", + " return local_path.resolve().relative_to(COSMOS_ROOT).as_posix()\n", + "\n", + "\n", + "def spec_size(spec: dict) -> str:\n", + " height = int(spec[\"resolution\"])\n", + " width_ratio, height_ratio = (int(part) for part in spec[\"aspect_ratio\"].split(\",\", 1))\n", + " width = round(height * width_ratio / height_ratio)\n", + " return f\"{width}x{height}\"\n", + "\n", + "\n", + "def load_transfer_spec(control: str) -> tuple[Path, dict]:\n", + " if control not in TRANSFER_CONTROLS:\n", + " raise ValueError(f\"control must be one of {TRANSFER_CONTROLS}, got {control!r}\")\n", + " spec_path = SPECS_DIR / f\"{control}.json\"\n", + " if not spec_path.is_file():\n", + " raise FileNotFoundError(spec_path)\n", + " return spec_path, json.loads(spec_path.read_text())\n", + "\n", + "\n", + "def build_transfer_request(control: str) -> tuple[dict[str, str], Path, Path]:\n", + " spec_path, spec = load_transfer_spec(control)\n", + " hint = dict(spec[control])\n", + " local_control_path = resolve_spec_path(spec_path, hint[\"control_path\"])\n", + " hint[\"control_path\"] = repo_relative_path(local_control_path)\n", + "\n", + " extra_params = {\n", + " \"use_resolution_template\": False,\n", + " \"use_duration_template\": False,\n", + " \"guardrails\": GUARDRAILS,\n", + " control: hint,\n", + " \"resolution\": spec[\"resolution\"],\n", + " \"control_guidance\": spec[\"control_guidance\"],\n", + " \"num_video_frames_per_chunk\": spec[\"num_video_frames_per_chunk\"],\n", + " \"num_conditional_frames\": spec.get(\"num_conditional_frames\", 1),\n", + " \"num_first_chunk_conditional_frames\": spec.get(\"num_first_chunk_conditional_frames\", 0),\n", + " \"share_vision_temporal_positions\": spec.get(\"share_vision_temporal_positions\", True),\n", + " \"max_frames\": spec[\"num_frames\"],\n", + " }\n", + " form = {\n", + " \"prompt\": compact_json_file(resolve_spec_path(spec_path, spec[\"prompt_path\"])),\n", + " \"negative_prompt\": compact_json_file(resolve_spec_path(spec_path, spec[\"negative_prompt_file\"])),\n", + " \"size\": spec_size(spec),\n", + " \"num_frames\": str(spec[\"num_frames\"]),\n", + " \"fps\": str(spec[\"fps\"]),\n", + " \"num_inference_steps\": \"50\",\n", + " \"guidance_scale\": str(spec[\"guidance\"]),\n", + " \"flow_shift\": \"10.0\",\n", + " \"seed\": \"2026\",\n", + " \"extra_params\": json.dumps(extra_params, separators=(\",\", \":\")),\n", + " }\n", + " output_path = OUTPUT_ROOT / control / f\"{spec['name']}.mp4\"\n", + " return form, local_control_path, output_path\n", + "\n", + "\n", + "def run_transfer(control: str) -> Path:\n", + " form, local_control_path, output_path = build_transfer_request(control)\n", + " output_path.parent.mkdir(parents=True, exist_ok=True)\n", + " error_path = output_path.with_suffix(\".error.txt\")\n", + " tmp_path = output_path.with_suffix(\".tmp\")\n", + " print(\"control:\", control)\n", + " print(\"local control:\", local_control_path)\n", + " print(\"size:\", form[\"size\"], \"frames:\", form[\"num_frames\"], \"fps:\", form[\"fps\"])\n", + " print(\"output:\", output_path)\n", + "\n", + " headers = {\"Accept\": \"video/mp4\"}\n", + " api_key = os.environ.get(\"COSMOS3_VLLM_API_KEY\")\n", + " if api_key:\n", + " headers[\"Authorization\"] = f\"Bearer {api_key}\"\n", + "\n", + " t0 = time.time()\n", + " response = requests.post(VIDEOS_SYNC_URL, data=form, headers=headers, timeout=3600)\n", + " if not response.ok:\n", + " error_path.write_text(response.text)\n", + " print(\"request failed:\", response.status_code)\n", + " print(response.text[:2000])\n", + " response.raise_for_status()\n", + " tmp_path.write_bytes(response.content)\n", + " tmp_path.replace(output_path)\n", + " print(f\"wrote {output_path} in {time.time() - t0:.1f}s\")\n", + " return output_path\n", + "\n", + "\n", + "def _ffmpeg_exe() -> str:\n", + " try:\n", + " import imageio_ffmpeg\n", + "\n", + " return imageio_ffmpeg.get_ffmpeg_exe()\n", + " except ImportError:\n", + " pass\n", + " exe = shutil.which(\"ffmpeg\")\n", + " if exe:\n", + " return exe\n", + " raise RuntimeError(\"Install imageio-ffmpeg or put ffmpeg on PATH to create compact previews.\")\n", + "\n", + "\n", + "def make_preview(src: Path, *, crf: int = 28) -> Path:\n", + " preview = src.with_name(f\"{src.stem}_preview.mp4\")\n", + " if not preview.exists() or preview.stat().st_mtime < src.stat().st_mtime:\n", + " subprocess.run(\n", + " [\n", + " _ffmpeg_exe(),\n", + " \"-y\",\n", + " \"-loglevel\",\n", + " \"error\",\n", + " \"-i\",\n", + " str(src),\n", + " \"-c:v\",\n", + " \"libx264\",\n", + " \"-crf\",\n", + " str(crf),\n", + " \"-preset\",\n", + " \"veryfast\",\n", + " \"-an\",\n", + " \"-pix_fmt\",\n", + " \"yuv420p\",\n", + " str(preview),\n", + " ],\n", + " check=True,\n", + " )\n", + " return preview\n", + "\n", + "\n", + "def display_video(path: Path, *, width: int = 720) -> None:\n", + " data = base64.b64encode(path.read_bytes()).decode(\"ascii\")\n", + " label = html.escape(str(path))\n", + " markup = f'''\n", + "\n", + "
{label}
\n", + "'''\n", + " display(HTML(markup))\n", + "\n", + "\n", + "def view_transfer(control: str, output_path: Path | None = None) -> None:\n", + " form, local_control_path, expected_output = build_transfer_request(control)\n", + " output_path = Path(output_path or expected_output)\n", + " if not output_path.is_file():\n", + " raise FileNotFoundError(f\"missing output: {output_path} (run {control} transfer first)\")\n", + " for label, src in [(\"control\", local_control_path), (\"generated\", output_path)]:\n", + " preview = make_preview(src)\n", + " print(f\"{control} {label}: {src.name} ({src.stat().st_size // 1024} KB -> {preview.stat().st_size // 1024} KB preview)\")\n", + " display_video(preview)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Preview Available Inputs\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "for control in TRANSFER_CONTROLS:\n", + " form, local_control_path, _ = build_transfer_request(control)\n", + " prompt = json.loads(form[\"prompt\"])\n", + " caption = prompt.get(\"temporal_caption\") or prompt.get(\"comprehensive_t2i_caption\") or prompt.get(\"extra\", {}).get(\"prompt\", \"\")\n", + " print(f\"{control}: {local_control_path.relative_to(COSMOS_ROOT)}\")\n", + " print(f\" size={form['size']} frames={form['num_frames']} fps={form['fps']}\")\n", + " print(f\" prompt={caption[:180]}{'...' if len(caption) > 180 else ''}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Edge (Canny) Transfer\n", + "\n", + "Run the `edge` transfer request through vLLM-Omni, then display the input control video and generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "edge_output = run_transfer(\"edge\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "view_transfer(\"edge\", edge_output)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Blur Transfer\n", + "\n", + "Run the `blur` transfer request through vLLM-Omni, then display the input control video and generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "blur_output = run_transfer(\"blur\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "view_transfer(\"blur\", blur_output)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. Depth Transfer\n", + "\n", + "Run the `depth` transfer request through vLLM-Omni, then display the input control video and generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "depth_output = run_transfer(\"depth\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "view_transfer(\"depth\", depth_output)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 9. Segmentation Transfer\n", + "\n", + "Run the `seg` transfer request through vLLM-Omni, then display the input control video and generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "seg_output = run_transfer(\"seg\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "view_transfer(\"seg\", seg_output)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 10. World Scenario Map Transfer\n", + "\n", + "Run the `wsm` transfer request through vLLM-Omni, then display the input control video and generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "wsm_output = run_transfer(\"wsm\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "view_transfer(\"wsm\", wsm_output)\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}