Add user guide for v2v and transfer for vllm-omni#247
Conversation
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Signed-off-by: Maciej Bala <mbala@nvidia.com>
|
@trungtpham @lfengad Could you please look? |
|
The MR looks good to me. |
|
@trungtpham - to confirm we will not have a separate notebook as run_with_vllm-omni ( along with PyTorch)? |
The MR adds [run_video_transfer_with_vllm_omni.ipynb]. Is this the one you are talking about? |
|
@trungtpham - ok saw that, looks good I will approve this PR, thank you! |
|
@lfengad Can you help approve this MR? Thanks :) |
|
@bastefaniak @lishunyang12 Some conflicts might be resolved? After that we could merge. THX! |
|
@bastefaniak @MaciejBalaNV. And I have a few comments below, then it is good to go. Thanks! |
|
|
||
| Set up the environment and start the server: | ||
| [vLLM-Omni setup](../../README.md#vllm-omni). Transfer controls require the | ||
| native vLLM-Omni install from `main` until the `vllm/vllm-omni:v0.23.0` Docker |
There was a problem hiding this comment.
We are about to release the Docker image for 0.24.0, as we only release Docker images for even-numbered versions.
There was a problem hiding this comment.
Updated to point to vllm/vllm-omni:v0.24.0
| Use vLLM-Omni for Generator production inference behind an OpenAI-compatible API. This integration loads the full Cosmos 3 checkpoint, including the Qwen3-VL-based reasoner path and the diffusion generation path. For understanding-only tasks that return text, use [Reasoner with vLLM](#reasoner-with-vllm) instead, which loads only the reasoner. | ||
|
|
||
| > **Compatibility status:** Cosmos 3 Generator support has landed in [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) `main`: text-to-image, text-to-video, and image-to-video ([#3454](https://github.com/vllm-project/vllm-omni/pull/3454)) and video-with-sound ([#4073](https://github.com/vllm-project/vllm-omni/pull/4073)) are merged; action (policy / forward-dynamics) is in review ([#4102](https://github.com/vllm-project/vllm-omni/pull/4102)) and video-to-video is planned. The `vllm/vllm-omni:cosmos3` Docker image remains the easiest all-in-one build. For current setup and per-modality usage, see the maintained recipes: [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). | ||
| > **Compatibility status:** Cosmos 3 Generator support is available in [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) `main` for text-to-image, text-to-video, image-to-video, video-to-video, transfer-control video-to-video, video-with-sound, and action generation. Transfer-control video-to-video requires installing vLLM-Omni from `main` until the `vllm/vllm-omni:v0.23.0` Docker image is released. For current setup and per-modality usage, see the maintained recipes: [Cosmos3-Nano](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [Cosmos3-Super](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). |
There was a problem hiding this comment.
Same as above. We can wait until 0.24.0 docker image is officially released in next couple of days.
There was a problem hiding this comment.
Ok, I'll resolve conflicts once new vllm-omni image drops
There was a problem hiding this comment.
Ok, I'll resolve conflicts once new vllm-omni image drops
Now, image is available. Please proceed.
There was a problem hiding this comment.
Updated to point to vllm/vllm-omni:v0.24.0
| (including action). Pull once: | ||
| The prebuilt image `vllm/vllm-omni:cosmos3` supports the released Generator | ||
| modalities. Transfer controls require the native vLLM-Omni install from `main` | ||
| until the `vllm/vllm-omni:v0.23.0` Docker image is released. Pull once: |
There was a problem hiding this comment.
Updated to point to vllm/vllm-omni:v0.24.0
| "\n", | ||
| "Start a vLLM-Omni Cosmos3 server before running the request cells. The examples assume the `cosmos` repo is mounted at `/workspace` and the server runs from that directory inside the container, so repo-local `cookbooks/...` paths resolve correctly.\n", | ||
| "\n", | ||
| "Transfer controls require installing vLLM-Omni from `main` until the `vllm/vllm-omni:v0.23.0` Docker image is released. Use the Docker command below only with a vLLM-Omni image that includes transfer support.\n", |
There was a problem hiding this comment.
Updated to point to vllm/vllm-omni:v0.24.0
|
@lfengad Can you take another look? |
|
@trungtpham @lfengad Could you please take a look? |
|
LGTM. |
# Conflicts: # cookbooks/cosmos3/generator/audiovisual/README.md # cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Signed-off-by: Maciej Bala <mbala@nvidia.com>
|
@trungtpham I fixed the conflicts, please approve again if possible |
|
@MaciejBalaNV I might need another one with write access to approve :) |
|
@rickzw Thanks for your approval. Can you safely merge this MR now? |
7 participants
This is a rebased copy of PR, as original contributor is OOO.
This PR adds user guide for v2v and transfer generation for vllm-omni. It uses the already merged transfer examples. The PR depends on this PR for vllm-omni: vllm-project/vllm-omni#4379, it should be merged after it.