关于vllm部分的修改

[Irelia314]

想问一下作者针对vllm做了哪些地方的修改？

[gjghfd]

Our implementation is based on vLLM v0.4.2. The major modifications are:

1. Startup optimizations

• Mitigate KV cache initialization overhead by replacing torch.zero with torch.empty.
• Disable the profiling run.
• Allow the model loader to directly use the loaded parameters on GPU.

2. Pipeline consolidation

• Implement pipeline parallelism support by modifying both executor and model-specific codes (e.g., model_executor/models/llama.py).
• Allow the model executor to initialize another worker with the whole model in the background. This is done by calling the init_dest_model_worker method of the GPUExecutor class.
• Enable request migration by modifying the step_async method of the _AsyncLLMEngine class.