vLLM is a language model serving engine designed for high throughput on GPU. It dynamically batches requests, manages the attention cache memory efficiently, and exposes an API compatible with OpenAI's. It suits use cases with many simultaneous requests.
Example
vllm serve mon-modele --port 8000