NVIDIA TensorRT-LLM Enhances Encoder-Decoder Models with In-Flight Batching

NVIDIA has announced a significant update to TensorRT-LLM, an open source library that includes support for the encoder-decoder model architecture with ongoing batch processing. According to NVIDIA, this development enhances generative AI applications on NVIDIA GPUs by further expanding the library’s capacity to optimize inference across a variety of model architectures.

Expanded model support

TensorRT-LLM has long been an important tool for optimizing inference on models such as decoder-only architectures such as Llama 3.1, expert mixture models such as Mixtral, and selective state space models such as Mamba. In particular, the addition of encoder-decoder models, including T5, mT5, and BART, has significantly expanded functionality. This update supports full tensor parallelism, pipeline parallelism, and hybrid parallelism for these models, ensuring robust performance across a variety of AI tasks.

Improved on-board batch processing and efficiency

In-flight batch integration, also known as continuous batching, plays a pivotal role in managing runtime differences in the encoder-decoder model. These models typically require complex processing for key-value cache management and batch management, especially in scenarios where requests are processed recursively. The latest improvements in TensorRT-LLM streamline this process, delivering high throughput while minimizing latency, which is critical for real-time AI applications.

Production-ready deployment

For companies looking to deploy these models in production, the TensorRT-LLM encoder-decoder model is supported by NVIDIA Triton Inference Server. This open source software simplifies AI inference, allowing you to efficiently deploy optimized models. The Triton TensorRT-LLM backend further improves performance, making it a good choice for production-ready applications.

Junior Adaptation Support

This update also introduces support for Low-Rank Adaptation (LoRA), a fine-tuning technique that reduces memory and compute requirements while maintaining model performance. This feature is particularly useful for customizing models for specific tasks, efficiently serving multiple LoRA adapters within a single deployment, and reducing memory footprint through dynamic loading.

Future improvements

In the future, NVIDIA plans to introduce FP8 quantization to further improve latency and throughput of the encoder-decoder model. These enhancements promise to strengthen NVIDIA’s commitment to advancing AI technology by delivering even faster and more efficient AI solutions.

Image source: Shutterstock