IBM Research announced a breakthrough in AI inference that combines speculative decoding and paging attention to improve the cost performance of large language models (LLMs). According to IBM Research, this development is expected to make customer care chatbots more efficient and cost-effective.
In recent years, LLM has improved the ability of chatbots to understand customer queries and provide accurate responses. However, the high cost and slow speed of delivering these models has led to broader adoption of AI. Speculative decoding emerges as an optimization technique that accelerates AI inference by generating tokens faster. This can improve customer experience by reducing latency by 2-3x.
Despite the benefits, reducing latency typically comes with a trade-off of increased operating costs due to reduced throughput or fewer users who can simultaneously utilize the model. IBM Research solved this problem by quadrupling throughput while halving the latency of the open source Granite 20B code model.
Speculative Decoding: Efficiency in Token Generation
LLM uses an inefficient translator architecture for text generation. Typically, forward passing is required to process each previously generated token before generating a new token. Speculative decoding modifies this process to evaluate multiple prospective tokens simultaneously. Once these tokens are verified, multiple tokens can be generated in one forward pass, improving inference speed.
This technique can be implemented in smaller, more efficient models or as part of the base model itself. Speculative decoding can maximize the efficiency of each GPU by processing tokens in parallel, doubling or tripling the speed of inference. While researchers at DeepMind and Google leveraged draft models when they first introduced speculative decoding, new methods like the Medusa speculator do not require auxiliary models.
IBM researchers tuned Medusa speculators by conditioning future tokens on each other rather than on the model’s next predicted token. This approach, combined with an efficient fine-tuning method using large and small batches of text, aligns the speculator’s responses closely with the LLM, significantly improving inference speed.
Paged Attention: Optimize memory usage
Reducing LLM latency often reduces throughput due to increased GPU memory strain. Dynamic batching can alleviate this, but not when speculative decoding is also competing for memory. IBM researchers solved this problem using paged attention, an optimization technique inspired by paging concepts in virtual memory and operating systems.
Existing attention algorithms store key-value (KV) sequences in contiguous memory, which results in fragmentation. However, paging attention breaks these sequences into smaller blocks, or pages, that can be accessed as needed. This method frees memory by minimizing redundant computations and allowing speculators to generate multiple candidates for each predicted word without duplicating the entire KV cache.
meaning of the future
IBM has integrated speculative decoding and attention into its Granite 20B code model. IBM Speculator has been made open source by Hugging Face so other developers can apply these technologies to their LLMs. IBM plans to implement these optimization technologies across all models of the watsonx platform to enhance enterprise AI applications.
Image source: Shutterstock