AI experts reveal improved algorithms that can increase performance by up to 2.8 times without data loss
In a groundbreaking development, researchers from the Weizmann Institute, Intel Labs, and d-Matrix have unveiled a suite of innovative algorithms designed to significantly improve the efficiency of serving large language models (LLMs). These advancements could pave the way for more widespread and cost-effective use of LLMs.
The three key algorithms are the Vocabulary Pruning Algorithm, the Token Translation Algorithm, and the Shared-Meaning Token Prioritization Algorithm.
The Vocabulary Pruning Algorithm focuses the drafter's vocabulary on "easy-to-predict" tokens, such as articles, prepositions, or token completions based on partial words. By limiting the vocabulary of the smaller "drafter" model, speculative decoding becomes more efficient as the drafter specializes in suggesting tokens with high confidence that the larger model would accept.
The Token Translation Algorithm enables a large language model to translate its internal token language into a shared token format understood by other models. This allows any smaller model to be paired with any larger model, overcoming the language barrier that previously required specialized small models trained on exactly the same tokenizer as the large model.
The Shared-Meaning Token Prioritization Algorithm directs the smaller drafter model to rely primarily on tokens that have the same meaning across models. This ensures the drafter's predictions more reliably align with the larger model's outputs, making speculative decoding feasible without retraining specialized drafters.
These innovations offer several benefits. They enable any small model to work collaboratively with any large model, significantly broadening the practical applicability of speculative decoding. They reduce wasted computation on unlikely or misaligned predictions, increasing throughput without sacrificing output quality. Moreover, they maintain 100% accuracy of the output distribution while substantially speeding up generation, which is crucial for efficiently serving large language models at scale.
The new algorithms, including the Token-Level-Intersection (TLI), String-Level Exact Match (SLEM), and String-Level Rejection Sampling (SLRS), represent a significant advance in serving large LLMs efficiently. For instance, SLRS uses a generalized drafter that considers probabilities over strings rather than tokens, offering a new spin on speculative decoding that boosts token generation rates by up to 2.8x without requiring specialized draft models.
The research into these algorithms is ongoing, with the team also exploring ways to address the explosive growth of model vocabularies and make draft models even faster. The ultimate goal is to democratize access to faster and more cost-effective LLM serving, making these powerful tools more accessible to a wider range of users and applications.
- The Shared-Meaning Token Prioritization Algorithm, a part of the suite, directs the smaller drafter model to focus on tokens that have common meanings across models, ensuring the drafter's predictions align more reliably with the larger model's outputs.
- The new algorithms, including the Token-Level-Intersection (TLI), String-Level Exact Match (SLEM), and String-Level Rejection Sampling (SLRS), are designed to boost the efficiency of serving large language models (LLMs) and offer advancements in data-and-cloud-computing, machine learning, and artificial-intelligence.
- The groundbreaking development in serving LLMs, led by researchers from the Weizmann Institute, Intel Labs, and d-Matrix, aims to democratize access to faster and more cost-effective LLM serving, making these powerful tools more accessible to the broader technology community and various applications, enhancing the field of science and AI.
