Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
Home»ADOPTION NEWS»TEAL, Introducing Training-Free Activation Sparsity to Improve LLM Efficiency
ADOPTION NEWS

TEAL, Introducing Training-Free Activation Sparsity to Improve LLM Efficiency

By Crypto FlexsSeptember 1, 20243 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
TEAL, Introducing Training-Free Activation Sparsity to Improve LLM Efficiency
Share
Facebook Twitter LinkedIn Pinterest Email

Jack Anderson
September 1, 2024 08:34

TEAL provides a learning-free approach to activation sparsity that significantly improves the efficiency of large-scale language models (LLMs) with minimal degradation.





TEAL (Training-Free Activation Sparsity in LLMs) has emerged as a groundbreaking approach to improve the efficiency of large-scale language models (LLMs) without additional training. According to together.ai, the method achieves 40-50% activation sparsity with minimal degradation by applying size pruning to the hidden state throughout the model. This innovation allows transferring fewer weights to on-chip memory, solving the memory-bound nature of LLM inference and translating into a 1.53-1.8x wall-clock speedup in single-batch decoding.

background

LLM is known for its enormous size, which makes it difficult during inference, mainly due to the speed limitation of transferring parameters from device memory to registers. Various techniques such as quantization, weight sparsity, and speculative decoding have been developed to address this ‘memory wall’. Activation sparsity, which utilizes zero values ​​in the hidden state, is a less explored method that avoids transferring unnecessary weight channels during decoding.

Older models like OPT-175B exhibit high activation sparsity, allowing significant speedups with methods like DejaVu. However, newer models like LLaMA have moved to SwiGLU variants, making these methods difficult to apply. Recent studies have attempted to ‘recover’ models that exhibit activation sparsity, but these models require extensive retraining on large datasets.

Motivational Research: Activation Distribution Characteristics of LLM

Studies have shown that the hidden states of LLM are outliers, zero-centered, and have similar distribution shapes across layers. Specifically, the states before MLP and Attention Blocks are Gaussian in shape, and the intermediate states are Laplacian in shape. This suggests that many low-amplitude activations can be eliminated with negligible model degradation, a notion also observed in other studies such as CATS.

teal

TEAL introduces optimizations by sparsifying all tensors in the model, achieving near-zero degradation at 25% sparsity and minimal degradation at 40% sparsity. At 50% sparsity, the Llama-3 variant shows slightly more degradation than its predecessors Llama-2 and Mistral. TEAL outperforms CATS by sparsifying all tensors and producing lower error by sparsifying the input.

Improved hardware recognition speed

To benchmark real-world speedups, TEAL is integrated with GPT-Fast, achieving significant speedups of up to 1.53x and 1.8x at 40% and 50% sparsity, respectively. The kernel is faster than cuBLAS at 0% sparsity, but there is still room for further optimization.

Compatibility with quantization

TEAL also demonstrates compatibility with quantization, another technique for efficient LLM inference. Combining activation sparsity and quantization opens up a new regime for transferring memory to GPU registers, leading to faster inference speeds.

Application

The most immediate application of TEAL is to accelerate inference in resource-constrained edge settings, especially in single-batch scenarios. It also enables inference providers like Together AI, which hosts over 100 open-source models on large fleets of GPUs, to serve their models more efficiently.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

AAVE Price Prediction: $100 is the wall. Factors that can destroy or bury a wall include:

July 25, 2026

Multicoin Capital has made its first Hyperliquid ecosystem investment in Trasia, an Asia-focused trading platform.

July 17, 2026

Polymarket Probability Price The probability that the United States will invade Iran before 2027 is 16.5%.

July 9, 2026
Add A Comment

Comments are closed.

Recent Posts

BC Game Launches 100 Million $BC Token Bonus Campaign Through Its Live-Market Crashx Game

November 20, 2026

How endpoint monitoring protects the people behind a crypto project

November 20, 2026

Bybit’s Global User Base Surpasses 90 Million as Company Expands Into Traditional Finance

November 20, 2026

Bybit Joins Proof of Talk in Abu Dhabi 2026 as Title Partner and Official Exchange Partner

November 19, 2026

Hashi Mainnet is Live With 20-Plus Partners, Bringing Bitcoin Finance to Sui at Scale

November 19, 2026

MEXC’s Hacken-Audited November 2026 Proof of Reserves Shows Reserve Ratios Above 100% Across Major Assets

November 19, 2026

My Wallet Introduces 1:1 Cross-Chain Stablecoin Swaps After Being Listed First on WalletConnect

November 19, 2026

The Promo Gaming -Discover Great Sportsbook & Casino Bonuses from Reliable Online Gaming Platforms

November 18, 2026

How to Read Online Gambling Statistics Without Being Misled

November 18, 2026

Tria and Noah Bring Bank Transfers Directly to Self-Custodial Wallets With Automatic EUR-to-USDC Conversion

November 18, 2026

Ondo Stocks to Launch ANTon with 24/7 Mint and Redeem, Live on IPO Day

November 18, 2026

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

BC Game Launches 100 Million $BC Token Bonus Campaign Through Its Live-Market Crashx Game

November 20, 2026

How endpoint monitoring protects the people behind a crypto project

November 20, 2026

Bybit’s Global User Base Surpasses 90 Million as Company Expands Into Traditional Finance

November 20, 2026
Most Popular

Mark Cuban announces Trump-inspired meme coin to pay off US debt

January 21, 2025

AriCoin: The next big thing in cryptocurrency? Find out why investors are flocking to this new coin – The Defi Info

January 9, 2024

Is BitVM the next evolution of Bitcoin smart contracts?

December 31, 2023
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2026 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.