Recent advances in AI have been greatly influenced by the Transformer architecture, a key component of large models across fields as diverse as language, vision, audio, and biology. However, the complexity of Transformer’s attention mechanism limits its application in processing long sequences. Even sophisticated models such as GPT-4 suffer from this limitation.
Breakthrough Advances with StripedHyena
To address these issues, Together Research recently open sourced StripedHyena, a language model that boasts a new architecture optimized for long contexts. StripedHyena can handle up to 128,000 tokens and has demonstrated improved performance over the Transformer architecture in both training and inference performance. It is the first model to have the performance of the best open source Transformer model for both short and long contexts. .
StripedHyena’s Hybrid Architecture
StripedHyena incorporates a hybrid architecture that combines multi-head, grouped query attention with gate convolution within hyena blocks. This design differs from traditional decoder-only Transformer models. Represent the convolution with a state-space model or truncated filter to decode it into a persistent memory of Hyena blocks. This architecture has lower latency, faster decoding, and higher throughput compared to Transformers.
Improve training and efficiency
StripedHyena improves performance by more than 30%, 50%, and 100% over existing Transformer in end-to-end training on 32k, 64k, and 128k token sequences, respectively. In terms of memory efficiency, it reduces memory usage during autoregressive generation by over 50% compared to Transformers.
Comparative performance using attention mechanisms
StripedHyena significantly reduces the quality gap through large-scale attention, reducing computational cost and providing similar disruption and downstream performance without the need for mixed attention.
Applications beyond language processing
StripedHyena’s versatility extends to image recognition. The researchers tested the applicability of Visual Transformers (ViT) to attention substitution and showed similar accuracy in an image classification task on the ImageNet-1k dataset.
StripedHyena represents an important advancement in AI architecture, providing a more efficient alternative to Transformer models, especially when processing long sequences. Its hybrid structure of training and inference and improved performance make it a promising tool for a wide range of applications in language and vision processing.
Image source: Shutterstock