Computing-in-Memory Transformer Accelerator

What is it about?

Transformer models have become the cornerstone of modern artificial intelligence for their ability to process sequential data with self-attention. Attention operations and large-scale matrix multiplications incur high latency and energy consumption. Hybrid computing-in-memory (CIM) architectures that integrate ReRAM and SRAM have shown great potential for Transformer acceleration. However, the inherent sparsity in the self-attention computation leads to redundant computation. Exploiting this sparsity often introduces significant overhead in Top-K query-key identification or causes accuracy degradation due to information loss from unselected keys in prior CIM accelerators. In this work, we propose HP-CIM, a hybrid CIM accelerator that employs a ReRAM-based hash predictor (ReHP) to efficiently leverage attention sparsity. The ReHP utilizes the intrinsic conductance variation of ReRAM cells for low-cost hash projection and integrates a ReRAM contentaddressable memory (CAM) with a K-winner-take-all (K-WTA) circuit for highly parallel, energy-efficient Top-K matching. Furthermore, an optimizable bias-filling strategy is introduced to restore model accuracy through a learnable bias that compensates for information loss during hash prediction. Experimental demonstration using multiple Transformer models with different tasks shows that HP-CIM achieves 9.05∼310.04× higher energy efficiency and 2.48∼16.93× speedup over state-of-the-art CIM-based Transformer accelerators.

Featured Image

Photo by Ecliptic Graphic on Unsplash

Photo by Ecliptic Graphic on Unsplash

Why is it important?

A new in-memory architecture Transformer accelerator has been proposed, which is beneficial for accelerating the deployment of edge-side AI models.

Perspectives