# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:
⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâto refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.
We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ
(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.
However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâ
to refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬
of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
{chunk_text}
'\n", + "html_str += '{chunk_text}
'\n", + "html_str += '{chunk_text}
'\n", - "html_str += '{chunk_text}
'\n", - "html_str += '# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:
⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâto refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.
We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ
(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.
However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâ
to refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬
of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
{chunk_text}
'\n", - "html_str += \"{chunk_text}
'\n", - "html_str += \"# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:
⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâto refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
# Mamba:Linear-Time Sequence Modeling with Selective State Spaces# Albert Gu*1 and Tri Dao*21Machine Learning Department, Carnegie Mellon University 2Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me# AbstractFoundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformersâcomputational ineï¬ciency on long sequences, but they have not performed as well as attention on important modalities such as language.We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements.First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token.Second, even though this change prevents the use of eï¬cient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.We integrate these selective SSMs into a simpliï¬ed end-to-end neural network architecture without attention or even MLP blocks (Mamba).Mamba enjoys fast inference (5Ãhigher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.# 1 IntroductionFoundation models (FMs), or large models pretrained on massive data then adapted for downstream tasks, have emerged as an eï¬ective paradigm in modern machine learning.The backbone of these FMs are often sequence models, operating on arbitrary sequences of inputs from a wide variety of domains such as language, images, speech, audio, time series, and genomics (Brown et al. 2020; Dosovitskiy et al. 2020; Ismail Fawaz et al. 2019; Oord et al. 2016; Poli et al. 2023; Sutskever, Vinyals, and Quoc V Le 2014).While this concept is agnostic to a particular choice of model architecture, modern FMs are predominantly based on a single type of sequence model: the Transformer (Vaswani et al. 2017) and its core attention layer (Bahdanau, Cho, and Bengio 2015) The eï¬cacy of self-attention is attributed to its ability to route information densely within a context window, allowing it to model complex data.
However, this property brings fundamental drawbacks: an inability to model anything outside of a ï¬nite window, and quadratic scaling with respect to the window length.An enormous body of research has appeared on more eï¬cient variants of attention to overcome these drawbacks (Tay, Dehghani, Bahri, et al. 2022), but often at the expense of the very properties that makes it eï¬ective.As of yet, none of these variants have been shown to be empirically eï¬ective at scale across domains.
Recently, structured state space sequence models (SSMs) (Gu, Goel, and Ré 2022; Gu, Johnson, Goel, et al. 2021) have emerged as a promising class of architectures for sequence modeling.These models can be interpreted as a combination of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), with inspiration from classical state space models (Kalman 1960).This class of models can be computed very eï¬ciently as either a recurrence or convolution, with linear or near-linear scaling in sequence length.
Additionally, they have principledEqual contribution.1mechanisms for modeling long-range dependencies (Gu, Dao, et al. 2020) in certain data modalities, and have dominated benchmarks such as the Long Range Arena (Tay, Dehghani, Abnar, et al. 2021).Many ï¬avors of SSMs (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Y.Li et al. 2023; Ma et al. 2023; Orvieto et al. 2023; Smith, Warrington, and Linderman 2023) have been successful in domains involving continuous signal data such as audio and vision (Goel et al. 2022; Nguyen, Goel, et al. 2022; Saon, Gupta, and Cui 2023).However, they have been less eï¬ective at modeling discrete and information-dense data such as text.We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.
Selection Mechanism.First, we identify a key limitation of prior models: the ability to eï¬ciently select data in an input-dependent manner (i.e. focus on or ignore particular inputs).Building on intuition based on important synthetic tasks such as selective copy and induction heads, we design a simple selection mechanism by parameterizing the SSM parameters based on the input.This allows the model to ï¬lter out irrelevant information and remember relevant information indeï¬nitely.Hardware-aware Algorithm.This simple change poses a technical challenge for the computation of the model; in fact, all prior SSMs models must be time- and input-invariant in order to be computationally eï¬cient.We overcome this with a hardware-aware algorithm that computes the model recurrently with a scan instead of convolution, but does not materialize the expanded state in order to avoid IO access between diï¬erent levels of the GPU memory hierarchy.The resulting implementation is faster than previous methods both in theory (scaling linearly in sequence length, compared to pseudo-linear for all convolution-based SSMs) and on modern hardware (up to 3Ãfaster on A100 GPUs).
Architecture.We simplify prior deep sequence model architectures by combining the design of prior SSM architectures (Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenous architecture design (Mamba) incorporating selective state spaces.Selective SSMs, and by extension the Mamba architecture, are fully recurrent models with key properties that make them suitable as the backbone of general foundation models operating on sequences. (i) High quality: selectivity brings strong performance on dense modalities such as language and genomics. (ii) Fast training and inference: computation and memory scales linearly in sequence length during training, and unrolling the model autoregressively during inference requires only constant time per step since it does not require a cache of previous elements. (iii) Long context: the quality and eï¬ciency together yield performance improvements on real data up to sequence length 1M.
We empirically validate Mambaâs potential as a general sequence FM backbone, in both pretraining quality and domain-speciï¬c task performance, on several types of modalities and settings:⢠Synthetics.On important synthetic tasks such as copying and induction heads that have been proposed as being key to large language models, Mamba not only solves them easily but can extrapolate solutions indeï¬nitely long (>1M tokens).⢠Audio and Genomics.Mamba out-performs prior state-of-the-art models such as SaShiMi, Hyena, and Transform- ers on modeling audio waveforms and DNA sequences, both in pretraining quality and downstream metrics (e.g. reducing FID on a challenging speech generation dataset by more than half).In both settings, its performance improves with longer context up to million-length sequences.⢠Language Modeling.
Mamba is the ï¬rst linear-time sequence model that truly achieves Transformer-quality performance, both in pretraining perplexity and downstream evaluations.With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).Our Mamba language model has 5Ãgeneration throughput compared to Transformers of similar size, and Mamba-3Bâs quality matches that of Transformers twice its size (e.g. 4 points higher avg. on common sense reasoning compared to Pythia-3B and even exceeding Pythia-7B).Model code and pre-trained checkpoints are open-sourced at https://github.com/state-spaces/mamba.
2# Selective State Space Model# with Hardware-aware State Expansion# Avuvy GPU SRAM Selection Mechanism esSelection MechanismFigure 1: (Overview.) Structured SSMs independently map each channel (e.g. ð· = 5) of an input ð¥ to output ð¦ through a higher dimensional latent state â(e.g. ð= 4).Prior SSMs avoid materializing this large effective state (ð·ð, times batch size ðµ and sequence length ð¿) through clever alternate computation paths requiring time-invariance: the (â, A, B, C) parameters are constant across time.Our selection mechanism adds back input-dependent dynamics, which also requires a careful hardware-aware algorithm to only materialize the expanded states in more efficient levels of the GPU memory hierarchy.# 2 State Space ModelsStructured state space sequence models (S4) are a recent class of sequence models for deep learning that are broadly related to RNNs, and CNNs, and classical state space models.They are inspired by a particular continuous system (1) that maps a 1-dimensional function or sequence ð
¥(ð¡) ââ⦠ð¦(ð¡) ââthrough an implicit latent state â(ð¡) ââð.Concretely, S4 models are deï¬ned with four parameters (â, A, B, C), which deï¬ne a sequence-to-sequence trans- formation in two stages.ââ²(ð¡) = Aâ
(ð¡) + Bð¥(ð¡) ð¦(ð¡) = Câ(ð¡)(1a) (1b) âð¡ = Aâð¡â1 + Bð¥ð¡ ð¦ð¡ = Câð¡ (2a) (2b) ðð² = (Cð©, Cð¨ð©, ⦠, Cð¨ ð¦ = ð¥ âð² ð©, â
¦ ) (3a) (3b)Discretization.The ï¬rst stage transforms the âcontinuous parametersâ(â, A, B) to âdiscrete parametersâ(A, B) through ï¬xed formulas A = ðð´(â, A) and B = ððµ(â, A, B), where the pair (ðð´, ððµ) is called a discretization rule.Various rules can be used such as the zero-order hold (ZOH) deï¬
ned in equation (4).A = exp(âA) B = (âA)â1(exp(âA) âI) ââB (4)Discretization has deep connections to continuous-time systems which can endow them with additional properties such as resolution invariance (Nguyen, Goel, et al. 2022) and automatically ensuring that the model is properly normalized (Gu, Johnson, Timalsina, et al. 2023; Orvieto et al. 2023).
It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.
However, from a mechanical point of view discretization can simply be viewed as the ï¬rst step of the computation graph in the forward pass of an SSM.Alternate ï¬avors of SSMs can bypass the discretization step and parameterize (A, B) directly instead (Zhang et al. 2023), which may be easier to reason about.Computation.After the parameters have been transformed from (â, A, B, C) ⦠(A, B, C), the model can be computed in two ways, either as a linear recurrence (2) or a global convolution (3).3Commonly, the model uses the convolutional mode (3) for eï¬cient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for eï¬cient autoregressive inference (where the inputs are seen one timestep at a time).Linear Time Invariance (LTI).An important property of equations (1) to (3) is that the modelâs dynamics are constant through time.
In other words (â, A, B, C), and consequently (A, B) as well, are ï¬xed for all time-steps.This property is called linear time invariance (LTI), which is deeply connected to recurrence and convolutions.Informally, we think of LTI SSMs as being equivalent to any linear recurrence (2a) or convolution (3b), and use LTI as an umbrella term for these classes of models.Thus far, all structured SSMs have been LTI (e.g. computed as convolutions) because of fundamental eï¬ciency constraints, discussed in Section 3.3.However, a core insight of this work is that LTI models have fundamental limitations in modeling certain types of data, and our technical contributions involve removing the LTI constraint while overcoming the eï¬ciency bottlenecks.
Structure and Dimensions.Finally, we note that structured SSMs are so named because computing them eï¬ciently also requires imposing structure on the A matrix.The most popular form of structure is diagonal (Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Smith, Warrington, and Linderman 2023), which we also use.In this case, the A ââðÃð, B ââðÃ1, C ââ1Ãðmatrices can all be represented by ðnumbers.To operate over an input sequence ð¥ of batch size ðµ and length ð¿ with ð
· channels, the SSM is applied independently to each channel.Note that in this case, the total hidden state has dimension ð·ðper input, and computing it over the sequence length requires ð(ðµð¿ð·ð) time and memory; this is the root of the fundamental eï¬ciency bottleneck addressed in Section 3.3.General State Space Models.We note that the term state space model has a very broad meaning which simply represents the notion of any recurrent process with a latent state.It has been used to refer to many disparate concepts in diï¬erent disciplines, including Markov decision processes (MDP) (reinforcement learning (Hafner et al. 2020)), dynamic causal modeling (DCM) (computational neuroscience (Friston, Harrison, and Penny 2003)), Kalman ï¬lters (controls (Kalman 1960)), hidden Markov models (HMM) and linear dynamical systems (LDS) (machine learning), and recurrent (and sometimes convolutional) models at large (deep learning).Throughout this entire paper we use the term â
SSMâ
to refer exclusively to the class of structured SSMs or S4 models (Gu, Goel, and Ré 2022; Gu, Gupta, et al. 2022; Gupta, Gu, and Berant 2022; Hasani et al. 2023; Ma et al. 2023; Smith, Warrington, and Linderman 2023) and use these terms interchangeably.For convenience we may also include derivatives of such models, such as those focusing on either the linear-recurrence or global-convolution viewpoints (Y.Li et al. 2023; Orvieto et al. 2023; Poli et al. 2023), and clarify nuances when necessary.SSM Architectures.SSMs are standalone sequence transformations that can be incorporated into end-to-end neural network architectures. (We also sometimes call SSM architectures SSNNs, which are to SSM layers as CNNs are to linear convolution layers.) We discuss some of the most well-known SSM architectures, many of which will also serve as our primary baselines.
⢠Linear attention (Katharopoulos et al. 2020) is an approximation of self-attention involving a recurrence which can be viewed as a degenerate linear SSM.⢠H3 (Dao, Fu, Saab, et al. 2023) generalized this recurrence to use S4; it can be viewed as an architecture with an SSM sandwiched by two gated connections (Figure 3).H3 also inserts a standard local convolution, which they frame as a shift-SSM, before the main SSM layer.⢠Hyena (Poli et al. 2023) uses the same architecture as H3 but replaces the S4 layer with an MLP-parameterized global convolution (Romero et al. 2021).â
¢ RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.4⢠RWKV (B.Peng et al. 2023) is a recent RNN designed for language modeling based on another linear attention approximation (attention-free Transformer (S.Zhai et al. 2021)).Its main âWKVâ
mechanism involves LTI recurrences and can be viewed as the ratio of two SSMs.Other closely related SSMs and architectures are discussed further in an extended related work (Appendix B).We highlight in particular S5 (Smith, Warrington, and Linderman 2023), QRNN (Bradbury et al. 2016), and SRU (Lei et al. 2017), which we view as the most closely related methods to our core selective SSM.# 3 Selective State Space ModelsWe motivate our selection mechanism using intuition from synthetic tasks (Section 3.1), then explain how to incorporate this mechanism into state space models (Section 3.2).The resulting time-varying SSMs cannot use convolutions, presenting a technical challenge of how to compute them eï¬
ciently.We overcome this with a hardware-aware algorithm that exploits the memory hierarchy on modern hardware (Section 3.3).We then describe a simple SSM architecture without attention or even MLP blocks (Section 3.4).Finally, we discuss some additional properties of selection mechanisms (Section 3.5).# 3.1 Motivation:Selection as a Means of CompressionWe argue that a fundamental problem of sequence modeling is compressing context into a smaller state.In fact, we can view the tradeoï¬s of popular sequence models from this point of view.For example, attention is both eï¬ective and ineï¬cient because it explicitly does not compress context at all.This can be seen from the fact that autoregressive inference requires explicitly storing the entire context (i.e. the KV cache), which directly causes the slow linear-time inference and quadratic-time training of Transformers.On the other hand, recurrent models are eï¬cient because they have a ï¬nite state, implying constant-time inference and linear-time training.However, their eï¬ectiveness is limited by how well this state has compressed the context.To understand this principle, we focus on two running examples of synthetic tasks (Figure 2).
⢠The Selective Copying task modiï¬es the popular Copying task (Arjovsky, Shah, and Bengio 2016) by varying the position of the tokens to memorize.It requires content-aware reasoning to be able to memorize the relevant tokens (colored) and ï¬lter out the irrelevant ones (white).⢠The Induction Heads task is a well-known mechanism hypothesized to explain the majority of in-context learning abilities of LLMs (Olsson et al. 2022).It requires context-aware reasoning to know when to produce the correct output in the appropriate context (black).These tasks reveal the failure mode of LTI models.From the recurrent view, their constant dynamics (e.g. the (A, B) transitions in (2)) cannot let them select the correct information from their context, or aï¬ect the hidden state passed along the sequence an in input-dependent way.From the convolutional view, it is known that global convolutions can solve the vanilla Copying task (Romero et al. 2021) because it only requires time-awareness, but that they have diï¬culty with the Selective Copying task because of lack of content-awareness (Figure 2).More concretely, the spacing between inputs-to-outputs is varying and cannot be modeled by static convolution kernels.
In summary, the eï¬ciency vs. eï¬ectiveness tradeoï¬
of sequence models is characterized by how well they compress their state: eï¬cient models must have a small state, while eï¬ective models must have a state that contains all necessary information from the context.In turn, we propose that a fundamental principle for building sequence models is selectivity: or the context-aware ability to focus on or ï¬lter out inputs into a sequential state.In particular, a selection mechanism controls how information propagates or interacts along the sequence dimension (see Section 3.5 for more discussion).
# Improving SSMs with SelectionOne method of incorporating a selection mechanism into models is by letting their parameters that aï¬ect interactions along the sequence (e.g. the recurrent dynamics of an RNN or the c
{chunk_text}
'\n", + "html_str += \"{chunk_text}
'\n", + "html_str += \"