{"id":"5a3768d4-bd2d-4863-b6e5-2fcd52ca77af","arxiv_id":"2412.14354","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Mamba-1 and Mamba-2 rerankers match comparably sized transformers on ranking accuracy but are less efficient in training and inference, with Mamba-2 improving on both fronts.","lead":"This paper benchmarks Mamba-style state space models against transformers for reranking search results. It finds Mamba models match transformer accuracy but run slower in practice, and Mamba-2 is both stronger and faster than Mamba-1.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining-token confound undermines the central architecture claim; add same-token decoder-only transformer baselines such as Pythia.","rationale":"The paper does substantial empirical work, releases code, and explicitly concedes the pretraining confound in its Limitations section. That concession is a point in its favor and narrows the issue: the architecture-level conclusion is conditionally supported, not established. The proposed Pythia comparison directly tests the load-bearing assumption because it controls for pretraining tokens and corpus while keeping the decoder-only family and the same fine-tuning pipeline. The efficiency findings and Mamba-2’s training-efficiency improvement are better supported, but the performance part of Mamba-2’s superiority is mixed in the BEIR and Dev tables and would also benefit from the controlled comparison or error bars. This aligns with the reader’s weakest_assumption and supports the existing CONDITIONAL verdict rather than changing it.","tokens_in":623,"tokens_out":4476,"duration_ms":86517,"concrete_test":"Fine-tune the same reranking pipeline on Pythia checkpoints (Pythia-160M, 410M, 1B, and 1.4B), which are decoder-only transformers pretrained on The Pile for 300B tokens, matching the Mamba pretraining budget and corpus more closely than the current transformer baselines. Use the released code and hyperparameters, the same BGE/BM25 negative sampling, and compare on MS MARCO Dev, DL19, DL20, and the 13 BEIR datasets. If similarly sized Pythia models match or exceed Mamba, the architecture-level performance claim is not supported; if Mamba still matches Pythia at equal token budget, the pretraining confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Mamba architectures are competitive text rerankers compared to transformer-based models of similar size (abstract claim 1; §4.2, §4.4) is supported by tables that compare models with very different pretraining budgets. Table 1 lists Mamba-1 and Mamba-2 at 300B tokens, while BERT/ELECTRA use 3.3B–33B, BART 33B, OPT 180B, and Llama-3.2 15T. Since reranking fine-tuning starts from these checkpoints, the observed ranking differences can be attributed to pretraining compute and data as much as to architecture. The paper’s Limitations section explicitly concedes this is not an apples-to-apples comparison. Without a same-token, same-family transformer baseline, the conclusion that SSMs are strong rerankers because of their architecture is not yet established. The efficiency claim is less affected because it is measured directly on the fine-tuned reranking task, and the Mamba-2-vs-Mamba-1 comparison does control for pretraining budget. However, the performance part of claim 3 is also inconsistent across sizes: for example, in Table 11 the BEIR average for Mamba-1-790M is 54.4 versus 53.9 for Mamba-2-780M, so the claim that Mamba-2 outperforms Mamba-1 in performance is not uniformly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmarking study of Mamba-1 and Mamba-2 state-space models as text rerankers, comparing them with transformer-based models (BERT, RoBERTa, ELECTRA, BART, OPT, Llama-3.2) across passage and document reranking. The authors fine-tune public checkpoints with a standard cross-encoder setup, evaluate on MS MARCO Dev, TREC DL19/DL20, and 13 BEIR datasets, and additionally measure training throughput, inference speed, and operator-level execution time. The paper claims that (1) Mamba models achieve competitive ranking performance compared with transformers of similar size, (2) they are less efficient than flash-attention transformers in training and inference, and (3) Mamba-2 outperforms Mamba-1 in both performance and efficiency.","tokens_in":23174,"tokens_out":6576,"duration_ms":51445,"significance":"The study is a useful and extensive empirical benchmark: it covers passage and document ranking, in-domain and out-of-domain evaluation, a wide range of parameter scales, training and inference throughput, and operator-level profiling. The code release and the use of public datasets and checkpoints are strengths, and all central claims are direct measurements rather than derived predictions, so there is no circularity. If the conclusions were fully supported, the paper would give the IR community a practical answer about whether SSM rerankers can substitute for transformers in the single-pass scoring setting. However, the headline architecture-level claims are currently stronger than the controlled comparisons and tables justify, and the Mamba-2-versus-Mamba-1 performance claim is not uniformly supported by the data.","major_comments":[{"comment":"The paper's central architecture-level claim that Mamba models are 'competitive, comparable to transformer-based models of similar size' is confounded by the pre-training token budget. Table 1 shows the Mamba checkpoints use 300B tokens, whereas the transformer counterparts use 3.3B–33B (BERT, ELECTRA, BART), 180B (OPT), or 15T (Llama-3.2) tokens. Since reranking fine-tuning starts from these checkpoints, the observed ranking differences can be attributed to pre-training data and compute as much as to architecture. The Limitations section explicitly concedes this is not apples-to-apples. In addition, Tables 9 and 10 show that global batch size and epochs differ across the compared models (for example, Mamba-1-130M uses batch size 8 while Mamba-2-130M uses batch size 4, and Mamba-1-790M trains for 1 epoch while smaller models train for 2 epochs), and the authors themselves note that batch size affects reranking quality. Please add same-token decoder-only transformer baselines such as Pythia, or equivalently reframe the claim as one about publicly available checkpoints rather than about the architecture per se.","section":"§4.1, Table 1, Limitations"},{"comment":"The claim that 'Mamba-2 outperforms Mamba-1 in both performance and efficiency' is not uniformly supported by the reported results. On the BEIR average in Table 11, Mamba-1-790M achieves 54.4 NDCG@10 versus Mamba-2-780M's 53.9, and Mamba-1-370M achieves 53.6 versus Mamba-2-370M's 53.0. In document reranking with the FirstP setting in Table 4, Mamba-1-370M beats Mamba-2-370M on Dev MRR@100 (42.5 vs 41.0) and DL19 NDCG@10 (67.8 vs 67.2). The performance superiority of Mamba-2 over Mamba-1 is therefore not a general result; it holds in the in-domain passage experiments but not in out-of-domain or long-document settings. Please either restrict the conclusion to the settings where it holds or provide significance tests or multiple-seed results that establish a consistent trend.","section":"§4.2, §4.4, Tables 11 and 4"},{"comment":"The efficiency claims are internally inconsistent with parts of the reported data. Section 4.4 states that Mamba-2 models 'in general require less GPU memory' during training, based on a FirstP run where Mamba-1-1.4B OOMs but Mamba-2-1.3B does not. However, the LongP block of Table 4 shows that both Mamba-1-1.4B and Mamba-2-1.3B OOM, and the Table 10 note confirms that Mamba-2-1.3B OOMs in the LongP setting despite all optimization techniques. The evidence supports only 'Mamba-2 fits in some configurations where Mamba-1 does not,' not a general memory-efficiency advantage. Similarly, the inference speed numbers in Table 5 do not uniformly support the abstract claim that Mamba models are less efficient than transformers with flash attention: Mamba-2-1.3B achieves 0.30 queries/second versus OPT-1.3B's 0.29 at length 512, and 0.29 versus 0.28 at length 1536. The efficiency conclusion is solid for Mamba-1 and for the 370M scale, but should be tempered for Mamba-2 at the 1.3B scale, and memory claims should report actual measured memory or be scoped to the configurations tested.","section":"§4.4, §4.5, Tables 4 and 5"}],"minor_comments":[{"comment":"The text says Llama-3.2-1B was 'trained on more tokens (15B)' but Table 1 lists the pre-training token count as 15T; please fix the unit.","section":"§4.2"},{"comment":"There are model-name inconsistencies in the document reranking table: 'Mamba1-790MD' lacks a hyphen, and 'Mamba-1-1.3BD' appears in the FirstP block while the model is called Mamba-1-1.4B in Table 1 and elsewhere. Please unify the names and sizes.","section":"Table 4"},{"comment":"The figure caption states that at batch size 8 all models except OPT-FlashAttn and Mamba-2 OOM with 48 GB VRAM, but the text says 'Mamba-1-370M does not train with batch size 8.' Please clarify whether the OOM statement applies to all Mamba-1 sizes and specify which model and batch size each throughput point corresponds to.","section":"§4.5, Figure 1"},{"comment":"The caption contains grammatical errors ('other models reranks') and mixes the reranking thresholds for RankLlama and the other models; please revise for clarity.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an empirical NLP/IR venue and the benchmark effort is substantial. The main risk is overclaiming: the architecture-level performance claim rests on pre-training token confounds, and the Mamba-2-versus-Mamba-1 and efficiency claims are contradicted by parts of the paper's own tables. These issues can be addressed by adding controlled baselines (e.g., Pythia at 300B tokens) and by carefully scoping the conclusions, so I do not see them as unfixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid, useful benchmark paper with an honest limitations section, but the headline that Mamba is competitive with transformers has to be read with pre-training token budgets in mind.\n\nWhat is actually new: Mamba-2 reranker results at four scales, document reranking with FirstP and LongP settings, BEIR out-of-domain evaluation, and operator-level profiling that explains why Mamba's asymptotic inference advantage does not show up in single-pass reranking. The benchmark is extensive and internally consistent, and the authors report hyperparameters, code, and checkpoints-to-come. That is real evidence and worth engaging with.\n\nThe efficiency claims are the strongest part. They are measured directly on the fine-tuned reranking task, and the Mamba-2-vs-Mamba-1 comparisons control for pre-training budget. Finding that Mamba is slower than transformers with flash attention in both training and inference, despite O(1) theory, is a useful empirical correction; the profiling shows the bottleneck is scalar-extraction ops in Mamba-1 and a still-unbalanced op mix in Mamba-2.\n\nThe soft spots are in the performance claims. Mamba checkpoints are 300B tokens, while BERT and ELECTRA are 3.3B-33B, OPT is 180B, and Llama-3.2 is 15T. So 'similar size' is not 'similar compute,' and the claim that Mamba architectures are competitive because of their architecture is not established. The Limitations section says this themselves, which is to their credit, but the abstract still overstates. Also, the memory-efficiency claim for Mamba-2 is over-generalized: Table 4 shows Mamba-2-1.3B OOM in LongP, same as Mamba-1, so 'requires less memory' only holds in the FirstP setting where they show it. And the Mamba-2-beats-Mamba-1 performance part is not uniform: on the BEIR average, Mamba-1-790M (54.4) edges Mamba-2-780M (53.9).\n\nNone of this is fatal. The benchmark has independent value as a public evidence base. But if I were refereeing, I would ask for same-token decoder-only transformer baselines such as Pythia to support the architecture claim, and ask them to qualify the memory and Mamba-2 performance generalizations. This paper deserves a serious referee. It is for IR researchers weighing architecture choices and for anyone working on SSM fine-tuning; a conditional accept with revisions seems about right.","headline":"Solid empirical benchmark of Mamba rerankers, but the 'competitive' claim is weaker than the title suggests once pretraining token budgets are accounted for.","tokens_in":23716,"tokens_out":2964,"would_cite":true,"duration_ms":22900,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba-based rerankers match transformer ranking quality at similar parameter counts, but trail transformers with Flash Attention in training and inference speed.","keywords":["state space models","Mamba","Mamba-2","text reranking","information retrieval","efficiency benchmark","Flash Attention","cross-encoder reranker"],"falsifier":"Train a Mamba and a transformer reranker from checkpoints pre-trained on the same corpus with matched token budgets and the same fine-tuning setup, then measure ranking metrics and per-query latency; if the ranking gap widens or the efficiency gap reverses under matched pre-training, the paper's architecture-level conclusions would not transfer, while if both hold, its claims are confirmed.","tokens_in":22702,"feed_emoji":"🔍","tokens_out":5990,"duration_ms":51675,"temperature":0.7,"pith_summary":"This paper asks whether state space models, specifically Mamba-1 and Mamba-2, can perform the fine-grained query-document interaction required by text reranking, a task transformers have dominated. It benchmarks rerankers built on Mamba and on transformer language models of similar size on MS MARCO passage and document ranking, plus BEIR out-of-domain sets. The central finding is threefold: Mamba rerankers match transformer rerankers of comparable parameter count on ranking accuracy; they are slower in both training and single-pass inference than transformers using Flash Attention; and Mamba-2 improves on Mamba-1 on both axes. A sympathetic reader takes this as evidence that SSMs are a viable architectural alternative for reranking, with efficiency still the open gap.","feed_headline":"State space models rank text as well as transformers","feed_subtitle":"Mamba-2 matches transformer accuracy but trails Flash Attention on speed in single-pass reranking.","key_machinery":"The central objects are the Mamba selective state space models. Mamba-1 makes the SSM parameters input-dependent and uses a hardware-aware selective scan, compressing context into a hidden state of size N; Mamba-2 restricts the A matrix to a scalar times identity, introduces an SSM head dimension analogous to transformer heads, and uses the structured state space duality algorithm so computation runs through matrix multiplications. The comparison's operational machinery is the standard cross-encoder reranker: query and document are concatenated into one input, a linear layer scores the final token, and training uses a softmax loss over one positive and hard negatives sampled from a first-stage retriever. That setup lets the authors attribute differences in ranking quality and speed primarily to the backbone architecture.","core_discovery":"The paper's central claim is that Mamba-based language models, despite compressing context into a fixed-size recurrent state, can learn the query-document interactions needed for text reranking and reach ranking quality comparable to transformer-based rerankers of similar scale. In passage reranking, Mamba-2-370M scores close to BERT-large on in-domain sets, and Mamba-2-1.3B averages 53.6 NDCG@10 across 13 BEIR datasets, slightly ahead of OPT-1.3B's 52.7. In document reranking, the best sub-1-billion-parameter model is the 780M Mamba-2 in the long-context setting. The paper also claims that the theoretical O(1) inference advantage of SSMs does not materialize on this task: Mamba models are slower in measured queries per second and substantially slower to train than transformers with Flash Attention, because reranking needs a single forward pass and Mamba's operators are less I/O-efficient. Within the SSM family, Mamba-2 outperforms Mamba-1 in both ranking performance and efficiency.","pith_inferences":["Because the comparison mixes pre-training budgets, the ranking parity result is most conservatively read as 'SSMs can be fine-tuned into competitive rerankers,' not as proof that architecture alone determines ranking quality; an apples-to-apples pre-training study could change the ranking comparison.","The profiling result suggests a concrete optimization target: if Mamba-1's scalar-extraction operations and Mamba-2's MambaSplitConv1D-dominated load are replaced by fused tensor kernels, the measured inference gap may close independently of architecture choice.","The same benchmarking template could be applied to dense retrieval with SSM encoders and to hybrid Mamba-transformer models, where the efficiency result may differ because retrieval and generation workloads have different forward-pass profiles."],"forward_implications":["Mamba-2 or similar SSMs could serve as the backbone of production rerankers when matching transformer ranking quality at a given parameter budget is the goal.","The advertised inference benefit of SSMs does not apply to reranking's single-pass workload, so IR deployments should expect SSM latency to be worse than Flash-Attention transformers until I/O optimizations improve.","Mamba-2's better training memory footprint lets larger SSM rerankers fit on the same GPU, as seen when Mamba-1 runs out of memory at 1.3B parameters while Mamba-2 trains.","Hybrid transformer-SSM models are a natural next step, since pure SSMs close the quality gap but not the efficiency gap in this workload."],"supporting_citations":[{"why":"Defines Mamba-1, the selective state space model architecture that is the primary object being benchmarked.","marker":"Gu and Dao, 2023"},{"why":"Defines Mamba-2 and the structured state space duality algorithm, plus the complexity analysis the efficiency comparison relies on.","marker":"Dao and Gu, 2024"},{"why":"Flash Attention 2 is the I/O-aware transformer optimization that serves as the efficiency baseline in training throughput and inference speed.","marker":"Dao, 2024"},{"why":"Supplies the RankLlama training methodology and the state-of-the-art reranking baseline the paper compares against.","marker":"Ma et al., 2023"},{"why":"Provides the established practice of sampling hard negatives from first-stage retrieval results that the reranker training setup follows.","marker":"Gao et al., 2021"},{"why":"Contributes the FirstP, MaxP, and LongP long-document reranking baselines and evaluation setup used for document ranking.","marker":"Boytsov et al., 2022"},{"why":"Prior empirical study of Mamba-based language models that shapes the paper's efficiency expectations and comparison points.","marker":"Waleffe et al., 2024"}],"fun_headline_variants":["Mamba matches transformers on reranking, but slower","SSMs rerank on par with transformers, but no speed win","State space models hold their own in text reranking","Mamba-2 catches up to BERT on reranking, speed lags","Reranking: SSMs match transformers, but not efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that publicly released checkpoints with very different pre-training token budgets and objectives, such as Mamba at roughly 300B tokens versus BERT at 3.3B-33B and Llama-3.2 at 15T tokens, can be compared as representative instantiations of their architectures, so observed ranking and efficiency differences are read as architectural rather than as artifacts of pre-training compute.","fun_headline_variants_meta":{"raw":{"variants":["Mamba matches transformers on reranking, but slower","SSMs rerank on par with transformers, but no speed win","State space models hold their own in text reranking","Mamba-2 catches up to BERT on reranking, speed lags","Reranking: SSMs match transformers, but not efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1379,"prompt_tokens":960,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":576,"tokens_out":419,"duration_ms":3606,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:18:31.682230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a Mamba and a transformer reranker from checkpoints pre-trained on the same corpus with matched token budgets and the same fine-tuning setup, then measure ranking metrics and per-query latency; if the ranking gap widens or the efficiency gap reverses under matched pre-training, the paper's architecture-level conclusions would not transfer, while if both hold, its claims are confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the established practice of sampling hard negatives from first-stage retrieval results that the reranker training setup follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the FirstP, MaxP, and LongP long-document reranking baselines and evaluation setup used for document ranking."}],"review_version":1}