Pith. sign in

REVIEW 3 major objections 4 minor 66 references

On the Scaling of Robustness and Effectiveness in Dense Retrieval

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dense retrieval robustness follows a power law.

desk verdict First quantitative scaling laws for robustness in dense retrieval with a Pareto-frontier framing, but the headline efficiency gains are extrapolations from BERT-scale fits and the Pareto-training validation is partly circular; still worth a serious referee. read the letter →

arxiv 2505.24279 v1 pith:QMDXBITR submitted 2025-05-30 cs.IR

Yu-An Liu , Ruqing Zhang , Jiafeng Guo , Maarten de Rijke , Yixing Fan , Xueqi Cheng This is my paper
classification cs.IR
keywords DenseretrievalRobustnessEffectivenessNeuralscalinglawParetoefficiencyAdversarialOut-of-distributionResourceallocation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether robustness in dense retrieval--the ability to keep ranking well on out-of-distribution data and under adversarial attacks--grows with model size and training data in the same predictable way that effectiveness does. The authors find that it does: both out-of-distribution and, after adversarial training, adversarial robustness follow power-law scaling curves. They also find that robustness and effectiveness scale differently, with robustness responding more to data size and effectiveness more to model size, so improving both at once through scaling alone is prohibitively expensive. The constructive claim is that the optimization objective is a third scaling axis: rebalancing the loss between robustness and effectiveness during training traces a Pareto frontier, and their 'Pareto training' method approaches that frontier, matching joint performance that would otherwise require several-fold more resources.

What carries the argument

Two objects carry the argument: the power-law scaling laws of Eqs. 7 and 8, measured through contrastive entropy, and the Pareto frontier of joint performance obtained by sweeping optimization strategies at fixed model and data sizes. Pareto training optimizes $\mathcal{L}_{\text{Pareto}}(\theta) = \omega \ell_R(\theta) + (1-\omega) \ell_E(\theta)$, updating the weight each step as $\omega_{t+1} = \omega_t - \eta(\ell_E^t/\ell_R^t - 1/\omega_0)$, clipped to $[0,1]$, following distributionally robust optimization. The joint data-model scaling law of Eq. 12 combines the single-factor laws additively, and the cost model of Eq. 13 converts that into budget-aware predictions for deployment.

What would settle it

Train a dense retrieval model outside the fitted range--say several billion parameters or several million training pairs--and compare the measured contrastive entropy on BEIR and under adversarial attacks with the curve extrapolated from Eqs. 7 and 8; if the gap grows systematically with scale, the power law bends and the extrapolated efficiency claims fail. A second check is to reproduce Pareto training at that larger scale and see whether the reported roughly 2.5x scaling-efficiency advantage over standard training persists.

Watch

Extended reading notes

Core claim

The central discovery is a pair of scaling laws plus a training method. Robustness follows the same power-law family as effectiveness when measured by contrastive entropy, a retrieval-specific metric in which lower values mean better ranking: loss scales as $(M/|f|)^\mu + \delta_f$ with model size and as $(D/|\mathcal{D}_{\text{train}}|)^\eta + \delta_{\mathcal{D}_{\text{train}}}$ with data size. Effectiveness is more sensitive to model size, while robustness is more sensitive to data size, and this asymmetry means jointly scaling both dimensions is resource-hungry--the paper estimates that a 10% joint improvement over a 7B-parameter state-of-the-art model would need a 175B-parameter model and ten times more data. Holding model and data fixed, different optimization strategies (standard, hard-negative, denoising, and adversarial training) trace a Pareto frontier between robustness and effectiveness, and most strategies sit away from it. Pareto training adaptively reweights the robustness and effectiveness losses each training step, using a distributionally robust optimization-style update, so the joint objective progresses toward the fitted frontier; the paper reports up to about 2.5x scaling-efficiency improvement and equivalence to several-fold resource scaling under unbalanced strategies.

Load-bearing premise

The joint scaling law of Eq. 12, which combines model and data scaling as an additive power law, is assumed without derivation, and the exponents fitted at BERT scale (up to 82M parameters and 480K training pairs) are extrapolated to models of 7B to 175B parameters and data scaled tenfold to support the 'several times' efficiency claim.

Editorial extensions

If this is right

  • Practitioners can predict robustness gains from model and data scaling before training, since out-of-distribution robustness follows the same power-law family as effectiveness.
  • Under a fixed budget, allocating resources should be asymmetric: data buys robustness, model size buys effectiveness, and over-scaling model size can hurt robustness as data budget shrinks.
  • Pareto training provides a resource-free lever: at the same data and compute as standard training, joint robustness-effectiveness performance can match several-fold scaling under unbalanced strategies.
  • The reported scaling-efficiency improvement of about 2.5x implies that rebalancing the optimization objective can be more cost-effective than increasing model or data size for realistic deployment budgets.
  • The extrapolation example--10% joint improvement over a 7B model needing 175B parameters and 10x data under naive scaling--frames the practical motivation for balanced training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the robustness scaling exponents hold across other encoder families, data-centric design may be a cheaper route to robustness than architecture scaling; one testable extension is measuring exponents for multi-vector and interaction-based retrievers.
  • The Pareto frontier is fitted from only four optimization strategies at BERT scale; whether its shape stabilizes for larger models and other objectives such as distillation or domain-adaptive training is an open, testable question.
  • The Pareto-training weight update depends on the ratio of robustness to effectiveness loss, so a similar adaptive-reweighting recipe might transfer to other multi-objective retrieval settings, including efficiency versus effectiveness or relevance versus diversity.
  • The cost model treats generated adversarial data as equivalent in cost to manual annotation; relaxing that assumption could shift the budget-allocation recommendations in deployment settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical study of scaling laws for two evaluation dimensions in dense retrieval: effectiveness (MS MARCO) and robustness (BEIR OOD plus adversarial attacks). Using BERT/ERNIE models from 0.5M to 87M non-embedding parameters and training sets from 30K to 480K pairs, it fits power laws of the form L = (M/|f|)^μ + δ_f and L = (D/|D|)^η + δ_D for both dimensions. It then argues that robustness is more sensitive to data size and effectiveness to model size, introduces a Pareto frontier over optimization strategies, and proposes Pareto training, which dynamically reweights robustness and effectiveness losses. The headline claims are that Pareto training improves scaling efficiency by up to 2.5x and, without extra resources, matches scaling up by several times.

Significance. The study is potentially valuable because robustness scaling laws in dense retrieval have not been systematically quantified; the reported R² values for OOD robustness are high, and the distinction between data-sensitive robustness and model-sensitive effectiveness is a useful observation. The practical resource-allocation analysis is interesting. However, the quantitative efficiency claims are predictions from an unvalidated joint scaling law and are evaluated with a partly circular procedure, so the paper's central contribution requires substantial strengthening before the quantitative conclusions can be accepted.

major comments (3)
  1. [§5.1, Eq. (12)] This equation is load-bearing for the resource-efficiency claims in §5.2 and Figure 6, but it is asserted as 'Following [6]' without derivation. It is not the sum of L(f) and L(Dtrain) from Eqs. (7)-(8), so its functional form needs justification or empirical validation. Moreover, the parameters in Table 4 are fitted on models up to 82M parameters and data up to 480K pairs, and the same equation is then used in §3.5 to make quantitative statements about Llama2Vec (7B) and GPT-4 (175B) and in §5.2 to claim 2.5x efficiency. The paper's own Limitation (vi) says validation on LLMs is future work; extrapolating two to three orders of magnitude beyond the fitted range without any held-out test makes those claims unsupported.
  2. [§4.2, Eqs. (9)-(10), Fig. 5] The empirical demonstration that Pareto training reaches the Pareto frontier is partly circular. The initial weight ω0 is computed from the ratio between robustness and effectiveness at the fitted Pareto frontier, and the same fitted frontier is used in Fig. 5 to show that Pareto training 'closely approaches' it. This does not validate the method as a way of discovering Pareto efficiency; it only shows that the training dynamics track the previously fitted ratio. An independent test, such as holding out optimization strategies from the frontier fit or evaluating on a validation set not used to compute the frontier, is needed.
  3. [§3 and Table 4] The scaling-law fits in Eqs. (7) and (8) are based on five model sizes and five data sizes per curve, and Table 4 estimates six parameters in the joint law. With this number of data points, high R² values alone provide limited statistical evidence; the paper should report confidence intervals, per-point residuals, or additional intermediate scales. This matters because the later claims about resource equivalence depend on the precise exponents.
minor comments (4)
  1. [§3.1] The text refers to 'MS MACRO' in the description of Figure 2(a); this should be 'MS MARCO'.
  2. [Eq. (12)] The coefficients M, D, μ, η, and δ are listed, but the relationship between δ and the irreducible losses δ_f and δ_D from Eqs. (7)-(8) is not defined; please clarify whether δ is a single fitted constant or derived from the earlier irreducible-loss terms.
  3. [Figure 2 and Section 3.1] The caption and text mention 'square points' for the official BERT checkpoints, but the figure legend does not clearly identify square markers; please align the legend and the text.
  4. [§5.2, Figure 6] The 2.5x scaling-efficiency claim is presented without sensitivity analysis; since Figure 6 is generated from Eq. (12) with fitted parameters, the paper should report how the ratio changes when the exponents are varied within their fitted uncertainty.

Circularity Check

2 steps flagged · score 6.0 of 10

Pareto-training validation is partly self-referential, and the headline 2.5×/several-times efficiency gain is a fit-derived prediction rather than an independent measurement.

  1. self definitional [Section 4.2 (Eqs. 9–11), Section 4.1 (Pareto efficiency), Figure 5]
    "To initialize the weight ω0 we calculate the ratio between robustness and effectiveness at Pareto efficiency. We jointly optimize the robustness and effectiveness objectives using the following loss: LPareto(θ)=ωℓ_R(θ)+(1−ω)ℓ_E(θ) ... In Figure 5 we observe that (i) Pareto training closely approaches Pareto efficiency in both model and data scaling. This indicates that our proposed training method can adaptively adjust the optimization direction, thereby improving the joint performance of robustness and effectiveness."

    The Pareto efficiency line is obtained by fitting the same robustness/effectiveness points of the tested optimization strategies at different weighting ratios (Section 4.1). Pareto training then sets its mixing weight ω0 to the robustness/effectiveness ratio at that fitted line, and Eq. 10 drives the loss ratio toward 1/ω0. Hence the target is baked into the initialization. Reporting that Pareto training 'closely approaches Pareto efficiency' and using this as 'indirectly validating the accuracy of the Pareto efficiency we have fitted' is circular: the fitted frontier supplies the hyperparameter and is then used as the success criterion. The independent content is the loss-to-performance mapping, but the claim that the front is validated by Pareto training is by construction.

  2. fitted input called prediction [Section 5.2, Eq. 12, Table 4, Figure 6]
    "By comparison, due to its ability to achieve efficient scaling, Pareto training exhibits a scaling efficiency improvement of about 2.5 times compared to standard training when the budget is 5,000. Figure 6: Predicted contrastive entropy of robustness and effectiveness for standard and Pareto training under limited budgets."

    Table 4's coefficients for Eq. 12 are numerically estimated from the same English-benchmark standard/Pareto-training runs, and Figure 6 is produced by plugging those coefficients into Eq. 12 under the budget model Eq. 13. The 2.5× factor is not an independent empirical result but a ratio of the two fitted scaling laws, labeled 'Predicted' without any held-out budget experiment or LLM-scale check. The paper's Limitations (vi) concedes that validation on large language models is future work, so the 'several times' headline rests on fit extrapolation, not measurement.

full rationale

The core scaling-law measurements for OOD and adversarial robustness (Sections 3.1–3.4) are empirical fits against external benchmarks (MS MARCO, BEIR, T2Ranking, BCIR); they are self-contained and not circular. The functional forms follow [6] (Fang et al., not the present authors), so this is borrowed ansatz, not self-citation. The circularity lies in the Pareto-training evaluation: the Pareto front is fitted from the tested optimization strategies, and Pareto training initializes ω0 from that fitted front, so the statement that Pareto training 'closely approaches Pareto efficiency' is partly self-fulfilling. The quantitative efficiency claims (2.5×, 'several times') are then computed from Eq. 12 with Table 4 parameters fitted to the same BERT-scale runs; Figure 6 is explicitly labeled 'Predicted,' and no held-out budget experiment is reported. Limitations (vi) says LLM validation is future work, yet Section 3.5 quantifies requirements for 7B/175B models from these BERT-scale exponents. These are not independent confirmations. The paper's empirical data are real, so not an 8–10; but the headline efficiency claim partially reduces to its own fit, giving 6.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The paper's quantitative claims rest on a set of fitted parameters for the scaling laws, a Pareto weight fitted to the observed frontier, and imported assumptions about metric validity, the form of the joint scaling law, and the cost model. No new theoretical entities are introduced.

free parameters (8)
  • M (model scaling intercept) = 3.70e4 (MS MARCO OOD), 4.23e6 (T2Ranking)
    Fitted in Eq. 7 for the model-size scaling law of OOD robustness.
  • mu (model scaling exponent) = 0.55 (MS MARCO OOD), 0.46 (T2Ranking)
    Fitted exponent describing how OOD robustness changes with model size.
  • delta_f (model irreducible loss) = 0.05 (MS MARCO OOD), 0.96 (T2Ranking)
    Fitted floor term in Eq. 7, attributed to annotation inconsistency.
  • D (data scaling intercept) = 4.34e3 (MS MARCO OOD), 1.93e5 (T2Ranking)
    Fitted in Eq. 8 for the data-size scaling law of OOD robustness.
  • eta (data scaling exponent) = 0.83 (MS MARCO OOD), 0.51 (T2Ranking)
    Fitted exponent describing how OOD robustness changes with data size.
  • delta_D (data irreducible loss) = 0.08 (MS MARCO OOD), 0.12 (T2Ranking)
    Fitted floor term in Eq. 8.
  • w0 (Pareto training initial weight) = not reported
    Initialized from the ratio between robustness and effectiveness at Pareto efficiency (Section 4.2), i.e., fitted to the observed frontier.
  • Joint scaling law coefficients = Table 4 (M, D, mu, eta, delta) for standard and Pareto training
    Numerically estimated for Eq. 12, used to compute the 2.5x budget efficiency claim in Section 5.2.
assumptions (6)
  • domain assumption Power-law scaling forms L(f)=(M/|f|)^mu + delta_f and L(D)=(D/|D|)^eta + delta_D hold for contrastive entropy.
    Assumed following [6]; no derivation is provided in this paper.
  • domain assumption Contrastive entropy is a valid proxy for ranking effectiveness and robustness, consistent with NDCG/MRR.
    Taken from prior work [6]; the paper does not revalidate this on discrete ranking metrics.
  • domain assumption Joint scaling law L(f,D) = [(M/|f|)^(mu/eta) + D/|D|]^eta + delta approximates combined model and data scaling.
    Imported from [6] and used without derivation in Eq. 12.
  • domain assumption The cost model Z = Z_data*|D| + Z_train*|f| + Z_infer*|f| with fixed dollar coefficients describes real resource tradeoffs.
    Coefficients taken from prior work [6]; the paper assumes adversarial and generated data costs equal manual annotation costs.
  • ad hoc to paper The loss ratio l_E/l_R is a reliable steering signal for reaching Pareto efficiency in the performance space.
    Eq. 10 uses this heuristic with no convergence argument; w0 is fitted to the same frontier the method is validated against.
  • domain assumption The fitted scaling law exponents remain valid outside the fitted range when extrapolated to 7B and 175B models.
    Section 3.5 relies on this to estimate resource requirements for Llama2Vec-scale models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Scaling of Robustness and Effectiveness in Dense Retrieval." pith.science (2026). https://pith.science/paper/QMDXBITR

@misc{pith2026250524279,
  author       = {Pith},
  title        = {Pith review of: On the Scaling of Robustness and Effectiveness in Dense Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMDXBITR}},
  note         = {Machine review of arXiv:2505.24279}
}
read the original abstract

Robustness and Effectiveness are critical aspects of developing dense retrieval models for real-world applications. It is known that there is a trade-off between the two. Recent work has addressed scaling laws of effectiveness in dense retrieval, revealing a power-law relationship between effectiveness and the size of models and data. Does robustness follow scaling laws too? If so, can scaling improve both robustness and effectiveness together, or do they remain locked in a trade-off? To answer these questions, we conduct a comprehensive experimental study. We find that:(i) Robustness, including out-of-distribution and adversarial robustness, also follows a scaling law.(ii) Robustness and effectiveness exhibit different scaling patterns, leading to significant resource costs when jointly improving both. Given these findings, we shift to the third factor that affects model performance, namely the optimization strategy, beyond the model size and data size. We find that: (i) By fitting different optimization strategies, the joint performance of robustness and effectiveness traces out a Pareto frontier. (ii) When the optimization strategy strays from Pareto efficiency, the joint performance scales in a sub-optimal direction. (iii) By adjusting the optimization weights to fit the Pareto efficiency, we can achieve Pareto training, where the scaling of joint performance becomes most efficient. Even without requiring additional resources, Pareto training is comparable to the performance of scaling resources several times under optimization strategies that overly prioritize either robustness or effectiveness. Finally, we demonstrate that our findings can help deploy dense retrieval models in real-world applications that scale efficiently and are balanced for robustness and effectiveness.

Figures

Figures reproduced from arXiv: 2505.24279 by the authors.

Figure 1
Figure 1. Joint scaling trends in robustness and effectiveness of the BERT model with respect to model size and data size across [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Scaling law for OOD robustness and effectiveness with model size (a) and data size (b) on MS MARCO and T2Ranking, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Scaling laws for OOD robustness with data size [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Scaling law for adversarial robustness and effectiveness with model size (a) and data size (b) on MS MARCO, respectively. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Joint scaling trends in robustness and effectiveness of the BERT model with respect to model size and data size across [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Predicted contrastive entropy of robustness and effectiveness for standard and Pareto training under limited budgets. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 54 canonical work pages

  1. [6]

    Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. 2024. Scaling Laws for Dense Retrieval. InSIGIR. 1339–1349

    work page 2024

  2. [1]

    Leif Azzopardi and Vishwa Vinay. 2008. Retrievability: An Evaluation Measure for Higher Order Information Access Tasks. InCIKM. 561–570

    work page 2008

  3. [2]

    Carlos Castillo and Brian D. Davison. 2011. Adversarial Web Search.Foundations and Trends in Information Retrieval4, 5 (2011), 377–486

    work page 2011

  4. [3]

    Xuanang Chen, Ben He, Kai Hui, Le Sun, and Yingfei Sun. 2023. Dealing with Textual Noise for Robust and Effective BERT Re-ranking.IPM60 (2023), 103135

    work page 2023

  5. [4]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin

  6. [5]

    Zhe Dong, Jianmo Ni, Daniel M Bikel, Enrique Alfonseca, Yuan Wang, Chen Qu, and Imed Zitouni. 2022. Exploring Dual Encoder Architectures for Question Answering. InEMNLP. 9414–9419

    work page 2022

  7. [7]

    Alexander Gelbukh and Grigori Sidorov. 2001. Zipf and Heaps Laws’ Coefficients Depend on Language. InCICLing. Springer, 332–335

    work page 2001

  8. [8]

    Joel Goh and Melvyn Sim. 2010. Distributionally Robust Optimization and Its Tractable Approximations.Operations Research58, 4-part-1 (2010), 902–917

    work page 2010

Show all 66 references
  1. [9]

    Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Sachdeva, and Chitta Baral. [n. d.]. Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness. InACL

  2. [10]

    Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng

  3. [11]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, et al. 2022. An Empirical Analysis of Compute-Optimal Large Lan- guage Model Training.NIPS35 (2022), 30016–30030

    2022

  4. [12]

    Nikolaus Howe, Ian McKenzie, Oskar Hollinsworth, Michał Zajac, Tom Tseng, Aaron Tucker, Pierre-Luc Bacon, and Adam Gleave. 2024. Effects of Scale on Language Model Robustness.arXiv preprint arXiv:2407.18213(2024)

    2024 arXiv

  5. [13]

    Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. InICLR

    2019

  6. [14]

    Hestness Joel, Narang Sharan, Ardalani Newsha, Diamos Gregory, and Zhou others. [n. d.]. Deep Learning Scaling Is Predictable, Empirically.arXiv preprint 1712.00409([n. d.])

    arXiv

  7. [15]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361(2020)

    2020 arXiv

  8. [16]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. InEMNLP. 6769–6781

    2020

  9. [17]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT

    2019

  10. [18]

    Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. InSIGIR. 39–48

    2020

  11. [19]

    Oren Kurland and Moshe Tennenholtz. 2022. Competitive Search. InSIGIR

    2022

  12. [20]

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent Retrieval for Weakly Supervised Open Domain Question Answering. InACL. 6086–6096

    2019

  13. [21]

    Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, and Defu Lian. 2024. Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Re- trieval. InACL. 3490–3500

    2024

  14. [22]

    Jimmy Lin. 2022. A Proposed Conceptual Framework for a Representational Approach to Information Retrieval. InACM SIGIR Forum, Vol. 55

    2022

  15. [23]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Wei Chen, and Xueqi Cheng. 2023. On the Robustness of Generative Retrieval Models. InGen-IR@SIGIR

    2023

  16. [24]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Xueqi Cheng. 2025. On the Robustness of Generative Information Retrieval Models. InECIR

    2025

  17. [25]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2025. Robust Information Retrieval. InWSDM

    2025

  18. [26]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Black-Box Adversarial Attacks against Dense Retrieval Models: A Multi-View Contrastive Learning Method. InCIKM. 1647–1656

    2023

  19. [27]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, and Xueqi Cheng. 2025. Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks against Black-box Neural Ranking Models. InAAAI

    2025

  20. [28]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Multi-granular Adversarial Attacks against Black-box Neural Ranking Models. InSIGIR. 1391–1400

    2024

  21. [29]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Robust Neural Information Retrieval: An Adversarial and Out-of- distribution Perspective.arXiv preprint arXiv:2407.06992(2024)

    2024 arXiv

  22. [30]

    Yu-An Liu, Ruqing Zhang, Mingkun Zhang, Wei Chen, Maarten de Rijke, Jiafeng Guo, and Xueqi Cheng. 2024. Perturbation-Invariant Adversarial Training for Neural Ranking Models. InAAAI, Vol. 38. 8832–8840

    2024

  23. [31]

    Dingkun Long, Qiong Gao, Kuan Zou, Guangwei Xu, Pengjun Xie, Ruijie Guo, Jian Xu, Guanjun Jiang, Luxi Xing, and Ping Yang. 2022. Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrieval. InSIGIR. 3046–3056

    2022

  24. [32]

    Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation. InECIR. 1075–1088

    2021

  25. [33]

    Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng

  26. [34]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. InCoCo@NIPS

    2016

  27. [35]

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al . 2022. Large Dual Encoders Are Generalizable Retrievers. InEMNLP. 9844–9855

    2022

  28. [36]

    Prop: Pre-training with Representative Words Prediction for Ad-hoc Re- trieval. InWSDM. 283–291

  29. [37]

    OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt

    2022

  30. [38]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021. KILT: a Benchmark for Knowledge Intensive Language Tasks. In NAACL. 2523–2544

    2021

  31. [39]

    Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction.arXiv preprint arXiv:1904.08375(2019)

    2019 arXiv

  32. [40]

    Nils Reimers and Iryna Gurevych. 2021. The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes. InACL. 605–611

    2021

  33. [41]

    Stephen E Robertson and Steve Walker. 1994. Some Simple Effective Approxima- tions to the 2-poisson Model for Probabilistic Weighted Retrieval. InSIGIR’94. Springer, 232–241

    1994

  34. [42]

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxi- ang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In NAACL. Association for Computational Linguistics

    2021

  35. [43]

    Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced Represen- tation Through Knowledge Integration.arXiv preprint 1904.09223(2019)

    2019 arXiv

  36. [44]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. InNIPS

    2021

  37. [45]

    Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Kai Zhang, and Daxin Jiang. 2023. Unifier: A Unified Retriever for Large-scale Retrieval. In SIGKDD. 4787–4799

    2023

  38. [46]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InNIPS, Vol. 30

    2017

  39. [47]

    Shaokun Wang, Tian Gan, Yuan Liu, Jianlong Wu, Yuan Cheng, and Liqiang Nie. 2022. Micro-influencer Recommendation by Multi-perspective Account Representation Learning.TMM(2022)

    2022

  40. [48]

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness May Be at Odds with Accuracy. InICLR

    2019

  41. [49]

    2013.Strategy: An Introduction to Game Theory(3 ed.)

    Joel Watson. 2013.Strategy: An Introduction to Game Theory(3 ed.). W. W. Norton and Company

    2013

  42. [50]

    Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2023. PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models.TOIS41, 4 (2023), Article 89

    2023

  43. [51]

    Shaokun Wang, Tian Gan, Yuan Liu, Li Zhang, JianLong Wu, and Liqiang Nie

  44. [52]

    Discover Micro-Influencers for Brands via Better Understanding.TMM (2021), 2595–2605

    2021

  45. [53]

    Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2015. Learning Maximal Marginal Relevance Model via Directly Optimizing Diversity Evaluation Measures. InSIGIR. 113–122

    2015

  46. [54]

    Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, et al . 2023. T2ranking: A Large-Scale Chinese Benchmark for Passage Ranking. InSIGIR. 2681–2690

    2023

  47. [55]

    Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2022. Are Neural Ranking Models Robust?TOIS41, 2 (2022), 1–36

    2022

  48. [56]

    Zhijing Wu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. In- vestigating Passage-Level Relevance and Its Role in Document-Level Relevance Judgment. InSIGIR. 605–614

    2019

  49. [57]

    Peng Zhang, Dawei Song, Jun Wang, and Yuexian Hou. 2013. Bias-Variance Decomposition of IR Evaluation. InSIGIR. 1021–1024

    2013

  50. [58]

    Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi- Scale Attentive Interaction Networks for Chinese Medical Question Answer Selection.IEEE Access6 (2018), 74061–74071. On the Scaling of Robustness and Effectiveness in Dense Retrieval SIGIR ’25, July ...

    2018

  51. [59]

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. InICLR

    2021

  52. [60]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. GLM-130B: An Open Bilingual Pre-trained Model. InICLR

    2022

  53. [61]

    2016.Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology

    George Kingsley Zipf. 2016.Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Ravenio Books

    2016

  54. [62]

    Xin Zou and Weiwei Liu. 2024. On the Adversarial Robustness of Out-of- Distribution Generalization Models.NIPS36 (2024)

    2024

  55. [63]

    Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2022. Dense Text Retrieval based on Pretrained Language Models: A Survey.arXiv preprint arXiv:2211.14876(2022)

    2022 arXiv

  56. [64]

    Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poison- ing Retrieval Corpora by Injecting Adversarial Passages. InEMNLP

    2023

  57. [2021]

    In SIGIR

    MS Marco: Benchmarking Ranking Models in the Large-Data Regime. In SIGIR. 1566–1576

  58. [2022]

    TOIS40, 4 (2022), 1–42

    Semantic Models for the First-stage Retrieval: A Comprehensive Review. TOIS40, 4 (2022), 1–42

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.