Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Towards Better Instruction Following Retrieval Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on 38,759 instruction-query-passage triples with poisoned-instruction and poisoned-query negatives teaches small embedding retrievers to follow user instructions, lifting p-MRR by 9.0 points on embedding backbones and 4.2 points…

desk verdict The dataset is a real contribution and the empirical gains look consistent, but Eq. (4)'s cross-attention degenerates to a linear projection as written, so the paper needs a formal fix before the claims about instruction-query attention can be trusted. read the letter →

arxiv 2505.21439 v1 pith:M27ZMKCC submitted 2025-05-27 cs.CL cs.IR

Yuchen Zhuang , Aaron Trinh , Rushi Qiang , Haotian Sun , Chao Zhang , Hanjun Dai , Bo Dai This is my paper
classification cs.CLcs.IR
keywords instruction-followingretrievaldensecontrastivelearninghardnegativeminingsynthetictrainingdatabi-encoderembeddingsp-MRRinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern retrieval models trained only on query-passage pairs miss what a user explicitly asks for in the instruction. The paper tries to establish that a training corpus of 38,759 instruction-query-passage positives, each paired with two hard negatives made by poisoning the instruction or the query, can teach small embedding-based retrievers to follow instructions, closing most of the gap previously reserved for large reranker models. It reports gains of 9.0 points in pairwise mean reciprocal rank (p-MRR) for embedding-based backbones and 4.2 points for auto-regressive language models across FollowIR, MAIR, and Bright. If the claim holds, instruction-aware retrieval no longer needs an expensive reranking stage, which matters for deployment at scale.

What carries the argument

The machinery has three parts. In the data, each positive $\langle\text{instruction}, \text{query}, \text{passage}\rangle$ triple $(I^+, Q^+, P^+)$ is paired with two hard negatives $(I^-, Q^+, P_1^-)$ and $(I^+, Q^-, P_2^-)$, where one component has been semantically poisoned, so the model must learn which instruction or query change flips relevance. A filtering step uses o3-mini to discard triplets whose relevance judgments are ambiguous across the three scenarios. In the model, InF-Embed is a dual encoder with separate passage and instruction-query towers; the instruction and query are combined either by concatenation into one input or by a cross-attention layer, and retrieval scores are cosine similarities between passage embeddings and instruction-conditioned query embeddings. The contrastive losses (univariate and multivariate variants) are made tractable by marginal negative sampling, which replaces the combinatorial denominator $\sum_{m,j,k} \exp(s_\theta(P_m, I_j, IQ_k))$ with three linear sums over a batch. The recommended configuration, $\ell^{\mathrm{multi}}_{P,I}$, jointly contrasts passages and instructions and is reported as the best across seven backbones.

What would settle it

A direct test is to collect fresh instruction-query pairs from real users in the same domains, get human relevance labels, and measure whether InF-Embed's p-MRR advantage over baselines survives; if it does not, the synthetic instruction distribution is carrying the result. A second check is to enlarge the human agreement study beyond three annotators and 100 samples, since the claim that o3-mini filtering matches human relevance currently rests on that small sample.

Watch

Extended reading notes

Core claim

The paper's central discovery is that instruction-following behavior can be trained directly into small dual-encoder retrievers rather than requiring large reranker models. Starting from MS MARCO query-passage pairs, the authors synthesize an instruction for each pair, then construct two hard negatives by poisoning either the instruction or the query; an o3-mini judge keeps only triplets with unambiguous relevance. This yields 38,759 positive and 77,518 negative triplets. Trained on this corpus, InF-Embed uses instruction-aware query representations and retrieves passages by cosine similarity; the authors report consistent p-MRR gains on every backbone tested, averaging 8.1% over competitive baselines and reaching +9.0 p-MRR for embedding models and +4.2 p-MRR for auto-regressive language models.

Load-bearing premise

The load-bearing premise is that synthetic instructions written by gpt-4o-mini and filtered by o3-mini capture how real users phrase and intend search instructions, so that gains on FollowIR, MAIR, and Bright transfer to other instruction-following queries.

Editorial extensions

If this is right

  • Instruction-following retrieval no longer requires a separate reranking stage: the same bi-encoder architecture used for ordinary dense retrieval can carry the instruction-aware behavior.
  • Both small encoder models and decoder-only LMs improve with InF-IR, so the corpus is not tied to one architecture family or scale.
  • Filtering synthetic triplets with a strong reasoning model is worth more than raw data volume; the paper reports that the unfiltered corpus performs worse.
  • The multivariate objective that contrasts passages and instructions jointly ($\ell^{\mathrm{multi}}_{P,I}$) is the configuration the paper recommends after ablating over seven backbones.
  • Models trained on InF-IR reach p-MRR levels competitive with much larger and proprietary retrievers on FollowIR, making efficient deployment viable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic instruction distribution is the real source of the gains, the same recipe should transfer to specialized corpora in law, medicine, or code; the paper only begins this by adding TREC Robust, Leetcode, and MetaMath sources in the appendix.
  • The o3-mini filtering step is a computational bottleneck; an untested extension is to distill its relevance judgments into a cheaper classifier so the pipeline can be rerun on new corpora without paying for a reasoning model on every triple.
  • The poisoning strategy targets instructions and queries that are semantic opposites, so it likely teaches models to respect stylistic and attitudinal instructions well; whether it captures complex reasoning constraints is an open question the paper leaves for reasoning-intensive retrieval.
  • Because the human validation of the filter used three coauthors and 100 samples, the claim that o3-mini matches human relevance is currently supported by a small study; a larger independent annotation effort would settle it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces InF-IR, a training corpus of 38,759 positive <instruction, query, passage> triplets and 77,518 hard negatives synthesized from MS MARCO by perturbing instructions and queries with gpt-4o-mini and filtering with o3-mini. It also proposes InF-Embed, a dual-encoder training framework with two instruction-query interaction strategies (concatenation and a cross-attention mechanism) and several univariate/multivariate contrastive objectives. Experiments on FollowIR, MAIR, and Bright report consistent p-MRR gains over base embedding models and decoder-only LMs, with a stated headline improvement of 8.1% p-MRR in the abstract and +9.0/+4.2 p-MRR in Section 7.

Significance. If the empirical claims hold, InF-IR addresses a real shortage: instruction-following training data for embedding-based retrieval, complementing reranker-oriented corpora. The paper's strengths include a publicly released dataset and code, a contamination check against evaluation sets, a broad battery of backbone models, and a systematic ablation of contrastive objectives. The headline gains are consistent in sign across many configurations, which lends some plausibility to the central claim. However, the technical presentation contains a formal error in the cross-attention equation, and the empirical evaluation lacks estimates of uncertainty; the small coauthor-only human validation is also thin. The core dataset contribution is likely salvageable, but the methodological claims require correction.

major comments (4)
  1. [§5.1, Eq. (4)] The cross-attention interaction as written is degenerate: the argument to the softmax is a scalar (the dot product of two d-dimensional vectors), so the softmax evaluates to 1 regardless of the instruction embedding i_j. Consequently iq_{j,k} = q_k W_{q,2}, and the instruction has zero influence on the query representation. This invalidates the claimed 'instruction-query attention mechanism' and is misleading in a contribution statement. Please specify the intended implementation (e.g., attention over instruction and query token sequences, or a non-scalar attention weight per query token) or, if attention was not actually used in the reported best models, remove or explicitly de-emphasize this mechanism.
  2. [Abstract and §7] The headline improvement is inconsistent across the manuscript. The abstract reports '8.1% in p-MRR', Section 7 and the conclusion report '+9.0 p-MRR' for embedding-based models and '+4.2 p-MRR' for auto-regressive LMs, and Section 6.2 reports '+1.36@p-MRR' and '+8.2@p-MRR' in the text while Table 2 shows different per-model deltas. Please state a single, precise aggregation rule (which benchmarks, which models, and how the average is computed) and use it consistently in the abstract, results, and conclusion.
  3. [§6.1, Table 2] The paper repeatedly claims that InF-Embed 'significantly surpasses' baselines, but no error bars, standard deviations, or significance tests are reported, and several evaluation sets are small (e.g., FollowIR has 104 queries). With single runs, the differences in p-MRR and nDCG could be within noise. Please provide multiple seeds or bootstrap confidence intervals for at least the main results in Table 2 and Figure 5, or temper the significance language accordingly.
  4. [Appendix F.1, Figure 4] The claim that o3-mini filtering is a reliable proxy for human relevance is supported only by a study with 3 coauthors and 100 samples. This is a load-bearing assumption for the data-quality claims, yet the sample size and rater pool are too small to establish strong agreement. Please expand the human study with independent annotators and a larger sample, or explicitly revise the conclusion to acknowledge the limited validation of the filtering step.
minor comments (6)
  1. [§6.1, Implementation Details] There is a typo: 'Llamma-3.2' should be 'Llama-3.2'.
  2. [Appendix C, Eqs. (13)-(14)] The equation labeled ℓuni_P,I,IQ appears twice with different expressions; the first instance (Eq. 13) is missing the P-contrast term and should be a two-term objective, while Eq. (14) has the full three-term form. Please renumber and correct.
  3. [Table 2, ModernBERT-base row] The entry '4.3 -1.' appears to be truncated; the p-MRR baseline value should be reported in full.
  4. [Figure 4] The bars and kappa values are not clearly matched to the four models named in the caption; please add a legend or label each bar directly.
  5. [Abstract and §4.1] The paper alternates between '38,759 positive samples' and 'over 38,000' in the abstract, and between '77,518' and 'more than 60,000' hard negatives in the conclusion; please make the numbers consistent throughout.
  6. [Introduction, Figure 1 caption] The text preceding Figure 1 ('Query Deforestationand urbanizationresult in increased emissions...') appears malformed and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training and evaluation pipeline is self-contained and the reported p-MRR gains are measured on external benchmarks.

full rationale

The paper's claimed derivation chain is not circular. InF-IR is built from MS MARCO seed pairs by synthesizing instructions and poisoned negatives with gpt-4o-mini and filtering with o3-mini; the resulting training triplets are distinct from the held-out evaluation sets (FollowIR, MAIR, Bright), and Section 6.1 reports a string-matching contamination check. The contrastive objectives in Section 5.2 are standard NCE losses applied to those triplets; the reported +9.0 p-MRR and +4.2 p-MRR gains are measured outcomes on external benchmarks, not quantities recovered from the training data or from fitted parameters. The o3-mini quality filter is validated against human annotations in Appendix F.1; even though that validation uses only 3 coauthors and 100 samples, it is an external agreement check rather than a definitional equivalence. No load-bearing self-citation or uniqueness theorem is invoked. Eq. (4) contains a formal error—the softmax is applied to a scalar, making the instruction term vacuous as written—but this is a correctness/implementation issue, not a circular derivation, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about the synthetic training pipeline; no invented physical entities or fitted scientific constants appear in the paper.

assumptions (4)
  • domain assumption Synthetic instructions generated by gpt-4o-mini are a faithful proxy for real user instructions.
    Section 4.1 Instruction Generation uses gpt-4o-mini to create instructions; the entire training corpus depends on this proxy.
  • domain assumption o3-mini relevance judgments align with human judgments well enough to filter training data.
    Section 4.1 Data Quality Check uses o3-mini to discard tuples; Appendix F.1 validates with only 3 coauthor annotators on 100 samples, which is weak support.
  • domain assumption MS MARCO query-passage pairs are an appropriate seed for general-domain instruction-following training.
    Section 4.1 Data Curation selects MS MARCO as the seed; generalizability to specialized domains is not established.
  • domain assumption The multivariate contrastive objectives with marginal negative sampling preserve sufficient signal for learning instruction-following.
    Section 5.2 defines the objectives; no analysis shows that the marginal strategy captures all relevant negative interactions, only an empirical comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Better Instruction Following Retrieval Models." pith.science (2026). https://pith.science/paper/M27ZMKCC

@misc{pith2026250521439,
  author       = {Pith},
  title        = {Pith review of: Towards Better Instruction Following Retrieval Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M27ZMKCC}},
  note         = {Machine review of arXiv:2505.21439}
}
read the original abstract

Modern information retrieval (IR) models, trained exclusively on standard <query, passage> pairs, struggle to effectively interpret and follow explicit user instructions. We introduce InF-IR, a large-scale, high-quality training corpus tailored for enhancing retrieval models in Instruction-Following IR. InF-IR expands traditional training pairs into over 38,000 expressive <instruction, query, passage> triplets as positive samples. In particular, for each positive triplet, we generate two additional hard negative examples by poisoning both instructions and queries, then rigorously validated by an advanced reasoning model (o3-mini) to ensure semantic plausibility while maintaining instructional incorrectness. Unlike existing corpora that primarily support computationally intensive reranking tasks for decoder-only language models, the highly contrastive positive-negative triplets in InF-IR further enable efficient representation learning for smaller encoder-only models, facilitating direct embedding-based retrieval. Using this corpus, we train InF-Embed, an instruction-aware Embedding model optimized through contrastive learning and instruction-query attention mechanisms to align retrieval outcomes precisely with user intents. Extensive experiments across five instruction-based retrieval benchmarks demonstrate that InF-Embed significantly surpasses competitive baselines by 8.1% in p-MRR, measuring the instruction-following capabilities.

Figures

Figures reproduced from arXiv: 2505.21439 by the authors.

Figure 1
Figure 1. Example of original information retrieval compared to instruction-following retrieval. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Hard negative samples in InF-IR generated by poisoning both instructions and queries. 3 Preliminaries Noise Contrastive Estimation. We begin by formulating a ranking-based noise contrastive estima￾tion (NCE) objective (Ma & Collins, 2018; Gutmann & Hyvärinen, 2010; Henderson et al., 2017; Yang et al., 2019) from a conditional modeling perspective. Specifically, consider a model that estimates a conditional distribut… view at source ↗
Figure 3
Figure 3. Visualization and diversity analysis of synthetic training samples from [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cohen’s kappa from 100 random samples. Data Quality Check. To ensure the quality and semantic consis￾tency of our synthetic data, we employ an advanced reasoning model, o3-mini, for quality evaluation. This validation procedure rigorously verifies whether the generated…
Figure 5
Figure 5. Figure 5: Comparative analysis of instruction-following capabilities on the Follow-IR benchmark [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Effect of quality filtering. Effect of Negative Pairs Synthesis. We analyze the impact of various training configurations in [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RecoWorld: Building Simulated Environments for Agentic Recommender Systems

    cs.IR 2025-09 conditional novelty 5.0 of 10

    A design proposal, not a tested system: a dual-view simulation loop in which an LLM-simulated user issues reflective instructions when about to disengage, and an instruction-following recommender adapts to maximize si...

  2. A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A research agenda calling for geo-temporal reasoning in deep research systems, with no experiments or system implementation.

Reference graph

Works this paper leans on

49 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    V., and Voorhees, E

    Allan, J., Harman, D., Kanoulas, E., Li, D., Gysel, C. V., and Voorhees, E. Trec 2017 common core track overview. In Proceedings of the Twenty-Sixth Text REtrieval Conference (TREC 2017), Gaithersburg, Maryland, USA, 2017. National Institute of Standards and Technology (NIST)

    work page 2017

  2. [2]

    Task-aware retrieval with instructions

    Asai, A., Schick, T., Lewis, P., Chen, X., Izacard, G., Riedel, S., Hajishirzi, H., and Yih, W.-t. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260, 2022

    arXiv 2022

  3. [3]

    Task-aware retrieval with instructions

    Asai, A., Schick, T., Lewis, P., Chen, X., Izacard, G., Riedel, S., Hajishirzi, H., and Yih, W.-t. Task-aware retrieval with instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 3650--3675, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:1...

    doi:10.18653/v1/2023.findings-acl.225 2023

  4. [4]

    Ms marco: A human generated machine reading comprehension dataset, 2018

    Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., and Wang, T. Ms marco: A human generated machine reading comprehension dataset, 2018. URL https://arxiv.org/abs/1611.09268

    arXiv 2018

  5. [5]

    Z., et al

    Chen, M., Li, T., Sun, H., Zhou, Y., Zhu, C., Yang, F., Zhou, Z., Chen, W., Wang, H., Pan, J. Z., et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025

    arXiv 2025

  6. [6]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. Scaling instruct...

    arXiv 2022

  7. [7]

    Deeprag: Thinking to retrieval step by step for large language models

    Guan, X., Zeng, J., Meng, F., Xin, C., Lu, Y., Lin, H., Han, X., Sun, L., and Zhou, J. Deeprag: Thinking to retrieval step by step for large language models. arXiv preprint arXiv:2502.01142, 2025

    arXiv 2025

  8. [8]

    and Hyv \"a rinen, A

    Gutmann, M. and Hyv \"a rinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 297--304. JMLR Workshop and Conference Proceedings, 2010

    2010

Show all 49 references
  1. [9]

    Efficient natural language response suggestion for smart reply, 2017

    Henderson, M., Al-Rfou, R., Strope, B., hsuan Sung, Y., Lukacs, L., Guo, R., Kumar, S., Miklos, B., and Kurzweil, R. Efficient natural language response suggestion for smart reply, 2017. URL https://arxiv.org/abs/1705.00652

    2017 arXiv

  2. [10]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  3. [11]

    Unsupervised dense information retrieval with contrastive learning

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021

    2021 arXiv

  4. [12]

    F ollow B ench: A multi-level fine-grained constraints following benchmark for large language models

    Jiang, Y., Wang, Y., Zeng, X., Zhong, W., Li, L., Mi, F., Shang, L., Jiang, X., Liu, Q., and Wang, W. F ollow B ench: A multi-level fine-grained constraints following benchmark for large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62...

    2024 doi

  5. [13]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Jin, B., Zeng, H., Yue, Z., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  6. [14]

    Dense passage retrieval for open-domain question answering

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...

    2020 doi

  7. [15]

    R., Hui, K., Boratko, M., Kapadia, R., Ding, W., et al

    Lee, J., Dai, Z., Ren, X., Chen, B., Cer, D., Cole, J. R., Hui, K., Boratko, M., Kapadia, R., Ding, W., et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024

    2024 arXiv

  8. [16]

    Fine-tuning llama for multi-stage text retrieval

    Ma, X., Wang, L., Yang, N., Wei, F., and Lin, J. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 2421--2425, 2024

    2024

  9. [17]

    and Collins, M

    Ma, Z. and Collins, M. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812, 2018

    2018 arXiv

  10. [18]

    Dqi: Measuring data quality in nlp

    Mishra, S., Arunkumar, A., Sachdeva, B., Bryan, C., and Baral, C. Dqi: Measuring data quality in nlp. arXiv preprint arXiv:2005.00816, 2020

    2005 arXiv

  11. [19]

    H., Dornbach, P., Zitouni, I., Alfonseca, E., and Dong, Z

    Moiseev, F., Abrego, G. H., Dornbach, P., Zitouni, I., Alfonseca, E., and Dong, Z. Samtone: Improving contrastive loss for dual encoder retrieval models with same tower negatives, 2023. URL https://arxiv.org/abs/2306.02516

    2023 arXiv

  12. [20]

    Moreira, G. d. S. P., Osmulski, R., Xu, M., Ak, R., Schifferer, B., and Oldridge, E. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831, 2024

    2024 arXiv

  13. [21]

    MTEB : Massive text embedding benchmark

    Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB : Massive text embedding benchmark. In Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 2014--2037, Dubrovnik, Cro...

    2014 doi

  14. [22]

    Generative representational instruction tuning

    Muennighoff, N., SU, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI, 2024. URL https://openreview.net/forum?id=8cQrRO9iFe

    2024

  15. [23]

    and Cho, K

    Nogueira, R. and Cho, K. Passage re-ranking with bert, 2020. URL https://arxiv.org/abs/1901.04085

    2020 arXiv

  16. [24]

    Document ranking with a pretrained sequence-to-sequence model

    Nogueira, R., Jiang, Z., Pradeep, R., and Lin, J. Document ranking with a pretrained sequence-to-sequence model. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp.\ 708--718, Online, November 2020. Association fo...

    2020 doi

  17. [25]

    Instructir: A benchmark for instruction following of information retrieval models

    Oh, H., Lee, H., Ye, S., Shin, H., Jang, H., Jun, C., and Seo, M. Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint arXiv:2402.14334, 2024

    2024 arXiv

  18. [26]

    S., Ladhak, F., and Hashimoto, T

    Oren, Y., Meister, N., Chatterji, N. S., Ladhak, F., and Hashimoto, T. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  20. [28]

    Kilt: a benchmark for knowledge intensive language tasks

    Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. Kilt: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association f...

    2021

  21. [29]

    X., She, Q., Wu, H., Wang, H., and Wen, J.-R

    Ren, R., Lv, S., Qu, Y., Liu, J., Zhao, W. X., She, Q., Wu, H., Wang, H., and Wen, J.-R. Pair: Leveraging passage-centric similarity relation for improving dense passage retrieval. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.\ 2173–2183. A...

    2021 doi

  22. [30]

    The probabilistic relevance framework: Bm25 and beyond

    Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3 0 (4): 0 333--389, 2009

    2009

  23. [31]

    E., Walker, S., Jones, S., Hancock-Beaulieu, M

    Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M. M., Gatford, M., et al. Okapi at trec-3. Nist Special Publication Sp, 109: 0 109, 1995

    1995

  24. [32]

    Overview of trec 2021

    Soboroff, I. Overview of trec 2021. In Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021), volume 500-335 of NIST Special Publication, Gaithersburg, Maryland, USA, 2022. National Institute of Standards and Technology (NIST)

    2021

  25. [33]

    IFIR : A comprehensive benchmark for evaluating instruction-following in expert-domain information retrieval

    Song, T., Gan, G., Shang, M., and Zhao, Y. IFIR : A comprehensive benchmark for evaluating instruction-following in expert-domain information retrieval. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter ...

    2025

  26. [34]

    A., Zettlemoyer, L., and Yu, T

    Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.-t., Smith, N. A., Zettlemoyer, L., and Yu, T. One embedder, any task: Instruction-finetuned text embeddings. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computation...

    2023 doi

  27. [35]

    S., Tang, M., et al

    Su, H., Yen, H., Xia, M., Shi, W., Muennighoff, N., Wang, H.-y., Liu, H., Shi, Q., Siegel, Z. S., Tang, M., et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. arXiv preprint arXiv:2407.12883, 2024

    2024 arXiv

  28. [36]

    J., Yan, L., Ma, X., Liu, Y., Cao, M., Yin, D., and Ren, Z

    Sun, W., Shi, Z., Long, W. J., Yan, L., Ma, X., Liu, Y., Cao, M., Yin, D., and Ren, Z. MAIR : A massive benchmark for evaluating instructed retrieval. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lang...

    2024 doi

  29. [37]

    BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models

    Thakur, N., Reimers, N., R \"u ckl \'e , A., Srivastava, A., and Gurevych, I. BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2),...

    2021

  30. [38]

    Overview of the trec 2004 robust retrieval track

    Voorhees, E. Overview of the trec 2004 robust retrieval track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), Gaithersburg, Maryland, USA, 2004. National Institute of Standards and Technology (NIST)

    2004

  31. [39]

    Text embeddings by weakly-supervised contrastive pre-training

    Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022 a

    2022 arXiv

  32. [40]

    Improving text embeddings with large language models

    Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023 a

    2023 arXiv

  33. [41]

    S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Pal, K

    Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi, M., P...

    2022

  34. [42]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  35. [43]

    Followir: Evaluating and teaching information retrieval models to follow instructions

    Weller, O., Chang, B., MacAvaney, S., Lo, K., Cohan, A., Van Durme, B., Lawrie, D., and Soldaini, L. Followir: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403.15246, 2024

    2024 arXiv

  36. [44]

    V., Lawrie, D., Paranjape, A., Zhang, Y., and Hessel, J

    Weller, O., Durme, B. V., Lawrie, D., Paranjape, A., Zhang, Y., and Hessel, J. Promptriever: Instruction-trained retrievers can be prompted like language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=o...

    2025

  37. [45]

    C-pack: Packed resources for general chinese embeddings

    Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., and Nie, J.-Y. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pp.\ 641--649, 2024

    2024

  38. [46]

    Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax

    Yang, Y., Hernandez Abrego, G., Yuan, S., Guo, M., Shen, Q., Cer, D., Sung, Y.-h., Strope, B., and Kurzweil, R. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Con...

    2019 doi

  39. [47]

    T., Li, Z., Weller, A., and Liu, W

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  40. [48]

    Beyond content relevance: Evaluating instruction following in retrieval models

    Zhou, J., Zheng, Y., Chen, W., Zheng, Q., Zeyuan, S., Zhang, W., Meng, R., and Shen, X. Beyond content relevance: Evaluating instruction following in retrieval models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/foru...

    2025

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.