Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DaRL, a distribution-aware robust learning framework, improves LLM-based relevance modeling on both in-distribution and out-of-distribution data, and its deployed version lifts Alipay insurance search's valid PV-CTR by 1.2% and PV-CVR by…

desk verdict DaRL is a sensible industrial recipe with solid ID results, but the OOD test set is selected by the same DASA module being evaluated, so the OOD generalization claim is not yet proven. read the letter →

arxiv 2412.12504 v1 pith:M7JP5IHR submitted 2024-12-17 cs.IR cs.LG

Hong Liu , Saisai Gong , Yixin Ji , Kaixin Wu , Jia Xu , Jinjie Gu This is my paper · ORCID
classification cs.IRcs.LG
keywords SearchRelevanceModelingLargeLanguageModelOut-of-distributionGeneralizationDistribution-AwareSampleAugmentationFine-tuningE-commerceMahalanobisDistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the two known weaknesses of fine-tuned LLMs in search relevance modeling—overconfidence on fine-grained relevance grades and performance collapse under distribution shift—can be addressed together by a distribution-aware robust learning framework called DaRL. DaRL combines three mechanisms: DASA selects unlabeled samples that the original training distribution poorly covers and adds them to the fine-tuning set; LPFT first probes a classification head, then full-fine-tunes, then interpolates the two weight sets; OCC regularizes the score distribution toward a smoothed prior. On Alipay insurance product search data, DaRL-0.3B improved out-of-distribution F1 from 0.6431 to 0.6678 and in-distribution F1 from 0.9252 to 0.9349 over the strongest baseline, with larger models gaining more; online A/B testing showed a 1.2% PV-CTR and 3.04% PV-CVR lift. The paper thus establishes that data-level augmentation guided by OOD detection plus robust fine-tuning is a practical path to closing the ID–OOD gap without scaling up model size.

What carries the argument

The load-bearing machinery is the interaction of three named mechanisms. DASA uses Mahalanobis distance under a Gaussian assumption on the training representations together with a cosine-distance nearest-neighbor score to flag out-of-distribution unlabeled samples, which are then labeled and merged into the training set. LPFT freezes the backbone to train a classification head, fine-tunes all parameters, and interpolates the two resulting weight vectors to avoid catastrophic forgetting. OCC adds a KL-divergence auxiliary loss that regularizes predicted relevance scores toward a hand-set smoothed prior distribution for the three relevance grades. Together they convert fine-tuned LLMs into models that keep in-distribution accuracy while gaining out-of-distribution robustness.

What would settle it

Run DASA (Eq. 4) on a public query-item relevance dataset whose training representation distribution is visibly multi-modal (e.g., estimated by clustering or a normality test), then compare OOD F1 against training on the same number of randomly selected labeled samples; if the random selection matches or beats DASA, the Gaussian assumption is load-bearing for the result. A second check: replace the Mahalanobis term with a non-parametric density estimate and see whether the OOD F1 gain on the Alipay-style data persists.

Watch

Extended reading notes

Core claim

DaRL is built from three components, each targeting a different failure mode of vanilla LLM fine-tuning. DASA (Distribution-Aware Sample Augmentation) assumes the training representations form a Gaussian and scores each unlabeled search-log sample by Mahalanobis distance plus cosine distance to its nearest training neighbor; samples exceeding both thresholds are labeled and added to the training set, giving the model exposure to the long tail of the query-item distribution. LPFT (Linear-Probing then Fine-Tuning, adapted from prior work) freezes the backbone and trains only a classification head, then fine-tunes all parameters, then linearly interpolates the two weight vectors, which reduces feature distortion and catastrophic forgetting. OCC (Over-Confidence Calibration) adds a KL-divergence loss pulling the predicted score distribution toward a smoothed prior for strong, weak, and irrelevant labels, preventing the model from collapsing all weak-relevance predictions to a single extreme. The paper shows on Alipay insurance search data that each module contributes to the gain, with DASA contributing the largest OOD improvement (about 6 F1 points alone), and that the deployed DaRL-0.3B model improved valid PV-CTR by 1.2% and PV-CVR by 3.04% relative to the BERT baseline in online A/B testing.

Load-bearing premise

The DASA module assumes the representation space of the training data is a single Gaussian, so a sample's Mahalanobis distance is a reliable score for how out-of-distribution it is; if the true representation manifold is multi-modal or non-Gaussian, DASA may select samples that are not actually informative and the reported OOD gains may shrink.

Editorial extensions

If this is right

  • DaRL-0.3B is the deployed model in Alipay insurance search; in online A/B tests it lifted valid PV-CTR by 1.2% and PV-CVR by 3.04% relative to the BERT baseline.
  • The DASA module lets a labeling budget go further: distribution-aware sample selection yields larger OOD F1 gains than the same number of randomly selected labeled samples (Fig. 3).
  • LPFT's weight interpolation at alpha = 0.6 yields better OOD F1 than either linear probing alone or full fine-tuning alone (Fig. 4), indicating a sweet spot between feature preservation and task adaptation.
  • OCC reduces score overlap between weak and strong relevance predictions (Fig. 2), which is what enables a downstream trained-to-rank layer to use the scores as graded signals.
  • Model scale still matters: DaRL-2B and DaRL-10B beat DaRL-0.3B on all metrics, but latency and compute made the 0.3B model the industrial choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because DASA scores samples with the pre-fine-tuning model's representations, the selected 'OOD' set is relative to the initial model; an iterative variant that re-runs DASA after each fine-tuning round (or after LPFT's interpolation step) could yield further OOD gains, but the paper does not explore this.
  • The Gaussian assumption in Eq. 2 is the framework's most fragile premise; on representation spaces that are naturally multi-modal (mixed item categories, domains, or languages), a non-parametric OOD score or a Gaussian-mixture alternative might change which samples DASA picks, and could either improve or degrade the final model.
  • The KL prior in Eq. 9 is defined for three relevance grades with a smoothing factor rho tuned by grid search; extending the same calibration idea to four-point or five-point grading scales would require redefining the prior and probably re-tuning rho, which is a simple but untested generalization.
  • The online gains (CTR, CVR) also reflect Alipay-specific traffic and labeling; transferring DaRL to other e-commerce verticals or to web search would test whether the ID–OOD gap behavior replicates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DaRL, a framework for fine-tuning LLMs for graded relevance modeling in e-commerce search (Alipay insurance search). DaRL has three components: (1) DASA, which uses Mahalanobis and kNN cosine distances to select out-of-distribution samples from an unlabeled pool for annotation and augmentation; (2) a multi-stage fine-tuning strategy LPFT that first trains a classification head with LoRA, then fine-tunes all parameters, and finally interpolates the two sets of weights; and (3) an over-confidence calibration (OCC) auxiliary loss that uses KL divergence to reshape the predicted score distribution toward a hand-defined prior per relevance grade. The authors report offline experiments on industrial data showing improvements in F1 and accuracy on in-distribution, out-of-distribution, and production test sets, plus online A/B testing results with improved PV-CTR, PV-CVR, and manual relevance metrics, and state that DaRL-0.3B has been deployed.

Significance. If the claimed OOD generalization improvements hold, the paper would make a useful practical contribution: it addresses a real problem (LLM fine-tuning degrading under distribution shift in search relevance) and combines data augmentation, multi-stage training, and calibration into one deployable system. The paper's strengths include real industrial data, a deployed system with online A/B evidence, and an ablation study that isolates the contribution of each component. However, the central OOD generalization claim is not certified by the current benchmark: the OOD test set is constructed by the very same DASA selection rule used to build the augmented training set, and the LPFT interpolation coefficient is tuned on that same OOD test set. This is a load-bearing issue because the paper's headline contribution is robustness to distribution shift. The ID-side results, the DASA-versus-random comparison, and the online A/B evidence are less affected, but the quantitative OOD improvements in Table 2 and Table 3 should be interpreted with caution.

major comments (4)
  1. [§5.1.1, Eq. (4), Eq. (5)] The OOD test set T_ood is not an independent distribution shift: §5.1.1 states that T_ood is 'constructed according to the DASA module,' and Eq. (4) shows that DASA selects samples for D_aug using the same Mahalanobis and kNN cosine thresholds derived from the training representation. A model trained on D_aug is therefore evaluated on a distribution that was explicitly selected to match the same criterion used for augmentation. This selection-matched evaluation can inflate T_ood gains without demonstrating robustness to other real-world shifts such as temporal drift, new query intents, or unseen item categories. To certify the OOD generalization claim, the authors should evaluate on an OOD test set constructed independently of the DASA selection rule, e.g., time-based splits, query-intent splits, or item-category splits.
  2. [§5.2.4, Fig. 4, Eq. (8)] The interpolation coefficient alpha in Eq. (8) is selected by maximizing F1 on the OOD test T_ood (Fig. 4 reports F1 on T_ood as a function of alpha, with the optimal value at alpha = 0.6). This means the reported T_ood numbers embed test-set hyperparameter tuning, which is a form of leakage and further weakens the claim that the method improves OOD generalization. The authors should select alpha (and any other hyperparameters) using a validation split that is disjoint from T_ood, or report T_ood results for a pre-specified alpha.
  3. [§4.3, Eq. (9)] The target distribution q(x) in Eq. (9) is not fully specified. The paper says q(x) is 'over the labels (0, 1)', but then presents intervals such as [1−ρ, ρ] for IR and [2ρ, 1−2ρ] for WR. For the default ρ = 0.1, these intervals are [0.9, 0.1] and [0.2, 0.8]; the first is written in descending order and appears to be a typo (likely should be [ρ, 1−ρ] if the label for IR is 0). More importantly, the KL divergence L_kl = KL(p(x), q(x)) in Eq. (10) is asymmetric, and it is unclear whether q(x) is a proper probability distribution over the binary label or over the score interval. If q is intended as a target distribution over the scalar score, the KL direction and the normalization need clarification. The authors should specify the exact form of p(x), the intended label convention, and why this particular set of target intervals is appropriate for graded relevance.
  4. [§4.1, Eq. (2)] The Gaussian assumption P_id = N(x̄, Σ) is a strong modeling assumption for the representation distribution. If the training representation is multimodal or non-Gaussian, Mahalanobis distance may not be a valid OOD score, and DASA may select uninformative or even adversarial samples. The paper does not provide any diagnostic evidence for Gaussianity, nor does it compare against non-parametric OOD scores beyond the kNN term already used. This is secondary to the selection-matched evaluation issue, but it affects the interpretation of the DASA ablation (Exp3 and Fig. 3); a robustness check on the choice of OOD score would strengthen the paper.
minor comments (6)
  1. [Table 2] Table 2 reports no variance or confidence intervals for any metric except the significance asterisks. Since the paper claims statistically significant improvements (t-test, p < 0.05), the reader cannot assess the stability of the results; reporting mean and standard deviation over multiple runs would be more informative, especially for the small OOD improvements (e.g., 0.6678 vs 0.6431 F1 on T_ood).
  2. [§5.3] The online A/B test compares DaRL against a BERT-325M baseline rather than a fine-tuned LLM baseline such as AntGLM or ProMoT. This is a weaker comparison for the paper's central claim that DaRL helps LLM-based relevance modeling; the offline comparison against ProMoT is more informative, and the online section should acknowledge this discrepancy or provide an online comparison against an LLM baseline if available.
  3. [§5.1.1, Table 1] The description of T_prod is vague: 'comprising samples from T_id and T_ood, along with long-tail samples' makes it unclear how the production test set relates to the other two test sets and whether it overlaps with the augmented training set. Please clarify the construction and overlap of T_prod.
  4. [§4.2, Eq. (8)] The symbol phi is reused: Eq. (8) defines phi as the interpolated weights, but Eq. (6) and Eq. (7) use phi_lp and phi_ft. This is not an error, but the notation could be made more distinct to avoid confusion, especially because Eq. (8) is central to the LPFT module.
  5. [§4.1, Eq. (4)] The threshold d1 and d2 are said to be computed using 'the best ID F1 scores on the validation set', but the validation set is not described in Section 5.1. Please specify the validation set construction and how exactly the thresholds are derived from it.
  6. [§3, Eq. (1)] Equation (1) defines the objective as a sum of ID and OOD expected losses, but it is not used subsequently in the method section (L_ce and L_kl appear in Eq. (10) without reference to the two-distribution objective). The connection between the formulation and the actual training loss could be made explicit.

Circularity Check

2 steps flagged · score 6.0 of 10

OOD gains are measured on a test set constructed by the same DASA selection rule used to build the augmented training set, and the LPFT interpolation coefficient is tuned on that same OOD test set.

  1. self definitional [Section 5.1.1 (Datasets) and Section 4.1, Eq. 4-5]
    "Test set T_ood is the out-of-distribution test dataset which is constructed according to the DASA module. ... U_ood = {x_u : d_Mahal(x_u,P_id) > d1 ∧ d_knn(x_u) > d2}. ... D_aug = D_id ∪ D_ood."

    The OOD test set is not an independent sample from a naturally shifted distribution; it is generated by the same Mahalanobis-plus-kNN selection rule (Eq. 4) that builds the augmented training set D_aug from the same unlabeled pool U. A model trained on D_aug is therefore evaluated on a distribution that the DASA module was explicitly designed to select and cover. Any model that fits the DASA-selected region will score well on T_ood, so the reported OOD gain partly reflects the selection criterion rather than robust generalization to unrelated real-world shifts (e.g., temporal drift, new query intents, unseen item categories).

  2. fitted input called prediction [Section 5.2.4, Fig. 4]
    "F1 results for the OOD test T_ood on the variation w.r.t. interpolation coefficient α in Eq.8 ... The optimal performance is attained at alpha = 0.6."

    The interpolation coefficient α is selected by sweeping F1 on the OOD test set T_ood itself, and the same T_ood numbers are then reported as the method's OOD performance. This embeds test-set hyperparameter tuning into the headline OOD result: the reported figure is fitted to the evaluation set rather than being a prediction on an untouched holdout. It inflates the absolute OOD score, although relative comparisons at a fixed α or on other test sets may remain informative.

full rationale

The central OOD claim is partially circular. T_ood is 'constructed according to the DASA module', the same module that selects D_ood from the unlabeled pool U to form D_aug, so the evaluation distribution is matched to the training augmentation criterion by construction. Additionally, the LPFT interpolation coefficient is tuned directly on T_ood (Fig. 4), making the reported OOD number a fitted result. The Gaussian-distribution assumption for the representation space is a secondary risk, not the main circularity. The in-distribution results, the threshold selection based on ID validation F1, the random-vs-DASA comparison at fixed settings, and the online A/B evidence are largely independent of this critique and appear self-contained. The score reflects partial circularity in the OOD evaluation, not a collapse of the whole paper.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

DaRL rests on four tunable parameters (rho, alpha, d1, d2) and three domain assumptions. No new physical or conceptual entities are introduced.

free parameters (4)
  • rho = 0.1 (grid-searched)
    Distribution smooth factor in Eq. 9; sets target probability distributions for IR/WR/SR classes and is tuned by grid search.
  • alpha = 0.6
    Interpolation coefficient in Eq. 8 for LPFT weight interpolation; optimal value found in Fig. 4.
  • d1 (Mahalanobis distance threshold) = not reported
    Threshold in Eq. 4 for selecting OOD samples; chosen using best ID F1 on validation set.
  • d2 (kNN cosine distance threshold) = not reported
    Threshold in Eq. 4 for selecting OOD samples; chosen using best ID F1 on validation set.
assumptions (3)
  • domain assumption Training representation distribution is Gaussian, P_id = N(xbar, Sigma).
    Section 4.1, Eq. 2; needed for Mahalanobis distance to be a meaningful OOD score.
  • domain assumption Unlabeled search logs contain OOD samples that are not covered by the training distribution and are worth labeling.
    Section 3 and Section 4.1; the entire DASA module depends on this premise.
  • ad hoc to paper The target distributions q(x) in Eq. 9 are appropriate calibration targets for relevance degrees.
    Section 4.3; the intervals are engineered for this task and rho is tuned on validation data, with no external justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning." pith.science (2026). https://pith.science/paper/M7JP5IHR

@misc{pith2026241212504,
  author       = {Pith},
  title        = {Pith review of: Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7JP5IHR}},
  note         = {Machine review of arXiv:2412.12504}
}
read the original abstract

With the rapid advancement of pre-trained large language models (LLMs), recent endeavors have leveraged the capabilities of LLMs in relevance modeling, resulting in enhanced performance. This is usually done through the process of fine-tuning LLMs on specifically annotated datasets to determine the relevance between queries and items. However, there are two limitations when LLMs are naively employed for relevance modeling through fine-tuning and inference. First, it is not inherently efficient for performing nuanced tasks beyond simple yes or no answers, such as assessing search relevance. It may therefore tend to be overconfident and struggle to distinguish fine-grained degrees of relevance (e.g., strong relevance, weak relevance, irrelevance) used in search engines. Second, it exhibits significant performance degradation when confronted with data distribution shift in real-world scenarios. In this paper, we propose a novel Distribution-Aware Robust Learning framework (DaRL) for relevance modeling in Alipay Search. Specifically, we design an effective loss function to enhance the discriminability of LLM-based relevance modeling across various fine-grained degrees of query-item relevance. To improve the generalizability of LLM-based relevance modeling, we first propose the Distribution-Aware Sample Augmentation (DASA) module. This module utilizes out-of-distribution (OOD) detection techniques to actively select appropriate samples that are not well covered by the original training set for model fine-tuning. Furthermore, we adopt a multi-stage fine-tuning strategy to simultaneously improve in-distribution (ID) and OOD performance, bridging the performance gap between them. DaRL has been deployed online to serve the Alipay's insurance product search...

Figures

Figures reproduced from arXiv: 2412.12504 by the authors.

Figure 1
Figure 1. Overview of our proposed framework. DaRL contains three major components: (1) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparing the effects of the Over-Confidence Cali [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. F1 comparison between the random sampling sam [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: F1 results for the OOD test T𝑜𝑜𝑑 on the variation w.r.t. interpolation coefficient 𝛼 in Eq.8 for the Linear-Probing then Fine-Tuning Module. 5.2.4 Parameter Sensitivity of Linear-Probing then Fine-Tuning Mod￾ule. Based on the trends depicted in [PITH_FULL_IMAGE:figure…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 38 canonical work pages

  1. [1]

    Cory Barr, Rosie Jones, and Moira Regelson. 2008. The Linguistic Structure of English Web-Search Queries. In 2008 Conference on Empirical Methods in Natural Language Processing. 1021–1030

    work page 2008

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  3. [3]

    Zeyuan Chen, Wei Chen, Jia Xu, Zhongyi Liu, and Wei Zhang. 2023. Beyond Se- mantics: Learning a Behavior Augmented Relevance Model with Self-supervised Learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4516–4522

    2023

  4. [4]

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre- Training with Whole Word Masking for Chinese BERT. https://doi.org/10.1109/ TASLP.2021.3124365

    arXiv 2021

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171–4186

    2019

  6. [6]

    Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2024. Short- cut Learning of Large Language Models in Natural Language Understanding. Commun. ACM 67, 1 (2024), 110–120

    work page 2024

  7. [7]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 320–335

    work page 2022

  8. [8]

    Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, and et al. 2023. Perspectives on Large Language Models for Rel- evance Judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. 39–50

    work page 2023

Show all 50 references
  1. [9]

    Yixing Fan, Jiafeng Guo, Xinyu Ma, Ruqing Zhang, Yanyan Lan, and Xueqi Cheng

  2. [10]

    Bruce Croft

    Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval. InProceedings of the 25th ACM International Conference on Information and Knowledge Management. 55–64

    2016

  3. [11]

    Bruce Croft

    Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. Semantic Match- ing by Non-Linear Word Transportation for Information Retrieval. InProceedings of the 25th ACM International Conference on Information and Knowledge Manage- ment. 701–710

    2016

  4. [12]

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a Unified View of Parameter-Efficient Transfer Learning. In The Tenth International Conference on Learning Representations

    2022

  5. [13]

    Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In Proceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Info...

    2021

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representa- tions. 4724 CIKM ’24, October 21–25, 2024, Boi...

    2022

  7. [15]

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management. 2333–2338

    2013

  8. [16]

    Shan Jiang, Yuening Hu, Changsung Kang, Tim Daly Jr., Dawei Yin, Yi Chang, and ChengXiang Zhai. 2016. Learning Query and Document Relevance from a Web-scale Click Graph. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Re...

    2016

  9. [17]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data 7, 3 (2021), 535–547

    2021

  10. [18]

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine-Tuning can Distort Pretrained Features and Underperform Out- of-Distribution. In The Tenth International Conference on Learning Representations

    2022

  11. [19]

    Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. In Advances in Neural Information Processing Systems 31. 7167–7177

    2018

  12. [20]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Sh...

    2022

  13. [21]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692

    2019 arXiv

  14. [22]

    Haodong Lu, Dong Gong, Shuo Wang, Jason Xue, Lina Yao, and Kristen Moore

  15. [23]

    Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. TwinBERT: Distilling Knowledge to Twin-Structured Compressed BERT Models for Large-Scale Retrieval. In Pro- ceedings of the 29th ACM International Conference on Information and Knowledge Management. 2645–2652

    2020

  16. [24]

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine- Tuning LLaMA for Multi-Stage Text Retrieval. arXiv:2310.08319

    2023 arXiv

  17. [25]

    Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Rel- evance Estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2230–2235

    2023

  18. [26]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. A Comprehensive Overview of Large Language Models. arXiv:2307.06435

    2023 arXiv

  19. [27]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and et al

  20. [28]

    Wainwright, and et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, and et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35. 27730–27744

    2022

  21. [29]

    Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Chen

  22. [30]

    Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning. 2024. eCeLLM: Gener- alizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data. arXiv:2402.08831 [31] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siam...

    2024 arXiv

  23. [32]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3, 4 (2009), 333–389

    2009

  24. [33]

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational ...

    2022

  25. [34]

    Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-Questions for Few- Shot Text Classification and Natural Language Inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. 255–269

    2021

  26. [35]

    Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. Learning Semantic Representations Using Convolutional Neural Networks for Web Search. In Proceedings of the 23rd International World Wide Web Conference, Companion Volume. 373–374

    2014

  27. [36]

    Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-Distribution Detection with Deep Nearest Neighbors. In International Conference on Machine Learning. PMLR, 20827–20840

    2022

  28. [37]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models can Accurately Predict Searcher Preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1930–1940

    2024

  29. [38]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. LLaMA: Open and Efficient Foundation ...

    2023 arXiv

  30. [39]

    Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit Dhillon, and Sanjiv Kumar. 2024. Two-stage LLM Fine-tuning with Less Spe- cialization and More Generalization. In The Twelfth International Conference on Learning Representations

    2024

  31. [40]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In The Twelfth International Conference on Learning Representations. https://openreview.net/fo...

    2024

  32. [41]

    Jun Xu, Xiangnan He, and Hang Li. 2020. Deep Learning for Matching in Search and Recommendation. Foundations and Trends in Information Retrieval 14, 2–3 (2020), 102–288

    2020

  33. [42]

    Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, and Wai Lam. 2024. Unveiling the Generalization Power of Fine-Tuned Large Language Models. arXiv:2403.09162

    2024 arXiv

  34. [43]

    Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021. Generalized Out-of-Distribution Detection: A Survey. arXiv:2110.11334

    2021 arXiv

  35. [44]

    Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. 2023. Out- of-Distribution Generalization in Natural Language Processing: Past, Present, and Future. In Proceedings of the 2023 Conference ...

    2023

  36. [45]

    Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective. In Findings of the Association for Computationa...

    2023

  37. [46]

    Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, and Jie Tang. 2022. Parameter- Efficient Tuning Makes a Good Classification Head. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 7576–7586

    2022

  38. [47]

    Wen Zan, Yaopeng Han, Xiaotian Jiang, Yao Xiao, Yang Yang, Dayao Chen, and Sheng Chen. 2023. SPM: Structured Pretraining and Matching Architectures for Relevance Modeling in Meituan Search. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Ma...

    2023

  39. [2016]

    In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence

    Text Matching as Image Recognition. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. 2793–2799

  40. [2021]

    In The Web Conference 2021

    A Linguistic Study on Relevance Modeling in Information Retrieval. In The Web Conference 2021. 1053–1064

    2021

  41. [2023]

    arXiv:2303.08774

    GPT-4 Technical Report. arXiv:2303.08774

    arXiv

  42. [2024]

    In The Twelfth International Conference on Learning Representations

    Learning with Mixture of Prototypes for Out-of-Distribution Detection. In The Twelfth International Conference on Learning Representations

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.