Pith. sign in

REVIEW 5 major objections 7 minor 46 references

CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CPRM, a three-module continual pre-training recipe for LLMs, lifts commercial search relevance by 2.48 accuracy, 2.47 F1, and 2.45 AUC points over GLM-2B and delivers a 0.32% online click-through lift.

desk verdict A plausible industrial recipe with a real data-leakage question mark; worth refereeing but the offline gains are not yet auditable. read the letter →

arxiv 2412.01269 v5 pith:ZWRZI4G7 submitted 2024-12-02 cs.AI cs.CLcs.IRcs.LG

Kaixin Wu , Yixin Ji , Zeyuan Chen , Qiang Wang , Cunxiang Wang , Hong Liu , Baijun Ji , Jia Xu
show 4 more authors
Zhongyi Liu Jinjie Gu Yuan Zhou Linjian Mo
This is my paper · ORCID
classification cs.AIcs.CLcs.IRcs.LG
keywords continualpre-trainingrelevancemodelingcommercialsearchin-contextlearninglargelanguagemodelsquery-itemmatchingreadingcomprehensiondistillationclicklogs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a three-module continual pre-training recipe lets a 2-billion-parameter LLM match or beat much larger models on commercial search relevance. The recipe jointly pre-trains on queries and multi-field item text, pre-trains on sequences of semantically related queries or items built from click logs, and distills reading-comprehension outputs from a larger teacher LLM. In offline evaluation the full framework improves accuracy by 2.48 points, F1 by 2.47 points, and AUC by 2.45 points over the GLM-2B baseline, putting it on par with GLM-10B. Online A/B testing reports a statistically significant 0.32% lift in valid page-view click-through rate, and the model has served traffic for over nine months. The central claim is that domain-specific and task-structured pre-training data, rather than raw scale, drive relevance-modeling gains.

What carries the argument

The load-bearing mechanism is the continual pre-training data construction. DKE joins queries and multi-field item text into a single pre-training sequence with 'startofpiece/endofpiece' segment tokens, trained with token-level and segment-level masked language modeling. ICP starts from click logs, builds Q2I and I2Q mappings sorted by click count, filters out pairs whose Contriever cosine similarity falls below a threshold σ (Eq. 4), and then orders the remaining items or queries by similarity before concatenating them into one pre-training example; the ordering is meant to teach the model graded relevance strength. RCD prompts a teacher LLM (Qwen2-72B) to summarize, paraphrase, generate background knowledge, and produce diverse queries with explanations from the item's structured fields, turning underused fields like description into pre-training text that transfers knowledge into the student.

What would settle it

Run ICP with a sweep of σ values (including no filtering) and with randomly shuffled item/query order within instances; if AUC gains over the GLM-2B baseline do not shrink or vanish when the semantic filter or the similarity ordering is removed, the paper's explanation of ICP's mechanism is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that relevance modeling in commercial search is improved more by how an LLM is continually pre-trained than by increasing model size alone. The authors build CPRM from three components: DKE jointly pre-trains a query with its top clicked item, using special segment tokens and a mix of token- and segment-level masked language modeling; ICP constructs training instances that concatenate a query with its semantically related items, or an item with its related queries, ordered by similarity; RCD uses a teacher LLM to generate summaries, background knowledge, and diversified queries from structured item text, then pre-trains the student on that generated data. When all three are combined on GLM-2B, the model reaches 93.64% accuracy, 94.42% F1, and 93.49% AUC, exceeding every baseline except the two largest GLM variants and matching GLM-10B's AUC within 0.13 points. The paper reports that each module contributes independently, with ICP giving the largest single AUC gain of 1.54 points, and that the full model maintains its advantage at every training step and on long queries.

Load-bearing premise

The strongest single module, ICP, depends on an unstated threshold σ in Eq. 4 that decides which click-derived pairs count as semantically related and on the similarity-based ordering that teaches relevance strength; if the threshold or ordering does not reflect true relevance structure, ICP's AUC gain of 1.54 points would not transfer to other settings, and the paper does not report σ or ablate it.

Editorial extensions

If this is right

  • Combining all three modules yields the best result: DKE+ICP+RCD tops every moderate-size baseline and matches GLM-10B, so the framework can substitute for parameter growth.
  • ICP alone contributes the largest single AUC gain (1.54 points), suggesting that context built from semantically ordered queries/items is the most powerful of the three signals.
  • The full CPRM model gains 15.85% AUC on queries longer than 15 characters, indicating that the pre-training recipe specifically improves long-query understanding.
  • The approach is stable across training steps: unlike the SFT-only baseline, which dips at 16K steps, all CPRM variants improve monotonically, implying the pre-training data eases domain adaptation.
  • Online, a 0.32% PVCTR lift at 95% confidence, a 0.75% Badcase@10 reduction, and a 4.71% error-filtering-rate decrease justify deployment; the model has been in production for over nine months.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ICP ordering-by-similarity design implies a soft graded-relevance signal is being learned without explicit labels; a natural extension is to test whether the learned score correlates with human graded judgments.
  • Because the teacher LLM generates queries and summaries without filtering, the RCD gains may partly reflect data augmentation rather than distillation; a controlled test with randomly selected generated data would separate these effects.
  • The σ threshold in Eq. 4 is never reported nor ablated, so a sensitivity analysis across σ values is the clearest way to test how much of ICP's gain depends on the semantic filter versus the click-sourced co-occurrence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes CPRM, a continual pre-training framework for LLM-based query-item relevance modeling in a commercial mini-app search engine. The framework combines domain knowledge enhancement via joint query-item masked language modeling, in-context pre-training on semantically related queries/items mined from click logs, and reading-comprehension distillation from a larger teacher LLM. Experiments on GLM-2B with a proprietary SFT dataset show additive gains of +2.48 Acc, +2.47 F1, and +2.45 AUC over GLM-2B (Table 2), and a two-week online A/B test reports a statistically significant 0.32% PVCTR lift and improvements in human-evaluated relevance metrics. The model is stated to have been serving production traffic for over nine months.

Significance. If the empirical claims hold up, this is a valuable industrial contribution: it provides a modular recipe for adapting open-weight LLMs to a specialized relevance domain, with a clear decomposition that lets practitioners see the contribution of each module. The paper also contains several good practices: it compares with a range of model sizes and architectures, includes an online A/B test, and reports long-query and training-step analyses. However, the entire case rests on self-reported, proprietary measurements. The absence of data-disjointness checks, threshold sensitivity analyses, and statistical details means the reader cannot currently distinguish a genuine transfer effect from memorization or from tuning to a specific filtering threshold. These concerns are addressable, but they are load-bearing for the central claim.

major comments (5)
  1. [Section 5.1, Table 2] The offline gains in Table 2 (e.g., rows 8 vs 18: +2.48 Acc, +2.47 F1, +2.45 AUC) are presented as evidence of generalization, but the manuscript never establishes that the continual pre-training corpora and the human-annotated test set are disjoint. Section 5.1 reports that DKE data are built from the top 500K most visited items, ICP from real-world click logs, and RCD from item-derived generated queries, while the test set (35,057 samples, 5,426 queries, 8,406 items) is also drawn from real mini-app search results. In such a setting, exact or near-duplicate query-item pairs are likely to appear in both; models could then memorize co-occurrence rather than learn transferable relevance. Please report overlap statistics at the query, item, and pair level between each pre-training corpus and the test set, and, if feasible, re-run the ablation on a disjoint held-out set.
  2. [Section 5.3, Table 2 and Section 6] The claim in Table 2's caption and Section 5.3 that 'improvements over variants are statistically significant with p < 0.05' is not backed by any description of the test, the number of runs, or the variance across runs. No confidence intervals are reported for Acc., F1, or AUC in Table 2 or Figure 4, and Section 6 similarly states a 'statistically significant increase of 0.32% in valid PVCTR at a 95% confidence level' without giving the confidence interval, the test statistic, or whether the 0.32% is relative or absolute. Please specify the significance test (e.g., paired bootstrap, McNemar), the number of model runs, and report confidence intervals for the main offline and online metrics.
  3. [Section 4.2, Eq. (4), Table 2] The ICP component, which provides the largest single-module gain (+1.54 AUC in Table 2, row 15), depends on the threshold σ in Eq. (4) and on the similarity-based reordering described in Section 4.2. The value of σ is never reported, and there is no sensitivity analysis or ablation of the fine-screening and reordering stages. If σ is chosen too loosely, semantically unrelated pairs remain; if too tightly, the constructed context loses diversity. Similarly, the choice of ascending order is asserted to 'implicitly indicate the strength of relevance' but is not compared with any alternative ordering. Please report σ, vary it over a reasonable range, and ablate the filtering and ordering choices to show that the ICP gains are not artifacts of a particular threshold.
  4. [Section 4.3, Table 2] The RCD module contributes +0.42 AUC in Table 2 (row 16), but the teacher-generated queries, summaries, and explanations are used without any quality filtering. The paper does not report manual evaluation of the generated data, a filtering criterion, or an analysis of how generation errors affect downstream relevance. Since a teacher LLM can produce fluent but semantically incorrect content, please provide evidence that the generated instances are accurate (e.g., sample-based human evaluation, agreement with click labels, or ablation excluding low-confidence generations).
  5. [Section 6] The online A/B test is the only out-of-sample support for the method, but the description is too brief to assess: no information is given on the number of users or queries in the 5% experiment traffic, the exact duration beyond 'two weeks,' how 'valid' clicks are defined, the baseline system's precise configuration, or the practical significance of a 0.32% lift. The human evaluation metrics (0.75% Badcase@10 reduction and 4.71% Error Filtering Rate decrease) also lack confidence intervals or sample sizes. Please expand this section with the missing experimental protocol details.
minor comments (7)
  1. [Abstract and Title] The title and abstract use 'A LLM-based'; the correct article is 'An LLM-based.' Similar grammar issues appear throughout the text (e.g., 'We conducts' in Section 5.3).
  2. [Table 2] Rows 8 and 13 report identical GLM-2B results (91.16 Acc, 91.95 F1, 91.04 AUC). Please state explicitly whether row 13 is the same checkpoint used as the starting point for continual pre-training or a separately trained baseline.
  3. [Section 4.1] The mixing coefficient α in Eq. (3) is set to 0.7 without any sensitivity analysis. Since DKE contributes +1.11 AUC, please report the influence of α and of the segment-masking ratio on downstream relevance performance.
  4. [Figure 4] The long-query analysis reports a 15.85% AUC improvement for queries longer than 15 characters, but the number of test queries in each length bucket is not given. Without confidence intervals and bucket sizes, this striking gain may be driven by a small subset of examples.
  5. [Figure 5] The y-axis label and metric for the training-step curves are missing. Please label the metric (presumably AUC) and add a legend identifying each series.
  6. [Section 5.2] ChatGPT and GPT-4 are evaluated with 8-shot prompting but without fine-tuning, so calling them 'strong baselines' in the abstract is misleading. Please describe them as zero/few-shot reference points rather than trained baselines.
  7. [Section 7] The deployment description does not clarify whether the online GLM-0.3B model is also CPRM-pretrained or only receives cached scores from the CPRM-trained GLM-2B model. Please state how the evaluated CPRM model relates to the deployed two-tier scoring system.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; CPRM's reported gains are empirical results, not consequences of a fitted constant or a self-referential definition.

full rationale

The paper's central claim is an empirical comparison: Table 2 reports that GLM-2B plus DKE, ICP, and RCD reaches 93.64% accuracy, 94.42% F1, and 93.49% AUC versus 91.16/91.95/91.04 for the GLM-2B baseline, and Section 6 reports a statistically significant 0.32% PVCTR lift in an online A/B test. None of these numbers is derived from the method's own parameters: the pre-training objectives in Eq. (3) are standard MLM losses, the ICP construction in Section 4.2 uses click logs filtered by an external Contriever model, and the RCD data in Section 4.3 is generated by a teacher LLM. The threshold sigma in Eq. (4) is a data-construction hyperparameter, not a fitted constant that is later reported as a prediction; its absence from the ablation study is a robustness gap, not circularity. The paper does cite prior Ant Group relevance-modeling papers (Chen et al., 2023, 2024; Liu et al., 2024) in the related-work discussion, but these citations are contextual and are not used to justify the framework's design or to exclude alternative methods, so they are not load-bearing. The offline evaluation could be threatened by overlap between the pre-training corpora and the SFT test set, but the paper never claims disjointness and such contamination would be a data-hygiene issue; it would not make the measured gains equivalent to the training inputs by construction. No equation in the paper defines a predicted quantity in terms of the same quantity from the data, so no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of click-log-based similarity data and on the teacher-generated synthetic data; both are domain assumptions without released artifacts or sensitivity analysis.

free parameters (2)
  • alpha (α) = 0.7
    Weight between token-level and segment-level MLM losses in Eq. 3, set by hand with no sensitivity analysis.
  • sigma (σ) = not reported
    Contriever cosine similarity threshold for fine screening in Eq. 4; the paper never states its value, and the ICP data quality depends on it.
assumptions (4)
  • domain assumption Click logs imply semantic relatedness between queries and items in the same Q2I or I2Q mapping
    Coarse screening in Section 4.2 treats co-clicked items under a query as a preliminary semantically related set, which is noisy.
  • domain assumption Contriever cosine similarity correctly separates semantically related from unrelated pairs at some threshold σ
    Fine screening in Eq. 4 filters instances based on the similarity between Query and Ik; errors in Contriever propagate to ICP data.
  • domain assumption Teacher LLM outputs (summaries, paraphrases, generated queries) are accurate and useful for pre-training
    RCD, Section 4.3, assumes the teacher's reading comprehension produces fluent domain knowledge and queries that improve the student; no quality filter on teacher outputs is described.
  • domain assumption The held-out test and validation sets do not overlap with the pre-training data
    No deduplication is reported between pre-training data (from click logs) and the SFT/valid/test sets, so leakage could inflate gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search." pith.science (2026). https://pith.science/paper/ZWRZI4G7

@misc{pith2026241201269,
  author       = {Pith},
  title        = {Pith review of: CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZWRZI4G7}},
  note         = {Machine review of arXiv:2412.01269}
}
read the original abstract

Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

Figures

Figures reproduced from arXiv: 2412.01269 by the authors.

Figure 1
Figure 1. Joint queries and multi-field item for pre [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. In-context pre-training instances construction. The left and right figures represent the ICP instances [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prompt for reading comprehension on item. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Performance of different training steps. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Deployment of the CPRM relevance model. functions to mini-apps for over nine months. These results suggest that our proposed method can effec￾tively enhance relevance models’ performance in real-world search systems. 7 Model Deployment LLMs have achieved significant pe…
Figure 7
Figure 7. Figure 7: Case study [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: SFT data examples [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 9 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038

    arXiv 2016

  6. [6]

    Zeyuan Chen, Wei Chen, Jia Xu, Zhongyi Liu, and Wei Zhang. 2023. Beyond semantics: Learning a behavior augmented relevance model with self-supervised learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4516--4522

    work page 2023

  7. [7]

    Zeyuan Chen, Haiyan Wu, Kaixin Wu, Wei Chen, Mingjie Zhong, Jia Xu, Zhongyi Liu, and Wei Zhang. 2024. http://arxiv.org/abs/2408.09439 Towards boosting llms-driven relevance modeling with progressive retrieved behavior-augmented prompting

    arXiv 2024

  8. [8]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. http://arxiv.org/abs/2306.16092 Chatlaw: Open-source legal large language model with integrated external knowledge bases

    arXiv 2023

Show all 46 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://api.semanticscholar.org/CorpusID:52967399 Bert: Pre-training of deep bidirectional transformers for language understanding . In North American Chapter of the Association for Computational Linguistics

    2019

  3. [11]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022

  4. [12]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. http://arxiv.org/abs/2305.09137 Pre-training to learn in context

    2023 arXiv

  5. [13]

    Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. http://arxiv.org/abs/2004.09733 Train no evil: Selective masking for task-guided pre-training

    2020 arXiv

  6. [14]

    Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. http://arxiv.org/abs/2004.10964 Don't stop pretraining: Adapt language models to domains and tasks

    2020 arXiv

  7. [15]

    Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. Advances in neural information processing systems, 27

    2014

  8. [16]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. http://arxiv.org/abs/2112.09118 Unsupervised dense information retrieval with contrastive learning

    2022 arXiv

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization

    2017 arXiv

  10. [18]

    Lakshya Kumar and Sagnik Sarkar. 2021. http://arxiv.org/abs/2107.08291 Neural search: Learning query and product representations in fashion e-commerce

    2021 arXiv

  11. [19]

    Yijiang Lian, Zhijie Chen, Jinlong Hu, Kefeng Zhang, Chunwei Yan, Muchenxuan Tong, Wenying Han, Hanju Guan, Ying Li, Ying Cao, Yang Yu, Zhigang Li, Xiaochun Liu, and Yue Wang. 2019. http://arxiv.org/abs/1902.00592 An end-to-end generative retrieval method for sponsored search ...

    2019 arXiv

  12. [20]

    Hong Liu, Saisai Gong, Yixin Ji, Kaixin Wu, Jia Xu, and Jinjie Gu. 2024. Boosting llm-based relevance modeling with distribution-aware robust learning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4718--4725

    2024

  13. [21]

    Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019 a . http://arxiv.org/abs/1909.07606 K-bert: Enabling language representation with knowledge graph

    2019 arXiv

  14. [22]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  15. [23]

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. http://arxiv.org/abs/2110.15943 Metaicl: Learning to learn in context

    2022 arXiv

  16. [24]

    OpenAI. 2024. http://arxiv.org/abs/2303.08774 Gpt-4 technical report

    2024 arXiv

  17. [25]

    Palangi, L

    H. Palangi, L. Deng, Y. Shen, J. Gao, X. He, J. Chen, X. Song, and R. Ward. 2015. http://arxiv.org/abs/1412.6629 Semantic modelling with long-short-term memory for information retrieval

    2015 arXiv

  18. [26]

    Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30

    2016

  19. [27]

    a ckstr \

    Ankur P Parikh, Oscar T \"a ckstr \"o m, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933

    2016 arXiv

  20. [28]

    Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. http://arxiv.org/abs/2001.04063 Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training

    2020 arXiv

  21. [29]

    Jinfeng Rao, Linqing Liu, Yi Tay, Wei Yang, Peng Shi, and Jimmy Lin. 2019. Bridging the gap between relevance matching and semantic matching for short text similarity modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9t...

    2019

  22. [30]

    Soumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang, Reid Pryzant, Wenhao Yu, Chenguang Zhu, and Xiang Ren. 2023. http://arxiv.org/abs/2212.09282 Apollo: A simple approach for adaptive pretraining of language models for logical reasoning

    2023 arXiv

  23. [31]

    Timo Schick and Hinrich Schütze. 2021. http://arxiv.org/abs/2001.07676 Exploiting cloze questions for few shot text classification and natural language inference

    2021 arXiv

  24. [32]

    Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gr \'e goire Mesnil. 2014. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pages 373--374

    2014

  25. [33]

    Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. 2024. http://arxiv.org/abs/2310.10638 In-context pretraining: Language modeling beyond document boundaries

    2024 arXiv

  26. [34]

    Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, and Yunlong Jiao. 2023. http://arxiv.org/abs/2305.13002 Rethinking semi-supervised learning with language models

    2023 arXiv

  27. [35]

    Rothschild, Daniel G

    Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman. 2023. http://arxiv.org/abs/2307.03744 Comparing traditional and llm-based search for consumer choice: A randomized experiment

    2023 arXiv

  28. [36]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542

    2023 arXiv

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  31. [39]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. http://arxiv.org/abs/2303.17564 Bloomberggpt: A large language model for finance

    2023 arXiv

  32. [40]

    Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Linlin Huang, Qian Wang, and Dinggang Shen. 2023. http://arxiv.org/abs/2304.01097 Doctorglm: Fine-tuning your chinese doctor is not a herculean task

    2023 arXiv

  33. [41]

    Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Hakkani-Tür. 2023. http://arxiv.org/abs/2302.09170 Kilm: Knowledge injection into encoder-decoder language models

    2023 arXiv

  34. [42]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  35. [43]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. http://arxiv.org/abs/1906.08237 Xlnet: Generalized autoregressive pretraining for language understanding

    2020 arXiv

  36. [44]

    Shaowei Yao, Jiwei Tan, Xi Chen, Juhao Zhang, Xiaoyi Zeng, and Keping Yang. 2022. Reprbert: Distilling bert to an efficient representation-based relevance model for e-commerce. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4363--4371

    2022

  37. [45]

    Wangchunshu Zhou, Ronan Le Bras, and Yejin Choi. 2023. http://arxiv.org/abs/2306.02388 Commonsense knowledge transfer for pre-trained language models

    2023 arXiv

  38. [46]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. 2024. http://arxiv.org/abs/2308.07107 Large language models for information retrieval: A survey

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.