Pith. sign in

REVIEW 4 major objections 5 minor 29 references

MST-R: Multi-Stage Tuning for Retrieval Systems and Metric Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-stage retrieval tuning recipe lifts Recall@10 to 0.8746 on ObliQA and proves the RePASs metric can be saturated by a trivial passage-concatenation answer.

desk verdict Useful engineering plus a genuinely cautionary metric analysis; retrieval claims need a clearer train/test split statement before they are fully trustworthy. read the letter →

arxiv 2412.10313 v1 pith:D7YNAPCN submitted 2024-12-13 cs.IR cs.LG

Yash Malviya , Karan Dhingra , Maneesh Singh This is my paper · ORCID
classification cs.IRcs.LG
keywords retrieval-augmentedgenerationregulatorycompliancemulti-stagetuninghybridretrievalreciprocalrankfusionhardnegativeminingRePASsmetricdomainadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a retrieval system for regulatory documents can be made dramatically more effective by a multi-stage tuning recipe: fine-tune dense encoders on the target corpus with hard negatives, fuse sparse and dense results by reciprocal rank fusion, then fine-tune a cross-encoder reranker on the top retrieved passages. On the ObliQA benchmark used by the RIRAG challenge, this system reaches Recall@10 of 0.8746 and MAP@10 of 0.7601, beating the BGE baseline by 12.1% and 23% respectively. The paper also tries to establish that the RePASs answer-quality metric is not measuring what it claims: a trivial answer formed by concatenating the retrieved passages receives RePASs=0.947, far above a Llama 3.1 answer at 0.41, and under two stated assumptions the metric can be driven to its maximum value of 1 by such a trivial model. The authors use this anomaly to argue that regulatory QA needs question-grounded or multi-faceted evaluation metrics, not just reference-free entailment coverage. A sympathetic reading: the contribution is a strong, reproducible retrieval pipeline plus a provable weakness in the challenge's scoring metric.

What carries the argument

The load-bearing mechanism is the multi-stage tuning pipeline (MST-R): (i) hard-negative contrastive fine-tuning of dense encoders against the target corpus, (ii) reciprocal rank fusion of sparse (BM25), dense (BGE, E5-FT), and query-to-query retrievers, and (iii) fine-tuning a cross-attention reranker on the top-K Level-1 outputs. The secondary mechanism is the analysis of the RePASs metric, defined as (Es − Cs + OCs + 1)/3 with sentence-level entailment/contradiction maxima and an obligation-coverage indicator; the paper proves that a trivial pass-through answer can saturate it, so the metric rewards copying rather than synthesis.

What would settle it

Take the ObliQA test queries, remove from the gold set all passages that are near-duplicates of the ground-truth passage (using string similarity or the NLI-based entailment criterion the paper uses), and re-run MST-R and the BGE baseline; if the Recall@10 gap narrows or reverses, the completeness assumption, not the retrieval method, drove the reported gains. Alternatively, run Passage Concat on a dataset where the retrieved passages are deliberately irrelevant to the query; if RePASs still approaches 1, the metric is confirmed to ignore question relevance entirely.

Watch

Extended reading notes

Core claim

The central claim is that domain adaptation of each retrieval stage—rather than a new architecture—accounts for the gains. Level 1 combines four retrievers: BM25, a prompt-conditioned BGE dense retriever, an E5 encoder fine-tuned on ObliQA with triplet loss and online hard mining, and a Q2Q retriever that matches new queries to similar training questions; their ranked lists are merged by reciprocal rank fusion. Level 2 fine-tunes a MiniLM cross-encoder on the Level-1 top-K results with hard and easy negatives, providing a reranking score. On the ObliQA test set the full system obtains Recall@10=0.8746 and MAP@10=0.7601, with Level 1 alone at 0.832 and 0.6914, showing the reranker adds roughly 5% relative Recall@10. For answer generation, the paper compares a Llama 3.1 Instruct 8B model with two trivial baselines—concatenating passages and joining them into a single line—and finds both outperform the LLM on RePASs; Passage Concat scores 0.947. The paper states and proves Claim 1: under Assumption 1 (a sentence entails itself and contradicts nothing) and Assumption 2 (no sentences in the corpus contradict each other), Passage Concat attains the maximum RePASs score of 1.

Load-bearing premise

The ObliQA ground truth is assumed to be the unique correct passage for each query; if that fails—and the paper's own Appendix A.4 shows near-duplicate passages carrying the same regulatory obligation—then the reported Recall@10 and MAP@10 understate true retrieval performance and the state-of-the-art claim depends on an incomplete gold list.

Editorial extensions

If this is right

  • The reported gains suggest that fine-tuning retriever components on domain data is sufficient to move regulatory retrieval from roughly 78% Recall@10 to 87.5%, without replacing the underlying encoders or changing the architecture.
  • The reranker contributes a measurable, though modest, improvement over the fused Level-1 list (0.832 to 0.8746 Recall@10), so the three-stage design is what the authors recommend for similar domain-adaptation tasks.
  • Because Passage Concat and Single Line beat a general-purpose LLM on RePASs, any leaderboard that relies on RePASs alone can be topped by a system that just echoes retrieved text.
  • The RePASs-N experiments with CNN/DailyMail show that increasing NLI context window size improves entailment and contradiction scores, suggesting the metric's behavior is sensitive to implementation details.
  • The paper's answer-generation module is intentionally frozen; the authors state the reported answer-quality numbers are a lower bound, since better or domain-adapted LLMs should improve them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the near-duplicate passages documented in Appendix A.4 are common in ObliQA, then the true Recall@10 of any retriever is higher than reported; a duplicate-aware evaluation would narrow the gap between MST-R and the baselines.
  • The gaming argument extends beyond RePASs to any NLI-based reference-free metric that scores entailment against retrieved context without checking whether the answer uses the question or adds new information.
  • A practical fix the authors do not explore: require the answer to be judged against the question (e.g., LLM-as-a-Judge) in addition to RePASs, and set a floor on answer length so trivial concatenation cannot dominate.
  • One testable prediction of the paper's analysis is that a retrieval system optimized with RePASs as the objective, rather than Recall@k, would converge to copying behavior and would score poorly on question-conditional judges.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MST-R, a multi-stage retrieval system for regulatory question answering, which combines BM25, few-shot BGE, fine-tuned E5, and a query-to-query (Q2Q) retriever via reciprocal rank fusion, followed by a fine-tuned cross-encoder reranker. On the ObliQA/RIRAG benchmark, the authors report Recall@10 = 0.8746 and MAP@10 = 0.7601, claiming 12.1% and 23% improvements over the BGE baseline. The paper also analyzes the RePASs evaluation metric, showing that a trivial passage-concatenation answer outscores a Llama-3.1-based generator (0.947 vs. 0.41), and proves a claim that such a trivial answer can attain maximum RePASs under idealized assumptions. The paper includes a 15-way ablation of Level 1 retrievers and releases code.

Significance. If the reported gains are robust, the multi-stage tuning recipe (hard-negative mining, hybrid fusion, reranker fine-tuning) is a useful, simple domain-adaptation baseline for regulatory retrieval, and the RePASs gaming analysis is a valuable cautionary result for the RIRAG/RegNLP evaluation suite. The paper's strengths include an external benchmark, a detailed 15-way ablation (Appendix A.3), a formal claim with proof (Appendix A.5), and a public code repository. However, the headline retrieval claims rest on an unspecified train/test split for all adaptation stages and on single-run point estimates without significance testing, and the leaderboard 'top rank' claim is not evidenced. These issues make the central claim currently unverifiable rather than wrong.

major comments (4)
  1. [Section 3.1.1 and Appendix A.2] The paper never states that the BGE few-shot conditioning samples, the E5/MPNet fine-tuning data, and the Q2Q memory are drawn exclusively from the official ObliQA training split. This matters because Section 3.1.1 describes Q2Q as retrieving ground-truth passages for 'previously seen questions... e.g. questions in the training data'; if any test query or its ground-truth passage appears in fine-tuning, conditioning, or Q2Q memory, the Recall@10 and MAP@10 gains in Table 1 (12.1% and 23%) could be inflated by memorization rather than genuine generalization. The authors should specify the exact split used for each adaptation stage and verify that the test set is disjoint from all training and conditioning data, including the passages used in hard-negative mining and reranker negative sampling.
  2. [Table 1 and Abstract/Introduction] The paper reports single-run point estimates for Recall@10 and MAP@10 with no error bars, confidence intervals, or significance tests, and the Abstract's claim of 'top rank on the RegNLP challenge leaderboard' is not supported by any leaderboard snapshot or comparison against other participating systems. Since the differences between MST-R (L1+L2) and RRF(L1) (0.8746 vs. 0.832) and between RRF(L1) and BGE (0.832 vs. 0.7796) may fall within run-to-run variance, the authors should provide multiple runs or bootstrap intervals and should either include the leaderboard evidence or temper the SOTA claim to 'outperforms these two baselines on this split.'
  3. [Appendix A.5, Claim 1] The proof that Passage Concat attains RePASs = 1 relies on Assumption 2, which is stated as requiring 'necessary (maybe unknown) contexts' for non-contradiction across sentences, but the RePASs metric in Equations (2) and (3) computes sentence-level entailment and contradiction as unconditional maxes over sentences. The authors acknowledge that this 'changes the definition of the RePASs metric,' so Claim 1 as written is not a theorem about the original RePASs metric. The empirical result in Table 2 (PC = 0.947 vs. Llama = 0.41) already demonstrates practical gaming, so the paper should either reformulate Claim 1 as an idealized worst-case argument or prove it under the metric's actual definition.
  4. [Appendix A.4 and Section 4] The paper's qualitative analysis shows multiple near-duplicate or significantly overlapping passages that carry the same regulatory obligation as the ground-truth passage (Tables 5-9), yet Recall@10 and MAP@10 treat only the single ground-truth passage as correct. This means the reported 0.8746 and 0.7601 are lower bounds on true retrieval accuracy, and the comparison with baselines may understate the relative advantage of methods that retrieve these equivalent passages. The paper should either define a lenient relevance set for a secondary evaluation or at least explicitly state in the main text that the absolute numbers are conservative lower bounds.
minor comments (5)
  1. [Appendix A.1] The motivation for RRF states 'P (rel|π) = exp(1/(π + β))'; since this quantity exceeds 1 for all π and β, the sentence should read 'exp(−1/(π + β))' or the quantity should be described as an unnormalized weight rather than a probability.
  2. [Section 3.1.1] The Q2Q module is cited as (Lewis et al., 2021), but the referenced PAQ paper is a question-answer dataset and does not appear to describe the query-to-query ANN retrieval with ground-truth-passage return; please clarify the provenance or add an appropriate reference for this design.
  3. [Throughout] The metric name is spelled inconsistently as 'RePaSs', 'RePASs', and 'RsPASs'; please standardize (e.g., 'RePASs' per Equation 1).
  4. [Appendix A.2] '8E + 4training steps' should be read as 8×10^4 training steps, and there are missing spaces in 'b = 400batches'; please fix the formatting.
  5. [Section 6, Limitations] 'per-formant' should be 'performant'; additionally, the limitations paragraph does not address the train/test disjointness issue raised in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: retrieval results are externally benchmarked and the RePASs-gaming analysis is a transparent metric critique, not a masked prediction.

full rationale

The paper's main retrieval claim is evaluated on the external ObliQA/RIRAG benchmark against external baselines (BM25 and BGE from Gokhan et al., 2024), and the multi-stage retriever combines standard components (BM25, BGE, E5, MPNet, RRF, and a cross-encoder reranker). No claimed performance number is derived from a parameter fitted to the target metric or from the paper's own prior results; the only dataset-specific tuning is fine-tuning on ObliQA, which is a normal supervised setup provided the official train/test split is respected. The RePASs analysis is also not circular: Claim 1 explicitly presents Passage Concat as a 'trivial answering model' and proves that it attains the maximum RePASs score under stated self-entailment and non-contradiction assumptions. That result follows from the metric's definition and is used to criticize the metric, not to support a positive system claim. The paper contains no load-bearing self-citation and does not import a uniqueness theorem from its own authors. One methodological caveat should be checked as a correctness risk, not circularity: the paper does not explicitly state that the 'five random samples' used for BGE few-shot conditioning and the fine-tuning data are disjoint from the test split, so a reviewer should verify this against the RIRAG challenge's official partition. Incomplete ground truth (Appendix A.4) is also a benchmark-quality concern, but it does not make any derivation equivalent to its inputs by construction. Overall, the derivation chain is self-contained relative to external benchmarks and baselines.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The pipeline's central claim rests on standard retrieval models plus fitted hyperparameters, and the RePASs analysis introduces two explicit assumptions about the corpus and NLI model. The unspecified train/test split is an additional unstated premise. No new physical entities are introduced.

free parameters (6)
  • beta (RRF regularizer) = 4
    Set to 4 in Appendix A.1 for reciprocal rank fusion; no sensitivity analysis reported.
  • hard-mining loop iterations n = 200
    Iterations of online hard negative mining in Appendix A.2, following standard practice.
  • batches per iteration b = 400
    Training batches per mining iteration in Appendix A.2.
  • contrastive batch size = 8
    Batch size for contrastive fine-tuning in Appendix A.2.
  • BGE few-shot conditioning samples = 5
    Five random ObliQA samples condition BGE-EN-ICL in Section 3.1.1; seed and split are unspecified.
  • RePASs-N context size N = 3
    Table 4 uses N=3 to report a 20% relative improvement; no principled selection criterion is given.
assumptions (5)
  • domain assumption A sentence from a regulatory corpus entails and does not contradict itself (Assumption 1).
    Used in Section 5.2 proof of Claim 1 to set Es=1 and OCs=1 for Passage Concat.
  • domain assumption No two sentences across the regulatory corpus contradict each other given necessary context (Assumption 2).
    Used in Section 5.2 to set Cs=0; Appendix A.6 shows NLI finds contradictions between actual ObliQA sentence pairs, so this assumption is empirically fragile.
  • domain assumption Queries satisfy standard regularity conditions for ANN search in the Q2Q retriever.
    Section 3.1.1 (d) assumes similar queries exist in training data; singleton Q2Q Recall@10 is only 0.35 in Appendix A.3, so the 1.5% fusion gain depends on this assumption.
  • domain assumption NLI model Deberta v3 gives trustworthy entailment and contradiction probabilities.
    Both the RePASs scores and the RePASs-N analysis take NLI probabilities as ground truth; the paper's own appendix shows NLI errors on short phrases.
  • domain assumption ObliQA fine-tuning and few-shot conditioning data come from a training split disjoint from the evaluation set.
    Section 3.1.1 and Appendix A.2 do not state the split; if this fails, the comparison is circular.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MST-R: Multi-Stage Tuning for Retrieval Systems and Metric Evaluation." pith.science (2026). https://pith.science/paper/D7YNAPCN

@misc{pith2026241210313,
  author       = {Pith},
  title        = {Pith review of: MST-R: Multi-Stage Tuning for Retrieval Systems and Metric Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D7YNAPCN}},
  note         = {Machine review of arXiv:2412.10313}
}
read the original abstract

Regulatory documents are rich in nuanced terminology and specialized semantics. FRAG systems: Frozen retrieval-augmented generators utilizing pre-trained (or, frozen) components face consequent challenges with both retriever and answering performance. We present a system that adapts the retriever performance to the target domain using a multi-stage tuning (MST) strategy. Our retrieval approach, called MST-R (a) first fine-tunes encoders used in vector stores using hard negative mining, (b) then uses a hybrid retriever, combining sparse and dense retrievers using reciprocal rank fusion, and then (c) adapts the cross-attention encoder by fine-tuning only the top-k retrieved results. We benchmark the system performance on the dataset released for the RIRAG challenge (as part of the RegNLP workshop at COLING 2025). We achieve significant performance gains obtaining a top rank on the RegNLP challenge leaderboard. We also show that a trivial answering approach games the RePASs metric outscoring all baselines and a pre-trained Llama model. Analyzing this anomaly, we present important takeaways for future research.

Figures

Figures reproduced from arXiv: 2412.10313 by the authors.

Figure 1
Figure 1. Multi Stage Tuning System Architecture Our system, depicted in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Es and Cs for RePASs-N. NLI Deberta v3(He et al., 2021) used with context length N. (Zheng et al., 2023) proposes a reference-free met￾ric requiring the original question and the generated answer. We evaluated Passage Concat and Llama an￾swers on this metric. Specifically, we used the prompt in (Roucher) to ask the LLM to judge whether the answer is relevant, direct, detailed, and addresses all the concerns in the q… view at source ↗
Figure 3
Figure 3. Distribution of entailment scores, Es (using NLI Deberta v3 Large), for T op@10 non-GT retrievals. We show five examples below which range from having almost duplicate wordings between the GT passage and another T op@10 retrieval, to having significant overlap or sharing of phrases/ keywords. • [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 4 canonical work pages

  1. [1]

    Arian Askari, Amin Abolghasemi, Gabriella Pasi, Wessel Kraaij, and Suzan Verberne. 2023. https://arxiv.org/abs/2301.09728 Injecting the bm25 score as text improves bert-based re-rankers . Preprint, arXiv:2301.09728

    work page Pith review arXiv 2023

  2. [2]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  3. [3]

    Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758--759

    2009

  4. [4]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217

    arXiv 2023

  5. [5]

    Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2g: Retrieve, rerank, generate. arXiv preprint arXiv:2207.06300

    arXiv 2022

  6. [6]

    Tuba Gokhan, Kexin Wang, Iryna Gurevych, and Ted Briscoe. 2024. Regnlp in action: Facilitating compliance through automated information retrieval and answer generation. arXiv preprint arXiv:2409.05677

    arXiv 2024

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  8. [8]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    arXiv 2021

Show all 29 references
  1. [9]

    Gautier Izacard and Edouard Grave. 2021. https://arxiv.org/abs/2007.01282 Leveraging passage retrieval with generative models for open domain question answering . Preprint, arXiv:2007.01282

    2021 arXiv

  2. [10]

    Omar Khattab and Matei Zaharia. 2020. https://arxiv.org/abs/2004.12832 Colbert: Efficient and effective passage search via contextualized late interaction over bert . Preprint, arXiv:2004.12832

    2020 arXiv

  3. [11]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163--177

    2022

  4. [12]

    Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. https://arxiv.org/abs/2102.07033 Paq: 65 million probably-asked questions and what you can do with them . Preprint, arXiv:2102.07033

    2021 arXiv

  5. [13]

    Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. https://arxiv.org/abs/2409.15700 Making text embedders few-shot learners . Preprint, arXiv:2409.15700

    2024 arXiv

  6. [14]

    Jing Lu, Keith Hall, Ji Ma, and Jianmo Ni. 2022. https://arxiv.org/abs/2212.10528 Hyrr: Hybrid infused reranking for passage retrieval . Preprint, arXiv:2212.10528

    2022 arXiv

  7. [15]

    Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. https://arxiv.org/abs/1602.06023 Abstractive text summarization using sequence-to-sequence rnns and beyond . Preprint, arXiv:1602.06023

    2016 arXiv

  8. [16]

    Nils Reimers and Iryna Gurevych. 2021. https://arxiv.org/abs/2012.14210 The curse of dense low-dimensional information retrieval for large index sizes . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint ...

    2021 arXiv

  9. [17]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  10. [18]

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. https://arxiv.org/abs/2010.04592 Contrastive learning with hard negative samples . Preprint, arXiv:2010.04592

    2021 arXiv

  11. [19]

    Using llm-as-a-judge for an automated and versatile evaluation

    Aymeric Roucher. Using llm-as-a-judge for an automated and versatile evaluation. https://huggingface.co/learn/cookbook/en/llm_judge#3-improve-the-llm-judge

  12. [20]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815--823

    2015

  13. [21]

    Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. 2016. Training region-based object detectors with online hard example mining. In Conference on Computer Vision and Pattern Recognition ( CVPR )

    2016

  14. [22]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297

    2020 arXiv

  15. [23]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020 a . https://arxiv.org/abs/2004.04228 Asking and answering questions to evaluate the factual consistency of summaries . Preprint, arXiv:2004.04228

    2020 arXiv

  16. [24]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    2022 arXiv

  17. [25]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020 b . https://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers . Preprint, arXiv:2002.10957

    2020 arXiv

  18. [26]

    Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. https://arxiv.org/abs/2305.06311 Automatic evaluation of attribution by large language models . Preprint, arXiv:2305.06311

    2023 arXiv

  19. [27]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  20. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.