Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Learning to Substitute Words with Model-based Score Ranking

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BARTScore can replace human labels for word-substitution training and evaluation.

desk verdict A label-free training recipe for word substitution that is clearly specified and code-released, but the evaluation is mostly circular — the gains over GPT-4o/LLaMA are on the metric that the training loss maximizes. read the letter →

arxiv 2502.05933 v2 pith:LWOQIPT2 submitted 2025-02-09 cs.CL cs.AI

Hongye Liu , Ricardo Henao This is my paper · ORCID
classification cs.CLcs.AI
keywords smartwordsubstitutionBARTScoremodel-basedevaluationmarginrankinglosspreferenceoptimizationBERTfine-tuninglearningwithouthumanlabelslexical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Smart word substitution—deciding which words to replace and what to replace them with—has relied on human-annotated datasets that are costly and inconsistent across annotators. This paper claims that a purely model-based score, BARTScore, can serve as both the training signal and the evaluation metric, eliminating the need for human labels while preserving substitution quality. The authors fine-tune BERT with a loss that couples margin ranking with an average-score term, and report that it outperforms masked language models (BERT, BART) and large language models (GPT-4o, LLaMA) on score alignment and quality across four datasets. If true, this points to a general recipe for refining text without manual annotation: take a trainable model, a frozen evaluator, and a ranking objective that aligns the two.

What carries the argument

The central machinery is BARTScore, a model-based evaluator defined as the conditional log-likelihood of a sentence under a pre-trained BART model. It plays three roles: it defines the reference distribution for a significance statistic of substitution quality, it supplies the target that the margin-ranking loss aligns the model's logits to, and it is itself the evaluation metric (CS and ABR). The proposed objective combines the margin ranking loss $L_{\text{MR}}$ with an average-score loss $L_{\text{AS}}$, so the model is trained to rank candidates like BARTScore and to raise the average BARTScore of its own candidates.

What would settle it

Take a few thousand substitution pairs, have a diverse panel of annotators rank them by preference, and compute the rank correlation between human preference and BARTScore; if the correlation is near zero or negative, the premise that BARTScore can stand in for human quality judgment collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that smart word substitution can be learned and evaluated without any human annotations, using BARTScore as a stand-in for human judgment. The authors define the quality of a candidate substitution as the BARTScore of the sentence with that substitution, $M(\tilde{X}_k) = \sum_{n=1}^N \log \hat{p}_\theta(\tilde{x}_n \mid \tilde{x}_{<n}, X)$, and build a reference (null) distribution for each token position by sampling candidates from the model and counting how often a given candidate outscores the others, yielding a p-value statistic. They then fine-tune BERT so that its logits order the candidates consistently with BARTScore (margin ranking loss) while also improving the average BARTScore of the candidates (average-score loss), giving the combined objective $L_{\text{MR+AS}} = L_{\text{MR}} + \gamma L_{\text{AS}}$. In experiments on SWS, LS07, LS14, and XSum, the trained model's top suggestions show higher cosine similarity with BARTScore and comparable or better average BARTScore ratios than BERT, BART, GPT-4o, and LLaMA, including on substitutions where the model and the single human annotator disagree.

Load-bearing premise

The load-bearing premise is that BARTScore's conditional log-likelihood reliably measures whether a word substitution improves a sentence as humans would judge it, since the same score is used to define the training signal, the evaluation metric, and the significance statistic.

Editorial extensions

If this is right

  • Writing-assistance systems can be trained for word substitution without human-annotated data, since the SWS training set used here is generated from a thesaurus rather than human judgments.
  • The paper's results imply that substitution disagreements between model and annotator are not automatically errors: many of those disagreements pass its model-based significance test, consistent with the subjectivity of word choice.
  • Ranking and score objectives are complementary: margin ranking alone improves alignment but lowers average quality, while the combined loss achieves the best alignment and the best quality trade-off across all ablations.
  • The pipeline is scorer-agnostic: replacing BARTScore with GPTScore preserves the relative advantage of MR+AS over the baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's training and evaluation both use BARTScore, its reported gains measure alignment with BARTScore; a larger human-preference study is the natural next test of whether that alignment transfers to human judgment.
  • The per-position p-value statistic could serve as an unsupervised confidence signal in a writing assistant, flagging substitutions that are statistically strong or near-ties for a reviewer.
  • If the premise holds, the same score-as-teacher recipe should transfer to other sentence edits, such as deleting, reordering, or rewriting phrases, turning general text revision into a label-free ranking problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MR+AS, a method for smart word substitution that fine-tunes BERT without human labels by using BARTScore as a reward. The training objective combines a margin ranking loss (Eq. 3) that aligns token logits with the order of BARTScore values and an average-score loss (Eq. 4) that maximizes a softmax-weighted sum of BARTScore. The authors compare MR+AS against MLMs (BERT, BART), a rule-based thesaurus method, and LLMs (GPT-4o, LLaMA) on four datasets, reporting that MR+AS achieves the best CS and ABR metrics, both computed from BARTScore. They also report a model-based p-value statistic (Eq. 2) and a small human study (25+25 cases) as external checks.

Significance. If the claims were established, the approach would be a useful contribution: it removes the need for human annotations in substitution tasks, gives a general recipe for aligning token predictions with a model-based quality score, and would show that a comparatively small BERT model can outperform much larger LLMs on a writing-assistance task. The manuscript is clearly organized, the code is released, and the ablation over MR, DPO variants, MR+BS, and MR+AS is informative. However, the main comparative claim is currently supported only by metrics that are essentially the training objective, and the external human validation is too thin to break the circularity. With additional independent evaluation, the result could be significant, but the present evidence is insufficient.

major comments (4)
  1. [Section 4, Evaluation Metrics and Tables 2-3] The headline comparison uses CS and ABR, both defined via the same BARTScore that appears in the training losses. Eq. (3) directly encourages logits to be ordered by BARTScore, and Eq. (4) directly maximizes a softmax-weighted average of BARTScore. Consequently, MR+AS's near-perfect CS (0.98-0.99) and its ABR advantage over baselines are expected by construction; they show that the model fits its reward, not that it produces better substitutions. The Appendix GPTScore tables do not resolve this, because the paper reports Spearman correlations of 0.929 and 0.921 between BARTScore and the two GPTScore variants. An independent evaluation—either a larger human study that pits MR+AS against GPT-4o/LLaMA head-to-head, or a held-out reward model not used in training—is needed before the 'outperforms' claim can be accepted.
  2. [Section 3.1 and Limitations] The validity of BARTScore as a proxy for human substitution quality is the load-bearing assumption of the whole method, and the paper's own Limitations section concedes that 'we cannot eliminate the possibility that BARTScore may not always accurately quantify the quality of token substitutions.' The only direct check is the human study in Section 4. It uses only 25+25 cases, reports replacement-agreement Cohen's kappa of 0.13±0.21, has 38.8% ties, and in the disagreement subset annotators sided with BARTScore only 67% of the time. This is far too thin to establish BARTScore as a valid reward, and it never compares MR+AS against GPT-4o or LLaMA on human judgment. Without a stronger external validation, the central claim that the method improves sentence quality is not supported.
  3. [Eq. (5)] The loss L_BS = max(0, (M(X) - M(\tilde X_1)) f(s_1)) contains an undefined function f. This makes the MR+BS variant (Eq. 7) unreproducible, since no definition or reference is given for f in the text, equations, or appendix. Either define f and specify its role, remove MR+BS from the ablation, or report the specific choice used to produce the MR+BS results in Table 2.
  4. [Section 3.2 and Appendix A] The DPO* and σDPO* losses are presented as derived from DPO, but the derivation replaces log p_θ(\tilde X_k) with the token logit s_k and drops the softmax denominator, yielding losses that are no longer DPO under either the Plackett-Luce or Bradley-Terry model. This substitution of token-level logits for sentence-level log-likelihoods is not justified in the paper. Since these variants underperform MR+AS, this does not overturn the main result, but as proposed the derivation is not valid and the name 'DPO*' is misleading.
minor comments (5)
  1. [Eq. (3)] The sentence defining the sorted order uses 'for i, j = 1, ..., K' but i is never used; the condition should be over pairs k, j.
  2. [Section 3.1, Eq. (2)] The quantity in Eq. (2) is described as a p-value and used in a hypothesis test, but it is an empirical proportion over the model's own candidate set. Calling it a p-value and 'rejecting the hypothesis' for p < α is statistically imprecise and could mislead readers about the meaning of Table 4.
  3. [Abstract and Section 1] The abstract and contributions state that the method 'defin[es] a distribution for each word substitution,' but the paper actually defines a rank statistic (Eq. 2) and a set of loss functions; there is no distribution over substitutions. Please rephrase to match the content.
  4. [Table 1] The second row of Table 1 is misaligned ('Not change14138'), making the counts hard to read; also, the row/column totals should be verified, since 14138 + 969 = 15107 appears to be the row total for 'Not change' tokens.
  5. [Section 4, Human Study] The human study reports only average agreement percentages without confidence intervals or per-item breakdowns for the 79% and 67% statistics; given the small sample, this makes the strength of the agreement hard to assess.

Circularity Check

3 steps flagged · score 8.0 of 10

The headline 'outperforms' claim is circular: the CS, ABR, and Eq. (2) statistics are BARTScore-based, while BARTScore is also the training reward; the GPTScore check is nearly collinear and the only human study is tiny.

  1. fitted input called prediction [Section 3.2, Eqs. (4)-(6); Section 4, 'Evaluation Metrics'; Table 3]
    "Conceptually, LAS in (4) seeks to maximize the (weighted) average of model-based scores from K predictions ... LMR+AS = LMR + γLAS , (6) ... We use the cosine similarity (CS) to measure the correlation between model predictions and model-based scores, i.e., BARTScores. ... calculate the average of BARTScore ratios as ABR = 1/K PK k=1 M ( ˜Xk)/M (X)."

    The same BARTScore M(·) is both the training reward and the headline evaluation quantity: LAS directly maximizes a softmax-weighted sum of BARTScore, LMR sorts logits to agree with BARTScore order, and CS/ABR measure exactly this alignment and average BARTScore. Thus Table 3's claim that MR+AS outperforms MLMs and LLMs is, on these metrics, a check of how well the model fits its own reward rather than an independent measure of substitution quality.

  2. self definitional [Section 3.2 (paragraph after Eq. 2); Section 4, 'Model-based Statistic Benchmark'; Table 4]
    "in principle, we seek to train a model such that for each token xn, the resulting p ˜Xk from (2) for a given substitute candidate wk producing ˜Xk is as small as possible. ... Table 4 ... results demonstrates that our model (MR+AS) not only produces the largest proportions, but it does so irrespective of the group (CA, CD or OMC). This suggests that substitutions made by our model are of good quality (using BARTScore as proxy for quality)."

    The statistic in Eq. (2) is defined via BARTScore comparisons among candidates, and the text explicitly states the model is trained to make this p-value as small as possible. Table 4 then evaluates all models by this same statistic, so MR+AS's high proportions are the trained objective itself, not evidence of quality independent of BARTScore.

1 more flagged steps
  1. other [Section 4, 'Model-based Statistic Benchmark' and Appendix Tables 9-10 (GPTScore check)]
    "To avoid problems associated with overfitting, thus making the evaluation more reliable, in Appendix Table 9 and Table 10 we extended the evaluation result by using the GPTScore ... The results show that our MR+AS still outperforms other baseline models. ... We also calculated the Spearman correlation between BARTScore and GPTScore (GPT2-medium and OPT-350M), which yielded results of 0.929 and 0.921, respectively."

    The GPTScore tables are presented as a more reliable, overfitting-avoiding check, but the paper's own reported Spearman correlations (0.929 and 0.921) make GPTScore nearly collinear with BARTScore, the training reward. The 'independent' metric therefore largely re-measures the same score and does not break the circularity of evaluating on the training objective.

full rationale

The paper's central comparative claim — that MR+AS outperforms BERT, BART, GPT-4o, and LLaMA — rests on Table 3, whose CS and ABR metrics are the same BARTScore quantities appearing in the training losses (Eqs. 3-6). Training directly increases softmax-weighted BARTScore (LAS) and logit-score alignment (LMR); at test time, CS measures that alignment and ABR measures average BARTScore, so the reported superiority is expected by construction. The Eq. (2) statistic used in Table 4 is likewise the explicit training target, and the GPTScore appendix is not independent because the paper reports Spearman correlations of 0.929 and 0.921 between BARTScore and GPTScore. The only external evidence, the human study, is small (50 cases; replacement-agreement Cohen's kappa 0.13±0.21; 38.8% ties) and never compares MR+AS directly with GPT-4o or LLaMA; the paper's own Limitations section concedes 'we cannot eliminate the possibility that BARTScore may not always accurately quantify the quality of token substitutions.' There is genuine method content in the losses, statistic, and DPO variants, but the headline empirical claim reduces to a fit to BARTScore, so the circularity score is 8.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on treating BARTScore as ground truth for substitution quality. Since the same score is used to define training losses and evaluation metrics, the improvement is partly self-confirming; external GPTScore checks are highly correlated and the human study is small.

free parameters (6)
  • lambda = 0.5
    Margin scaling in the ranking loss (Eq. 3), selected by cross-validation; controls the required logit gap between ranked candidates.
  • gamma = 1
    Weight balancing the margin ranking loss and the score-improving loss in Eqs. (6)-(7), selected by cross-validation.
  • K = 5
    Number of substitute candidates sampled per token for training and test candidate pools; chosen by the authors.
  • K_s = 1000
    Number of candidates used to build the reference score distribution in Eq. (2); a computational accuracy trade-off.
  • alpha = 0.01
    Significance threshold for declaring a substitute statistically superior in the model-based statistic.
  • tokens_per_sentence = 5
    Randomly selected tokens per sentence used for fine-tuning and evaluation; affects the training signal and change rates.
assumptions (4)
  • domain assumption BARTScore is a valid proxy for human judgment of word-substitution quality.
    The training losses and primary evaluation metrics define quality through BARTScore (Eq. 1). If BARTScore diverges from human judgment, the reported improvements do not correspond to human-perceived quality.
  • domain assumption A reference distribution of candidates sampled from the model itself is a meaningful null distribution for substitution quality.
    Eq. (2) compares a candidate against substitutes sampled from the same model under evaluation, which can bias the statistic toward the model's own preferences.
  • ad hoc to paper Token logits can replace sentence-level log-probabilities in DPO-style losses.
    Eqs. (9)-(10) substitute log p_theta(tilde X_k) with the token logit s_k without derivation; this affects the DPO variants but not the final MR+AS model.
  • domain assumption BARTScore values are comparable across sentences and can be used as constants in a loss.
    All losses in Section 3.2 treat M as a fixed, meaningful score and take expectations over modified sentences; no calibration is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Substitute Words with Model-based Score Ranking." pith.science (2026). https://pith.science/paper/LWOQIPT2

@misc{pith2026250205933,
  author       = {Pith},
  title        = {Pith review of: Learning to Substitute Words with Model-based Score Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWOQIPT2}},
  note         = {Machine review of arXiv:2502.05933}
}
read the original abstract

Smart word substitution aims to enhance sentence quality by improving word choices; however current benchmarks rely on human-labeled data. Since word choices are inherently subjective, ground-truth word substitutions generated by a small group of annotators are often incomplete and likely not generalizable. To circumvent this issue, we instead employ a model-based score (BARTScore) to quantify sentence quality, thus forgoing the need for human annotations. Specifically, we use this score to define a distribution for each word substitution, allowing one to test whether a substitution is statistically superior relative to others. In addition, we propose a loss function that directly optimizes the alignment between model predictions and sentence scores, while also enhancing the overall quality score of a substitution. Crucially, model learning no longer requires human labels, thus avoiding the cost of annotation while maintaining the quality of the text modified with substitutions. Experimental results show that the proposed approach outperforms both masked language models (BERT, BART) and large language models (GPT-4, LLaMA). The source code is available at https://github.com/Hyfred/Substitute-Words-with-Ranking.

Figures

Figures reproduced from arXiv: 2502.05933 by the authors.

Figure 1
Figure 1. CS distribution results for SWS test data. [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 3
Figure 3. p-value distributions [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. p-value distributions [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: p-value distributions [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: p-value distributions [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Results for LLaMA and GPT-4o with and without a prompt encouraging ranking. Order Agreement Annotator1 Annotator2 Annotator3 Annotator4 Annotator5 Annotator1 - 0.28 0.51 0.75 0.44 Annotator2 - - 0.12 0.19 0.2 Annotator3 - - - 0.41 0.44 Annotator4 - - - - 0.68 Annotator…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 8 canonical work pages

  1. [1]

    I-Chun Chern, Zhiruo Wang, Sanjan Das, Bhavuk Sharma, Pengfei Liu, Graham Neubig, et al. 2023. Improving factuality of abstractive summarization via contrastive reward learning. arXiv preprint arXiv:2307.04507

    arXiv 2023

  2. [2]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  3. [3]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  4. [4]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    arXiv 2023

  5. [5]

    Zhiwei He, Xing Wang, Wenxiang Jiao, Zhuosheng Zhang, Rui Wang, Shuming Shi, and Zhaopeng Tu. 2024. Improving machine translation with human feedback: An exploration of quality estimation as a reward model. arXiv preprint arXiv:2401.12873

    arXiv 2024

  6. [6]

    all-words

    Gerhard Kremer, Katrin Erk, Sebastian Pad \'o , and Stefan Thater. 2014. What substitutes tell us-analysis of an “all-words” lexical substitution corpus. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 540--549

    work page 2014

  7. [7]

    M Lewis. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

    arXiv 2019

  8. [8]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023 a . Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1--35

    work page 2023

Show all 35 references
  1. [9]

    Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203.16804

    2022 arXiv

  2. [10]

    Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023 b . On learning to summarize with large language models as references. arXiv preprint arXiv:2305.14239

    2023 arXiv

  3. [11]

    Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F Wong, and Dacheng Tao. 2022. Toward human-like evaluation for natural language generation with error analysis. arXiv preprint arXiv:2212.10179

    2022 arXiv

  4. [12]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models. arXiv preprint arXiv:2303.13809

    2023 arXiv

  5. [13]

    Diana McCarthy and Roberto Navigli. 2007. Semeval-2007 task 10: English lexical substitution task. In Proceedings of the fourth international workshop on semantic evaluations (SemEval-2007), pages 48--53

    2007

  6. [14]

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. 2024. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  7. [15]

    George Michalopoulos, Ian McKillop, Alexander Wong, and Helen Chen. 2021. Lexsubcon: Integrating knowledge from lexical resources into contextual embeddings for lexical substitution. arXiv preprint arXiv:2107.05132

    2021 arXiv

  8. [16]

    Jason H Moore. 1999. Bootstrapping, permutation testing and the method of surrogate data. Physics in Medicine & Biology, 44(6):L11

    1999

  9. [17]

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745

    2018 arXiv

  10. [18]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  11. [19]

    Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Associa...

    2015

  12. [20]

    Jipeng Qiang, Kang Liu, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. Parals: lexical substitution via pretrained paraphraser. arXiv preprint arXiv:2305.08146

    2023 arXiv

  13. [21]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  14. [22]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

    2020 arXiv

  16. [24]

    Alexander J Smola. 2000. Advances in large margin classifiers. MIT press

    2000

  17. [25]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  18. [26]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  19. [27]

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944

    2023 arXiv

  20. [28]

    Chenshuo Wang, Shaoguang Mao, Tao Ge, Wenshan Wu, Xun Wang, Yan Xia, Jonathan Tien, and Dongyan Zhao. 2023. Smart word suggestions for writing assistance. arXiv preprint arXiv:2305.09975

    2023 arXiv

  21. [29]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263--27277

    2021

  22. [30]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  23. [31]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  24. [32]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  25. [33]

    Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. Bert-based lexical substitution. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 3368--3373

    2019

  26. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.