DIVE: Diversified Iterative Self-Improvement

Pengfei Liu; Yiwei Qin; Yixiu Liu

REVIEW 4 major objections 6 minor 44 references

DIVE: Diversified Iterative Self-Improvement

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By expanding the candidate pool to include every previous iteration's generations and then selecting diverse, high-quality preference pairs, DIVE raises the output diversity of iteratively self-improving language models by 10–45% on MATH…

desk verdict A plausible and useful anti-collapse recipe for iterative preference learning, with real empirical support but headline stats that need variance estimates and a spelled-out baseline. read the letter →

arxiv 2501.00747 v1 pith:UWXCVOLR submitted 2025-01-01 cs.CL

Yiwei Qin , Yixiu Liu , Pengfei Liu This is my paper · ORCID

classification cs.CL

keywords iterativeself-improvementoutputdiversitypreferencelearningdirectoptimizationmathematicalreasoningdataselectionsamplepoolexpansionmodelcollapse

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Iterative self-improvement (ISI) lets a language model train on its own outputs round after round, but the model tends to converge on a narrow set of response styles, which is especially costly in math reasoning where many valid solution paths exist. DIVE adds two steps to that loop: it expands the pool of candidate responses (sampling more answers per question and reusing every previous iteration's generations) and then selects preference pairs that are both high-quality and diverse, using embedding-based outlier filtering followed by greedy diversity maximization. On GSM8k and MATH with Mistral-7B, the paper reports a 10–45% relative increase in four output-diversity metrics for both correct and incorrect responses, while top-1 and top-50 accuracy stay essentially level with vanilla ISI. The paper argues that the diversity of the selected training examples, not the raw size of the pool, is what determines whether the trained model stays diverse. If it holds, DIVE offers a plug-in remedy for diversity collapse in preference-based self-improvement that costs nothing in accuracy.

What carries the argument

The load-bearing mechanism is the Data Selection pipeline acting on an expanded candidate pool. The pool is expanded two ways: sampling $K=10$ or $50$ responses per question instead of fewer, and accumulating generations across iterations into a global pool $\mathcal{D}^t_{\mathrm{pool}} = \bigcup_{i=1}^{t} \mathcal{D}^i_{\mathrm{pool}}$. Selection then runs Isolation Forest (an outlier-detection method) on Sentence-BERT embedding distances to discard low-quality outliers, and a greedy algorithm that iteratively adds the remaining response whose inclusion maximizes the embedding-based diversity of the chosen set, building preference pairs from the correct and incorrect halves of that curated pool. The induced preference pairs feed the DPO objective (with an additional negative-log-likelihood term on chosen sequences), so diversity is injected at the training-data level rather than through decoding-time tricks.

What would settle it

Take the DIVE-selected preference pairs and score their reasoning-path diversity with a surface-independent measure—for example, structural equation-chain similarity on GSM8k solutions, or a human annotation of solution strategy—and compare with vanilla ISI selection at matched pool size; if the 10–45% diversity advantage shrinks or vanishes under that measure, the embedding signal was tracking style rather than reasoning diversity. A cheaper check: measure how much of DIVE's reported diversity gain survives when the evaluation metric is replaced by distinct-answers or distinct-equation-chains, which do not overlap with the Sentence-BERT selection signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that diversity collapse in iterative preference learning can be reversed by controlling what goes into the preference pairs, without giving up accuracy. Specifically, DIVE combines Sample Pool Expansion—sampling $K$ responses per question and pooling generations from all previous iterations into $\mathcal{D}^t_{\mathrm{pool}} = \bigcup_{i=1}^{t} \mathcal{D}^i_{\mathrm{pool}}$—with Data Selection that first removes semantic outliers via Isolation Forest on Sentence-BERT embeddings (vector representations of response meaning) and then greedily picks responses that maximize the embedding-distance diversity of the selected set. On the MATH and GSM8k datasets, with Mistral-7B as the base model and up to six iterations of direct preference optimization (DPO) plus a negative-log-likelihood term, DIVE yields a 10–45% relative improvement in four diversity metrics (distinct n-grams and the complement of Sentence-BERT cosine similarity) for both correct and incorrect responses, while @1 and @50 accuracy remain comparable to vanilla ISI. Ablations show that pooling alone does not consistently help, that selection is the component that reliably raises diversity, and that greedy selection without the Isolation Forest filter can hurt accuracy. The authors further show that the diversity gains generalize to seven additional metrics, including equation-chain diversity and distinct-answer counts, and that DIVE helps across all question difficulty levels, with the largest relative diversity loss in vanilla ISI occurring on the easiest questions.

Load-bearing premise

The assumption the whole result rests on is that Sentence-BERT embedding distance measures diversity of mathematical reasoning paths; if it mostly captures wording style, DIVE's selection inflates the very metrics used to evaluate it, and the reported gains would not reflect genuinely different solution strategies.

Editorial extensions

If this is right

Applying DIVE's two components—pool expansion and embedding-based selection—to any DPO-style iterative self-improvement loop should slow or stop the per-iteration decline in output diversity on mathematical reasoning tasks.
Diverse preference data can be chosen without a reward model or extra human labels: correctness is judged by the gold final answer and diversity by embedding distance, so the method is fully self-supervised once gold answers exist.
Larger sampling (K=50) combined with global pooling and selection gives the best diversity, whereas larger sampling without selection reduces diversity relative to K=10.
Because self-improvement on these benchmarks mostly re-ranks the sample space (stable @50 accuracy), DIVE preserves the breadth of that sample space rather than changing the model's ceiling.
The diversity gains are not confined to the metrics used during training: they transfer to distinct-equation-chain and distinct-answer metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If embedding distance truly captures solution-path diversity, the same selection recipe should transfer to other self-improvement tasks with multiple valid answers—code generation, open-ended instruction following, and multi-step planning—where the style-versus-strategy trade-off would need to be re-checked.
DIVE's habit of keeping every previous iteration's generations in the pool suggests a testable scheduling principle: an ISI loop should maintain a reservoir of older, more diverse samples and decide by selection, not recency.
One could adapt DIVE to online preference learning by using the greedy selector as a diversity-aware buffer sampler, turning the per-iteration selection cost into a streaming update rather than a full re-scan of the pool.
A harder test the paper does not run: whether the 10–45% diversity gains translate into accuracy gains under self-consistency decoding (majority vote over diverse paths); if they do, DIVE would improve not just sample-space breadth but downstream task performance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: DIVE is a useful, honest empirical paper on countering diversity collapse in iterative self-improvement. The two-part recipe (pool expansion plus isolation-forest-filtered greedy selection) is not present in the cited prior work, and the paper mostly delivers what it claims. I am fairly convinced the effect is real, but the headline 10-45% numbers are not statistically anchored as reported, so the paper needs revision rather than rejection.

What is actually new: the combination of a global accumulated sample pool with outlier-filtered greedy diversity selection for DPO-style iterative training is a reasonable and underexplored design. The Q1 analysis comparing global pooling against merely sampling more per question is the most useful part: it shows you get more diversity per unit compute by reusing old data, which is a practical insight. The difficulty-level analysis in Q2 is also a nice addition, and the paper is clearly written. They released code, which is credit.

The soft spots are real but not fatal. Tables 1 and 2 report the best iteration out of six for each method, with no error bars or repeated runs. That makes small differences hard to interpret. The iteration plots in Figure 2 (and Appendix Figure 6 for alternative metrics) show the trend more honestly, and those plots do support the qualitative claim. Still, the paper ought to give fixed-iteration comparisons and variance estimates. The vanilla baseline's preference-pair construction is never specified — Section 2.1 just says "we select P responses" without saying how. Given the counter-intuitive result that vanilla diversity drops when K goes from 10 to 50, the selection rule matters a lot. The circularity concern (selecting on the same Sentence-BERT/distinct-n-gram signals used for evaluation) is partially mitigated by the alternative metrics in Section 4.3, but those are also shown without variance, so the mitigation is incomplete. The small accuracy drops on MATH (e.g., @50 from 0.606 to 0.586) are honestly reported, though "maintaining performance" is a bit generous for a 3% drop.

Who this is for: people working on post-training diversity, DPO variants, and model collapse. It deserves a serious referee; I would send it out and ask for fixed-iteration tables, repeated seeds, and an explicit description of the vanilla baseline's selection mechanism. With those, the core claim would be credible enough for publication.

Referee Report

4 major / 6 minor

Summary. The paper proposes DIVE (Diversified Iterative Self-Improvement), a framework that modifies iterative Direct Preference Optimization (DPO) training to counteract the loss of output diversity in self-improvement. Two components are introduced: Sample Pool Expansion, which increases the number of sampled responses per question and reuses responses from all previous iterations, and Data Selection, which filters responses with an Isolation Forest on Sentence-BERT embeddings and then greedily selects preference pairs to maximize diversity. Experiments on GSM8K and MATH with Mistral-7B compare Vanilla ISI, Global, Selection, and Global+Selection over six iterations. The paper reports that Global+Selection yields a 10–45% relative increase in diversity metrics (Distinct N-grams and Sentence-BERT cosine similarity) while keeping accuracy roughly unchanged. Ablations and additional diversity metrics (NV-Embed, Stella, equation chains, distinct answers) are provided, together with an analysis of diversity across question difficulty levels.

Significance. If the central empirical claim holds, DIVE is a useful and simple contribution to an important problem: diversity collapse in iterative preference learning for reasoning tasks. The paper is clearly written, ships code, includes ablations for both proposed components, and partially addresses the metric-circularity concern by evaluating with alternative embedding and task-specific diversity metrics. The difficulty-level analysis (Section 4.2) is a nice addition. However, the load-bearing quantitative claim is not yet statistically anchored: the headline tables report the best of six iterations per method, there are no error bars or repeated-seed runs, and the vanilla baseline's selection mechanism is left unspecified. These issues are fixable and do not invalidate the approach, but they must be resolved before the 10–45% claim can be accepted.

major comments (4)

[§3.2, Tables 1–2] The headline diversity and accuracy results are taken from the best-performing of six iterations per method, with no standard deviations, confidence intervals, or repeated-seed runs. Because the pipeline is stochastic (nucleus sampling, DPO training, greedy selection with a random start), this protocol can systematically overstate the advantage of DIVE: a lucky iteration is reported as representative. This is load-bearing for the central '10–45% relative increase' claim. Please report per-iteration results with variance across seeds (or at least the full per-iteration distribution for every method) and assess the statistical significance of the differences.
[§2.1 and §3.1.3] The vanilla baseline is under-specified. Section 2.1 Step 2 states only that 'we select P responses' from the correct and incorrect pools, and Section 3.1.3 gives K and P but not the selection rule used by Vanilla. If vanilla selects, for example, the first P responses in generation order or by a fixed score ordering, then the comparison against Global+Selection is not a controlled test of the proposed selection mechanism. This also makes the counter-intuitive result that vanilla diversity decreases when K grows from 10 to 50 (Table 1) impossible to interpret. The authors should specify the vanilla pair-construction rule and ensure that the only differences between conditions are the proposed components.
[§2.2.2 and §3.1.2] The data-selection pipeline uses the same diversity signals that are the primary evaluation metrics: the greedy selection maximizes a diversity score on the candidate list, and the Isolation Forest filter uses Sentence-BERT embedding distances, while Section 3.1.2 evaluates diversity with Sentence-BERT similarity and distinct N-grams. Consequently, part of the reported gain is selected for by construction. The alternative metrics in Section 4.3 (NV-Embed, Stella, equation chains) are a useful partial response, but they are shown only for the best-accuracy iteration and without variance estimates, so the circularity concern is not fully resolved. Please either measure diversity with held-out metrics not used in selection, or report all alternative metrics with the same statistical rigor as the headline numbers.
[Tables 1–2 and §3.2] The quality-preservation claim is not statistically anchored. Several accuracy comparisons are close but in the wrong direction (e.g., MATH Sample 50 @50: 0.586 for Global+Selection vs 0.606 for Vanilla; GSM8k Sample 50 @50: 0.972 vs 0.975). Without error bars or a repeated-seed analysis, 'maintaining performance quality' cannot be distinguished from a small but real degradation, and the reader cannot judge whether the trade-off is acceptable. Report confidence intervals for @1 and @50 and discuss the accuracy/diversity trade-off explicitly.

minor comments (6)

[§2.2.2] The greedy selection step does not specify which diversity measure is optimized in Step 2; please state whether it is the same Distinct-N/Sentence-BERT metric used in evaluation and how any hyperparameters (e.g., the n-gram range) are chosen.
[§3.1.3] The text says models are trained 'on all the preference pairs constructed so far,' which appears to conflict with the per-iteration D_t_pref definition in Section 2.1; please clarify whether the DPO training set accumulates across iterations and whether the same holds for the vanilla baseline.
[§3.1.3, footnote 1] The number of preference pairs per iteration varies between 10k and 30k depending on the setting, and questions without both a correct and an incorrect response are skipped; please report pair counts per method and iteration, since differences in dataset size alone could affect both accuracy and diversity.
[§2.2.2] The Isolation Forest contamination threshold (or equivalent outlier-rate setting) is not reported; this parameter directly controls how aggressively the response pool is pruned and should be stated for reproducibility.
[Figure 5 caption] The caption says 'Only the results from the iteration with the highest accuracy are shown' while Figure 6 reports all iterations; please use a consistent protocol and explain why the choice of iteration differs from that in Tables 1–2.
[§1] The phrase 'the first study focused on this problem' is a strong claim; please either qualify it with the specific setting (diversity-preserving data selection in ISI for math reasoning) or cite concurrent work.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: headline SentBERT and Distinct-N diversity gains are selected for by the same metrics used in DIVE's data selection; alternative metrics in §4.3 provide independent support, keeping this at partial rather than total circularity.

fitted input called prediction [Section 2.2.2 (Data Selection); Section 3.1.2 (Evaluation Metrics); Section 4.3 (Alternative Metrics)]
"Once the response pool is filtered, we apply the greedy selection method to maximize diversity among the remaining high-quality responses. ... To evaluate the diversity of the generated responses, we use the following metrics, in line with Kirk et al. (2023): Distinct N-grams ... Sentence-BERT Embedding Cosine Similarity ... Notably, while our method uses computationally efficient metrics (SentBERT and Distinct-N) during training, the improvements generalize to more sophisticated metrics, confirming the robustness of our approach."

DIVE's Data Selection step selects preference pairs by maximizing diversity as measured with Sentence-BERT and Distinct-N, and Section 4.3 explicitly confirms these are the metrics used during training. The headline Tables 1–2 then report improvements on exactly those two metrics (Distinct-N and SentBERT) as evidence that DIVE increases output diversity. For those metrics, the reported gain is at least partly a direct consequence of the selection objective—the evaluation signal is the same signal used to construct the training data—so calling the improvement a 'prediction' is partially circular.

full rationale

The paper's central claim is that DIVE raises output diversity by 10–45% while preserving accuracy. The most circular element is the identity between the training-time selection signal and the headline evaluation metrics: the greedy selection and Isolation-Forest filter are driven by Sentence-BERT embeddings, and the paper states that SentBERT and Distinct-N are used during training, while Tables 1–2 report gains on SentBERT and Distinct-N. That portion of the claim is selected for by construction. However, the paper does not rest solely on those metrics: Section 4.3 and Figures 5–6 show consistent improvements on NV-Embed, Stella, equation-chain diversity, and distinct-answer diversity, metrics that were not part of the selection procedure. Those results give the core claim independent empirical content, so this is not a case where the derivation reduces entirely to its input. The self-citation to Wu et al. (2024)—co-authored by a co-author of this paper—is used only to motivate the diversity-loss problem and to interpret the stable @50 accuracy, not to justify the claimed improvements, so it is not load-bearing. The best-iteration reporting and absence of error bars noted in the manuscript are validity concerns about statistical anchoring, but they are not circularity under the definitions used here; they do not change the partial-circularity verdict. Overall score 6: one headline 'prediction' (SentBERT/Distinct-N diversity gains) reduces substantially to the selection objective, while the alternative-metric results prevent a score of 8 or 10.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claims depend on hand-set hyperparameters and on assumptions that answer matching is a reliable correctness signal and that embedding distances measure reasoning diversity. No quantities are fitted to data in a derivation sense, but the unstated Isolation Forest threshold and random greedy seed directly control the quality-diversity trade-off and should be reported.

free parameters (5)

Isolation Forest contamination threshold = unspecified
This threshold controls how many responses are discarded as outliers, effectively setting the quality-diversity trade-off in DIVE; it is not reported or swept in the paper.
Greedy selection random seed = unspecified
The greedy algorithm begins with a randomly chosen response, so the selected subset and its diversity score depend on this random seed, which is not provided.
alpha (DPO/NLL balance) = 0.5
The weight between the DPO preference loss and the NLL stability loss is chosen by hand and not ablated in the main experiments.
beta (DPO coefficient) = 0.4
The DPO scaling factor is a hand-set hyperparameter that affects how strongly preferences are enforced.
K (samples per question) = 10 or 50
The reported diversity gains are partly driven by the larger sampling budget K=50, so the claim is conditional on these specific budgets.

assumptions (3)

domain assumption Correctness of a generated solution is determined solely by exact equality of its final answer with the gold answer
This rule builds the correct and incorrect pools in Section 2.1, step 2. Solutions with correct final answers but flawed reasoning are treated as preferred responses, which could distort preference pairs and the diversity-quality balance.
domain assumption Sentence-BERT embedding distances and distinct n-gram counts are valid proxies for diversity of mathematical reasoning paths
These signals drive both the Isolation Forest filter and the greedy selection (Section 2.2.2) and also serve as the main evaluation metrics (Section 3.1.2). If they measure style rather than strategy, the reported diversity gains are inflated.
domain assumption Isolation Forest outlier removal preserves diverse high-quality solutions and removes only low-quality ones
Section 2.2.2 uses the filter to 'maintain response quality'. If outliers in embedding space include rare but valid solution strategies, the method would sacrifice the diversity it aims to increase.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DIVE: Diversified Iterative Self-Improvement." pith.science (2026). https://pith.science/paper/UWXCVOLR

@misc{pith2026250100747,
  author       = {Pith},
  title        = {Pith review of: DIVE: Diversified Iterative Self-Improvement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWXCVOLR}},
  note         = {Machine review of arXiv:2501.00747}
}

read the original abstract

Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at https://github.com/qinyiwei/DIVE.

Figures

Figures reproduced from arXiv: 2501.00747 by the authors.

**Figure 2.** Evolution of diversity metrics and model performance across iterations (M0-M6) for both GSM8k and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Comparison of different sampling strategies for GSM8k dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Diversity trends across different difficulty levels (Level 1-5) for positive and negative examples. The plots [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Results of different diversity metrics for both the GSM8k and MATH datasets. Only the results from the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Results of all iterations across different diversity metrics for both the GSM8k and MATH datasets. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

2024
[4]

Dhruv Batra, Payman Yadollahpour, Abner Guzman-Rivera, and Gregory Shakhnarovich. 2012. Diverse m-best solutions in markov random fields. In Computer Vision--ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 1--16. Springer

work page 2012
[5]

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

2017
[6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021
[7]

Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. 2024. https://arxiv.org/abs/2402.07043 A tale of tails: Model collapse as a change of scaling laws . Preprint, arXiv:2402.07043

arXiv 2024
[8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

Show all 44 references

[9]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

2024 arXiv
[10]

Jonathan St BT Evans. 2010. Intuition and reasoning: A dual-process perspective. Psychological Inquiry, 21(4):313--326

2010
[11]

Roberts, Diyi Yang, David L

Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. 2024. https://arxiv.org/abs/2404.01413 Is model collap...

2024 arXiv
[12]

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

2023 arXiv
[13]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

2021 arXiv
[14]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[15]

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452

2023 arXiv
[16]

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055

2015 arXiv
[17]

Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562

2016 arXiv
[18]

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. Making large language models better reasoners with step-aware verifier. arXiv preprint arXiv:2206.02336

2022 arXiv
[19]

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. https://doi.org/10.1109/ICDM.2008.17 Isolation forest . In 2008 Eighth IEEE International Conference on Data Mining, pages 413--422

2008 doi
[20]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

2024
[21]

Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830

2024 arXiv
[22]

Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831

2024 arXiv
[23]

Niklas Muennighoff, Nouamane Tazi, Lo \" c Magne, and Nils Reimers. 2022. https://doi.org/10.48550/ARXIV.2210.07316 Mteb: Massive text embedding benchmark . arXiv preprint arXiv:2210.07316

2022 doi arXiv
[24]

R Naik, V Chandrasekaran, M Yuksekgonul, H Palangi, and B Nushi. 2023. Diversity of thought improves reasoning abilities of llms. arXiv preprint

2023
[25]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2022 arXiv
[26]

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. https://doi.org/10.48550/ARXIV.2404.19733 Iterative reasoning preference optimization . CoRR, abs/2404.19733

2024 doi arXiv
[27]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[28]

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

2019 arXiv
[29]

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2024. https://arxiv.org/abs/2305.17493 The curse of recursion: Training on generated data makes models forget . Preprint, arXiv:2305.17493

2024 arXiv
[30]

Keith E Stanovich. 2012. On the distinction between rationality and intelligence: Implications for understanding individual differences in reasoning. The Oxford handbook of thinking and reasoning, pages 343--365

2012
[31]

Guy Tevet and Jonathan Berant. 2020. Evaluating the evaluation of diversity in natural language generation. arXiv preprint arXiv:2004.02990

2020 arXiv
[32]

Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

2018
[33]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[34]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[35]

Ting Wu, Xuefeng Li, and Pengfei Liu. 2024. https://doi.org/10.48550/ARXIV.2407.05013 Progress or regress? self-improvement reversal in post-training . CoRR, abs/2407.05013

2024 doi arXiv
[36]

Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. 2024. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36

2024
[37]

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682

2023 arXiv
[38]

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. 2024. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline. arXiv preprint arXiv:2404.02893

2024 arXiv
[39]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

2024
[40]

Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. 2024. Flow of reasoning: Efficient training of llm policy with divergent thinking. arXiv preprint arXiv:2406.05673

2024 arXiv
[41]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. https://openreview.net/forum?id=0NphYCmgua Self-rewarding language models . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria...

2024
[42]

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

2022
[43]

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

2023 arXiv
[44]

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

2019 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

2024

[4] [4]

Dhruv Batra, Payman Yadollahpour, Abner Guzman-Rivera, and Gregory Shakhnarovich. 2012. Diverse m-best solutions in markov random fields. In Computer Vision--ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 1--16. Springer

work page 2012

[5] [5]

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

2017

[6] [6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021

[7] [7]

Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. 2024. https://arxiv.org/abs/2402.07043 A tale of tails: Model collapse as a change of scaling laws . Preprint, arXiv:2402.07043

arXiv 2024

[8] [8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024