Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

PD$^3$: A Project Duplication Detection Framework via Adapted Multi-Agent Debate

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A round-robin multi-agent debate retrieves duplicate project references more accurately than word, vector, and single-LLM baselines.

desk verdict Round-robin MAD is a real idea, but the paper's own tables contradict its headline improvement claims and the evaluation lacks significance testing, so it needs major revision before it can be trusted. read the letter →

arxiv 2505.17492 v1 pith:BDHTBKT7 submitted 2025-05-23 cs.AI cs.CLcs.LG

Dezheng Bao , Yueci Yang , Xin Chen , Zhengxuan Jiang , Zeguo Fei , Daoze Zhang , Xuanwen Huang , Junru Chen
show 3 more authors
Chutian Yu Xiang Yuan Yang Yang
This is my paper
classification cs.AIcs.CLcs.LG
keywords projectduplicationdetectionmulti-agentdebateround-robincompetitiontop-KretrievalLLM-as-a-judgepowerprojectsexpertannotationreviewplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PD³ is a framework for detecting duplicate research projects: given a new project, it retrieves the five most relevant past projects and produces a duplication score plus qualitative comparison. The paper's central claim is that its round-robin multi-agent debate—where small groups of candidate projects are debated in parallel tournaments and winners are aggregated by voting—retrieves more relevant references than word-frequency, vector-distance, single-LLM, and vanilla multi-agent baselines on 833 real power projects. The paper also argues that LLM-as-a-judge feedback, combining a 1–10 duplication score with similarity conclusions and original-text comparisons, better matches expert preferences than scores from baselines. The motivating insight is that project relevance is not a strict partial order: a project can be more relevant than another on content, on technology, or on applications in ways that cycle, so global debate rather than pairwise ranking is needed.

What carries the argument

The central mechanism is the round-robin competition format for multi-agent debate. Thirty candidates from embedding-based preliminary retrieval are randomly split into six sets of five, then organized into 15 non-repeating 5-out-of-20 sub-competitions. In each sub-competition, three expert LLM agents each select their top-5, debate for two rounds after an initial round, and a senior judge picks the group top-5; results are aggregated across groups by voting to yield the global top-5. This structure keeps context short (20 projects per debate), gives every candidate equal participation, and runs the 15 groups in parallel; the paper argues this balances the 'lost in the middle' context degradation against the need for global information.

What would settle it

Re-run Task 1 with an independent expert panel annotating the same 331 test items and measure inter-annotator agreement; if the annotator disagreement on the top-5 sets is comparable to or larger than PD³'s 0.0079 Precision@5 margin over the best baseline, the central claim would be falsified. Alternatively, a randomized permutation of candidate grouping within the round-robin structure that still yields the same top-5 would show the debate adds no information beyond the candidate pool.

Watch

Extended reading notes

Core claim

On real data from 833 power-industry projects, PD³'s round-robin MAD retrieval reaches Precision@5 of 0.4423 against expert annotations on 331 test items, edging the strongest baseline (MAD Sliding Window) at 0.4344, and its feedback module achieves 0.6700/0.6467 accuracy on the weighted-group scoring task compared with the best baseline's 0.6500/0.6233. The paper attributes the gain to the round-robin format's equal exposure of every candidate, which avoids the unfairness of traversal and the context-length degradation of direct MAD, while still giving agents global information through repeated sub-competitions and a senior-judge vote.

Load-bearing premise

The results rest on the assumption that the expert annotations—three reviewers' judgments on 331 retrieval tasks and 100 paired scoring tasks—are a stable, unbiased ground truth; if those labels are noisy or systematically differ between annotators, the reported gains of a few percentage points may not be real.

Editorial extensions

If this is right

  • If the paper is right, embedding-prefiltered candidate pools of about 30 can be safely used, and the round-robin debate will recover relevant items that static similarity misses.
  • The adapted MAD format can be reused for other selection tasks where relevance is multi-dimensional and not a strict partial order.
  • The integrated qualitative feedback (similarity conclusion with original text comparison) means a duplication detection system can return not just a rank but an audit trail for human experts.
  • A live deployment on 118 real proposals led to 20 projects being flagged ineligible and prevented an estimated $5.73M in spending, suggesting operational feasibility.
  • The cost profile (about 3 minutes and roughly $1.34 per project with parallel execution) makes the framework practical for routine screening.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The round-robin-voting design is essentially a ranking aggregation scheme; one could test whether the debate records themselves, rather than just the final votes, contain signals that smaller or cheaper models could distill, since the paper only reports the final top-5.
  • The paper's own ablation on a 60-item subset shows that three debate rounds outperform the deployed two-round setting, so the reported performance may underestimate the framework's ceiling under higher compute.
  • The 'no strict partial order' motivation suggests PD³ might transfer to peer review, grant triage, or patent prior-art search, where multi-criteria relevance also cycles; the paper does not test these domains.
  • The duplication score is prompt-defined and domain-specific; the same judge pipeline could be re-parameterized for novelty assessment if the scoring criteria were inverted, but the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PD3, a framework for duplicate detection of scientific project proposals. It first embeds project texts and retrieves a 30-candidate shortlist by vector similarity; then a multi-agent debate with a round-robin group-competition design selects the top-5 most relevant reference projects; finally, LLM-based agents output a quantitative duplication score and qualitative similarity conclusions and text comparisons. The authors evaluate on 833 State Grid Corporation power projects, using 331 annotated retrieval tasks and 100 annotated pairwise scoring tasks, and claim improvements of 7.43% and 8.00% over baselines in the two downstream tasks. They also describe the deployed platform Review Dingdang and report cost savings of 5.73 million USD during a live test.

Significance. If the quantitative claims were reliable, the paper would present a practical, deployed application of multi-agent debate to a real-world retrieval and feedback task, with a concrete cost-saving result. The framework is clearly described and the runtime/cost analysis is a useful inclusion. However, the central empirical claims are not supported by the paper's own tables: the reported improvement percentages are not reproducible, some settings show PD3 below the best baseline, and the observed gains over the strongest baselines are small and not tested for statistical significance. The evaluation also has methodological weaknesses, including test-set-derived hyperparameter choices and a lack of annotation-reliability reporting. As written, the current evidence is not strong enough to establish the claimed superiority.

major comments (4)
  1. [Abstract; §3.3; Tables 1 and 2] The central quantitative claims are contradicted by the paper's own tables. The abstract and Section 3.3 state that PD3 'outperforms existing approaches by 7.43% and 8.00%' and achieves 'consistent gains in Match@K (K=1-5)' with improvements of 5.14%, 11.62%, 11.74%, 5.09%, and 2.32%. In Table 1, PD3's Precision@5 is 0.4423 versus 0.4344 for the best baseline MAD Sliding Window, a relative improvement of about 1.8%, not 7.43%; the Match@3 count is identical (133/331) and the Match@4 count (41/331) is lower than MAD Sliding Window's 42/331, so the 'consistent gains' claim is false. In Table 2, under Human Retrieval Origin Group, gte-1.5B AVG achieves 0.6500 while PD3 feedback achieves 0.6400, directly contradicting the claim of outperforming baselines 'in both evaluation settings under all retrieval settings'. None of the reported improvement percentages can be derived from the tables under any stated comparison rule. This is a load-bearing issue because the paper's main contribution is the claimed empirical superiority.
  2. [§2.1; §3.4; Figure 1(b); Tables 5-7] Hyperparameter choices appear to be made on the evaluation test set, which can inflate the reported gains. Section 2.1 states that the preliminary retrieval count N=30 is the 'optimal count, as validated in preliminary experiments' with Figure 1(b) showing top-5 coverage versus candidate count, but the same 331 test items are later used for the final evaluation. Section 3.4 and Table 5 additionally select the group size M=20 from results on the full 331-item test set, while debate rounds and agent counts are tuned on a 60-sample subset without a separate holdout (Tables 6 and 7). There is no held-out validation split or nested selection procedure, so the reported precision numbers are at risk of overfitting to the test set. The authors should report results on a separate validation set or use nested cross-validation.
  3. [§3.2; §3.3; Appendix A.3; Table 2] The statistical significance and annotation reliability of the Task 2 results are not established. The observed differences in Table 2 are a few percentage points (for example, 0.6400 vs 0.6500 in one setting), yet no confidence intervals, bootstrap estimates, or significance tests are reported for either task. For Task 2, the ground truth is a three-expert majority vote on only 100 pairs, but no inter-annotator agreement measure (e.g., Cohen's kappa) is given. Appendix A.3 introduces an ad-hoc tie-breaking rule: when an LLM assigns equal scores to both projects, the score is 0 under the Origin Group and the minority-label value under the Weighted Group. This rule can shift accuracy by several points, which is on the same scale as the claimed improvements. Without annotation-quality metrics and sensitivity analyses, the reported superiority may not be robust.
  4. [§2.2; Tables 1 and 2] The randomized round-robin design and the stochasticity of LLM-based methods are not sufficiently characterized. Section 2.2 describes randomly partitioning the 30 candidates into 6 sets and forming G=15 sub-competitions, but the paper does not report the number of random partition seeds or the variance of results across repeated runs. Since the debate, scoring, and voting procedures all involve stochastic LLM generations, retrieval and scoring outcomes could vary substantially between runs; the single-run numbers in Tables 1 and 2 are insufficient to establish that the differences between PD3 and the baselines are not due to randomness. The authors should report mean and standard deviation (or per-run results) over multiple seeds.
minor comments (5)
  1. [§2.2] There is a typo: 's unique challenge' should be 'a unique challenge'.
  2. [Table 1] The header 'Match@K K= 1K= 2K= 3K= 4K= 5(Hit Rate@5)' is hard to parse; the count and ratio columns should be clearly separated, and the definition of 'Hit Rate@5' should be aligned with the metric description in §3.2.
  3. [§3.3] Two different sets of Match@K improvement percentages are given in the same subsection (5.14%, 11.62%, 11.74%, 5.09%, 2.32% and later 5.81%, 19.27%, 41.26%, 69.78%, 332.86%); they are mutually inconsistent and neither set is derivable from Table 1, so they should be reconciled or removed.
  4. [Abstract; §4] The cost-saving figure of 5.73 million USD is stated without a derivation; the authors should explain how this amount is computed from the 20 prevented projects and the total project budget.
  5. [§1; Related Work] The claim of being 'the first LLM-based framework for project duplication detection' may be too strong in light of existing LLM-based plagiarism and similarity detection systems; the authors should temper the claim or cite and differentiate from relevant prior work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PD3's retrieval and scoring claims rest on external expert annotations and are not defined in terms of its own outputs.

full rationale

PD3's central retrieval claim is an empirical comparison against expert-annotated ground truth: 331 projects with expert top-5 sets and 100 pairwise expert votes. Those annotations are external to the model, so the framework is not defined in terms of its own target. The round-robin MAD module outputs a candidate ranking through sub-competition voting, and the reported Precision@5 and Match@K metrics are computed against the expert sets, not against PD3's own scores. Task 2's duplication score is evaluated by comparing predicted pairwise preference to three experts' votes; the scoring prompt's 'review experts' conclusion' is PD3's own qualitative summary, which creates a potential self-consistency concern in the conclusion-augmented ablation, but it is not a circular derivation because the conclusion does not encode the pairwise ground-truth labels and the score is still compared with independent human votes. Hyperparameters such as M=20 and agent/round counts are tuned on separate preliminary or 60-sample experiments, and the paper even reports final results with a suboptimal 2-round configuration, weakening any claim that the main results are forced by chosen settings. The only self-citation, reference [35] (RE2), is used as background for LLM peer review and is not load-bearing for the paper's contributions. The limitations section explicitly acknowledges annotation-cost and base-model generalizability constraints, which are external validity concerns rather than circularity. No equation or construction in the paper makes a predicted quantity equal to an input or a fitted parameter by definition.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on domain assumptions about the structure of relevance, the recall of the embedding prefilter, the reliability of expert labels, and the value of multi-agent debate. Hyperparameters such as pool size, group size, agent count, and debate rounds are fitted to the same benchmark, which adds a tuning burden the paper does not fully account for.

free parameters (4)
  • Candidate pool size after preliminary retrieval (N) = 30
    Chosen to balance recall against context length; Figure 1(b) claims about 90% top-5 coverage at 30 candidates, and sensitivity is not fully reported across domains.
  • Round-robin group size (M) = 20
    Selected as the optimum in Table 5 on the same benchmark, a test-set-derived choice that affects retrieval quality.
  • Number of debate agents = 3
    Selected from Table 7 as the optimum on a 60-sample subset; the deployed configuration uses this value.
  • Number of debate rounds = 2
    The paper reports 3 rounds as the empirical optimum in Table 6 but deploys 2 rounds due to resource constraints, so the reported configuration is not the tested optimum.
assumptions (5)
  • domain assumption Relevance among projects is not a strict partial order, so point-wise and pair-wise comparisons are insufficient.
    Section 2.1 and Figure 1(a) motivate the global round-robin debate design; if relevance were a strict partial order, simpler ranking methods would suffice.
  • domain assumption The 30-candidate vector prefilter contains the relevant reference projects with sufficient recall.
    Section 2.2 and Figure 1(b) assert that 30 candidates cover roughly 90% of top-5 relevant projects; any relevant project missing from the 30 cannot be retrieved by the debate stage.
  • domain assumption Expert annotations on 331 tasks and 100 pairs are a reliable ground truth for duplication relevance and relative duplication.
    Section 3.2 defines Tasks 1 and 2 using expert labels, but no inter-annotator agreement or adjudication details are reported.
  • domain assumption Three independent LLM expert agents plus a senior judge produce more reliable rankings than a single LLM.
    Section 2.2 and the related work assume MAD improves reasoning through debate; this is the design premise tested by the baselines but not independently proven.
  • domain assumption LLM-as-a-judge scores on a 1-10 scale correspond to human expert judgments.
    Section 2.3 uses a hand-written scoring rubric and averages three LLM ratings; the score distribution is not calibrated against human score distributions beyond task-level accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PD$^3$: A Project Duplication Detection Framework via Adapted Multi-Agent Debate." pith.science (2026). https://pith.science/paper/BDHTBKT7

@misc{pith2026250517492,
  author       = {Pith},
  title        = {Pith review of: PD$^3$: A Project Duplication Detection Framework via Adapted Multi-Agent Debate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDHTBKT7}},
  note         = {Machine review of arXiv:2505.17492}
}
abstract

Project duplication detection is critical for project quality assessment, as it improves resource utilization efficiency by preventing investing in newly proposed project that have already been studied. It requires the ability to understand high-level semantics and generate constructive and valuable feedback. Existing detection methods rely on basic word- or sentence-level comparison or solely apply large language models, lacking valuable insights for experts and in-depth comprehension of project content and review criteria. To tackle this issue, we propose PD$^3$, a Project Duplication Detection framework via adapted multi-agent Debate. Inspired by real-world expert debates, it employs a fair competition format to guide multi-agent debate to retrieve relevant projects. For feedback, it incorporates both qualitative and quantitative analysis to improve its practicality. Over 800 real-world power project data spanning more than 20 specialized fields are used to evaluate the framework, demonstrating that our method outperforms existing approaches by 7.43% and 8.00% in two downstream tasks. Furthermore, we establish an online platform, Review Dingdang, to assist power experts, saving 5.73 million USD in initial detection on more than 100 newly proposed projects.

Figures

Figures reproduced from arXiv: 2505.17492 by the authors.

Figure 1
Figure 1. (a) Case on the lack of strict partial order. This case shows exhibiting ranking intransitivity: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Review Dingdang, the power project duplication platform based on PD3 . set is required. Nevertheless, current point-wise or pair-wise scoring methods (also including vector distance-based methods) incorrectly assume partial ordering, limiting their effectiveness. From a methodological viewpoint, an effective detection method should support multi-perspective analysis to ensure comprehensive project comparisons. Conve… view at source ↗
Figure 3
Figure 3. Overview of PD3 framework. According to the I/O sequence of the project under detection, the framework consists of four parts: data pre-processing, database and preliminary retrieval, MAD￾based round-robin retrieval and LLM-as-a-Judge-based Feedback. we also implement an online platform, Review Dingdang, achieving practical impact by preventing millions of USD in redundant investments during live test for new projec… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: A case study from the Review DingDang detection process conducted by PD3 . In one sub-competition phase, three experts first independently select their top-5 choices. After debating their differing perspectives, Expert A ultimately accepts the views of Experts B and C.…
Figure 4
Figure 4. Figure 4: (a) Performance comparison across different group size. (b) Performance comparison [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: Prompt template used in the initial round of MAD-based round-robin retrieval. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Prompt template used in the debate round of MAD-based round-robin retrieval. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt template used to set senior expert round in MAD-based round-robin retrieval. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompt template used to generate duplication score of LLM-as-a-Judge-based feedback. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

    cs.SE 2026-07 accept novelty 6.0 of 10

    A systematic review of 141 papers derives a three-axis taxonomy of multi-agent debate design (participants, interaction, agreement) and shows the field has converged on a narrow default pattern.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Intrinsic plagiarism detection using n-gram classes

    Imene Bensalem, Paolo Rosso, and Salim Chikhi. Intrinsic plagiarism detection using n-gram classes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1459–1464, 2014

    work page 2014

  2. [2]

    volcengine, 2025

    ByteDance. volcengine, 2025. URLhttps://www.volcengine.com/. Accessed: 2025-05-15

    work page 2025

  3. [3]

    Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

    arXiv 2023

  4. [4]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024

    arXiv 2024

  5. [5]

    Debate-feedback: A multi-agent framework for efficient legal judgment prediction

    Xi Chen, Mao Mao, Shuo Li, and Haotian Shangguan. Debate-feedback: A multi-agent framework for efficient legal judgment prediction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 462–470, 2025

    work page 2025

  6. [6]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. URLhttps://arxiv.org/abs/2412.19437

    arXiv 2024

  7. [7]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  8. [8]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first International Conference on Machine Learning, 2023

    2023

Show all 37 references
  1. [9]

    Multi-llm debate: Framework, principals, and interventions.Advances in Neural Information Processing Systems, 37:28938–28964, 2024

    Andrew Estornell and Yang Liu. Multi-llm debate: Framework, principals, and interventions.Advances in Neural Information Processing Systems, 37:28938–28964, 2024

    2024

  2. [10]

    Towards an ai co-scientist.arXiv preprint arXiv:2502.18864, 2025

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864, 2025

    2025 arXiv

  3. [11]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    V oting or consensus? decision-making in multi-agent debate.arXiv preprint arXiv:2502.19130, 2025

    Lars Benedikt Kaesberg, Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. V oting or consensus? decision-making in multi-agent debate.arXiv preprint arXiv:2502.19130, 2025

    2025

  6. [14]

    Making large language models a better foundation for dense retrieval.arXiv preprint arXiv:2312.15503, 2023

    Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval.arXiv preprint arXiv:2312.15503, 2023

    2023

  7. [15]

    Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281, 2023

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  8. [16]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Proc...

    2024

  9. [17]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  10. [18]

    Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023

    2023 arXiv

  11. [19]

    Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024. 11

    2024

  12. [20]

    Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781, 2013

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  13. [21]

    Let models speak ciphers: Multiagent debate through embeddings

    Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. In The Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  15. [23]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009

    2009

  16. [24]

    Mred: A meta-review dataset for structure-controllable text generation

    Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. Mred: A meta-review dataset for structure-controllable text generation. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, 2022

    2022

  17. [25]

    Should we be going mad? a look at multi-agent debate strategies for llms

    Andries Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms. InProceedings of the 41st International Conference on Machine Learning, pages 45883–45905, 2024

    2024

  18. [26]

    jina-embeddings-v3: Multilingual embeddings with task lora.arXiv preprint arXiv:2409.10173, 2024

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings-v3: Multilingual embeddings with task lora.arXiv preprint arXiv:2409.10173, 2024

    2024 arXiv

  19. [27]

    Beir: A hetero- geneous benchmark for zero-shot evaluation of information retrieval models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A hetero- geneous benchmark for zero-shot evaluation of information retrieval models. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (R...

    2021

  20. [28]

    The State Grid Corporation of China 2023 Social Re- sponsibility Report, 2024

    The State Grid Corporation of China. The State Grid Corporation of China 2023 Social Re- sponsibility Report, 2024. URL http://www.sgcc.com.cn/u/cms/sgcc_main/other/202408/ 20240806163601853978745.pdf. [Online; accessed 2025-05-15]

    2023

  21. [29]

    Department of Energy

    The U.S. Department of Energy. Smart Grid Grants | Department of Energy, 2024. URL https: //www.energy.gov/gdo/smart-grid-grants. [Online; accessed 2025-05-15]

    2024

  22. [30]

    MMLU-Pro Leaderboard, 2025

    TIGER AI Lab. MMLU-Pro Leaderboard, 2025. URL https://huggingface.co/spaces/ TIGER-Lab/MMLU-Pro. [Online; accessed 2025-05-15]

    2025

  23. [31]

    Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? In62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 6106–6131

    Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? In62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 6106–6131. Association for Computational Ling...

    2024

  24. [32]

    Debate as optimization: Adaptive conformal prediction and diverse retrieval for event extraction

    Sijia Wang and Lifu Huang. Debate as optimization: Adaptive conformal prediction and diverse retrieval for event extraction. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 16422–16435, 2024

    2024

  25. [33]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  26. [34]

    Yang Yu, Lei Liu, Xiangyi Xu, and Shaoqian Bai.Text detection methods, devices, computing equipment, and computer-readable storage media, cn108829780b edition, 2022

    2022

  27. [35]

    Re2: A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions.arXiv preprint arXiv:2505.07920, 2025

    Daoze Zhang, Zhijian Bao, Sihang Du, ZHiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang. Re2: A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions.arXiv preprint arXiv:2505.07920, 2025

    2025

  28. [36]

    Zhenhai Zhang and Xiongyong Sun.A method and system for automatically detecting academic misconduct literature, cn101833579b edition, 2012

    2012

  29. [37]

    society of minds

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 12 A APPEND...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.