{"id":"918ba01e-cbae-4d12-ac05-a92e2aa0c613","arxiv_id":"2411.19039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-agent preference optimization method that uses pooled correct answers from several LLMs as shared positives and each model's own errors as negatives improves math reasoning accuracy on GSM8K and MATH.","lead":"The paper introduces Mars-PO, a training method where several math-focused AI models generate answers to the same problems; correct answers from all models are pooled and used to teach each model, while each model's own mistakes are used as negative examples. The method lifts accuracy on standard math benchmarks, for example raising Llama3.1-8B on the MATH test from about 50% to 58%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manuscript never states whether the GSM8K/MATH data in D is the official train split or includes evaluation problems; if test problems are in D, the reported accuracy gains reflect memorization rather than improved reasoning.","rationale":"The reader's CONDITIONAL verdict is appropriate. I agree that the reward-model assumption is a weakness, but I find the missing train/test split more load-bearing because it threatens the validity of all reported numbers, not just the attribution of the gain to reward-model quality. The paper's method is otherwise plausible: the hybrid positive pool plus agent-specific negatives is a reasonable extension of DPO, and the gains are consistent across agents and iterations. However, without a stated split, the experiments are not verifiable. The reward-model concern is secondary: positive samples are already filtered by answer correctness, so even a random ranking among correct outputs would still yield a valid positive set; the reward model only affects which correct reasoning trace is used. Therefore the split is the first thing to check. If the split is clean, the remaining issues (no error bars, no code, reward model not ablated) would justify a conditional acceptance pending release; if the split is contaminated, the paper should be rejected. My recommendation is to keep the reader's CONDITIONAL verdict, as the requested clarification would settle the central question.","tokens_in":8599,"tokens_out":5486,"duration_ms":53652,"concrete_test":"Ask the authors to release the exact training split (or a hash of D) and run an overlap check between D and the GSM8K/MATH test sets. Report the number of overlapping problems. If the overlap is zero, retrain Llama3.1-8B-Instruct with Mars-PO on the official train split and confirm the 57.82% MATH result; if the overlap is nonzero, the headline claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Mars-PO improves mathematical reasoning on held-out GSM8K and MATH. This requires that the response-generation and training set D (defined in §3.1) be disjoint from the evaluation sets. The paper never states this. §4.2 lists GSM8K and MATH as evaluation benchmarks but gives no train/test split; §4.4 describes generating N solutions 'for each problem' and training on the resulting pairs, again without specifying the split. Because the method trains on the correct answers (yi) of the same benchmark, any overlap between D and the evaluation problems would let the model memorize answers, and the reported Llama3.1 MATH gain (50.38% → 57.82%) would not demonstrate improved reasoning. No code or data is released, so the split cannot be checked from the manuscript alone. This is the most load-bearing concern: unlike the reward-model assumption, which only affects which correct solution is chosen, a split violation invalidates every quantitative result in Table 1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Mars-PO proposes a multi-agent preference optimization framework for improving mathematical reasoning of instruction-tuned LLMs. The method (i) samples multiple responses per prompt from each of three agents (Qwen2.5-Math-7B-Instruct, DeepSeek-Math-7B-RL, Llama3.1-8B-Instruct), (ii) classifies responses by final-answer correctness, (iii) selects a hybrid positive set by scoring correct responses with an external reward model (Qwen2.5-Math-RM-72B), and (iv) trains each agent with DPO using the shared hybrid positives and agent-specific negatives, plus an NLL term, over three iterations. The authors report accuracy gains on GSM8K and MATH for all three agents, with the largest gain for Llama3.1-8B-Instruct on MATH (50.38% to 57.82%).","tokens_in":8800,"tokens_out":4709,"duration_ms":62808,"significance":"If the reported gains are reproducible and not an artifact of data leakage, the framework is a simple and potentially useful extension of DPO to multi-agent settings: sharing correct solutions across agents in preference pairs is a plausible way to transfer reasoning strengths, and the gains for DeepSeek-Math and Llama are nontrivial. The paper also includes a useful negative result for vanilla DPO (large degradation on these benchmarks) and an SFT-with-positive-samples baseline. However, the significance is currently limited by missing train/test split details, absent statistical uncertainty quantification, and a likely sign error in the NLL loss, all of which prevent a reader from verifying the central empirical claim.","major_comments":[{"comment":"The manuscript never states whether the training dataset D in §3.1 is the official train split of GSM8K/MATH or includes the evaluation problems. Since the method trains on the correct answers yi from D, any overlap between D and the test sets used to produce Table 1 would make all reported accuracy gains reflect memorization rather than improved reasoning. The authors must explicitly specify the train/test split, state how prompts were sampled, and ideally release code/data to allow verification. This is the most load-bearing issue: without this clarification, every quantitative conclusion in the paper is unverifiable.","section":"§3.1, §4.2, §4.4"},{"comment":"No error bars, confidence intervals, or significance tests are reported. For Qwen2.5-Math-7B-Instruct, the Mars-PO iter1 gains are only +0.15 on GSM8K and +0.16 on MATH; on a benchmark with hundreds to thousands of problems, these differences are within the scale of one or two questions and are indistinguishable from sampling noise. The paper's claim of 'consistent' improvements across all agents is therefore not statistically supported for the strongest baseline model. The authors should report variance across seeds or a significance test, especially for small-magnitude gains.","section":"Table 1, §5.1"},{"comment":"The NLL term is written as L_NLL = log Mtheta(cw, aw|xi) / (|cw| + |yw|). This is a positive (length-normalized) log-likelihood of the chosen response. Since the total loss is L = L_DPO + alpha * L_NLL and the objective is minimization, this term would minimize the likelihood of the chosen response, which is the opposite of the intended NLL regularization described in the text. A missing minus sign appears likely. The sign convention must be corrected and clarified, and the broken reference 'Equation ??' in §4.4 should be fixed.","section":"§3.3, Eq. (4)"},{"comment":"The reward model Qwen2.5-Math-RM-72B belongs to the same model family as one of the three agents (Qwen2.5-Math-7B-Instruct). The paper does not analyze the composition of the hybrid positive sample set or ablate the reward model choice. If the reward model systematically favors Qwen-style solutions, the observed pattern of tiny gains for Qwen and larger gains for DeepSeek/Llama could stem from style transfer toward Qwen-like reasoning rather than from the claimed diversity-based benefit. The authors should report the distribution of selected positives across the three agents and add an ablation with a different reward model or no reward-model selection.","section":"§3.2, §4.1"},{"comment":"Eq. (3) uses Mk as the reference model in the DPO ratio, but the paper does not state whether Mk is frozen at its initial weights or updated across the three iterations. In iterative preference optimization, the choice of reference model changes the DPO objective; using the current policy as reference would break the standard DPO derivation. Please specify the reference-modelling procedure for each iteration and justify it.","section":"§3.3, §5.2"}],"minor_comments":[{"comment":"The header 'Postive SFT' is a typo and should read 'Positive SFT'.","section":"Table 1"},{"comment":"The sentence 'Note that these are the results after the first iteration of training' is contradicted by Table 1, which reports iter1, iter2, and iter3 results. The authors should clarify whether the numbers discussed in §5.1 refer to iter1 only or to all iterations, and which iteration is used in the abstract's headline gain.","section":"§5.1, Table 1"},{"comment":"The sentence 'To sum up, our main contribution are as follows' should be 'our main contributions are as follows'.","section":"§1"},{"comment":"The phrase 'as shown in in Table 1' contains a duplicated 'in'.","section":"§5.1"},{"comment":"The paper cites Touvron et al. (2023) for Llama3.1-8B-Instruct, but the cited reference is the original Llama paper; the Llama3.1 technical report or a proper citation for the 8B Instruct model should be used.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The train/test split ambiguity is the decisive issue for this paper. If the authors cannot provide a clear statement and supporting data that the training set is disjoint from the evaluation sets, the paper should not be accepted. In its current form, the missing split specification and the absence of code/data, combined with the NLL sign error, prevent independent verification of the central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick bottom line: Mars-PO is a reasonable and clearly explained extension of DPO—use a hybrid positive pool selected by a reward model from multiple agents' correct outputs, pair with agent-specific negatives, add NLL, iterate. The reported gains are consistent across three agents and two benchmarks, including a +7.4 point MATH improvement on Llama3.1-8B. That is the kind of result worth checking.\n\nWhat is actually new: the hybrid positive sample set construction plus agent-specific negatives, as a multi-agent preference optimization scheme. It is a natural extension, but I don't know of prior work doing exactly this. The experimental comparisons against vanilla DPO, DPO+NLL, and positive-only SFT are the right controls, and the finding that vanilla DPO hurts math reasoning while Mars-PO helps is coherent with recent work (e.g., Iterative RPO, Step-DPO).\n\nNow the soft spots, in order of severity.\n\nFirst and decisive: the paper never states whether the GSM8K and MATH problems used for generating responses and training (§3.1, §4.4) are the official train splits or the full benchmark including the test/evaluation problems. It says \"the used training dataset is D\" and \"we generate N solutions for each problem,\" but doesn't say D is the train split of GSM8K/MATH. If any evaluation problems appear in D, the accuracy gains in Table 1 reflect memorization, not improved reasoning. No code or data is released, so I can't check. This is the single most load-bearing missing detail.\n\nSecond, the empirical reporting is thin: no error bars, no repeated runs, no significance tests. The differences are mostly large (5–7 points), so they might survive, but with no variance estimate I'd want at least a few seeds.\n\nThird, the positive sample selection depends on Qwen2.5-Math-RM-72B scoring. That's a sensible choice, and the paper does not claim the reward model is perfect. The risk is that the hybrid pool is biased toward Qwen-like solutions, which could penalize the other agents' training. Worth a discussion or ablation, though not necessarily fatal.\n\nFourth, minor mechanical issues: a reference to \"Equation ??\" in §4.4, and some overstatements in the intro/abstract (e.g., \"over 10%\" vs +5.10 points on Llama MATH; \"up to about 8%\" is vague relative to the table). These are fixable.\n\nThe citation pattern looks fine; the related work covers the key DPO-for-math papers. The central claim is plausible but not yet substantiated because of the split ambiguity. This is exactly what peer review should sort out. I'd send it to review with a strong request for code/data and explicit train/test split statements, plus repeated-run statistics. If the split turns out clean and the gains replicate, it's a useful contribution to the alignment subfield.","headline":"A plausible multi-agent DPO variant with consistent reported gains, but the paper never states whether it trained on the GSM8K/MATH test splits—and that omission is decisive.","tokens_in":9308,"tokens_out":2508,"would_cite":false,"duration_ms":21616,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mars-PO pools the best correct answers from several agents and pairs them with each agent's mistakes, lifting Llama3.1-8B-Instruct on MATH from 50.38% to 57.82%.","keywords":["mathematical reasoning","direct preference optimization","multi-agent LLM system","hybrid positive samples","reward model scoring","iterative preference training","GSM8K","MATH benchmark"],"falsifier":"Run the full pipeline twice with everything fixed except the hybrid positive set: once with reward-model-selected correct answers and once with a same-size random sample of correct answers; if the MATH accuracy matches, reward-model selection is not what carries the result.","tokens_in":8404,"feed_emoji":"🧮","tokens_out":10632,"duration_ms":81739,"temperature":0.7,"pith_summary":"The paper sets out to show that a multi-agent twist on direct preference optimization (DPO) can push already instruction-tuned language models further on mathematical reasoning. For each problem, several agents generate many candidate solutions; the correct ones are pooled, scored by a reward model, and the highest-scoring correct solutions become a shared positive set, while each agent keeps its own wrong answers as negatives. Training each agent on these pairs with a DPO loss plus a next-token likelihood loss is reported to beat vanilla DPO, DPO with an extra NLL term, and supervised fine-tuning on the same positives. The headline result is Llama3.1-8B-Instruct improving from 50.38% to 57.82% on the MATH benchmark over three iterations.","feed_headline":"Multi-agent preference training lifts MATH score from 50.38% to 57.82%","feed_subtitle":"Mixing three agents' best correct answers with each agent's errors beats vanilla DPO on GSM8K and MATH.","key_machinery":"The load-bearing object is the hybrid positive sample set: correct answers sampled from all agents, scored by a reward model, and reduced to a shared high-quality subset. The paper pairs this shared positive set with each agent's own negative samples and optimizes $\\mathcal{L} = \\mathcal{L}_{\\mathrm{DPO}} + \\alpha \\mathcal{L}_{\\mathrm{NLL}}$ over three iterations of sampling and retraining. This construction is what converts multi-agent diversity into stronger preference pairs and is the claimed reason Mars-PO outperforms single-agent DPO and positive-sample SFT.","core_discovery":"The central discovery is that the quality and diversity of positive samples, not just the contrast with negatives, is what makes preference optimization work for mathematical reasoning. By merging correct outputs across agents and filtering them with a reward model, the authors construct what they call a hybrid positive sample set; pairing it with agent-specific negative samples avoids the degradation that vanilla DPO causes on already fine-tuned models. The paper reports consistent gains across all three agents on both GSM8K and MATH, with the largest gains on the weakest agent, and shows that supervised fine-tuning on the same positive samples without the contrastive negatives gives much smaller improvements.","pith_inferences":["A natural testable extension is to replace the 72-billion-parameter reward model with a smaller or process-based scorer; if the gains shrink, reward-model quality is the real driver rather than multi-agent pooling itself.","The method implicitly assumes the reward model's score tracks solution quality; a version using human-rated correct solutions could separate those two factors.","The same hybrid-positive, agent-specific-negative recipe could transfer to other verdictable generation tasks, such as code generation, where correctness can be checked and a scorer exists."],"forward_implications":["If the mechanism holds, improving math reasoning by preference optimization does not require step-level error annotations; final-answer correctness plus reward-model selection is enough.","Multi-agent alignment can amplify the weakest model in the group, since the largest reported gains are on Llama3.1-8B-Instruct, the weakest of the three agents.","Vanilla DPO should be expected to hurt already math-tuned models, while shared positives paired with agent-specific negatives should improve them.","Iterative retraining on freshly sampled responses continues to add accuracy, so the framework can be run as a loop rather than a one-shot post-training step."],"supporting_citations":[{"why":"Supplies the DPO objective that Mars-PO extends to a multi-agent setting.","marker":"Rafailov et al., 2024"},{"why":"Provides the 72-billion-parameter reward model used to score and select hybrid positive samples.","marker":"Qwen, 2024"},{"why":"Presents Qwen2.5-Math, one of the three agents and the source of the reward model design.","marker":"Yang et al., 2024"},{"why":"Presents DeepSeek-Math, one of the three agents and a math reasoning baseline.","marker":"Shao et al., 2024"},{"why":"Motivates iterative reasoning preference optimization and the NLL-augmented DPO loss used as a baseline.","marker":"Pang et al., 2024"},{"why":"A prior step-controlled DPO variant for math reasoning that motivates the comparison set.","marker":"Lu et al., 2024b"},{"why":"A prior step-wise preference method for long-chain reasoning used as a comparison point.","marker":"Lai et al., 2024"},{"why":"Provides the GSM8K benchmark used for evaluation.","marker":"Cobbe et al., 2021"},{"why":"Provides the MATH benchmark used for evaluation.","marker":"Hendrycks et al., 2021"}],"fun_headline_variants":["Mars-PO: Hybrid positive sample set beats vanilla DPO on math benchmarks","Multi-agent correct answers merged into preference pairs lift MATH by 7 points","Why mixing agents' correct outputs beats single-agent preference optimization","Shared positives plus agent-specific negatives boost Llama3.1-8B on MATH","Preference optimization with hybrid positive set raises MATH accuracy to 57.82%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the reward model reliably ranks the quality of correct solutions, so the highest-scoring correct answers across agents genuinely teach each agent better than its own correct answers.","fun_headline_variants_meta":{"raw":{"variants":["Mars-PO: Hybrid positive sample set beats vanilla DPO on math benchmarks","Multi-agent correct answers merged into preference pairs lift MATH by 7 points","Why mixing agents' correct outputs beats single-agent preference optimization","Shared positives plus agent-specific negatives boost Llama3.1-8B on MATH","Preference optimization with hybrid positive set raises MATH accuracy to 57.82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1870,"prompt_tokens":858,"completion_tokens":1012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":913}},"tokens_in":474,"tokens_out":1012,"duration_ms":10728,"temperature":1.0,"reasoning_tokens":913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:35:25.547288+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline twice with everything fixed except the hybrid positive set: once with reward-model-selected correct answers and once with a same-size random sample of correct answers; if the MATH accuracy matches, reward-model selection is not what carries the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 72-billion-parameter reward model used to score and select hybrid positive samples."}],"review_version":1}