{"id":"94f9978e-0515-463f-9f71-99d1e9734607","arxiv_id":"2608.04071","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MCTS-Report applies Monte Carlo Tree Search to multimodal table-to-report generation, reaching a 77.9 overall score on the new MMRBench benchmark, but its evaluation relies on a single, unvalidated LLM judge.","lead":"The paper presents MCTS-Report, which uses Monte Carlo Tree Search to guide a large language model in building text-and-chart reports from tables, step by step. It also introduces MMRBench, a benchmark of 185 tables and 386 report tasks, and reports a 77.9 overall score that beats twelve baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed +15.2 SOTA gain rests entirely on a single unvalidated GPT-4o judge, so the central result is not yet supported.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing issue: the evaluation relies on a single unvalidated LLM judge, and the headline 15.2-point improvement could reflect judge bias rather than genuine quality. My stress-test confirms this is the most consequential gap, and adds that the absence of any human-judge correlation, significance testing, or variance reporting means the central empirical claim is currently unfalsifiable from the paper alone. The framework itself is plausible and the SQL-based factual check is a creditable design choice, but the reported numbers cannot yet be taken as evidence of SOTA performance. I therefore agree with the reader's CONDITIONAL verdict: the paper should be accepted only if the evaluation is strengthened and the statistical inconsistencies in MMRBench are resolved. No deeper objection to the MCTS formulation itself was found; the action-space size and transition constraints are coherent, and the ablation results are directionally consistent with the framework's logic, though they inherit the same judge-dependence.","tokens_in":10822,"tokens_out":2903,"duration_ms":28726,"concrete_test":"On the 50-task subset already scored by human analysts, have those analysts blindly score MCTS-Report (DeepSeek-R1) and the DeepSeek-R1 baseline reports, and compute Spearman correlation between GPT-4o judge scores and human scores. Separately, re-run the full benchmark evaluation with a different judge model (e.g., Claude-4.5-Sonnet or Gemini-2.5-Pro) and report the MCTS-Report minus DeepSeek-R1 overall gap. If the gap is not robust across judges or if GPT-4o-human correlation is below roughly 0.6, the headline SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that MCTS-Report outperforms the best baseline by 15.2 points (77.9 vs. 62.7) and rivals human chart-text alignment, is measured exclusively by GPT-4o as an LLM judge over four dimensions (Table 4; Evaluation Protocol). No human correlation is reported, despite a human baseline being collected on 50 tasks; no inter-judge agreement, confidence intervals, or significance tests are given; and the paper does not state the number of independent runs or their variance. Because the benchmark judge is an LLM, its scores may reward report length, formatting, or self-consistent narrative rather than factual quality, and the claim that MCTS improves factual reliability is therefore not verified independently of the judge. The risk is compounded for MCTS-Report (GPT-4o), where the generator and judge share a model family, but even the headline DeepSeek-R1 variant's reports are optimized against a similar self-supervised reward, which could induce properties that the GPT-4o judge systematically favors. Separate internal inconsistencies in benchmark statistics (79+131=210 vs. 185 tables; 386/185=2.09 vs. stated 1.84 tasks per table) further undermine confidence that the evaluation setup is reliably curated. The self-supervised SQL-based fact check is a genuine strength of the framework, but it is used for search guidance, not for the reported benchmark scores.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MCTS-Report, a Monte Carlo Tree Search framework that reformulates table-to-multimodal report generation as a structured search process over atomic actions (chapter planning, visualization, chart generation, insight writing, refinement). A single LLM serves as the action-generation engine, and a self-supervised reward combining SQL-verified fact accuracy, structural completeness, chart quality via OCR-based data fidelity, and a novelty penalty guides the search. The authors also introduce MMRBench, a bilingual benchmark of 185 tables and 386 queries with expert-refined reference reports and keypoints. On MMRBench, evaluated by a GPT-4o judge, MCTS-Report over DeepSeek-R1 achieves an overall score of 77.9, surpassing the best baseline (62.7) by 15.2 points and approaching human-level chart-text alignment (88.7 vs. 89.5). Ablations attribute gains to MCTS planning, the self-supervised reward, and the number of rollouts.","tokens_in":11126,"tokens_out":2659,"duration_ms":25136,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it demonstrates that a single LLM with MCTS-style search and self-supervised rewards can improve factual reliability and cross-modal coherence in a task where linear pipelines dominate. The SQL-based fact verification and the separation between internal search guidance and external evaluation are principled strengths, and MMRBench could be a valuable resource for the community. However, the central claim—the +15.2 SOTA gain and the near-human alignment score—rests entirely on a single unvalidated LLM judge. No human-correlation analysis, confidence intervals, or significance tests are provided, and the evaluation lacks a critical best-of-N control that would isolate the contribution of tree search from the mere effect of sampling multiple rollouts. The internal benchmark statistics also contain inconsistencies. These issues must be resolved before the headline result can be accepted.","major_comments":[{"comment":"The central empirical claim (77.9 vs. 62.7, +15.2) is measured exclusively by GPT-4o as a single LLM judge over four dimensions. The paper reports a human baseline on 50 tasks but never reports the correlation between the GPT-4o judge and human ratings, nor does it provide inter-judge agreement, confidence intervals, or significance tests. Given the paper's own acknowledgement that LLM-as-a-judge is vulnerable to reward hacking (Introduction), the absence of any validation of the judge is load-bearing. Please add (a) judge–human correlation on the 50-task subset (and ideally on more tasks), (b) a second judge or permutation-based significance testing, and (c) variance over independent runs or bootstrapped CIs. Without these, the +15.2 improvement may reflect judge self-preference—particularly for MCTS-Report (GPT-4o), where generator and judge share a model family.","section":"Evaluation Protocol / Table 4"},{"comment":"The paper lacks a best-of-N or sample-then-rank baseline, which is essential for isolating the contribution of MCTS. MCTS-Report performs N=10 rollouts and (presumably) selects the best report according to the self-supervised reward; Variant A is a single rollout, and Variant C uses 5 rollouts. The reported gains could therefore stem from selecting the maximum of N samples rather than from tree-structure-aware search. Please include a control that draws N independent samples with the same LLM and selects the one with the highest self-supervised reward (without any tree, UCT, or backpropagation). This control is necessary to support the claim in Finding 1 that 'MCTS-guided search consistently boosts performance across all base models.'","section":"Experiments / Ablation Studies (Figure 2)"},{"comment":"The benchmark statistics are internally inconsistent. The text states '79 tables are in English and 131 are in Chinese' for a total of 185 tables, but 79+131=210. Table 3 reports 'Average tasks per table 1.84', but 386 queries divided by 185 tables equals approximately 2.09. Also, 'Average cells per table 420,000' appears implausibly high for the described real-world tables and is not reconciled with the later statement that 38 tables are 'Extremely large tables (>50K cells)'. These inconsistencies undermine confidence in the curation and reliability of the benchmark. Please correct the numbers and explain any discrepancy between the stated totals and the sum of the stated subsets.","section":"MMRBench Construction / Table 3"},{"comment":"The paper does not specify how the final report is selected after the N=10 MCTS rollouts, nor how the rollout-based 'fast policy' interacts with the reward used to rank reports. This is needed to understand whether the headline numbers are the result of a single run or a selection among rollouts. Additionally, the reward function's r_vis relies on 'OCR + parsing' of rendered charts and r_novel on a cosine-similarity threshold against a template bank; neither component is validated for reliability, and both are used to backpropagate search guidance. If the final report is the argmax of this composite reward, then the evaluation is indirectly optimizing the same (unvalidated) proxy. Please state the selection rule explicitly and, if possible, report the correlation between the self-supervised reward and the GPT-4o judge scores.","section":"Phase 3: Simulation / Final Report Selection"}],"minor_comments":[{"comment":"The abstract says 'MCTS-Report significantly outperforms strong baselines,' but no significance tests are reported anywhere; please temper the wording or add statistical support.","section":"Abstract / Conclusion"},{"comment":"The transition constraints table uses '-' for the root's valid next action; 'a1' appears to be the only valid action. Consider making the root row explicit with 'a1' rather than '-' for clarity.","section":"Table 2"},{"comment":"The ablation results are reported only in a figure; please include the numerical values in a table so that exact scores and differences can be verified against the text.","section":"Figure 2"},{"comment":"A few references lack author names or venue details (e.g., the 'Gemini Deep Research' and 'Qwen3-VL-235B' entries), and the formatting of the 'GPT-4 Technical Report' citation is inconsistent with the rest of the bibliography.","section":"References"},{"comment":"The manuscript text contains several missing spaces and word-joining artifacts (e.g., 'comprisingbothtextualanalysisandvisualchartsfromstruc-turedtablardata'), which hinder readability; a careful proofread is needed.","section":"Page 1 (header)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a publishable core idea, but the empirical validation is not yet at the level the claims require. The lack of a best-of-N control, the unvalidated single LLM judge, and the inconsistent benchmark statistics are all fixable in a revision, in my assessment. I would look favorably on a revised version that provides judge–human correlation, variance estimates, a sampling control, and corrected benchmark numbers. That said, if the authors are unable to provide human-correlation evidence, the central claim should be substantially downgraded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the MCTS-Report paper. The core idea is genuinely sensible: treat multimodal report generation as a search over atomic construction actions, use one LLM as the action generator, and guide the search with a reward that includes SQL-based fact verification. The method description is clear, and the ablation results suggest the search planning does contribute. MMRBench, if released in usable form, fills a real gap: a shared benchmark with tables, charts, and verifiable keypoints.\n\nWhat is new here is the formulation, not the algorithm. MCTS is decades old, and applying it to LLM generation is not itself novel. But the specific action decomposition and the transition constraint matrix are a reasonable contribution, and the self-supervised reward (SQL fact checks, chart fidelity, novelty penalty) is the most interesting part.\n\nThe soft spots are in the evaluation, not the method. The entire empirical claim rests on GPT-4o as a single judge. The paper collects human ratings on 50 tasks but never reports the correlation between judge and humans. Without that, the +15.2 gain over the best baseline could be an artifact of LLM self-preference, especially for the GPT-4o variant where generator and judge share a model family. There are also no confidence intervals, significance tests, or even a statement of how many independent runs were performed.\n\nSeparately, the benchmark statistics do not add up: 79 English + 131 Chinese = 210, not 185 tables; and 386 tasks / 185 tables = 2.09, not the stated 1.84. These are small enough to be typos, but in a benchmark paper they erode confidence in the curation. The related work section also omits prior work on MCTS for LLM planning (Tree of Thoughts, LATS, etc.), which would be the natural comparison and sharpens the novelty claim. That is minor relative to the evaluation issue.\n\nBottom line: this is a serious system paper with a plausible method and a useful benchmark artifact, but the headline result is not yet supported. The authors need to validate the judge against human scores, report variance, fix the benchmark statistics, and release code and data. I would send it to review with a request for major revision, because the core idea is worth circulating and the evaluation can be fixed.","headline":"Plausible MCTS framework and new benchmark, but the headline SOTA gain is unverified until the evaluation is tightened.","tokens_in":11653,"tokens_out":3274,"would_cite":false,"duration_ms":27289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Table-to-report generation is better as a search problem than a fixed pipeline.","keywords":["Monte Carlo Tree Search","multimodal report generation","table-to-report generation","self-supervised reward","chart-text alignment","LLM-as-a-judge","data analysis agents","MMRBench"],"falsifier":"Re-score the 50 human-scored tasks with an independent judge or human panel and compare with GPT-4o scores; if the correlation is weak, or if a different judge shrinks or reverses the +15.2 point lead over DeepSeek-R1, the central claim is an evaluation artifact.","tokens_in":10597,"feed_emoji":"📊","tokens_out":6732,"duration_ms":52326,"temperature":0.7,"pith_summary":"The paper tries to establish that professional multimodal reports from structured tables should be built by search, not by a fixed linear pipeline. It proposes MCTS-Report, which treats a report as a path through a tree of atomic construction actions and uses a single LLM to choose and execute those actions, guided by a reward that verifies numbers against the source tables. On the paper's new MMRBench benchmark, this raises the best baseline score from 62.7 to 77.9 overall, with chart-text alignment nearly matching human experts. A sympathetic reader would care because the result suggests that accuracy and cross-modal consistency in data journalism and business reporting can be improved without hand-crafting multi-agent systems.","feed_headline":"Turning report writing into a search lifts scores by 15 points","feed_subtitle":"A single LLM searching over report-building actions beats fixed pipelines on accuracy, alignment, and novelty.","key_machinery":"The carrying object is the search tree of partial reports: the root is an empty report skeleton, edges are atomic actions (chapter planning, visualization task identification, chart generation, chart modification, insight organization, chapter summary, title optimization, narrative flow, transition design, termination), and each path is a complete report. MCTS explores this tree with UCT selection, LLM-driven expansion and simulation, and backpropagation of a self-supervised reward. That reward is the load-bearing mechanism: SQL checks verify numerical claims, a completeness check enforces chapter structure, chart checks test executable code and data fidelity via OCR, and a cosine-similarity template bank penalizes trivial insights. Because the reward is computed from the tables and rules rather than from a judge model, the search can optimize factual fidelity during generation.","core_discovery":"The central claim is that reformulating table-to-report generation as Monte Carlo Tree Search over partial report states, with one LLM generating actions and a self-supervised reward scoring completed drafts, jointly improves factual accuracy, visual quality, and narrative coherence. On MMRBench, MCTS-Report using DeepSeek-R1 reaches an overall score of 77.9, compared with 62.7 for the strongest baseline, and its chart-text alignment of 88.7 approaches the human expert score of 89.5. Ablations attribute 13.4 points to the search itself and 10.6 points to reward-guided selection. The paper also finds that novelty remains a bottleneck, with 61.3 versus the human 88.5.","pith_inferences":["Because the paper's GPT-4o variant is generated by GPT-4o and judged by GPT-4o, part of the reported gain may be judge self-preference; a multi-judge or human re-scoring of the same outputs is the natural test (an inference, not a paper claim).","The framework implies a compute-quality tradeoff: richer reward signals, such as cross-table join checks or semantic novelty scoring, could further reduce multi-table confusion and paraphrase insights beyond the levels reported here.","Search trajectories produced by MCTS could be distilled into a single-pass generator, potentially delivering search-level accuracy at inference cost closer to direct generation."],"forward_implications":["MCTS improves every base model it is applied to, so search-based planning can be layered on existing report generators without retraining them.","SQL-based fact verification during search cuts numerical hallucination in sampled errors from 38.5% for the best baseline to 24.6%.","Chart-text alignment reaches 88.7, close to the human 89.5, showing cross-modal consistency is largely a search problem.","Increasing search depth from 5 to 10 rollouts raises the overall score from 70.4 to 77.9, so more exploration keeps paying.","Insight novelty remains the weakest dimension (61.3 vs. 88.5 human), so original analysis, not fact checking, is the next frontier."],"supporting_citations":[{"why":"Supplies the Monte Carlo Tree Search algorithm with selection, expansion, simulation, and backpropagation that organizes the report search space.","marker":"Coulom 2007"},{"why":"DeepSeek-R1 is the base model that, wrapped in MCTS, produces the headline 77.9 overall score.","marker":"Guo et al. 2025"},{"why":"GPT-4 is the base of a variant and the single judge model used to score all reports.","marker":"Achiam et al. 2023"},{"why":"Documents reward hacking in LLM-as-a-judge settings, motivating the paper's self-supervised reward separated from evaluation.","marker":"Zheng et al. 2023"},{"why":"Shows reasoning LLM judges can be deceived by high-reward adversarial outputs, supporting the strict separation of evaluation and reward.","marker":"Liu et al. 2026b"},{"why":"T2R-Bench defines the table-to-report task and is the source of the real-world tables used to build MMRBench.","marker":"Zhang et al. 2025"},{"why":"MMDR-Bench is the prior multimodal report benchmark that MMRBench extends by adding table grounding and verifiable keypoints.","marker":"Huang et al. 2026"},{"why":"WikiTableQuestions is the early table benchmark that the task formulation explicitly moves beyond.","marker":"Pasupat and Liang 2015"}],"fun_headline_variants":["Monte Carlo search lifts AI reports 15 points","Tree search for reports: 77.9 score, near-human charts","Search-based drafting beats pipelines by 15 points","AI report generation gets 15-point boost with MCTS","Smarter search, sharper reports: 15-point gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire scoreboard depends on GPT-4o as the judge, and the paper reports human scores on only 50 tasks without showing that judge and human rankings agree.","fun_headline_variants_meta":{"raw":{"variants":["Monte Carlo search lifts AI reports 15 points","Tree search for reports: 77.9 score, near-human charts","Search-based drafting beats pipelines by 15 points","AI report generation gets 15-point boost with MCTS","Smarter search, sharper reports: 15-point gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00091,"raw_usage":{"total_tokens":3919,"prompt_tokens":964,"completion_tokens":2955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2874}},"tokens_in":580,"tokens_out":2955,"duration_ms":18762,"temperature":1.0,"reasoning_tokens":2874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:45:43.836049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the 50 human-scored tasks with an independent judge or human panel and compare with GPT-4o scores; if the correlation is weak, or if a different judge shrinks or reverses the +15.2 point lead over DeepSeek-R1, the central claim is an evaluation artifact.","supporting_citations":[],"review_version":2}