{"id":"007a84c5-25a6-41dc-aade-bc028266505a","arxiv_id":"2511.22884","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"InsightEval re-annotates 100 business-analytics tables with 1,000 expert-reviewed insights and proposes recall/precision F1 plus LLM-judged novelty metrics for evaluating data-analysis agents.","lead":"A research team built a new benchmark, InsightEval, to test how well AI agents discover insights from business data tables, along with new scoring metrics. It is a direct attempt to fix flaws in the existing InsightBench benchmark and to better measure agents' exploration quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1 'better reflects ability' is supported by an unreleased 30-instance comparison with no rater-reliability statistics, and the proposed F1 can be inflated by duplicate insights that a set-based precision would penalize.","rationale":"The reader identified the gold-set/LLM-judgment circularity as the weakest assumption. I agree that is a core risk, but my stress-test sharpens it in two ways. First, the specific evidence for the F1 claim is too thin: 30 instances, no agreement statistics, no inferential test. Second, the F1 definition itself is vulnerable to duplicate inflation, which interacts with the paper's own Finding 1 that agents produce redundant insights. This means even if the gold set were perfect, the proposed F1 may not measure what is claimed. I would not reject the paper because these issues are addressable by public release and additional human validation; the conditional verdict remains appropriate. The concern is not that the authors are wrong, but that the central comparative claims currently outrun the evidence.","tokens_in":19966,"tokens_out":6401,"duration_ms":67107,"concrete_test":"Release the dataset and evaluation code, then recompute Figure 5 on the 30 held-out instances in two ways: (1) per-instance Spearman correlation between G-Eval Insight F1 and the mean of the ten expert scores, with a confidence interval; (2) a one-to-one greedy matching version of precision where each ground-truth insight can be matched at most once, so duplicates do not inflate precision, and compare that F1 against human scores. If the correlation is not significantly positive or the duplicate-insensitive F1 is no closer to human judgment than plain recall, the 'better reflects ability' claim should be withdrawn or substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Insight F1 'can better reflect the agent's insight discovery ability' (Sec. 6.2) rests almost entirely on Figure 5, which compares G-Eval recall and F1 with expert scores on only 30 data points. The paper reports no inter-rater reliability, no per-instance correlation, and no confidence intervals; Appendix C describes expert ratings but no protocol, scale, or blinding. The metric also has a structural weakness: Eqs. 1-3 use expectation over max pairwise similarity without one-to-one matching. If an agent emits many near-identical copies of one correct insight, precision (and therefore F1) stays high, whereas set-based precision would fall. Finding 1 states that agents 'generate the most confidently correct insights while avoiding uncertain or exploratory outputs' and produce 'substantial redundancy', so duplicate inflation is not a corner case. In addition, ground-truth insights were generated with LLM assistance (o3-mini, GPT-4o) and evaluated by LLMs (GPT-3.5-Turbo, Gemini 2.5 Pro), so the F1 comparison may partly measure stylistic similarity to LLM-generated text. Section 8 concedes ground-truth annotations are 'necessarily incomplete'; the novelty metric intended to compensate is itself an unvalidated LLM judgment with an arbitrary G-Eval>5 threshold. The 40-instance quality audit in Table 2 has the same missing reliability evidence. None of this proves the benchmark is bad, but the comparative superiority claims are currently under-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the existing InsightBench benchmark is flawed and introduces InsightEval, a new benchmark of 100 instances and 1,000 insights (10 per table) built through an LLM-assisted pipeline with human verification. It also proposes an evaluation framework with insight recall, precision, an Insight F1 Score, and a novelty metric, and it benchmarks two agent frameworks (Pandas Agent and Agent Poirot) on this dataset. The central claims are that the Insight F1 Score 'can better reflect the agent's insight discovery ability' and that InsightEval provides a 'more comprehensive and deeper assessment' than existing benchmarks.","tokens_in":20341,"tokens_out":5334,"duration_ms":48624,"significance":"If properly validated, the dataset and evaluation framework would be a useful community resource: the error analysis of InsightBench in Table 1 and Figure 1 is concrete and actionable, the six-type taxonomy is a reasonable extension, and the move from a single LLM evaluator to multiple evaluators plus recall/precision/novelty is a step in the right direction. However, the paper's headline comparative claims currently rest on a small human-evaluation sample with no reliability statistics, a metric with a structural duplicate-inflation problem, and an unvalidated novelty threshold. These issues do not disprove the value of the resource, but they do mean that the stated superiority claims are under-supported as written.","major_comments":[{"comment":"The Insight F1 metric is defined through expectations over max pairwise similarity without any one-to-one matching. If an agent emits many near-identical copies of one correct insight, Eq. (2) assigns each duplicate the same high precision score and Eq. (1) is unaffected, so F1 can be inflated while set-based precision would fall. Finding 1 in §6.2 explicitly reports that agents produce 'substantial redundancy,' so this is not a corner case. Since the paper's central claim is that Insight F1 'can better reflect the agent's insight discovery ability,' the metric should either enforce one-to-one matching (e.g., greedy or Hungarian matching) or be supplemented by a set-level precision that penalizes duplicates.","section":"§4.3.1–§4.3.3, Eqs. (1)–(3)"},{"comment":"The key evidence for the claim that Insight F1 aligns better with human judgment is a comparison on only 30 sampled data points. No inter-annotator agreement is reported, no per-instance correlation or confidence intervals are given, and Appendix C does not describe the rating scale, the number of raters per instance, or whether raters were blinded to the condition. The conclusion that F1 is 'closer to Human Evaluation Scores' is based on aggregate means and is not statistically supported. This needs a more rigorous human-evaluation protocol and analysis before the central claim can be accepted.","section":"§6.2, Figure 5, Appendix C"},{"comment":"The novelty metric depends on classifying insights as correct when their G-Eval score 'exceeding 5,' but §6.1.3 states that G-Eval scores are normalized to facilitate comparison with ROUGE-1. If the normalized range is [0,1], the threshold >5 is either impossible or trivially satisfied; if the raw 1–5/1–10 scale is used, the threshold is arbitrary and uncalibrated. In addition, the LLM-majority judgment of novelty is never validated against human novelty judgments. Since novelty is a stated contribution of the paper, this inconsistency and lack of validation are load-bearing.","section":"§4.3.4 and §6.1.3"},{"comment":"Ground-truth insights are generated with LLM assistance (o3-mini and GPT-4o) and evaluated by LLMs (GPT-3.5-Turbo, Gemini 2.5 Pro) plus ROUGE, while the quality audit in Table 2 covers only 40 randomly sampled instances and reports no inter-annotator agreement or confidence intervals. The comparison against InsightBench in Appendix D is reported without the annotation protocol. Section 8 concedes that 'ground-truth annotations are necessarily incomplete.' The paper should explicitly address the risk that its F1 and novelty scores partly measure stylistic similarity to LLM-generated reference text rather than true insight quality—for example, by validating on a fully human-curated subset, ablating evaluator choice, or measuring agreement with expert judgments of insight completeness.","section":"§4.2, Table 2, Appendix D, §8"}],"minor_comments":[{"comment":"The example dataset contains near-duplicate insights: Insight 9 and Insight 10 both state that every caller submitted 125 incidents, with only a slight interpretative difference. This undercuts the de-duplication claim in §4.2.3; please replace the example or revisit the de-duplication procedure.","section":"Table 11"},{"comment":"The variables M, N, and δ are introduced only in prose; define them directly in or below the equation to avoid ambiguity.","section":"Eq. (4)"},{"comment":"The bar chart shows only aggregate values. A per-agent scatter plot or error bars would help readers see variance across the 30 instances.","section":"Figure 5"},{"comment":"No dataset URL or code release is mentioned. For a benchmark paper, an artifact link is essential for reproducibility and community adoption.","section":"General"},{"comment":"ROUGE-1 and G-Eval rank the agents differently (e.g., Pandas Agent (GPT-4o) has the highest ROUGE-1 F1 but is below Agent Poirot (GPT-4o) in G-Eval). A short discussion of what each metric captures would help readers interpret the divergence.","section":"Table 4"},{"comment":"The reported error rates (57.0%, 20.8%, 14.5%, 32.8%, 18.3%) sum to more than 100%. A sentence clarifying that the error types are not mutually exclusive would prevent confusion.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the error analysis of InsightBench and the dataset construction pipeline are potentially valuable contributions, but the comparative superiority claims are not yet supported by the evidence. In particular, the Insight F1 metric's duplicate-inflation issue is a real structural flaw that interacts with the paper's own Finding 1 about redundancy, and the human-evaluation evidence is too thin for the central claim. I would not recommend rejection, but the authors need to strengthen the metric definition and validation before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Start with the punchline: the most useful thing in this paper is the error analysis of InsightBench. The taxonomy — vague goals, phantom column names, irrational insights, duplicates — is concrete and matches what you'd expect if you'd actually run these agents. The curation pipeline that follows (goal refinement, question validation, LLM-assisted insight generation with human checks) is a reasonable fix, and the two new insight types (Evaluative, Exploratory) make sense.\n\nWhat's genuinely new: a re-annotated 100-table / 1000-insight dataset with six types, a novelty scoring protocol, and the explicit attempt to measure precision as well as recall. That's a real step beyond InsightBench's single-evaluator recall setup.\n\nWhere it gets soft: the claim that Insight F1 \"better reflects\" ability is not established. The human comparison is 30 instances, no inter-rater reliability, no protocol details, and Appendix C is one paragraph. The metric itself has a structural flaw: equations 1 and 2 take a max over all pair similarities without one-to-one matching, so an agent that emits five near-copies of one correct insight keeps a high precision and high F1. Finding 1 says agents produce \"substantial redundancy,\" so this isn't a corner case. The ground truth is LLM-assisted (o3-mini) and the evaluation is other LLMs (GPT-3.5, Gemini), so the F1 comparison partly measures stylistic match to LLM-generated text. The novelty metric's G-Eval>5 threshold is arbitrary and also unvalidated. The paper itself concedes in Section 8 that the ground truth is necessarily incomplete.\n\nNone of this is fatal. The error analysis stands on its own, and the dataset could become useful if released. But the comparative superiority claims need more than 40 + 30 human-annotated instances without reliability statistics.\n\nWho this is for: people building or evaluating data-analysis agents. It deserves a serious referee, but not a quick accept — I'd want public release, stronger human validation (including inter-annotator agreement and a real protocol), and a fix or at least a discussion of the duplicate-inflation problem in the F1 computation. I'd recommend sending it to peer review with the expectation of major revision; I'd hold off citing it until the dataset and code are out.","headline":"Useful error analysis and a plausible re-annotated benchmark, but the F1-superiority claim rests on a 30-instance human comparison and a metric that rewards duplicated insights.","tokens_in":20839,"tokens_out":2771,"would_cite":false,"duration_ms":26209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes InsightEval, a curated benchmark of 1,000 insights across six types, and an Insight F1 metric that claim to better measure how LLM-driven data-analysis agents discover insights.","keywords":["insight discovery","benchmark","LLM agents","evaluation metric","data analysis","novelty","data curation"],"falsifier":"If a second, independent expert annotation of a random sample of the same tables orders the agent configurations differently under Insight F1 than the published order, the benchmark's reliability is not established. Concretely, re-annotate 30 tables with a new expert team and check whether the agent F1 ranking matches; if it flips, the metric is measuring the annotation rather than the agents.","tokens_in":19861,"feed_emoji":"🔍","tokens_out":7581,"duration_ms":64078,"temperature":0.7,"pith_summary":"This paper argues that current benchmarks for evaluating LLM-driven data-analysis agents are flawed, and that a new curated dataset and metric are needed. The authors introduce InsightEval, a dataset of 100 tables with 1,000 expert-checked insights spanning six types, built through a four-step pipeline that refines goals, validates questions, generates answers and insights, and synthesizes summaries. They also propose an Insight F1 Score that combines recall and precision, plus a novelty score, to measure whether agents rediscover known insights without generating noise and whether they find unannotated ones. If the paper is right, the field gains a more reliable yardstick for comparing data agents and a sharper picture of what they still fail to do.","feed_headline":"New benchmark and metric sharpen AI agents' insight grading","feed_subtitle":"Curated tables and human-checked insights reveal which data agents find real patterns.","key_machinery":"The central machinery is the data-curation pipeline and the evaluation metric. The pipeline (goal refinement, question validation and generation, answering and insight generation, summary synthesis) enforces three requirements: clearly defined goals, high-quality questions and insights, and multi-perspective automatic evaluation. The evaluation uses iterative matching between ground-truth and agent insights to compute recall and precision, combined into the Insight F1 Score, plus a novelty score that asks three LLMs to judge whether unmatched agent insights are correct and original.","core_discovery":"The paper's central claim is that InsightEval is a higher-quality benchmark than the existing one it audits, and that its Insight F1 Score better reflects an agent's insight-discovery ability than the recall-only evaluation used previously. The audit identifies five error types in the prior benchmark—ambiguous goals, undefined data types, erroneous questions, irrational insights, and redundant insights—and the new dataset corrects them. On this dataset, the authors benchmark two agent frameworks built on three common large language models, finding that precision typically exceeds recall, meaning agents generate confident but repetitive insights rather than exploring broadly; that novelty sco","pith_inferences":["The paper's own limitation note says ground-truth annotations are incomplete; this implies the F1 score could reward agents that paraphrase the annotations rather than find truly better insights, so a stronger test would use multiple independent gold sets.","Because novelty is judged by LLMs, the score is time- and context-dependent; re-scoring the same agent outputs months later could shift rankings as model knowledge evolves.","The curation pipeline's criteria—clear goals, tight question-data alignment, multi-perspective evaluation—could transfer to other domains, like scientific data, but would need domain-expert involvement in annotation.","A testable extension: compare Insight F1 against human rankings on a held-out set of fresh tables to see if the metric generalizes beyond the curated instances."],"forward_implications":["The benchmark gives the field a shared testbed, making future comparisons of data-analysis agents reproducible.","Because precision is included, an agent that dumps many shallow or duplicate insights is penalized, encouraging focused exploration.","The two added insight types (Evaluative and Exploratory) expose capability gaps that the original four types miss.","The finding that agents favor prescriptive and exploratory outputs while missing ground-truth recall suggests current agents need better exploration strategies.","The novelty score rewards discoveries not in the gold set, pushing agents toward genuinely new findings."],"fun_headline_variants":["New benchmark with better metric grades AI data agents' insights","InsightEval: sharper grading for AI insight discovery","Curated benchmark reveals AI agents find confident, repetitive insights","Better insight metric exposes AI agents' exploration gaps","InsightEval: fixing the flaws in AI insight benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's scores assume the annotated ground-truth insights are a complete and unbiased gold set, and that LLM judges can reliably tell which agent outputs match those insights and which are genuinely novel.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark with better metric grades AI data agents' insights","InsightEval: sharper grading for AI insight discovery","Curated benchmark reveals AI agents find confident, repetitive insights","Better insight metric exposes AI agents' exploration gaps","InsightEval: fixing the flaws in AI insight benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2476,"prompt_tokens":687,"completion_tokens":1789,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":1723}},"tokens_in":431,"tokens_out":1789,"duration_ms":10581,"temperature":1.0,"reasoning_tokens":1723,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:37:30.449132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a second, independent expert annotation of a random sample of the same tables orders the agent configurations differently under Insight F1 than the published order, the benchmark's reliability is not established. Concretely, re-annotate 30 tables with a new expert team and check whether the agent F1 ranking matches; if it flips, the metric is measuring the annotation rather than the agents.","supporting_citations":[],"review_version":1}