{"id":"13b44bfb-bd10-4605-ba18-31e30234943a","arxiv_id":"2505.00903","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"NeMo-Inspector combines interactive inference, side-by-side and multi-generation comparison, custom Python statistics, and LaTeX/Markdown rendering to help developers clean and improve LLM-generated datasets.","lead":"NeMo-Inspector is an open-source tool for inspecting, annotating, and cleaning LLM-generated synthetic datasets, with built-in inference and multi-generation comparison. The paper demonstrates the tool on two math dataset tasks and reports cleaned data and small accuracy gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 46.99% to 19.51% low-quality reduction is not a valid population estimate because the manual check in Section 4.1 is conditioned on model-incorrect responses and no independent definition of low quality is given.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the quantitative quality reduction and accuracy gains are not rigorously measured. The open-source tool and the linked dataset preparation script make the proposed reproduction test feasible, which supports keeping the verdict conditional rather than moving to reject. The qualitative tool description and the existence of the released artifact are genuine contributions, but the abstract's headline numbers need to be backed by a defined labeling protocol, unbiased sampling, and controlled comparisons. If the test fails, the paper should present these cases as anecdotal demonstrations rather than measured improvements; if it passes, the conditional acceptance can be upgraded. I found no internal inconsistency in the tool's design itself; the concern is about the strength of the empirical evidence supporting the central claims.","tokens_in":6932,"tokens_out":8146,"duration_ms":87394,"concrete_test":"Reproduce the cleaning pipeline from the NeMo-Skills prepare.py link given in Section 4.1 on the public GSM-Plus dataset. Have two annotators, blind to source, label a random sample of at least 100 problems per perturbation type from the original and cleaned datasets, using a pre-specified rubric for low quality (wrong answer key, ambiguous wording, extra or missing questions). Compare the estimated rates with confidence intervals to 46.99% and 19.51%. Also compute Llama-3-70B-Instruct accuracy on both versions under the decoding settings in Section 3.1.1; if accuracy improves only because hard problems were removed, the drop in the low-quality rate does not demonstrate improved data quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (abstract, Section 4.1) is that NeMo-Inspector reduced low-quality GSM-Plus samples from 46.99% to 19.51%. The supporting evidence does not establish this. No definition of 'low-quality' is provided, and the stated manual check samples 'ten random samples per perturbation category where the model generated incorrect responses.' This is a conditional sample: it cannot estimate the prevalence of defective samples in the full dataset. If the 46.99% figure is instead the model's error rate, the decrease is partly mechanical—removing problems the model finds hard, even when those problems are valid, will lower the error rate on the remaining set. The persistence statistic in Listing 1 measures answer stability across random seeds, not dataset correctness; a hard but valid problem can have high persistence if the model consistently produces the same wrong answer. The downstream claims in Section 4.2 have a similar attribution gap: the 1.92% MATH and 4.17% GSM8K gains are reported with no error bars, no statement of absolute versus relative improvement, and no controlled comparison isolating the identified fix as the only changed variable. The dangling reference at the end of the conclusion is an editing artifact and does not affect this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NeMo-Inspector, an open-source visualization tool for inspecting and curating LLM-generated datasets. The tool combines an Inference page for interactive prompt design with an Analyze page supporting intra-sample, comparative, and inter-parameter views, custom Python statistics, manual editing and labeling, and Markdown/LaTeX rendering. The authors demonstrate the tool through two application cases: cleaning a synthetically generated GSM-Plus dataset, where they report reducing low-quality samples from 46.99% to 19.51%, and error analysis of OpenMath models, where fixes led to reported accuracy gains of 1.92% on MATH and 4.17% on GSM8K. The paper is primarily an engineering contribution with illustrative case studies.","tokens_in":7111,"tokens_out":3679,"duration_ms":36294,"significance":"If adopted as described, the tool addresses a real gap in LLM dataset analysis: no existing open tool combines integrated inference, multi-generation comparison, custom statistics, manual editing, and flexible rendering in one workflow. The open-source release under Apache 2.0 and the concrete workflow descriptions, such as detecting double question marks and comparing original versus perturbed samples, are valuable and give the paper practical relevance. The central qualitative claim, that NeMo-Inspector facilitates dataset exploration, is plausible and supported by the case-study narratives. However, the quantitative claims in Section 4 are not rigorously supported and should be treated as anecdotal illustrations unless the evidence is strengthened.","major_comments":[{"comment":"The headline reduction from 46.99% to 19.51% is not established by the evidence. The manual assessment samples only incorrect-response cases ('ten random samples per perturbation category where the model generated incorrect responses'), so it cannot provide a population-level estimate of low-quality prevalence in the full dataset; moreover, 'low-quality' is never defined independently of the model's error behavior. If the 46.99% figure is the model's error rate, then removing hard-but-valid samples would mechanically lower the measured rate, making the claimed quality improvement circular. The authors should either provide an independent quality definition with an unbiased sampling protocol, or reframe these numbers as exploratory observations.","section":"Section 4.1"},{"comment":"The accuracy improvements of 1.92% on MATH and 4.17% on GSM8K are reported without error bars, without clarification of absolute versus relative improvement, and without a controlled comparison in which the identified fix is the only changed variable. The attribution of these gains to the tool's error analysis is therefore not demonstrated; the gains could stem from fine-tuning noise or from changes in the training data unrelated to the identified patterns. The authors should either provide a proper ablation and multiple-run statistics, or soften the causal claim.","section":"Section 4.2"},{"comment":"The persistence statistic measures answer stability across random seeds, not dataset correctness. A valid but hard problem can have high persistence when the model consistently produces the same wrong answer, and a defective sample can have low persistence if the model produces different wrong answers. Using persistence as evidence of low-quality samples is not justified without validation against an independent quality definition, which weakens the Section 4.1 cleaning workflow.","section":"Section 3.1.2, Listing 1"}],"minor_comments":[{"comment":"The conclusion ends with a dangling citation '(Ando and Zhang, 2005)' that is unrelated to the sentence it follows; it should be removed or substantiated.","section":"Section 5"},{"comment":"The first sentence of the Analyze-page description contains a typo: 'andthe Analyze page' should read 'and the Analyze page'.","section":"Section 2.2"},{"comment":"The sentence 'if the model generates the answer A1 in n1 run' should be pluralized ('in n1 runs'), and the definition of persistence could benefit from a concrete example if it is intended as a quality statistic.","section":"Section 3.1.2"},{"comment":"The Vieta's formula example describes a cubic polynomial but states that the equation 'is a quadratic equation'; this should be 'cubic equation'.","section":"Section 3.3.2"},{"comment":"The claim 'less than 5% in each category' is not backed by the per-category percentages shown anywhere in the paper; a small table or figure would make this claim checkable.","section":"Section 4.1"},{"comment":"The abstract and Section 4.2 refer to 'OpenMath models' while Section 3.3 uses the specific name 'OpenMath-Mistral-7B-v0.1'; the naming should be made consistent throughout.","section":"Abstract and Section 4.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NeMo-Inspector is a legitimate piece of engineering: an open-source tool that combines streamlined inference, homogeneous and heterogeneous multi-generation analysis, custom Python statistics, and LaTeX/Markdown rendering in one interface. If you work on synthetic data cleaning or LLM error analysis, it's worth a look, and the Apache-2.0 release makes the underlying functionality independently checkable. The paper is clearly written and the three usage scenarios (quality check, annotation, prompt adjustment) map onto real workflows.\n\nThe tool's novelty is the combination, not any single feature. The comparison with Lilac, KNIME, LLM-Attributor, and LLM-Comparator is fair, and the homogeneous multi-generation aggregation with custom functions is something I haven't seen in those tools. The first case study on GSM-Plus shows how persistence and error-rate sorting can surface problematic questions, and the second shows how code-execution and arithmetic errors can be identified and addressed.\n\nThe soft spot is the quantitative evaluation. The 46.99% to 19.51% reduction in 'low-quality' samples is presented as a headline result, but the methods don't support it as a population estimate. The manual check samples only model-incorrect responses, so it can't tell you the full dataset's defect rate. There's also no independent definition of 'low-quality'—the persistence statistic measures answer stability, which is not the same as correctness. And in Section 4.2, the accuracy gains (1.92% and 4.17%) are reported without error bars, without a clear statement of absolute vs. relative improvement, and without a controlled comparison isolating the dataset fix as the only change. These problems are real, but they're the kind you'd expect in a tool paper's evaluation section, and they don't invalidate the tool's usefulness. The dangling (Ando & Zhang, 2005) reference in the conclusion is minor sloppiness.\n\nMy take: the paper deserves a serious referee, but the authors should be asked to either soften the quantitative claims or add a proper experimental appendix with error bars, a definition of the quality metric, and a controlled setup. As it stands, the case studies are illustrative, not evidence of a precise effect size. I'd bring it to a reading group that cares about data quality tooling, and I'd cite it if I were writing about practical LLM debugging.","headline":"Useful open-source tool for LLM dataset inspection, but the headline quality-reduction numbers are not statistically backed and should be treated as illustrative.","tokens_in":7673,"tokens_out":2696,"would_cite":true,"duration_ms":25511,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A visualization tool for LLM outputs uses a persistence statistic to flag defective synthetic samples, and the fixes it surfaced improved math benchmarks by up to 4.17 points.","keywords":["LLM generation analysis","synthetic data quality","data visualization","custom statistics","persistence","math reasoning benchmarks","prompt engineering","multi-seed generation"],"falsifier":"Have independent annotators, who have not seen the tool's labels, mark the same original GSM-Plus samples as low-quality or not; if their estimate differs from the paper's 46.99% by more than about five percentage points, the headline quality reduction is not a stable measurement. Alternatively, retrain the same fine-tuned model on data with the two error fixes reverted; if MATH and GSM8K accuracy stay within noise, the reported gains are not caused by the tool's error identification.","tokens_in":6699,"feed_emoji":"📊","tokens_out":10835,"duration_ms":99177,"temperature":0.7,"pith_summary":"This paper introduces NeMo-Inspector, an open-source visualization tool for inspecting LLM outputs on synthetic datasets, and claims it makes the quality-control loop practical. The tool groups repeated generations from the same model into homogeneous sets, computes custom aggregate statistics such as 'persistence'—the maximum number of identical answers across seeds—and lets users sort, label, edit, compare, and rerun inference from one interface. In two case studies the authors report that tool-guided cleaning lowered low-quality samples in the GSM-Plus dataset from 46.99% to 19.51%, and that fixing errors the tool exposed raised math accuracy by 1.92 percentage points on MATH and 4.17 percentage points on GSM8K for a fine-tuned model. If the reported numbers hold, the consequence is that a well-designed inspection interface, not a more powerful generator, can be the bottleneck-breaking step in synthetic-data quality.","feed_headline":"Tool cuts bad synthetic math data from 47% to 19.5%","feed_subtitle":"Multi-seed inspection flags defective synthetic samples; the revealed fixes lift MATH and GSM8K accuracy.","key_machinery":"The load-bearing object is the Analyze page's inter-parameter analysis view, backed by custom Python statistics. The paper's example statistic is 'persistence': across many random-seed generations of the same prompt, count how often the most common identical answer appears. A high-persistence wrong answer marks a question whose signal is likely a bad expected answer or a misleading prompt, rather than sampling luck. Around this statistic the tool organizes three perspectives—intra-sample inspection with rendered Markdown/LaTeX, comparative analysis of outputs across models or prompts, and inter-parameter aggregation such as majority voting and correctness ratios—while the inference page closes the loop by letting the user test a prompt hypothesis on a single failing question immediately after inspecting it.","core_discovery":"The central claim is that a single tool can cover the full loop of synthetic-data quality work—generating multiple answers per prompt, aggregating them into interpretable statistics, sorting to the suspicious cases, manually editing or filtering, and immediately testing prompt changes by running inference—and that this loop produces measurable gains. The authors define 'persistence' as the count of the most frequent identical answer across generations and use it, together with accuracy and side-by-side comparisons, to flag questions whose expected answers are wrong or whose wording is ambiguous, such as prompts containing two question marks. On GSM-Plus, filtering with these signals reduced the fraction of low-quality samples from 46.99% to 19.51%; on math instruction data, the same workflow surfaced two recurring error classes—code-execution errors and arithmetic mistakes—and the resulting fixes improved accuracy by 1.92% on MATH and 4.17% on GSM8K.","pith_inferences":["The persistence statistic could be automated into a one-number quality score for any multi-generation dataset, turning a human-in-the-loop inspection aid into a scalable filter for data curation.","The paper's 46.99% to 19.51% reduction was measured by the same authors using a manual ten-sample-per-category check; an independent, blinded relabeling would clarify how much of the drop is real defect removal and how much is label noise.","A natural testable extension is to apply persistence-guided filtering to a non-math task, such as code generation or instruction following, and compare downstream accuracy against equal-size random filtering to isolate the tool's contribution.","High-persistence wrong answers also suggest a curriculum signal: training on low-persistence, high-accuracy examples first may stabilize learning before hard or defective examples are introduced."],"forward_implications":["Synthetic-data teams can move from random auditing to targeted auditing: generate multiple seeds, rank questions by persistence and accuracy, and inspect only the top suspicious cases.","Systematic artifacts like double question marks or mismatched expected answers can be filtered out of augmented benchmarks without retraining, which may change published robustness numbers.","The code-execution retry and arithmetic-decomposition fixes are cheap, model-agnostic interventions that any code-augmented math generator can adopt.","Because the tool accepts any JSONL text dataset, the same inspect–label–edit–infer loop transfers to non-math domains where formatting or execution errors dominate."],"supporting_citations":[{"why":"Supplies the GSM-Plus synthetic dataset being cleaned and defines the perturbation categories used in the quality analysis.","marker":"(Li et al., 2024)"},{"why":"Defines GSM8K, the human-written source of GSM-Plus problems and one of the two evaluation benchmarks.","marker":"(Cobbe et al., 2021)"},{"why":"Provides the MATH benchmark on which the tool-guided fixes are reported to improve accuracy by 1.92%.","marker":"(Hendrycks et al., 2021)"},{"why":"Introduces the OpenMath models and the OpenMathInstruct dataset; its in-depth error examples ground the code-execution and calculation error taxonomy used in Section 4.2.","marker":"(Toshniwal et al., 2024)"},{"why":"Documents the Nemotron-4 340B generator whose outputs, after fine-tuning, form the data that the tool's fixes improve.","marker":"(Adler et al., 2024)"},{"why":"Specifies the Llama-3 model used for the multi-seed GSM-Plus generations that feed the persistence statistic.","marker":"(AI@Meta, 2024)"},{"why":"Supplies the few-shot instruction-following setup used to generate the solutions analyzed for GSM-Plus quality.","marker":"(Chung et al., 2022)"}],"fun_headline_variants":["NeMo-Inspector: Tool cuts bad synthetic math data by 58%","AI tool flags defective synthetic samples, boosts math accuracy","Slash defective synthetic data from 47% to 19.5% with NeMo-Inspector","Inspect LLM synthetic data: fix errors, boost MATH and GSM8K","Open-source tool cleans synthetic data, lifts LLM math accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that high-persistence wrong answers reliably indicate defective prompts or expected answers, and that manually reviewing a handful of flagged cases per category is enough to know which fixes actually improve dataset quality.","fun_headline_variants_meta":{"raw":{"variants":["NeMo-Inspector: Tool cuts bad synthetic math data by 58%","AI tool flags defective synthetic samples, boosts math accuracy","Slash defective synthetic data from 47% to 19.5% with NeMo-Inspector","Inspect LLM synthetic data: fix errors, boost MATH and GSM8K","Open-source tool cleans synthetic data, lifts LLM math accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3588,"prompt_tokens":930,"completion_tokens":2658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2557}},"tokens_in":546,"tokens_out":2658,"duration_ms":19380,"temperature":1.0,"reasoning_tokens":2557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:31:09.967004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators, who have not seen the tool's labels, mark the same original GSM-Plus samples as low-quality or not; if their estimate differs from the paper's 46.99% by more than about five percentage points, the headline quality reduction is not a stable measurement. Alternatively, retrain the same fine-tuned model on data with the two error fixes reverted; if MATH and GSM8K accuracy stay within noise, the reported gains are not caused by the tool's error identification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Specifies the Llama-3 model used for the multi-seed GSM-Plus generations that feed the persistence statistic."}],"review_version":1}