{"id":"7d856be9-c781-43ec-8fc7-a309de710f37","arxiv_id":"2508.11715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A generated benchmark for Excel runtime-error repair and a baseline LLM evaluation, but with weak human-LLM judge agreement.","lead":"The paper introduces FoRepBench, a synthetic benchmark of 618 Excel formula repair cases spanning five runtime error types, generated from 59 forum-seeded examples via LLM prompting and automated validation. It also evaluates four LLMs on a context-aware repair task, reporting up to 80% output-match accuracy with GPT-4.1.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-as-a-Judge acceptance is the load-bearing risk: kappa 0.42 vs humans, admitted unrealistic samples, and an internal kappa inconsistency (0.42 vs 0.25) mean dataset validity is unproven.","rationale":"The reader identified the same weakest assumption: the LLM-as-a-judge validation is the load-bearing point. The paper's own reported low agreement (kappa 0.42) and admitted unrealistic samples directly undermine the claim that FoRepBench is a high-quality, reusable benchmark. The internal inconsistency between Table 4 (0.42) and Section 7 (0.25) further suggests the validation evidence is shaky. I agree that the dataset's validity is the crux: if the validator is too permissive, then every downstream evaluation on FoRepBench is potentially measuring performance on a polluted test set, including the headline 0.80 execution-match for GPT-4.1. The paper is transparent about this limitation and proposes future improvements, so the current evidence does not justify outright rejection, but it does not yet support full acceptance either. The proposed concrete test—a larger, independent human audit—would directly quantify the risk and could resolve whether the dataset is sound or requires substantial filtering. Given the reader's CONDITIONAL verdict, my concern does not change that verdict, so I recommend UNCHANGED.","tokens_in":12790,"tokens_out":3727,"duration_ms":39172,"concrete_test":"Take a random sample of 100 accepted FoRepBench samples that were not part of the original 24-sample annotation. Have three Excel experts independently label each sample as valid/invalid using the same criteria as Section 3.1.2 (correct formula satisfies intent, faulty formula yields the stated error, table is realistic). Compute the proportion of samples judged invalid by a majority, and compute pairwise kappa between experts and the LLM Validator. Also re-check the kappa inconsistency by re-running the original annotation or verifying the reported values. If the majority-invalid proportion exceeds 10%, or if expert-LLM kappa is below 0.5 on this larger sample, the dataset quality claim and all execution-match results on it are not trustworthy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution of FoRepBench is a high-quality, realistic benchmark for Excel formula repair. The gating step that determines whether a generated sample becomes part of the dataset is the LLM Validator (Section 3.2.3). If this validator is over-permissive, the benchmark contains invalid or unrealistic samples and the reported Execution Match numbers (e.g., 0.80 for GPT-4.1) are not measuring genuine repair ability. The paper's own evidence raises this exact concern: pairwise agreement between each human annotator and the LLM Validator is only Cohen's kappa = 0.42 (Table 4), and Section 7 even states a different value (0.25), an internal inconsistency. Section 6.2 admits that \"some unrealistic samples may persist\" and gives Figure 8, where the validator accepts a table with the string \"Three\" in a numeric Unit Price column, which both annotators reject. Since only 24 synthetic samples were human-annotated (Section 5), the true proportion of flawed samples among the 618 accepted ones is unknown. If, say, 20% of the samples have implausible contexts or incorrect ground-truth formulas, the dataset's validity as an evaluation resource is compromised, and the 0.80 execution-match headline becomes an artifact of a polluted test set. This is load-bearing because the paper's main deliverable is the dataset itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FoRepBench, a benchmark dataset of 618 synthetic Excel formula repair examples covering five runtime error types (#DIV/0!, #N/A, #NAME?, #REF!, #VALUE!). Each sample includes spreadsheet context, a faulty formula, a corrected formula, and a user utterance. The construction pipeline starts from 59 manually curated seed samples scraped from the MrExcel forum, generates new samples via one-shot GPT-4o prompting, filters with Calc.ts execution checks, and validates semantic quality with a chain-of-thought LLM-as-a-Judge (the \"LLM Validator\"). The paper also proposes a context-aware baseline repair technique that feeds the LLM the nearest table rows, faulty formula, error type, and optional utterance, and evaluates GPT-4.1, GPT-4o, Phi-3, and Mistral using syntax validity, can-execute, and execution-match metrics. On FoRepBench, execution match reaches 0.80 for GPT-4.1, with substantially lower scores on the seed dataset. The paper reports human annotation quality on a 24-sample subset, agreement statistics, function/error distributions, and cost estimates.","tokens_in":13218,"tokens_out":3282,"duration_ms":39915,"significance":"If the dataset is indeed high quality, FoRepBench would be a useful resource for a relatively under-served task: context-aware repair of semantic runtime errors in Excel formulas. The authors have released the dataset, and the use of Calc.ts execution checks provides an objective grounding for the claim that faulty formulas produce the intended error type and corrected formulas execute without error. The cost analysis and pipeline description are also valuable for reproducibility. However, the central validity claim rests on the LLM Validator's ability to filter out semantically implausible or incorrect samples. The paper's own evidence shows only moderate human-LLM agreement (kappa 0.42, and inconsistently reported as 0.25 in Section 7), with the validator accepting examples that human annotators reject as unrealistic. With only 24 human-annotated samples, none drawn from the final accepted set, the benchmark's quality is not yet established. The self-consistency confound—GPT-4o generates the samples, the LLM Validator (from the same model family) filters them, and GPT-4o/GPT-4.1 are evaluated on them—further weakens the interpretation of the headline execution-match numbers. Thes","major_comments":[{"comment":"The paper reports two different Cohen's kappa values for human-LLM Validator agreement: 0.42 in Section 6.2 and Table 4, and 0.25 in Section 7. This internal inconsistency must be resolved. Whichever value is correct, the agreement is at best moderate, and Section 7's lower value would indicate only fair agreement. Since the LLM Validator is the sole quality gate for the final dataset, this is load-bearing: the claim that FoRepBench contains \"high-quality samples\" is not supported if the validator's judgments diverge substantially from human judgments. Please provide the correct value, the confusion matrix, and a per-criterion breakdown (error resolution, intent alignment, realism), and discuss how the validator's biases affect dataset composition.","section":"Section 6.2 / Section 7 / Table 4"},{"comment":"The human quality assessment is based on only 24 samples annotated by two of the authors/teammates, and these samples were drawn before the LLM Validator filter, not from the final 618-sample benchmark. With n=24, the confidence interval for the estimated contamination rate is very wide, and the two annotators themselves only reach kappa 0.60. The paper admits in Section 6.2 that \"some unrealistic samples may persist in the dataset,\" and Figure 8 gives a concrete accepted example that both annotators reject. To establish the benchmark's validity, the authors should sample from the final accepted set and report the proportion of samples that human annotators judge as semantically correct and realistic, ideally with annotators who are not co-authors and with a larger sample size.","section":"Section 5 / Section 6.2"},{"comment":"There is a circularity risk that is not addressed: GPT-4o generates the candidate samples, the LLM Validator (the same model family) filters them, and GPT-4o and GPT-4.1 are then evaluated on the resulting set. The ground-truth correct formulas are also GPT-4o outputs. Consequently, the high execution-match rates on FoRepBench (0.73 and 0.80) may reflect self-consistency with GPT-4-style repair patterns rather than general repair ability. The substantially lower scores on the human-curated seed dataset (0.35 and 0.41) support this concern. I am not claiming the results are meaningless, but the paper should include a concrete analysis separating these effects: for example, evaluate a non-GPT model on samples where the validator agreed with humans versus where it disagreed, or measure how often GPT-4o's repair is exactly the ground-truth formula versus a different but equivalent formula.","section":"Section 3.2.3 / Table 5 / Section 7"},{"comment":"The large gap between FoRepBench and seed dataset performance (e.g., GPT-4.1 execution match 0.80 vs. 0.41) is discussed as a complexity difference, but it also means the synthetic benchmark is much easier than the real-world seed data. The paper claims FoRepBench is suitable for training and evaluating repair models for realistic runtime errors, yet the RQ1 analysis (Figures 5 and 6) shows a skew toward \"easy\" samples, and Section 7 acknowledges that synthetic examples \"lack the semantic complexity of real-world formulas.\" This is not a fatal flaw, but the claim that execution match on FoRepBench approximates \"real-life formula repair scenarios\" (Section 4) is overstated. Please qualify the claim and provide a more direct comparison of complexity (e.g., nesting depth, edit distance, validator-assigned difficulty) between the two datasets, or adjust the conclusions accordingly.","section":"Section 6.3 / Table 5"},{"comment":"The paper claims FoRepBench is \"the first large-scale dataset of Excel formula repair examples for runtime errors.\" The related-work section mentions RING, FLAME, and Singh et al.'s synthetic formula validation, but it does not explicitly state whether these systems release or rely on any existing repair datasets (syntactic or otherwise). To make the novelty claim precise, please name the closest existing datasets and state what runtime-error repair data, if any, they contain. This is relevant to the contribution's significance.","section":"Section 3.1 / Section 2.2"}],"minor_comments":[{"comment":"The text says \"as illustrated in Figure 3\" when contrasting a synthetic example; based on the surrounding discussion, it likely means Figure 4. Please verify all figure cross-references.","section":"Section 6.3"},{"comment":"The function column \"AVRG\" appears in FoRepBench's table. This may be a legitimate function in some locales, but it is not standard Excel. If it is a typo for AVERAGE, correct it; if it is intentional, explain why a formula with a non-standard function is considered valid.","section":"Table 3"},{"comment":"The number of seed samples is given as 59 in Table 2, but the text does not explicitly state this until later. Please state the seed dataset size in Section 3.1.","section":"Section 5"},{"comment":"Minor capitalization inconsistencies: \"Gpt-4.1\" and \"Gpt-4o\" appear in Section 5 and Table 5; use consistent \"GPT-4.1\" and \"GPT-4o\".","section":"Abstract / Section 4"},{"comment":"The LLM Validator is described as using chain-of-thought reasoning, but no example of the prompt or the exact acceptance criterion is provided. Adding the prompt template in an appendix would improve reproducibility.","section":"Section 3.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable contribution to a benchmark-oriented venue, and the pipeline is clearly described with execution checks and released data. My main concern is that the dataset's validity is not yet demonstrated: the LLM Validator's agreement with humans is low-to-moderate, the human sample is tiny and not drawn from the final set, and the benchmark creation/evaluation loop uses the same model family, which inflates the reported repair accuracy. These concerns are addressable with additional experiments and reporting, so I recommend major revision rather than rejection. I would also suggest the editors ask the authors to clarify the provenance of the seed dataset and whether any existing dataset (e.g., from RING or FLAME) already covers runtime errors, as this affects the novelty claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to have a look at this one. The paper delivers what it says: a 618-sample benchmark (FoRepBench) for Excel runtime-error repair, each sample with spreadsheet context, faulty/correct formulas, and a user utterance. The pipeline is clearly described—seed curation from MrExcel, one-shot generation with GPT-4o, Calc.ts execution checks, then an LLM-as-a-judge filter—and the dataset is public. That is a legitimate contribution. They also run a sensible context-aware baseline across four models and report execution-match metrics, not just string overlap. The results on FoRepBench (up to 0.80 for GPT-4.1) are plausible, and the sharp drop on the harder seed dataset (0.41) is a good sanity check; it shows the synthetic set is easier, which they admit.\n\nThe soft spot is the one you'd expect: the LLM Validator is the gatekeeper, and its agreement with humans is weak. Table 4 says kappa 0.42 for annotator-vs-validator, but Section 7 says 0.25—that inconsistency needs fixing. Only 24 synthetic samples were human-annotated, so we don't know how many of the 618 accepted samples are unrealistic or have wrong ground truths. Their own Figure 8 shows the validator accepting a table with 'Three' in a numeric column, which both annotators reject. That's exactly the kind of implausible-context failure that matters for a benchmark meant to be realistic. Add the same-model bias—GPT-4o generates, validates, and is also evaluated—and the 0.80 number is partly self-consistency. They don't hide this; Section 6.2 and 7 are upfront about it, which I credit.\n\nIs it fatal? Not in my view. The benchmark is still usable, especially for comparing models or for training data, and the execution checks catch the clear-cut cases. The flaws are real but addressable: a stricter judge, more human annotation, and a held-out manually verified test set. The paper deserves a serious referee. I'd send it to review with a request for clarification on the kappa inconsistency and a more guarded interpretation of the headline number. If I were working on spreadsheet repair, I'd cite it and use the dataset. Not a home run, but a solid, honest contribution.","headline":"FoRepBench is a real, reproducible dataset for Excel formula repair with an honest evaluation, but the LLM-judge validation is the soft spot—worth refereeing, not desk rejecting.","tokens_in":13661,"tokens_out":1555,"would_cite":true,"duration_ms":15851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces FoRepBench, the first large-scale dataset of Excel formula repair examples for runtime errors, plus a synthetic generation pipeline and a context-aware LLM baseline that reaches up to 0.80 execution match.","keywords":["Excel formulas","formula repair","synthetic data generation","large language models","runtime errors","benchmark dataset","LLM-as-a-judge","spreadsheet automation"],"falsifier":"Have two spreadsheet experts independently audit a random sample of, say, 100 of the 618 released FoRepBench examples, flagging any with implausible table semantics (e.g., text in numeric columns) or where the 'correct' formula does not actually satisfy the utterance. If a substantial proportion (e.g., >20%) are flagged, then the claim that the benchmark consists of high-quality, realistic repair examples is falsified.","tokens_in":12761,"feed_emoji":"📊","tokens_out":5596,"duration_ms":53268,"temperature":0.7,"pith_summary":"This paper aims to fill a gap in automated Excel formula repair: there has been no large-scale benchmark for semantic runtime errors such as #DIV/0!, #VALUE!, and #N/A. The authors build FoRepBench, a dataset of 618 formula-repair examples, each with spreadsheet context, faulty formula, corrected formula, and a user utterance. They create it by bootstrapping from 59 manually curated forum examples, using few-shot LLM prompting to synthesize new cases, and filtering through execution checks plus an LLM-as-a-judge validation. They also propose a context-aware baseline repair technique and show that GPT-4.1 achieves an execution-match rate of 0.80 on the benchmark, while noting the synthetic data is easier than real-world seed data. The contribution is a reusable resource and a scalable methodology that could extend to other low-resource formula or script languages.","feed_headline":"Top LLM fixes 80% of Excel formula errors on new benchmark","feed_subtitle":"FoRepBench gives researchers 618 realistic repair cases and a scalable way to build more.","key_machinery":"The central object is FoRepBench, a benchmark dataset where each sample couples a faulty formula, a corrected formula, spreadsheet context (cell values and headers), and a natural-language utterance. The key mechanism that carries the argument is the Bootstrap Generator: a few-shot prompting step that generates new samples from curated seed examples, followed by a two-stage validator—execution checks via Calc.ts and an LLM-as-a-judge (LLM Validator) using chain-of-thought reasoning—to filter for correctness and semantic fidelity. The repair baseline uses a single LLM call with a structured prompt that includes nearby table context.","core_discovery":"The central claim is that a small pool of real-world repair examples can be synthetically expanded into a large, validated benchmark for Excel formula repair. The pipeline, called Bootstrap Generator, uses one-shot prompting with each seed sample, followed by two validation stages: execution-based verification with Calc.ts to confirm that the faulty formula produces the intended runtime error and the corrected formula executes cleanly, and a chain-of-thought LLM validator that checks semantic alignment with user intent. The result is FoRepBench, with 618 samples spanning five runtime error types. The authors report that LLMs with spreadsheet context (table headers and sample rows) can repair","pith_inferences":["The pipeline could be adapted to generate repair benchmarks for formulas in Google Sheets or other end-user programming environments that share the event-driven, layout-sensitive structure of Excel.","One testable extension: introducing an explicit difficulty target (e.g., minimum nesting depth or edit distance between faulty and correct formulas) during prompting could close the simplicity gap between synthetic and real data.","The dependence on a single forum (MrExcel) as the seed source may bias the dataset toward the error patterns and functions discussed there; using other forums and support channels would diversify the seed distribution.","If the LLM validator is used to filter training data, its bias toward accepting logically consistent but contextually implausible tables could propagate into fine-tuned repair models; a data-valuation step that measures downstream repair performance per sample could mitigate this."],"forward_implications":["If correct, FoRepBench provides the first common testbed for training and evaluating Excel formula repair models that target runtime errors rather than syntax errors.","The generation pipeline's low cost (about $0.02 per accepted sample) makes it practical to produce similar benchmarks for other low-resource formula languages or script dialects.","The reported gap between synthetic and seed data suggests that current synthetic generation underrepresents deep-nesting, multi-edit repairs, guiding future data-generation work toward harder examples.","Context-aware repair (feeding table headers and sample rows) appears to materially help LLMs, since the strongest model reaches 0.80 execution match.","The moderate human–judge agreement (kappa 0.42) implies that an LLM-based validator alone is not a sufficient quality gate; human-in-the-loop or stricter plausibility checks are needed."],"supporting_citations":[{"why":"LaMirage: prior neurosymbolic repair for low-code formula languages, which the paper positions as syntax-focused and context-free—the gap FoRepBench addresses.","marker":"[3]"},{"why":"RING: prompting-based repair system that suggests fixes without using spreadsheet context; the paper contrasts its approach with this.","marker":"[4]"},{"why":"FLAME: small language model for spreadsheet formulas, operating solely on formula syntax; used to motivate adding context and runtime-error focus.","marker":"[15]"},{"why":"Singh et al.: automated validation of synthetic NL-to-formula data; the paper adapts its LLM-validation idea to formula repair.","marker":"[22]"},{"why":"GPT-4o: the LLM used as the generator for synthetic samples and as one of the repair baselines; also the paper's main generation model.","marker":"[13]"}],"fun_headline_variants":["New benchmark: LLMs repair 80% of Excel formula errors","FoRepBench: 618 validated Excel repair cases for LLMs","Synthetic expansion yields Excel formula repair benchmark","Excel repair test set built from LLM-validated samples","Best LLM hits 80% on new Excel formula repair benchmark"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The benchmark's trustworthiness rests on the LLM-as-a-judge validator correctly identifying samples that are both semantically correct and realistic, yet the paper reports only moderate agreement with human annotators (kappa 0.42) and concedes that some unrealistic samples persist after filtering.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark: LLMs repair 80% of Excel formula errors","FoRepBench: 618 validated Excel repair cases for LLMs","Synthetic expansion yields Excel formula repair benchmark","Excel repair test set built from LLM-validated samples","Best LLM hits 80% on new Excel formula repair benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001024,"raw_usage":{"total_tokens":4180,"prompt_tokens":792,"completion_tokens":3388,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":3303}},"tokens_in":536,"tokens_out":3388,"duration_ms":27517,"temperature":1.0,"reasoning_tokens":3303,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:13:13.369319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two spreadsheet experts independently audit a random sample of, say, 100 of the 618 released FoRepBench examples, flagging any with implausible table semantics (e.g., text in numeric columns) or where the 'correct' formula does not actually satisfy the utterance. If a substantial proportion (e.g., >20%) are flagged, then the claim that the benchmark consists of high-quality, realistic repair examples is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RING: prompting-based repair system that suggests fixes without using spreadsheet context; the paper contrasts its approach with this."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FLAME: small language model for spreadsheet formulas, operating solely on formula syntax; used to motivate adding context and runtime-error focus."},{"cited_title":"An Empirical Study of Validating Synthetic Data for Formula Generation","cited_arxiv_id":"2407.10657","evidence_quote":"Singh et al.: automated validation of synthetic NL-to-formula data; the paper adapts its LLM-validation idea to formula repair."}],"review_version":1}