Pith. sign in

REVIEW 5 major objections 5 minor 30 references

Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces FoRepBench, the first large-scale dataset of Excel formula repair examples for runtime errors, plus a synthetic generation pipeline and a context-aware LLM baseline that reaches up to 0.80 execution match.

desk verdict FoRepBench is a real, reproducible dataset for Excel formula repair with an honest evaluation, but the LLM-judge validation is the soft spot—worth refereeing, not desk rejecting. read the letter →

arxiv 2508.11715 v1 pith:JG76IPAR submitted 2025-08-14 cs.SE cs.AI

Ananya Singha , Harshita Sahijwani , Walt Williams , Emmanuel Aboah Boateng , Nick Hausman , Miguel Di Luca , Keegan Choudhury , Chaya Binet
show 5 more authors
Vu Le Tianwei Chen Oryan Rokeah Chen Sulaiman Vesal Sadid Hasan
This is my paper · ORCID
classification cs.SEcs.AI
keywords ExcelformulasformularepairsyntheticdatagenerationlargelanguagemodelsruntimeerrorsbenchmarkdatasetLLM-as-a-judgespreadsheetautomation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to fill a gap in automated Excel formula repair: there has been no large-scale benchmark for semantic runtime errors such as #DIV/0!, #VALUE!, and #N/A. The authors build FoRepBench, a dataset of 618 formula-repair examples, each with spreadsheet context, faulty formula, corrected formula, and a user utterance. They create it by bootstrapping from 59 manually curated forum examples, using few-shot LLM prompting to synthesize new cases, and filtering through execution checks plus an LLM-as-a-judge validation. They also propose a context-aware baseline repair technique and show that GPT-4.1 achieves an execution-match rate of 0.80 on the benchmark, while noting the synthetic data is easier than real-world seed data. The contribution is a reusable resource and a scalable methodology that could extend to other low-resource formula or script languages.

What carries the argument

The central object is FoRepBench, a benchmark dataset where each sample couples a faulty formula, a corrected formula, spreadsheet context (cell values and headers), and a natural-language utterance. The key mechanism that carries the argument is the Bootstrap Generator: a few-shot prompting step that generates new samples from curated seed examples, followed by a two-stage validator—execution checks via Calc.ts and an LLM-as-a-judge (LLM Validator) using chain-of-thought reasoning—to filter for correctness and semantic fidelity. The repair baseline uses a single LLM call with a structured prompt that includes nearby table context.

What would settle it

Have two spreadsheet experts independently audit a random sample of, say, 100 of the 618 released FoRepBench examples, flagging any with implausible table semantics (e.g., text in numeric columns) or where the 'correct' formula does not actually satisfy the utterance. If a substantial proportion (e.g., >20%) are flagged, then the claim that the benchmark consists of high-quality, realistic repair examples is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a small pool of real-world repair examples can be synthetically expanded into a large, validated benchmark for Excel formula repair. The pipeline, called Bootstrap Generator, uses one-shot prompting with each seed sample, followed by two validation stages: execution-based verification with Calc.ts to confirm that the faulty formula produces the intended runtime error and the corrected formula executes cleanly, and a chain-of-thought LLM validator that checks semantic alignment with user intent. The result is FoRepBench, with 618 samples spanning five runtime error types. The authors report that LLMs with spreadsheet context (table headers and sample rows) can repair

Load-bearing premise

The benchmark's trustworthiness rests on the LLM-as-a-judge validator correctly identifying samples that are both semantically correct and realistic, yet the paper reports only moderate agreement with human annotators (kappa 0.42) and concedes that some unrealistic samples persist after filtering.

Editorial extensions

If this is right

  • If correct, FoRepBench provides the first common testbed for training and evaluating Excel formula repair models that target runtime errors rather than syntax errors.
  • The generation pipeline's low cost (about $0.02 per accepted sample) makes it practical to produce similar benchmarks for other low-resource formula languages or script dialects.
  • The reported gap between synthetic and seed data suggests that current synthetic generation underrepresents deep-nesting, multi-edit repairs, guiding future data-generation work toward harder examples.
  • Context-aware repair (feeding table headers and sample rows) appears to materially help LLMs, since the strongest model reaches 0.80 execution match.
  • The moderate human–judge agreement (kappa 0.42) implies that an LLM-based validator alone is not a sufficient quality gate; human-in-the-loop or stricter plausibility checks are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pipeline could be adapted to generate repair benchmarks for formulas in Google Sheets or other end-user programming environments that share the event-driven, layout-sensitive structure of Excel.
  • One testable extension: introducing an explicit difficulty target (e.g., minimum nesting depth or edit distance between faulty and correct formulas) during prompting could close the simplicity gap between synthetic and real data.
  • The dependence on a single forum (MrExcel) as the seed source may bias the dataset toward the error patterns and functions discussed there; using other forums and support channels would diversify the seed distribution.
  • If the LLM validator is used to filter training data, its bias toward accepting logically consistent but contextually implausible tables could propagate into fine-tuned repair models; a data-valuation step that measures downstream repair performance per sample could mitigate this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces FoRepBench, a benchmark dataset of 618 synthetic Excel formula repair examples covering five runtime error types (#DIV/0!, #N/A, #NAME?, #REF!, #VALUE!). Each sample includes spreadsheet context, a faulty formula, a corrected formula, and a user utterance. The construction pipeline starts from 59 manually curated seed samples scraped from the MrExcel forum, generates new samples via one-shot GPT-4o prompting, filters with Calc.ts execution checks, and validates semantic quality with a chain-of-thought LLM-as-a-Judge (the "LLM Validator"). The paper also proposes a context-aware baseline repair technique that feeds the LLM the nearest table rows, faulty formula, error type, and optional utterance, and evaluates GPT-4.1, GPT-4o, Phi-3, and Mistral using syntax validity, can-execute, and execution-match metrics. On FoRepBench, execution match reaches 0.80 for GPT-4.1, with substantially lower scores on the seed dataset. The paper reports human annotation quality on a 24-sample subset, agreement statistics, function/error distributions, and cost estimates.

Significance. If the dataset is indeed high quality, FoRepBench would be a useful resource for a relatively under-served task: context-aware repair of semantic runtime errors in Excel formulas. The authors have released the dataset, and the use of Calc.ts execution checks provides an objective grounding for the claim that faulty formulas produce the intended error type and corrected formulas execute without error. The cost analysis and pipeline description are also valuable for reproducibility. However, the central validity claim rests on the LLM Validator's ability to filter out semantically implausible or incorrect samples. The paper's own evidence shows only moderate human-LLM agreement (kappa 0.42, and inconsistently reported as 0.25 in Section 7), with the validator accepting examples that human annotators reject as unrealistic. With only 24 human-annotated samples, none drawn from the final accepted set, the benchmark's quality is not yet established. The self-consistency confound—GPT-4o generates the samples, the LLM Validator (from the same model family) filters them, and GPT-4o/GPT-4.1 are evaluated on them—further weakens the interpretation of the headline execution-match numbers. Thes

major comments (5)
  1. [Section 6.2 / Section 7 / Table 4] The paper reports two different Cohen's kappa values for human-LLM Validator agreement: 0.42 in Section 6.2 and Table 4, and 0.25 in Section 7. This internal inconsistency must be resolved. Whichever value is correct, the agreement is at best moderate, and Section 7's lower value would indicate only fair agreement. Since the LLM Validator is the sole quality gate for the final dataset, this is load-bearing: the claim that FoRepBench contains "high-quality samples" is not supported if the validator's judgments diverge substantially from human judgments. Please provide the correct value, the confusion matrix, and a per-criterion breakdown (error resolution, intent alignment, realism), and discuss how the validator's biases affect dataset composition.
  2. [Section 5 / Section 6.2] The human quality assessment is based on only 24 samples annotated by two of the authors/teammates, and these samples were drawn before the LLM Validator filter, not from the final 618-sample benchmark. With n=24, the confidence interval for the estimated contamination rate is very wide, and the two annotators themselves only reach kappa 0.60. The paper admits in Section 6.2 that "some unrealistic samples may persist in the dataset," and Figure 8 gives a concrete accepted example that both annotators reject. To establish the benchmark's validity, the authors should sample from the final accepted set and report the proportion of samples that human annotators judge as semantically correct and realistic, ideally with annotators who are not co-authors and with a larger sample size.
  3. [Section 3.2.3 / Table 5 / Section 7] There is a circularity risk that is not addressed: GPT-4o generates the candidate samples, the LLM Validator (the same model family) filters them, and GPT-4o and GPT-4.1 are then evaluated on the resulting set. The ground-truth correct formulas are also GPT-4o outputs. Consequently, the high execution-match rates on FoRepBench (0.73 and 0.80) may reflect self-consistency with GPT-4-style repair patterns rather than general repair ability. The substantially lower scores on the human-curated seed dataset (0.35 and 0.41) support this concern. I am not claiming the results are meaningless, but the paper should include a concrete analysis separating these effects: for example, evaluate a non-GPT model on samples where the validator agreed with humans versus where it disagreed, or measure how often GPT-4o's repair is exactly the ground-truth formula versus a different but equivalent formula.
  4. [Section 6.3 / Table 5] The large gap between FoRepBench and seed dataset performance (e.g., GPT-4.1 execution match 0.80 vs. 0.41) is discussed as a complexity difference, but it also means the synthetic benchmark is much easier than the real-world seed data. The paper claims FoRepBench is suitable for training and evaluating repair models for realistic runtime errors, yet the RQ1 analysis (Figures 5 and 6) shows a skew toward "easy" samples, and Section 7 acknowledges that synthetic examples "lack the semantic complexity of real-world formulas." This is not a fatal flaw, but the claim that execution match on FoRepBench approximates "real-life formula repair scenarios" (Section 4) is overstated. Please qualify the claim and provide a more direct comparison of complexity (e.g., nesting depth, edit distance, validator-assigned difficulty) between the two datasets, or adjust the conclusions accordingly.
  5. [Section 3.1 / Section 2.2] The paper claims FoRepBench is "the first large-scale dataset of Excel formula repair examples for runtime errors." The related-work section mentions RING, FLAME, and Singh et al.'s synthetic formula validation, but it does not explicitly state whether these systems release or rely on any existing repair datasets (syntactic or otherwise). To make the novelty claim precise, please name the closest existing datasets and state what runtime-error repair data, if any, they contain. This is relevant to the contribution's significance.
minor comments (5)
  1. [Section 6.3] The text says "as illustrated in Figure 3" when contrasting a synthetic example; based on the surrounding discussion, it likely means Figure 4. Please verify all figure cross-references.
  2. [Table 3] The function column "AVRG" appears in FoRepBench's table. This may be a legitimate function in some locales, but it is not standard Excel. If it is a typo for AVERAGE, correct it; if it is intentional, explain why a formula with a non-standard function is considered valid.
  3. [Section 5] The number of seed samples is given as 59 in Table 2, but the text does not explicitly state this until later. Please state the seed dataset size in Section 3.1.
  4. [Abstract / Section 4] Minor capitalization inconsistencies: "Gpt-4.1" and "Gpt-4o" appear in Section 5 and Table 5; use consistent "GPT-4.1" and "GPT-4o".
  5. [Section 3.2.3] The LLM Validator is described as using chain-of-thought reasoning, but no example of the prompt or the exact acceptance criterion is provided. Adding the prompt template in an appendix would improve reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

FoRepBench's ground-truth formulas are GPT-4o outputs filtered by an LLM Validator with weak human agreement, so the reported GPT-4o Execution Match is substantially a self-consistency measure.

  1. self definitional [Section 5 (Experimental Setup) and Section 4.1 (Baseline Repair Technique)]
    "We applied Bootstrap Generator(Section 3.2) to generate our dataset using GPT-4o as the LLM. ... In total, we generated 1095 samples, out of which 618 passed LLM V alidator. ... The repaired formula is subsequently evaluated by comparing it with the ground truth correct formula from the benchmark dataset."

    The 'ground truth correct formula' in each FoRepBench sample is a GPT-4o output, and GPT-4o is then evaluated by comparing its repair to that same GPT-4o-generated correct formula. The only semantic filter is the LLM Validator, whose agreement with human annotators is moderate (kappa 0.42 in Table 4; Section 7 reports 0.25). Thus GPT-4o's Execution Match (0.73) largely measures how well the model reproduces its own generation distribution, not an independently validated repair target. For GPT-4.1 (0.80), the score measures alignment with GPT-4o-authored ground truth, which is still a same-family self-consistency effect rather than an externally established correctness standard.

  2. self definitional [Section 3.2.3 (Validating Generations with LLM-as-a-Judge Approach), Section 6.2 (RQ2), Section 7 (Discussion and Conclusion)]
    "To ensure reliability of the generated synthetic data, the LLM-judge approach we implemented leverages Chain-of-Though (CoT) reasoning for systematic assessment. We refer to this model as LLM V alidator. ... some unrealistic samples may persist in the dataset despite passing automated filtering."

    The dataset's 'high-quality' status is operationalized as acceptance by the LLM Validator: samples that passed both correctness and quality checks were added to the final dataset. The abstract then claims '618 high-quality samples.' But the only human check covers 24 samples and shows only moderate agreement (kappa 0.42 in Table 4, contradicted by 0.25 in Section 7). The paper even admits unrealistic samples persist. The central quality claim therefore reduces to the validator's own judgment, with no reliable independent ground truth to confirm the benchmark's validity.

full rationale

The paper's main contribution is a benchmark dataset and an evaluation of LLM formula repair on it. The load-bearing circularity is that the benchmark's correct formulas are generated by GPT-4o, that the same model family is then evaluated against those formulas, and that the semantic gatekeeper for dataset inclusion is an LLM Validator whose agreement with humans is weak and internally inconsistent (0.42 vs 0.25). This makes the GPT-4o Execution Match (0.73) and, to a lesser degree, GPT-4.1's (0.80) partly self-consistency measurements rather than independent repair-accuracy measurements. The paper honestly discloses the judge's limitations and the persistence of unrealistic samples, which supports rather than refutes the circularity concern. There is no load-bearing self-citation or imported uniqueness theorem here; the circularity is in the dataset construction/evaluation loop. The score is 6 rather than higher because execution-based checks via Calc.ts and the human-curated seed data provide some independent content, so the result is not wholly forced by definition; nevertheless, the semantic ground truth of the benchmark is not independent of the evaluated model family.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark construction depends on several modeling choices and implicit thresholds (temperature, judge criteria, seed corpus) and on the assumption that the LLM-as-a-judge can stand in for human quality assessment. No new physical or formal entities are introduced.

free parameters (2)
  • temperature for GPT-4o generation = 0.64
    Chosen to promote diversity among generated examples; a free design choice not grounded in external optimization.
  • LLM Validator acceptance criterion = not formally specified
    The judge's pass/fail decision and any internal threshold are not described, so the dataset composition depends on an implicit, unreported parameter.
assumptions (3)
  • domain assumption Calc.ts execution confirms that a formula is semantically correct, not merely error-free
    Invoked in Section 3.2.2 to validate generated formulas; executing without errors does not guarantee the formula meets user intent.
  • ad hoc to paper GPT-4-based LLM Validator reliably assesses semantic alignment and realism of synthetic examples
    Proposed in Section 3.2.3; Section 6.2 shows only moderate agreement with human annotators (kappa 0.42), weakening this assumption.
  • domain assumption The MrExcel-derived seed data, after manual editing, represents realistic runtime-error repair scenarios
    Section 3.1 describes curation from one forum; the paper later acknowledges synthetic data is simpler than this seed set, indicating limited coverage of real-world complexity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs." pith.science (2026). https://pith.science/paper/JG76IPAR

@misc{pith2026250811715,
  author       = {Pith},
  title        = {Pith review of: Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JG76IPAR}},
  note         = {Machine review of arXiv:2508.11715}
}
read the original abstract

Excel is a pervasive yet often complex tool, particularly for novice users, where runtime errors arising from logical mistakes or misinterpretations of functions pose a significant challenge. While large language models (LLMs) offer promising assistance by explaining formula errors, the automated correction of these semantic runtime errors remains an open problem. A primary challenge to advancing models for such scenarios is the severe lack of high-quality, comprehensive datasets for training and rigorous evaluation. This paper addresses this gap by introducing a novel approach for constructing a benchmark dataset specifically designed for Excel formula repair. We propose a data generation pipeline, which leverages a small set of curated seed samples from online forums to synthetically expand the dataset. Our pipeline integrates few-shot prompting with LLMs and employs a robust \textit{LLM-as-a-Judge} validation framework, combined with execution-based checks to ensure the correctness and semantic fidelity of the generated data. This process produced a benchmark dataset of 618 high-quality samples, covering common runtime errors. Furthermore, we propose a context-aware baseline technique for Excel formula repair that utilizes LLMs to leverage both the faulty formula, and relevant spreadsheet context. We evaluate the performance of various LLMs (GPT-4o, GPT-4.1, Phi-3, Mistral) on our newly generated benchmark using execution-based metrics. Our analysis demonstrates the dataset's quality through manual annotation and provides insights into error and function distributions. The proposed generation methodology is highly scalable and can be readily adapted to create evaluation benchmarks for similar code repair tasks in other low-resource programming languages.

Figures

Figures reproduced from arXiv: 2508.11715 by the authors.

Figure 1
Figure 1. Overall workflow illustrating the sequential steps in the seed data curation process. MrExcel forum is scraped to get [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Pipeline overview of the synthetic data generation system. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An example from the seed dataset where the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An example from FoRepBench. The faulty for￾mula does not handle the exception when the value A10 is not present in the table. The correct formula fixes this by checking if the value exists, and returning "Not Found" if it doesn’t. the input and attempts to generate a c…
Figure 7
Figure 7. Figure 7: Example from the synthetic dataset before being passed through LLM Validator. where annotators disagree on validity. The formula includes cell A2 in a summation, but A2 contains a string. One annotator considers it valid, interpreting it as a possible typo, while the o…
Figure 6
Figure 6. Figure 6: Distribution of sample difficulty levels by Excel [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024)

    arXiv 2024

  2. [2]

    Daniel W Barowy, Shan Gao, Alvin Cheung, and Brad A Myers. 2014. ExceLint: Automatically detecting spreadsheet formula errors. In Proceedings of the 36th International Conference on Software Engineering . ACM, 460–470

    work page 2014

  3. [3]

    Rohan Bavishi, Harshit Joshi, José Cambronero, Anna Fariha, Sumit Gulwani, Vu Le, Ivan Radiček, and Ashish Tiwari. 2022. Neurosymbolic repair for low-code formula languages. Proc. ACM Program. Lang. 6, OOPSLA2, Article 164 (Oct. 2022), 30 pages. doi:10.1145/3563327

    doi:10.1145/3563327 2022

  4. [4]

    Rohan Bavishi, Harshita Joshi, Jorge Cambronero, Ayesha Fariha, Sumit Gul- wani, Vu Le, Ivan Radiček, and Aditya Tiwari. 2022. Neurosymbolic Repair for Low-Code Formula Languages. Proceedings of the ACM on Programming Languages 6, OOPSLA2 (2022)

    work page 2022

  5. [5]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (2020)

    work page 2020

  6. [6]

    Binyuan Chen, Qian Liu, Jinjie Jiang, and et al. 2023. CodeLLM: Evaluating Large Language Models on Code Generation. arXiv preprint arXiv:2305.14335 (2023)

    work page Pith review arXiv 2023

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al

    work page

  8. [8]

    Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. 2021. SpreadsheetCoder: Formula Prediction from Semi- structured Context. arXiv:2106.15339 [cs.SE] https://arxiv.org/abs/2106.15339

    work page Pith review arXiv 2021

Show all 30 references
  1. [9]

    Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie Van Deursen

  2. [10]

    Aniruddh Gudibande, Xisen Li, Ethan Chi, Percy Liang, and Yuxin Wu. 2023. False sense of security: Evaluation misalignment in language models. arXiv preprint arXiv:2304.09106 (2023)

    2023 arXiv

  3. [11]

    Sumit Gulwani. 2011. Automating string processing in spreadsheets using input- output examples. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Sym- posium on Principles of Programming Languages (Austin, Texas, USA) (POPL ’11). Association for Computing Machinery, New Yor...

    2011

  4. [12]

    Felienne Hermans, Martin Pinzger, and Arie van Deursen. 2016. Detecting errors in spreadsheets. In Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 818–828

    2016

  5. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    2023 arXiv

  7. [15]

    Harshit Joshi, Abishai Ebenezer, José Cambronero Sanchez, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. 2024. Flame: A small language model for spreadsheet formulas. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 12995–13003

    2024

  8. [16]

    Sean Kandel, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2011. Wrangler: Interactive visual specification of data transformation scripts. In Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM, 3363–3372

    2011

  9. [17]

    Xi Li et al. 2022. Competition-Level Code Generation with AlphaCode. In Pro- ceedings of the International Conference on Machine Learning (ICML)

    2022

  10. [18]

    Liu et al

    A. Liu et al . 2022. InCoder: A Generative Model for Code Infill. arXiv:2210.00745 [cs.CL]

    2022 arXiv

  11. [19]

    Fangyu Liu, Yuxian Wu, Yixuan Liu, and et al. 2023. GPTEval: NLG evaluation using GPT-4 as the reference-free evaluator. arXiv preprint arXiv:2305.04648 (2023)

    2023 arXiv

  12. [20]

    Martin Monperrus. 2018. Automatic Software Repair: A Bibliography. ACM Comput. Surv. 51, 1, Article 17 (Jan. 2018), 24 pages. doi:10.1145/3105906

    2018 doi

  13. [21]

    Eric Nijkamp, Christopher Rosin, Antonio Martins, Adam Rogers, Thomas Wolf, Mikel Artetxe, Victor Costa, Sudheer Banerjee, Binh Shih, Emelie Siktberg, et al

  14. [22]

    Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, and Gust Verbruggen. 2024. An Empirical Study of Validating Synthetic Data for Formula Generation. arXiv:2407.10657 [cs.CL] https://arxiv. org/abs/2407.10657

    2024 arXiv

  15. [23]

    Alex Wang and Ellie Pavlick. 2023. ChatGPT as a judge: Linguistic acceptability judgments. arXiv preprint arXiv:2304.03442 (2023)

    2023 arXiv

  16. [24]

    Shuai Wang, Feng Li, Jia Zhou, Ruo Yan, and Jie Chen. 2021. CodeT5: Identifier- aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  17. [25]

    Pengcheng Yin and Graham Neubig. 2018. Learning to Represent Programs with Graphs. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 195–206

    2018

  18. [26]

    Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, and Haidong Zhang. 2024. NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries. arXiv:2402.14853 [cs.CL] https://arxiv. org/abs/2402.14853

    2024 arXiv

  19. [27]

    Lei Zheng, Xiaowei Wang, Baoxu Peng, Xin Wang, and Minlie Huang. 2023. Judging code generation with large language models: A comparative study.arXiv preprint arXiv:2305.17951 (2023)

    2023 arXiv

  20. [2015]

    In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering, Volume 1

    Work practices and challenges in pull-based development: The integrator’s perspective. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering, Volume 1. IEEE, 358–368

  21. [2021]

    arXiv preprint arXiv:2107.03374 (2021)

    Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  22. [2022]

    arXiv preprint arXiv:2203.13474 (2022)

    CodeGen: An Open Large Language Model for Code Generation. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.