Pith. sign in

REVIEW 3 major objections 7 minor 31 references

CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark shows that simple program transformations make state-of-the-art LLMs much worse at deciding whether two programs compute the same function.

desk verdict Builds a useful perturbation benchmark for LLM code-equivalence checking and finds a believable, large accuracy drop; the main soft spot is unstated handling of non-preserving label errors, which matters for fine-grained numbers but not for the headline result. read the letter →

arxiv 2506.04019 v1 pith:BEIA4LWE submitted 2025-06-04 cs.SE cs.CLcs.LGcs.PL

Neeva Oza , Ishaan Govil , Parul Gupta , Dinesh Khandelwal , Dinesh Garg , Parag Singla This is my paper
classification cs.SEcs.CLcs.LGcs.PL
keywords code-equivalencecheckingprogramtransformationsLLMevaluationsemanticequivalencefine-tuningcompetitiveprogrammingcoderobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models cannot reliably tell whether two programs compute the same function once the programs differ by small mechanical edits. To test this, the authors build a benchmark that starts from real competitive-programming solutions and applies sequences of one to five simple transformations, such as renaming variables, swapping if/else blocks, or flipping conditions, producing both semantically equivalent and non-equivalent pairs. Across four LLMs, including a leading closed model, accuracy on equivalence checking drops substantially after such perturbations, especially for pairs that originally were identical or functionally equivalent. The authors conclude that current models lean on surface form rather than functional semantics, and they show that fine-tuning on perturbed pairs partially restores performance. If true, this matters for any downstream use of LLMs for code rewriting, translation, or repair, where equivalence checking is the safety check.

What carries the argument

The load-bearing mechanism is the CETBench generation pipeline. Starting from clusters of human-written solutions to competitive-programming problems, the pipeline samples a base pair whose relationship is labelled id (identical), fe (functionally equivalent), ne (one correct, one incorrect), or di (different problems). It then selects a perturbation count k between 1 and 5, decides whether the net effect should preserve semantics, and applies a random sequence of seven transformation types, each available in a semantic-preserving and a semantic-non-preserving variant; every generated pair receives the label \langle prefix, k, flag\rangle. This construction isolates the effect of small syntactic changes on equivalence judgment while keeping the underlying task content fixed.

What would settle it

Re-run the main accuracy comparisons after relabelling every pair by formal equivalence or by running both programs on a much larger fuzzed input set, then check whether the reported post-perturbation drops persist; if the drops mostly disappear, the conclusion is an artifact of test-based labels.

Watch

Extended reading notes

Core claim

The central claim is that code-equivalence checking is far from solved by current LLMs, and that the gap is exposed by perturbations so small that a human reader would spot them immediately. For pairs of programs that are literally identical or solving the same task, models score near ceiling before perturbation; after one or a few mechanical edits, accuracy collapses, and on semantically non-preserving edits of identical code some models drop to near zero. The paper interprets this as evidence that LLMs answer equivalence questions using statistical and syntactic pattern matching rather than a functional model of what the code computes. It also demonstrates that targeted fine-tuning on original plus perturbed pairs can lift small open models above an unfine-tuned frontier model, which it reads as a sign that the capability exists but needs explicit supervision.

Load-bearing premise

The labels say two programs are equivalent or not based on whether they pass the original problem's test suite, not on a proof of universal equivalence; incomplete tests would mislabel some pairs.

Editorial extensions

If this is right

  • Equivalence checking is not solved by model scale alone: GPT-4o also shows large drops on perturbed pairs, and its chain-of-thought prompting can hurt rather than help.
  • Pairs that start identical or functionally equivalent are the most fragile; models often judge two copies of the same code inequivalent after a small edit.
  • Logic-changing edits (boolean inversion, if-condition flipping, if-else swapping) degrade accuracy more than surface-level edits (variable renaming, statement reordering), which points to where semantic understanding fails first.
  • Fine-tuning on a mix of perturbed and unperturbed pairs restores much of the lost accuracy and lets small open-weight models beat the best zero-shot GPT-4o result.
  • The drop in accuracy is roughly constant across problem difficulty levels, so the effect is not limited to easy or hard programs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper does not pursue: if its labels are cleaned with formal equivalence or broader differential testing, the reported accuracy drops may shrink; the paper's reported error rates up to 11.68 percent on some non-preserving edits suggest label noise is not negligible.
  • The same pipeline could be turned into a robustness probe for code-repair and translation models, where equivalence checking is the acceptance criterion, rather than only a standalone LLM evaluation.
  • The fine-tuning recovery may reflect learning perturbation-specific patterns; holding out unseen transformation types would tell whether the model learned semantics or heuristics.
  • A stronger version of the benchmark would include programs that differ stylistically but are provably equivalent with no edits, to separate the model's ability to handle natural diversity from its sensitivity to synthetic perturbation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces CETBench, a benchmark for evaluating LLMs on code-equivalence checking. The dataset is constructed by applying sequences of semantic-preserving (sp) and semantic-non-preserving (np) transformations to pairs of programs drawn from the CodeContests repository, producing eight categories based on the original pair relationship (identical, functionally equivalent, one-correct-one-incorrect, different problems) and the net semantic flag. The authors evaluate four LLMs (GPT-4o, Qwen2.5-Coder-7B, Mistral-7B, DeepSeek-Coder-6.7B) on original and perturbed pairs, reporting large accuracy drops on perturbed pairs, and then present a LoRA fine-tuning approach that partially recovers performance. The central claim is that simple transformations cause substantial performance degradation, indicating that LLMs rely on surface form rather than on semantic understanding of code.

Significance. If the central result holds, CETBench is a useful and timely benchmark for code-equivalence checking, complementing recent benchmarks such as SeqCoBench and EquiBench by adding composed sequences of transformations and a finer-grained category structure. The paper's strengths include a generic perturbation pipeline, ablations over problem difficulty and transformation type, variance analysis across sampled pairs, explicit error-rate reporting, and a simple fine-tuning baseline that demonstrates the effect of targeted supervision. The finding that even trivial syntactic changes sharply reduce model accuracy is an informative stress test for LLM code reasoning, and the dataset, if made available, could support further research on semantic robustness.

major comments (3)
  1. [Section 3.3 and Appendix D (Table 15)] The manuscript states that instances where an intended semantic-preserving transformation fails the test-suite validation are excluded from the benchmark, but it does not state whether semantic-non-preserving transformations that fail to break behavior (i.e., the perturbed program still passes the original test suite) are also excluded. Table 15 reports np error rates up to 11.68% for if-else swapping, and the text describes these as percentages of erroneous samples generated over perturbation attempts without clarifying whether such samples end up in the final dataset. If they are retained, a non-negligible fraction of np-labeled pairs are actually equivalent, so a model answering 'Yes' is correct but is counted as wrong. This would systematically deflate np accuracy and inflate the reported performance drop, which is the paper's central claim. The authors should either state clearly that np failures are discarded, or re-run the evaluation after removing them and provide a sensitivity analysis showing that the reported drop survives label correction.
  2. [Section 4.2, Table 2] The structure of the results tables is not self-explanatory. The top row for each model is labeled 'original' and the bottom row 'perturbed', but both rows are split into np/sp columns. Original pairs have not undergone any perturbation, so the np/sp distinction does not directly apply; presumably the original rows are split according to the flag of the corresponding perturbed pair, but this mapping is never stated. Without this explanation, the reader cannot verify the magnitude of the drop from original to perturbed rows, which is the main empirical result. Please define the original-row split explicitly and consider presenting original rows with a single value per category or with an explicit pairing explanation.
  3. [General (data availability)] The paper does not provide an availability statement for the dataset, the perturbation-generation code, or the evaluation scripts. Since the stated primary contribution is the CETBench dataset, a benchmark paper should include a link or a clear release plan; otherwise the contribution cannot be used by other researchers or independently verified. Please add a data/code availability statement.
minor comments (7)
  1. [Section 2 vs Section 3.4] Section 2 says CETBench applies 'composed sequences of up to seven transformations,' but Section 3.4 states that the benchmark test set uses Kmax = 5, so k ranges from 1 through 5. Reconcile this inconsistency.
  2. [Section 4.3.2] There is a typo: 'correspnding' should be 'corresponding'.
  3. [Figures 1 and 2] The captions say 'Absolute Accuracy or its Delta (%)' but the plots show multiple curves that appear to mix absolute accuracy and delta values. Clarify which curves are absolute and which are deltas, and state the units precisely.
  4. [Appendix D, Table 15] The heading 'Error bounds' is better described as 'error rates'; define the denominator (number of perturbation attempts) and state explicitly whether erroneous samples are discarded before the benchmark is finalized.
  5. [Section 4.4, Table 4] Table 4 reports fine-tuning results for Qwen, Mistral, and DSC but not for GPT-4o. State explicitly that GPT-4o was not fine-tuned and explain the reason (e.g., API constraints), since the text says 'best models for each LLM.'
  6. [Section 2] The abstract and introduction describe code-equivalence checking as 'relatively unexplored,' but Section 2 cites two closely related parallel benchmarks (SeqCoBench and EquiBench). Consider tempering the novelty claim or explaining more concretely how CETBench differs in scope and construction.
  7. [Appendix C] Variance analysis is reported only for the Qwen model. Since the paper's central claim concerns four models, either report variance or confidence intervals for the other models as well, or justify why Qwen is representative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark labels and the measured LLM accuracy drops are produced by independent pipelines, and the only self-referential aspect (fine-tuning and test sets from the same generator) is explicitly disclosed as a generalization limitation.

full rationale

CETBench's central claims are empirical observations about LLM accuracy on program pairs, not a derived prediction that reduces to its inputs. Labels are generated by an external pipeline: solution clusters from CodeContests, a fixed set of transformation rules, and test-suite validation in Section 3.3, all independent of the LLMs being benchmarked. No parameter is fitted to the evaluation data and then renamed as a prediction, and no self-citation carries the load of the argument. The fine-tuning experiments use train and validation splits generated by the same Algorithm 1 as the test set, but this is a standard supervised-learning setup and the paper's Limitations section explicitly states that generalization to unseen perturbation types or real-world scenarios remains unclear, so the limitation is disclosed rather than hidden. Appendix D's error rates (up to 5.35% for sp and 11.68% for np) and the test-suite-based validation in Section 3.3 are label-quality and correctness concerns, not circularity, because ground truth is determined by the perturbation/validation pipeline rather than by the evaluated models; even the largest reported np error cannot by itself account for drops of nearly 100 accuracy points in id-np categories such as Mistral's 0.2% versus 100% on the original pairs. The paper's conclusion that LLMs rely on surface patterns is an inductive interpretation of those independent measurements, not a definitional restatement of the benchmark construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on test-suite-based labels, automated perturbation implementations, and CodeContests metadata; no formal verification or released artifacts independently certify the ground truth.

free parameters (4)
  • Kmax = 5
    Maximum number of perturbations applied to a pair in the test set; chosen by hand and controls dataset difficulty. Section 3.4 sets Kmax=5 while the related work section says 'up to seven'.
  • p_same
    Probability of sampling both programs from the same solution cluster; referenced in Algorithm 1 but the value is not stated, and it determines the mix of identical and different-problem pairs.
  • max_tries
    Retry limit when a perturbation attempt fails; not specified in Algorithm 1, affects which samples enter the benchmark.
  • preserve_probability = 0.5
    Bernoulli(0.5) decides whether a perturbation sequence is net semantic-preserving, creating a 75/25 negative/positive class imbalance in the perturbed test set, as noted in Appendix B.
assumptions (4)
  • domain assumption CodeContests test cases are correct and comprehensive enough to certify functional equivalence for validated perturbed pairs.
    Used in Section 3.3 to accept semantic-preserving transformations; no formal verification is performed.
  • domain assumption The perturbation implementations reliably produce code with the intended semantic effect and the ast/astor transformations do not introduce syntax errors beyond the reported retries.
    Algorithm 1 relies on automated rewriting; Appendix D reports error rates but does not describe corrections.
  • domain assumption Codeforces difficulty ratings are a meaningful proxy for problem difficulty in the LLM ablations.
    Section 4.3.2 varies difficulty 900 to 2200 using Codeforces ratings as the difficulty measure.
  • domain assumption Parsing the model outputs for 'Yes' or 'No' and for the COT format captures the model's intended judgment.
    Evaluation depends on exact output parsing; no manual verification of parsed judgments is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking." pith.science (2026). https://pith.science/paper/BEIA4LWE

@misc{pith2026250604019,
  author       = {Pith},
  title        = {Pith review of: CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BEIA4LWE}},
  note         = {Machine review of arXiv:2506.04019}
}
read the original abstract

LLMs have been extensively used for the task of automated code generation. In this work, we examine the applicability of LLMs for the related but relatively unexplored task of code-equivalence checking, i.e., given two programs, whether they are functionally equivalent or not. This is an important problem since benchmarking code equivalence can play a critical role in evaluating LLM capabilities for tasks such as code re-writing and code translation. Towards this end, we present CETBench - Code Equivalence with Transformations Benchmark, constructed via a repository of programs, where two programs in the repository may be solving the same or different tasks. Each instance in our dataset is obtained by taking a pair of programs in the repository and applying a random series of pre-defined code transformations, resulting in (non-)equivalent pairs. Our analysis on this dataset reveals a surprising finding that very simple code transformations in the underlying pair of programs can result in a significant drop in performance of SOTA LLMs for the task of code-equivalence checking. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, and can be used with repositories with varying program difficulty levels and allows for applying varying numbers as well as kinds of transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.

Figures

Figures reproduced from arXiv: 2506.04019 by the authors.

Figure 1
Figure 1. Change in Accuracy and F1 with varying Difficulty Level for different Models [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Change in Accuracy and F1 with varying Perturbation Types for different Models [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Example of perturbation on programs. To keep our results reproducible, we use these hyperparameters of LLMs in our experiments: We set sampling to False, temperature to 0. For finetuning, we used LoRA finetuning using A100 GPU. We finetuned for nearly 48 GPU hours and selected the model with the least validation loss. The training and validation datasets were generated by the same algorithm 1. We initially split the… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Change in Weighted accuracy and F1 with varying Difficulty Level and varying Perturbation Types for [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 20 canonical work pages

  1. [1]

    Ajmain I Alam, Palash R Roy, Farouq Al-Omari, Chanchal K Roy, Banani Roy, and Kevin A Schneider. 2023. Gptclonebench: A comprehensive benchmark of semantic clones and cross-language clones using gpt-3 model and semanticclonebench. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 1--13. IEEE

    work page 2023

  2. [2]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. https://api.semanticscholar.org/CorpusID:237142385 Program synthesis with large language models . ArXiv, abs/2108.07732

    arXiv 2021

  3. [3]

    Sahar Badihi, Yi Li, and Julia Rubin. 2021. Eqbench: A dataset of equivalent and non-equivalent program pairs. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 610--614. IEEE

    work page 2021

  4. [4]

    Ira D Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant'Anna, and Lorraine Bier. 1998. Clone detection using abstract syntax trees. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272), pages 368--377. IEEE

    work page 1998

  5. [5]

    Saikat Chakraborty, Shuvendu Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy. 2023. Ranking llm-generated loop invariants for program verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9164--9175

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \'e , Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 34 others. 2021. https://api.semanticscholar.org/CorpusID:235755472 Evaluating lar...

    arXiv 2021

  7. [7]

    Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990--998. PMLR

    work page 2017

  8. [8]

    Yijiang Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126--10141

    work page 2024

Show all 31 references
  1. [9]

    Aram Ebtekar. 2019. https://codeforces.com/blog/entry/68288 How to interpret contest ratings . Codeforces Blog. Accessed: 2025-05-18

    2019

  2. [10]

    Mark Gabel, Lingxiao Jiang, and Zhendong Su. 2008. Scalable detection of semantic clones. In Proceedings of the 30th international conference on Software engineering, pages 321--330

    2008

  3. [11]

    Robert Goldblatt and Marcel Jackson. 2012. Well-structured program equivalence is highly undecidable. ACM Transactions on Computational Logic (TOCL), 13(3):1--8

    2012

  4. [12]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  5. [13]

    Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246

    2025 arXiv

  6. [14]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2025. https://arxiv.org/abs/2408.02479 From llms to llm-based agents for software engineering: A survey of current, challenges and future . Preprint, arXiv:2408.02479

    2025 arXiv

  7. [15]

    Heejung Kim, Yungbum Jung, Sunghun Kim, and Kwankeun Yi. 2011. Mecc: memory comparison-based clone detector. In Proceedings of the 33rd International Conference on Software Engineering, pages 301--310

    2011

  8. [16]

    Jens Krinke and Chaiyong Ragkhitwetsagul. 2022. Bigclonebench considered harmful for machine learning. In 2022 IEEE 16th International Workshop on Software Clones (IWSC), pages 1--7. IEEE

    2022

  9. [17]

    Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated program repair. Communications of the ACM, 62(12):56--65

    2019

  10. [18]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, and 11 ot...

    2022

  11. [19]

    Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2025. Exploring the potential of general purpose llms in automated software refactoring: an empirical study. Automated Software Engineering, 32(1):26

    2025

  12. [20]

    Nickil Maveli, Antonio Vergari, and Shay B Cohen. 2025. https://aclanthology.org/2025.findings-naacl.382/ What can large language models capture about code functional equivalence? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6865--6903, Albuq...

    2025

  13. [21]

    MikeMirzayanov. 2018. https://codeforces.com/blog/entry/62865 Codeforces: Problem difficulties . Codeforces Blog. Accessed: 2025-05-18

    2018

  14. [22]

    Miguel Romero Rosas, Miguel Torres Sanchez, and Rudolf Eigenmann. 2024. Should ai optimize your code? a comparative study of current large language models versus classical optimizing compilers. arXiv preprint arXiv:2406.12146

    2024 arXiv

  15. [23]

    Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in neural information processing systems, 33:20601--20611

    2020

  16. [24]

    Kathryn T Stolee, Sebastian Elbaum, and Daniel Dobos. 2014. Solving the search for source code. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(3):1--45

    2014

  17. [25]

    Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. 2014. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476--480. IEEE

    2014

  18. [26]

    Weixi Tong and Tianyi Zhang. 2024. Codejudge: Evaluating code generation with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20032--20051

    2024

  19. [27]

    Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yaofeng Sun, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, and 1 others. 2025. Equibench: Benchmarking code reasoning capabilities of large language models via equivalence checking. arXiv preprint arXiv:2502.12466

    2025

  20. [28]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Informa...

    2023

  21. [29]

    Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2232--2242

    2024

  22. [30]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  23. [31]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.