Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read METAMON flags code-documentation mismatches with 0.72 precision and 0.48 recall.

desk verdict METAMON's core idea—using generated regression tests as behavioral anchors for LLM-based doc-consistency checking—is genuinely good, but the headline precision/recall are likely overstated because the ground truth labels treat every mutant-killed test as a documentation inconsistency. read the letter →

arxiv 2502.02794 v1 pith:6F4KFRY5 submitted 2025-02-05 cs.SE

Hyeonseok Lee , Gabin An , Shin Yoo This is my paper · ORCID
classification cs.SE
keywords codedocumentationinconsistencyLLMreasoningmetamorphictestingself-consistencyregressiontestoraclesearch-basedgenerationJavaJavadocproblem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

METAMON is a method for checking whether Java method documentation matches what the code actually does. It captures behavior not by reading the source to the model, but by first generating regression tests whose assertions record current outputs, then asking a language model whether each assertion agrees with the documented specification. The paper's central hypothesis is that metamorphic prompt pairs—the original assertion plus a version with the assertion negated—make the model's judgments reliable enough to triage stale documentation automatically. On 9,482 documentation-test pairs drawn from five open-source Java projects, the method reports precision 0.72 and recall 0.48 for flagging inconsistencies; if correct, this gives teams a low-cost filter that surfaces only the pairs worth human review.

What carries the argument

The load-bearing object is the metamorphic prompt pair: an original prompt containing a generated test assertion and a transformed prompt in which the assertion is negated (for example, assertEquals becomes assertNotEquals). The identity that carries the argument is the metamorphic relation $$R = \{(a_1,a_2,\mathrm{Exec}(t,a_1),\mathrm{Exec}(t,a_2)) \mid a_1 = \neg a_2 \to \mathrm{Exec}(t,a_1) = \neg \mathrm{Exec}(t,a_2)\}$$, where $\mathrm{Exec}(t,a)$ is whether test $t$ passes assertion $a$; it says the two opposite assertions cannot both pass on the same program. A language model that understands the documentation should therefore give opposite labels to the two prompts; METAMON scores both, inverts the transformed-prompt scores, and sums them to get a consistency score. The score is sharpened by chain-of-thought prompting, by querying each prompt $n$ times and taking a sum, and by allowing an <undecidable> label so the model does not invent a verdict when the documentation is too vague.

What would settle it

Take the same five projects and have human annotators judge, without seeing mutants, which documentation-test pairs are genuinely inconsistent, then compare METAMON's threshold classification against those annotations; if precision at the -0.1 threshold falls well below 0.72, the mutant-based oracle labels were inflating the result.

Watch

Extended reading notes

Core claim

The paper claims that an LLM's correctness judgment of a generated regression-test oracle is a valid proxy for consistency between documentation and behavior. To establish this, METAMON injects first-order mutants into documented methods so that generated tests on the mutated code fail against the original version; those failing tests are the ground-truth 'incorrect oracle' cases, and passing tests are the 'correct oracle' cases. The LLM labels each oracle <correct>, <undecidable>, or <incorrect>, and the labels are aggregated into a normalized score in [-1,1]. The discovery is that the aggregate score separates the two classes: at threshold -0.1, precision is 0.722 and recall 0.480, and stricter thresholds push precision toward 1.0. The paper's own framing is that this turns documentation checking from a human-only task into an automated, tunable triage step.

Load-bearing premise

The evaluation labels a generated test oracle as incorrect exactly when it fails on the original, unmutated program, which assumes that every mutation-induced behavior change counts as a documentation inconsistency; if a mutation changes only undocumented behavior, the ground-truth labels overstate real-world inconsistency.

Editorial extensions

If this is right

  • Teams can delegate first-pass documentation triage to METAMON and review only the pairs whose normalized score falls below a chosen threshold, converting a manual scan into a focused review.
  • The threshold controls the precision-recall tradeoff: at score $\leq -0.1$ precision is 0.722 with recall 0.480, while at $\leq -0.6$ precision rises to 0.967 as recall falls to 0.099.
  • Metamorphic prompting is the decisive component: using only the original prompts gives near-random separation, whereas adding the transformed prompts makes low scores strongly indicative of inconsistent oracles.
  • Self-consistency and the <undecidable> label both contribute to the monotonic relationship between score and inconsistency ratio, with diminishing returns as the number of queries grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to use the normalized score as a prioritization key for documentation repair, reviewing the lowest-scoring methods first rather than applying a single binary threshold; the reported monotonic relationship between score and inconsistency ratio suggests this would concentrate effort where inconsistencies are likeliest.
  • The metamorphic-prompt consistency check is not tied to Javadoc specifics; adapting the assertion transformations to other languages or test frameworks would let the same technique audit documentation in other ecosystems, although the empirical numbers here are Java-only.
  • Because the ground-truth labels come from mutant-induced test failures, the 0.72 precision is likely an upper-bound estimate for real stale-documentation settings, where many mutations may affect behavior the documentation never specifies; a human-annotated evaluation would clarify the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes METAMON, an automated pipeline that checks for inconsistencies between Java method documentation and program behavior. The pipeline first filters methods whose Javadoc contains @param and @return descriptions, then generates regression tests with EvoSuite (on mutated versions of the methods) to capture behavior as executable oracles. An LLM (GPT-3.5-Turbo) is prompted, with chain-of-thought, to label each test oracle as <correct>, <incorrect>, or <undecidable> against the documentation; each prompt is paired with a metamorphically transformed version in which the assertion is negated, and the LLM is queried multiple times. The responses are aggregated into a normalized consistency score, and a threshold on this score classifies the oracle as inconsistent with the documentation. The authors evaluate on 9,482 test-documentation pairs built from five Defects4J projects using Major mutants, reporting precision 0.722 and recall 0.480 at a score threshold of -0.1, and an ablation study showing positive contributions from metamorphic prompting, self-consistency, and the <undecidable> label.

Significance. If the reported effectiveness generalizes, METAMON would be a useful triage tool for stale or inaccurate documentation, combining concrete program behavior captured by search-based testing with LLM reasoning. The metamorphic-prompting idea is well-motivated and is a practical way to check LLM consistency without requiring a second model or human oracle. The paper also contributes a sizeable public dataset and replication materials, and its ablation study is carefully designed to isolate component contributions. The main weakness is that the ground truth labels are constructed from mutant execution results rather than from the documentation itself, so the headline precision and recall may not measure the stated construct; this concern is central to the empirical claim and needs additional validation.

major comments (4)
  1. [Section IV-A.4, Oracle Identification] The ground-truth label for an "incorrect oracle" is defined by executing the EvoSuite-generated test against the fixed program version: a failing test is labeled incorrect. This operationalizes a documentation inconsistency as "a mutant changed some behavior observable by a generated test," but it does not verify that the documentation actually specifies the changed behavior. The paper's own RQ3 analysis (Section V-C, Fig. 5 and Fig. 6) identifies cases where the documentation lacks specification detail or contextual information, and in those cases a failing test does not entail a documentation-code mismatch. Because such pairs are counted as ground-truth positives, the reported precision and recall are estimates of a proxy construct rather than of the stated claim. I recommend adding a human-annotated validation subset (or restricting the evaluation to methods whose Javadoc gives explicit output constraints) to show that the execution-based labels align with genuine documentation inconsistencies.
  2. [Section V-A, Table III] The headline precision of 0.722 and recall of 0.480 are reported at a threshold (-0.1) selected from the same data used to evaluate the classifier, and no threshold-selection procedure or separate validation set is described. This makes the numbers optimistic as performance estimates. The paper should report a threshold-independent measure (e.g., area under the precision-recall curve) or a cross-validated threshold-selection protocol, and should provide confidence intervals given that GPT-3.5-Turbo is sampled stochastically with temperature 0.7 and only one run is reported.
  3. [Section V, RQ1 and RQ2] The evaluation does not include a baseline that asks the LLM to judge code-and-documentation consistency directly from source code and documentation, without first capturing behavior as generated test oracles. The ablation study compares variants of METAMON (original prompt, transformed prompt, metamorphic prompt, with and without <undecidable>), but all variants use EvoSuite tests. Since one of the paper's central claims is that capturing behavior through regression tests is beneficial, the absence of this baseline leaves open the possibility that the LLM's judgment, rather than the test-generation step, drives the result. Adding a raw code-and-doc baseline would make the contribution of the behavioral-capture component explicit.
  4. [Section IV-A, Dataset construction] The evaluation samples multiple mutants per method and reports test-level precision and recall without accounting for clustering by method or project. Mutants of the same method produce related test-documentation pairs, so the effective number of independent samples is smaller than 9,482. Reporting per-project results and some measure of variance (e.g., by method-level bootstrap or mixed-effects analysis) would make the empirical claims more robust.
minor comments (6)
  1. [Section IV-A, text before Table II] The text says the dataset contains 4,800 tests with incorrect oracles and an equal number with correct oracles, but Table II lists 4,741 and 4,741, totaling 9,482; this numerical inconsistency should be corrected.
  2. [Abstract and Introduction] There is a typo in "langauges" in the Introduction; also, the rendered spacing in "M ETAMON" and "M ETAMON" should be fixed throughout.
  3. [Section II-B] The reference list includes the Chain-of-Thought paper twice ([16] and [28]); the duplicate should be removed and the citation in Section III-C updated accordingly.
  4. [Figure 1 and Figure 2] The string "line.seperator" appears to be a typo for "line.separator" in the examples; the text in Fig. 2 also uses "seperator" in one prompt, which may confuse readers.
  5. [Fig. 4 caption] The label "Orginal prompt" in Fig. 4a should be "Original prompt."
  6. [Section V-C, Fig. 7 title] The heading "Underperformace of LLMs" should read "Underperformance of LLMs," and the figure caption should be checked for grammatical completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: execution-based ground truth is independent of the LLM scoring procedure.

full rationale

The paper's central derivation chain is not circular. Ground-truth labels for 'incorrect oracles' are obtained by executing the EvoSuite-generated tests against the fixed version of the program (Section IV-A.4: 'Automatically generated tests are executed against the latest fixed version of the program, identifying the outcomes as failing or passing'). This label source is external to the LLM and independent of the LLM's responses. The METAMON score (Section III-E) is an aggregate of LLM judgments over original and transformed prompts; the inverted scoring for transformed prompts is a designed consistency check, not a fitted transform. The metamorphic relation in Eq. (1) is a logical property of assertion negation, not an empirically fitted parameter. No parameter is fitted to the ground-truth labels and then reported as a prediction. The only self-citation, reference [11] (Kang, Yoon, and Yoo), is used as background evidence that LLMs can reproduce bugs from reports; it is not load-bearing for METAMON's core mechanism or for excluding alternatives. The RQ3 qualitative analysis (Section V-C) does identify limitations where a mutant-killing test may not correspond to a true documentation inconsistency, such as 'Lack of Specification Detail' and 'Need for Contextual Information'; this is a legitimate construct-validity threat to the reported precision and recall, but it is not a circularity in the derivation chain. Similarly, the threshold-based precision/recall table (Table III) is descriptive across operating points; even if a threshold were selected using the same evaluation data, that would be evaluation overfitting rather than a definitional reduction. Therefore the paper's empirical claim is self-contained against an externally grounded benchmark and exhibits no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no free parameters beyond experimental settings; the core novelty is the metamorphic prompt protocol, which is evaluated on synthetic mutant-based ground truth. No new physical or theoretical entities are introduced.

free parameters (2)
  • decision threshold = -0.1 (headline operating point)
    The normalized score is converted to a binary 'inconsistent' label by thresholding at -0.1 for the headline precision/recall; the paper also reports other thresholds, indicating this is a user-selected operating point rather than a fitted model parameter.
  • number of LLM queries n = 5
    The number of queries per prompt version is set to 5. The ablation in Fig. 4b shows performance improves with n, so this is a cost-accuracy tradeoff rather than a value fitted to the evaluation data.
assumptions (4)
  • domain assumption Methods with @param and @return tags provide sufficient specification detail to evaluate generated test oracles.
    Used in Section IV-A.1 to filter the dataset; if the Javadoc lacks detail, the LLM's undecidable responses make recall low.
  • domain assumption A generated test that fails on the fixed program version is a valid ground-truth label for 'oracle inconsistent with documentation.'
    Used in Section IV-A.4 (Oracle Identification); assumes mutant-induced behavior changes are within the documented contract.
  • domain assumption EvoSuite-generated assertions capture behavior relevant to the documentation rather than only incidental implementation details.
    Used throughout Step B; if assertions depend on implementation internals, the LLM's evaluation against Javadoc is ill-posed.
  • domain assumption Assertion negation (for example assertEquals to assertNotEquals) yields an exact semantic inverse for the LLM query, so a consistent model should give opposite labels.
    Introduced in Section III-C, Eq. (1); the metamorphic scoring relies on this inverse relation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries." pith.science (2026). https://pith.science/paper/6F4KFRY5

@misc{pith2026250202794,
  author       = {Pith},
  title        = {Pith review of: METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6F4KFRY5}},
  note         = {Machine review of arXiv:2502.02794}
}
read the original abstract

Code documentation can, if written precisely, help developers better understand the code they accompany. However, unlike code, code documentation cannot be automatically verified via execution, potentially leading to inconsistencies between documentation and the actual behavior. While such inconsistencies can be harmful for the developer's understanding of the code, checking and finding them remains a costly task due to the involvement of human engineers. This paper proposes METAMON, which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases, and subsequently uses LLM-based code reasoning to identify the generated regression test oracles that are not consistent with the program specifications in the documentation. METAMON is supported in this task by metamorphic testing and self-consistency. An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that METAMON can classify the code-and-documentation inconsistencies with a precision of 0.72 and a recall of 0.48.

Figures

Figures reproduced from arXiv: 2502.02794 by the authors.

Figure 1
Figure 1. An example of a buggy source code along with its corresponding Javadoc and EvoSuite-generated regression test case [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. An example of metamorphic prompt 1) Documentation Quality Assessment: For each project, we examine the documentation quality of each method to confirm it contains descriptions for both parameters and return conditions within the latest fixed version for each project (e.g., Chart-1f, Closure-1f). Only methods with documentation satisfying these criteria move to the next step. As illustrated in Table II, despite its l… view at source ↗
Figure 3
Figure 3. Incorrect oracle detection of METAMON TABLE III: Precision, Recall, and F1 at different thresholds Score Pre. Rec. F1 Score Pre. Rec. F1 ≤ −0.1 0.722 0.480 0.576 ≤ −0.6 0.967 0.099 0.180 ≤ −0.2 0.808 0.361 0.499 ≤ −0.7 0.971 0.064 0.120 ≤ −0.3 0.873 0.267 0.409 ≤ −0.8 0.973 0.046 0.087 ≤ −0.4 0.926 0.199 0.328 ≤ −0.9 1.000 0.021 0.042 ≤ −0.5 0.952 0.134 0.235 ≤ −1.0 1.000 0.014 0.027 to detect inconsistencies accura… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An impact of metamorphic relations, self-consistency, and labels [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: An example of Need for Contextual Information public void test() throws Throwable { Fraction fraction0 = Fraction.ONE_HALF; Fraction fraction1 = Fraction.ONE_THIRD; int int0 = fraction0.compareTo(fraction1); assertEquals((-1), int0); } Answer 1: ... The expected result…
Figure 7
Figure 7. Figure 7: An example of Underperformace of LLMs method is supposed to convert special characters to JavaScript escape sequences, it does not provide concrete examples of special characters. This lack of detailed specification makes it challenging to assess the correctness of the…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. We Must Have Missed This Comment: Detecting and Repairing Stale Function References in Linux Kernel Comments

    cs.SE 2026-08 accept novelty 6.0 of 10

    ReCite detects 869 stale function references in Linux kernel v6.18-rc1 comments, with 89% of sampled repairs judged useful and 42.5% directly applicable.

Reference graph

Works this paper leans on

29 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Which documentation for software maintenance?

    S. C. B. de Souza, N. Anquetil, and K. M. de Oliveira, “Which documentation for software maintenance?” Journal of the Brazilian Computer Society, vol. 12, pp. 31–44, 2006

    work page 2006

  2. [2]

    Examining the evolution of code comments in postgresql,

    Z. M. Jiang and A. E. Hassan, “Examining the evolution of code comments in postgresql,” in Proceedings of the 2006 International Workshop on Mining Software Repositories , ser. MSR ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 179–180. [Online]. Available: https://doi.org/10.1145/1137983.1138030

    arXiv 2006

  3. [3]

    /* icomment: Bugs or bad comments?*,

    L. Tan, D. Yuan, G. Krishna, and Y . Zhou, “/* icomment: Bugs or bad comments?*,” in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles , 2007, pp. 145–158

    2007

  4. [4]

    acomment: mining annotations from comments and code to detect interrupt related concurrency bugs,

    L. Tan, Y . Zhou, and Y . Padioleau, “acomment: mining annotations from comments and code to detect interrupt related concurrency bugs,” inPro- ceedings of the 33rd international conference on software engineering , 2011, pp. 11–20

    work page 2011

  5. [5]

    @ tcomment: Testing javadoc comments to detect comment-code inconsistencies,

    S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens, “@ tcomment: Testing javadoc comments to detect comment-code inconsistencies,” in 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 2012, pp. 260–269

    work page 2012

  6. [6]

    Detecting fragile comments,

    I. K. Ratol and M. P. Robillard, “Detecting fragile comments,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2017, pp. 112–122

    work page 2017

  7. [7]

    Coherence of comments and method implementations: a dataset and an empirical investigation,

    A. Corazza, V . Maggio, and G. Scanniello, “Coherence of comments and method implementations: a dataset and an empirical investigation,” Software Quality Journal , vol. 26, pp. 751–777, 2018

    work page 2018

  8. [8]

    Deep just- in-time inconsistency detection between comments and source code,

    S. Panthaplackel, J. J. Li, M. Gligoric, and R. J. Mooney, “Deep just- in-time inconsistency detection between comments and source code,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 1, 2021, pp. 427–435

    work page 2021

Show all 29 references
  1. [9]

    Detecting code comment inconsistency using siamese recurrent network,

    F. Rabbi and M. S. Siddik, “Detecting code comment inconsistency using siamese recurrent network,” in Proceedings of the 28th International Conference on Program Comprehension , 2020, pp. 371–375

    2020

  2. [10]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al. , “Competition- level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  3. [11]

    Large language models are few- shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few- shot testers: Exploring llm-based general bug reproduction,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2312–2323

    2023

  4. [12]

    Adaptive test generation using a large language model,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “Adaptive test generation using a large language model,” arXiv e-prints, pp. arXiv–2302, 2023

    2023

  5. [13]

    Mutation-based consistency testing for eval- uating the code understanding capability of llms,

    Z. Li and D. Shin, “Mutation-based consistency testing for eval- uating the code understanding capability of llms,” arXiv preprint arXiv:2401.05940, 2024

    2024 arXiv

  6. [14]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232 , 2023

    2023 arXiv

  7. [15]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ser. ESEC/FSE ’11. New York, NY , USA: Association for ...

    2011

  8. [16]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  9. [17]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  10. [18]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Pro- ceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  11. [19]

    Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,

    J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hem- mati, “Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,” arXiv preprint arXiv:2310.10508, 2023

    arXiv

  12. [20]

    Automatic code summarization via chatgpt: How far are we?

    W. Sun, C. Fang, Y . You, Y . Miao, Y . Liu, Y . Li, G. Deng, S. Huang, Y . Chen, Q. Zhang et al., “Automatic code summarization via chatgpt: How far are we?” arXiv preprint arXiv:2305.12865 , 2023

    2023 arXiv

  13. [21]

    Improving few- shot prompts with relevant static analysis products,

    T. Ahmed, K. S. Pai, P. Devanbu, and E. T. Barr, “Improving few- shot prompts with relevant static analysis products,” arXiv preprint arXiv:2304.06815, 2023

    2023 arXiv

  14. [22]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  15. [23]

    A survey on metamorphic testing,

    S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cort ´es, “A survey on metamorphic testing,” IEEE Transactions on software engineering , vol. 42, no. 9, pp. 805–824, 2016

    2016

  16. [24]

    Assessing robust- ness of ml-based program analysis tools using metamorphic program transformations,

    L. Applis, A. Panichella, and A. van Deursen, “Assessing robust- ness of ml-based program analysis tools using metamorphic program transformations,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 1377–1381

    2021

  17. [25]

    Properties of machine learning applications for use in metamorphic testing,

    C. Murphy, G. E. Kaiser, and L. Hu, “Properties of machine learning applications for use in metamorphic testing,” 2008

    2008

  18. [26]

    Testing and validating machine learning classifiers by metamorphic testing,

    X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y . Chen, “Testing and validating machine learning classifiers by metamorphic testing,” Journal of Systems and Software , vol. 84, no. 4, pp. 544–558, 2011

    2011

  19. [27]

    On Testing Non-Testable Programs,

    E. J. Weyuker, “On Testing Non-Testable Programs,” The Computer Journal, vol. 25, no. 4, pp. 465–470, 11 1982

    1982

  20. [28]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22...

    2024

  21. [29]

    Major: An efficient and extensible tool for mutation analysis in a java compiler,

    R. Just, F. Schweiggert, and G. M. Kapfhammer, “Major: An efficient and extensible tool for mutation analysis in a java compiler,” in 2011 26th IEEE/ACM International Conference on Automated Software En- gineering (ASE 2011) . IEEE, 2011, pp. 612–615

    2011

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.