Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Combining Type Inference and Automated Unit Test Generation for Python

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By wrapping arguments in transparent proxies during test execution, this paper infers Python parameter and return types on the fly and shows the inferred types raise branch coverage and mutation scores of generated tests.

desk verdict Real dynamic type tracing result with a solid coverage evaluation, but the type-quality claim needs more runs and the abstract overstates the gain. read the letter →

arxiv 2507.01477 v2 pith:UVGVPYO2 submitted 2025-07-02 cs.SE

Lukas Krodinger , Stephan Lukasczyk , Gordon Fraser This is my paper · ORCID
classification cs.SE
keywords typetracingdynamicinferenceautomatedunittestgenerationPythonproxy-basedbranchcoveragemutationtestinghints
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that automated test generation for dynamically typed Python can recover the missing type information it needs by observing how candidate tests use their own arguments. The key move is to wrap arguments in transparent proxies during a small fraction of executions, recording which attributes, methods, and type checks the code under test performs, and to record the types of values routines return. The paper shows that feeding this traced information back into argument selection raises branch coverage, with up to 90.0% relative improvement on real-world modules, and slightly improves mutation scores. A fair reader would care because this turns test generation into a self-improving loop: every execution both measures coverage and learns the types needed to cover more.

What carries the argument

The central object is the ObjectProxy, a wrapper that intercepts dunder methods and attribute access, records the interaction in a usage trace, and then forwards the operation to the wrapped object so that it remains transparent to duck-typed code. Around it sit an isinstance shim that records type checks, an attribute-to-class mapping built from __init__ assignments and static attributes, and a consistency relation that lets the generator choose concrete subtypes of an inferred type. The mechanism is completed by probabilistic proxied execution: only 5% of executions pay the overhead of tracing, which lets the search budget be spent mostly on exploration while still accumulating type evidence.

What would settle it

Take a set of classes whose distinguishing attributes are assigned outside __init__, such as in a configure method or dynamically, and check whether type tracing infers the correct parameter types while the attribute map is restricted to __init__. If coverage and inferred-type F1 do not drop relative to a version of the map augmented with the runtime attributes, the mechanism does not depend on that map; if they do drop, the map is load-bearing.

Watch

Extended reading notes

Core claim

Type tracing treats the test generator's own executions as the data source for type inference. When a candidate test is run, arguments are wrapped in a proxy that logs every operation, such as attribute reads, method calls, comparisons, and isinstance checks, before forwarding it to the wrapped object, and the same execution records the types of returned values. The logs are merged into a type cluster that maps attribute names to the classes offering them, and later argument choices are drawn from the inferred candidate types with probability weights. Because a proxied execution can behave differently for C-implemented operations, tracing runs as a separate execution with a tuned probability, 5% in the evaluation. The paper reports that on 466 real-world modules this raises mean branch coverage from 67.7% with no type information to 71.0% when combined with developer hints, yields up to 90.0% relative coverage on individual modules, improves mutation scores slightly, and produces parameter and return types whose precision, recall, and F1 scores exceed those of all non-LLM comparison tools on the benchmark.

Load-bearing premise

The load-bearing premise is that the statically built attribute map accurately approximates which attributes instances of a class actually have; Python objects can change layout at runtime, and if the map misses attributes, observed accesses will not resolve to the right candidate types.

Editorial extensions

If this is right

  • On unannotated modules, type tracing alone raises mean branch coverage from 67.7% to 69.8%, so test generators can stop relying on random type selection when hints are absent.
  • Combining traced types with existing developer hints gives the best results, 71.0% mean coverage and 90.0% relative coverage on some modules, so tracing is an addition rather than a replacement for annotations.
  • Inferred parameter and return types are of comparable or better quality than those of the non-LLM static inference tools compared, meaning the approach can double as a side-effect type inference for unannotated code.
  • Mutation scores improve only slightly, and using all available type information is still the best configuration, so the main benefit of tracing is coverage rather than fault detection.
  • Type quality correlates with coverage, with a reported Pearson coefficient of 0.546, implying that deeper exploration produces better types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the inferred types are valid input types rather than necessarily the developer's intended annotations, the same mechanism could be used to suggest broader parameter types that still execute successfully, which may reveal hidden guard branches or design simplifications.
  • The probabilistic tracing idea transfers to any dynamic language or test generator that repeatedly executes candidate tests; the 5% probability is an empirical knob that would need retuning for each language and search budget.
  • An obvious hybrid, left implicit in the paper, is to seed type tracing with LLM or static predictions and let runtime tracing refine them, combining high initial accuracy with execution-grounded evidence.
  • The positive correlation between coverage and type quality suggests that any future improvement in object instantiation, which the paper names as an open challenge, would automatically improve the inferred types as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes type tracing, a dynamic analysis embedded in the Pynguin test generator for Python. During selected test executions, arguments are wrapped in transparent proxies that record attribute accesses, method calls, and isinstance checks, while return types are recorded from the values returned by successfully executed routines. These observations are combined with an approximate static inventory of class attributes (Section 3.2.3) and a weighted type-selection mechanism (Section 3.4) to guide subsequent input generation. The evaluation tunes the probability of proxied execution on a separate dataset (RQ1), compares four configurations (NoTypeHints, TypeHints, NoTypeHints-TypeTracing, TypeHints-TypeTracing) on 466 modules for branch coverage (RQ2) and mutation score (RQ3), and compares the inferred types against six non-LLM tools and GPT-4o-mini using TypeEvalPy (RQ4). The paper reports mean branch coverage of 71.0% for TypeHints-TypeTracing versus 67.7% for NoTypeHints, and parameter/return F1 scores of 0.215/0.387, and it claims improvements in coverage, mutation score, and type quality comparable to state-of-the-art approaches.

Significance. The core idea is timely and the RQ2 evidence is solid in its experimental design: 30 repeated runs per configuration, Mann-Whitney tests, Vargha-Delaney effect sizes, and a separate tuning dataset avoid the most common circularity pitfalls. The observation that type tracing improves coverage on 162 modules and worsens it on only 19 relative to NoTypeHints (Table 1) is a credible and practically relevant result. However, the type-quality comparison (RQ4) rests on a single stochastic Pynguin execution, and the wording of the headline coverage claim overstates what the relative-coverage metric shows. Both issues are load-bearing for the abstract and conclusions and need to be corrected before the paper can be accepted.

major comments (3)
  1. [Section 4.1.4, Tables 6 and 7] The RQ4 claim that type tracing produces type information of 'similar quality' to other state-of-the-art tools is not established because the comparison is based on a single Pynguin execution. Section 4.1.4 states that Pynguin was executed only once when answering RQ4, despite Pynguin being a stochastic evolutionary search with random type selection and a 5% probability of proxied execution. A single run provides no variance estimate, so the observed F1 differences (parameter 0.215 vs. HiTyper 0.104; return 0.387 vs. 0.194) may be within run-to-run noise. Repeated runs (e.g., 30, as in RQ2/RQ3) with confidence intervals, or a substantially weakened claim that does not appear in the abstract, are needed.
  2. [Abstract, Section 1, and Section 4.4, Table 2] The claim of 'up to 90.0% more branch coverage' (also given as 87.8% in the abstract) is not supported by the data. Table 2 reports a mean relative coverage of 90.0% for TypeHints-TypeTracing, where relative coverage is defined in Section 4.1.3 as (cov - min)/(max - min). That value means the configuration reaches 90% of the observed coverage range, not that it achieves 90% more coverage than a baseline. The actual mean branch coverage gain is 71.0% versus 67.7% for NoTypeHints (Section 4.4). The abstract and conclusions should be rephrased to report the relative-coverage metric accurately and the two numerical versions should be reconciled.
  3. [Section 4.5, Tables 4 and 5] The conclusion that 'using as much type information as possible yields the best results' for fault finding is not supported by the aggregate mutation scores. Table 4 shows TypeHints-TypeTracing with a mean mutation score of 24.2%, which is lower than TypeHints at 24.4%. Table 5 shows only a small positive effect size (0.508) for TypeHints-TypeTracing versus TypeHints, with 141 better but 137 worse modules. The paper should either report that the mutation-score benefit of combining type hints with type tracing is not consistently supported, or explain why the module-level effect size should be preferred over the mean mutation score for this conclusion.
minor comments (6)
  1. [Section 4.1.4 (RQ1 procedure)] The number of repetitions per probability setting in RQ1 is not stated explicitly. It should be confirmed that the 30-run policy described in Section 4.1.2 applies to every probability configuration used to tune the 5% value.
  2. [Abstract vs. full text] The abstract reports 'up to 87.8% more branch coverage,' while the introduction and conclusions report 'up to 90.0%.' These numbers should be unified and described as relative coverage rather than 'more branch coverage.'
  3. [Section 4.3, Figure 10] The differences in mean coverage across probability settings are very small (roughly 53.9% to 54.6%) and the reported effect sizes are around 0.5. A brief discussion of how stable the choice of 5% is across these near-ties would strengthen the RQ1 conclusion.
  4. [Section 4.1.2] The omega weights for type selection were chosen by hand without tuning. This is acknowledged as a limitation, but a sensitivity analysis of these weights would increase confidence in the RQ2 results.
  5. [Section 4.1.3] The definition of relative coverage is clear, but the paper should avoid using the word 'up to' together with a mean value; the 90.0% is a mean relative coverage, not a maximum increase.
  6. [Sections 4.4-4.6, general copyediting] There are several typos, including 'overead' in Section 4.4, 'qualitatiy' and 'constition' in Section 4.6, and 'Automatc' in the reference to Gruber et al. These should be corrected in a final copyedit.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the type-tracing evaluation is anchored to external ground truth and a separately tuned parameter, with only a minor non-load-bearing self-citation.

full rationale

The paper's derivation chain is self-contained and empirically anchored. RQ1 tuning of the proxy execution probability is conducted on DS1-Tuning and then fixed at 5% on DS2-Evaluation (Sections 4.1.1 and 4.3), so the headline coverage results are not fits of the tuned parameter. The type-selection weights in Sections 3.4 and 4.1.2 were chosen by hand ('we chose values that provided reasonable results for a manual small-scale evaluation'), not optimized against the reported RQ2 or RQ3 outcomes. Inferred type quality (RQ4) is measured against an external ground truth consisting of developer type hints extracted from DS2-Evaluation and typeshed (Sections 4.1.3 and 4.1.4), and the comparison uses external TypeEvalPy baseline tools; no inferred type is defined in terms of the target metric. The only notable weakness is that RQ4 executes Pynguin only once ('To ensure fairness toward the other tools, we executed Pynguin only once'), which is a statistical validity threat about variance rather than a circular reduction. The citations to the authors' previous Pynguin work describe the framework being extended, not a theorem that forces the results. No equation in the paper equals its input by construction, and no fitted parameter is relabeled as a prediction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The ObjectProxy and the trace-recording shim are implementation artifacts of the method, not unexplained entities. The central claim rests on transparency of proxies, the attribute mapping approximation, and the quality of the ground-truth annotations.

free parameters (5)
  • type_tracing_probability = 5%
    Probability of adding a proxied execution after a regular execution. Chosen as best on DS1-Tuning over 0%, 5%, ..., 100%; central to RQ2 experiments.
  • omega_ann = 10
    Weight for developer-annotated type when selecting a parameter type. Chosen manually, not tuned.
  • omega_none = 1
    Weight for None parameter type; manual choice.
  • omega_any = 5
    Weight for Any; manual choice.
  • omega_union = 10
    Weight for union of traced types; manual choice.
assumptions (4)
  • domain assumption ObjectProxy forwards all Python-level operations transparently, so proxies reveal the same usage as the wrapped object.
    Section 3.2.1 claims the proxy is indistinguishable in duck-typed code; Section 3.3.1 admits C-implemented methods can reject proxies, motivating separate proxied executions.
  • domain assumption isinstance checks are the dominant runtime type-check mechanism; type() checks are negligible.
    Section 3.2.2 explicitly forgoes type() support; Section 4.2.1 gives GitHub search counts 700k vs 90k as evidence.
  • domain assumption Static attribute collection from __init__ assignments and class vars(), plus the inheritance-based attribute mapping, approximates the attributes available on instances.
    Section 3.2.3 says Python can change object layout at runtime, so this is an approximation; inference quality depends on it.
  • domain assumption Existing type annotations and merged typeshed annotations are correct enough to act as ground truth.
    Used as ground truth in RQ4 and Table 3; Section 4.4 discusses alias-induced mismatches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combining Type Inference and Automated Unit Test Generation for Python." pith.science (2026). https://pith.science/paper/UVGVPYO2

@misc{pith2026250701477,
  author       = {Pith},
  title        = {Pith review of: Combining Type Inference and Automated Unit Test Generation for Python},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UVGVPYO2}},
  note         = {Machine review of arXiv:2507.01477}
}
read the original abstract

Automated unit test generation is an established research field that has so far focused on statically-typed programming languages. The lack of type information in dynamically-typed programming languages, such as Python, inhibits test generators, which heavily rely on information about parameter and return types of functions to select suitable arguments when constructing test cases. Since automated test generators inherently rely on frequent execution of candidate tests, we make use of these frequent executions to address this problem by introducing type tracing, which extracts type-related information during execution and gradually refines the available type information. We implement type tracing as an extension of the Pynguin test-generation framework for Python, allowing it (i) to infer parameter types by observing how parameters are used during runtime, (ii) to record the types of values that function calls return, and (iii) to use this type information to increase code coverage. The approach leads to up to 87.8 % more branch coverage, improved mutation scores, and to type information of similar quality to that produced by other state-of-the-art type-inference tools.

Figures

Figures reproduced from arXiv: 2507.01477 by the authors.

Figure 1
Figure 1. Simplified excerpt from py-backwards.transformers.class_without_bases. one with and one without type hints. Type hints are not only beneficial for type checking (e.g., statically checking whether functions are called with arguments of valid types) but also because they provide test generators with crucial information [27]. The version with type hints (Fig. 1b) allows a test generator for Python to inspect the parame… view at source ↗
Figure 2
Figure 2. A simple function to show the recording of argument interactions. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Pseudo code showing how our proxy implementation works. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: A (simplified) code snippet of how the shim for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Example for mapping attribute names to classes which have that attribute. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Example for a native method not accepting a proxied argument. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: A code example showing the original test case in function [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: An example of how types can affect the discovery of mutants regardless of the code coverage. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Prompt used for the GPT-4o-mini model. We used Pynguin’s implementation of the DynaMOSA [37] algorithm for our executions, which was effective for test generation in Pynguin, along with the same parameter settings used in previous work [27]. We chose the weights for th…
Figure 10
Figure 10. Figure 10: Mean branch coverage after 600 s on the DS1-Tuning dataset for different probabilities of proxied execution. 4.3 RQ1: Type Tracing Probability We executed Pynguin on DS1-Tuning as outlined in Section 4.1.4, varying the probability of proxied execution (see Section 3.3…
Figure 11
Figure 11. Figure 11: Development of the branch coverage of the four configurations over the algorithm runtime of [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Simplified excerpt from py-backwards.transformers.base where the expected parameter type is known but Pynguin fails to instantiate the required object. def foo ( a : int) : if isinstance (a , str) : raise ValueError () [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: A code example showing a function that expects an [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: A code example showing a type alias along with a function that uses the alias. [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: A method requiring an object with an endswith attribute, such as a string, as input to not raise an AttributeError [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Type Score (MATCH over ALL) against average code coverage. The correlation line is shown in red. The correlation is [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 18
Figure 18. Figure 18: gpt-4o-mini is not able to infer the correct type for the node parameter of the visit_import method, while Pynguin infers the correct type ast.Import among other possible types. def get_domain () -> Optional [ str ]: """ Returns the domain for the current VCS : return…
Figure 19
Figure 19. Figure 19: Pynguin is the only tool that infers the correct return type str | None of the method get_domain. process is not triggered. While such edge cases exist, the overall correlation confirms that coverage drives type quality, which is best understood through a qualitative …
Figure 20
Figure 20. Figure 20: Pynguin fails to overcome the input verification of convert_path if os.sep == '/' which is not dependent on the input type or value. def optimizeConfigs ( self , interpreter : ATNSimulator ) : if self . readonly : raise IllegalStateException ( " This ␣ set ␣ is ␣ read…
Figure 21
Figure 21. Figure 21: Pynguin infers None as the return type of the optimizeConfigs method, while the ground truth is ASTNSimulator. generate a test that achieves higher coverage and it correctly inferred the type. In contrast, gpt-4o-mini fails to identify the correct type in this example…
Figure 22
Figure 22. Figure 22: Type counts of the most frequent types. These successes and failures raise a more fundamental question about what constitutes a ‘correct’ type in a dynamic context. We designed type tracing to find any parameter type that allows for successful execution, which may not…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 30 canonical work pages

  1. [1]

    Andrews, Lionel C

    James H. Andrews, Lionel C. Briand, and Yvan Labiche. 2005. Is Mutation an Appropriate Tool for Testing Experiments?. In International Conference on Software Engineering (ICSE) . ACM, 402–411. doi:10.1145/1062455.1062530

    arXiv 2005

  2. [2]

    Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18, 3 (2013), 594–623. doi:10.1007/s10664-013-9249-9

    doi:10.1007/s10664-013-9249-9 2013

  3. [3]

    Stefan Bucur, Johannes Kinder, and George Candea. 2014. Prototyping symbolic execution engines for interpreted languages. SIGARCH Comput. Archit. News 42, 1 (Feb. 2014), 239–254. doi:10.1145/2654822.2541977

    arXiv 2014

  4. [4]

    José Campos, Yan Ge, Nasser Albunian, Gordon Fraser, Marcelo Eler, and Andrea Arcuri. 2018. An empirical evaluation of evolutionary algorithms for unit test suite generation. Information & Software Technology 104 (2018), 207–235. doi:10.1016/j.infsof.2018.08.010

    doi:10.1016/j.infsof.2018.08.010 2018

  5. [5]

    Luca Cardelli. 2004. Type Systems

    work page 2004

  6. [6]

    Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet?. In International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, 429–440. doi:10.1109/ASE.2015.89

    doi:10.1109/ase.2015.89 2015

  7. [7]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre- trained Large Language Models and mutation testing. Information and Software Technology 171 (July 2024), 107468. doi:10.1016/j.infsof.2024.107468 Manuscript submitted to ACM Combining Type Inference and Automated Unit ...

    arXiv 2024

  8. [8]

    Xuefeng Ding, Wanyu Huang, Ying Liu, Chen Wantao, and Ding Xuyang. 2016. Dynamic Symbolic Execution Tool for Python Programs. In 2016 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS) . 212–217. doi:10.1109/ICITBS.2016.88

    work page doi:10.1109/icitbs.2016.88 2016

Show all 65 references
  1. [9]

    Dwyer, and Mary Lou Soffa

    Swaroopa Dola, Matthew B. Dwyer, and Mary Lou Soffa. 2021. Distribution-Aware Testing of Neural Networks Using Generative Models. In International Conference on Software Engineering (ICSE) . IEEE, 226–237. doi:10.1109/ICSE43902.2021.00032

    2021

  2. [10]

    Nicolas Erni, Al-Ameen Mohammed Ali Mohammed, Christian Birchler, Pouria Derakhshanfar, Stephan Lukasczyk, and Sebastiano Panichella. 2024. SBFT Tool Competition 2024 - Python Test Case Generation Track. CoRR abs/2401.15189 (2024). arXiv:2401.15189

    2024 arXiv

  3. [11]

    Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE Transactions on Software Engineering 39, 2 (2013), 276–291. doi:10.1109/ TSE.2012.14

    2013

  4. [12]

    Liang Gong, Michael Pradel, Manu Sridharan, and Koushik Sen. 2015. DLint: Dynamically Checking Bad Coding Practices in JavaScript. In International Symposium on Software Testing and Analysis (ISSTA) . ACM, 94–105. doi:10.1145/2771783.2771809

    2015

  5. [13]

    Luca Di Grazia and Michael Pradel. 2022. The Evolution of Type Annotations in Python: An Empirical Study. InJoint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE) . ACM, 209–220. doi:10.1145/3540250.3549114

    2022

  6. [14]

    Martin Gruber, Muhammad Firhard Roslan, Owain Parry, Fabian Scharnböck, Phil McMinn, and Gordon Fraser. 2024. Do Automatc Test Generation Tools Generate Flaky Tests. In International Conference on Software Engineering (ICSE) . ACM, 47:1–47:12. doi:10.1145/3597503.3608138

    2024

  7. [15]

    Mostafa Hassan, Caterina Urban, Marco Eilers, and Peter Müller. 2018. MaxSMT-Based Type Inference for Python 3. In International Conference on Computer Aided Verification (CA V) (Lecture Notes in Computer Science, Vol. 10982). Springer, 12–19. doi:10.1007/978-3-319-96142-2_2

    2018 doi

  8. [16]

    Hellendoorn, Christian Bird, Earl T

    Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. 2018. Deep Learning Type Inference. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE) . ACM, 152–162. doi:10.11...

    2018 doi

  9. [17]

    Alex Holkner and James Harland. 2009. Evaluating the dynamic behaviour of Python applications. InAustralasian Computer Science Conference (ACSC) (CRPIT, Vol. 91). Australian Computer Society, 17–25. http://crpit.scem.westernsydney.edu.au/abstracts/CRPITV91Holkner.html

    2009

  10. [18]

    Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62

    2011 doi

  11. [19]

    Ernst, Reid Holmes, and Gordon Fraser

    René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In International Symposium on Foundations of Software Engineering (FSE) . ACM, 654–665. doi:10.1145/26358...

    2014

  12. [20]

    Sebastian Kleinschmager, Stefan Hanenberg, Romain Robbes, Éric Tanter, and Andreas Stefik. 2012. Do Static Type Systems Improve the Maintain- ability of Software Systems? An Empirical Study. In International Conference on Program Comprehension (ICPC) . IEEE Computer Society, 1...

    2012

  13. [21]

    Combining Type Inference and Automated Unit Test Generation for Python

    Lukas Krodinger, Stephan Lukasczyk, and Gordon Fraser. 2025. Artifact for the paper "Combining Type Inference and Automated Unit Test Generation for Python" submitted to TOSEM 2025 . doi:10.5281/zenodo.15788696

    2025 doi

  14. [23]

    Li Li, Jiawei Wang, and Haowei Quan. 2022. Scalpel: The Python Static Analysis Framework. CoRR abs/2202.11840 (2022). arXiv:2202.11840

    2022 arXiv

  15. [24]

    Yun Lin, You Sheng Ong, Jun Sun, Gordon Fraser, and Jin Song Dong. 2021. Graph-based Seed Object Synthesis for Seach-Based Unit Testing. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE) . A...

    2021

  16. [25]

    Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: Automated Unit Test Generation for Python. In International Conference on Software Engineering Companion (ICSE Companion) . IEEE/ACM, 168–172. doi:10.1145/3510454.3516829

    2022

  17. [26]

    Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2020. Automated Unit Test Generation for Python. In International Symposium on Search Based Software Engineering (SSBSE) (Lecture Notes in Computer Science, Vol. 12420) . Springer, 9–24. doi:10.1007/978-3-030-59762-7_2

    2020 doi

  18. [27]

    Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2023. An empirical study of automated unit test generation for Python. Empirical Software Engineering 28, 2 (2023), 36:1–36:46. doi:10.1007/s10664-022-10248-w

    2023 doi

  19. [28]

    Donaldson

    David MacIver and Alastair F. Donaldson. 2020. Test-Case Reduction via Test-Case Generation: Insights from the Hypothesis Reducer (Tool Insights Paper). In European Conference on Object-Oriented Programming (ECOOP) (Leibnitz International Proceedings in Informatics (LIPIcs), V...

    2020 doi

  20. [29]

    David MacIver and Zac Hatfield-Dodds. 2019. Hypothesis: A new approach to property-based testing. Journal of Open Source Software 4, 43 (2019),

    2019

  21. [30]

    Magnus Madsen. 2015. Static Analysis of Dynamic Languages . phdthesis

    2015

  22. [31]

    Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. 2019. NL2Type: Inferring JavaScript Function Types from Natural Language Information. In International Conference on Software Engineering (ICSE) . IEEE/ACM, 304–315. doi:10.1109/ICSE.2019.00045

    2019

  23. [32]

    Mann and Donald R

    Henry B. Mann and Donald R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.The Annals of Mathematical Statistics 18, 1 (1947), 50–60. doi:10.1214/aoms/1177730491

    1947

  24. [33]

    Nevena Milojkovic, Mohammad Ghafari, and Oscar Nierstrasz. 2017. It’s Duck (Typing) Season!. In International Conference on Program Comprehen- sion (ICPC). IEEE Computer Society, 312–315. doi:10.1109/ICPC.2017.10 Manuscript submitted to ACM 36 Lukas Krodinger, Stephan Lukasczy...

    2017 doi

  25. [34]

    Mir, Evaldas Latoskinas, Sebastian Proksch, and Georgios Gousios

    Amir M. Mir, Evaldas Latoskinas, Sebastian Proksch, and Georgios Gousios. 2022. Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python. In International Conference on Software Engineering (ICSE) . ACM, 2241–2252. doi:10.1145/3510003.3510124

    2022

  26. [35]

    Milos Ojdanic, Aayush Garg, Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. 2023. Syntactic Versus Semantic Similarity of Artificial andReal Faults in Mutation Testing Studies.IEEE Transactions on Software Engineering 49, 7 (2023), 3922–3938. doi:10.1109/TS...

    2023

  27. [36]

    Lahiri, Michael D

    Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In International Conference on Software Engineering (ICSE) . IEEE Computer Society, 75–84. doi:10.1109/ICSE.2007.37

    2007 doi

  28. [37]

    Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets. IEEE Transactions on Software Engineering 44, 2 (2018), 122–158. doi:10.1109/TSE.2017.2663435

    2018

  29. [38]

    Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are Mutation Scores Correlated with Real Fault Detection?. In International Conference on Software Engineering (ICSE) . ACM, 537–548. doi:10.1145/3180155.3180183

    2018

  30. [39]

    Zvonimir Pavlinovic. 2019. Leveraging Program Analysis for Type Inference . phdthesis

    2019

  31. [40]

    Karl Pearson. 1895. Note on Regression and Inheritance in the Case of Two Parents. In Proceedings of the Royal Society of London , Vol. 58. 240–242

  32. [41]

    Yun Peng, Cuiyun Gao, Zongjie Li, Bowei Gao, David Lo, Qirun Zhang, and Michael Lyu. 2022. Static Inference Meets Deep Learning: A Hybrid Type Inference Approach for Python. In International Conference on Software Engineering (ICSE) . ACM, 2019–2030. doi:10.1145/3510003.3510038

    2022

  33. [42]

    Yun Peng, Chaozhen Wang, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. 2023. Generative Type Inference for Python. In International Conference on Automated Software Engineering (ASE) . IEEE, 988–999. doi:10.1109/ASE56229.2023.00031

    2023

  34. [43]

    Mauro Pezzè and Michal Young. 2007. Software testing and analysis - process, principles and techniques . Wiley

    2007

  35. [44]

    Benjamin C. Pierce. 2002. Types and Programming Languages. MIT Press

    2002

  36. [45]

    Juan Altmayer Pizzorno and Emery D. Berger. 2024. CoverUp: Coverage-Guided LLM-Based Test Generation. doi:10.48550/arXiv.2403.16218 arXiv:2403.16218

    2024 doi arXiv

  37. [46]

    Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chendra. 2020. TypeWriter: Neural Type Prediction with Search-Based Validation. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE) . AC...

    2020

  38. [47]

    Milanova, Martin Hirzel, and Julian Dolby

    Ingkarat Rak-amnouykit, Daniel McCrevan, Ana L. Milanova, Martin Hirzel, and Julian Dolby. 2020. Python 3 Types in the Wild: A Tale of Two Type Systems. In ACM SIGPLAN International Symposium on Dynamic Languages (DLS) . ACM, 57–70. doi:10.1145/3426422.3426981

    2020

  39. [48]

    Big Code

    Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from “Big Code”. In Symposium on Principles of Programming Languages (POPL). ACM, 111–124. doi:10.1145/2676726.2677009

    2015

  40. [50]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proceedings of the ACM on Software Engineering 1, FSE (Ju...

    2024 doi

  41. [51]

    Samir Sapra, Marius Minea, Sagar Chaki, Arie Gurfinkel, and Edmund M. Clarke. 2013. Finding Errors in Python Programs Using Dynamic Symbolic Execution. In IFIP International Conference on Testing Software and Systems (Lecture Notes in Computer Science, Vol. 8254) . Springer, 2...

    2013 doi

  42. [52]

    Sina Shamshiri, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges. In International Conference on Automated Software Engineering (ASE) . ...

    2015 doi

  43. [53]

    Siek and Walid Taha

    Jeremy G. Siek and Walid Taha. 2007. Gradual Typing for Objects. In European Conference on Object-Oriented Programming (ECOOP) (Lecture Notes in Computer Science, Vol. 4609). Springer, 2–27. doi:10.1007/978-3-540-73589-2_2

    2007 doi

  44. [54]

    Dimitri Michel Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2022. Guess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference. In International Symposium on Search Based Software Engineering (SSBSE) (Lecture Notes in Computer Sci...

    2022 doi

  45. [56]

    Daniel Trübenbach, Sebastian Müller, and Lars Grunske. 2022. A Comparative Evaluation on the Quality of Manual and Automatic Test Case Generation Techniques for Scientific Software — A Case Study of a Python Project for Material Science Workflows. InInternational Workshop on S...

    2022

  46. [57]

    András Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. journaltitle of Educational and Behavioral Statistics 25, 2 (2000), 101–132. doi:10.3102/10769986025002101

    2000 doi

  47. [58]

    Mir, Li Li, and Eric Bodden

    Ashwin Prasad Shivarpatna Venkatesh, Samkutty Sabu, Jiawei Wang, Amir M. Mir, Li Li, and Eric Bodden. 2023. TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference. CoRR abs/2312.16882 (2023). arXiv:2312.16882

    2023 arXiv

  48. [59]

    Ashwin Prasad Shivarpatna Venkatesh, Jiawei Wang, Li Li, and Eric Bodden. 2023. Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis. In International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 391–401. doi:10.1109/...

    2023

  49. [60]

    Păsăreanu, and Sarfraz Khurshid

    Willem Visser, Corina S. Păsăreanu, and Sarfraz Khurshid. 2004. Test Input Generation with Java PathFinder. InInternational Symposium on Software Testing and Analysis (ISSTA). ACM, 97–107. doi:10.1145/1007512.1007526

    2004

  50. [61]

    Stefan Wappler and Frank Lammermann. 2005. Using Evolutionary Algorithms for the Unit Testing of Object-Oriented Software. In Annual Conference on Genetic and Evolutionary Computation (GECCO) . ACM, 1053–1060. doi:10.1145/1068009.1068187

    2005

  51. [62]

    Jiayi Wei, Greg Durrett, and Isil Dillig. 2023. TypeT5: Seq2seq Type Inference using Static Analysis. doi:10.48550/arXiv.2303.09564 arXiv:2303.09564 [cs]

    2023 doi arXiv

  52. [63]

    Jifeng Wu and Caroline Lemieux. 2024. QuAC: Quick Attribute-Centric Type Inference for Python. Reproduction Package for Article ‘QuAC: Quick Attribute-Centric Type Inference for Python‘ 8, OOPSLA2 (Oct. 2024), 343:2040–343:2069. doi:10.1145/3689783

    2024 doi

  53. [64]

    Danni Xiao, Yimeng Guo, Yanhui Li, and Lin Chen. 2024. Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical Study. In Proceedings of the 15th Asia-Pacific Symposium on Internetware (Internetware ’24) . Association for Computing Machinery, New Y...

    2024

  54. [65]

    Yanyan Yan, Yang Feng, Hongcheng Fan, and Baowen Xu. 2023. DLInfer: Deep Learning with Static Slicing for Python Type Inference. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, Melbourne, Australia, 2009–2021. doi:10.1109/ICSE48619.2023.00...

    2023

  55. [66]

    Chen Yang, Junjie Chen, Bin Lin, Jianyi Zhou, and Ziqi Wang. 2024. Enhancing LLM-based Test Generation for Hard-to-Cover Branches via Program Analysis. doi:10.48550/arXiv.2404.04966 arXiv:2404.04966

    2024 doi arXiv

  56. [67]

    Ruofan Yang, Xianghua Xu, and Ran Wang. 2025. LLM-enhanced evolutionary test generation for untyped languages. Automated Software Engineering 32, 1 (Feb. 2025), 20. doi:10.1007/s10515-025-00496-7 Manuscript submitted to ACM

    2025 doi

  57. [1891]

    doi:10.21105/joss.01891

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.