REVIEW 5 major objections 5 minor 22 references
Can LLMs Test Terminal User Interfaces?
T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Automated TUI testing is feasible but far from solved, a 197-app study finds.
desk verdict First real TUI testing benchmark with a plausible core finding; the crash oracle needs an independent audit before the numbers are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a headless TUI-testing toolchain: each application is packaged as an instrumented container image that runs in a pseudo-terminal, the exploration framework issues raw key, mouse, and resize events and records the rendered character grid after each action, and the coverage tool merges native line coverage with an experimental widget-coverage signal. The decisive measurement device is the content-aware crash oracle: instead of counting non-zero exits, it scans the post-termination terminal screen for tracebacks, panics, or fatal signals and excludes harness and environment noise, with all ambiguous cases adjudicated by two independent labelers. The other load-bearing mech
What would settle it
Re-label the 1,033 raw crash events with a different oracle: have the maintainers of the affected applications confirm or reject each of the 179 'valid' faults and independently judge the 233 ambiguous exits. If most of the 179 turn out to be expected behavior, or if random's per-interaction rate rises to LLM-guided levels, the central comparison collapses.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that automated TUI testing is feasible but far from solved. Driving 197 real terminal-interface applications through a pseudo-terminal under equal 600-second budgets, the study finds that no frontier LLM dominates, that random exploration's per-run crash lead is a throughput artifact rather than evidence of better exploration, and that per interaction LLM guidance is roughly an order of magnitude more crash-efficient (2.41 versus 0.19 valid faults per 1,000 steps) and uniquely reaches input-gated faults. The largest practical lever is automatically deriving launch inputs, which rescues applications that otherwise exit before their interface appe
Load-bearing premise
The crash numbers rest on the assumption that the screen-scraping classifier and the manual review correctly separate genuine TUI crashes from harmless exits like usage messages, Ctrl+C, and tooling failures.
Editorial extensions
If this is right
- Future TUI test oracles should classify crashes from rendered terminal output, since 82% of raw non-zero exits in this study were noise.
- Comparisons of LLM-driven and random TUI testing should report faults per unit of interaction, not only per run, because time-equalized budgets give random an order-of-magnitude throughput advantage.
- Automatically deriving launch inputs and fixtures is the highest-leverage improvement and should be treated as a distinct subproblem in TUI testing.
- Line coverage should not be the primary proxy for TUI test effectiveness; a criterion keyed to reachable interactive states is the paper's suggested direction.
- Hybrid testers that use LLMs to unlock state and random input to stress it follow directly from the complementary fault sets observed.
Reading between the lines
- The coverage-fault decoupling likely generalizes to any crash-terminated interactive testing regime: a crash caps the coverage that session can accrue, so the runs that find faults are penalized on coverage. The paper's session-truncation reasoning gives a testable prediction for GUI fuzzing as well.
- The flat widget-coverage ordering across strategies suggests that how an explorer chooses the next action matters less than whether it can get the application into meaningful states; a benchmark variant measuring coverage of reachable interactive states rather than lines could test this directly.
- Because per-step LLM calls dominate throughput, a natural next experiment is a cheaper classify-then-act policy that uses one model call to choose among a small menu of actions, testing whether it preserves the per-interaction efficiency while narrowing the throughput gap.
- The released framework could be extended to non-crash faults by differencing rendered screens across sessions, giving the domain ground truth for misrenderings and state-transition bugs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents the first cross-framework empirical study of automated TUI testing. It mines 197 real-world TUI applications (ratatui/Rust, bubbletea/Go, textual/Python, ink/TypeScript), packages them as instrumented Docker images, and compares four LLM-driven exploration settings against a random baseline under equalized 600-second wall-clock budgets. The authors also survey existing TUI test suites, finding that interface-level testing is scarce and often non-interactive. They report three main results: no LLM dominates; random exploration wins per-run crash yield only because of higher throughput, while LLM guidance is more efficient per interaction and reaches input-gated faults; and line coverage is a poor proxy for crash discovery. The paper releases the tuicov and tuibot toolchains.
Significance. If the results hold, this is a valuable contribution. It provides the first dedicated benchmark and infrastructure for TUI testing, a domain that has been largely bypassed by CLI and GUI techniques. The paper is commendably honest about confounds: it uses equal wall-clock budgets, normalizes crash finding by interaction, reports a matched-step truncation check, and adopts a rendered-screen crash oracle rather than raw exit codes. The conclusion that coverage and fault-finding decouple in this setting, if supported, challenges a widely used proxy for test effectiveness. The released tools (tuicov, tuibot) are an additional strength, as is the explicit attention to measurement validity. The central claims are plausible but rest on measurement steps that need stronger validation before the headline numbers can be taken as settled.
major comments (5)
- [§3.4, Tables 6-7] The crash oracle is load-bearing for the paper's central quantitative claims, and its validation is incomplete. The classifier hard-excludes 'coverage-runtime emit failures' as noise, but no breakdown of how many of the 849 noise events fall in this bucket is given, and these events are not part of the 233 ambiguous events that were manually adjudicated. Because a genuine panic or abort can prevent the coverage runtime from flushing cleanly, hard-excluding this bucket risks systematically discarding real faults. With random's headline yield being 30 of 197 runs and LLM-guided-derived's 9.0%, a small number of reclassifications can flip the per-run ranking. Please report the size of the cov-emit bucket, perform a sensitivity analysis (e.g., reclassify 1-10% of these events as valid and recompute Table 7), and release the per-event triage labels. The 93% raw agreement is useful but is not
- [§5.4, RQ3] The claim that 'line coverage is a poor proxy for crash-finding' is supported only by four setting-level aggregates: the line-coverage column in Table 4 versus the crash columns in Table 7. This is not a statistical test of a relationship; it is a post-hoc reading of four points. The session-truncation confound is asserted but not modeled. A per-app or per-run regression of valid faults on coverage, controlling for steps taken and application, with confidence intervals, is needed. Without such an analysis, the abstract and conclusion overstate what the data show.
- [§5.2, Table 7, footnote 8] The matched-step truncation check is under-specified and the uncertainty around the throughput/efficiency numbers is not reported. 'Truncating every arm to a matched step budget, at k≤12 steps' does not state how steps are counted across the episodes within a run, how random's many relaunches are handled, or how many runs remain in each arm. The paper also reports point estimates (15.2%, 2.41, etc.) without confidence intervals or significance tests. Given that random's lead is 30 vs. ~17 runs, the variance across apps and seeds is likely material. In addition, footnote 8 says random has 197 runs, while §4.1 says every configuration is repeated three times per app; this inconsistency must be resolved for the reader to interpret the denominators.
- [§5.3, RQ1] The paper's unique-fault counts and complementarity claims depend on '47 unique fault signatures' and on categorizations such as '15 apps reached only by an LLM arm' and '10 random-only'. No definition or procedure is given for what counts as a unique signature, how signatures are deduplicated across apps and runs, or how the only-by assignments are determined. Without a reproducible deduplication protocol, the RQ1 comparison (Gemini 27, Claude 24, GPT-5.5 18, DeepSeek 16) and the 'strategies are complementary' argument are not independently checkable.
- [§4.1, §5.2, Table 4] The claim that automatically deriving launch inputs is the largest practical lever is central to the paper, but §4.1 does not describe the derivation procedure: what inputs are derived, which applications receive them, how success is measured, or what happens when derivation fails. The coverage gain in Table 4 (LLM-guided-derived 30.4% vs. LLM-guided 26.4%) could be due to a few high-impact applications or to incidental differences in prompt design rather than to a general effect. Please provide a detailed description of the derivation mechanism and a per-application breakdown of its effect.
minor comments (5)
- [Table 4] The row 'LLM-guided-derived' has '30.475,988' — a missing space between the percentage and the token count; also the 'Steps' column in Table 4 (means) and the 'Med. steps' column in Table 7 (medians) are not labeled as such in the text, which is confusing.
- [§4.1 / footnote 8] As noted in Major Comment 3, the definition of a 'run' is inconsistent: §4.1 says metrics are averaged over three runs per app, while footnote 8 says random has 197 runs. Please clarify whether 'run' means an app-level aggregate or a single session.
- [Table 6 and §3.4] Table 6 labels the noise bucket 'Go cov-emit', but the text says the largest category is 'coverage-runtime emit failure' across runtimes. These should be made consistent, and the size of each noise sub-bucket should be reported.
- [§2] The survey classification is described as 'LLM-assisted and verified by manual inspection,' but the verification protocol is not described (sample size, agreement, how disagreements were resolved). A brief description would help readers assess the survey numbers.
- [§5.2] The choice of k≤12 for the matched-step truncation check is not justified. If the goal is to compare arms at the LLM arms' median, the paper should also report results at several other budgets (e.g., 5, 10, 20) to show the conclusion is not an artifact of a single threshold.
Circularity Check
No circularity: benchmark comparisons rest on separately measured coverage and manually adjudicated crash events; the only author-overlap citation is a non-load-bearing related-work reference.
full rationale
The paper's central claims are not definitionally tied to their inputs. The 197-app benchmark, four exploration settings, and the random baseline are externally comparable; measurements (line/widget coverage, rendered terminal states, raw crash events) are collected by independent tooling and then reduced to a valid-fault set by a rendered-screen classifier plus manual adjudication (Section 3.4). That classifier is a measurement premise, not a circular step: the paper does not define 'valid fault' as 'what the prediction wants'; it reports raw exits, noise buckets, and 233 individually adjudicated ambiguous events, and the 179 valid faults are external observations against which technique rankings are computed. No parameter is fitted to the target result, and no prediction is constructed from the outcome it claims. RQ3's coverage-fault decoupling is an empirical correlation over that adjudicated set, not an identity or a fitted proxy. The only author-overlap citation is [22] (Tian et al.) in Related Work, used merely as an example of an empirical 'how far are we' study; it does not supply a uniqueness theorem, an ansatz, or any load-bearing evidence. The manual adjudication's lack of kappa/third-party release is a reproducibility/validity threat, not circularity. Hence no circular step is present; score 0.
Assumptions & free parameters
free parameters (1)
- matched step budget k=12 =
12 steps (median of LLM-guided arms)
assumptions (4)
- domain assumption The four frameworks (ratatui, bubbletea, textual, ink) are the dominant TUI frameworks and representative of the TUI class.
- domain assumption A TUI's observable state is fully captured by the rendered character grid and raw events in a pseudo-terminal.
- domain assumption Crash validity can be determined from the post-termination rendered screen plus human adjudication.
- domain assumption Line and widget coverage instrumentation does not materially distort application behavior in ways that confound crash findings.
Cite this review
Pith. "Pith review of Can LLMs Test Terminal User Interfaces?." pith.science (2026). https://pith.science/paper/P73WH55B
@misc{pith2026260803743,
author = {Pith},
title = {Pith review of: Can LLMs Test Terminal User Interfaces?},
year = {2026},
howpublished = {\url{https://pith.science/paper/P73WH55B}},
note = {Machine review of arXiv:2608.03743}
}
read the original abstract
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil A. Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, and Stefan Wagner. Guidelines for empirical studies in software engin...
arXiv 2025
-
[2]
On the reliability of coverage- based fuzzer benchmarking
Marcel Böhme, László Szekeres, and Jonathan Metzman. On the reliability of coverage- based fuzzer benchmarking. InProceedings of the 44th International Conference on Software Engineering (ICSE), pages 1621–1633. ACM, 2022. doi: 10.1145/3510003.3510230
arXiv 2022
-
[3]
Bubble tea: A powerful little TUI framework
Charm. Bubble tea: A powerful little TUI framework. https://github.com/ charmbracelet/bubbletea, 2024. Accessed: 2026-06-24
work page 2024
-
[4]
Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. Automated test input generation for Android: Are we there yet? InProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 429–440. IEEE, 2015. doi: 10.1109/ASE.2015.89
-
[5]
Ink: React for interactive command-line apps
Vadim Demedes. Ink: React for interactive command-line apps. https://github.com/ vadimdemedes/ink, 2024. Accessed: 2026-06-24
work page 2024
-
[6]
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pages 423–435. ACM, 2023. doi: 10.1145/3597926.3598067
arXiv 2023
-
[7]
Coverage is not strongly correlated with test suite effectiveness
Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. InProceedings of the 36th International Conference on Software Engineering (ICSE), pages 435–445. ACM, 2014. doi: 10.1145/2568225.2568271
arXiv 2014
-
[9]
Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 1–12. ACM, 2024. doi: 10.1145/3597503.3639118
arXiv 2024
Show all 22 references
-
[10]
An empirical study of automated unit test generation for Python.Empirical Software Engineering, 28(2), 2023
Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. An empirical study of automated unit test generation for Python.Empirical Software Engineering, 28(2), 2023. doi: 10.1007/ s10664-022-10248-w
2023
-
[11]
Dynodroid: An input generation system for Android apps
Aravind Machiry, Rohan Tahiliani, and Mayur Naik. Dynodroid: An input generation system for Android apps. InProceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), pages 224–234. ACM, 2013. doi: 10.1145/2491411.2491450
2013
-
[12]
Sapienz: Multi-objective automated testing for Android applications
Ke Mao, Mark Harman, and Yue Jia. Sapienz: Multi-objective automated testing for Android applications. InProceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA), pages 94–105. ACM, 2016. doi: 10.1145/2931037.2931054. 16
2016
-
[13]
TUI Test: End-to-end terminal testing framework
Microsoft. TUI Test: End-to-end terminal testing framework. https://github.com/ microsoft/tui-test, 2024. Accessed: 2026-06-24
2024
-
[14]
Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults
Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. InProceedings of the 40th International Conference on Software Engineering (I...
2018
-
[15]
Ratatui: A Rust library to build rich terminal user interfaces
Ratatui Developers. Ratatui: A Rust library to build rich terminal user interfaces. https: //ratatui.rs, 2024. Accessed: 2026-06-24
2024
-
[16]
Deep reinforcement learning for black-box testing of Android apps.ACM Transactions on Software Engineering and Methodology, 31(4):1–29, 2022
Andrea Romdhana, Alessio Merlo, Mariano Ceccato, and Paolo Tonella. Deep reinforcement learning for black-box testing of Android apps.ACM Transactions on Software Engineering and Methodology, 31(4):1–29, 2022. doi: 10.1145/3502868
2022 doi
-
[17]
An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering, 50(1):85–105, 2024
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering, 50(1):85–105, 2024. doi: 10.1109/TSE.2023.3334955
2024
-
[18]
FLACOCO: Fault localization for Java based on industry-grade coverage.arXiv preprint arXiv:2111.12513, 2021
André Silva, Matias Martinez, Benjamin Danglot, Davide Ginelli, and Martin Monperrus. FLACOCO: Fault localization for Java based on industry-grade coverage.arXiv preprint arXiv:2111.12513, 2021
2021 arXiv
-
[19]
Relationships between test suites, faults, and fault detection in GUI testing
Jaymie Strecker and Atif Memon. Relationships between test suites, faults, and fault detection in GUI testing. In2008 1st International Conference on Software Testing, Verification, and Validation (ICST), pages 12–21. IEEE, 2008. doi: 10.1109/ICST.2008.26
2008 doi
-
[20]
Guided, stochastic model-based GUI testing of Android apps
Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. Guided, stochastic model-based GUI testing of Android apps. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), pages 245–256. AC...
2017
-
[21]
Textual: A Rapid Application Development framework for Python
Textualize. Textual: A Rapid Application Development framework for Python. https: //textual.textualize.io, 2024. Accessed: 2026-06-24
2024
-
[22]
Bissyandé
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F. Bissyandé. Is ChatGPT the ultimate programming assistant – how far is it?arXiv preprint arXiv:2304.11938, 2023
2023 arXiv
-
[23]
Cohen, and Atif M
Xun Yuan, Myra B. Cohen, and Atif M. Memon. GUI interaction testing: Incorporating event context.IEEE Transactions on Software Engineering, 37(4):559–574, 2011. doi: 10.1109/TSE. 2010.50. 17
2011 doi
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.