Pith. sign in

REVIEW 4 major objections 5 minor 90 references

ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ViseGPT claims that automatically turning a user's natural-language prompt into executable test cases, and showing the results in a Gantt chart, makes LLM-generated data-wrangling scripts faster and more reliable to debug than chat-only…

desk verdict A well-designed debugging tool with a promising interaction pattern, but the evidence for its headline efficiency claim is thinner than the abstract suggests. read the letter →

arxiv 2508.01279 v1 pith:HSCBLN6Z submitted 2025-08-02 cs.HC

Jiajun Zhu , Xinyu Cheng , Zhongsu Luo , Yunfan Zhou , Xinhuan Shu , Di Weng , Yingcai Wu This is my paper
classification cs.HC
keywords datawranglingscriptsLLM-generatedcodeunittestgenerationconstraintextractionGanttchartvisualizationdebugginghuman-AIinteractionpromptalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that the best way to check an LLM-generated data-wrangling script is not to make the user read and trace code, but to turn the user's own prompt into executable test cases the script must pass. ViseGPT extracts constraints from natural language, runs the generated script against test data, and visualizes which columns and steps fail in a Gantt chart. The motivation is practical: a formative study found that users wanted direct validation of whether a script matches their requirements, rather than just help understanding the script. If the approach works, it gives non-experts a low-barrier way to verify and iteratively repair AI-generated wrangling code.

What carries the argument

The load-bearing mechanism is a prompt-to-test pipeline: an analysis program hosted on Llama-3.3-70b parses the user's natural-language request into output constraints under eight categories — Type, Format, Range, Order, Exception (missing values), Unique, Forbidden Value, and Relation — and turns them into executable test cases, using static checks for some categories and LLM-based natural-language checks for others. On the display side, a tailored Gantt chart maps each script step to the columns it touches, color-codes pass and fail status, and lets users click through failing test cases and step-level data details; the test report can then be attached to the next LLM prompt for script regeneration.

What would settle it

Take a set of data-wrangling prompts with deliberately hidden requirements, have human annotators list the constraints each prompt entails, and compare ViseGPT's generated test cases against that list; if the generated tests regularly miss constraints or mark non-conforming output as passing, the tool's alignment guarantee fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, ViseGPT establishes that automatically generated unit tests, derived from constraints in the user prompt, can serve as the reliability verification layer for LLM-generated data-wrangling scripts. Because the tests are tied to the prompt rather than to hand-written expected outputs, they expose semantic mismatches such as wrong grouping columns, date-format drift, missing-value propagation, and non-compliant rounding. A user study with 18 participants reports that this approach significantly reduces debugging time on most tasks, improves success rates, and shifts users' iteration strategy toward attaching test reports to follow-up prompts.

Load-bearing premise

The whole debugging advantage rests on the LLM-based test case generator extracting from the user's prompt the constraints that actually matter; if those tests are wrong or incomplete, ViseGPT can display a green checkmark for a script that still does not match the request.

Editorial extensions

If this is right

  • A user can validate an LLM-generated script without reading every line, by checking whether each prompt-derived constraint passes at each column and step.
  • Failures that propagate through dependent columns become visible as chains of red rectangles, pointing toward the root-cause step rather than every downstream symptom.
  • After a failed test, the user can attach the report to the next prompt, so LLM-based script repair is driven by concrete failing cases rather than vague descriptions.
  • The same constraint taxonomy can be extended with user-defined test categories, since the system's backend is largely model-agnostic on the LLM side.
  • A fully green Gantt chart means all generated test cases passed, not that the script is provably correct against every possible interpretation of the prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the constraint-extraction step is accurate enough, the same prompt-to-tests pipeline could be applied to other LLM code outputs, such as SQL queries, spreadsheet formulas, or visualization specifications, where correctness is also a matter of satisfying semantic constraints.
  • The user study combines the test generator and the Gantt visualization in one condition; an obvious next experiment is to isolate the two to see which component carries the efficiency gain.
  • The eight-category constraint taxonomy sets the ceiling on what ViseGPT can detect, so adding user-defined or learned constraint categories would directly enlarge the space of bugs the tool can catch.
  • Because a green checkmark could mislead when extraction misses a hidden requirement, tools of this kind should surface test coverage or confidence estimates alongside pass-and-fail status.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents ViseGPT, a system that supports debugging of LLM-generated data-wrangling scripts by automatically extracting constraints from natural-language prompts, synthesizing test cases from those constraints, executing the tests against the generated script, and presenting pass/fail results in a Gantt-chart visualization. A formative study (N=8) motivates the design, and a within-subjects user study (N=18) compares ViseGPT with a chat-like Baseline across four debugging tasks. The paper reports higher success rates and lower completion times for ViseGPT on most tasks, along with significantly better UEQ ratings, and argues that the tool improves debugging efficiency, issue detection, and workflow experience.

Significance. If the central claims hold, ViseGPT is a useful contribution to HCI for LLM-generated code: it operationalizes unit-test-style validation for natural-language data-wrangling requests, provides a visual mapping of data-flow steps to test outcomes, and closes a debugging loop by feeding test reports back into the LLM. The paper's strengths include a formative study that grounds task selection, a user study with a Latin-square design and behavioral metrics, and an honest discussion of several limitations in Section 6.3. However, the load-bearing claim of 'significantly improved debugging efficiency' rests on descriptive statistics without inferential tests, and the core constraint-extraction/test-generation mechanism is never validated against human-annotated ground truth. Because ViseGPT's interface gives users an authoritative green/red checkmark, the trustworthiness of the automatically generated tests is central to the claimed benefit; this gap currently prevents the paper from supporting its strongest conclusions.

major comments (4)
  1. [§5.3.1, Table 2, Figure 5] The abstract and conclusion claim that ViseGPT 'significantly improves debugging efficiency,' but Section 5.3.1 reports no significance tests or effect sizes for completion times or success rates. The paper gives means (e.g., Task A 100% vs. 78%, Task B 100% vs. 33%) and a figure, but no p-values, confidence intervals, or paired-test results for these behavioral measures. Given the within-subjects design, the authors should report appropriate paired tests (e.g., Wilcoxon signed-rank for times, McNemar for binary success) and effect sizes. Without these, the word 'significantly' is unsupported.
  2. [§5.3.1, Table 2] The text states that Task C had 'comparable success rates' (56% vs. 22%), but a 34-percentage-point difference with only nine participants per condition is not self-evidently comparable, and the paper does not report a test for this difference. The subsequent narrative about the 6/9 participants who did not address the root cause is plausible, but the success-rate claim needs either a statistical test or more precise wording (e.g., 'not statistically distinguishable given the sample size').
  3. [§4.2, §4.5, §5.3.4] The central mechanism—automatic extraction of constraints from natural-language prompts and conversion into executable test cases—is never validated against human-annotated ground truth. The paper does not report precision or recall of extracted constraints, nor does it check whether the generated test cases would catch the seeded bugs when the scripts are correct. Since the test-case generator and the code agent are built on the same LLM family, systematic misinterpretations of a prompt could appear in both the script and the tests, making the green checkmark an unreliable oracle. Participants themselves expressed this concern (P3, Section 5.3.4). The authors should either supply an independent evaluation of extraction/test-case quality (e.g., human-annotated constraints, mutation-style bug injection) or substantially temper the claims about reliability and coverage.
  4. [§5.1, §6.3] The evaluation tasks are one-bug scripts whose violations are directly visible in output data (as acknowledged in Section 6.3: 'common violations ... that directly manifest in output data'). This limits generalization to prompts with implicit, ambiguous, or domain-specific constraints—precisely the hard cases discussed in Section 2.2 and Section 4.2. The paper should either add tasks with less explicit constraints or narrow the conclusion to the demonstrated scenario. The current abstract's general statement about 'comprehensive test cases' and 'reliability' overstates what the evidence supports.
minor comments (5)
  1. [Table 2 caption] '(Value): standard deviance' should read 'standard deviation.'
  2. [References] References [73] and [74] are identical (same title, authors, venue, and page numbers); one should be removed or replaced with the correct distinct source.
  3. [§6.3] The text says 'the four tasks (Section 6.1)' but the tasks are described in Section 5.1; the cross-reference should be corrected.
  4. [§6.1] The phrase 'infomation presentation medium' contains a typo; it should be 'information presentation medium.'
  5. [§5.3.2] The UEQ analysis is described as using paired t-tests (alpha 0.05), but the paper does not report the exact p-values or effect sizes for the six dimensions; adding these would make the claim of significant improvement more checkable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the user-study evidence is external, and the unvalidated constraint-extraction step is a validity gap rather than a definitional reduction.

full rationale

I find no circular step that reduces a prediction to its inputs by construction. ViseGPT's headline claim is an empirical user-study result: participants debugging LLM-generated scripts were faster and more successful with ViseGPT than with a matched Baseline, measured by completion time and success rate (Section 5.3.1). That evidence is external to the system's own definition of correctness and therefore not circular. The closest candidate is the automatic constraint-extraction component (Section 4.2), where the test cases are generated from the same natural-language prompt that drives script generation, and the same Llama-3.3-70b model hosts both the code agent and the test-case generator (Section 4.5). This creates a real shared-bias risk: an incomplete or misinterpreted constraint set can yield a false green checkmark, which participants themselves flagged (P3, Section 5.3.4). However, an unvalidated component is not circularity by the standard used here: the paper never equates the extracted constraints with ground truth, and it explicitly acknowledges coverage limits in Section 6.3 ('This may not fully capture nuanced or domain-specific requirements, leading to potential gaps in test coverage'). The constraint taxonomy is adapted from the authors' prior Ferry paper (Section 4.2, [47]), which is a self-citation, but the taxonomy is an input design choice rather than a load-bearing prediction, and it is not used to derive the user-study results. The paper also reports an honest negative result: Task C showed no significant improvement (ViseGPT M=625s vs. Baseline M=645s, success 56% vs. 22%), which is inconsistent with a system whose evaluation is forced by construction. Overall, the central claims are grounded in a comparative user study, and the weaknesses are validity threats, not circular reasoning.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central mechanism rests on several unverified assumptions: the constraint taxonomy is hand-designed, prompts are assumed to contain sufficient constraints, the LLM-generated test cases are assumed correct without external validation, and scripts are assumed linear. The paper acknowledges some of these limitations in Section 6.3.

free parameters (2)
  • Constraint taxonomy composition = 8 categories (Type, Format, Range, Order, Exception, Unique, Forbidden Value, Relation)
    Hand-selected by the authors, extending Ferry's categories with Unique, Forbidden Value, and Relation. There is no data-driven or theoretical justification, and these categories bound what test cases ViseGPT can generate, so they materially shape the system's behavior.
  • Default date format regex = \d{2}-\d{2}-\d{4}
    Chosen as the default validation regex when the user prompt does not specify a date format. In the usage scenario (Section 4.1), this default produced false test failures for yyyy-mm-dd data, requiring manual test case editing, showing it is an ad hoc choice.
assumptions (4)
  • domain assumption Data wrangling scripts can be decomposed into a linear sequence of atomic steps without branches or loops.
    Section 4.3 states ViseGPT focuses on sequential execution and does not support branching or loops; if real scripts contain such logic, the tool's step-based visualization and testing would be incomplete.
  • domain assumption The LLM-based analysis program extracts constraints from natural language prompts accurately enough to validate script correctness.
    Section 4.2 relies on the analysis program to derive output constraints; no precision or recall evaluation against human annotations is provided, and some participants (P3, Section 5.3.4) worried about undetected issues.
  • domain assumption LLM-generated test results can serve as ground truth for script correctness.
    The LLM code agent generates the script, the test case generator generates assertions, and LLMs are used for some constraint matching categories such as Format, Forbidden Value, and Relation; the validation loop is self-referential without an external oracle.
  • domain assumption User prompts contain sufficient and explicit constraints for generating meaningful test cases.
    The usage scenario (Section 4.1) shows Mary's prompt lacked a date format, causing the system to apply a default regex and generate false failures; the system then required manual test case customization, indicating prompts are often incomplete.
invented entities (1)
  • Constraint categories (Unique, Forbidden Value, Relation)
    purpose: Conceptual framework for converting natural language requirements into testable assertions on data columns.
    These categories are introduced as extensions to the same lab's Ferry taxonomy; they are not derived from data or external theory, and their adequacy is not benchmarked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts." pith.science (2026). https://pith.science/paper/HSCBLN6Z

@misc{pith2026250801279,
  author       = {Pith},
  title        = {Pith review of: ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSCBLN6Z}},
  note         = {Machine review of arXiv:2508.01279}
}
read the original abstract

Large language models (LLMs) enable the rapid generation of data wrangling scripts based on natural language instructions, but these scripts may not fully adhere to user-specified requirements, necessitating careful inspection and iterative refinement. Existing approaches primarily assist users in understanding script logic and spotting potential issues themselves, rather than providing direct validation of correctness. To enhance debugging efficiency and optimize the user experience, we develop ViseGPT, a tool that automatically extracts constraints from user prompts to generate comprehensive test cases for verifying script reliability. The test results are then transformed into a tailored Gantt chart, allowing users to intuitively assess alignment with semantic requirements and iteratively refine their scripts. Our design decisions are informed by a formative study (N=8) that explores user practices and challenges. We further evaluate the effectiveness and usability of ViseGPT through a user study (N=18). Results indicate that ViseGPT significantly improves debugging efficiency for LLM-generated data-wrangling scripts, enhances users' ability to detect and correct issues, and streamlines the workflow experience.

Figures

Figures reproduced from arXiv: 2508.01279 by the authors.

Figure 1
Figure 1. ViseGPT automatically extracts test cases from the user’s prompts for efficient and accuracy debugging. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A screenshot of the ViseGPT user interface. After users send a prompt to ViseGPT (A), the response content in the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Two issues in the usage scenario. (A) Format issues [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: ViseGPT Workflow. (A) Test Case Generatorgenerates test cases based on data constraints extracted from users’ [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Completion time in four tasks [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 34 canonical work pages

  1. [1]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2024. A3test: Assertion-augmented automated test case generation. Information and Software Technology 176 (2024), 107565. doi:10.1016/j.infsof.2024.107565 UIST ’25, September 28-October 1, 2025, Busan, Republic of Korea Zhu et al

    work page arXiv 2024

  2. [2]

    Aleftina Andrianova and Vladimir Itsykson. 2013. Generating unit tests using static analysis and contracts. InProceedings of Tools & Methods of Program Analysis. 83–88. doi:10.1109/TMPA.2013.7163725

    work page arXiv 2013

  3. [3]

    Anthropic. 2025. Claude 3.7 Sonnet and Claude Code. Retrieved February 25, 2025 from https://www.anthropic.com/news/claude-3-7-sonnet

    2025

  4. [4]

    Antoy, P

    S. Antoy, P. Forcheri, and M.T. Molfino. 1990. Specification-based code generation. In Proceedings of Annual Hawaii International Conference on System Sciences . 165–

    1990

  5. [5]

    R. Barták. 1999. Constraint Programming: What is Behind. In Proceedings of Workshop on Constraint Programming for Decision and Control . 7–15

    1999

  6. [6]

    Hongbo Chen, Yifan Zhang, Xing Han, Huanyao Rong, Yuheng Zhang, Tianhao Mao, Hang Zhang, XiaoFeng Wang, Luyi Xing, and Xun Chen. 2024. With- eredLeaf: Finding Entity-Inconsistency Bugs with LLMs. https://arxiv.org/abs/ 2405.01668

    work page arXiv 2024

  7. [7]

    Weihao Chen, Xiaoyu Liu, Jiacheng Zhang, Ian Iong Lam, Zhicheng Huang, Rui Dong, Xinyu Wang, and Tianyi Zhang. 2023. MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence. In Proceedings of Annual ACM Symposium on User Interface Software and Technology . Association for Computing Machinery. https://doi.org/10.1145/3586183.3606720

    arXiv 2023

  8. [8]

    Wei-Hao Chen, Weixi Tong, Amanda Case, and Tianyi Zhang. 2025. Dango: A Mixed-Initiative Data Wrangling System using Large Language Model. In Pro- ceedings of CHI Conference on Human Factors in Computing Systems . Association for Computing Machinery. https://doi.org/10.1145/3706598.3714135

    arXiv 2025

Show all 90 references
  1. [9]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Compan- ion Proceedings of ACM International Conference on the Foundations of Software Engineering. 572–576. doi:10.1145/3663529.3663801

    2024

  2. [10]

    Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, Jianye Hao, Hangyu Mao, and Fuzheng Zhang. 2025. SheetAgent: Towards a Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models. In Proceedings of ACM on Web Conference . Association ...

    2025

  3. [12]

    Tamraparni Dasu and Theodore Johnson. 2003. Exploratory Data Mining and Data Cleaning. John Wiley & Sons, Inc., New York, NY

    2003

  4. [13]

    A.M. Dean. 2001. Experimental Design: Overview. InInternational Encyclopedia of the Social & Behavioral Sciences . 5090–5096. doi:10.1016/B0-08-043076-7/00417-4

    2001 doi

  5. [14]

    Ximing Dong, Dayi Lin, Shaowei Wang, and Ahmed E. Hassan. 2024. A Frame- work for Real-time Safeguarding the Text Generation of Large Language Model. https://arxiv.org/abs/2404.19048

    2024 arXiv

  6. [15]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. Code- Score: Evaluating Code Generation by Learning Code Execution. ACM Transac- tions on Software Engineering and Methodology 34, 3 (2025). doi:10.1145/3695991

    2025 doi

  7. [16]

    Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Inter- pretable Machine Learning. https://arxiv.org/abs/1702.08608

    2017 arXiv

  8. [17]

    Guo, Robert DeLine, and Sumit Gulwani

    Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, and Sumit Gulwani. 2020. Wrex: A Unified Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists. In Proceedings of CHI Conference on Human Factors in Computing Systems . Association for Compu...

    2020

  9. [18]

    David Evans, John Guttag, James Horning, and Yang Meng Tan. 1994. LCLint: a tool for using specifications to check code. ACM SIGSOFT Software Engineering Notes (1994), 87–96. doi:10.1145/195274.195297

    1994

  10. [19]

    Yingchaojie Feng, Xingbo Wang, Bo Pan, Kam Kwai Wong, Yi Ren, Shi Liu, Zihan Yan, Yuxin Ma, Huamin Qu, and Wei Chen. 2024. XNLI: Explaining and Diagnosing NLI-Based Visual Data Analysis. IEEE Transactions on Visualization and Computer Graphics 30, 7 (2024), 3813–3827. doi:10.1...

    2024

  11. [20]

    Gordon, Carina Negreanu, Nadia Polikarpova, Advait Sarkar, and Benjamin Zorn

    Kasra Ferdowsi, Jack Williams, Ian Drosos, Andrew D. Gordon, Carina Negreanu, Nadia Polikarpova, Advait Sarkar, and Benjamin Zorn. 2023. COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code. In Proceedings of IEEE Symposium on Visual Languages and Human-Centr...

    2023

  12. [21]

    Afonso Fontes, Gregory Gay, Francisco Gomes de Oliveira Neto, and Robert Feldt

  13. [22]

    Martin Fowler and Matthew Foemmel. 2006. Continuous integration

    2006

  14. [23]

    Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology 24, 2, Article 8 (2014). doi:10.1145/2685612

    2014 doi

  15. [24]

    Google. 2024. Gemini Advanced: Release updates. Retrieved December 17, 2024 from https://gemini.google.com/updates

    2024

  16. [25]

    Vitor Guilherme and Auri Vincenzi. 2023. An initial investigation of ChatGPT unit test generation capability. In Proceedings of Brazilian Symposium on Systematic and Automated Software Testing. 15–24. doi:10.1145/3624032.3624035

    2023

  17. [26]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions ...

    2025 doi

  18. [27]

    Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. 2024. Semantic-guided Prompt Organization for Universal Goal Hijacking against LLMs. https://arxiv.org/abs/2405.14189

    2024 arXiv

  19. [28]

    Anita M. Hubley. 2017. Expanding Views on Response Processes Evidence for Validity. Measurement: Interdisciplinary Research and Perspectives (2017), 140–142. doi:10.1080/15366367.2017.1404366

    2017

  20. [29]

    IEEE. 1990. IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12-1990 (1990), 1–84. doi:10.1109/IEEESTD.1990.101064

    1990

  21. [30]

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2024. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. https://arxiv.org/ abs/2410.00752

    2024 arXiv

  22. [31]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards Mitigating LLM Hallucination via Self Reflection. In Proceedings of Findings of the Association for Computational Linguistics: EMNLP . 1827–1843. doi:10.18653/v1/2023.findings-emnlp.123

    2023 doi

  23. [32]

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Pooven- dran. 2025. ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates. https://arxiv.org/abs/2406.12935

    2025 arXiv

  24. [33]

    Sean Kandel, Jeffrey Heer, Catherine Plaisant, Jessie Kennedy, Frank van Ham, Nathalie Henry Riche, Chris Weaver, Bongshin Lee, Dominique Brodbeck, and Paolo Buono. 2011. Research directions in data wrangling: Visualizations and transformations for usable and credible data. In...

    2011 doi

  25. [34]

    Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wran- gler: interactive visual specification of data transformation scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 3363–3372. doi:10.1145/1978942.1979444

    2011

  26. [35]

    Aryan Keluskar, Amrita Bhattacharjee, and Huan Liu. 2024. Do LLMs Un- derstand Ambiguity in Text? A Case Study in Open-world Question Answer- ing. In Proceedings of IEEE International Conference on Big Data . 7485–7490. doi:10.1109/BigData62323.2024.10825265

    2024

  27. [36]

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. SPoC: Search-based Pseudocode to Code. In Proceedings of Advances in Neural Information Processing Sys- tems, Vol. 32. https://proceedings.neurips.cc/paper_files/paper/20...

    2019

  28. [37]

    Rob Law. 1997. An overview of debugging tools. ACM SIGSOFT Software Engi- neering Notes (1997), 43–47. doi:10.1145/251880.251926

    1997

  29. [39]

    Haber, Tara Matthews, and Tessa Lau

    Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: automating & sharing how-to knowledge in the enterprise. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems . Association for Computing Machinery, 1719–1728. https://doi.org/10....

    2008

  30. [40]

    Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2023. SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models. In Proceedings of Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 4952–4984

    2023

  31. [41]

    Xue Li and Till Döhmen. 2024. Towards Efficient Data Wrangling with LLMs using Code Generation. In Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning. 62–66. doi:10.1145/3650203.3663334

    2024

  32. [42]

    Chi-Liang Liu, Hung yi Lee, and Wen tau Yih. 2022. Structured Prompt Tuning. https://arxiv.org/abs/2205.12309

    2022 arXiv

  33. [43]

    Lei Liu, So Hasegawa, Shailaja Keyur Sampat, Maria Xenochristou, Wei-Peng Chen, Takashi Kato, Taisei Kakibuchi, and Tatsuya Asai. 2024. AutoDW: Au- tomatic Data Wrangling Leveraging Large Language Models. In Proceedings of IEEE/ACM International Conference on Automated Softwar...

    2024

  34. [44]

    What It Wants Me To Say

    Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code- Generating Large Language Models. In Proceedings of CHI C...

    2023

  35. [45]

    Shaoying Liu, Tetsuo Tamai, and Shin Nakajima. 2011. A framework for inte- grating formal specification, review, and testing to enhance software reliability. International Journal of Software Engineering and Knowledge Engineering (2011), 259–288

    2011

  36. [46]

    Ziao Liu, Xiao Xie, Moqi He, Wenshuo Zhao, Yihong Wu, Liqi Cheng, Hui Zhang, and Yingcai Wu. 2025. Smartboard: Visual Exploration of Team Tactics with LLM ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts UIST ’25, September 28-October ...

    2025

  37. [47]

    Zhongsu Luo, Kai Xiong, Jiajun Zhu, Ran Chen, Xinhuan Shu, Di Weng, and Yingcai Wu. 2025. Ferry: Toward Better Understanding of Input/Output Space for Data Wrangling Scripts. IEEE Transactions on Visualization and Computer Graphics 31, 1 (2025), 1202–1212. doi:10.1109/TVCG.202...

    2025

  38. [48]

    Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Kenneth Koedinger, and Tongshuang Wu. 2024. What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use. https://arxiv.org/abs/2409.08775

    2024 arXiv

  39. [49]

    Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. 2015. User Interac- tion Models for Disambiguation in Programming by Example. In Proceedings of Annual ACM Symposium on User Interface Software...

    2015

  40. [50]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishin- skaya, Maja Trebacz, and Jan Leike. 2024. LLM Critics Help Catch LLM Bugs. https://arxiv.org/abs/2407.00215

    2024 arXiv

  41. [51]

    Behrang Mehrparvar and Sandro Pezzelle. 2024. Detecting and Translating Language Ambiguity with Multilingual LLMs. In Proceedings of Workshop on Multilingual Representation Learning. 310–323. doi:10.18653/v1/2024.mrl-1.26

    2024 doi

  42. [52]

    Miller and Brad A

    Robert C. Miller and Brad A. Myers. 2001. Outlier finding: focusing user attention on possible errors. In Proceedings of Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, 81–90. https: //doi.org/10.1145/502348.502361

    2001

  43. [53]

    Dinh-Trong

    Audris Mockus, Nachiappan Nagappan, and Trung T. Dinh-Trong. 2009. Test coverage and post-verification defects: A multiple case study. In Proceedings of International Symposium on Empirical Software Engineering and Measurement . 291–301. doi:10.1109/ESEM.2009.5315981

    2009

  44. [54]

    MODUGNO and B.A

    F. MODUGNO and B.A. MYERS. 1997. Visual Programming in a Visual Shell—A Unified Approach. Journal of Visual Languages & Computing 8, 5 (1997), 491–522. doi:10.1006/jvlc.1997.0049

    1997

  45. [55]

    Nelson, Halden Lin, Adam M

    Dominik Moritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. 2019. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2...

    2019

  46. [56]

    Zan Ahmad Naeem, Mohammad Shahmeer Ahmad, Mohamed Eltabakh, Mourad Ouzzani, and Nan Tang. 2024. RetClean: Retrieval-Based Data Cleaning Using LLMs and Data Lakes. Proceedings of the VLDB Endowment 17, 12 (2024), 4421–

    2024

  47. [57]

    Arpit Narechania, Adam Fourney, Bongshin Lee, and Gonzalo Ramos. 2021. DIY: Assessing the Correctness of Natural Language to SQL Systems. In Proceedings of International Conference on Intelligent User Interfaces . 597–607. doi:10.1145/ 3397481.3450667

    2021

  48. [58]

    Donald A Norman. 1986. Cognitive engineering. User centered system design 31, 61 (1986), 2

    1986

  49. [59]

    OpenAI. 2024. Data analysis with ChatGPT. Retrieved June 1, 2024 from https://help.openai.com/en/articles/8437071-data-analysis-with-chatgpt

    2024

  50. [60]

    Newman, and Mohamed Wiem Mkaouer

    Anthony Peruma, Taryn Takebayashi, Rocky Huang, Joseph Carmelo Averion, Veronica Hodapp, Christian D. Newman, and Mohamed Wiem Mkaouer. 2024. On the Rationale and Use of Assertion Messages in Test Code: Insights from Software Practitioners. InProceedings of IEEE International ...

    2024

  51. [61]

    Luigi Pinca. 2025. WebSockets. https://developer.mozilla.org/en-US/docs/Web/ API/WebSockets_API

    2025

  52. [63]

    KK Ramachandran and KK Karthick. 2019. Gantt chart: An important tool of management. International Journal of Innovative Technology and Exploring Engineering 8, 7 (2019), 140–142

    2019

  53. [64]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85–105. doi:10.1109/TSE. 2023.3334955

    2024

  54. [65]

    Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2014. Applying the User Experience Questionnaire (UEQ) in Different Evaluation Scenarios. In Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience. 383–392

    2014

  55. [66]

    Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo

    Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. In Proceedings of Annual ACM Symposium on User Interface Software and Tech...

    2024

  56. [67]

    O’Reilly Media, Inc

    James Shore and Shane Warden. 2021. The art of agile development . " O’Reilly Media, Inc. "

    2021

  57. [68]

    Nischal Shrestha, Titus Barik, and Chris Parnin. 2021. Unravel: A Fluent Code Explorer for Data Wrangling. In Proceedings of Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, 198–

    2021

  58. [69]

    Meta Open Source. 2024. React. https://react.dev/

    2024

  59. [70]

    John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science 12, 2 (1988), 257–285. doi:10.1016/0364-0213(88)90023-7

    1988 doi

  60. [72]

    Team UEQ. 2024. UEQ. https://www.ueq-online.org/

    2024

  61. [74]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. InProceedings of IEEE/ACM International Conference on Automated Software Engineering . 1258–1268. doi:10.1145/3691620. 3695501

    2024 doi

  62. [75]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshy- vanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of ACM/IEEE International Conference on Software Engineering . 1398–

    2020

  63. [76]

    James M. Wilson. 2003. Gantt charts: A centenary appreciation. European Journal of Operational Research 149, 2 (2003), 430–437. doi:10.1016/S0377-2217(02)00769-5

    2003 doi

  64. [77]

    Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian. 2024. WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. InProceedings of Annual ACM Symposium on User Interface Software and Technology . Article...

    2024

  65. [78]

    Tao Xie and David Notkin. 2006. Tool-assisted unit-test generation and selection based on operational abstractions. Automated Software Engineering 13, 3 (2006), 345–371. doi:10.1007/s10851-006-8530-6

    2006 doi

  66. [79]

    Kai Xiong, Siwei Fu, Guoming Ding, Zhongsu Luo, Rong Yu, Wei Chen, Hujun Bao, and Yingcai Wu. 2023. Visualizing the Scripts of Data Wrangling With Somnus. IEEE Transactions on Visualization and Computer Graphics (2023), 2950–

    2023

  67. [80]

    Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. 2024. On the Evaluation of Large Language Models in Unit Test Generation. https: //arxiv.org/abs/2406.18181

    2024 arXiv

  68. [81]

    Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. 2024. MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization. https://arxiv.org/abs/2402.11453

    2024 arXiv

  69. [82]

    Xin Yin, Chao Ni, Xiaodan Xu, and Xiaohu Yang. 2024. What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation. https: //arxiv.org/abs/2412.00828

    2024 arXiv

  70. [83]

    Andy Yu. 2023. Improving Efficiency in Data Wrangling With Semantic Type Detection. Ph. D. Dissertation. University of Hawai’i at Manoa

    2023

  71. [84]

    Zeyu Zhang, Paul Groth, Iacer Calixto, and Sebastian Schelter. 2024. Directions Towards Efficient and Automated Data Wrangling with Large Language Models. In Proceedings of IEEE International Conference on Data Engineering Workshops . 301–304. doi:10.1109/ICDEW61823.2024.00044

    2024

  72. [85]

    Zhanhui Zhou, Man To Tang, Qiping Pan, Shangyin Tan, Xinyu Wang, and Tianyi Zhang. 2022. INTENT: Interactive Tensor Transformation Synthesis. In Proceedings of Annual ACM Symposium on User Interface Software and Technology . Association for Computing Machinery. https://doi.org...

    2022

  73. [86]

    Zhichao Zhou, Yuming Zhou, Chunrong Fang, Zhenyu Chen, Xiapu Luo, Jingzhu He, and Yutian Tang. 2024. Coverage Goal Selector for Combining Multiple Criteria in Search-Based Unit Test Generation. IEEE Transactions on Software Engineering 50, 4 (2024), 854–883. doi:10.1109/TSE.20...

    2024

  74. [87]

    Xinhua Zhu, Zhongjie Kuang, and Lanfang Zhang. 2023. A prompt model with combined semantic refinement for aspect sentiment analysis. Information Pro- cessing & Management 60, 5 (2023), 103462. doi:10.1016/j.ipm.2023.103462

    2023

  75. [88]

    Jan Felix Zolitschka. 2020. A novel multi-agent-based chatbot approach to orches- trate conversational assistants. In Proceedings of Business Information Systems International Conference. 103–117

    2020

  76. [173]

    doi:10.1109/HICSS.1990.205185

    1990

  77. [207]

    https://doi.org/10.1145/3472749.3474744

  78. [1409]

    doi:10.1145/3377811.3380429

  79. [2023]

    In Optimising the Software Development Process with Artificial Intelligence

    Automated support for unit test generation. In Optimising the Software Development Process with Artificial Intelligence . 179–219. doi:10.1007/978-981-19- 9948-2_7

    doi

  80. [2052]

    doi:10.1145/3691620.3695267

  81. [2964]

    doi:10.1109/TVCG.2022.3144975

    2022

  82. [4424]

    doi:10.14778/3685800.3685890

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.