Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that turning a model's chain-of-thought into an interactive topic tree gives users a sense of control over, and understanding of, the reasoning behind an LLM's answer.

desk verdict Timely HCI prototype for editing chain-of-thought, but the steering claim outruns the self-report evidence—worth refereeing nonetheless. read the letter →

arxiv 2506.23678 v1 pith:J35VJDTM submitted 2025-06-30 cs.HC cs.AI

Rock Yuren Pang , K. J. Kevin Feng , Shangbin Feng , Chu Li , Weijia Shi , Yulia Tsvetkov , Jeffrey Heer , Katharina Reinecke This is my paper
classification cs.HCcs.AI
keywords interactivereasoningchain-of-thoughtLLMtreevisualizationhuman-AIinteractiontest-timescalinguserstudyAIdecision-making
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Interactive Reasoning, an interaction design that turns a large language model's verbose, linear chain-of-thought into a hierarchical tree of topics that users can directly edit. It implements this in Hippo, a prototype for AI-assisted decision-making under trade-offs, and reports a within-subjects study of 16 participants. The study finds that, compared to a baseline interface that also allows editing but in plain text, Hippo significantly increases users' perceived control, sense-making, layout comprehension, and awareness of assumptions in the reasoning. The promise is that users can catch and interrupt wrong turns, add their own context and priorities, and receive a final answer that feels personalized rather than generic. The paper frames this as a new paradigm: putting human oversight inside the reasoning step, not just at the final answer.

What carries the argument

The central object is the interactive reasoning tree, a depth-first progressively generated topic hierarchy created by a few-shot prompt that structures raw chain-of-thought text into <topic> and <branch> XML-like tags. A Clarify operator flags feedback nodes where user input would be valuable, and a Link operator uses zero-shot natural-language inference to highlight which reasoning nodes support which sentences of the final response. User edits (add, edit, delete, regenerate, or branch with a custom prompt) are collected, and the updated reasoning is placed back inside <think>...</think> tags to condition the model's final generation.

What would settle it

A controlled experiment where users insert a node asserting a clearly false or contradictory premise (for example, 'the user has no budget constraints' when the query is about budget travel) and the final response is compared against a baseline run without that edit; if the response does not systematically and reliably incorporate the edit, the steering mechanism fails. Repeating across multiple users and edits would show whether the effect is real or merely perceived.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that representing a model's chain-of-thought as an interactive topic tree, rather than as a linear text wall, lets users feel they are steering the model, and this sense of agency extends to the final response. In the user study, perceived control rose from a mean of 4.19 to 5.75 (p = 0.003), sense-making from 5.19 to 6.44 (p = 0.004), layout comprehensibility from 3.44 to 6.00 (p = 0.009), and awareness of assumptions from 4.69 to 6.25 (p = 0.012). Participants described outputs as personalized, saying the model 'hears my voice rather than generating a generic answer,' and a case-study user called the tree 'essentially a plan for me to consider' in financial analysis. The paper is careful not to claim that editing the chain improves the factual correctness of outputs; it claims an improved sense of control, understanding, and personalization from the end user's perspective.

Load-bearing premise

The load-bearing premise is that editing the visible chain-of-thought text and feeding it back inside <think> tags actually steers the model's final answer in the direction the user intends; if the written chain is not a faithful or effective control handle, the claimed benefits are illusory even if users feel more confident.

Editorial extensions

If this is right

  • Reasoning-model interfaces can be redesigned around review-and-edit, letting users interrupt wrong assumptions before an answer is finalized.
  • In high-stakes decisions, the reasoning chain itself becomes a plan the user can refine, while the final response becomes a personalized summary rather than the main deliverable.
  • Tree visualization may reduce cognitive load enough that users actually read reasoning they would otherwise skim or ignore.
  • The Link highlight feature offers traceability from each response sentence back to the reasoning step that produced it, which can support appropriate trust calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: This study measures perceived control and understanding, not whether edited chain-of-thought text actually changes the model's final output; the paper itself cites evidence that chain-of-thought may not faithfully reflect a model's real reasoning.
  • Editorial inference: The feedback-node mechanism may act as a cognitive forcing function, so part of the measured awareness gain could come from being asked questions rather than from the tree layout itself.
  • Editorial inference: A testable extension would insert a deliberately contradictory or false premise into a node and measure whether the final response systematically reflects that edit across repeated runs, which would separate real steering from perceived steering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Interactive Reasoning, an interaction design that converts an LLM's chain-of-thought into a hierarchical tree of topics, allows users to add, edit, delete, and regenerate nodes, and generates a final answer conditioned on the edited thinking. This is implemented in Hippo, a prototype for decision-making under trade-offs. The authors report a within-subjects user study (N=16) comparing Hippo to an editable linear-text baseline, finding significant self-reported improvements in control, sense-making, layout, awareness of assumptions, and confidence, plus qualitative evidence of personalization and repurposing of the final response. Two follow-up case studies illustrate use for information seeking and financial planning. The paper claims that interactive reasoning lets users 'efficiently steer the model towards customized responses' and proposes a new paradigm for human oversight during test-time scaling.

Significance. If the steering claim holds, this is a timely and useful contribution to human-AI interaction: it directly addresses a real usability problem with long, unstructured chain-of-thought output and offers a concrete design space for test-time scaling control. The work's strengths include a thoughtful derivation of design goals from prior HCI/XAI literature, a fair baseline that is also editable, rich qualitative analysis, and unusually candid limitations. I see no circularity in the empirical evaluation, which is an external user study. However, the paper's most attention-grabbing claim, that edited chain-of-thought text inside <think> tags steers the final output, is not tested objectively, and the headline statistical results are based on a small number of unadjusted Likert comparisons. These issues are load-bearing because the system is framed as a control mechanism rather than a visualization only. The paper is worth publishing after the causal and statistical concerns are addressed.

major comments (3)
  1. [Abstract; §4.2.3; §8] The abstract states that Hippo lets users 'efficiently steer the model towards customized responses', and contribution (2) asserts that users reported 'increased confidence in making the final decision' with Hippo. Yet §8 explicitly says that 'our current approach does not systematically analyze the model behaviors given users’ feedback' and that the paper 'does not claim that user feedback led to more personalized or accurate reasoning chains or final model outputs'. These statements are in tension. The mechanism in §4.2.3, which places edited reasoning inside <think>...</think> tags, is a causal assumption: the final response should change in the direction of the user's edits. The user study provides only self-reported Likert scales and qualitative anecdotes (e.g., P8, P10), with no objective measure of whether the final output actually changed in the intended direction. This is not merely a presentation issue; the central value proposition of Hippo is control and steering, not just display. I recommend either softening the abstract and contribution claims to describe perceived control, or adding a systematic evaluation of edit-to-output alignment (e.g., a small controlled experiment with blind raters or automated entailment/similarity metrics comparing responses with and without a given edit).
  2. [§5.3.1; §5.3.2] The headline quantitative findings rest on nine Likert items tested with Wilcoxon signed-rank tests without any correction for multiple comparisons. With a conservative Bonferroni threshold of α=0.0056 for nine tests, control (p=0.003) and sense-making (p=0.004) remain significant, but layout (p=0.009), awareness (p=0.012), and confidence (p=0.049) would not survive. At minimum, the authors should report effect sizes and confidence intervals for each comparison, use a mixed-effects model or otherwise account for the multiple outcomes, and temper the wording in the abstract and Section 5.3 that presents all five results as equally robust. The lack of correction is especially relevant because the measures are correlated (e.g., layout and sense-making) and the sample is only N=16.
  3. [§5.3.3; §5.1] RQ3 concerns how users interact with the reasoning tree, but the paper provides no quantitative interaction logs. Statements such as 'we observed that participants rarely made edits', 'only four users typed their own experiences and opinions', and 'zero participants used the collapse function during tree generation' are based on informal observation. Since the system was implemented with a logging backend (Next.js/Flask), the authors can easily report counts of add/edit/delete/regenerate operations, time-to-first-edit, and usage of the collapse feature. Such logs would also help distinguish perceptual effects in RQ1–RQ2 from actual behavioral changes, and would strengthen the design implications in Section 7.
minor comments (6)
  1. [§2.2] The phrase 'long reasonig chain' contains a typo; it should read 'long reasoning chain'.
  2. [§5.3.1] The section heading 'A wareness' should be 'Awareness'.
  3. [Figure 4 caption] The figure caption says 'Review and edit the reaching chain below!' but should say 'reasoning chain'.
  4. [Appendix B.1.1] The literal placeholder '[More of this example]' appears inside the few-shot prompt; this should be removed or clearly marked as an abridged prompt rather than part of the actual prompt text.
  5. [§4.2.2; Appendix B.1.2] The duplicate-clarification threshold (cosine similarity > 0.8 with all-MiniLM-L6-v2) and the group-thoughts paragraph limit (8 paragraphs) are introduced without any sensitivity analysis or justification; a brief ablation or note on their influence would improve reproducibility.
  6. [§5.2] The power analysis is reported as Cohen's d = 0.8, α = 0.05, N = 16, but the assumed effect size and the paired versus independent design are not justified; please specify the target power and whether the test is two-sided.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the central evidence is an external user study, and the untested steering assumption is an acknowledged correctness limitation rather than a circular loop.

full rationale

The paper's central claims are empirical and are not derived from fitted parameters or from the authors' prior theorems. The quantitative findings in §5.3.1 (control p=0.003, sense-making p=0.004, layout p=0.009, awareness p=0.012) are Wilcoxon tests on 16 participants' Likert ratings comparing Hippo to an editable baseline; these are observations from an external user study, not predictions generated by the system's own construction. The pipeline in §4.2 uses few-shot and zero-shot GPT-4o prompts for structuring, clarifying, and linking, and DeepSeek-R1 for the reasoning chain; no parameter is fit to the outcome being claimed, and no quantity is defined in terms of the result it is supposed to explain. The assumption that edited chain-of-thought text inside <think> tags steers the final response is arguably load-bearing for the design, but the paper candidly flags it in §8: 'our current approach does not systematically analyze the model behaviors given users' feedback,' and it cites [10] to note that CoTs may not faithfully represent a model's actual reasoning. That is an admitted lack of causal validation, which is a correctness risk, not a circularity. Self-citations (e.g., [13], [46], [53], [76]) appear only as related-work background and do not supply the argument's conclusion; there is no imported uniqueness theorem, no ansatz smuggled in by citation, and no renaming of a known result presented as a derivation. Because no circular step can be exhibited with the required specificity, the appropriate score is 1, reflecting only minor, non-load-bearing self-citations.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rely on the reliability of LLM-based structuring, clarifying, and linking operators, on a hand-set deduplication threshold, and critically on the assumption that the visible chain-of-thought is a causally effective steering handle. These are domain assumptions rather than derived results.

free parameters (2)
  • Duplicate clarification detection threshold = 0.8 cosine similarity
    Hand-chosen in §4.2.2 to deduplicate Clarify questions; affects which Feedback nodes appear and was not empirically tuned or validated.
  • Group Thoughts paragraph limit = at most 8 paragraphs
    Prompt rule in appendix B.1.2 constraining the theme grouping step; controls tree granularity and was set by the authors without a comparative evaluation.
assumptions (4)
  • domain assumption LLM-based few-shot tagging with XML-like tags reliably recovers a topic hierarchy from raw CoT text.
    Invoked in §4.2.1 and appendix B.1; no accuracy metrics are reported for this structuring pipeline.
  • domain assumption GPT-4o can reliably identify which reasoning nodes would benefit from user clarification.
    Invoked in §4.2.2 and appendix B.2; relies on LLM classification ability, with pilot feedback but no systematic validation against human judgments.
  • domain assumption Zero-shot NLI linking between reasoning nodes and response sentences is accurate enough for the highlighting feature.
    Invoked in §4.2.4 and appendix B.3; the earlier bart-large approach was dropped for latency, and no gold-label evaluation of the zero-shot linker is provided.
  • domain assumption Editing visible chain-of-thought text and placing it back into <think> tags steers the final model output as the user intends.
    Invoked in §4.2.3; the paper does not systematically test this causal link and cites [10], which shows CoT may not faithfully represent model reasoning. This is the weakest load-bearing premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/J35VJDTM

@misc{pith2026250623678,
  author       = {Pith},
  title        = {Pith review of: Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J35VJDTM}},
  note         = {Machine review of arXiv:2506.23678}
}
read the original abstract

The output quality of large language models (LLMs) can be improved via "reasoning": generating segments of chain-of-thought (CoT) content to further condition the model prior to producing user-facing output. While these chains contain valuable information, they are verbose and lack explicit organization, making them tedious to review. Moreover, they lack opportunities for user feedback, such as to remove unwanted considerations, add desired ones, or clarify unclear assumptions. We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics and enables user review and modification. We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs. In a user study with 16 participants, we find that interactive reasoning in Hippo allows users to quickly identify and interrupt erroneous generations, efficiently steer the model towards customized responses, and better understand both model reasoning and model outputs. Our work contributes to a new paradigm that incorporates user oversight into LLM reasoning processes.

Figures

Figures reproduced from arXiv: 2506.23678 by the authors.

Figure 1
Figure 1. Hippo includes a tree visualization of the reasoning steps and allows users to directly control when models need users’ feedback. Users input their query in the input bar A③. Then, the reasoning tree progressively generates nodes B③following a preorder (depth-first) tree traversal order. Users can branch out a reasoning node C③by providing a customized prompt, which will add a new child node F③. Hippo halts the tree… view at source ↗
Figure 2
Figure 2. Hippo highlights the tree nodes G③and sentences in the reasoning model final response H③. structure of reasoning) and tags (capturing the low-level details and important entities given structural constraints). This process requires structuring the raw text from the model’s reasoning into a hierarchy of smaller components, drawing out components that require human intervention, and linking the final output to compo￾n… view at source ↗
Figure 3
Figure 3. The Interactive Reasoning pipeline fetches the initial reasoning chain, structures the reasoning into topical hierarchy, flags text that might benefit from user intervention. The final output is directed back to the updated reasoning chain. challenges in the case of long reasoning chains, which usually gen￾erate numerous reasoning nodes and response paragraphs. Process￾ing times in batch for visual highlighting freq… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The baseline system. The reasoning steps show up [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Participants’ responses to the Likert-scale questions, contrasting the baseline and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An example of the current DeepSeek-R1 platform [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations

    cs.AI 2026-04 conditional novelty 7.0 of 10

    GROVE visualizes distributions of language model generations as overlapping paths through a text graph, with user studies showing that graph summaries aid structural judgments like diversity assessment while raw outpu...

  2. InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluation

    cs.HC 2025-12 conditional novelty 6.0 of 10

    An interactive tree visualization of LLM sampling lets evaluators cover the same harmful-response space as random sampling with up to 5x fewer samples.

Reference graph

Works this paper leans on

91 extracted references · 21 canonical work pages · cited by 2 Pith papers

  1. [2]

    Glassman

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L. Glassman. 2024. ChainForge: A Visual Toolkit for Prompt Engineer- ing and LLM Hypothesis Testing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Asso- ciation for Computing Machinery, New York, NY, USA, Article 3...

    arXiv 2024

  2. [3]

    Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi

    Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926 [cs.AI] https://arxiv.org/abs/2503.11926

    arXiv 2025

  3. [4]

    Jeff Baker, Donald Jones, and Jim Burkman. 2009. Using visual representations of data to enhance sensemaking in data exploration tasks. Journal of the Association for Information Systems 10, 7 (2009), 2

    2009

  4. [5]

    Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing ...

    arXiv 2021

  5. [7]

    Virginia Braun and Victoria Clarke and. 2006. Using the- matic analysis in psychology. Qualitative Research in Psy- chology 3, 2 (2006), 77–101. doi:10.1191/1478088706qp063oa arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa

    doi:10.1191/1478088706qp063oa 2006

  6. [8]

    Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI- assisted Decision-making. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 188 (April 2021), 21 pages. doi:10.1145/3449287

    doi:10.1145/3449287 2021

  7. [9]

    Boqi Chen, Fandi Yi, and Dániel Varró. 2023. Prompting or fine-tuning? a com- parative study of large language models for taxonomy construction. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and arXiv, June, 2025 Pang et al. Systems Companion (MODELS-C). IEEE, 588–596

    2023

  8. [10]

    Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. arXiv:2505.05410 [cs.CL] https://arxiv.org/abs/2505.05410

    arXiv 2025

Show all 91 references
  1. [11]

    Yu Ying Chiu, Liwei Jiang, and Yejin Choi. 2024. Dailydilemmas: Revealing value preferences of llms with quandaries of daily life. arXiv preprint arXiv:2410.02683 (2024)

    2024 arXiv

  2. [12]

    Hai Dang, Sven Goller, Florian Lehmann, and Daniel Buschek. 2023. Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ...

    2023

  3. [13]

    Fisher, Jennifer Pan, Yulia Tsvetkov, and Katharina Reinecke

    Jillian Fisher, Shangbin Feng, Robert Aron, Thomas Richardson, Yejin Choi, Daniel W. Fisher, Jennifer Pan, Yulia Tsvetkov, and Katharina Reinecke. 2025. Biased AI can Influence Political Decision-Making. arXiv:2410.06415 [cs.HC] https://arxiv.org/abs/2410.06415

    2025

  4. [14]

    Austin J Freeley. 2009. Argumentation and debate: Critical thinking for reasoned decision making

    2009

  5. [16]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. DeepSeek- R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501.12948

    2025 arXiv

  6. [17]

    Mary E Guy. 1990. Ethical decision making in everyday work situations . Blooms- bury Publishing

    1990

  7. [19]

    Jeffrey Heer. 2019. Agency plus automation: Designing artificial intelligence into interactive systems. Proceedings of the National Academy of Sciences 116, 6 (2019), 1844–1850

    2019

  8. [20]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275 (2020)

    2020 arXiv

  9. [21]

    Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer

  10. [22]

    Shelton, Fanny Cheva- lier, Kari Kraus, and Niklas Elmqvist

    Md Naimul Hoque, Tasfia Mashiat, Bhavya Ghai, Cecilia D. Shelton, Fanny Cheva- lier, Kari Kraus, and Niklas Elmqvist. 2024. The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Inter- active Visualization. InProceedings of the...

    2024

  11. [23]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    1999

  12. [24]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. OpenAI o1 System Card. arXiv:2412.16720 [cs.AI] https://arxiv.org/abs/2412. 16720

    2024 arXiv

  13. [26]

    Zhuohang Jiang, Pangjing Wu, Ziran Liang, Peter Q Chen, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter HF Ng, and Qing Li. 2025. HiBench: Bench- marking LLMs Capability on Hierarchical Structure Reasoning. arXiv preprint arXiv:2503.00912 (2025)

    2025 arXiv

  14. [27]

    Philip N Johnson-Laird, Sangeet S Khemlani, and Geoffrey P Goodwin. 2015. Logic, probability, and human reasoning. Trends in cognitive sciences 19, 4 (2015), 201–214

    2015

  15. [28]

    Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. 2024. LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models. In Extended Abstracts of t...

    2024

  16. [29]

    Hita Kambhamettu, Jamie Flores, and Andrew Head. 2025. Traceable Texts and Their Effects: A Study of Summary-Source Links in AI-Generated Summaries. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25) . Association ...

    2025

  17. [31]

    I’m Not Sure, But

    Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jen- nifer Wortman Vaughan. 2024. "I’m Not Sure, But... ": Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust. In Proceedings of the 2024 ACM Conference on ...

    2024

  18. [33]

    Help Me Help the AI

    Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (...

    2023

  19. [34]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. EvalLM: Interactive Evaluation of Large Language Model Prompts on User- Defined Criteria. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Asso...

    2024 doi

  20. [35]

    Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2024. Understanding Users’ Dissatisfaction with ChatGPT Responses: Types, Resolv- ing Tactics, and the Effect of Knowledge Level. In Proceedings of the 29th In- ternational Conference on Intelligent User Interfac...

    2024

  21. [36]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916 [cs.CL] https://arxiv.org/abs/2205.11916

    2023 arXiv

  22. [37]

    Lam, Janice Teoh, James A

    Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High- Level Concepts Using LLooM. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA...

    2024

  23. [39]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58t...

    2020 doi

  24. [40]

    Diya Li, Yue Zhao, Zhifang Wang, Calvin Jung, and Zhe Zhang. 2024. Large Lan- guage Model-Driven Structured Output: A Comprehensive Benchmark and Spa- tial Data Generation Framework. ISPRS International Journal of Geo-Information (2024). https://api.semanticscholar.org/CorpusI...

    2024

  25. [41]

    Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Ha- jishirzi, and Asli Celikyilmaz. 2024. Don’t throw away your value model! Gener- ating more preferable text with Value-Guided Monte-Carlo Tree Search decoding. arXiv:2309.15028 [cs.CL] https://arxiv.org/ab...

    2024 arXiv

  26. [42]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. Interactive Reasoning ...

    2024

  27. [43]

    Xingyu Bruce Liu, Haijun Xia, and Xiang Anthony Chen. 2025. Interacting with Thoughtful AI. arXiv preprint arXiv:2502.18676 (2025)

    2025 arXiv

  28. [44]

    Yang Liu, Alex Kale, Tim Althoff, and Jeffrey Heer. 2021. Boba: Authoring and Visualizing Multiverse Analyses. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2021), 1753–1763. doi:10.1109/TVCG.2020.3028985

    2021

  29. [46]

    Katelyn Xiaoying Mei, Rock Yuren Pang, Alex Lyford, Lucy Lu Wang, and Katha- rina Reinecke. 2025. Passing the Buck to AI: How Individuals’ Decision-Making Patterns Affect Reliance on AI. arXiv:2505.01537 [cs.HC] https://arxiv.org/abs/ 2505.01537

    2025

  30. [47]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? ArXiv abs/2202.12837 (2022). https: //api.semanticscholar.org/CorpusID:247155069

    2022 arXiv

  31. [48]

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229 (2024)

    2024 arXiv

  32. [49]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv:2501.19393 [cs.CL] https: //arxiv.org/abs/2501.19393

    2025 arXiv

  33. [50]

    Richard Nordquist. 2019. What is deductive reasoning? https://www.thoughtco. com/deduction-logic-and-rhetoric-1690422

    2019

  34. [51]

    OpenAI. 2025. Detecting Misbehavior in Frontier Reasoning Models. https: //openai.com/index/chain-of-thought-monitoring/. Accessed Apr 08, 2025

    2025

  35. [52]

    Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions. In Proceedings of the 2023 Conference on Empirical Method...

    2023 doi

  36. [53]

    Rock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, and Danielle Bragg. 2025. Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review. In Proceedings of the 2025 CHI Conferenc...

    2025

  37. [54]

    Joon Sung Park, Rick Barber, Alex Kirlik, and Karrie Karahalios. 2019. A slow algorithm improves users’ assessments of the algorithm’s accuracy. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–15

    2019

  38. [55]

    Andreas Peldszus and Manfred Stede. 2013. From argument diagrams to argumen- tation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI) 7, 1 (2013), 1–31

    2013

  39. [56]

    Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Zifeng Wang, Jiaqi Mu, Hao Zhang, Chen-Yu Lee, and Nan Hua. 2024. LMDX: Language Model-based Document Information Extraction and Localization. In Findings of the Association for ...

    2024 doi

  40. [57]

    Kevin Pu, K. J. Kevin Feng, Tovi Grossman, Tom Hope, Bhavana Dalvi Mishra, Matt Latzke, Jonathan Bragg, Joseph Chee Chang, and Pao Siangliulue. 2025. IdeaSynth: Iterative Research Idea Development Through Evolving and Com- posing Idea Facets with Literature-Grounded Feedback. ...

    2025

  41. [58]

    Sebastian Raschka. 2025. Understanding reasoning llms. https://magazine. sebastianraschka.com/p/understanding-reasoning-llms

    2025

  42. [59]

    Chris Reed, Douglas Walton, and Fabrizio Macagno. 2007. Argument diagram- ming in logic, law and artificial intelligence. The Knowledge Engineering Review 22, 1 (2007), 87–109

    2007

  43. [60]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

  44. [61]

    Max Schemmer, Niklas Kuehl, Carina Benz, Andrea Bartos, and Gerhard Satzger

  45. [62]

    Vera Liao, and Ziang Xiao

    Nikhil Sharma, Q. Vera Liao, and Ziang Xiao. 2024. Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Compu...

    2024

  46. [63]

    Ben Shneiderman. 1983. Direct manipulation: A step beyond programming languages. Computer 16, 08 (1983), 57–69

    1983

  47. [64]

    Ben Shneiderman. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human–Computer Interaction 36 (2020), 495 – 504. https://api.semanticscholar.org/CorpusID:211259461

    2020

  48. [65]

    Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. 2024. StructuredRAG: JSON Response Formatting with Large Language Models. ArXiv abs/2408.11061 (2024). https://api.semanticscholar.org/CorpusID:271916259

    2024 arXiv

  49. [66]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=4FWAwZtd2n

    2025

  50. [67]

    Paul M Sniderman and Sean M Theriault. 2004. The structure of political argument and the logic of issue framing. Studies in public opinion: Attitudes, nonattitudes, measurement error, and change 3, 03 (2004), 133–65

    2004

  51. [68]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

    2022

  52. [69]

    Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics 43, 3 (2017), 619–659

    2017

  53. [70]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling Multilevel Exploration and Sensemaking with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). ...

    2023

  54. [71]

    Gajos, and Finale Doshi-Velez

    Siddharth Swaroop, Zana Buçinca, Krzysztof Z. Gajos, and Finale Doshi-Velez

  55. [72]

    Peter Szolovits, Ramesh S Patil, and William B Schwartz. 1988. Artificial intelli- gence in medical diagnosis. Annals of internal medicine 108, 1 (1988), 80–87

    1988

  56. [73]

    Together AI. 2025. Together API. https://api.together.xyz/. Accessed April 08, 2025

    2025

  57. [74]

    Douglas N Walton. 1990. What is reasoning? What is an argument? The journal of Philosophy 87, 8 (1990), 399–419

    1990

  58. [75]

    Douglas N Walton and Lynn M Batten. 1984. Games, graphs and circular argu- ments. Logique et Analyse 27, 106 (1984), 133–164

    1984

  59. [76]

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. 2023. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems 36 (2023), 30840– 30861

    2023

  60. [77]

    Xinru Wang and Ming Yin. 2021. Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making. In Proceedings of the 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for ...

    2021

  61. [78]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amir- reza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. a...

    2022 arXiv

  62. [79]

    Wang, Chinmay Kulkarni, Lauren Wilcox, Michael Terry, and Michael Madaio

    Zijie J. Wang, Chinmay Kulkarni, Lauren Wilcox, Michael Terry, and Michael Madaio. 2024. Farsight: Fostering Responsible AI Awareness During AI Applica- tion Prototyping. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA)(CHI ’2...

    2024

  63. [80]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  64. [81]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Syste...

    2022

  65. [82]

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA)(CHI ’22). Associ...

    2022

  66. [83]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724 (2024)

    2024 arXiv

  67. [84]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451 (2024)

    2024 arXiv

  68. [85]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.n...

    2023

  69. [86]

    Ryan Yen, Jiawen Stefanie Zhu, Sangho Suh, Haijun Xia, and Jian Zhao. 2024. CoLadder: Manipulating Code Generation via Multi-Level Blocks. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association fo...

    2024

  70. [87]

    Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Understanding the Effect of Accuracy on Trust in Machine Learning Models. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Mac...

    2019

  71. [88]

    Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2024. Natural Language Reasoning, A Survey. ACM Comput. Surv. 56, 12, Article 304 (Oct. 2024), 39 pages. doi:10.1145/3664194

    2024 doi

  72. [89]

    Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2024. Natural language reasoning, a survey. Comput. Surveys 56, 12 (2024), 1–39

    2024

  73. [90]

    Zamfirescu-Pereira, Richmond Y

    J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang

  74. [91]

    Zhang, Lea Verou, and David Karger

    Amy X. Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging Discussion Forums and Wikis Using Recursive Summarization. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for...

    2017

  75. [92]

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025. A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well? arXiv:2503.24235 [cs.CL...

    2025 arXiv

  76. [93]

    Vera Liao, and Rachel K

    Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20)...

    2020

  77. [94]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470 (2024)

    2024 arXiv

  78. [95]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23)

    Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. doi:10...

    2023 doi

  79. [100]

    I don’t know X

    Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schu- urmans, and Hanjun Dai. 2023. Large language models can learn rules. arXiv preprint arXiv:2310.07064 (2023). A An example of DeepSeek-R1 response with the reasoning steps Intermediate reasoning steps ... [...

    2023 arXiv

  80. [2021]

    arXiv preprint arXiv:2104.08315 (2021)

    Surface form competition: Why the highest probability answer isn’t always right. arXiv preprint arXiv:2104.08315 (2021)

    2021 arXiv

  81. [2022]

    In International Conference on Learning Representations

    Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations . https://openreview.net/ forum?id=9Vrb9D0WI4

  82. [2023]

    In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23)

    Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23). Association for Computing Machinery, New York, NY, USA, 410–422. doi:10.1...

  83. [2024]

    In Proceedings of the 29th International Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI ’24)

    Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure. In Proceedings of the 29th International Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI ’24). Association for Computing Machinery, New York, NY, USA, 138–154. doi:10.1145/3640543.3645206

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.