{"id":"03f3f1e3-8574-43d5-8320-365261221ae8","arxiv_id":"2608.09294","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 16-participant qualitative study finds that using LLMs to turn routine conversations into knowledge graphs causes entity fragmentation because users narrate hierarchically and out of order, leading to design heuristics for JITAIs.","lead":"This paper studies an AI voice assistant that converts casual talk about a daily routine into a structured knowledge graph, then asks 16 volunteers to judge whether the machine's model matches their real lives. It reports a 'mental-model gap' where non-linear, hierarchical storytelling clashes with linear machine extraction, and it proposes design rules for future health interventions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution of extraction failures to inherent LLM linearity is untested; implementation confounds could fully account for the reported mental-model gap.","rationale":"The reader's weakest assumption correctly identifies that the study attributes errors to LLM linearity without varying the extractor, prompt, or conversation flow. I agree this is the most load-bearing concern. The paper's strengths are real: a careful qualitative protocol, member-checking with visual artifacts, reported inter-rater reliability (Cohen's kappa 0.71-0.79), and concrete design heuristics that follow from the observations. These support a conditional acceptance. However, the central theoretical claim depends on showing that the failures are due to a fundamental linear-vs-hierarchical mismatch rather than to specific implementation choices. A single-system study with 16 participants cannot establish that without some comparison. The proposed test is feasible with existing data and would settle the question. Since the issue is addressable and the reader already conditioned acceptance on similar revisions, my read does not change the verdict.","tokens_in":18473,"tokens_out":2875,"duration_ms":30262,"concrete_test":"Re-run extraction on the existing 16 transcripts under a factorial comparison: (a) current gpt-4o-mini LangExtract pipeline; (b) same pipeline with an added coreference-resolution instruction and an explicit hierarchical event schema; (c) a stronger model such as gpt-4o with identical prompts; (d) the same extractor applied to cleaned, structured textual versions of the transcripts (removing conversational disfluency). Measure duplicate-node rate, missing-event rate, and ordering-error rate per condition. If fragmentation remains high across (a)-(c) but drops in (d), the linear-extraction attribution is supported. If (b) or (c) substantially reduces errors, the reported gap is partly implementation-specific and the design-heuristic claims must be re-scoped to this system. Publish transcripts, prompts, and merge rules to allow independent replication.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a fundamental 'mental-model gap' exists between non-linear human narration and linear LLM extraction, causing severe entity fragmentation (Sections 1, 5.1, 6.1). The evidence shows participants narrated hierarchically and non-linearly, and that this specific system produced duplicate nodes, missing events, and ordering errors. But the pipeline confounds several factors: the extractor (gpt-4o-mini via LangExtract), the state-machine conversation flow with rigid goal-driven prompts, the graph-merge logic, and the ASR front-end. Section 6.1 attributes failures to 'standard zero-shot LLM extractors' and 'strict linear architecture' without varying any of these components. Several reported failures—co-referent duplicates, 'gym captured as dream', stated commitments resurfacing as availability—could stem from model capability, prompt design, merge rules, or ASR errors rather than an inherent linearity property. The state machine itself may induce the non-linear corrections and looping users exhibited, partly manufacturing the 'gap'. Because the paper generalizes to design heuristics for all conversational JITAIs, this un-tested attribution is load-bearing. If a stronger model or a coreference-aware prompt eliminates most fragmentation, the gap as a critical fundamental property is overstated and the contribution reduces to a single-system case study. The limitations section acknowledges lab setting and small n but does not acknowledge or test this attributional confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a qualitative lab study of a neurosymbolic conversational agent (LLM extractor paired with a Neo4j knowledge graph) that elicits users' daily routines through voice dialogue and converts them into structured schedule data for Just-In-Time Adaptive Interventions (JITAIs). Sixteen participants interacted with the agent, then reviewed state-machine traces, knowledge graphs, and natural-language summaries in a member-checking phase. Thematic analysis of 221 coded units yields three reported themes: a 'mental-model gap' between non-linear, hierarchical human narration and 'linear' LLM extraction; positive conversational naturalness but rigidity and looping in the state-machine-driven dialogue; and users' willingness to bridge the gap when given structural transparency. The paper argues these findings reveal fundamental structural friction and derives design heuristics (hierarchical extraction, clarification over assumption, routine piggybacking, adaptive negotiation, scalable transparency).","tokens_in":18744,"tokens_out":4357,"duration_ms":42250,"significance":"If the central claim held, the work would be a significant empirical contribution to JITAI design, showing that routine elicitation requires adaptive, hierarchical, and negotiable representations rather than flat extraction. The paper has clear strengths: it follows a codebook thematic-analysis procedure with reported inter-rater reliability (Cohen's kappa improving from 0.32 to 0.71 to 0.79), uses human-centric evaluation via natural-language playbacks, and offers concrete, actionable design heuristics. The transparency artifacts (state-machine trace, knowledge-graph trace) are a useful methodological contribution. However, the paper's central causal attribution—that observed extraction failures are caused by the inherent linearity of LLM extraction—is not supported by the single-system study design. The contribution is better characterized as a detailed single-system case study of conversational routine elicitation; the broader 'fundamental' claims require comparative evidence or substantial reframing.","major_comments":[{"comment":"The central claim that 'standard zero-shot LLM extractors operate bottom-up and linearly' and that this 'strict linear architecture causes severe node duplication and fragmentation' is an attribution to inherent properties of linear extraction, but the study only evaluates one specific pipeline: LangExtract with gpt-4o-mini, a particular state-machine prompt regime, a graph-merge implementation, and an ASR front-end. The paper never varies the extractor, the prompt templates, the conversation flow, or the merge logic, nor does it compare against a non-linear or human extraction baseline. The reported failures—co-referent duplicates (n=12), missing information (n=20), wrong order (n=20), and even speech-recognition errors such as 'gym captured as dream' (P6)—could stem from model capability, prompt design, merge rules, or ASR errors rather than from a fundamental property of linearity. Because the paper's title, abstract, and design heuristics generalize beyond this system, this untested attribution is load-bearing and needs either direct comparative evidence or a clear reframing to single-system findings.","section":"Section 6.1 and Theme 1 (Section 5.1)"},{"comment":"The study design has no baseline or control condition and reports no objective error counts. All evidence of the 'mental-model gap' comes from participants' subjective identification of errors during the member-checking phase, after they have been shown the system's own representations. Additionally, the codebook was developed by the same authors who designed the system, so the themes (especially Theme 1) may be shaped by the system's design stance. To support the claim that the gap is a property of human-algorithm translation rather than of this particular implementation, the paper would need a comparative condition (e.g., form-based elicitation, a human transcription baseline, or a different extractor) or a substantially more cautious interpretation.","section":"Section 4 (Study Design)"},{"comment":"The state machine's rigid goal-driven behavior—looping (n=12), insistence (n=14), and perceived lack of adaptivity—is described as evidence of a general 'ecological mismatch' and is used to motivate the 'adaptive negotiation' heuristic. However, this behavior is a designed property of the particular state machine and prompt templates, not a general property of LLM extraction or neurosymbolic systems. The state machine itself may even induce the non-linear corrections and looping users exhibited, partly manufacturing the observed 'gap.' Without varying the state machine (e.g., testing an adaptive variant) or at least acknowledging this as a design choice rather than an inherent limitation, the recommendation for adaptive negotiation is not empirically grounded by this study.","section":"Section 6.2 and Theme 2 (Section 5.2)"},{"comment":"The text states that the 27 codes were grouped into 'five themes reported below,' but then says 'We report three themes' and presents only three subsections (5.1, 5.2, 5.3). This inconsistency is confusing and should be corrected, either by reporting all five themes or by revising the earlier sentence to match the actual number.","section":"Section 5 (Qualitative Analysis)"}],"minor_comments":[{"comment":"There is a typo: 'independetly' should be 'independently.'","section":"Section 5"},{"comment":"Reference [23] contains the incomplete placeholder 'Accessed: [Insert access date here].' This should be filled in or removed.","section":"References"},{"comment":"The words 'fundamental structural paradox' and 'fundamental mental-model gap' are stronger than the single-system evidence supports; consider tempering the language unless comparative evidence is added.","section":"Section 6.1 and Conclusion"},{"comment":"Reporting the distribution of the 27 codes across the themes in a summary table would improve transparency and help readers assess the relative weight of the reported frequencies.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The qualitative data and the member-checking methodology are valuable, but the paper's contribution is overstated as a general claim about LLM linearity versus human narration. The core fixable issue is the attribution problem: the authors should either add a comparative condition (even a small one, such as a different extractor or a prompt variation) or reframe the paper as a single-system case study with design implications rather than fundamental structural claims. The inconsistent theme count and incomplete citation are minor and easily fixed. I would not recommend rejection because the empirical material and design heuristics have clear value for the HCI community if appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you. This is a carefully done qualitative HCI study with a genuinely useful evaluation method, and a central interpretive claim that outruns the evidence. The system is a neurosymbolic pipeline—LLM extractor, Neo4j knowledge graph, state-machine dialogue—and the evaluation shows users the extracted graph, the state machine trace, and a summary, then asks them to audit it. That 'user as arbiter of representational fidelity' move is the real contribution; I haven't seen it applied to knowledge-graph extraction from conversation elsewhere. The coding is serious: 221 coded units, codebook thematic analysis, kappas reported at 0.32, 0.71, and 0.79, with the inter-rater process documented. The themes—non-linear and hierarchical narration, unstated context, fuzzy time references, and users who voluntarily repair the system's model when they can see it—are credible and well-supported by quotes. For someone working on conversational JITAI or personal knowledge graphs, the design heuristics (hierarchical extraction, clarification over assumption, conversational playback) have a plausible empirical basis.\n\nThe soft spot is the paper's central claim. Section 6.1 states that 'strict linear architecture causes severe node duplication and fragmentation,' and the paper generalizes to a 'mental-model gap' inherent to LLM extraction. But the study never varies the extractor, the prompts, the merge logic, or the ASR. Every failure came from one pipeline: gpt-4o-mini via LangExtract, a specific state machine, and a voice front-end. The participants' non-linear narration is real, but the reported failures—co-referent duplicates, missing events, reversed order—could just as plausibly come from the model's coreference weakness, prompt design, or ASR errors, not from linearity as such. The state machine's rigid goal-driven loop may even help manufacture the very non-linearity and looping the users complained about. The limitations section notes the lab setting and small n, but does not acknowledge this attributional confound. That is load-bearing because the design heuristics are presented as responses to a fundamental property. If a stronger model or coreference-aware prompt eliminates most fragmentation, the contribution shrinks to a single-system case study.\n\nAdditionally, no data, prompt templates, or code are released, which weakens a study whose evidence is otherwise qualitative and whose system is described only loosely. And the citation used to support the 'linear extraction' claim ([72], a temporal reasoning benchmark) does not actually back it up.\n\nBottom line: it deserves a serious referee, but the referees should require either a comparison condition (stronger extractor, different prompts, or manual annotation of the same transcripts) or a rewrite that frames the gap as system-specific rather than fundamental. With that change, it is a solid venue paper; as written, it overstates. I'd send it to review with major revisions in mind.","headline":"Solid qualitative HCI study with a genuinely new evaluation method; the 'mental-model gap' attribution to inherent LLM linearity is untested and should not survive review as stated.","tokens_in":19275,"tokens_out":4466,"would_cite":true,"duration_ms":42173,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neurosymbolic voice agent that turns everyday talk into schedule knowledge fails precisely where linear extraction meets hierarchical human storytelling.","keywords":["Just-In-Time Adaptive Interventions","conversational agents","neurosymbolic AI","knowledge graph extraction","mental-model gap","routine elicitation","LLM temporal reasoning","receptivity"],"falsifier":"Run the same elicitation task on matched transcripts with several different LLM extractors and with a variant that allows hierarchical nesting and clarification; if duplicate nodes, missing events, and ordering errors largely disappear or change character across conditions, the 'mental-model gap' is an implementation artifact rather than a structural property of linear extraction.","tokens_in":18318,"feed_emoji":"🗣️","tokens_out":9042,"duration_ms":78132,"temperature":0.7,"pith_summary":"Just-in-time health interventions depend on knowing when a person is genuinely available to act, and this paper asks whether a conversational agent can learn that from a short spoken account of a day. The authors built a neurosymbolic pipeline—an LLM extracts events, locations, and free-time windows from each utterance and merges them into a persistent knowledge graph—and tested it by having 16 people narrate their routines and then audit the graphs and state-machine traces the system produced. They argue that a 'mental-model gap' separates human routine narratives, which are hierarchical, non-linear, and approximate, from the flat, turn-by-turn extraction LLMs perform, and that this gap produces duplicate entities, missing events, wrong ordering, and invented precision. They further identify an 'ecological mismatch' in which a schedule can show free time the user has no energy or willingness to use. If the paper is right, proactive health agents should extract hierarchically, ask instead of assume, piggyback new behaviors onto existing routines, negotiate adaptively, and scale their transparency to the user.","feed_headline":"Human routines are hierarchical, so linear AI extraction fails","feed_subtitle":"A 16-person study shows where voice assistants break when turning everyday talk into health-nudge schedules.","key_machinery":"The load-bearing mechanism is a neurosymbolic conversation loop. A state machine drives a voice agent through schedule elicitation, and at each turn an LLM extractor converts the latest transcript into typed entities (events, locations, free-time windows) and temporal relations, which a persistent knowledge graph accumulates across the conversation. Transition rules fire on quantitative conditions read from the graph or semantic conditions judged by a second LLM, letting the agent clarify ambiguous details, confirm understood events, and return to earlier states when the user revises something. Afterward, two log-derived artifacts—the state-machine trace and the knowledge-graph trace—are shown to the user; their job is to make the system's internal model inspectable so that participants can point to the exact node, missing event, or transition where their story was distorted. This combination of symbolic persistence and visual traceability is what turns an abstract 'gap' into a catalog of concrete, user-identified failures.","core_discovery":"The paper's central claim is that the friction between conversational routine elicitation and structured schedule data is not random noise but a systematic collision of two mental models. People narrate their day top-down and out of order, treating 'heading home' as one event that contains walking and taking the train, and assuming the listener already knows that 'office' means 'workplace' or that weekday patterns differ from Saturdays. The LLM extractor, by contrast, works bottom-up and linearly, so identical places become duplicate nodes, contained activities are split into unrelated events, stated commitments vanish or are reordered, and vague times are pinned to exact timestamps. The paper grounds this in a 16-participant member-checking study in which users audited the generated knowledge graph and state-machine trace, and it argues that the same collision extends from facts to feelings: a schedule can show free time that the user has no energy or motivation to use, which the authors call ecological mismatch. The proposed resolution is a set of design heuristics rather than a new algorithm: hierarchical extraction, clarification over assumption, contextual anchoring of nudges, adaptive negotiation, and scalable transparency.","pith_inferences":["Because the extractor, prompts, and state machine are never varied, a comparison across different LLM extractors and conversation flows would settle whether the 'mental-model gap' is intrinsic to linear extraction or specific to this implementation.","The fact that users eagerly corrected the graph when shown after the conversation suggests a live, in-chat editing surface—merging duplicate nodes or re-parenting sub-events while talking—could close the gap in real time, though the paper only tests post-hoc review.","The same hierarchy-versus-linearity friction should appear in any proactive agent that must act on user-described routines, such as calendar assistants or reminder systems, so the design heuristics may transfer beyond health interventions.","Combining this elicitation layer with wearable or physiological signals would let future work measure the ecological mismatch quantitatively, learning when a user's free time and receptive time diverge rather than only documenting the divergence qualitatively."],"forward_implications":["Routine-elicitation systems for JITAIs should store and reason over hierarchical event structures with parent events and sub-events, so that 'heading home' can contain 'walking' and 'taking the train' without duplication.","Extraction modules should treat vague time references as fuzzy bounds and respond to missing context by asking a clarifying question instead of assigning an arbitrary exact timestamp.","Intervention timing should piggyback on existing behavioral transitions, such as lengthening a commute walk, rather than proposing isolated high-effort activities, so nudges align with real energy levels.","Conversational agents should recognize user pushback, fatigue, or boundary-setting as a signal to pause their extraction agenda and negotiate, rather than looping to complete the state-machine goal.","Transparency should be role-scaled: developers and researchers get the full knowledge graph for debugging, while end users get natural-language playbacks that let them correct the model without managing graph data."],"supporting_citations":[{"why":"Documents LLMs' temporal-reasoning weaknesses, the evidence base for the paper's claim that linear extraction causes wrong ordering and invented precision.","marker":"[72]"},{"why":"Defines the zero-shot extraction paradigm through which the LLM turns narrative dialogue into typed entities and relations.","marker":"[73]"},{"why":"Establishes structured conversational memory as the relevant baseline and flags the lack of user-centered evaluation of extraction fidelity.","marker":"[79]"},{"why":"Defines JITAIs and the centrality of receptivity, the application context the system is designed for.","marker":"[50]"},{"why":"Shows mental workload predicts opportune moments for interruption, grounding the ecological-mismatch finding.","marker":"[30]"},{"why":"Supplies the fuzzy temporal constraint machinery that motivates the design heuristic of clarification over assumption.","marker":"[9]"},{"why":"Shows how granularity choices in LLM domain modeling produce duplicate and fragmented entities, explaining node duplication.","marker":"[59]"},{"why":"Provides the event-segmentation account of hierarchical event structure underlying the mental-model gap.","marker":"[77]"}],"fun_headline_variants":["Telling your day to AI fragments your routine","Linear AI extraction breaks hierarchical human stories","AI misses your energy when building your schedule","Why voice assistants botch your daily routine narrative","Design heuristics to align AI with human routine logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The core bet is that the observed failures come from the inherent linearity of LLM extraction rather than from this particular extractor, these prompts, and this state machine, which are never varied in the study.","fun_headline_variants_meta":{"raw":{"variants":["Telling your day to AI fragments your routine","Linear AI extraction breaks hierarchical human stories","AI misses your energy when building your schedule","Why voice assistants botch your daily routine narrative","Design heuristics to align AI with human routine logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1950,"prompt_tokens":944,"completion_tokens":1006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":937}},"tokens_in":560,"tokens_out":1006,"duration_ms":10547,"temperature":1.0,"reasoning_tokens":937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:57:20.491479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same elicitation task on matched transcripts with several different LLM extractors and with a variant that allows hierarchical nesting and clarification; if duplicate nodes, missing events, and ordering errors largely disappear or change character across conditions, the 'mental-model gap' is an implementation artifact rather than a structural property of linear extraction.","supporting_citations":[{"cited_title":"Iqbal and Brian P","cited_arxiv_id":null,"evidence_quote":"Shows mental workload predicts opportune moments for interruption, grounding the ecological-mismatch finding."},{"cited_title":"Reinhartz-Berger, S","cited_arxiv_id":null,"evidence_quote":"Shows how granularity choices in LLM domain modeling produce duplicate and fragmented entities, explaining node duplication."}],"review_version":1}