{"id":"d50204ec-b342-4c9e-8430-f93068a93080","arxiv_id":"2504.17334","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new system called DataScout lets data storytellers retrieve supporting and opposing data facts from the World Bank database through an LLM-based agent and a mind-map interface.","lead":"DataScout is an interactive system that helps people writing data stories by automatically finding facts that support or challenge a claim. It uses an AI agent to build a search tree shaped like a mind map, so authors can steer the search in the direction they need.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness claim rests on untested fact accuracy; Section 7.2 admits no quality evaluation of retrieved facts.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: the utility of DataScout depends on the LLM-based pipeline producing accurate, well-attributed, stance-consistent data facts. My reading of the paper supports this. The case studies and expert interviews are useful for demonstrating perceived value and usability, but they do not measure whether the retrieved facts are actually correct. The paper's own limitation statement, Section 7.2, explicitly says that the quality of retrieved data facts was not evaluated and that hallucination-induced failures were observed. This makes the effectiveness claim plausible but unsupported on the dimension that matters most for the stated goal of helping users verify statements and enhance credibility. I do not see this as a reason to reject the paper: the system is well-motivated, the design is clearly described, and the authors are transparent about limitations. The appropriate response is to require additional evidence, such as a fact-level accuracy audit, baseline comparison, and release of prompts and code, which is what the reader's conditional verdict already asks for. Therefore I recommend no change to the verdict.","tokens_in":16817,"tokens_out":2674,"duration_ms":26053,"concrete_test":"Run DataScout on the three statements from Section 6.1 using the same World Development Indicators database. Collect every fact in the resulting retrieval trees. Have two independent annotators, blind to the system's labels, check each fact against the source sub-table: are the numerical values, time range, aggregation, and caption consistent with the data? Also compare the predicted stance label with an independent stance judgment. Report fact-level precision and stance accuracy per statement. If precision or stance accuracy falls materially below the level experts assume, for example below 80%, the central claim should be qualified as useful but not yet reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DataScout can effectively retrieve multifaceted data facts from different stances, helping users verify statements and enhance credibility. For that claim to hold, retrieved facts must be numerically correct, faithful to their source sub-tables, and correctly labeled by stance. The paper's evaluation does not test any of these properties. Section 6 reports three case studies and expert interviews in which experts selected and praised facts, but no ground-truth audit of fact correctness, caption consistency, or stance agreement is reported. Section 7.2 explicitly states: 'We did not compare the user experience of DataScout with other data fact retrieval systems, nor did we evaluate the quality of the retrieved data facts.' The same section documents three hallucination-induced failures: caption-visualization mismatch, non-standard fact structure, and non-data-oriented queries. Because experts were shown generated charts and descriptions rather than independently verified numbers, their positive assessments establish perceived utility, not reliability. If a non-trivial fraction of facts are inaccurate or mismatched, using DataScout to augment statements could damage the credibility the system is designed to support. The load-bearing assumption, accurate, well-attributed, stance-consistent retrieval, is therefore both untested and partially contradicted by the paper's own limitation section.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DataScout is an interactive system for data-storytelling that uses an LLM-based agent (GPT-4o) to decompose a user's statement and stance into sub-queries, search a curated World Development Indicators database via embedding similarity and text-to-SQL, extract structured data facts with Chain-of-Thought prompting, and present results in a mind-map retrieval interface. The paper reports a formative study with four experts, three design goals, a system description, three expert case studies on climate change, gender gap, and aging population, and semi-structured interviews with three experts. The central claim is that DataScout can effectively retrieve multifaceted, stance-aligned data facts that help users verify statements and enhance story credibility.","tokens_in":17005,"tokens_out":2093,"duration_ms":20934,"significance":"If the central claim were fully supported, DataScout would be a useful contribution to data storytelling and human-LLM collaboration: it addresses a real workflow bottleneck (data fact search and extraction), proposes a concrete agent architecture with query decomposition, data search, fact extraction, and planning modules, and introduces a mind-map interface for steering multi-step retrieval. The formative study is a genuine empirical contribution, and the three case studies provide illustrative evidence of the system's interactive workflow. The paper also ships a clearly described pipeline that is reproducible in principle. However, the evidence presented does not yet establish the effectiveness claim, because the evaluation lacks any independent check of the correctness, faithfulness, or stance accuracy of the retrieved facts, and the paper explicitly acknowledges this gap in Section 7.2.","major_comments":[{"comment":"The central claim, stated in the abstract and Section 8, is that DataScout 'can effectively retrieve multifaceted data facts from different stances.' The evaluation in Section 6 consists of three expert case studies and interviews, but it contains no ground-truth audit of whether retrieved numbers are correct, whether captions match the visualizations, whether facts are truly generated from the displayed sub-tables, or whether the predicted stance labels agree with independent human judgment. Section 7.2 explicitly states: 'We did not compare the user experience of DataScout with other data fact retrieval systems, nor did we evaluate the quality of the retrieved data facts.' This admission is directly load-bearing: if a non-trivial fraction of facts are hallucinated, mismatched, or mislabeled, then using DataScout to augment statements could damage the very credibility the system is designed to support. The effectiveness claim therefore is not established by the reported evidence. I would like to see either a correctness audit on a sample of retrieved facts, a baseline comparison, or at minimum a systematic error analysis that quantifies the frequency and severity of the failure modes listed in Section 7.2.","section":"§7.2 (Limitations) and §6 (Evaluation)"},{"comment":"The stance probability is computed by asking the same LLM that generated the fact to label its own output as supporting or opposing the statement. This creates a self-referential loop: the fact-extraction prompt already instructs the LLM to produce stance-aligned facts, so the subsequent stance prediction is partly a restatement of the generation instruction rather than an independent measurement. The relevance score at least uses embedding similarity, which is independent of the generator, but the stance probability does not. The paper should either use a separate model or independent human raters for stance validation, or explicitly discuss this circularity as a limitation and temper the claims about stance alignment.","section":"§4.3.4 (Fact Extraction)"},{"comment":"The data search module's accuracy is not evaluated at all: there is no report of how often the text-to-SQL generation produced correct SQL, how often the top-three field selection retrieved relevant datasets, or how often query decomposition led to non-answerable sub-queries. Section 7.2 lists 'non-data-oriented query' as a known failure mode (e.g., Query 1-3-2 in Case Study III), which suggests this is not rare. Because the entire pipeline depends on these intermediate steps, the paper should report at least simple success-rate statistics for query decomposition, text-to-SQL execution, and field selection on the three case-study statements.","section":"§4.3.3 (Data Search)"}],"minor_comments":[{"comment":"The text says 'As shown in Fig x' but no figure number or figure itself is provided; this unresolved placeholder should be fixed.","section":"§4.3.2"},{"comment":"The phrase 'organizing them into q cohesive narrative' contains a typo ('q' should be 'a').","section":"§2.1"},{"comment":"Figure references are inconsistent: the paper uses 'Fig. 3-1', 'Fig. 3-(a)', and 'Fig. 6(d)' in close proximity. Please standardize the citation style for sub-figures.","section":"§6.3–6.5 and §5"},{"comment":"The references section contains placeholder tokens: reference [66] uses 'Year' and 'Accessed: YYYY-MM-DD', and the ACM Reference Format block contains '2018' and a dummy DOI. These should be updated before submission.","section":"§6.2 and §7.2"},{"comment":"The phrase 'as noted by E3-5' is ambiguous; it is unclear whether this means experts E3 through E5 or a different notation, and it should be clarified.","section":"§7.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the system is interesting, but the evaluation is substantially weaker than the claims. The authors' own Section 7.2 concedes the two most load-bearing missing pieces: no comparison with other systems and no evaluation of retrieved-fact quality. I see this as fixable with a nontrivial empirical addition rather than as a fatal flaw. Also, the POMDP formulation in Section 4.2 is not used in the evaluation; consider removing it or connecting it to measurable quantities, since as written it appears decorative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DataScout is a well-motivated systems paper with a genuinely new idea, but its central effectiveness claim is not supported by the current evaluation. The new bit is stance-based retrieval: decomposing a user's statement into support/oppose sub-queries, running an LLM agent with text-to-SQL and chain-of-thought fact extraction, and presenting the retrieval tree as a collaborative mind map. That integration is not in DataShot, Calliope, DataDive, or Self-Retrieval. The formative study with four experts is sensible and the design goals follow from it. The reference list is standard for the area and the writing is clear.\n\nThe soft spot is exactly where the stress-test note lands. The abstract says DataScout 'effectively retrieves multifaceted data facts' that help users verify statements and enhance credibility. For that to be true, facts need to be numerically correct, faithful to their source sub-tables, and correctly stance-labeled. The evaluation tests none of that. Three case studies and expert interviews show perceived utility; they did not audit fact accuracy, did not compare with any baseline, and reported no quantitative metrics. Section 7.2 is honest about this: 'We did not compare the user experience of DataScout with other data fact retrieval systems, nor did we evaluate the quality of the retrieved data facts.' The same section documents three hallucination-induced failures. Given the experts were reacting to generated charts and captions, not independently verified numbers, their positive comments carry weight for usability, not reliability. The mild self-referential loop (same LLM extracts facts and predicts their stance) is worth noting but secondary.\n\nI don't think this is a fatal flaw for a systems paper, but it is load-bearing. The contribution is the integration and the interaction design, both plausible. What it needs is a fact-accuracy evaluation: a sample of retrieved facts checked against the source database for numeric correctness, caption consistency, and stance agreement, plus at least a simple baseline comparison. Releasing prompts and code would strengthen it a lot.\n\nWho is this for: HCI and visualization people working on data storytelling and LLM-based agents. They'll get a clear idea of a promising interaction pattern and a reminder about evaluation gaps. I'd cite it cautiously for the stance-decomposition and mind-map tree idea.\n\nRecommendation: send to peer review with the expectation of major revision. This is not a desk reject.","headline":"DataScout introduces a genuinely new stance-based data fact retrieval workflow, but the paper's effectiveness claim rests on expert praise, not on any test of fact accuracy.","tokens_in":17537,"tokens_out":2712,"would_cite":true,"duration_ms":26341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DataScout claims an LLM-based agent can automatically retrieve stance-aligned data facts that let authors verify and augment written statements during data-story authoring.","keywords":["data fact retrieval","data storytelling","large language models","human-AI collaboration","stance-based retrieval","retrieval tree","text-to-SQL query generation","mind map interface"],"falsifier":"Take a fixed set of, say, fifty statements across the paper's three topics, run DataScout, and have independent raters check every retrieved fact against its displayed source for four properties: the visualization matches the caption, the fact has a valid five-part structure, the stance label agrees with the rater's judgment, and the query is answerable by the retrieved data. If the share of facts passing all four checks is low, or if stance labels agree with raters only at chance level, the central utility claim would be refuted.","tokens_in":16627,"feed_emoji":"🌳","tokens_out":6662,"duration_ms":58793,"temperature":0.7,"pith_summary":"DataScout is an interactive system whose stated goal is to automatically retrieve data facts that support or challenge a user's written statement, so that data-story authors can verify claims and add credible evidence without spending hours searching for data. The paper argues that a stance-aware LLM agent can do this by building a retrieval tree: the statement sits at the root, and each expansion decomposes the current query into multifaceted sub-queries, searches a curated international-development-indicators database, and extracts structured facts aligned with the requested stance. The retrieval tree is shown as a mind map, letting users steer the next branch while the agent recommends promising nodes. Through three expert case studies and interviews, the authors report that the system retrieved stance-aligned facts, such as rising particulate-emission damage and widening labor-force participation gaps, and that experts found the opposing-stance facts useful for reflecting on objectivity. The central claim, as stated, is that DataScout effectively retrieves multifaceted data facts that help users verify statements and enhance story credibility.","feed_headline":"DataScout mines supporting and opposing facts from any statement","feed_subtitle":"A mind-map retrieval tree lets authors verify claims and add credible evidence without manual data search.","key_machinery":"The retrieval tree is the central object. The user's statement is the root; every child node holds a sub-query, the data sub-table it produced, and the extracted facts, and expansion is modeled as a Partially Observable Markov Decision Process in which the user's choice of node-and-stance is the action and an LLM policy recommends the next node. The agent that drives the tree has four modules: planning, query decomposition, data search (embedding-based field matching followed by generated-and-executed SQL), and fact extraction (chain-of-thought selection of a {type, subspace, breakdown, measure, focus} tuple, followed by relevance and stance scoring). The interface renders the tree as a mind map with green and red stance colors and size-encoded relevance, which is what makes the agent's reasoning legible and steerable.","core_discovery":"The central claim is that stance-conditioned, tree-structured retrieval can supply relevant, attributable data facts that match a desired argumentative direction, and that surfacing the tree as a mind map gives authors enough control to make the automation trustworthy. On the paper's own terms, the contribution is the combination: a planning module recommends which node to expand next, a query decomposition module generates stance-aware sub-queries, a search module maps those sub-queries to data fields and generates SQL filters, and a fact-extraction module uses step-by-step reasoning to produce facts in a canonical five-part form and scores each by relevance and predicted stance. The authors report that experts consistently selected highly relevant, clearly stanced facts, expanded the agent-highlighted nodes, and described the system as broadening their thinking and improving efficiency. They conclude that stance-based retrieval helps validate the rigor and objectivity of an argument while enriching the narrative.","pith_inferences":["The stance and relevance scores could be turned into a quantitative benchmark: have independent raters judge whether retrieved facts genuinely support or oppose the statement and whether the top-ranked facts are the ones an author would use; the paper's expert interviews suggest but do not measure this.","A head-to-head comparison against plain LLM extraction or enumeration-based fact generation would isolate whether the retrieval tree itself, rather than the underlying LLM, drives the reported gains; the paper notes that no such comparison was run.","If the fixed development-indicators database were replaced by live web search, the same agent could cover arbitrary topics, but the paper's own failure cases, such as non-data-oriented queries and redundant fields, suggest query validation and source filtering would become the critical components.","The POMDP formulation invites treating retrieval quality as an optimizable objective, but the system uses it descriptively rather than learning a policy from reward; training the planning module on user feedback is a testable next step."],"forward_implications":["Data-story authors can verify and enrich a statement in minutes instead of spending the 10 to 30 percent of authoring time that the paper's formative study attributes to data search.","Opposing-stance facts become ordinary outputs of the tool, so writers can stress-test their claims and build more balanced narratives rather than only reinforcing an initial viewpoint.","Novices can follow the visible reasoning trail of the tree, offloading analytical skill to the agent while keeping control of retrieval direction.","The four-module decompose–search–extract–plan pipeline can be applied to other multi-step analytical tasks, such as retrieval-augmented generation and question answering over tables.","Because every fact carries an editable configuration and its source sub-table, stories can stay grounded in checkable data rather than in unverifiable LLM output."],"supporting_citations":[{"why":"Supplies the underlying large language model that performs decomposition, SQL generation, extraction, and planning in the agent.","marker":"[31]"},{"why":"Supplies the curated development-indicators database that the search module queries; the system's coverage is bounded by it.","marker":"[66]"},{"why":"Provides the five-tuple data-fact definition, type, subspace, breakdown, measure, and focus, that the extraction module is prompted to produce.","marker":"[47]"},{"why":"Provides the chain-of-thought prompting technique the extraction module uses to reason through the fact structure step by step.","marker":"[64]"},{"why":"Supplies the sentence-embedding model used to match sub-queries to data fields and to compute fact relevance scores.","marker":"[35]"},{"why":"Inspires the question-decomposition approach and the stepwise extraction strategy the agent adopts.","marker":"[16]"}],"fun_headline_variants":["DataScout: both-sides fact finder for any claim","LLM agent builds your evidence tree, all stances","Turn statements into credible stories with DataScout","Mind-map retrieval tree for stance-aware facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's usefulness collapses if the LLM pipeline frequently produces hallucinated numbers, mismatched captions, malformed fact structures, or queries that no dataset can answer; the paper's own Section 7.2 documents these failure modes, so the central claim depends on how often they occur.","fun_headline_variants_meta":{"raw":{"variants":["DataScout: both-sides fact finder for any claim","LLM agent builds your evidence tree, all stances","Turn statements into credible stories with DataScout","Mind-map retrieval tree for stance-aware facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1229,"prompt_tokens":865,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":481,"tokens_out":364,"duration_ms":4288,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:42:07.349811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of, say, fifty statements across the paper's three topics, run DataScout, and have independent raters check every retrieved fact against its displayed source for four properties: the visualization matches the caption, the fact has a valid five-part structure, the stance label agrees with the rater's judgment, and the query is answerable by the retrieved data. If the share of facts passing all four checks is low, or if stance labels agree with raters only at chance level, the central utility claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the underlying large language model that performs decomposition, SQL generation, extraction, and planning in the agent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the curated development-indicators database that the search module queries; the system's coverage is bounded by it."}],"review_version":1}