{"id":"1956026a-d688-4298-892d-dbaae9b11112","arxiv_id":"2507.18971","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dataset search interface with LLM-generated query reformulations, semantic column and granularity filters, and task-specific relevance indicators that helped 12 study participants explore and make sense of dataset search results.","lead":"DataScout is a dataset search tool that uses large language models to suggest better search queries, filter by column meaning and data granularity, and explain why each result fits the user's task. In a 12-person study, users found these features useful not just for narrowing results but for learning what data exists and refining their search goals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Success-rate comparison is confounded by per-task corpus augmentation: DataScout and the semantic baseline had 300 task-specific Kaggle results injected into their search space, so the 10/12 vs 7/12 and 6/12 success counts do not isolate interface benefit.","rationale":"The reader's stated weakest assumption is LLM metadata accuracy, and that is a real transferability risk. However, the most load-bearing condition for the central claim is that the observed improvement is attributable to DataScout's features. The task-level augmentation in Section 6 breaks that condition for every quantitative success and efficiency metric in Table 4, because the semantic conditions searched a corpus seeded with each participant's own top Kaggle results. This is not a minor limitation: the study's success definition is self-reported 'promising' datasets, and the augmented datasets are by construction the most likely to be judged promising. The paper is transparent about the augmentation and about many other limitations, which is to its credit, but the quantitative comparison cannot support the abstract's implication of measurable improvement until the confound is removed. The qualitative findings about proactive reformulations, filter suggestions, and relevance indicators remain valuable and are largely independent of the success-rate comparison; they support a conditional acceptance with the quantitative claims narrowed or repaired.","tokens_in":22528,"tokens_out":4955,"duration_ms":51973,"concrete_test":"From the session logs, determine for each participant in the DataScout and semantic-baseline conditions whether the dataset(s) they judged successful came from the 300 task-specific augmented datasets or from the original 6,500. If the majority of successful finds are from the injected 300, the success-rate advantage is explained by corpus curation. As a complementary check, rerun the semantic retrieval for each participant's first successful query with the injected datasets removed and compute the top-10 hit rate; if it drops to baseline levels, the augmentation is load-bearing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 6 states that after indexing 6,500 Kaggle datasets, the authors 'augmented our initial dataset collection by indexing 300 additional datasets, containing top 25 Kaggle dataset search results for each participant's task.' This was intended to prevent the semantic conditions from being restricted to popular datasets, but it makes the central comparison unfair. The Kaggle keyword baseline is the live Kaggle site, searched without any task-specific seed injection, while DataScout and the semantic baseline search a corpus that was deliberately curated to contain the top Kaggle results for each participant's own task. The success metric counts whether a participant found at least one dataset they judged promising; with high-relevance datasets injected into the semantic index, the reported 10/12 vs 7/12 and 6/12 success counts and the faster time-to-target (5.1 vs 7.0/7.5 min) are not identifiable as effects of DataScout's interface. The paper discloses the augmentation, but disclosure does not remove the confound. The qualitative sensemaking results may stand, but the quantitative claim that DataScout 'measurably improves' task success in dataset discovery does not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces DataScout, a dataset search interface that combines semantic search with three LLM-powered proactive assistance features: query reformulation suggestions grounded in the current result space, semantic attribute and granularity filters, and task-specific dataset relevance indicators. The authors motivate the design with a formative study (n=8) and evaluate with a within-subjects user study (n=12) comparing DataScout against Kaggle keyword search and a stripped semantic baseline, reporting quantitative metrics (success counts, time-to-target, ease-of-use ratings) and qualitative themes about steering, sensemaking, and conceptual model building. The paper argues that DataScout measurably improves exploratory dataset discovery and that its features are uniquely used for structured exploration and for gleaning feedback on search queries.","tokens_in":22766,"tokens_out":4344,"duration_ms":42010,"significance":"If the findings hold, DataScout makes a worthwhile contribution to HCI for data science: it operationalizes design considerations for exploratory dataset search and provides qualitative evidence that search-space-aware LLM suggestions function as feedback mechanisms. The paper is transparent about implementation, with appendices containing the prompts for metadata generation and online assistance, and the system description is detailed enough to reproduce. The strengths are the formative study, the within-subjects design, and the rich qualitative analysis with participant quotes. However, the headline quantitative claim of improved task success is currently threatened by a corpus-augmentation confound, and the system's reliance on unvalidated LLM metadata creates a transferability risk.","major_comments":[{"comment":"The comparison underlying the paper's headline success/failure numbers is confounded. The authors state that after indexing 6,500 Kaggle datasets they 'augmented our initial dataset collection by indexing 300 additional datasets, containing top 25 Kaggle dataset search results for each participant's task.' This means DataScout and the semantic baseline searched a corpus explicitly seeded with task-relevant datasets, while the Kaggle keyword baseline (condition A) was the live Kaggle site with no such task-specific seed injection. The success counts (10/12 vs 7/12 and 6/12), datasets explored (6.02 vs 3.33/4.25), and time-to-first-target (5.1 vs 7.0/7.5 min) therefore cannot be identified as effects of DataScout's interface; they may partly reflect the different, task-curated search space. Disclosure of the augmentation in the text does not remove the confound. Please either compare against a keyword baseline over the same augmented corpus, analyze the non-augmented subset separately, or substantially soften the claim that DataScout 'measurably improves' task success; also report whether any difference in success counts is statistically meaningful at this sample size.","section":"Section 6, Table 4"},{"comment":"DataScout's core features are built on LLM-generated metadata (summaries, purposes, column descriptions, granularity tags) and LLM-generated relevance indicators, with no ground-truth verification or error analysis. The prompt in Appendix C instructs the model to 'Stay factual' and 'Avoid hallucination,' but there is no mechanism that checks the generated text against the actual dataset contents; Section 5.3.4 even generates 'utilities' and 'limitations' from descriptions and example rows. If these outputs hallucinate or mischaracterize data, users would be misled, and the positive qualitative findings about relevance indicators would not transfer to other corpora or LLM backends. Please add a validation study (e.g., manual evaluation of a random sample of metadata and indicators, or a comparison against dataset-level ground truth) and discuss how accuracy failures affect the design conclusions.","section":"Sections 5.1 and 5.3.4"},{"comment":"The paper reports quantitative advantages for DataScout—'found relevant datasets sooner' (5.1 vs 7.0/7.5 min) and faster suitability assessment (37s vs 115/134s)—but does not report significance tests or per-participant distributions for these measures, and the success criterion is self-judged 'promising' with no fixed target datasets. With n=12 and three conditions, these numbers could be driven by a few participants or by the corpus difference noted above. Please report the underlying distributions, conduct appropriate paired tests (or justify their absence), and clearly label the success comparison as exploratory rather than confirmatory.","section":"Section 7, Table 4"}],"minor_comments":[{"comment":"The text 'DataScoutassists users' is missing a space between 'DataScout' and 'assists'; please fix the typo.","section":"Section 1"},{"comment":"The callout labels in Figure 2 are difficult to parse; for example, the label 'A' appears alongside '(11)' and 'B', and the caption does not map all labeled panels to the prose. Please clean up the labels and ensure each panel is referenced consistently in the text.","section":"Figure 2"},{"comment":"The sentence 'We hypothesize that the presence of relevance indicators but facilitate meta-cognition' appears to contain a typo: 'but' should likely be 'may'. Please correct it.","section":"Section 7.2.1"},{"comment":"The limitation that only two observations per condition order were recorded is acknowledged, but the paper should also state the implication for the order-effect claims in Section 7, where differences in feature appreciation by exposure order are reported without supporting statistical or systematic evidence.","section":"Section 8.3"},{"comment":"The citation [18] for the within-subjects repeated-measures design appears to be Hearst's 'Search User Interfaces'; consider citing a standard methods or experimental-design reference instead.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems-and-study paper with a clear design contribution, and the qualitative findings are likely salvageable. The main issue is that the abstract and Section 7 overstate the quantitative success comparison because of the corpus-augmentation confound and the absence of significance tests for success and timing metrics. My recommendation of major revision is based on the need to either re-run the comparison without the confound, analyze the non-augmented subset, or substantially qualify the claims. The lack of validation for LLM-generated metadata should also be addressed, as it affects the transferability of the positive results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nDataScout is a genuine attempt to fix a real bottleneck: dataset search. The paper's contribution is an integrated system that combines query reformulations generated from the actual search space, semantic filters over attributes and granularity, and on-the-fly relevance indicators. That combination is new relative to Olio and similar tools. The formative study (n=8) and the within-subjects evaluation (n=12) produce a rich set of observations — users treat reformulations and filter suggestions as implicit feedback, use them to build mental models of the dataset landscape, and even adjust their queries to steer the relevance indicators. The qualitative analysis is careful, with quotes that actually support the claims, and the authors are transparent about their method.\n\nThe main soft spot is the corpus augmentation. After indexing 6,500 Kaggle datasets, they added top-25 Kaggle search results for each participant's task. That means DataScout and the semantic baseline had task-specific relevant datasets seeded into their index, while the Kaggle baseline was the live site. The success-rate comparison (10/12 vs 7/12 vs 6/12) is therefore not a clean test of interface benefit. To their credit, the authors disclose this and say it was meant to avoid restricting the semantic conditions to popular datasets. But disclosure doesn't remove the confound. The comparison between DataScout and the semantic baseline is less affected, since both had the same augmentation — and yet DataScout still did better, which suggests the interface itself helps. Still, the paper should either reanalyze with augmentation removed or present the success counts as descriptive rather than causal.\n\nA second concern: the metadata and relevance indicators are generated by GPT-4o/mini without validation. The prompts explicitly instruct the model not to hallucinate, but there's no check against ground truth. If the LLM misdescribes a dataset's utilities or limitations, the user could be misled, and the positive findings about relevance indicators might not transfer. This is a real limitation, though not fatal for the qualitative claims.\n\nThe paper is well-written, the related work is well-situated, and the limitations section acknowledges most of these issues. The quantitative comparisons should be taken with a grain of salt, but the design considerations and the observed user behaviors are worth engaging with. This deserves a serious referee — I'd send it out. My own verdict: conditional accept with requests for a reanalysis of the success metrics and some validation of the LLM outputs.","headline":"DataScout is a solid, honest HCI systems paper with a real contribution, but the success-rate comparison is contaminated by task-specific corpus augmentation — read the qualitative findings, treat the numbers cautiously.","tokens_in":23314,"tokens_out":2827,"would_cite":false,"duration_ms":27602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Proactive AI assistance changes how users explore and understand dataset search spaces.","keywords":["exploratory dataset search","dataset discovery","LLM assistance","query reformulation","semantic search","relevance indicators","sensemaking","human-AI interaction"],"falsifier":"Take a held-out set of tables, run the same metadata-augmentation and relevance-indicator prompts, and compare every generated utility and limitation statement and granularity tag against the actual rows and a human label; if a material fraction of indicators assert facts not present in, or contradicted by, the preview rows (for example a time range, geographic scope, or \"No significant utilities\" verdict the data does not support), the trust and sensemaking effects reported in the study would not generalize.","tokens_in":22322,"feed_emoji":"🔍","tokens_out":6641,"duration_ms":67682,"temperature":0.7,"pith_summary":"Dataset search fails because users cannot express half-formed preferences and cannot see what datasets exist or why results appear. DataScout attacks both failures by making the search itself a conversation: it reformulates the user's query into options grounded in what is actually in the result set, offers filters over column meaning and row granularity, and generates per-dataset explanations of utility and limitation for the stated task. In a within-subjects study against keyword and plain semantic search, users of DataScout assessed dataset suitability faster, reached a promising dataset sooner, and succeeded more often. The qualitative finding is that users treated these proactive features as feedback channels, rewriting queries to steer the explanations and using filter suggestions as sanity checks that they were in the right part of the search space.","feed_headline":"Proactive AI search helps users judge datasets in 37 seconds","feed_subtitle":"DataScout grounds query and filter suggestions in live results, letting searchers sense the landscape and find useful data sooner.","key_machinery":"The load-bearing mechanism is a hybrid offline and online pipeline. Offline, each table is augmented by an LLM with summaries, column descriptions, source and collection details, granularity tags, and purposes, then embedded three ways: whole dataset, individual attributes, and purpose, with indexes built for dataset and attribute retrieval. Online, the user's query is expanded into three hypothetical table schemas whose embeddings are averaged against the dataset index; the retrieved result set is then clustered over purpose embeddings for reformulation suggestions and over attribute embeddings for column-concept filters, with the most frequent granularity tags offered as filters; and the top results receive LLM-generated utility and limitation indicators conditioned on the query and filters. This makes every suggestion a function of what the search actually returned, which is what lets DataScout ground reformulations in available data rather than open-ended generation.","core_discovery":"On its own terms, the paper claims that proactive, search-space-aware assistance changes how people discover datasets. DataScout generates three hypothetical schemas from the user's task to retrieve semantically relevant datasets, clusters the retrieved datasets by purpose to propose query reformulations that are guaranteed to have matching results, and clusters attributes to propose column-concept filters and granularity filters drawn from the result set. It then produces task-specific utility and limitation statements for inspected datasets. The user study reports that this combination shortened average suitability assessment from about two minutes in the baselines to 37 seconds, raised success from 7 of 12 and 6 of 12 participants to 10 of 12, and most distinctively led participants to treat suggestions as implicit feedback about query quality and dataset availability, building a conceptual model of the search space and even transferring that awareness back to baseline tools.","pith_inferences":["The reported \"query as knobs\" behavior suggests a design principle: in exploratory search, users actively steer the explanation generator; a testable extension is making the dimensions of relevance indicators explicit and user-selectable to see whether the sensemaking benefits persist.","If LLM-generated metadata is the weak link, the same interaction design could be re-run with human-verified or schema-derived annotations; the study does not establish how performance degrades under annotation error.","Grounding query suggestions in clusters of actual results is a transferable pattern: code search, document corpora, and other open-ended discovery tasks face the same gap between user intent and available items, and could reuse the cluster-then-reformulate mechanism.","The qualitative mechanisms of feedback loops, sanity checks, and conceptual model building could be quantified in a larger study by logging reformulation decisions and measuring post-task transfer to unassisted search."],"forward_implications":["Relevance indicators double as query feedback: when top-ranked datasets show \"No significant utilities,\" users infer that the search space lacks what they asked for and reformulate, turning a retrieval list into a diagnostic.","Attribute-level semantic filters recover datasets that exact column-name matching misses, including for expert users in their own domains; one participant found a relevant art-history collection they had not previously encountered.","Users apply the conceptual model learned from DataScout's suggestions even after switching to baseline tools, for example checking dataset upload dates more deliberately, so the sensemaking benefit is not confined to the interface itself.","Supporting both broad starting points of 50 or more results and targeted narrowing to 10 to 12 results fits the observed workflow of exploration followed by refinement, rather than a single retrieval mode."],"supporting_citations":[{"why":"Defines task-based dataset search and the broad challenges DataScout is positioned against.","marker":"[8]"},{"why":"Documents why practitioners find dataset search hard and motivates DataScout's design considerations.","marker":"[22]"},{"why":"Supplies the iterative exploratory-search model that DataScout's reformulation loop operationalizes.","marker":"[37]"},{"why":"Provides the information-foraging vocabulary of scents and patches used to describe search-space navigation.","marker":"[44]"},{"why":"Grounds the constraint-versus-preference discussion and the Information Diet Model referenced in the discussion.","marker":"[45]"},{"why":"The closest prior semantic dataset search system, which DataScout's semantic baseline builds on.","marker":"[51]"},{"why":"The sensemaking-based explanation framework used to interpret the timing effects of relevance indicators.","marker":"[24]"},{"why":"Related query-suggestion tool whose context-grounded suggestions DataScout extends to dataset search.","marker":"[43]"}],"fun_headline_variants":["Proactive search cuts dataset judging to 37 seconds","DataScout steers users to datasets in 37 seconds","Search-space-aware AI speeds dataset discovery","Users build search models from DataScout's proactive hints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes the LLM-generated dataset summaries, granularity tags, and relevance indicators are faithful to what the tables actually contain; if those annotations hallucinate or misdescribe the data, users would be misled and the study's measured benefits would not transfer beyond this corpus.","fun_headline_variants_meta":{"raw":{"variants":["Proactive search cuts dataset judging to 37 seconds","DataScout steers users to datasets in 37 seconds","Search-space-aware AI speeds dataset discovery","Users build search models from DataScout's proactive hints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1411,"prompt_tokens":930,"completion_tokens":481,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":418}},"tokens_in":546,"tokens_out":481,"duration_ms":5118,"temperature":1.0,"reasoning_tokens":418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:03:51.024901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of tables, run the same metadata-augmentation and relevance-indicator prompts, and compare every generated utility and limitation statement and granularity tag against the actual rows and a human label; if a material fraction of indicators assert facts not present in, or contradicted by, the preview rows (for example a time range, geographic scope, or \"No significant utilities\" verdict the data does not support), the trust and sensemaking effects reported in the study would not generalize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the iterative exploratory-search model that DataScout's reformulation loop operationalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the constraint-versus-preference discussion and the Information Diet Model referenced in the discussion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior semantic dataset search system, which DataScout's semantic baseline builds on."}],"review_version":2}