{"id":"5374bdff-1bfc-47c7-b784-b1e9d0c18128","arxiv_id":"2412.19718","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Text2Insight combines an LLM text-to-SQL step with a rule-based chart predictor and BERT-based question answering and prediction, but its end-to-end performance claims rest on circular or missing evaluation.","lead":"This thesis-style preprint describes Text2Insight, a pipeline that turns natural language queries into SQL, charts, and text insights using Llama3, Spacy, and BERT. The headline claim of 99 percent accuracy is not supported by the evaluation in the report, which relies on a circular BLEU-threshold measure and a handful of example outputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BLEU threshold is used to define SQL correctness, so the 99–100% accuracy claims are circular and unsupported.","rationale":"The reader's verdict is REJECT with high confidence, and my analysis supports that verdict without modification. The central claim is the abstract's 99% accuracy/precision/recall/F1 for Text2Insight. The only quantitative support for that claim is the Text2SQL evaluation in §5.2.1, where correctness is defined by a BLEU threshold of 0.5 and the confusion matrix is derived from that threshold. That is a circular evaluation: the metric does not measure whether the generated SQL returns the correct data; it only measures n-gram similarity to a reference string, and the threshold is selected after the fact to make all examples look correct. BLEU is inappropriate for SQL semantic equivalence because surface differences (column aliases, condition ordering, redundant parentheses, or equivalent joins) can produce low BLEU for correct queries, while high BLEU can accompany genuinely wrong queries that select the wrong column or filter incorrectly. The paper reports that with this threshold the accuracy, precision, recall, and F1 are all 1.0, which is an artifact of the thresholding procedure, not evidence of correct end-to-end behavior. Table 5.1 reports 99.25% syntactic validity, but syntactic validity is also not semantic correctness; a valid SQL query can still be semantically wrong. The end-to-end examples in §5.3 are five hand-selected cases with no evaluation protocol, no random sampling, and no error-rate computation, so they cannot support a 99% claim either. \n\nThe reader's stated weakest_assumption is that Spacy's en_core_web_sm similarity can match LLM-generated SQL keywords to actual CSV column names. This is a reasonable failure hypothesis, and the proposed execution-accuracy test would also expose failures in that refinement step. However, I see the more fundamental load-bearing concern as the circular BLEU-based evaluation: even if Spacy worked perfectly, the paper still would not have measured whether the final SQL is correct. The BLEU-based confusion matrix is a purely internal validity problem, and it is the actual basis for the headline numbers. Therefore I partially agree with the reader: they flagged the BLEU threshold in their rationale, but their formal weakest_assumption singled out Spacy rather than the circular evaluation. My proposed concrete test replaces the BLEU proxy with execution accuracy on the same dev set, which would settle whether the SQL generation and refinement steps actually produce correct queries. If execution accuracy is high, the central claim gains support; if it is low, the claim collapses. Either way, the reader's REJECT verdict remains the correct decision given the current evidence, so no verdict change is needed.","tokens_in":42454,"tokens_out":3485,"duration_ms":36620,"concrete_test":"Run the 665 spider-clean dev queries through the complete Text2Insight pipeline (Llama3 generation, Spacy refinement, SQLite/pandasql execution) and compare the result set of each generated query against the result set of the gold SQL executed on the same database, scoring exact result-set equality without using BLEU or any threshold. If execution accuracy does not approach 99%, the headline metrics are invalid; the same run will also reveal how often the Spacy similarity refinement produces non-executable or wrong-column queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline metrics in the abstract (99% accuracy, 100% precision, 99% recall, 99% F1) rest on the Text2SQL evaluation in §5.2.1. There, the authors define the \"correctness\" of a generated SQL query as BLEU ≥ 0.5 against the reference query, and then compute the confusion matrix in Figure 5.2 from that thresholded BLEU score. This is circular: the evaluation metric is used to construct the ground-truth labels. BLEU measures n-gram surface overlap, not whether the query returns the correct data. A query with high BLEU can still select the wrong columns, miss conditions, or produce an incorrect join; a semantically equivalent query with different aliases, ordering, or formatting can receive low BLEU. The reported 100% accuracy therefore only shows that every generated query reached a post-hoc BLEU threshold of 0.5, not that any query correctly retrieves the intended subset. The end-to-end evaluation in §5.3 adds only five hand-picked examples, one of which is an error case, and provides no sampling procedure, baseline, or error rate. Thus the central quantitative claim lacks any non-circular measure of SQL correctness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This thesis-style manuscript proposes Text2Insight, a multi-model pipeline that converts natural-language queries over a user-supplied CSV file into SQL, executes the SQL to obtain a data subset, predicts a chart type, generates a chart with Matplotlib/Seaborn, and produces natural-language insights with Llama3. The system also includes a BERT-based question-answering model and a BERT-based predictive model for cricket match outcomes. The headline empirical claims are 99% accuracy, 100% precision, 99% recall, and 99% F1-score with a BLEU score of 0.5 for the full Text2Insight system, plus 89% QA accuracy and roughly 70% accuracy for the binary predictive model.","tokens_in":42666,"tokens_out":4449,"duration_ms":46124,"significance":"If the stated performance were real, the work would be a useful demonstration of composing off-the-shelf LLMs, NER-based refinement, and rule-based chart selection into an NL-to-visualization system. The authors do describe a plausible pipeline and honestly list several limitations in Section 6.2. However, the central quantitative claims are not supported by the evaluation: the 99% overall accuracy is built on a circular BLEU-based correctness definition, the end-to-end evaluation is a set of five hand-picked examples with no scoring protocol, the chart-type predictor has no measured accuracy, and the insight-generation component is not evaluated at all. The paper ships no code, no data, and no reproducible evaluation harness, so the contribution is currently a system description rather than a validated result.","major_comments":[{"comment":"The reported 100% accuracy, precision, recall, and F1 for the Text2SQL model are circular. The paper defines a generated SQL query as 'correct' when its BLEU score against the reference query is at least 0.5, and then computes the confusion matrix from exactly that thresholded BLEU score. BLEU measures n-gram surface overlap, not whether the query returns the intended rows and columns; a high-BLEU query can still select wrong columns or omit conditions, while a semantically equivalent query with different aliases or ordering can receive low BLEU. The abstract's 99% accuracy claim and the end-to-end 'BLEU score of 0.5' therefore rest on this circular definition. A non-circular evaluation, such as execution-match accuracy on the Spider dev split, is required before any correctness claim can be made.","section":"§5.2.1 (Table 5.2, Figure 5.2)"},{"comment":"The end-to-end evaluation consists of five hand-picked input queries on a single ODI dataset, one of which is an intentional error case. There is no sampling procedure, no error rate, no comparison baseline, and no scoring rubric for what constitutes a correct chart or insight. These examples can at most demonstrate that the pipeline produces plausible output on selected inputs; they cannot support the headline '99% accuracy' statement. A systematic evaluation over a defined query set with human or automated correctness labels is needed.","section":"§5.3 (Figures 5.4–5.8)"},{"comment":"The chart-type prediction method is not actually evaluated. Table 5.3 restates the method's own decision rules (e.g., categorical plus quantitative implies bar chart) rather than reporting accuracy on a test set. The text immediately notes that the method fails on negative preferences such as 'not a bar chart' and does not generalize to subsets with more than five columns. No confusion matrix, accuracy figure, or error analysis is provided, so the chart predictor's contribution to the claimed end-to-end performance is unmeasured.","section":"§5.2.2 (Table 5.3)"},{"comment":"The insight-generation model receives no evaluation of its own. The section defers to Figure 5.3, which is reproduced from Meta's official page for Llama3, and the accompanying text cites the model's pre-training on 15 trillion tokens. These are not measurements of the quality of the insights generated by the Text2Insight pipeline. Since insight generation is an advertised component of the system, the absence of any metric for it means that no part of the '99% accuracy' claim is accounted for by this stage.","section":"§5.2.3 (Figure 5.3)"},{"comment":"The SQL refinement step is load-bearing for the system's claim of working on arbitrary user-supplied CSVs. It uses Spacy's 'en_core_web_sm' similarity index to replace words in the LLM-generated SQL with the closest column names in the user's schema. The paper provides no evaluation of this mapping on unseen schemas, no ablation showing how much the refinement improves execution success, and no failure analysis. The only evaluation involving refinement uses the Spider dev dataset, where the schema of the reference query is known in advance, so it does not test the difficult case of an arbitrary CSV schema. Given that the Section 6.2 discussion acknowledges the model performs best on small subsets, the robustness of this step remains an open question that directly affects the end-to-end claim.","section":"§3.3.3 and §6.2"}],"minor_comments":[{"comment":"The statistical analysis states there are 1,093 matches, 12 teams, and 34 cities, while the chapter summary in Section 4.7 reports 19 teams and 36 cities; these numbers need to be reconciled.","section":"§4.4.1 vs. §4.7"},{"comment":"The paper describes Llama3 as 'trained on 15 trillion parameters' in Section 3.3.2 but later says '15 trillion tokens' in Section 6.2; the latter is the correct quantity for pre-training data.","section":"§3.3.2 and §6.2"},{"comment":"The text says the spider-clean-text-to-sql dataset is divided into train (6,016), dev (665), and test (1,929), then states that 'due to the smaller size of the train dataset, the study utilizes only the train component,' yet the reported evaluation uses 665 queries, which is the dev-set size. Please clarify which split was actually used.","section":"§5.2.1"},{"comment":"Table 5.4 lists a runtime of 14.2854 without units, while the text says 'an evaluation runtime of 14.28 minutes'; the units should be made consistent.","section":"§5.4 (Table 5.4)"},{"comment":"Figure 5.3 is captioned 'Performance Matrix for Llama 3' but is taken from Meta's official page; it should be explicitly labeled as an external source and clearly distinguished from the paper's own experimental results.","section":"§5.2.3 (Figure 5.3)"},{"comment":"The title and abstract use 'Text2Insight' while the body consistently uses 'Text2Insights'; pick one spelling and use it throughout.","section":"Title and Abstract"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a Master's thesis rather than a peer-reviewed archival paper. The central quantitative claims are unsupported because the main evaluation is circular (BLEU threshold defines correctness and then yields 100%) and the end-to-end evaluation is anecdotal. While the system description could be useful as a technical report, the level of validation is well below the standard for publication in a serious journal. A revision would require a fundamentally new evaluation with execution-match accuracy on a standard benchmark, a properly sampled end-to-end user study, and an evaluation of the chart predictor and insight generator; that is beyond a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a thesis write-up, not a research paper. It builds a natural-language-to-visualization pipeline out of existing components (Llama3, Spacy, BERT) plus a hand-written chart-predictor rule list, and the headline 99% accuracy numbers don't survive contact with the evaluation section. The BLEU-based SQL correctness evaluation is circular: the authors define a \"correct\" query as one with BLEU >= 0.5, then report that all queries exceed that threshold. The end-to-end evaluation is five hand-picked examples, one of which is an error case. The chart predictor is \"evaluated\" by the authors observing that their own rules work.\n\nThat said, the paper does a few things well. The literature review is broad and current, covering Text2Chart, ChartLlama, Chat2VIS, NL4DV, and the Text-to-SQL and NER lines. The architecture is described in enough detail that someone could reimplement the pipeline, and the handling of invalid queries (asking the user for a better input) is a sensible practical choice. The QA and predictive BERT models are fine-tuned on a home-grown IPL dataset and report plausible standalone numbers—though without baselines or dataset release, those numbers are hard to interpret.\n\nThe soft spots are load-bearing. The 99/100/99/99 figures in the abstract come from the circular BLEU evaluation, not from any non-circular measure of SQL correctness. The Spacy similarity refinement step (Section 3.3.3) is also under-specified: the paper doesn't explain how the similarity index maps arbitrary LLM-generated keywords to actual column names, and the stress-test note's concern is real. The chart predictor is a hand-coded decision list, which is fine as an engineering choice but not a research contribution, and the paper itself admits it fails on negative preferences and datasets with more than five columns.\n\nWho is this for? A reader curious about the anatomy of a student thesis on applying LLMs to visualization might skim it. But as a research submission, it lacks baselines, code, data, and a non-circular evaluation. I would not send this to peer review; I would desk-reject it with an invitation to resubmit if the authors add a real evaluation, compare against Chat2VIS or a simple \"generate code and execute\" baseline, and release their code and datasets.\n\nRecommendation: reject.","headline":"A thesis-style engineering write-up whose headline accuracy numbers are circular; the literature review and architecture description have some value, but the evaluation does not support the claims.","tokens_in":43205,"tokens_out":2747,"would_cite":false,"duration_ms":28091,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text2Insight claims a five-model pipeline can turn a plain-English data request into an accurate SQL query, a fitting chart, and a written insight.","keywords":["natural language to visualization","text-to-SQL","chart type prediction","Llama3","Spacy similarity","BERT question answering","predictive modeling","cricket dataset"],"falsifier":"Run Text2Insight on a fresh table with non-obvious column names (for example, 'TTL' instead of 'total_runs') and a query asking for 'total runs'; if the Spacy refinement replaces the generated column token with a dissimilar schema name, or leaves it unmatched, the SQL query fails or returns the wrong rows, and the claimed end-to-end accuracy does not reproduce. A quantitative version would compare 100 generated-and-refined SQL queries against reference SQL on a held-out table and count errors the paper's metrics did not capture.","tokens_in":42208,"feed_emoji":"📊","tokens_out":7711,"duration_ms":72001,"temperature":0.7,"pith_summary":"Text2Insight claims to remove the static, pre-set limits of current visualization tools by converting a natural-language request and a CSV file into a customized chart and a written insight. If the claim holds, non-specialists could ask data questions in plain English and receive an accurate SQL-generated data subset, a sensible chart type, and a concise summary without manual configuration. The paper reports 99.25% syntactic accuracy, 100% precision, 99.25% recall, and an F1 of 99.62% for the text-to-SQL stage on the Spider development set, with a BLEU score of 0.5 at the chosen similarity threshold. It also reports 89% accuracy for a BERT question-answering model and 70% accuracy for a binary predictive model, both trained on Indian Premier League cricket data. The contribution is an engineering architecture rather than a new mathematical or algorithmic identity.","feed_headline":"A five-model pipeline turns plain English into charts and insights","feed_subtitle":"Text2Insight claims 99% accuracy converting natural-language requests into SQL, charts, and insight text","key_machinery":"The load-bearing object is the sequential multi-model pipeline: the input CSV is read with Pandas and NumPy to extract shape, column names, data types, primary key, and schema, then loaded into SQLite; a pre-trained decoder-only Llama3 model converts natural-language text into an SQL query; Spacy's en_core_web_sm model refines the query by matching tokens in the generated SQL to the closest column names via word similarity; the refined query executes on the SQLite database to produce a data subset; a chart-predictor method inspects the subset's data types (categorical, continuous, univariate, bivariate, multivariate, time-series) and selects among ten chart types, or honors an explicit chart preference; Matplotlib or Seaborn render the chart; and Llama3 again generates insights. Two BERT models extend the system: a distilbert question-answering model with TF-IDF context retrieval, and a bert-base-uncased sequence classifier for binary and ten-class match prediction.","core_discovery":"The paper claims that Text2Insight, a multi-model pipeline, reliably turns a user's natural-language request and a tabular CSV file into a validated SQL query, a chart, and a written insight. In the pipeline, a pre-trained Llama3 model generates an initial SQL query; Spacy's en_core_web_sm similarity index refines it by replacing column-like tokens with the closest names in the input schema; the refined query runs against a SQLite version of the CSV; a chart-predictor method chooses among ten chart types from the data types of the resulting subset; and Llama3 again produces a 500-word insight summary. The paper also reports a BERT-based question-answering model at 89% accuracy and a BERT-based match-outcome predictor at roughly 70% (binary) and 49% (ten-class) accuracy on IPL cricket data. The headline claim is that Text2Insight's text-to-SQL stage reaches 99.25% syntactic accuracy with BLEU-score correctness at a 0.5 threshold, and that the end-to-end model handles simple and complex queries on its cricket dataset.","pith_inferences":["The reported 99% figures describe SQL syntax and BLEU similarity at a 0.5 threshold; they do not by themselves establish that the final chart answers the user's intent, since a syntactically valid query can still select the wrong columns.","The chart predictor's ten-chart, data-type-driven rules suggest the system is most reliable on small subsets; wide tables or mixed-type schemas would likely fall through to the default bar or line chart, so cross-domain generality is an open question.","The QA and predictive models were trained and tested only on IPL cricket contexts; applying the same recipe to healthcare, finance, or research tables would require new labeled context datasets and would likely shift the reported accuracies.","A stronger test than BLEU would be execution-based: compare the rows returned by the refined SQL with the rows a human expects, since that directly checks the semantic correctness the pipeline is designed for."],"forward_implications":["Users can obtain a chart and a 500-word insight from a plain-English query without manually choosing chart type or axes.","The SQL-refinement step lets the same pipeline work with arbitrary CSV column names by similarity matching, rather than requiring a fixed schema vocabulary.","Queries that do not specify a chart still receive a visualization, because the chart predictor selects among ten chart types based on the data types in the result subset.","The BERT question-answering model answers factual natural-language questions about historical match data, including player scores and match outcomes.","The BERT predictive model forecasts match winners in both binary (two teams) and ten-team classification settings from natural-language match contexts."],"supporting_citations":[{"why":"Supplies the nvBench dataset that defines the natural-language-to-visualization task and the benchmark context this system builds on.","marker":"Luo et al., 2021"},{"why":"Supplies the Spider development set used to measure the Text2SQL stage's syntactic accuracy and BLEU score.","marker":"Yu et al., 2018"},{"why":"Provides the Chat2VIS LLM-based text-to-visualization approach that this work positions against.","marker":"Maddigan and Susnjak, 2023"},{"why":"Provides the Text2Chart multi-stage generator whose x- and y-axis mapping limitations motivate the chart-prediction component.","marker":"Rashid et al., 2021"},{"why":"Supplies the survey of text-to-SQL and text-to-visualization datasets and LLM methods that frames the architecture.","marker":"Zhang et al., 2023b"},{"why":"Provides the NL4DV toolkit that turns natural-language queries into analytic specifications, a related text-to-visualization baseline.","marker":"Narechania et al., 2021"}],"fun_headline_variants":["Text2Insight turns plain English into charts with 99% accuracy","Five-model pipeline: natural language to SQL, charts, insights","Text2Insight: 99% accuracy from English text to data visuals","Plain English to SQL and charts: Text2Insight hits 99%","Text2Insight's text-to-SQL achieves 99.25% syntactic accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on Spacy's word-similarity step matching whatever column-like words the LLM puts in the SQL query to the actual column names in a user-supplied CSV; if that lexical match fails, the query errors or silently returns the wrong subset.","fun_headline_variants_meta":{"raw":{"variants":["Text2Insight turns plain English into charts with 99% accuracy","Five-model pipeline: natural language to SQL, charts, insights","Text2Insight: 99% accuracy from English text to data visuals","Plain English to SQL and charts: Text2Insight hits 99%","Text2Insight's text-to-SQL achieves 99.25% syntactic accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":4061,"prompt_tokens":1069,"completion_tokens":2992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":2893}},"tokens_in":685,"tokens_out":2992,"duration_ms":20278,"temperature":1.0,"reasoning_tokens":2893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:55:46.609125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Text2Insight on a fresh table with non-obvious column names (for example, 'TTL' instead of 'total_runs') and a query asking for 'total runs'; if the Spacy refinement replaces the generated column token with a dissimilar schema name, or leaves it unmatched, the SQL query fails or returns the wrong rows, and the claimed end-to-end accuracy does not reproduce. A quantitative version would compare 100 generated-and-refined SQL queries against reference SQL on a held-out table and count errors the paper's metrics did not capture.","supporting_citations":[{"cited_title":"Text2Chart: A Multi-Staged Chart Generator from Natural Language Text","cited_arxiv_id":"2104.04584","evidence_quote":"Provides the Text2Chart multi-stage generator whose x- and y-axis mapping limitations motivate the chart-prediction component."},{"cited_title":"and Stasko, J., (2021) NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries","cited_arxiv_id":null,"evidence_quote":"Provides the NL4DV toolkit that turns natural-language queries into analytic specifications, a related text-to-visualization baseline."}],"review_version":1}