REVIEW 4 major objections 5 minor 17 references
Towards Probabilistic Question Answering Over Tabular Data
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that combining automatically learned Bayesian networks with LLM-based query translation outperforms retrieval-only and SQL-based approaches on probabilistic table questions.
desk verdict LUCARIO is a genuinely new benchmark resource, but the headline results measure LLM query translation against the benchmark's own ground-truth BN, not probabilistic reasoning over the table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the automatically learned Bayesian network over the table's columns, together with the premise and insight artifacts derived from it. A premise is a one-hop conditional statement, P(child | parents), covering one combination of node states; an insight is a small subset of premises ranked by KL divergence within the node's Markov blanket. The pipeline has four stages: learn the BN from the table; use the LLM to map the question to a target node and evidence node-state pairs; run exact inference (variable elimination) on the BN; and have the LLM verbalize the resulting probability. The BN carries the argument because all calibrated probability mass comes from exact symbolic inference, while the LLM is confined to the translation step, which the paper identifies as the main source of remaining errors.
What would settle it
Create a second version of a benchmark table whose ground-truth probabilities are computed from an independent source—say, a human-validated causal model or a network learned from a held-out portion of the rows—and run Auto-BN on the same questions. If most of its accuracy advantage over the baselines disappears, the edge came from sharing the same BN at benchmark construction and inference, not from probabilistic reasoning.
Extended reading notes
Core claim
The paper's central claim is that a hybrid symbolic-neural pipeline—auto-learned Bayesian network induction plus LLM-based query translation—substantially outperforms neural-only and retrieval-only alternatives for probabilistic QA over large real-world tables. On LUCARIO, Auto-BN with GPT-4o reaches 38.2% Acc0.02 and a mean absolute error of 0.103, compared with 24.4% Acc0.02 for the best LLM+Table baseline, and it does so while evaluated only on the hardest 'natural' question type to prevent lexical shortcuts. The authors interpret this as evidence that deterministic SQL retrieval cannot express conditional uncertainty, that large premise sets overwhelm LLM reasoning, and that grounding the LLM's answer in exact inference is what drives accuracy.
Load-bearing premise
The evaluation assumes the auto-learned Bayesian network is the correct source of ground truth, because the same learned network is used both to build the benchmark answers and to run Auto-BN, so the non-circular skill being measured is mainly how well the LLM translates questions into that network's node-state vocabulary.
Editorial extensions
If this is right
- Auto-BN can answer conditional-likelihood questions that NL2SQL systems cannot express, because deterministic SQL execution has no semantics for conditional uncertainty.
- Distilling a large premise space into a few high-impact insights improves LLM accuracy more than retrieving many premises, suggesting that focused evidence beats exhaustive evidence for probabilistic reasoning.
- Because Bayesian networks are learned automatically from data, the approach scales to tables with many columns and millions of rows without expert-curated structures.
- The residual error rate is driven by the LLM's node-state translation step, so improving query translation—not the inference engine—is the clear next lever for accuracy.
- LUCARIO's four question variants make benchmark scores robust to lexical shortcutting, since Auto-BN is only scored on the paraphrased Natural type.
Reading between the lines
- If the shared-BN circularity holds in practice, LUCARIO today measures question-to-vocabulary translation more than probability estimation; an external ground-truth source would certify the method's statistical claims.
- The same grounding recipe could transfer to causal questions by replacing exact BN inference with an interventional or counterfactual engine, retaining the LLM only for query parsing.
- The insight-distillation result suggests a cheap general recipe for retrieval-augmented LLM reasoning: rank a large rule base by divergence from the marginal distribution and prompt with the top few rules, instead of relying on retriever similarity alone.
- A robustness probe: run Auto-BN with a different structure-learning algorithm or a human-validated network; if accuracy is stable across structures, the method is robust to structural uncertainty.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LUCARIO, a benchmark for probabilistic question answering over tabular data, built from 10 BIRD tables, with automatically learned Bayesian networks, premise and insight artifacts, and four linguistic question types. Ground-truth probabilities for each query are computed by exact inference over the Bayesian network induced from the corresponding table. The proposed Auto-BN framework learns a BN from the table, uses an LLM to translate natural-language questions into structured probabilistic queries, performs exact inference, and optionally verbalizes the result. Experiments report that Auto-BN with GPT-4o achieves 38.2% Acc0.02 and MAE 0.103, outperforming LLM+Table, NL2SQL, and premise-retrieval baselines, and the paper concludes that hybrid symbolic-neural reasoning is effective for probabilistic tabular QA.
Significance. If the evaluation were sound, the paper would address a real gap: a large-scale, data-driven benchmark for probabilistic QA over real-world tables, with linguistically varied questions and structured artifacts such as premises and insights. The authors deserve credit for assembling large tables, defining four question types, performing exact inference, and taking steps to avoid lexical leakage by restricting Auto-BN to the Natural question type. However, the central claim is not supported as stated. Because the benchmark's ground truth is generated by the same Bayesian network that Auto-BN uses for inference, the reported Acc0.02 and MAE values primarily measure how often the LLM translator recovers the hidden structured query, not how well the system performs probabilistic reasoning over the table's actual distribution. In addition, Auto-BN is evaluated only on the Natural question type while Table 4 appears to aggregate question types, and all results are single runs with no significance testing. The benchmark may still be a useful resource if its ground-truth validity is established independently, but the current evidence does not support the paper's headline claims.
major comments (4)
- [§3.4, Appendix D.1] Section 3.4 states that the ground-truth probability for each query is computed by exact inference over a Bayesian Network induced from the corresponding table, and Appendix D.1 states that "the same BN is used during both benchmark construction and inference." Therefore, whenever the LLM correctly translates the natural question into the intended target variable and evidence node-states, Auto-BN returns the benchmark's ground-truth probability by construction. Consequently, the headline numbers in §5.2 (38.2% Acc0.02, MAE 0.103) measure translation fidelity into the BN's node-state vocabulary, not the quality of probabilistic reasoning over the table's actual data distribution. This is a load-bearing issue for the central claim, and the paper's own Section 8 concedes that the auto-learned structures "may not represent ground truth." The authors should either construct ground truth from an independent source (for example, empirical conditional frequencies or an externally validated BN) or explicitly reframe the benchmark as measuring query-to-structured-query translation.
- [§5.1, §5.2, Table 4, Appendix E] Section 5.1 says Auto-BN is "evaluated exclusively on the most challenging 'natural' question type," while §5.2 says Table 4 reports metrics "computed per (question type, baseline) pair." Table 4, however, has no question-type dimension and lists Auto-BN alongside baselines without indicating that its row is Natural-only. Appendix E provides question-type-level results for the baselines but not for Auto-BN. If the baseline rows aggregate all four question types while the Auto-BN row reports only Natural questions, the comparison is not apples-to-apples and the claim that Auto-BN "substantially outperforms all baselines across all metrics besides error-rate" is not justified. The authors should report per-question-type results for every method or evaluate all methods on the same question types.
- [Appendix D.1, Section 4.2, Section 8] Appendix D.1 refers to "a modified structure learning algorithm" but never specifies the algorithm, its hyperparameters, or the discretization procedure, so the reader cannot assess the quality of the induced BNs. Moreover, no experiment compares BN-inferred marginal or conditional probabilities with direct empirical estimates from the same tables, and Section 8 explicitly warns that the learned structures "may not represent ground truth." Since the benchmark answers are defined as exact inferences over these very BNs, the validity of the benchmark itself is unestablished. At minimum, the authors should validate the learned BNs against empirical frequencies or an independent gold-standard structure and report sensitivity to the discretization and structure-learning hyperparameters.
- [Footnote 2, Table 4] Footnote 2 states that each result is a single run per method per dataset, and the standard deviations in Table 4 are computed across the 10 datasets, not across repeated runs. Several key comparisons differ by only a few percentage points in Acc0.02 or by roughly 0.02–0.05 in MAE, yet no significance tests or confidence intervals are reported. The claim in §5.2 that Auto-BN "substantially outperforms all baselines across all metrics besides error-rate" is therefore not statistically supported, especially because the error-rate for Auto-BN (6.5%) is not lower than all baselines and the differences may reflect run-to-run variability.
minor comments (5)
- [Table 4] The row label "LLM+Premise(Vector))" contains an extra closing parenthesis, and Table 4 uses "Mixtral-8x21B" while Appendix D.7 and the reference list use "Mixtral-8x22B"; these should be reconciled.
- [Appendix D.1] There is a typo in "grouth truth" that should read "ground truth," and the sentence "This conservative setup enforces a strict separation between the input modality and the inference mechanism" would benefit from a clearer statement of what exactly is being separated.
- [Table 1] Table 1 uses ✓ and ✗ symbols without a legend; the authors should state explicitly what each symbol means (e.g., "supported" vs. "not supported") and define abbreviations such as "Prob Artifacts (Insights)."
- [Appendix B] The linguistic-quality check uses only 10 randomly sampled premises and 10 questions per table, which is a small sample; the Grammarly score of 96 should be presented as anecdotal rather than as a rigorous evaluation of fluency.
- [§3.2] The phrase "contains 10 real-world large-scale tables (table SQLite file and description)" is grammatically awkward, and "The statistics of table source included" should be rephrased; also, the paper does not clearly explain how the "causal" edges in the auto-learned BNs were validated beyond the small expert sample in Appendix B.
Circularity Check
Ground-truth probabilities and Auto-BN inference come from the same auto-learned Bayesian network, so the reported gains reduce to LLM query-translation accuracy.
-
self definitional
[Section 3.4 (ground-truth definition) and Appendix D.1 (Auto-BN inference)]
"Similar to existing benchmarks, the ground truth probability for each query is computed by performing exact inference over a Bayesian Network (induced from the corresponding table), based on a structured query derived from the natural language input. ... Because the same BN is used during both benchmark construction and inference, we take special care to prevent lexical leakage or shortcut exploitation."
Auto-BN's only symbolic component is exact inference over the BN; the LLM merely translates the question into target and evidence node-state pairs. The benchmark's ground truth is defined as exact inference over that same BN from the same type of structured query. Therefore, whenever the LLM translation is correct, Auto-BN's output equals the ground truth by construction, and the probabilistic-inference part cannot fail. The measured Acc0.02/MAE thus quantify how often the LLM recovers the hidden structured query, not whether the system reasons correctly about the table's actual distribution.
-
fitted input called prediction
[Section 3.3 (BN artifact), Section 5.2 (main results), Table 4]
"Bayesian Network (BN): A graphical model representing the dependencies and independencies among the table’s columns, automatically constructed based on the data distribution. ... Auto-BN substantially outperforms all baselines across all metrics besides error-rate."
The BN is automatically fit to each table and then serves simultaneously as (a) the source of benchmark ground-truth probabilities and (b) Auto-BN's inference engine. The headline results (38.2% Acc0.02, MAE 0.103) therefore report how often the LLM maps natural language into the BN's node-state vocabulary, not how well either the learned BN or Auto-BN matches the table's true distribution. No comparison of BN-inferred probabilities to direct empirical estimates from the data is provided, so the learned structure is never externally validated.
1 more flagged steps
-
other
[Section 8 (Ethical Considerations)]
"The learned structures reflect statistical estimations based solely on observed records, without incorporating broader world knowledge. As such, they may not represent ground truth and should not be used as generic justification or reliable decision making, such as investigations or healthcare applications."
The paper itself concedes that the auto-learned Bayesian networks 'may not represent ground truth,' yet the benchmark defines its ground-truth answers as exact inference over those same networks. This makes the evaluation self-referential: Auto-BN is scored against the very model it uses, so a high score does not establish that either the BN or Auto-BN captures the true probabilities of the table. The concession highlights that the non-circular residual is only the LLM's query-to-node translation.
full rationale
The central evaluation is circular in a specific, quotable way. Section 3.4 defines every ground-truth probability as exact inference over a Bayesian Network induced from the corresponding table. Appendix D.1 explicitly states that the same BN is used during both benchmark construction and Auto-BN inference. Consequently, whenever the LLM translator correctly identifies the target variable and evidence node-states, Auto-BN returns the benchmark's ground-truth probability by construction; the symbolic probabilistic reasoning is guaranteed correct because it is the same computation that generated the label. The only non-circular competence being measured is natural-language-to-structured-query translation, and the paper provides no baseline that pairs the same translator with an independent probability estimator. The paper's own Section 8 further concedes that the learned structures 'may not represent ground truth,' undermining the use of those same structures as ground truth. This is not a case of mere self-citation; it is a reduction of the headline claim to a fit by construction. The score is 8 rather than 10 because a genuine, non-circular component remains: the LLM's ability to map paraphrased natural questions to the correct node-state pairs, which is exactly the error rate the authors observe.
Assumptions & free parameters
free parameters (5)
- state_discretization_bins =
not reported (states per node averages 4.94 in Table 3)
- structure_learning_hyperparameters =
not reported
- premise_retrieval_top_k =
40
- insight_count =
20
- fallback_uniform_prior =
1/num_states
assumptions (4)
- domain assumption The auto-learned Bayesian network from each table is the correct probabilistic model for the benchmark questions.
- domain assumption Each natural language question maps uniquely to node-state pairs of the induced BN.
- domain assumption Learned edge directions can support the 'causal inference' query labels.
- standard math Standard Bayesian network factorization and exact variable elimination inference are valid.
Cite this review
Pith. "Pith review of Towards Probabilistic Question Answering Over Tabular Data." pith.science (2026). https://pith.science/paper/Y64ZEF7F
@misc{pith2026250620747,
author = {Pith},
title = {Pith review of: Towards Probabilistic Question Answering Over Tabular Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y64ZEF7F}},
note = {Machine review of arXiv:2506.20747}
}
read the original abstract
Current approaches for question answering (QA) over tabular data, such as NL2SQL systems, perform well for factual questions where answers are directly retrieved from tables. However, they fall short on probabilistic questions requiring reasoning under uncertainty. In this paper, we introduce a new benchmark LUCARIO and a framework for probabilistic QA over large tabular data. Our method induces Bayesian Networks from tables, translates natural language queries into probabilistic queries, and uses large language models (LLMs) to generate final answers. Empirical results demonstrate significant improvements over baselines, highlighting the benefits of hybrid symbolic-neural reasoning.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[4]
arXiv preprint arXiv:2406.05516
Verbalized probabilis- tic graphical modeling with large language models. arXiv preprint arXiv:2406.05516. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al
-
[5]
Mixtral of experts.arXiv preprint arXiv:2401.04088. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, et al
-
[6]
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang
Cladder: Assessing causal rea- soning in language models.Advances in Neural In- formation Processing Systems, 36:31038–31065. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024a. The dawn of natural lan- guage to sql: are we fully ready?arXiv preprint arXiv:2406.01265. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bail...
-
[7]
InFindings of the Association for Computational Linguistics ACL 2024, pages 9215–9235
Are llms capable of data-based statistical and causal reasoning? bench- marking advanced quantitative reasoning with data. InFindings of the Association for Computational Linguistics ACL 2024, pages 9215–9235. Aliakbar Nafar, Kristen Brent Venable, and Parisa Ko- rdjamshidi
work page 2024
-
[8]
Gpt-4o system card.Preprint, arXiv:2410.21276. Judea Pearl. 1988.Probabilistic Reasoning in Intelli- gent Systems: Networks of Plausible Inference. Mor- gan Kaufmann Publishers Inc., San Francisco, CA, USA. Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik
arXiv 1988
-
[9]
Nils Reimers and Iryna Gurevych
Chase-sql: Multi-path reason- ing and preference optimized candidate selection in text-to-sql.arXiv preprint arXiv:2410.01943. Nils Reimers and Iryna Gurevych
-
[11]
QUITE: Quantify- ing uncertainty in natural language text in Bayesian reasoning scenarios. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 2634–2652, Miami, Florida, USA. Association for Computational Linguistics. Chen Shen, Jin Wang, Sajjadur Rahman, and Eser Kan- dogan
work page 2024
-
[12]
Chess: Contextual harnessing for efficient sql synthe- sis.arXiv preprint arXiv:2405.16755. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al
Show all 17 references
-
[13]
Jianbin Zhao, Pengfei Zhang, Yuzhen Wang, Rui Xin, Xiuyuan Lu, Ripeng Li, Shuai Lyu, Zhonghong Ou, and Meina Song
Llama: Open and effi- cient foundation language models.arXiv preprint arXiv:2302.13971. Jianbin Zhao, Pengfei Zhang, Yuzhen Wang, Rui Xin, Xiuyuan Lu, Ripeng Li, Shuai Lyu, Zhonghong Ou, and Meina Song
-
[16]
slightly increase
and the abstraction techniques in CLADDER (Jin et al., 2023), we re- strict Auto-BN’s evaluation to the most challenging question category: theNaturaltype. These ques- tions describe variable states using paraphrased linguistic expressions (e.g., “slightly increase”) that do n...
2023
-
[17]
Given a question, the LLM first identifies target and evidence vari- ables
strategy of retrieving causal/premise statements and prompting the LLM to chain them in multiple steps. Given a question, the LLM first identifies target and evidence vari- ables. We then retrieve up to 40 relevant premises from a pre-generated store, using either BM25, dense ...
2024
-
[25]
• HybridRetrieve from both Vector and BM25 Retriever, combine and rerank the results
method that effectively ranks premises based on query term occurrence and rarity across the corpus. • HybridRetrieve from both Vector and BM25 Retriever, combine and rerank the results. 3https://www.grammarly.com D Baselines Implementation Details We compare our Auto-BN framew...
2024
-
[2007]
InIJCAI 2007, Proceedings of the 20th international joint conference on artificial intelligence, pages 2462–2467
Problog: A probabilistic prolog and its appli- cation in link discovery. InIJCAI 2007, Proceedings of the 20th international joint conference on artificial intelligence, pages 2462–2467. IJCAI-INT JOINT CONF ARTIF INTELL. Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu
2007
-
[2019]
InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics. Timo Pierre Schrader, Lukas Lange, Simon Razniewski, and Annemarie Friedrich
2019
-
[2023]
Violet Yinuo Han, Tianyi Wang, Hyunsung Cho, Kashyap Todi, Ajoy Savio Fernandes, Andre Levi, Zheng Zhang, Tovi Grossman, Alexandra Ion, and Tanya R
Text-to-sql empowered by large language mod- els: A benchmark evaluation.arXiv preprint arXiv:2308.15363. Violet Yinuo Han, Tianyi Wang, Hyunsung Cho, Kashyap Todi, Ajoy Savio Fernandes, Andre Levi, Zheng Zhang, Tovi Grossman, Alexandra Ion, and Tanya R. Jonker
-
[2024]
In Proceedings of the ACM Web Conference 2024, pages 2204–2215
Causal question answering with reinforcement learning. In Proceedings of the ACM Web Conference 2024, pages 2204–2215. Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, Axel-Cyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Ha- gen, and Martin...
2024
-
[2025]
price in- creases from 1% to 5%
Tqagent: Enhancing table- based question answering with knowledge graphs and tree-structured reasoning.Applied Sciences (2076- 3417), 15(7). A Benchmark Detailed Statistics As shown in Table 5, the Bayesian Networks in- duced in LUCARIO are substantially larger and more struct...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.