Pith. sign in

REVIEW 3 major objections 5 minor 18 references

ReliableTableQA shows that a few hundred stratified examples teach an LLM to flag statistically unreliable table answers, and that GRPO pays off only when supervised fine-tuning is under-trained.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:29 UTC pith:4YPZXN5R

load-bearing objection The headline result—200 SFT examples suffice and GRPO is null—does not survive Section 5.1: the model is fed oracle profiler statistics from the gold SQL, so the claims describe a supervised classifier head, not end-to-end reliability annotation. the 3 major comments →

arxiv 2607.20537 v1 pith:4YPZXN5R submitted 2026-07-10 cs.LG cs.AI

ReliableTableQA:How Much Supervision Does Reliability Annotation Need?

Huei-Chung Hu , Hsin-Tai Wu , Koyo Kobayashi This is my paper
classification cs.LG cs.AI
keywords reliability annotationtabular question answeringtext-to-SQLdata efficiencysupervised fine-tuningGRPOsynthetic data generationUCAR
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that teaching an LLM to judge whether a computed table answer is statistically trustworthy—as opposed to merely syntactically correct—is mostly a data-selection problem, not a scaling or reinforcement-learning problem. The authors build a ten-hazard taxonomy (R1–R10) for answerable-but-unreliable results, generate 50,000 labeled examples with a grammar-first pipeline, and show that a schema-stratified subset of about 200 examples lifts reliability-flag F1 from 0.61 to 0.98, pushes parse rate to 1.00, and eliminates high-confidence wrong answers (UCAR = 0). A held-out retail domain is handled without retraining. The same controlled sweep shows GRPO, widely assumed essential, yields measurable gains only when the SFT base is under-trained (at 100 examples) and is flat-to-negative once 200 adequate examples are seen. A sympathetic reader would care because the result reframes reliability annotation as a budgeted-data task and gives practitioners a concrete rule: invest in a small stratified set before reaching for RL.

Core claim

On the paper's own terms, the central discovery is that a small, schema-stratified supervised set—not a large corpus and not reinforcement post-training—is the load-bearing ingredient for calibrated reliability annotation. A model fine-tuned on only 200 examples drawn to balance all ten hazard categories across schemas reaches reliability-flag F1 0.978 and a parse rate of 1.000, drives the Unreliable Confident Answer Rate to essentially zero, and transfers to a fourth, unseen retail domain with Rel-F1 0.997. GRPO with executable rewards rescues the under-trained 100-example model (+0.06–0.16 exact-flag-set match, in and out of distribution) but contributes +0.000 on a hard compound-hazard sl

What carries the argument

The load-bearing object is the deterministic reliability profiler combined with a grammar-first synthetic-data generator. A context-free grammar emits structurally diverse SQL queries over public retail schemas; the profiler executes each query in a SQL engine and deterministically fires a ten-category hazard vector (R1–R10: small-sample aggregates, wide confidence intervals, multiple-comparison inflation, missing-data bias, subgroup imbalance, single-observation dominance, Simpson-reversal risk, distribution-tail mismatch, rare-event proportions, cross-tab sparseness). This yields exact multi-label ground truth that the LLM is trained to reproduce from the profiler statistics block, and the

Load-bearing premise

The model annotates reliability from a precomputed profiler statistics block, not from raw data; if those statistics are missing, insufficient, or wrong in production, the data-efficiency result reduces to a classification head on a hand-built deterministic profiler rather than autonomous reliability reasoning.

What would settle it

Collect a new schema with heavy missing data and skew (so R4/R8 can fire), provide the model only the raw table and question (no profiler statistics), and measure reliability-flag F1: if the SFT(200) model performs no better than zero-shot, the claimed reliability reasoning is just profiler-statistics mapping. Alternatively, if a random (non-stratified) 200-example set achieves the same 0.98 Rel-F1, then stratification is not the load-bearing mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can treat reliability annotation as a few-hundred-example task: a schema-stratified SFT set of roughly 200 examples suffices, and scaling to thousands adds nothing.
  • GRPO post-training should be reserved for under-trained bases; at adequate SFT it is a no-op and can slightly hurt answer accuracy on the hardest records.
  • The grammar-first, program-first generation pipeline yields dense coverage of rare hazards (R3, R8) that paraphrase-first LLM generation collapses.
  • A model trained on the three schemas transfers to an unseen retail domain without retraining, indicating it learns general reliability reasoning rather than schema-specific patterns.
  • The risk of confidently answering unreliable table results (UCAR) is essentially eliminated by the small SFT set alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the profiler statistics block is unavailable in production, the model's reliability judgment is expected to degrade toward the zero-shot baseline; an extension would be to train on raw tables with the profiler as a hidden teacher, testing whether the reasoning can be internalized.
  • The sharp threshold suggests a curriculum: first learn format and flag mapping from stratified examples, then use RL only if a held-out evaluation shows the base is under-trained; the paper's data could support a decision rule for when to deploy RL.
  • A schema-stratified selection procedure (balancing every hazard/schema cell) might be the actual driver of data efficiency rather than the number of examples; an ablation with random 200-example subsets would isolate this.
  • The taxonomy is a starting point; extending it to causal confounding, temporal autocorrelation, or measurement error would test whether the data-efficiency result holds for harder hazard spaces.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ReliableTableQA introduces a ten-category statistical-reliability taxonomy (R1–R10) for tabular QA, a program-first data pipeline that synthesizes 50k labeled examples from a CFG over retail schemas, and a controlled SFT-size ablation with GRPO. The central empirical claims are that 200 schema-stratified SFT examples suffice to reach reliability-flag F1 0.98, parse rate 1.0, and UCAR 0.0; that this transfers to a held-out retail domain; and that GRPO improves reliability annotation only when SFT is under-trained, with no measurable benefit once SFT is adequate. The paper also proposes UCAR as a metric for unreliable confident answers.

Significance. If the central claims hold, the paper usefully reframes reliability annotation as a data-efficiency problem and provides a concrete, deterministic, machine-checkable label-generation pipeline. The controlled SFT-size sweep, the hard compound-flag slice, and the held-out OOD evaluation are well-designed stress tests. The executable profiler and DuckDB-backed rewards are strengths: they make labels reproducible and avoid learned-reward-model noise. The significance is, however, bounded by the fact that the reliability metrics are evaluated on profiler statistics that appear to be oracle-derived from the gold SQL rather than from the model's own generated SQL. The data-efficiency and GRPO-null results may therefore describe a supervised classifier over oracle features rather than end-to-end reliability annotation of the system's actual answer.

major comments (3)
  1. [§5.1, §2, Table 3] The input context includes 'the full profiler statistics block (group sizes, CI width, null rates, etc.)' and the model 'annotates reliability given the profiler statistics, not by recomputing them from raw data.' Since the model emits its SQL in the same JSON response, a single pass cannot compute those statistics from the model's own SQL; no two-pass procedure is described. The only consistent reading is that the statistics block is derived from the gold SQL. Table 3 supports this: at SFT(200), answer accuracy is 0.723 while Rel-F1 is 0.978; if flags were based on statistics from the model's own (27.7% wrong) SQL, such high agreement with gold labels would be implausible. Therefore Rel-F1 and UCAR measure a classifier from oracle features to flags, not end-to-end reliability of the generated answer. UCAR=0.000 means only that the model never confidently answers gold-unreliable queries
  2. [§6.2, Table 3] At SFT(50), parse rate is 0.153 yet UCAR is reported as 0.000. Since UCAR is necessarily computed only on parseable outputs, the metric excludes 85% of generations at this setting. Reporting UCAR=0.000 without the denominator or a parse-conditional caveat is misleading. The same issue affects the SFT(100) row (parse rate 0.521). Please report UCAR on the full evaluation set with unparseable outputs counted as failures, and clearly state the conditional denominator in the table caption.
  3. [§6.3, Table 4] The central null claim — 'GRPO provides no measurable benefit once SFT is adequate' — rests on single-run deltas: +0.009 in-distribution, +0.000 on the hard slice, and -0.004 OOD. No confidence intervals, number of seeds, or significance tests are reported. The hard-slice discriminative check (+0.158 at S1) is a good idea, but the S2 differences are small enough that sampling noise could change the qualitative conclusion. Please provide at least a few random seeds or bootstrap confidence intervals for the exact-match deltas, and report the per-schema variance behind the aggregate in-distribution numbers.
minor comments (5)
  1. [§8] The limitation section says that in production a profiler must run before the model responds, but it does not acknowledge that the current evaluation uses oracle statistics from the gold SQL. Please add an explicit statement about which SQL is used to compute the statistics block in evaluation.
  2. [Figure 1] The inference panel shows SQL generation, execution, and reliability flags as a single flow, but the input to the model already contains the profiler statistics block. The figure should clarify whether statistics are computed from the generated SQL (two-pass) or supplied from the gold SQL (one-pass classification).
  3. [§8 / Answer accuracy] The paper notes that multi-row GROUP BY comparisons use exact row-set equality after JSON serialization without ORDER BY, and that DuckDB row ordering is non-deterministic. This is a real limitation; a canonical ordering (e.g., ORDER BY all columns) would make the metric reproducible. The current text correctly says it affects all models equally, but the non-determinism also affects the GRPO reward signal during training, which is worth stating.
  4. [§6.3] The text says 'the S2 contribution is exactly +0.000' on the hard slice. Given that exact-match deltas are computed from single runs and rounded to three decimals, '+0.000' should be reported with the underlying raw counts (e.g., 349 records) to convey the resolution of the measurement.
  5. [§4 and Table 2] The label distribution table reports R4 as 0.0% for Olist and Dunnhumby and R8 as 0.1% for SynCust. This is acknowledged, but a reader may infer per-label F1 from the aggregate Rel-F1. Please state explicitly that per-flag conclusions for R4 and R8 are not supported outside the schemas where those hazards occur.

Circularity Check

0 steps flagged

No circular derivation: the reported F1/UCAR numbers are held-out supervised-learning results on a self-defined profiler task; the oracle-statistics caveat is disclosed in Limitations, so the score is 1 rather than 0.

full rationale

The derivation chain is a standard supervised-learning loop: the deterministic profiler (Table 7) generates multi-hot R1-R10 labels, an LLM is SFT/GRPO-trained to reproduce those labels from the profiler statistics block, and performance is measured on held-out records. There is no equation in which a fitted parameter is renamed as a prediction, and no load-bearing self-citation. The paper is explicit that the model 'annotates reliability given the profiler statistics, not by recomputing them from raw data' (Section 5.1) and lists the resulting profiler dependence as a limitation (Section 8). The 200-example sufficiency and the GRPO null are therefore honest empirical findings about learning a deterministic threshold mapping from oracle features, not a derivation that assumes its conclusion. The reason the circularity score is not 0 is that the task and the R1-R10 threshold space are self-defined with no external benchmark: the high Rel-F1 and UCAR=0 validate the model against the same profiler that produced the labels, so the headline 'generalizes to an unseen domain' should be read as generalization of the statistics-to-flags mapping, not as evidence of autonomous reliability reasoning from raw table data. This is a scope caveat, not a circular step.

Axiom & Free-Parameter Ledger

11 free parameters · 6 axioms · 2 invented entities

The central claims rest on a substantial set of hand-chosen thresholds, reward weights, and sampling marginals, plus the design decision that the model sees precomputed profiler statistics. The taxonomy itself is the ground truth, which makes the benchmark self-contained but not externally validated. The GRPO null result is conditional on the specific reward weighting and single training runs.

free parameters (11)
  • R1 sample-size thresholds = n<5 abstain, n<30 warn
    Hand-set thresholds that define the most common hazard and the abstention decision; changing them changes ground-truth labels and all reported F1/UCAR numbers.
  • R2 CI-width threshold = 0.5 x |estimate|
    Hand-set; defines when a confidence interval is 'too wide'.
  • R4 null-rate threshold = >10%
    Hand-set missing-data bias criterion.
  • R5 imbalance thresholds = min/max < 0.1, or both < 10
    Hand-set subgroup-imbalance firing condition.
  • R6 driver-deletion count = 1-2 rows
    Hand-set jack-knife criterion for single-observation dominance.
  • R8 tail/skew thresholds = |skew|>2, mean/median>1.5
    Hand-set distribution-tail mismatch criteria; R8 label density varies strongly by schema.
  • R9 numerator threshold = numerator <30
    Hand-set rare-event proportion criterion.
  • R10 expected-cell threshold = expected cell count <5
    Hand-set chi-square rule-of-thumb for cross-tab sparseness.
  • GRPO reward weights = answer +0.30, reliability-F1 +0.25, SQL validity +0.15
    Chosen by hand; the GRPO null result could shift under different reward weighting.
  • GRPO KL coefficient = 0.04
    Hand-set hyperparameter that controls drift from the SFT reference policy.
  • Layer-1 sampling marginals = aggregate uniform; group-by .30/.50/.20; predicates .20/.40/.30/.10; etc.
    Hand-set grammar weights in Table 6 determine the hazard coverage of the 50k pool.
axioms (6)
  • domain assumption Statistical meaningfulness of a query result is fully determined by executed rows and column statistics.
    Underlies R1-R10 and the profiler-visible inference design; no external knowledge or causal model is needed. Invoked throughout Section 3.
  • ad hoc to paper The ten hand-thresholded hazards constitute the reliability label space.
    R1-R10 thresholds are chosen by the authors; every F1 and UCAR number is relative to this label space. Stated in Table 1 and Section 3.
  • domain assumption DuckDB execution plus exact-match or 1% tolerance is a valid oracle for answer correctness.
    Used for answer-accuracy rewards and evaluation; Section 5.3 and 6.1.
  • standard math Bootstrap CI, BH-FDR, Student-t, and Wilson interval procedures are correct statistical tools.
    Invoked in the deterministic profiler; Section 4 Layer 2 and Appendix A.3.
  • domain assumption Qwen3-4B-Instruct is a representative base model and the LoRA/full-FT setups are appropriate proxies for standard practice.
    All results are from one base model family and one training configuration; Section 5.
  • domain assumption Public retail schemas plus a synthetic table are representative of enterprise tabular QA.
    The OOD test is a single easier retail domain; generalization to proprietary enterprise schemas is acknowledged as unstudied. Section 6.4 and 8.
invented entities (2)
  • R1-R10 reliability taxonomy no independent evidence
    purpose: Defines the label space for post-execution statistical hazards in tabular QA
    The ten hazard definitions and thresholds are introduced by this paper and validated only against the authors' own profiler-generated ground truth, not against an external benchmark or domain-expert adjudication.
  • UCAR (Unreliable Confident Answer Rate) no independent evidence
    purpose: Quantifies the rate at which a system confidently answers an unreliable query without flagging it
    A new metric defined by the authors; its headline value of 0.000 is partly an artifact of excluding malformed outputs at low parse rates.

pith-pipeline@v1.3.0-alltime-deepseek · 11994 in / 12367 out tokens · 130910 ms · 2026-08-02T07:29:09.321979+00:00 · methodology

0 comments
read the original abstract

We introduce ReliableTableQA, a framework for training an LLM to annotate the statistical reliability of tabular QA results, not whether the query is answerable, but whether the computed answer is statistically meaningful. In real enterprise analytics, a syntactically correct SQL query can return a value that is based on too small a sample, has an excessively wide confidence interval, or is too confounded to support action. Existing systems answer confidently in all such cases, a failure we quantify as the Unreliable Confident Answer Rate (UCAR). We contribute (1) a ten-category reliability taxonomy (R1-R10) covering hazards such as small-sample aggregates, multiple-comparison inflation, and distribution-tail mismatch; (2) a program-first data pipeline that generates 50,000 reliability-labeled training examples from a context-free grammar over public retail schemas, with schema-stratified SFT/GRPO splits; and (3) a controlled study of how much supervision calibrated reliability annotation actually requires. We find that a small, schema-stratified SFT set is remarkably sufficient: 200 examples raise reliability-flag F1 from 0.61 to 0.98 and parse rate from 0.52 to 1.00, drive UCAR to zero, and yield a model that generalizes to an unseen retail domain (Rel-F1 0.997 on held-out H&M). Against this strong SFT baseline, GRPO, commonly assumed to be essential, helps only when SFT is under-trained (+0.06-0.16 exact-flag-set match at 100 examples, in- and out-of-distribution) and provides no measurable benefit once SFT is adequate, a null result we confirm across a hard compound-flag slice, a strict exact-match metric, and out-of-distribution evaluation. Our findings reframe reliability annotation as a data-efficiency problem and delineate precisely when reinforcement fine-tuning does and does not pay off.

Figures

Figures reproduced from arXiv: 2607.20537 by Hsin-Tai Wu, Huei-Chung Hu, Koyo Kobayashi.

Figure 1
Figure 1. Figure 1: Overview of the ReliableTableQA framework. (A) Data generation: a context-free grammar (Layer 1) emits structurally diverse SQL queries, a deterministic profiler (Layer 2) assigns R1–R10 reliability labels, deduplication and balancing (Layer 3) produce a 50 K labeled pool, and a small LLM (Layer 4) realizes natural-language questions. SynCust is an abbreviation of Synthetic Customer. (B) Fine-tuning: Qwen3… view at source ↗
Figure 2
Figure 2. Figure 2: Reliability annotation is data-efficient: a threshold, not a scaling, phenomenon. SFT-only (no GRPO) performance versus the number of schema-stratified SFT examples k (log scale), macro-averaged over the three in-distribution schemas (Synthetic Customer, Olist, Dunnhumby; 500 records each). All three metrics—output parse rate (valid JSON / SQL), reliability-F1, and answer accuracy—rise steeply across the s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [1]

    Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R

    Tao Yu, Rui Zhang, Kai-Chou Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Z. Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    doi:10.18653/v1/d18-1425 2018

  2. [2]

    Can LLM already serve as a database interface? A BIg bench for large-scale database grounded text-to-SQLs

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? A BIg bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing ...

    2023

  3. [3]

    AbstentionBench: Reasoning LLMs fail on unanswerable questions.arXiv preprint arXiv:2506.09038, 2025

    Polina Kirichenko et al. AbstentionBench: Reasoning LLMs fail on unanswerable questions.arXiv preprint arXiv:2506.09038, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    TrustSQL: Benchmarking text-to-SQL reliability with penalty-based scoring.arXiv preprint arXiv:2403.15879, 2024

    Gyubok Lee, Woosog Chay, Seonhee Cho, and Edward Choi. TrustSQL: Benchmarking text-to-SQL reliability with penalty-based scoring.arXiv preprint arXiv:2403.15879, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Reliable text-to-SQL with adaptive abstention

    Kaiwen Chen, Yueting Chen, Xiaohui Yu, and Nick Koudas. Reliable text-to-SQL with adaptive abstention. Proceedings of the ACM on Management of Data, 2025. doi: 10.1145/3709719

    doi:10.1145/3709719 2025

  6. [6]

    Carlin, Hal S

    Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin.Bayesian Data Analysis. CRC Press, 3 edition, 2013

    2013

  7. [7]

    Tukey.Exploratory Data Analysis

    John W. Tukey.Exploratory Data Analysis. Addison-Wesley, 1977

    1977

  8. [8]

    Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995

    Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. doi: 10.1111/j.2517-6161.1995.tb02031.x

    arXiv 1995

  9. [9]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025. 9 ReliableTableQA: How Much Supervision Does Reliability Annotation Need?A PREPRINT

    2025

  10. [10]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  11. [11]

    TRL: Transformer reinforcement learning, 2020

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforcement learning, 2020. URL https://github.com/huggingface/trl

    2020

  12. [12]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Yu Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Calibrating LLMs for text-to-SQL parsing by leveraging sub-clause frequencies

    Terrance Liu, Shuyi Wang, Daniel Preotiuc-Pietro, Yash Chandarana, and Chirag Gupta. Calibrating LLMs for text-to-SQL parsing by leveraging sub-clause frequencies. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. URLhttps://aclanthology.org/2025.emnlp-main.859/

    2025

  14. [14]

    Node-level uncertainty estimation in LLM-generated SQL.arXiv preprint arXiv:2511.13984, 2025

    Hilaf Hasson and Ruocheng Guo. Node-level uncertainty estimation in LLM-generated SQL.arXiv preprint arXiv:2511.13984, 2025

    arXiv 2025

  15. [15]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, and Sercan O. Arik. Reasoning-SQL: Reinforcement learning with SQL tailored partial rewards for reasoning-enhanced text-to-SQL.arXiv preprint arXiv:2503.23157, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Ilyas, and Christopher Ré

    Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré. HoloClean: Holistic data repairs with probabilistic inference.Proceedings of the VLDB Endowment, 10(11):1190–1201, 2017. doi: 10.14778/3137628. 3137631

    work page doi:10.14778/3137628 2017

  18. [18]

    how many orders per state?

    Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. Raha: A configuration-free error detection system. InProceedings of the 2019 International Conference on Management of Data, pages 865–882, 2019. doi: 10.1145/3299869.3324956. A Program-First Data Generation This appendix expands th...

    arXiv 2019