{"id":"78f880ab-f6bf-49f1-8287-3c26d5ae160b","arxiv_id":"2506.01710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"Reinforcement learning with rule-based rewards improves table question answering and SQL generation over supervised fine-tuning, though several 'first' claims are undercut by concurrent work.","lead":"The paper trains 7-billion-parameter language models with reinforcement learning on a wide range of table-reading tasks, reporting better accuracy than supervised fine-tuning and average scores above a large commercial model. It also introduces a position reward that encourages models to point to the exact cells they use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline RL-over-SFT margins and the Claude-surpassing comparison are selected on the test set, with no validation split or seeds; this can inflate the central empirical claim.","rationale":"The paper is a broad, useful empirical study, and the direction of the result—RL improves over SFT for table reasoning—is consistent across many datasets and aligns with concurrent RL-for-reasoning work such as DeepSeek-R1 and DAPO. The most load-bearing vulnerability is not the sign of the effect but the magnitude and the SOTA comparison: because all ablations and hyperparameter choices are made on the same test sets used for the headline numbers, the reported margins are not protected against selection on the evaluation metric. This is especially important for the 4.0-point Claude margin, which is an average over seven tasks and comes from a configuration chosen after inspecting test results. The absence of error bars or multiple seeds compounds the issue. The 'first application of RL to table reasoning' claim is also contradicted by the paper's own citations (Table-R1, SQL-R1, Reasoning-SQL), but that is a novelty overstatement rather than a threat to the empirical trend. The reader's weakest-assumption focus on reward fidelity is reasonable, yet for the primary tableQA and text-to-SQL experiments the rule-based reward is essentially the evaluation metric, so the larger correctness risk is the test-set selection protocol. If the proposed validation re-run holds, the empirical claims should be accepted conditionally; if it does not, the headline comparisons should be weakened accordingly.","tokens_in":26812,"tokens_out":6627,"duration_ms":70986,"concrete_test":"Rerun the unified tableQA experiments with a held-out validation split (e.g., 10% of each training set) used to select reward weights (λ1, λ2), data-ablation choice, early stopping, and any extra reward such as the SQL N-gram term. Then evaluate the single selected configuration on the official test sets with three random seeds, reporting mean ± std. If RL-zero and Reason-SFT+RL still beat both SFT baselines on the seven-task average and still exceed Claude-3.7-Sonnet by roughly 4 points, the central claim survives. Additionally, tune the BIRD N-gram reward on a validation subset rather than on the reported BIRD dev numbers, and report whether the selected configuration differs from the one in Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that GRPO with rule-based rewards beats SFT and reaches Claude-3.7-Sonnet-level tableQA—rests on a protocol in which modeling choices are made on the test set. Table 1 explicitly states: \"We only use the train set for training and the test set for evaluation, without using the validation set.\" The reward-configuration study in §5.1 (Tables 3–4), the data-ablation study in §5.2 (Fig. 3), the BIRD N-gram addition (§5.1, App. C.2), and the model-size analysis are all reported on the same test/dev sets used for the headline numbers. Selecting λ1, λ2, data filtering, and extra rewards from test performance can inflate the 60.12/62.62 averages and the 4.0-point margin over Claude, especially with no error bars and apparently single-seed runs. The reader's reward-proxy concern is real but secondary here: for tableQA and text-to-SQL the rule-based reward is essentially the reported evaluation metric, so optimizing it is not obviously a corruption; the unprotected evaluation protocol is the more load-bearing threat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Reasoning-Table, a GRPO-based reinforcement-learning pipeline for table reasoning. The method trains Qwen2.5-7B on tableQA, fact verification, table-to-text, and text-to-SQL using a rule-based outcome reward, optionally combined with a format reward and a novel position-evidence reward. The main empirical claims are that RL outperforms SFT across all task families, that unified RL training on seven tableQA datasets reaches an average of 62.62 and surpasses Claude-3.7-Sonnet by about 4.0 points, and that the method improves robustness and out-of-domain generalization. The authors release their curated datasets and code.","tokens_in":27071,"tokens_out":4412,"duration_ms":49341,"significance":"If the central claims hold, the paper would be a valuable contribution: it demonstrates that simple rule-based outcome rewards can drive reinforcement learning for structured table reasoning, introduces a position-evidence reward that is intuitively useful for table-grounded reasoning, and provides a broad empirical comparison across four task families with released data and code. The robustness and generalization experiments are also informative. However, the significance is currently bounded by the evaluation protocol: the headline margins and the comparison to proprietary models are selected or reported without a held-out validation set, multiple seeds, or confidence intervals, and the 'first application of RL to table reasoning' claim is explicitly contradicted by the paper's own related-work section. These issues need to be addressed before the quantitative claims can be taken at face value.","major_comments":[{"comment":"The paper states that 'We only use the train set for training and the test set for evaluation, without using the validation set.' Yet all ablation and configuration decisions—reward weights in Tables 3 and 4, the data-version choice in Figure 3, the N-gram reward added in App. C.2, and the model-size analysis in Figure 4—are evaluated on the same test/dev sets used for the headline numbers. Selecting λ1, λ2, data filtering, and extra rewards from test performance can inflate the reported RL-over-SFT margins (60.12 and 62.62 averages) and the 4.0-point margin over Claude-3.7-Sonnet. The authors should either introduce a held-out validation split and report the final numbers on a truly untouched test set, or report multiple seeds with means and confidence intervals for all key configurations.","section":"§4.1, Table 1 footnote"},{"comment":"The comparison to Claude-3.7-Sonnet, GPT-4o, and Qwen2.5-32B-Instruct appears to be zero-shot prompting, since these models are not fine-tuned on the training portions of the benchmarks, whereas Reasoning-Table is trained on those exact datasets. The abstract's claim of 'surpassing larger proprietary models' therefore conflates a fine-tuned model with an inference-only baseline. The paper should explicitly state the inference protocol for all baselines and either add stronger fine-tuned baselines or soften the claim from 'state-of-the-art' to 'state-of-the-art among fine-tuned open 7B models at the time of writing.' There is also an internal inconsistency: §4.2 says 'Reason-SFT+RL surpasses Claude-3.5-Sonnet', while the abstract and Table 1 refer to Claude-3.7-Sonnet.","section":"§4.2, Table 1"},{"comment":"The rule-based outcome reward is a central design choice, but the paper itself provides evidence that it is an imperfect proxy: in §5.1, the LLM-as-a-judge reward improves WikiTQ from 80.35 to 81.23, yet the paper does not adopt it in the main training runs. Since the rule-based reward is essentially the same metric used for evaluation, this is not a circularity, but it is a correctness-risk concern. If acceptable answers are systematically mislabeled as incorrect, the RL signal is corrupted and the RL-over-SFT conclusion may partly reflect reward optimization rather than genuine reasoning improvement. The paper should quantify agreement between the rule-based reward and the LLM judge, and justify the decision to keep the rule-based reward in the main experiments despite the WikiTQ result.","section":"§3.4, §5.1, Table 3"},{"comment":"The abstract and introduction repeatedly claim that Reasoning-Table is 'the first application of reinforcement learning (RL) to table reasoning.' However, §6 cites Table-R1 (Yang et al., 2025; Wu et al., 2025b) as sharing the same motivation and applies RL to table reasoning. This is an internal contradiction in the novelty claim. The 'first' wording should be relaxed, for example to 'one of the first' or 'the first unified systematic study across four table-reasoning task families,' unless the authors can specify a precise technical sense in which their contribution is strictly prior.","section":"Abstract vs. §6 (Related Work)"}],"minor_comments":[{"comment":"The caption says 'using the GPRO RL method'; the algorithm used throughout the paper is GRPO. Please correct the typo.","section":"Figure 1 caption"},{"comment":"The headings 'Reason-SFT Data Quailty Control' and 'SFT Data Quailty Control' contain a misspelling of 'Quality.'","section":"App. A.1, App. A.2 headings"},{"comment":"In Table 12, both FeTaQA and TAT-QA are attributed to (Nan et al., 2022); TAT-QA should be attributed to Zhu et al. (2021), and FeTaQA to Nan et al. (2022).","section":"Table 12"},{"comment":"The notation 'T5 i=1 Pi' for the union of position sets is ambiguous; the 'T' likely represents a union symbol that did not render. Please use a clear union notation such as ⋃_{i=1}^{5} P_i.","section":"§3.2"},{"comment":"The column headers in Table 8 lack spacing ('Origin(No-Reason Data)After Stage1After Stage2'), making the table difficult to read.","section":"Table 8"},{"comment":"The text says 'using the strongest OmniSQL model as the base, Reasoning-Table achieves a performance of 69.0 in the majority voting setting'; Figure 5 appears to show multiple lines, and it would help to mark explicitly which line corresponds to the 69.0 value and whether this is the same as the 68.3 reported in the abstract or a different configuration.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The core empirical direction is promising and the released assets are valuable, but the evaluation protocol is the main risk. Because all configuration choices are reported on the test/dev sets that also produce the headline numbers, the authors cannot currently distinguish a genuine RL advantage from selection on the evaluation set. This is fixable with a held-out validation framework and/or multi-seed reporting. The 'first' novelty claim should also be reconciled with the cited concurrent Table-R1 work before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper's headline claim—'first application of RL to table reasoning'—is wrong, and the paper itself disproves it by citing Table-R1, SQL-R1, and Reasoning-SQL. What's left after that overclaim is still a solid, useful empirical study. GRPO with rule-based outcome rewards beats SFT across nearly every dataset in a sweep covering tableQA, fact verification, table-to-text, and text-to-SQL, and the unified-training numbers on 7 tableQA datasets are consistently better than the SFT baselines. I believe that trend is real; it matches concurrent work and the direction of the field.\n\nWhat the paper does well: the benchmark sweep is genuinely broad—15 datasets, 4 task families, single and unified training. The position reward, which rewards annotating the specific cells used in reasoning, is a sensible extension of schema linking to cell-level grounding, and the ablation suggests it helps (0.6–1.1 points on some sets). They also release data and code, report out-of-domain generalization (AIT-QA, TableBench), test-time scaling, and robustness to perturbation. Those are the parts I'd want to read carefully, but they're the right experiments to run.\n\nThe soft spots are mostly about the evaluation protocol. Table 1 explicitly states that they use the test set for evaluation without validation. Then the reward configuration (§5.1), the data ablation (§5.2), the N-gram addition, and the model-size analysis are all reported on the same test sets used for the headline numbers. With no error bars and apparently single-seed runs, the reported margins—especially the 4.0-point edge over Claude—are likely inflated by test-set selection. The comparison to proprietary models is also zero-shot, not fine-tuned, so 'surpassing Claude' is not an apples-to-apples result. The rule-based reward being equal to the eval metric is not itself a corruption; it means the RL is optimizing what gets measured. The more serious issue is choosing λ1, λ2, data filters, and extra rewards based on test performance.\n\nMinor: the position evidence is generated by a Qwen2.5 model, same family as the policy, which is a mild circularity. And the paper's own §5.1 shows an LLM-as-judge reward helps WikiTQ but isn't used in the main runs, which suggests the rule-based reward is imperfect on at least one benchmark. Those are minor compared to the test-set problem.\n\nVerdict: this deserves a serious referee. The empirical core is plausible and reproducible; the overclaim about 'first' needs to go, and the evaluation protocol needs a validation split or at least a disclosure of what was selected on test. I'd recommend conditional accept with major revisions, or, for a workshop, a strong poster with the overclaims fixed. I'd bring it to a reading group and would cite it for the breadth of the RL-over-SFT evidence.","headline":"A broad RL-over-SFT sweep for table reasoning whose 'first' claim is contradicted by its own citations, but whose empirical core is plausible and worth refereeing seriously.","tokens_in":27619,"tokens_out":3006,"would_cite":true,"duration_ms":29893,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GRPO-based reinforcement learning with simple rule-based outcome rewards outperforms supervised fine-tuning across tableQA, fact verification, table-to-text, and text-to-SQL.","keywords":["table reasoning","reinforcement learning","GRPO","rule-based reward","text-to-SQL","supervised fine-tuning","table question answering","large language models"],"falsifier":"Compare the paper's rule-based reward against an oracle semantic-equivalence reward (human- or LLM-judged) on a dataset like WikiTQ where many gold answers have multiple valid phrasings, training identical GRPO runs otherwise. If the oracle-reward model beats the rule-based-reward model by a large margin on human-judged correctness, the rule-based reward is not a faithful correctness proxy and the central claim fails for answer-equivalence tasks.","tokens_in":1734,"feed_emoji":"📊","tokens_out":1934,"duration_ms":96023,"temperature":0.7,"pith_summary":"Reasoning-Table aims to establish that reinforcement learning—specifically GRPO (group-relative policy optimization) with rule-based outcome rewards—can replace supervised fine-tuning as the post-training method for LLMs on table reasoning. The paper argues that SFT's imitation bias limits generalization and robustness, whereas RL lets the model explore reasoning paths and optimize against a direct correctness signal. Across tableQA, table fact verification, table-to-text, and text-to-SQL, RL-zero beats both No-Reason SFT and Reason-SFT, and cold-starting RL from SFT reasoning data improves results further. In unified training over seven tableQA datasets, the 7B model reaches 62.62 average EM, about four points above Claude-3.7-Sonnet, and on the BIRD text-to-SQL benchmark it achieves 68.3% execution accuracy. The paper also claims improved robustness to table perturbations and stronger out-of-domain generalization.","feed_headline":"RL training beats supervised fine-tuning on table reasoning","feed_subtitle":"A 7B model tops Claude-3.7-Sonnet on tableQA and hits 68.3% on BIRD text-to-SQL","key_machinery":"The load-bearing machinery is the reward function $$R(o_i)=R_{\\text{ans}}(o_i)\\times(1+\\lambda_1 R_{\\text{pos}}(o_i))+\\lambda_2 R_{\\text{fmt}}(o_i),$$ trained with GRPO's group-relative advantage estimate. $R_{\\text{ans}}$ is a binary rule-based outcome reward defined per task: exact match for short answers, F1 threshold for long strings, SQL execution match for text-to-SQL, and BLEU threshold for long-form and table-to-text outputs. $R_{\\text{pos}}$ rewards cell-level annotations that match ground-truth reference cells, and the multiplicative coupling ensures it only contributes when the answer is correct. This mechanism lets the model learn from its own rollouts rather than from teacher-generated chain-of-thought, with an optional cold start from filtered Reason-SFT data.","core_discovery":"The central claim, stated on the paper's own terms, is that applying GRPO reinforcement learning with simple rule-based outcome rewards to table reasoning produces a model that outperforms SFT on every task family tested. The final reward couples answer correctness with table-specific evidence: $R_{\\text{ans}}$ is exact match, F1 threshold, SQL execution match, or BLEU threshold, while a position reward $R_{\\text{pos}}$ pays for annotating the actual cells and columns used in reasoning, and a format reward enforces the expected output structure. In the unified tableQA setting, RL-zero raises the average from 49.80 (No-Reason SFT) to 60.12, and Reason-SFT+RL reaches 62.62, surpassing Claude-3.7-Sonnet by about 4.0 points. On text-to-SQL, the method reaches 68.3% execution accuracy on the BIRD dev set with a 7B model. The paper further claims that RL, unlike SFT, remains robust under column and row perturbations and transfers to out-of-domain table datasets.","pith_inferences":["The paper leaves implicit that its data-ablation result—training on 'challenging' examples beats using all data—points toward curriculum or difficulty-sampled RL as a natural way to improve sample efficiency.","Because the LLM-as-judge reward improved WikiTQ over rule-based matching, replacing the rule-based reward with an equivalence-aware judge is a plausible extension that could lift performance on free-form and multi-format answers.","The position reward only contributes when the answer is correct; a natural extension would be to convert it into a dense process signal during rollouts for multi-hop table reasoning.","The authors' stated lack of a code-execution module for tableQA implies that tool-augmented RL, executing Python or SQL during rollout, is a likely next step for reducing arithmetic hallucinations."],"forward_implications":["GRPO reinforcement learning with rule-based outcome rewards can replace or complement SFT across tableQA, table fact verification, table-to-text, and text-to-SQL, with larger gains on reasoning-intensive tasks.","Unified multi-task RL lets a 7B model beat a much larger proprietary model (Claude-3.7-Sonnet) on tableQA benchmarks, indicating that task diversity in RL is a scaling axis.","RL-trained models are more robust than SFT models to table serialization changes and column or row order perturbations, and transfer better to out-of-domain tables.","Test-time scaling works: increasing rollouts improves pass@n and majority-voting accuracy on BIRD, with the best 7B configuration reaching 68.3% execution accuracy.","Data selection matters: filtering RL training data to 'challenging' examples yields better performance than using all data or only unstable examples."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that the RL training loop is built on.","marker":"Shao et al. (2024)"},{"why":"Establishes the R1 paradigm that RL elicits reasoning traces and motivates the cold-start and exploration setup.","marker":"Guo et al. (2025a)"},{"why":"Provides the empirical contrast that SFT memorizes while RL generalizes, which frames the paper's central comparison.","marker":"Chu et al. (2025)"},{"why":"TableGPT2 is the table-specific domain model baseline that Reasoning-Table must outperform.","marker":"Su et al. (2024)"},{"why":"OmniSQL is the strong SQL base model used for the best text-to-SQL results on BIRD.","marker":"Li et al. (2025)"},{"why":"DAPO provides the Clip-Higher and response-truncation modifications adopted in the RL training.","marker":"Yu et al. (2025)"},{"why":"Defines the BIRD benchmark and execution-accuracy metric used for text-to-SQL evaluation.","marker":"Li et al. (2024)"},{"why":"Defines the Spider benchmark used for text-to-SQL evaluation.","marker":"Yu et al. (2018)"}],"fun_headline_variants":["Reinforcement learning outperforms supervised tuning for table reasoning","RL-trained 7B model surpasses Claude-3.7-Sonnet on table benchmarks","Table reasoning via RL hits 68.3% on BIRD with a 7B model","First RL approach to table reasoning beats SFT across tasks","Rule-based rewards make RL win on tableQA and text-to-SQL"],"cache_read_input_tokens":29696,"weakest_assumption_plain":"The rule-based outcome reward—exact match, F1 threshold, SQL execution match, or BLEU threshold—faithfully identifies every acceptable answer; if it mislabels correct answers as wrong or wrong answers as right, the reinforcement signal is corrupted.","fun_headline_variants_meta":{"raw":{"variants":["Reinforcement learning outperforms supervised tuning for table reasoning","RL-trained 7B model surpasses Claude-3.7-Sonnet on table benchmarks","Table reasoning via RL hits 68.3% on BIRD with a 7B model","First RL approach to table reasoning beats SFT across tasks","Rule-based rewards make RL win on tableQA and text-to-SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3813,"prompt_tokens":961,"completion_tokens":2852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2752}},"tokens_in":577,"tokens_out":2852,"duration_ms":20404,"temperature":1.0,"reasoning_tokens":2752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:35:30.065878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the paper's rule-based reward against an oracle semantic-equivalence reward (human- or LLM-judged) on a dataset like WikiTQ where many gold answers have multiple valid phrasings, training identical GRPO runs otherwise. If the oracle-reward model beats the rule-based-reward model by a large margin on human-judged correctness, the rule-based reward is not a faithful correctness proxy and the central claim fails for answer-equivalence tasks.","supporting_citations":[],"review_version":1}