{"id":"e1c81120-8f25-4e13-b99f-34e195044d3d","arxiv_id":"2412.01230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GraphOTTER represents complex tables as graphs and uses LLM step-by-step actions to answer questions, outperforming prior implicit-reasoning baselines on Hitab and AIT-QA.","lead":"GraphOTTER turns complex tables into graphs of cells and has an LLM reason step-by-step on that graph, using actions like 'get shared neighbors.' On two benchmarks, it beats existing table-QA methods by an average of about 4.8 percent. The main idea is to make the reasoning process visible and to let the model focus on only the cells that matter, rather than reading the whole table at once.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AIT-QA results are reported on a reduced, non-random subset (80 tables/367 QA pairs) while the main text presents them as 'AIT-QA'; the largest gains and the +1.63% average over MIX-SC rest on this filtered set and single-run point estimates.","rationale":"The reader's weakest_assumption is the sufficiency of the undirected same-row/same-column graph representation without header annotations. That assumption is partly supported by the paper's own ablation: GraphOTTER w/ known headers (Table 4) is only about 0.7-1.2 points better than GraphOTTER on Hitab and AIT-QA, suggesting that self-inferred headers are not the main source of the gain. The more vulnerable point is evaluation validity: the AIT-QA numbers come from a filtered subset described only in Appendix B, while the main text presents the results as applying to AIT-QA without qualification. This matters because the largest gains are on AIT-QA and because the filtering criterion (header positions not reconstructable without annotation) is correlated with the complex-table property the method targets. The comparison against baselines is internally coherent because all methods are evaluated on the same filtered set, so this is not a fatal flaw, but it makes the strong claim 'outperforms on AIT-QA' conditional on a modified benchmark. The absence of significance testing compounds the issue, especially for the sub-one-point Hitab margins. Since the reader's verdict is already CONDITIONAL and explicitly cites the filtered AIT-QA test set, I retain that verdict rather than moving to a different one. A full-AIT-QA rerun is the single check that would resolve the concern.","tokens_in":101,"tokens_out":12181,"duration_ms":302298,"concrete_test":"Run GraphOTTER and MIX-SC on the full original AIT-QA test set (or at minimum on the discarded tables) using the same prompts, graph construction, and evaluator, and report per-metric differences. If the AIT-QA gain over MIX-SC in Table 2 shrinks below zero or reverses, the headline claim is an artifact of the filtering; if the advantage persists at a similar magnitude, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the validity of the AIT-QA half of the headline comparison. Appendix B states that tables whose header cells cannot be positionally reconstructed without annotations were removed, leaving 80 tables and 367 QA pairs (Table 6). The main text, however, presents Table 2 as results on 'AIT-QA' and claims an average +1.63% gain over MIX-SC there, with the largest single-cell margin being Gemini EM 81.47 vs 69.95 (+11.52). The removed tables are not a random sample: they are exactly the tables with the most ambiguous header structures, which is the phenomenon the method is designed to handle. Thus the filtered subset may not represent the benchmark the claim names. Because all baselines are rerun on the same subset, the relative comparison is internally coherent, but the external claim 'GraphOTTER outperforms baselines on AIT-QA' is not yet established. Additionally, no variance or significance testing is reported, and several Hitab margins are under one accuracy point (e.g., Qwen2 LLM Eval 77.37 vs 77.08, EM 73.74 vs 73.42), so the asserted consistency across settings is partly a point-estimate artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GraphOTTER, a method for complex table question answering that converts a table into an undirected graph whose nodes are cells and whose edges encode same-row and same-column relations, then prompts an LLM to reason step-by-step using four actions (VisitNode, GetAllNeighbours, GetSharedNeighbours, AnswerQuestion). A reasoning trace is initialized with a small set of retrieved nodes and dynamically updated as the LLM performs actions. The authors evaluate GraphOTTER on HiTab and AIT-QA using Gemini-1.5-flash and Qwen2-72B-Instruct, compare it against one-shot implicit, iterative implicit, and iterative explicit baselines, and include ablations on graph-vs-implicit reasoning, known-vs-self-inferred headers, iteration efficiency, and initialization size. The central claim is that explicit graph reasoning improves complex-table QA by filtering out irrelevant information and by not requiring pre-annotated headers.","tokens_in":19595,"tokens_out":9433,"duration_ms":77955,"significance":"If the reported results hold, the paper makes a useful contribution to complex-table QA: it offers a concrete way to make LLM reasoning explicit over a graph representation, it releases code and prompts, and its ablations support the view that combining graph representations with explicit reasoning helps. The method is clearly described and the action set is simple enough to be adapted by others. However, the evaluation has several load-bearing issues: the AIT-QA experiments are run on a heavily filtered subset that is not clearly labeled as such in the main text, the headline performance gains are not reproducible from the reported table, and there is no variance or significance analysis for several very small margins. These issues weaken the support for the claim that GraphOTTER consistently outperforms the best baselines across benchmarks.","major_comments":[{"comment":"The main text presents Table 2 as results on 'AIT-QA' and draws the headline conclusion that GraphOTTER outperforms all baselines there, but the experiments use only 80 tables and 367 QA pairs after removing AIT-QA tables whose header cells cannot be positionally reconstructed without annotations. This removal is not a random subsample: it preferentially drops tables with the most flexible or ambiguous header structures, which are precisely the cases GraphOTTER was designed to handle. The relative comparison is internally coherent because all methods are evaluated on the same subset, but the external claim about AIT-QA as a benchmark is not established. Please report the filtering criterion and counts in the main text, and evaluate on the full AIT-QA test set or on a justified random subset that preserves the hard cases.","section":"Section 4.2 and Appendix B, Table 6"},{"comment":"The claimed gains are not reproducible from Table 2. The abstract states an average gain of +4.77% over the best baseline; Section 4.2 states that GraphOTTER surpasses MIX-SC by an average of +9.07% on HiTab and +1.63% on AIT-QA. Computing from Table 2, GraphOTTER over MIX-SC averages +1.66% on HiTab and +6.03% on AIT-QA, and +3.43% if the best per-cell baseline is used instead. Please correct the arithmetic or state clearly what averaging protocol produces the quoted numbers; as written, the headline number is not supported by the reported table.","section":"Abstract and Section 4.2, Table 2"},{"comment":"All results are single-run point estimates with no variance, confidence intervals, or significance tests. Several claimed wins are very small (for example, on Qwen2, HiTab EM 73.74 vs 73.42 and LLM Eval 77.37 vs 77.08), so the statement that GraphOTTER 'consistently outperforms' all baselines rests on differences that may be within run-to-run noise. Please report multiple seeds, bootstrap intervals, or a significance test for at least the main comparisons, especially given that the LLM evaluator itself introduces additional variance.","section":"Table 2 and Section 4.2"},{"comment":"The top-K initialization size is selected after inspecting sensitivity curves computed on the same test sets used for the main results, and the paper then states that all experiments use K=8 'without further tuning.' Because Figure 5 shows that performance varies with K, selecting K from test-set performance is a form of test-set tuning and can inflate the reported numbers. Please perform this sensitivity analysis on a validation split, or report the selection as a tuned hyperparameter and soften the 'without further tuning' claim.","section":"Section 4.4, Figure 5"},{"comment":"The baseline comparison is not fully controlled for header information. Only TableParser is explicitly run with header cells removed to simulate the real-world scenario, while the other baselines use their official code or prompts that assume headers at the top or left of the table. Since GraphOTTER's advantage is partly that it does not need header annotations, this asymmetry may favor GraphOTTER. Please report baselines both with and without header annotations, or justify why the official header assumptions are appropriate for all tables in both datasets.","section":"Appendix A.1 and Table 2"}],"minor_comments":[{"comment":"The axes of Figure 3 are confusing: the x-axis is labeled 'LLM Eval (%)' while the y-axis appears to be 'Average cell utilization rate per question (%)', and the figure also contains oddly placed text fragments. Please redraw the figure with clear axis labels and a proper legend.","section":"Section 4.3, Figure 3"},{"comment":"Figure 4 has duplicated and truncated labels (for example, 'Gemini on AIT-QA' appears twice and 'Agerave' should be 'Average'), and the numerical annotations near the bars are difficult to read. Please clean up the figure formatting.","section":"Section 4.4, Figure 4"},{"comment":"The header-related analysis uses AIT-QA annotations, but the paper does not say this explicitly in the main text; please state that the HRQ/HUQ split is available only for AIT-QA. Also, in Table 3 the Gemini-1.5 HRQ result for TableParser (71.05) is actually higher than GraphOTTER (69.74); the text should acknowledge this exception rather than saying GraphOTTER achieves promising results on both question types without qualification.","section":"Tables 3 and 5"},{"comment":"The VisitNode and GetAllNeighbours actions can return multiple candidate nodes for duplicate cell values; the paper mentions presenting structural information to the LLM, but it would be helpful to specify how ties are resolved when the LLM query matches many nodes with identical content.","section":"Section 3.3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The arithmetic discrepancy in the headline gain (Abstract vs Table 2) should be checked at the start of the revision; it is a simple correction but it undermines confidence in the reported numbers. The AIT-QA filtering also deserves an explicit request for full-dataset results or a clearly labeled random subset before the external claim about AIT-QA is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take. The paper is a solid, clearly-written empirical contribution to complex table QA. The core idea is simple: turn the table into an undirected graph (cells as nodes, same-row/same-column edges), and have the LLM walk it with four actions (visit, get neighbors, get shared neighbors, answer), maintaining a reasoning trace. That is genuinely new relative to Chain-of-Table's tabular operations and to GNN-based graph encoders, and it's a clean application of recent LLM graph reasoning. The paper also does the right things: it releases code and data, includes full prompts, runs ablations that separate the graph representation from the explicit reasoning, and shows self-inferred headers are nearly as good as ground truth. The limitations section is candid about cost and prompt sensitivity.\n\nThe main soft spot is the AIT-QA evaluation. Appendix B says tables whose header cells can't be positionally reconstructed without annotations were removed, leaving 80 tables and 367 QA pairs. The main text presents Table 2 as 'AIT-QA' without flagging that subset. The removed tables are exactly the ambiguous-header cases GraphOTTER is designed for, so the subset likely favors the method. The comparison is internally fair—all baselines were rerun on the same subset—but the claim about beating baselines on AIT-QA as a benchmark is not yet established. I'd want the authors to either report full AIT-QA results, or clearly label the subset and justify its representativeness. This is fixable, but it is load-bearing for the headline numbers.\n\nTwo smaller issues. The authors modified TableParser by removing header annotations to match the no-header setting; that's defensible but handicaps a baseline built for headers. And there is no variance or significance testing; several Hitab margins with Qwen2 are below a point, which is noise territory. The AIT-QA gains are large, but the Hitab point estimates alone wouldn't convince me.\n\nThe central claim—explicit graph reasoning helps complex table QA—holds up in the within-paper ablations, so I'd send this to a serious referee. The revision should fix the AIT-QA framing, report variance or repeated runs, and clarify the baseline modifications. My verdict is conditional, not reject.","headline":"A solid, clearly-written empirical paper on LLM graph reasoning for complex table QA; the AIT-QA filtering is a real but fixable wrinkle.","tokens_in":20206,"tokens_out":4692,"would_cite":true,"duration_ms":39507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphOTTER claims that complex table question answering is best done by converting the table into an undirected graph and having a language model reason step by step with graph actions, and reports consistent gains over implicit…","keywords":["complex table question answering","graph reasoning","large language models","reasoning trace","table representation","self-inferred headers","HiTab","AIT-QA"],"falsifier":"Run GraphOTTER on a held-out set of complex tables whose hierarchical headers are not aligned with simple row/column adjacency (for example, headers that apply diagonally or span only part of a row/column), keeping the same prompt and action design; if accuracy falls to or below the best implicit baseline on that set, the claim that the graph representation generalizes beyond header positions would be falsified.","tokens_in":19162,"feed_emoji":"📊","tokens_out":4352,"duration_ms":33669,"temperature":0.7,"pith_summary":"GraphOTTER is a method for answering questions about complex tables that converts the table into an undirected graph—cells are nodes, same-row/same-column cells are edges—and then has a language model reason step by step over that graph using a small set of actions (visit a node, get neighbours, get shared neighbours, answer). The paper's central claim is that this explicit, traceable reasoning is more accurate than feeding the whole table into a prompt, because it lets the model ignore irrelevant cells. Across the HiTab and AIT-QA benchmarks and two LLM backbones, the method reports an average gain of +4.77% over the best baseline. The paper argues the gain comes from filtering out irrelevant information, and shows the model uses far fewer table cells when producing answers.","feed_headline":"Graph reasoning beats whole-table prompting on complex tables","feed_subtitle":"Step-by-step reasoning over a cell graph filters irrelevant data and lifts accuracy by +4.77% over the strongest baseline.","key_machinery":"The central object is an undirected graph G=(N,E,V) built from the table, where each node is a triple (row index, column index, cell value), merged cells have sets of indices, and edges connect cells in the same row or column with a label saying which. Reasoning is driven by a Thought-Action-Update loop: the LLM maintains a reasoning trace (the visited nodes and their connections), selects one of four actions—VisitNode, GetAllNeighbours, GetSharedNeighbours, AnswerQuestion—executes it against the graph, and updates the trace. This machinery lets the model expand only the relevant neighbourhood of cells and decide when it has enough information to answer.","core_discovery":"On its own terms, the paper's discovery is that explicit graph-based reasoning is a better inductive bias for complex table QA than implicit whole-table prompting. GraphOTTER treats header and data cells uniformly as nodes, so it does not need pre-annotated headers or assumptions about header location. The step-by-step actions and the reasoning trace—the growing subgraph of visited nodes—turn a complex table into a navigable structure, and the final answer is generated from only the local state, not the entire graph. The paper shows this outperforms one-shot and iterative implicit baselines as well as the iterative explicit Chain-of-Table method on both datasets and both backbones.","pith_inferences":["The graph representation with only same-row/same-column edges may not capture hierarchical header semantics in all complex tables; a directed or hypergraph representation might be needed for tables where headers span multiple columns in non-rectangular ways.","The filtered AIT-QA test set (tables where header positions cannot be reconstructed) means the reported gains may not hold on the full AIT-QA distribution; this is a caveat the reader should weigh.","A natural extension is to combine one-shot and iterative reasoning to reduce LLM calls, as the paper itself suggests, and to add n-hop neighbour actions for longer-range reasoning."],"forward_implications":["If the central claim is correct, explicit graph-based reasoning becomes a strong default for complex table QA, especially where headers are not at the top or left.","The reasoning trace provides an inspectable explanation of which cells led to the answer, enabling human verification of the model's reasoning path.","The method's few iterations compared to ReAct-style baselines suggest it is cost-effective for iterative reasoning, despite being iterative itself.","The self-inferred header mechanism implies the method transfers to new table layouts without annotation, which is a practical advantage over header-dependent baselines.","The sensitivity to trace initialization (4–8 cells) indicates that a hybrid retrieval/model-selection step is important for performance."],"supporting_citations":[{"why":"Supplies the HiTab dataset, one of the two complex-table QA benchmarks used for evaluation.","marker":"(Cheng et al., 2022)"},{"why":"Supplies the AIT-QA dataset, the airline-industry benchmark with header-related and header-unrelated questions.","marker":"(Katsis et al., 2021)"},{"why":"MIX-SC is the strongest baseline that GraphOTTER claims to outperform by an average of +4.77%.","marker":"(Liu et al., 2024)"},{"why":"Chain-of-Table provides the iterative explicit reasoning baseline that GraphOTTER distinguishes itself from.","marker":"(Wang et al., 2024)"},{"why":"ReAct is the iterative implicit baseline that shares the Thought-Action style and is a reference for reasoning on tables.","marker":"(Yao et al., 2023)"},{"why":"TableParser is a one-shot implicit baseline and also provides the tuple-based representation and LLM-evaluator methodology.","marker":"(Zhao et al., 2023)"},{"why":"TableReasoner is a one-shot implicit baseline using Markdown table representations.","marker":"(Chen, 2022)"},{"why":"E5 is an iterative implicit baseline that uses a code interpreter and self-described table structure.","marker":"(Zhang et al., 2024b)"}],"fun_headline_variants":["GraphOTTER: graph reasoning outperforms whole-table prompts","Cell-graph reasoning filters noise, boosts table QA by 4.77%","Step-by-step graph reasoning beats implicit table prompting","Graph-based reasoning trims irrelevant data for sharper answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The undirected same-row/same-column graph is assumed to preserve enough of the table's semantic structure—including merged cells and flexible header locations—for the LLM to reason accurately without explicit header annotations; the paper does not validate this independently of its aggregate accuracy results, which themselves rely on a filtered AIT-QA test set.","fun_headline_variants_meta":{"raw":{"variants":["GraphOTTER: graph reasoning outperforms whole-table prompts","Cell-graph reasoning filters noise, boosts table QA by 4.77%","Step-by-step graph reasoning beats implicit table prompting","Graph-based reasoning trims irrelevant data for sharper answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1210,"prompt_tokens":879,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":495,"tokens_out":331,"duration_ms":3951,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:32:42.583939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GraphOTTER on a held-out set of complex tables whose hierarchical headers are not aligned with simple row/column adjacency (for example, headers that apply diagonally or span only part of a row/column), keeping the same prompt and action design; if accuracy falls to or below the best implicit baseline on that set, the claim that the graph representation generalizes beyond header positions would be falsified.","supporting_citations":[],"review_version":1}