Pith. sign in

REVIEW 3 major objections 9 minor 40 references

Team Anotheroption at SemEval-2025 Task 8: Bridging the Gap Between Open-Source and Proprietary LLMs in Table QA

T0 review · 3 major / 9 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open-source pipeline with an LLM orchestrator reaches 88% on table QA dev set, beating GPT-4o's 74%, and 80% official.

desk verdict Solid SemEval system report with an externally validated official score, but the headline 'surpassing GPT-4o' claim rests on an uncontrolled comparison and should be toned down. read the letter →

arxiv 2506.09657 v2 pith:2Z3TQ7FF submitted 2025-06-11 cs.CL

Nikolas Evkarpidi , Elena Tutubalina This is my paper
classification cs.CL
keywords tablequestionansweringlargelanguagemodelstext-to-SQLtext-to-codegenerationretrieval-augmentedLLMorchestratoropen-sourceLLMsDataBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a fully open-source system for answering questions over tabular data in SemEval-2025 Task 8. It combines several independent answer generators - SQL queries, Pandas code, and a direct end-to-end model - along with retrieval and a final orchestrator LLM that chooses the most plausible successful answer. On the authors' 100-question development set this setup reaches 88% accuracy, above the 74% they measured for GPT-4o in the same pipeline, and on the official test set it scores 80%, ranking 13th among 38 open-source teams. The authors take this as evidence that orchestrated open-source LLMs can rival proprietary models on structured-data QA.

What carries the argument

The load-bearing component is the orchestrator, a Llama-3.3-70B instruction-tuned model that receives every successfully executed candidate answer - including the generated code and a truncated result - and must return the number of the most plausible one. It supports the ensemble by resolving conflicts between divergent outputs, rejecting mismatched formats, and confirming consensus, while all other components (retrieval, column selection, code generation, self-correction) feed it candidates.

What would settle it

Run the full pipeline on the 400 held-out DataBench QA pairs not used for development: if accuracy falls well below 80% or below a comparably run GPT-4o pipeline, the claim that orchestrated open-source models rival proprietary models would be weakened.

Watch

Extended reading notes

Core claim

The central claim is that an ensemble of open-source LLMs, coordinated by a selection oracle, can match proprietary models on table QA. The system generates candidate answers via text-to-SQL and Pandas code, augments prompts with retrieved rows and predicted relevant columns, includes an end-to-end model for questions that need semantic reading, and then asks a Llama-3.3-70B model to pick the best successful candidate. On the development set the full pipeline reaches 88%, exceeding the 74% achieved by GPT-4o running in the same pipeline, and the official test set score is 80%. The paper argues this narrows the practical gap between open-source and proprietary solutions in this task.

Load-bearing premise

The load-bearing premise is that the first 100 QA pairs used as the development set are representative of the full DataBench benchmark; if they are not, the reported 88% and the qualitative comparisons against GPT-4o may not hold on the official test distribution.

Editorial extensions

If this is right

  • If the claim holds, competitive table QA is achievable without proprietary APIs by combining several open-source generators with an orchestrator.
  • The ablation results indicate that combining multiple code generators and an end-to-end model improves over any single model, while question reformulation hurts accuracy.
  • The orchestrator's success depends on at least one correct candidate; when all candidates are wrong, the system cannot recover, as the paper acknowledges.
  • The official score of 80% versus the 26% stable-code baseline shows a large improvement, though the gap to the top open-source team's 95% remains sizeable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 8-point drop from dev (88%) to test (80%) suggests the first-100 dev subset is not fully representative, so readers should treat the dev-set comparisons as diagnostic rather than predictive.
  • Beyond the paper: the pattern of independent solvers plus a strong selector should transfer to other structured-data QA domains, including database question answering over enterprise tables.
  • Beyond the paper: replacing the orchestrator with a learned router that predicts whether a question needs SQL, Python, or direct semantic answering is the natural next step, especially because the orchestrator actively adjudicated almost 37% of dev questions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. The paper describes a table QA system developed for SemEval 2025 Task 8 (DataBench). The pipeline combines text-to-SQL and text-to-Pandas code generation, retrieval-augmented generation, an end-to-end answering module, a self-correction mechanism, and a Llama-3.3 orchestrator that selects among candidate answers. On the official test set the system reports 80.08% accuracy, ranking 13th among 38 open-source teams and 20th among 53 teams overall. On a 100-question development set, the full open-source pipeline reaches 88% accuracy, and the paper claims this surpasses GPT-4o's 74%. Ablations examine question reformulation, column selection, column renaming, and the orchestrator's decision behavior, and the paper includes an error analysis of code-based failures.

Significance. If the claims were fully supported, this would be a useful system-description contribution to table QA: it shows an orchestrated open-source pipeline can be competitive in a shared task, it provides a detailed component analysis, and it makes code available. The official competition result is externally validated, and the publicly verifiable test score is a genuine strength. However, the headline comparison against GPT-4o is not a controlled experiment, and the ablation conclusions rest on only 100 development questions without error bars or significance testing. These issues do not invalidate the competition result, but they materially weaken the paper's central 'bridging the gap' claim and require revision before the paper can be accepted.

major comments (3)
  1. [Abstract; Sec. 6] The headline claim that open-source models surpass GPT-4o (88% vs. 74%) is not supported by a controlled comparison. The 88% is the full open-source pipeline (two code generators, E2E, RAG, column selection, and a Llama-3.3 orchestrator), whereas the 74% appears to be GPT-4o without the pipeline. The paper itself reports in Sec. 6 that 'Our pipeline applied to GPT4o (w/o orchestrator) also performs well (87%)', reducing the gap to one point. The missing condition—GPT-4o with the full pipeline including the orchestrator—is exactly what is needed to test the superiority claim. Without it, the open-source advantage is unestablished; at most, the results support 'comparable' performance under an uncontrolled comparison. Since this comparison appears in the abstract, introduction, and conclusion, it is load-bearing and requires either a matched experiment or a reformulated claim.
  2. [Sec. 4; Sec. 5 (Fig. 2)] All ablations and qualitative conclusions are based on a development set of only the first 100 QA pairs, with no confidence intervals or significance testing. With n=100, differences of a few percentage points (e.g., the 88% vs. 87% full-pipeline comparison, or the reformulation contrasts such as 68% vs. 84%) are within sampling noise. The official test set provides only the final system's accuracy, so the internal claims about which components help or hurt are not externally validated. The authors should report interval estimates, validate the main ablation conclusions on a larger sample from the benchmark, or explicitly restrict the conclusions to the observed dev subset.
  3. [Sec. 5.1; Sec. 8.3] The paper makes contradictory statements about self-correction: Sec. 5.1 says it 'improved accuracy by refining failed queries', while Sec. 8.3 says it was 'largely ineffective' because at least one Python and one SQL agent usually produce a runnable solution. These statements need reconciliation with an ablation that isolates self-correction. Without such an experiment, the contribution of this component is unclear, and the reader cannot tell whether the self-correction mechanism should be retained in the described pipeline.
minor comments (9)
  1. [Throughout] There are repeated typos, including 'Codetral' for 'Codestral' (e.g., Sec. 5 and Fig. 2) and 'accuarcy' in Sec. 6; a proofreading pass is needed.
  2. [Sec. 3; Sec. 8.1] Cross-reference errors: Sec. 3 says 'as shown in Fig. 3' but the system overview is Fig. 1, while the orchestrator decision distribution is Fig. 3; all figure references should be checked.
  3. [Sec. 4; Table 4] Table 4 reports 'Unique datasets 49' but Sec. 4 says the dataset comprises 65 tables; since 65 × 20 = 1300 QA pairs, the table entry appears to be a typo and should be corrected.
  4. [Abstract; Sec. 11] The abstract and conclusion mention a GitHub repository but no URL is provided in the manuscript; include the repository link for reproducibility.
  5. [Sec. 7] The abstract says 'top-13 ranking among the 38 participating teams' but Sec. 7 makes clear this is the open-source-only section; in the global ranking the system is 20th of 53 teams, so the abstract should specify 'open-source' to avoid ambiguity.
  6. [Sec. 7] The paper says the system achieved 80% on both the DataBench evaluation and on the lite part, but no separate lite results are reported; specify the score or remove the claim.
  7. [Sec. 3.4] The retrieval component does not specify the embedding model or the exact retrieval implementation, which is needed for reproducibility.
  8. [Sec. 8.1] The orchestrator decision analysis is based on 41 questions, but the paper does not explain how these 41 questions were selected from the 100-question development set.
  9. [Fig. 10] In the SQL generation prompt, Example 2 uses square brackets around 'salary' while the stated rule says to use double quotes; the example should match the rule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracies are external benchmark measurements, and the only noted weakness is a non-circular baseline-comparison limitation.

full rationale

The paper contains no circular derivation. Its central results are external-benchmark measurements: the 88% development-set accuracy in Section 5 and Figure 2 and the official 80.08% test score in Section 7 (Table 2) are obtained by running a fixed pipeline on DataBench QA pairs, not by fitting a parameter to the target and then reading it back out. The development set (the first 100 QA pairs) is used for ablation and prompt decisions, while the official test set is held out by the competition, so the reported test result is an independent evaluation. The GPT-4o comparison in Section 6 is a baseline comparison, not a derived quantity; the paper even discloses that GPT-4o with the same pipeline minus the orchestrator reaches 87%, which weakens the superiority claim but does not make it circular. The cited prior work by the authors (Somov et al., 2024; Somov and Tutubalina, 2025) appears only in related work and carries no load-bearing justification. The Limitations section acknowledges that orchestration can fail when all candidates are wrong, which is an honest boundary condition rather than a circular step. No equation or fitted constant is redefined as a prediction, and no uniqueness theorem or ansatz is imported from self-citations.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new postulates or entities. It relies on standard LLM capabilities and a small dev set for tuning. The main design choice that functions as a free parameter is the retrieval row count, which is not ablated.

free parameters (1)
  • Number of rows retrieved for RAG (top-3) = 3
    Chosen by hand without ablation; the paper does not report experiments varying this number.
assumptions (4)
  • domain assumption The 100-question dev set is representative of the DataBench test distribution.
    All ablations and qualitative conclusions (e.g., reformulation is counterproductive, open-source surpasses GPT-4o) are drawn from this small subset, and the paper assumes it generalizes.
  • domain assumption The official SemEval evaluation framework correctly computes exact-match accuracy.
    The paper relies on the competition's evaluation rules and reported scores without independent verification.
  • domain assumption Chain-of-thought prompting and structured checklists improve LLM code generation and orchestration.
    The paper invokes prior work (Wei et al., 2022; Cook et al., 2024) and applies these techniques without a dedicated controlled ablation for each component.
  • domain assumption The selected open-source models (Llama, Qwen, Codestral, MiniMax) are reliable enough for the pipeline.
    Model choices are based on benchmark performance and open-source availability, but the paper does not compare against alternative open-source models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Team Anotheroption at SemEval-2025 Task 8: Bridging the Gap Between Open-Source and Proprietary LLMs in Table QA." pith.science (2026). https://pith.science/paper/2Z3TQ7FF

@misc{pith2026250609657,
  author       = {Pith},
  title        = {Pith review of: Team Anotheroption at SemEval-2025 Task 8: Bridging the Gap Between Open-Source and Proprietary LLMs in Table QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Z3TQ7FF}},
  note         = {Machine review of arXiv:2506.09657}
}
read the original abstract

This paper presents a system developed for SemEval 2025 Task 8: Question Answering (QA) over tabular data. Our approach integrates several key components: text-to-SQL and text-to-code generation modules, a self-correction mechanism, and a retrieval-augmented generation (RAG). Additionally, it includes an end-to-end (E2E) module, all orchestrated by a large language model (LLM). Through ablation studies, we analyzed the effects of different parts of our pipeline and identified the challenges that are still present in this field. During the evaluation phase of the competition, our solution achieved an accuracy of 80%, resulting in a top-13 ranking among the 38 participating teams. Our pipeline demonstrates a significant improvement in accuracy for open-source models and achieves a performance comparable to proprietary LLMs in QA tasks over tables. The code is available at GitHub repository.

Figures

Figures reproduced from arXiv: 2506.09657 by the authors.

Figure 1
Figure 1. Overview of our system, featuring two so [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System performance on the dev set. Llama3.3-70b-instruct is used as an orchestrator. As a full pipeline, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of LLM orchestrator decision [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: GPT4o comparison with and w/o our pipeline. Llama3.3-70b-instruct used as orchestrator. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Structured data questions and formats [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Prompt for python generation (dialogue) [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Prompt for python generation (without dialogue) [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Prompt for self-correction [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Prompt for orchestrator [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Prompt for SQL-generation [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Prompt for E2E model [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 37 canonical work pages

  1. [1]

    You are presented with a dataframe and a question

    You are two of the most esteemed Pandas DataScientists engaged in a heated and truth-seeking debate. You are presented with a dataframe and a question. Begin dialogue by rigorously discussing your reasoning step by step, ensuring to address all aspects of the checklist. In your discourse, meticulously articulate the variable type necessary to derive the a...

    work page

  2. [2]

    Refrain from importing any additional libraries beyond pandas and numpy

    work page

  3. [3]

    In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 431–438, Mexico City, Mexico

    AIRI NLP team at EHRSQL 2024 shared task: T5 and logistic regression to the rescue. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 431–438, Mexico City, Mexico. Association for Computational Linguistics. Oleg Somov and Elena Tutubalina. 2025. Confidence estimation for error detection in text-to-sql systems. Proceedings of t...

    arXiv 2024

  4. [4]

    If the question requires multiple entries, always utilize .tolist() to present the results

    work page

  5. [5]

    You MUST FOLLOW THE CHECKLIST, ANSWER EACH OF ITS QUESTIONS (REASONING STEP), AND ONLY THEN OUTPUT THE FINAL ANSWER BASED ON THOSE ANSWERS:

    If the question seeks a single entry, ensure that only one value is output , even if multiple entries meet the criteria. You MUST FOLLOW THE CHECKLIST, ANSWER EACH OF ITS QUESTIONS (REASONING STEP), AND ONLY THEN OUTPUT THE FINAL ANSWER BASED ON THOSE ANSWERS:

    work page

  6. [6]

    CODE: ‘‘‘SELECT CASE WHEN EXISTS(SELECT 1 FROM temp_table WHERE "age" > 30) THEN ’True’ ELSE ’False’ END;‘‘‘ Example 2: Task: Count the number of entries with a salary above 50000

    Verify that the query outputs ’True’ or ’False’ when presented with a yes or no question. CODE: ‘‘‘SELECT CASE WHEN EXISTS(SELECT 1 FROM temp_table WHERE "age" > 30) THEN ’True’ ELSE ’False’ END;‘‘‘ Example 2: Task: Count the number of entries with a salary above 50000. REASONING:

    work page

  7. [8]

    The dataframe, df, is already populated with data for your analysis; do not initialize it, but focus solely on manipulating df to arrive at the answer

    work page

  8. [11]

    How many values should be in the output?

    work page

Show all 40 references
  1. [12]

    Values (or one value) from which column (only one!) should the answer consist of?

  2. [13]

    What should be the type of value in the answer? Example of a task: Question: Identify the top 3 departments with the most employees. <Columns> = [’department’, ’employee_id’] <First_row> = (’department’: ’HR’, ’employee_id’: 101) Reasoning: Count the number of employees in eac...

  3. [14]

    The output should consist of 3 values

  4. [15]

    The values should come from the ’department’ column

  5. [16]

    The type of value in the answer should be a list of strings. Code: result = df[’department’].value_counts().nlargest(3).index.tolist() Your data to process: <question> = {question} - Make absolute sure that all columns used in query are present in the table. <columns_in_the_ta...

  6. [17]

    You are given a dataframe and a question

    You are a best in the field Pandas DataScientist. You are given a dataframe and a question. You should spell out your reasoning step by step and only then provide code to answer the question. In the reasoning state it is essentianl to spell out the answers’ variable type that ...

  7. [18]

    Avoid importing any additional libraries than pandas and numpy

  8. [19]

    All data is already loaded into df dataframe for you, you MUST NOT initialise it, rather present only manipulations on df to calculate the answer

  9. [20]

    If the question ask for several entries alsways use .tolist()

  10. [21]

    The following solutions failed for the task: \

    If the question ask for one entry, make sure to output only one, even if multiple qualify. <...> (same as previous prompt) Figure 7: Prompt for python generation (without dialogue) prompt for self-correction "The following solutions failed for the task: \"{question}\"\n\n" + ’...

  11. [22]

    Do we have respondents who have shifted their voting preference?

    If the question is "Do we have respondents who have shifted their voting preference?" the answer type is **Boolean** because the response should be True/False

  12. [23]

    How many respondents participated in the survey?

    If the question is "How many respondents participated in the survey?" the answer type is **Integer**

  13. [24]

    List the respondents who preferred candidate X?

    If the question is "List the respondents who preferred candidate X?" the answer type is **List** because the response requires a collection of values

  14. [25]

    What is the average age of respondents?

    If the question is "What is the average age of respondents?" the answer type is **Number** because the response should be a decimal value

  15. [26]

    What is the name of the candidate with the highest votes?

    If the question is "What is the name of the candidate with the highest votes?" the answer type is **String** because the response is a single textual value. Given the following solutions and their results for the task: "{question}" {’ ’.join([f’Solution Number {i+1}: Code: {r[...

  16. [27]

    Identify the column of interest, which is ’age’

  17. [28]

    Determine the condition to check, which is ’age > 30’

  18. [29]

    Use the SELECT statement to retrieve a boolean result indicating the presence of such entries

  19. [30]

    Apply the WHERE clause to filter rows based on the condition ’age > 30’

  20. [31]

    Use the EXISTS clause to ensure the query outputs ’True’ if any row matches the condition, otherwise ’False’

  21. [33]

    Identify the column of interest, which is ’salary’

  22. [34]

    Determine the condition to filter the data, which is ’salary > 50000’

  23. [35]

    Use the SELECT COUNT(*) statement to count the number of rows that meet the condition

  24. [36]

    Apply the WHERE clause to filter rows based on the condition ’salary > 50000’

  25. [37]

    {col}: {str(type(df[col].iloc[0])) }

    Ensure the table name is ’temp_table’ and the column name is enclosed in double quotes to handle any spaces or special characters. CODE: ‘‘‘SELECT COUNT(*) FROM temp_table WHERE [salary] > 50000;‘‘‘ Write a correct fault-proof SQL SELECT query that solves this precise task. Ru...

  26. [38]

    Boolean (True/False)

  27. [39]

    List (e.g., [’Tree’, ’Stone’])

  28. [40]

    Final Answer:

    String (e.g., ’Spanish’) Give extensive reasoning and then fianlly provide the answer starting with string "Final Answer:" in one of the four formats presented above ( Boolean, List, Number, String). Your response should then be finished. Figure 11: Prompt for E2E model

  29. [2017]

    Life Sciences

    Seq2sql: Generating structured queries from natural language using reinforcement learning. Preprint, arXiv:1709.00103. A Appendix A.1 Checklists and Dialogue-Inducing Prompts During testing, models often skipped crucial in- structions, leading to incorrect code generation. To ...

    2024 arXiv

  30. [2023]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou

    Plan-and-solve prompting: Improving zero- shot chain-of-thought reasoning by large language models.Preprint, arXiv:2305.04091. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elici...

    2022 arXiv

  31. [2024]

    InProceedings of the 2024 Joint In- ternational Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 13471–13488, Torino, Italia

    Question answering over tabular data with DataBench: A large-scale empirical evaluation of LLMs. InProceedings of the 2024 Joint In- ternational Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 13471–13488, Torino, Italia....

    2024

  32. [2025]

    InProceedings of the 19th Interna- tional Workshop on Semantic Evaluation (SemEval- 2025), Vienna, Austria

    SemEval-2025 task 8: Question answering over tabular data. InProceedings of the 19th Interna- tional Workshop on Semantic Evaluation (SemEval- 2025), Vienna, Austria. Association for Computa- tional Linguistics. Zhongyi Qiu, Kangyi Qiu, Hanjia Lyu, Wei Xiong, and Jiebo Luo. 20...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.