{"id":"bf63c644-d271-4e9e-9243-ad79ef16584c","arxiv_id":"2411.18564","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A DSPy-orchestrated LLM plus Answer Set Programming pipeline reports 82% average accuracy on StepGame and 69% on SparQA, well above direct prompting baselines.","lead":"This paper combines large language models with answer set programming in a DSPy-based pipeline that turns natural language spatial descriptions into logical facts, lets a solver reason over them, and iteratively repairs failed programs. On StepGame and SparQA, the pipeline reports large accuracy gains over direct prompting, though no code, error bars, or statistical tests are provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSPy optimization data is never specified; if it overlaps the evaluation sets, the reported gains are test-set overfitting, not reasoning improvements.","rationale":"The reader's chosen weakest assumption (label noise) affects the magnitude of the reported accuracy but not the core comparison, because baselines face the same noisy labels. A more damaging threat is uncontrolled optimization data: DSPy compiles by tuning prompts on examples, and the paper never states that the evaluation samples were held out. This is precisely the kind of leakage that can manufacture large apparent gains. Internal contradictions in the reported numbers further suggest the results have not been carefully audited. The appropriate verdict remains CONDITIONAL because the issue is addressable by releasing code and a clear data split; if the proposed check shows overlap, the paper would need to be rejected. Since the reader already required artifacts and the same conditional outcome, I keep the verdict unchanged.","tokens_in":12298,"tokens_out":7448,"duration_ms":65321,"concrete_test":"Obtain the DSPy compilation script and the list of instances used for prompt optimization. Verify that the optimization set is disjoint from the 3,000 StepGame (300 per hop times 10) and 220 SparQA evaluation instances. Then re-run the full pipeline with a strictly disjoint optimization set (e.g., 50 examples per dataset) and recompute Table 1 and Table 2. If the DSPy pipeline's accuracy drops substantially relative to baselines, the reported gains are due to evaluation-set contamination rather than the neural-symbolic architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the DSPy-based LLM+ASP pipeline 'significantly outperforms' direct prompting (82% on StepGame, 69% on SparQA). This claim requires that the DSPy optimizer, which 'iteratively generates and refines prompts' (Section 3.2), does not tune on the same instances used for evaluation. The evaluation sections (Section 4.1.1 and 4.2.1) describe only the construction of the evaluation samples (300 per StepGame hop; 220 SparQA examples) and never specify a disjoint training/validation set for DSPy compilation. If the optimizer's few-shot prompts or instruction weights were learned from these same test instances, the pipeline's advantage over the non-optimized Direct baseline would be inflated. The paper also contains numerical inconsistencies that compound the uncertainty: Table 1 overall averages give 81.4% rather than the abstract's 82%; Section 5 cites 'StepGame (92%) and SparQA (65%)' matching no table; and Section 4.3 feedback-loop accuracies (70.5%, 71.8%, 70.9%) differ from Table 2 overalls (67.2, 69.4, 70.3). The 10% label-noise issue (Section 4.1.3) is real but secondary: label noise affects all methods roughly equally, so the qualitative finding of large gains would likely survive even if the absolute numbers shift.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a neural-symbolic pipeline for spatial reasoning in LLMs. The pipeline uses DSPy to structure LLM calls that convert natural-language context and questions into Answer Set Programming (ASP) facts and queries, iteratively refines the ASP program using solver error feedback, and then evaluates the solver output against a synonym dictionary. The authors compare three strategies—direct prompting, Facts+Rules prompting, and the DSPy-based LLM+ASP pipeline—on StepGame and SparQA, using DeepSeek, Llama3, and GPT-4.0 mini. They report that the LLM+ASP pipeline substantially outperforms direct prompting on StepGame, with smaller gains on SparQA, and they include an ablation of the iterative feedback loop. The central claim is that separating semantic parsing from symbolic reasoning and adding iterative solver feedback yields large, generalizable improvements over direct prompting.","tokens_in":12576,"tokens_out":4907,"duration_ms":40796,"significance":"If the reported gains are robust, the paper would make a useful empirical contribution to neural-symbolic integration for spatial reasoning. The strengths are the comparison across three different LLMs and two quite different benchmarks, the inclusion of a lightweight Facts+Rules intermediate method, and the explicit analysis of feedback-loop effects on program executability. However, the paper currently does not ship code, prompts, or a clear description of the data used for DSPy optimization, and the quantitative claims are internally inconsistent across the abstract, introduction, tables, and conclusion. The significance is therefore conditional on resolving these verification and reproducibility issues; the qualitative direction of the results is plausible, but the specific numbers and the generalizability claim are not yet established.","major_comments":[{"comment":"The paper never specifies which data were used by DSPy's optimizer to compile and refine prompts, although §3.2 says that the optimizer 'iteratively generates and refines prompts' and 'optimizes LLM prompts and weights.' The evaluation sections describe only the construction of the test samples (300 per StepGame hop; 220 hand-selected SparQA examples). If the DSPy compilation used the same instances that are later scored, the reported gains over the non-optimized Direct baseline would be inflated by test-set overfitting. Please specify the disjoint training/validation split used for DSPy optimization, and if the optimizer was run on the evaluation instances, report results with prompts frozen on truly held-out data.","section":"§3.2, §4.1.1, §4.2.1"},{"comment":"The central quantitative claims are internally inconsistent. The DSPy overall accuracies in Table 1 are 87.7%, 75.6%, and 80.8%, whose mean is 81.4%, not the abstract's 82%. Section 5 cites 'StepGame (92%) and SparQA (65%)', which appear in no table. Section 4.3 reports final feedback-loop accuracies of 70.5%, 71.8%, and 70.9% for DeepSeek, Llama3, and GPT-4.0 mini, which differ from the Table 2 overall accuracies of 67.2%, 69.4%, and 70.3%. The introduction's claim of '40% on StepGame and 20% on SparQA' also conflicts with the abstract and conclusion's '40-50% and 8-15%.' These discrepancies need to be reconciled before the quantitative claims can be taken at face value.","section":"Abstract, §1, §5, Table 1, Table 2, §4.3"},{"comment":"The paper reports that about 10% of StepGame instances have labeling errors and that these ambiguities accumulate with reasoning depth, yet all reported StepGame accuracies in Table 1 are computed against the original labels without correction or sensitivity analysis. Because the abstract's headline results are absolute accuracy values (82% on StepGame) and improvements of 40-50% over direct prompting, a label-error rate of about 10% could materially shift the absolute numbers. Please evaluate on a cleaned subset, quantify the sensitivity of each method to label noise, or clearly caveat the absolute accuracy figures.","section":"§4.1.3, §3.1"},{"comment":"The SparQA evaluation is based on a hand-selected subset of 220 examples (55 per question type), and the relation rules are described as 'manually designed and updated' for this dataset, while the StepGame knowledge module is adapted from Yang et al. (2023b). With no error bars or significance tests, the reported 8-15% improvement over direct prompting on SparQA could lie within sampling noise, and the manual per-dataset rule engineering weakens the paper's generalizability claim. Please report confidence intervals or significance tests, justify the representativeness of the 220-example selection, and document the degree to which the gains depend on the manually engineered rules.","section":"§4.2.1, §4.2.2"},{"comment":"The manuscript refers to an appendix for 'code and samples' and repeatedly relies on prompt-engineering details, but no code, prompts, or sample outputs are provided in the text and no repository link is given. Since the method's contribution is a specific DSPy pipeline with iterative feedback and error handling, the absence of these artifacts prevents verification and replication. Please include the full prompts, the DSPy program, the ASP rule sets, and the evaluation scripts, or provide a persistent public repository.","section":"§3.2, §4.2.1"}],"minor_comments":[{"comment":"The figure numbering is inconsistent: the caption says 'Figure 1: LLM +ASP Pipeline' while the text says 'The framework of this pipeline is shown in Fig.2.' Please unify the figure references and ensure the figure is actually included.","section":"§3.2, Figure 1"},{"comment":"The description of the direct prompting baseline contains a duplicated sentence: 'As the most straightforward approach, it involves presenting the task to the LLM without additional guidance. ... This straightforward approach involves presenting the task to the LLM without additional guidance.' Please remove the repetition.","section":"§3.3"},{"comment":"The introduction cites '(Yu et al, 2021)' but this reference is missing from the reference list; please add the full citation or remove the citation.","section":"§1, References"},{"comment":"There are numerous typos and grammatical errors, including 'collpase' (§2.2), 'satifiable' (§4.2.3), 'overr' (§4.2.2), 'I If' (§4.2.2), 'perforamce' and 'pipleine' (§5), and 'liketree of thoughts' (§2.1). A thorough proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a system paper, not a methods breakthrough. The recipe—LLM translates natural language to ASP, a solver reasons, and error feedback closes the loop—appears in LOGIC-LM and Yang et al. (2023a), including on StepGame. What's new here is the DSPy orchestration, a lighter Facts+Rules variant, and a three-model evaluation on StepGame and SparQA.\n\nThe paper does some things well. The experiments are internally coherent in direction: the DSPy pipeline beats direct prompting on both datasets for Deepseek, Llama3, and GPT-4.0 mini, and the feedback-loop ablation (Figure 2) shows large gains in executability and accuracy. The error analysis—parsing, grounding, satisfiability—is a useful practical contribution. And the observation that ~10% of StepGame labels are wrong is honest and consistent with prior work.\n\nThe soft spots are real, though. No code or prompts are shipped; the text says the appendix has them, but the appendix isn't in this version. The SparQA evaluation is 220 hand-chosen examples with no documented randomization; StepGame uses 300 per hop without a stated split. More seriously, the paper never says whether DSPy's optimizer, which 'iteratively generates and refines prompts,' was run on the same instances used for evaluation. If it was, the gains over the non-optimized Direct baseline are inflated by test-set overfitting. I'm not accusing them—they may have used a proper dev set—but the omission makes the central claim unverifiable.\n\nThe numbers also contradict each other. The abstract's 82%/69% match the averages of Tables 1 and 2, but Section 5 cites 'StepGame (92%) and SparQA (65%),' which match no table. The feedback-loop accuracies in Section 4.3 (70.5/71.8/70.9) don't match Table 2's overalls (67.2/69.4/70.3). The claimed 40-50% improvement on StepGame is actually 46-56 percentage points across the three models. And there are no error bars or significance tests anywhere.\n\nThat said, the qualitative finding is plausible: adding a symbolic solver with feedback should help structured spatial reasoning, and the effect sizes are huge. Label noise at ~10% affects all methods roughly equally, so the main result would likely survive with cleaner labels and a proper dev split. As written, the paper needs a revision that ships code and prompts, documents the DSPy optimization split, adds basic statistics, and fixes the internal contradictions. It deserves a serious referee, but the reviewer should focus on the data-split question above all.","headline":"A plausible but under-documented system result: DSPy-orchestrated LLM+ASP beats direct prompting on two spatial benchmarks, yet the evaluation lacks artifacts and a clear data split, so the exact numbers shouldn't be trusted yet.","tokens_in":13116,"tokens_out":4492,"would_cite":false,"duration_ms":37029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural-symbolic pipeline with solver feedback achieves 82% on StepGame and 69% on SparQA.","keywords":["spatial reasoning","neural-symbolic","Answer Set Programming","DSPy","iterative feedback","large language models","StepGame","SparQA"],"falsifier":"Take a random sample of StepGame instances, hand-correct the labels, and recompute accuracy against the corrected gold standard. If the LLM+ASP advantage over direct prompting largely disappears, the central claim fails.","tokens_in":12069,"feed_emoji":"🧩","tokens_out":13173,"duration_ms":88138,"temperature":0.7,"pith_summary":"This paper sets out to show that spatial reasoning in large language models can be substantially improved by pairing the model with a symbolic solver and letting the two exchange information. The authors build a DSPy-orchestrated pipeline in which an LLM translates spatial-language descriptions into Answer Set Programming facts and rules, the Clingo solver reasons over them, and solver error messages are fed back to the LLM to refine the formal program over up to three iterations. Across three LLM architectures (Deepseek, Llama3, and GPT-4.0 mini) the pipeline averages 82% accuracy on StepGame and 69% on SparQA, compared with roughly 30% and 57% for direct prompting. The paper argues that the gains come from separating semantic parsing from logical deduction, from the iterative repair loop that raises the share of executable ASP programs, and from explicit handling of parsing, grounding, and solving failures.","feed_headline":"LLM+ASP feedback loop lifts StepGame accuracy to 82%","feed_subtitle":"A modular pipeline with solver feedback beats direct prompting by 40–50 points on StepGame and 8–15 on SparQA.","key_machinery":"The load-bearing mechanism is the iterative feedback loop between the LLM and the ASP solver, embedded in a DSPy pipeline. DSPy, a Python framework that compiles declarative LLM calls into self-improving pipelines, orchestrates four stages: fact generation (LLM produces ASP facts and a query), program refinement (LLM edits the ASP over up to three rounds using solver diagnostics), symbolic reasoning (the Clingo solver computes stable models), and answer mapping (solver output is matched to candidates via a synonym dictionary). The loop matters because LLMs often generate syntactically invalid or ungroundable ASP; feeding solver errors back lets the LLM repair its own program, which is what raises executability and accuracy.","core_discovery":"The central claim is that a modular neural-symbolic pipeline, where an LLM writes ASP code and a solver's diagnostics drive iterative repair, reliably outperforms direct prompting and a lighter natural-language 'Facts+Rules' strategy on spatial reasoning benchmarks. The reported numbers: average 82% accuracy on StepGame (87.7% for Deepseek, 75.6% for Llama3, 80.8% for GPT-4.0 mini) and 69% on SparQA, with StepGame gains of roughly 40–50 percentage points over direct prompting. The authors attribute the improvement to three mechanisms: modular separation of parsing from reasoning, iterative feedback that raises the proportion of executable ASP programs, and targeted error handling for parsing, grounding, and solving failures. On SparQA the feedback loop alone lifts program execution rates from roughly 35–46% to 73–80% and accuracy from 24–42% to 70–72% across the three models.","pith_inferences":["Because the paper itself reports roughly 10% labeling errors in StepGame, some measured accuracy may reflect agreement with wrong labels; correcting those labels could shrink the reported gap to baselines. This is our inference, not the paper's claim.","The same LLM-plus-solver feedback design could transfer to other structured reasoning tasks—temporal reasoning, planning, constraint satisfaction—wherever the bottleneck is producing executable formal code from natural language.","The error taxonomy suggests model-specific tuning matters: Deepseek's syntax errors, GPT-4.0 mini's predicate-argument ordering, and Llama3's satisfiability failures each need different prompts or validation rules, so a single fixed pipeline may underperform on new models.","Since Facts+Rules nearly matches the full ASP pipeline on SparQA, the added value of formal code generation appears task-dependent; a cost-aware system might choose the lighter method for language-heavy questions and reserve ASP for multi-hop chain reasoning."],"forward_implications":["Across all three tested LLM architectures, the LLM+ASP pipeline beats both direct prompting and Facts+Rules on StepGame at every reasoning depth from 1 to 10 hops.","On SparQA, the iterative feedback loop raises ASP program execution rates from roughly 35–46% to 73–80% and accuracy from 24–42% to 70–72%, showing the loop is the main driver of the gain.","Facts+Rules, which applies the same logical rules inside natural-language prompts, reaches accuracy close to the full LLM+ASP pipeline on SparQA while avoiding solver and code-generation overhead.","The pipeline also functions as a dataset-quality checker, identifying about 10% labeling errors in StepGame instances, consistent with earlier findings.","The consistency of gains across Deepseek, Llama3, and GPT-4.0 mini suggests the mechanism transfers across LLM families, not just to one model."],"supporting_citations":[{"why":"provides the StepGame benchmark, whose 10-hop spatial layouts define the main testbed and the template-error issue.","marker":"Shi et al. (2022)"},{"why":"provides the SparQA benchmark, the harder dataset with quantifier questions and varied relation types.","marker":"Mirzaee and Kordjamshidi (2022)"},{"why":"introduces DSPy, the orchestration framework that compiles the LLM calls into the iterative pipeline.","marker":"Khattab et al. (2023)"},{"why":"LOGIC-LM, the iterative LLM-solver feedback framework that this pipeline adapts and extends.","marker":"Pan et al. (2023)"},{"why":"shows LLM+logic programming can reach about 90% on StepGame with GPT-3, the direct precursor and baseline of the approach.","marker":"Yang et al. (2023a)"},{"why":"supplies the coordinate-based knowledge module used in StepGame and first documents its labeling errors.","marker":"Yang et al. (2023b)"},{"why":"documents that natural-language-to-program parsing success can be as low as 17%, motivating the feedback loop.","marker":"Feng et al. (2024)"},{"why":"defines Answer Set Programming and stable-model semantics, the formal foundation of the solver side.","marker":"Brewka et al. (2011)"}],"fun_headline_variants":["DSPy pipeline boosts LLM spatial reasoning to 82% on StepGame","Iterative LLM+ASP feedback lifts spatial accuracy 40-50 pts","Modular neural-symbolic loop beats direct prompting on spatial tasks","DSPy+ASP feedback raises LLM spatial reasoning by 40-50%","82% StepGame, 69% SparQA via LLM+ASP feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies and improvements assume the benchmark labels are correct, but the paper itself finds that about 10% of StepGame instances carry labeling errors.","fun_headline_variants_meta":{"raw":{"variants":["DSPy pipeline boosts LLM spatial reasoning to 82% on StepGame","Iterative LLM+ASP feedback lifts spatial accuracy 40-50 pts","Modular neural-symbolic loop beats direct prompting on spatial tasks","DSPy+ASP feedback raises LLM spatial reasoning by 40-50%","82% StepGame, 69% SparQA via LLM+ASP feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000512,"raw_usage":{"total_tokens":2527,"prompt_tokens":1024,"completion_tokens":1503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1400}},"tokens_in":640,"tokens_out":1503,"duration_ms":10336,"temperature":1.0,"reasoning_tokens":1400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:04:29.640256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of StepGame instances, hand-correct the labels, and recompute accuracy against the corrected gold standard. If the LLM+ASP advantage over direct prompting largely disappears, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the StepGame benchmark, whose 10-hop spatial layouts define the main testbed and the template-error issue."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents that natural-language-to-program parsing success can be as low as 17%, motivating the feedback loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines Answer Set Programming and stable-model semantics, the formal foundation of the solver side."}],"review_version":1}