{"id":"0b33e06a-9d09-4935-b184-ec2e7cb2736a","arxiv_id":"2412.13952","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A prompting strategy that walks LLMs through the steps of the PC algorithm improves their accuracy at inferring causal relations from correlation statements on the Corr2Cause benchmark.","lead":"This paper introduces PC-SubQ, a prompting strategy that breaks the task of inferring causation from correlation into fixed subquestions that follow the steps of the classic PC causal discovery algorithm. The authors report consistent F1 gains over standard prompting methods on the Corr2Cause benchmark across five large language models, with robustness to renamed variables and paraphrased queries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F1 gain of PC-SubQ over few-shot COT is confounded with a much larger in-context demonstration budget (about 28 shots and 16 model calls versus 6 shots and 1 call), so the PC-specific decomposition is not established as the cause of the improvement.","rationale":"The paper's central empirical claim is that decomposing NL-CD into eight PC-mirroring subquestions raises F1 over standard prompting strategies; the reader's strongest_claim adds that the LLM is 'steered through a formal algorithmic procedure.' For that mechanism to be the cause of the gains, the comparison must rule out the obvious alternative that the gains come from the much richer in-context supervision and compute. PC-SubQ as specified in Appendix A contains roughly 28 demonstration examples and requires 16 sequential calls, while the few-shot COT baseline has 6 demonstrations and one call. Increasing the number of demonstrations and the length of the model's 'thinking' (via chained reasoning calls) is known to improve few-shot performance in many LLM settings. The 11-shot SubQ8 stage is particularly suspect: it contains worked answers for each hypothesis family present in Corr2Cause, so the final validity check is nearly a template match. Without a matched-shot control or a non-PC chain with the same decomposition budget, the headline F1 differences (e.g., PaLM 2 from 0.30 to 0.64) could reflect supervision quantity rather than algorithmic steering. This concern is concrete: it is a standard experimental design issue, fully testable from the published prompts and test set. It does not impugn the integrity of the work; the appendices give all prompts, and the reported single-case traces in Appendix E provide some evidence that the chain can execute PC correctly. But a few correct traces do not license the attribution. The paper itself acknowledges the lack of intermediate evaluation (Discussion, Sec. 6), which is a related but distinct limitation: even if intermediate errors were abundant, F1 could still be high; the shot-count confound directly attacks whether the PC decomposition is doing the work. The robustness claim in the abstract is also somewhat overstated because the refactoring/paraphrasing experiments perturb the prompt examples rather than the test queries, with only two natural-story queries shown. These issues together support keeping the CONDITIONAL verdict rather than ACCEPT, but they do not warrant rejection: the reported numbers, if taken at face value, do show that PC-SubQ outperforms the specified baselines on Corr2Cause. A matched control would settle whether the PC-specific design is the active ingredient.","tokens_in":29137,"tokens_out":10288,"duration_ms":93682,"concrete_test":"Run a matched-control chain on at least PaLM 2 L and GPT-4-turbo with the same 8-subquestion / 16-call structure and the same per-step shot counts (1,1,3,4,3,2,3,11), but replace the PC-specific steps with generic ones (e.g., SubQ1 list variables; SubQ2 list correlations; SubQ3 list all conditional independencies per pair; SubQ4 construct an undirected graph; SubQ5 orient edges if constraints force it; SubQ6 combine; SubQ7 refine; SubQ8 use the final graph to answer the hypothesis with the identical 11-shot final exemplars). Compare F1/accuracy on the Corr2Cause test set. In addition, run a flat 28-shot few-shot COT baseline (the 6 existing shots plus 22 additional curated demonstrations) with a single call. If either control matches PC-SubQ's F1 within a few points, the reported improvement is not attributable to the PC decomposition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PC-SubQ's advantage over the baseline prompting strategies is not an apples-to-apples comparison. The baseline few-shot COT uses 6 demonstrations total (Appendix B) and a single LLM call. PC-SubQ uses 28 demonstrations total (Tables 1-10: SubQ1, 1; SubQ2, 1; SubQ3, 3; SubQ4, 4; SubQ5, 3; SubQ6, 2; SubQ7, 3; SubQ8, 11) and 16 LLM calls (two per subquestion). The SubQ8 step alone contains 11 shots spanning every hypothesis type (directly affects, together cause, common effect, mediation), nearly double the baseline's entire shot budget, and includes explicit reminders that directed edges differ from undirected ones. This creates a plausible alternative explanation for the headline F1 improvement: the model is given far more in-context supervision and a much larger reasoning/compute budget, and the final answer step is heavily scaffolded into a near pattern-matching task. The paper never includes a control that matches shot count or compute without the PC structure, so the central claim that 'guiding the LLM through PC algorithm steps' drives the gain is untested. The qualitative Appendix E trace shows one correct PC execution, but single examples do not establish that the mechanism is responsible for the aggregate F1 numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PC-SubQ, a prompting strategy that decomposes the task of inferring causal relationships from natural-language correlation statements into eight fixed subquestions modeled on the steps of the PC algorithm. The prompts are chained by passing the answer of each subquestion to the next, with few-shot chain-of-thought demonstrations prepended to each subquestion. The authors evaluate PC-SubQ on the Corr2Cause benchmark using five LLMs, reporting F1 improvements over zero-shot, zero-shot COT, few-shot, and few-shot COT baselines, and they present robustness experiments involving variable renaming, paraphrasing, and two natural-story examples.","tokens_in":29375,"tokens_out":4796,"duration_ms":43031,"significance":"If the reported results hold, PC-SubQ is a valuable contribution to formal causal reasoning in LLMs: it shows that a fixed prompt chain can steer an off-the-shelf LLM through a standard causal discovery algorithm without retraining, while providing transparent intermediate reasoning. The paper has notable strengths: the full few-shot prompts are provided in the appendix, making the experiments reproducible; no parameters are fitted to the benchmark; the Corr2Cause benchmark is external; and the robustness analysis is motivated by prior work on fine-tuned models. The interpretability benefit is demonstrated concretely in Appendix E. However, the central comparison is confounded by unequal demonstration and inference budgets, and the absence of error bars or significance tests leaves the magnitude of the reported gains uncertain.","major_comments":[{"comment":"The headline comparison is confounded by unequal demonstration and inference budgets. PC-SubQ uses 28 few-shot demonstrations (SubQ1: 1, SubQ2: 1, SubQ3: 3, SubQ4: 4, SubQ5: 3, SubQ6: 2, SubQ7: 3, SubQ8: 11, from Tables 1-10) and 16 LLM calls (two per subquestion, as stated in Appendix A), whereas the few-shot COT baseline uses 6 demonstrations and a single LLM call (Appendix B). SubQ8 alone contains 11 demonstrations, nearly double the baseline's entire shot budget, and includes explicit reminders that directed and undirected edges differ. The paper does not include a control that matches shot count or compute without the PC structure, so the specific claim that guiding the LLM through the PC algorithm steps drives the improvement is not established. I ask for such a control, for example a flat 28-shot prompt without the PC chain, or a non-PC multi-step decomposition with the same call budget.","section":"Section 5 (Fig. 4) vs. Appendix B and Tables 1-10"},{"comment":"No error bars, significance tests, or sampling details are reported for any of the LLM experiments. LLM outputs are stochastic, and footnote 5 states that GPT-4-turbo was evaluated on only a random 20% subset of the dataset, but no seed or repeated-run information is provided. As a result, the reported F1 differences, some of which are modest, cannot be distinguished from run-to-run variation. Please report the number of runs, seeds, standard deviations or confidence intervals, and, where feasible, significance tests on the F1 differences.","section":"Section 5"},{"comment":"The manuscript explicitly acknowledges that only the final SubQ8 output is evaluated, with no ground-truth labels for the intermediate SubQ1-7 answers. Since PC-SubQ's claimed mechanism is faithful execution of the PC algorithm, an incorrect intermediate graph that still yields a correct final label would go undetected; Appendix E is a single qualitative trace. To support the mechanistic claim, the authors should evaluate intermediate outputs, at least SubQ7's inferred graph against the MEC/CPDAG ground truth, or otherwise quantify error propagation across the chain.","section":"Section 6 (Discussion and Conclusion)"},{"comment":"The robustness to variable refactoring and paraphrasing is demonstrated only for PaLM 2 (Fig. 5, left), and the natural-story evidence consists of two qualitative examples (Fig. 5, right). The abstract states that \"Results are robust to causal query perturbations\" without restricting this claim to one model. Please either test robustness across the five LLMs or soften the claim to avoid overgeneralization.","section":"Section 5 (Robustness analysis)"}],"minor_comments":[{"comment":"The premise says \"All the statistical relations among these 3 variables\" although the example has 4 variables (A, B, C, D); this should be corrected.","section":"Table 5, second exemplar"},{"comment":"In the refactored and paraphrased shots, the reasoning line restates the hypothesis incorrectly: the original \"A and B together cause\" becomes \"A and E together cause\", and the refactored version says \"Z and V\" instead of \"Z and Y\". These typos should be fixed since the appendix is intended to support reproduction.","section":"Appendix C"},{"comment":"The phrase \"constrained-based causal discovery\" should be \"constraint-based causal discovery\".","section":"Related Work"},{"comment":"For PaLM 2 L, the reasoning parts of the last three baseline shots were shortened to fit the context limit; please state whether the same truncation was applied to PC-SubQ prompts or discuss the potential effect on the comparison.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a strong group and the appendix-level transparency is a real strength. The main risk is the uncontrolled comparison between PC-SubQ and the baselines, which I believe is fixable with additional controls and statistical reporting. I would not reject the paper, but the revisions described in the major comments are necessary before the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: PC-SubQ is a clean idea and the paper is honestly written, but the main empirical claim is not established. Treat the F1 gains as real but unexplained.\n\nWhat's new: fixing the decomposition to the steps of the PC algorithm, with each of eight subquestions carrying its own few-shot CoT examples, and passing only the previous answer forward. That is a sensible, interpretable recipe, and the appendix gives every prompt, so it is reproducible without code. The robustness check on variable renaming and paraphrasing of the prompts themselves is a plus, and the two natural-story traces are suggestive if not conclusive.\n\nThe soft spot is the comparison. PC-SubQ uses roughly 28 in-context demonstrations and 16 LLM calls per query. The few-shot COT baseline uses 6 demonstrations and a single call. SubQ8 alone has 11 shots, nearly double the baseline's entire budget, and they cover every hypothesis type. The headline improvement could therefore come from extra demonstrations, extra compute, or heavy scaffolding of the final step, rather than from the PC-structured decomposition per se. There is no control matching shot count or model calls without the PC structure. That is a load-bearing gap for the central claim.\n\nOther issues are comparatively minor: no error bars or significance tests, GPT-4-turbo only on a 20% random subset, and natural-story robustness shown for one model and two examples. The paper also evaluates only the final SUBQ8 answer; intermediate graph errors are unmeasured, so a wrong skeleton that still yields the right label would be invisible. The authors flag this limitation themselves, which is honest. The benchmark is built from MECs, so it rewards PC-style reasoning by construction; that is a feature, not a bug, given the task.\n\nThis paper deserves peer review: the method is novel, the prompts are open, and the confound is fixable with ablations, such as a decomposed prompt with the same shot budget but an arbitrary decomposition, or a single-call prompt with all 28 shots. As written, I would not accept it without those controls. I would send it to review with a request for a matched-budget ablation. For readers: useful if you work on LLM reasoning or causal discovery and want a concrete, reproducible baseline; do not cite it as evidence that PC decomposition helps until the ablation exists.","headline":"A clean, fully-specified prompt chain for LLM causal discovery, but the headline gain is confounded with a much larger demonstration and compute budget, so the PC decomposition's contribution is unproven.","tokens_in":29981,"tokens_out":4300,"would_cite":false,"duration_ms":37402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PC-SubQ prompts large language models through the steps of the PC algorithm, and on the Corr2Cause benchmark this beats standard prompting across five LLMs.","keywords":["causal reasoning","large language models","prompting strategies","PC algorithm","causal discovery","correlation","chain-of-thought","Corr2Cause"],"falsifier":"Run PC-SubQ on Corr2Cause but also score the intermediate answers (skeleton after SubQ2, oriented graph after SubQ7) against the ground-truth CPDAGs; if intermediate accuracy is low or a wrong intermediate graph still yields the right final label, the reported F1 gain would not demonstrate that the LLM is actually following the PC steps.","tokens_in":28907,"feed_emoji":"🧩","tokens_out":6669,"duration_ms":50584,"temperature":0.7,"pith_summary":"The paper claims that a fixed, eight-question prompt chain—each question corresponding to one step of the PC algorithm for causal discovery—lets large language models infer causal relations from natural-language correlation statements far better than standard prompting. On the Corr2Cause benchmark the strategy, called PC-SubQ, outperforms zero-shot, zero-shot chain-of-thought, few-shot, and few-shot chain-of-thought prompting across five LLMs as measured by F1. Because the steps are explicit, the model's reasoning is interpretable and errors can be traced to a specific step, and no fine-tuning is needed. If right, this shows that steering a language model through a formal algorithmic procedure by prompts alone can substitute for retraining on causal tasks.","feed_headline":"Eight-step PC prompt chain lifts causal inference F1 across five LLMs","feed_subtitle":"Fixed subquestions mirroring the PC algorithm beat zero-shot and few-shot prompting on Corr2Cause.","key_machinery":"The load-bearing object is a chain of eight fixed prompts, PC-SubQ, whose subquestions correspond to the steps of the PC algorithm—a constraint-based causal discovery method that starts from a complete undirected graph, deletes edges via conditional-independence information, orients v-structures, and then orients remaining edges without introducing new v-structures, yielding a partially directed graph (a CPDAG). The mechanism is sequential prompting: each subquestion is asked in its own LLM call, the next prompt is augmented with only the prior answer (not the full history), and two calls per subquestion separate the reasoning text from the extracted answer. This lets the model execute a formal procedure stepwise, producing interpretable intermediate outputs that can be inspected to localize mistakes.","core_discovery":"PC-SubQ breaks natural-language causal discovery into eight subquestions that mirror the PC algorithm: initialize a fully connected undirected graph, remove edges using the stated conditional independencies, enumerate paths of length two, identify candidate v-structures, orient v-structures, merge orientations into a partially directed graph, orient remaining edges without creating new v-structures, and finally read off whether the queried hypothesis holds in the inferred graph. The LLM is prompted with one subquestion at a time; only the answer to the previous subquestion is inserted into the next prompt, and each subquestion is preceded by few-shot chain-of-thought demonstrations. The paper reports that this chain raises F1 relative to all baselines on all five models tested, is robust to variable renaming and paraphrasing of the prompts, and produces correct answers on natural-story examples even though the demonstrations use only symbolic variables.","pith_inferences":["A testable extension the paper does not run is to score the intermediate outputs (skeleton after SubQ2, partial graph after SubQ7) against ground-truth graphs; that would reveal whether the F1 gain comes from faithful PC execution or from the final hypothesis step compensating for earlier mistakes.","If the chain's bottleneck is SubQ2, as the paper's own limitation discussion hints, then paraphrasing the premise or reordering the conditional-independence statements might change accuracy without changing the underlying graph—an experiment that would isolate the model's ability to map premises to edge removals.","The same fixed-chain recipe could be applied to other constraint-based discovery algorithms, such as variants that allow latent confounders, by replacing the subquestions; the paper explicitly frames the approach as general to tasks with a common underlying algorithm.","For natural stories, formal and commonsense reasoning are entangled; a useful extension would compare PC-SubQ with and without semantic variable names to measure how much of the gain is formal versus semantic."],"forward_implications":["On Corr2Cause, PC-SubQ yields higher F1 than zero-shot, zero-shot COT, few-shot, and few-shot COT prompting for all five LLMs tested (Gemini Pro 1.0, Gemini Ultra 1.0, PaLM 2 L, GPT-3.5-turbo, GPT-4-turbo).","The strategy is robust to prompt perturbations: performance does not drop under variable refactoring and drops only slightly under paraphrasing, a failure mode that the paper reports for fine-tuned models.","PC-SubQ transfers, without modification, to natural-language stories with natural variable names, producing correct reasoning and answers despite seeing only symbolic-name demonstrations.","Because each step is visible, a wrong final answer can be traced to the specific subquestion that failed, making the LLM's causal reasoning auditable.","The approach requires no fine-tuning and uses only a handful of demonstrations per subquestion, suggesting a general recipe: decompose a task along the steps of a shared algorithmic procedure."],"supporting_citations":[{"why":"Supplies the PC algorithm whose steps the eight subquestions mirror, providing the formal backbone of PC-SubQ.","marker":"Spirtes et al., 2000"},{"why":"Provides the Corr2Cause benchmark, the task formulation, and the robustness tests (variable refactorization and paraphrasing) used to evaluate PC-SubQ and to compare against fine-tuned baselines.","marker":"Jin et al., 2024"},{"why":"Chain-of-thought prompting is used both inside each PC-SubQ subquestion's few-shot demonstrations and as the few-shot COT baseline strategy.","marker":"Wei et al., 2022"},{"why":"Least-to-most prompting is the decomposition antecedent that PC-SubQ adapts by replacing inferred subquestions with a fixed set tied to a common algorithm.","marker":"Zhou et al., 2023a"},{"why":"Zero-shot chain-of-thought prompting is one of the baseline strategies that PC-SubQ is compared against.","marker":"Kojima et al., 2022"}],"fun_headline_variants":["PC-SubQ: eight-step prompt chain walks LLMs through PC algorithm","Subquestions mirroring PC steps improve causal inference in five LLMs","Prompt chain breaks correlation into causal steps, boosting LLM F1","Algorithmic prompting: LLMs infer causation via PC-subquestions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The chain works only if the language model can faithfully carry out each PC step from the natural-language premise and pass its answer forward without errors snowballing, and the paper checks only the final answer, not the intermediate graphs.","fun_headline_variants_meta":{"raw":{"variants":["PC-SubQ: eight-step prompt chain walks LLMs through PC algorithm","Subquestions mirroring PC steps improve causal inference in five LLMs","Prompt chain breaks correlation into causal steps, boosting LLM F1","Algorithmic prompting: LLMs infer causation via PC-subquestions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1764,"prompt_tokens":886,"completion_tokens":878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":802}},"tokens_in":502,"tokens_out":878,"duration_ms":6407,"temperature":1.0,"reasoning_tokens":802,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:36:41.853988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PC-SubQ on Corr2Cause but also score the intermediate answers (skeleton after SubQ2, oriented graph after SubQ7) against the ground-truth CPDAGs; if intermediate accuracy is low or a wrong intermediate graph still yields the right final label, the reported F1 gain would not demonstrate that the LLM is actually following the PC steps.","supporting_citations":[{"cited_title":"N., and Scheines, R","cited_arxiv_id":null,"evidence_quote":"Supplies the PC algorithm whose steps the eight subquestions mirror, providing the formal backbone of PC-SubQ."},{"cited_title":"T., and Sch \\\"o lkopf, B","cited_arxiv_id":null,"evidence_quote":"Provides the Corr2Cause benchmark, the task formulation, and the robustness tests (variable refactorization and paraphrasing) used to evaluate PC-SubQ and to compare against fine-tuned baselines."},{"cited_title":"V., Zhou, D., et al","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting is used both inside each PC-SubQ subquestion's few-shot demonstrations and as the few-shot COT baseline strategy."},{"cited_title":"S., Reid, M., Matsuo, Y., and Iwasawa, Y","cited_arxiv_id":null,"evidence_quote":"Zero-shot chain-of-thought prompting is one of the baseline strategies that PC-SubQ is compared against."}],"review_version":1}