{"id":"366f7c2d-ad9c-428d-b41f-4960872c99ee","arxiv_id":"2506.09433","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning a 3B LLM on randomly symbolized reasoning questions reduces spurious-correlation failures and improves OOD accuracy on CLadder and PrOntoQA.","lead":"This paper proposes CAPT, a fine-tuning recipe that rewrites causal and logical reasoning questions into random symbols before fine-tuning a small language model, so the model learns structure instead of word associations. On two reasoning benchmarks, a 3B model trained this way on 100 examples generalizes to out-of-distribution questions better than standard fine-tuning and, on OOD test sets, beats larger models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated assumption that event estimation by GPT-4o-mini is accurate on OOD inputs; the paper's <3% in-domain drop is end-to-end evidence and does not isolate that step.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: event estimation by a pretrained LLM is treated as unbiased and transferable, but the paper does not measure event-estimation error directly on distribution-shifted inputs. My stress-test pass sharpens this in two ways. First, the evidence cited in Section 4.2 is an end-to-end accuracy drop on the commonsense set, which cannot separate errors in event estimation from errors in the subsequent Qwen reasoning step; a 3% drop could conceal substantial estimation errors that happen to cancel or that affect only easy questions. Second, the headline comparison to larger LLMs is a system-level comparison, since GPT-4o-mini performs event estimation at inference time and is not part of the fine-tuned 3B model; if event estimation is wrong on OOD inputs, the claimed decomposition into unbiased steps does not go through. The empirical OOD improvements in Table 2 are plausible and the method is simple, but the central theoretical claim is conditional on an unverified oracle-like capability. This does not move the reader's CONDITIONAL verdict; it reinforces exactly why the revision should add direct event-estimation metrics and an oracle-symbol comparison before the causal-decomposition claim can be accepted at face value.","tokens_in":28192,"tokens_out":5611,"duration_ms":68659,"concrete_test":"Build gold event annotations for the anti-sense and non-sense splits (CLadder's true causal graphs; PrOntoQA's true facts). Run the exact Appendix C event-estimation prompt with GPT-4o-mini and compute symbol-to-entity match rate and per-question structure preservation. Then fine-tune Qwen under CAPT using (a) gold symbols and (b) GPT-4o-mini symbols, and compare results on the correct/incorrect estimation subsets. If correct-estimation questions reproduce the reported OOD gains while incorrect ones collapse, the assumption is load-bearing and currently unverified; if gold and estimated symbols give the same OOD numbers, the concern is resolved. Report variance across three seeds at the 100-sample setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CAPT's guarantee (Eqs. 2-5) requires that event estimation produces the correct event set E and preserves the logical structure S before symbolization. Section 3.3.1 explicitly makes this the load-bearing assumption: pretrained LLMs have \"diluted and negligible\" colliding bias in P(E|X). The only validation offered in Section 4.2 is that applying CAPT drops commonsense accuracy by less than 3%. That is a joint check of estimation + intervention + fine-tuning on in-distribution inputs; it says nothing about the per-question error rate of GPT-4o-mini's event estimation on anti-sense/non-sense inputs, where event identities are deliberately chosen to conflict with pretraining priors. If GPT-4o-mini maps an OOD event incorrectly, omits a variable, or conflates two events, the symbolized prompt given to Qwen contains a corrupted S, so the reported OOD gains would measure the robustness of the anonymization wrapper rather than the causal decomposition. Appendix B admits dependence on \"accurate event estimation,\" but no estimate of that accuracy is reported. The claim that 3B Qwen outperforms larger LLMs is also system-level: the pipeline uses GPT-4o-mini as a non-frozen preprocessing oracle at both train and test time. This does not invalidate the method, but it makes the transferability assumption the single point of failure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Causality-Aware Post-Training (CAPT), a fine-tuning strategy for mitigating event-level spurious correlations in LLM reasoning. The authors model data generation as E→X←S→Y, argue that collider bias in this process injects spurious E–Y associations during pre-training via an unobserved confounder U, and propose two steps: event estimation, in which a separate pretrained model (GPT-4o-mini) extracts the events from a prompt under the stated assumption that event estimation is a transferable, bias-diluted capability; and event intervention, in which all events are replaced by randomly assigned alphabet symbols at both training and inference time before SFT of Qwen2.5-3B on 100–200 symbolized samples with chain-of-thought traces. On CLadder and PrOntoQA with commonsense, anti-sense, and non-sense test splits, the paper reports that CAPT reduces the standard deviation across splits and improves OOD accuracy over standard SFT, and in several settings exceeds GPT-4o with CoT prompting without fine-tuning.","tokens_in":28559,"tokens_out":15370,"duration_ms":155962,"significance":"If the transferability assumption holds, CAPT is an attractive and practical recipe: a small model fine-tuned on roughly a hundred samples becomes markedly more robust to event-level shifts, with large, reproducible gains on PrOntoQA anti-sense (e.g., 100-sample CAPT CoT reaches 82.5 versus 70.75 for Original CoT). The paper deserves credit for stating its key assumption explicitly (Section 3.3.1), shipping the full prompts and in-context examples (Appendix C), reporting hyperparameters (Appendix D.1), and including an ablation that isolates the random-assignment component (Figures 3–4), all of which make the method easy to re-implement and to falsify. The comparison against standard SFT is a meaningful external check even though the SCM derivation is best read as a motivating model rather than a proof. The three qualifications that follow — an internal inconsistency in Eqs. (3)–(5), the unmeasured event-estimation accuracy, and an abstract stronger than Table 2 — are all addressable within the manuscript's scope.","major_comments":[{"comment":"The derivation mixes direct and indirect effects of the unobserved confounder U without stating the required assumption. Equation (3) factorizes the biased prediction with P(Y|s,u), so U appears as a direct parent of Y, but the red spurious path drawn in Figure 1(b) is X←E←U→S→Y, which contains no U→Y arrow. In Equation (4), the step from Σ_{e,s,u} P(Y,u|s)P(e)P(s)P(X|s,e)/P(X) to Σ_{e,s} P(Y|s)P(s|X)P(e|X,s) is justified only by 'U has been marginalized out'; this is valid only under Y⊥⊥U|S, i.e., the absence of a direct U→Y edge. If such an edge exists, intervening on E (do(E)) removes U→E but leaves U's influence on Y through P(Y|s,u) intact, so the claimed bias elimination does not follow. The paper must either remove u from P(Y|s,u) in Eq. (3) and align the figure, or state the no-direct-effect assumption and explain why it holds for spurious attention patterns.","section":"§3.2, Eqs. (3)–(5); Figure 1(b)–(d)"},{"comment":"The transferability of event estimation is the single point of failure of the entire pipeline, but it is never measured directly. The only validation offered in Section 4.2 (the less-than-3% drop on commonsense sets) is an end-to-end check of estimation + intervention + fine-tuning on in-distribution inputs; it does not give the per-question error rate of GPT-4o-mini's event estimation on anti-sense and non-sense inputs, where event identities are deliberately chosen to conflict with pretraining priors. If an event is omitted, conflated, or mis-mapped on an OOD input, the symbolized prompt contains a corrupted logical structure S, and the OOD gains would then measure the robustness of the symbolization wrapper rather than the causal decomposition. Since CLadder and PrOntoQA are template-generated, gold event sets are available; the paper should report event-estimation accuracy against gold annotations on held-out ID and OOD subsets, plus an error-propagation analysis showing how estimation failures affect final answers. Appendix B's acknowledgment that 'CAPT relies on accurate event estimation' does not substitute for such a measurement.","section":"§3.3.1, §4.2, Appendix B"},{"comment":"The abstract's claim that with 100 ID fine-tuning samples CAPT enables 3B-scale models to 'outperform both larger LLMs and standard fine-tuning approaches on in-distribution (ID) and OOD tasks' is contradicted by Table 2 in the CLadder Non-sense column: 100-sample CAPT CoT scores 69.43 versus 72.36 for GPT-4o CoT, and the CAPT Answer-only configuration (67.21 ID / 64.42 anti-sense / 63.57 non-sense) is below GPT-4o CoT on all three CLadder columns. The claim should be qualified per dataset and per configuration (e.g., 'CAPT with CoT outperforms standard SFT across all splits and outperforms GPT-4o CoT on PrOntoQA and on the CLadder ID and anti-sense splits'). In addition, the comparison is a system-level one: the CAPT pipeline uses GPT-4o-mini as a non-frozen preprocessing oracle at both training and test time, while the GPT-4o baseline numbers are obtained without symbolization; the text should say this explicitly and, ideally, report the effect of applying the same symbolization to the larger baseline models.","section":"Abstract; Table 2"}],"minor_comments":[{"comment":"The sentence claiming that random reassignment 'ensures permutation invariance and breaks any selection bias, enforcing the condition E⊥⊥S|X' asserts rather than derives the independence used in Eq. (5); at most, random reassignment can make E and S independent in the empirical training distribution under the additional assumption that the symbolized prompt does not reveal event identity, which is in tension with the fact that event estimation is supposed to recover E from X. Please state the exact independence condition the procedure establishes and prove Eq. (5) from it.","section":"§3.3.2, Eq. (5)"},{"comment":"The unobserved confounder U is introduced in Section 3.2 without being part of the SCM defined in Section 3.1; specify the edges incident to U explicitly and reconcile the P(Y|s,u) term in Eq. (3) with the red path X←E←U→S→Y drawn in Figure 1(b).","section":"§3.2, Figure 1(b)"},{"comment":"The label 'Efficient SFT (≤5%)' is ambiguous without dataset totals; state the total training-set sizes for CLadder and PrOntoQA so that '100 ID fine-tuning samples' can be interpreted, and clarify how many samples were used for the full SFT rows.","section":"Table 2"},{"comment":"The STD columns report standard deviations across the three test splits, not across fine-tuning runs; since the sample-efficiency comparisons sometimes hinge on small margins, report means and standard deviations over multiple random seeds.","section":"§4.2, Table 2"},{"comment":"The CAPT=order ablation is presented only in figures without converged numeric values or run-to-run variance; report the final accuracies in a table so the magnitude of the random-assignment benefit can be assessed.","section":"Appendix D.3, Figures 3–4"},{"comment":"In the second CLadder in-context example, the quantity 0.36−0.48 appears once as '−0.13' and once as '−0.12' across the raw and transformed reasoning traces, and it is labeled a 'natural indirect effect' estimand; fix the arithmetic and verify the estimand naming, since these traces serve as the formatting exemplars for event transformation.","section":"Appendix C.2"},{"comment":"The sentence 'more biased knowledge is also injected into the LLMs during the pertaining' contains a typo ('pertaining' should be 'pre-training').","section":"§2"},{"comment":"GPT-4o-mini is cited to the GPT-4 technical report [Achiam et al., 2023], which does not describe that model; cite the appropriate OpenAI model documentation instead.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid candidate if the assumptions are validated. I recommend requiring the event-estimation accuracy experiment as a condition for acceptance; without it, the central attribution of the OOD gains to the causal decomposition is unverifiable. At the revision stage it would also be worth probing: (i) whether the same symbolization wrapper applied to GPT-4o (as in the non-sense oracle rows of Table 1) would shrink the claimed gap over larger LLMs; (ii) whether the 100-sample gains survive different random seeds, given the absence of repeated-run variance; and (iii) whether the abstract and introduction can be aligned with the CLadder non-sense results in Table 2. The overall idea is simple and publishable if these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the recipe: estimate events in a reasoning prompt, replace them with randomly assigned symbols, and SFT on the symbolized version. That is a clean, practical extension of entity-level debiasing to event-level reasoning, and the experiments show it buys something real. On PrOntoQA anti-sense, 100-sample CAPT CoT gets 82.5 versus 70.75 for standard SFT CoT; on CLadder ID and anti-sense it also beats both standard SFT and GPT-4o with CoT. The ablation in Appendix D.3, random versus deterministic assignment, is a good check that the randomization itself matters. Credit where due: this is a simple, sample-efficient recipe that deserves scrutiny.\n\nThe soft spots are proportionate but real. First, the theory doesn't hold up as written. Equations 3 to 5 mix conditioning and intervention, and the step enforcing E⊥⊥S|X by randomization is an assumed condition, not a derived result. Second, the abstract overclaims. Table 2 shows CAPT CoT underperforms standard SFT on PrOntoQA in-distribution: 87.5 versus 99.5 with 100 samples, 95 versus 100 with 200. So the paper is really about OOD robustness, not ID parity. Third, the load-bearing assumption about event estimation is under-tested. Section 3.3.1 and Appendix B both admit dependence on accurate event estimation, but the only evidence is the <3% in-domain drop, which is a joint check of estimation plus intervention plus fine-tuning. It does not measure GPT-4o-mini's per-question error on anti-sense or non-sense inputs, where event identities deliberately conflict with pretraining priors. If the oracle mis-symbolizes an OOD question, the downstream model reasons over a corrupted structure. Fourth, no artifacts are released and no seed variance is reported.\n\nWho is this for? People working on spurious correlation mitigation, causal reasoning benchmarks, and sample-efficient post-training. The method is simple enough to reproduce once code and data are out. This paper deserves a serious referee. The fixes are straightforward: correct the ID overclaim, report variance across seeds, and either measure event-estimation accuracy on OOD inputs or soften the mechanistic interpretation. Send it to review with those requests.","headline":"CAPT's random event-symbolization recipe is a genuinely simple and plausible debiasing trick with real OOD gains, but the abstract overclaims ID performance and the event-estimation assumption is never directly tested.","tokens_in":711,"tokens_out":792,"would_cite":false,"duration_ms":34658,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAPT rewrites the events in a reasoning question as random symbols before fine-tuning, so the model learns the underlying reasoning structure rather than correlations tied to event names.","keywords":["spurious correlations","causal reasoning","out-of-distribution generalization","event abstraction","symbolic reasoning","supervised fine-tuning","chain-of-thought","sample-efficient fine-tuning"],"falsifier":"Annotate an out-of-distribution version of CLadder or PrOntoQA with ground-truth event-to-symbol mappings, run the paper's extraction prompt, and compare the extracted symbols with the ground truth; if extraction accuracy is well below the accuracy the fine-tuned model shows on those same questions, then the assumption that $\\mathbb{P}(E|X)$ is unbiased is refuted.","tokens_in":28018,"feed_emoji":"🔀","tokens_out":6276,"duration_ms":59589,"temperature":0.7,"pith_summary":"The paper argues that LLM reasoning failures on out-of-distribution inputs are largely spurious correlations at the level of events, not just entities, and that these correlations can be broken after pre-training. It proposes CAPT, which splits the biased prediction into two steps: a pretrained LLM first names the events in the question, then a random symbol assignment erases their identity before the model is fine-tuned. This recoding makes the event variable independent of the reasoning structure, leaving the model to learn only the invariant reasoning trace. The authors report that a 3B-scale model trained with CAPT on 100 in-distribution samples beats both standard fine-tuning and larger models on CLadder and PrOntoQA, including out-of-distribution test sets. The reason the result would matter, if it holds, is that robustness can be bought without collecting hard-to-label out-of-distribution data.","feed_headline":"Randomized symbol recoding lets 3B models beat larger LLMs on OOD","feed_subtitle":"A symbolization step before fine-tuning removes event-level biases, so 100 samples suffice","key_machinery":"The machinery is a structural causal model with four roles: $E$ (events or contextual content), $S$ (latent reasoning structure), $X$ (the prompt), and $Y$ (reasoning trace plus answer), with $X$ as a collider and $Y$ dependent on $S$ alone once events are controlled. CAPT is the two-step pipeline built on this graph: a frozen pretrained LLM performs event estimation by mapping the prompt into placeholders $\\{\\text{symbol}_1,\\text{symbol}_2,\\dots\\}$, and event intervention randomly reassigns those placeholders to capital letters separately for each input. Random assignment is what enforces permutation invariance and blocks new associations during fine-tuning; the frozen estimator never sees the answer, so the spurious path through the confounder is cut.","core_discovery":"The central claim is that the standard learned conditional $\\mathbb{P}(Y|X)$ for a reasoning question is contaminated because the prompt $X$ acts as a collider between the event content $E$ and the latent reasoning structure $S$. The paper formalizes this contamination as $$\\mathbb{P}(Y|X)=\\sum_{e,s}\\mathbb{P}(Y|s)\\mathbb{P}(s|X)\\mathbb{P}(e|X,s)$$ and shows that the last factor is what carries the spurious dependence of the answer $Y$ on events $E$. CAPT removes it in two steps: event estimation extracts $E$ from $X$ using a pretrained LLM, and event intervention rewrites every identified event as a randomly assigned symbol so that $E\\perp\\!\\!\\perp S|X$. The result is $$\\mathbb{P}(Y|X)=\\sum_s \\mathbb{P}(Y|s)\\mathbb{P}(s|X),$$ a prediction driven only by the reasoning trace, which is what the model is fine-tuned to follow. The same intervention marginalizes out the unobserved confounder $U$ representing pre-training bias.","pith_inferences":["If event estimation is transferable, CAPT should extend beyond formal causal and logical benchmarks to any task with identifiable but semantically irrelevant surface features, for example arithmetic word problems or knowledge-grounded question answering; a quick test would be applying the same two-step pipeline to a math word-problem suite with permuted variable names.","The method's ceiling is set by the frozen estimator's event-extraction accuracy on shifted inputs; the reported within-3% in-domain drop is not a direct measurement of that error, so measuring extraction error on an OOD set with known ground-truth symbols would predict where CAPT starts to fail.","Because only one random assignment is used per input at inference, a natural variant is to sample several assignments and aggregate the model's answers; such ensembling could reduce the variance the authors already observe in converged results and would cheaply test how much of the gain is due to randomization.","The argument implies that pre-training bias and fine-tuning bias are separable interventions, which suggests a modular deployment where a stronger frozen event estimator is paired with a small fine-tuned reasoning module and the two are updated independently."],"forward_implications":["For causal and syllogistic tasks, a 3B model fine-tuned with CAPT on 100 samples is claimed to match or beat larger models that use chain-of-thought prompting, suggesting that scale is not the only route to out-of-distribution robustness.","The method needs no out-of-distribution training data: the same randomized symbolization applied at test time maps an OOD question into the same representation space as the in-distribution training samples.","Deterministic symbol assignments, such as alphabetic order, converge faster but lead to worse final results, so the randomization step, not symbolization alone, is carrying the fine-tuning debias.","When the space of reasoning structures is large, training with chain-of-thought traces is necessary; answer-only supervision can leave the correct reasoning path underdetermined.","CAPT's gains are largest precisely on the OOD sets where standard fine-tuning collapses, which is where the paper's separation of pre-training bias from fine-tuning bias makes a concrete practical difference."],"supporting_citations":[{"why":"supplies the CLadder benchmark and the CausalCOT reasoning format used for causal inference evaluation.","marker":"[Jin et al., 2023]"},{"why":"supplies the PrOntoQA logical reasoning benchmark and its evaluation setup.","marker":"[Saparov and He, 2023]"},{"why":"the premise that modeling the data-generating process is necessary for OOD generalization motivates the SCM decomposition.","marker":"[Kaur et al., 2022]"},{"why":"provides the structural causal model and intervention vocabulary used to formalize the two-step debiasing.","marker":"[Pearl et al., 2016]"},{"why":"supplies the front-door and backdoor adjustment ideas the paper adapts, together with part of the commonsense material behind CLadder.","marker":"[Pearl and Mackenzie, 2018]"},{"why":"chain-of-thought prompting is the baseline and training target whose debiasing role CAPT revisits.","marker":"[Wei et al., 2022]"},{"why":"invoked to justify why OOD generalization requires explicit assumptions, here the name-agnostic generation mechanism.","marker":"[Lin et al., 2022]"},{"why":"supports the invariant data-generation perspective used to separate spurious from causal factors.","marker":"[Arjovsky et al., 2019]"}],"fun_headline_variants":["Symbolizing events in post-training beats spurious correlations","Causal post-training: 3B models outperform larger ones on OOD","Random event symbols strip LLM bias: 100 samples enough","Event intervention cuts spurious links, boosting OOD accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on a pretrained LLM being able to identify the events in a question correctly and without bias, even when the question is out-of-distribution; if that event-estimation step is wrong, the whole symbolization passes the error through and the method's stated sample and model gains do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Symbolizing events in post-training beats spurious correlations","Causal post-training: 3B models outperform larger ones on OOD","Random event symbols strip LLM bias: 100 samples enough","Event intervention cuts spurious links, boosting OOD accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2443,"prompt_tokens":937,"completion_tokens":1506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":1435}},"tokens_in":553,"tokens_out":1506,"duration_ms":11388,"temperature":1.0,"reasoning_tokens":1435,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:48:12.272514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate an out-of-distribution version of CLadder or PrOntoQA with ground-truth event-to-symbol mappings, run the paper's extraction prompt, and compare the extracted symbols with the ground truth; if extraction accuracy is well below the accuracy the fine-tuned model shows on those same questions, then the assumption that $\\mathbb{P}(E|X)$ is unbiased is refuted.","supporting_citations":[{"cited_title":"Cladder: Assessing causal reasoning in language models","cited_arxiv_id":null,"evidence_quote":"supplies the CLadder benchmark and the CausalCOT reasoning format used for causal inference evaluation."},{"cited_title":"Language models are greedy reasoners: A systematic formal analysis of chain-of-thought","cited_arxiv_id":null,"evidence_quote":"supplies the PrOntoQA logical reasoning benchmark and its evaluation setup."},{"cited_title":"Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization","cited_arxiv_id":"2206.07837","evidence_quote":"the premise that modeling the data-generating process is necessary for OOD generalization motivates the SCM decomposition."},{"cited_title":"Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35: 0 24529--24542, 2022","cited_arxiv_id":null,"evidence_quote":"invoked to justify why OOD generalization requires explicit assumptions, here the name-agnostic generation mechanism."}],"review_version":1}