{"id":"47fd6b76-22f0-40fa-85b6-1c7bbcb4ec0d","arxiv_id":"2506.10753","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A neuro-symbolic system constructs a causal graph from perceived collisions and uses it to selectively start physical simulation, improving counterfactual question answering on CLEVRER and CRAFT.","lead":"This paper introduces CRCG, a method that uses answer set programming to build a causal graph of collisions in a video and then uses that graph to decide when to trust real perception states versus simulated physics. On the CLEVRER benchmark the method raises counterfactual question accuracy, and it also improves GPT-3.5/GPT-4 answers on the CRAFT benchmark by rephrasing questions as factual ones when the causal graph says the answer is determined.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is the CRCG_approx 'determined' criterion: an original-video collision can be classified as persisting in the counterfactual even when a cascade invisible in the causal graph should change it, and the paper's own Figure 5 acknowledges this without quantifying it.","rationale":"The paper does two things: full CRCG with frame-by-frame simulation, and an approximation for blackbox simulators. The most complete version of the claim, that symbolic orchestration improves VRDP on CLEVRER, is supported by Table 1 and uses Algorithm 1, which includes the collision check that mitigates the cascade failure. The approximate version drives the CRAFT results and the NS-DR enhancement, and there the determined criterion is doing all the work. Without a quantified estimate of how often the Figure 5 failure occurs, a reader cannot tell whether the approximate gains are a robust property of causal reasoning or an artifact of rare cases in these benchmarks. This is an addressable empirical question, not a demonstrated flaw: the paper explicitly acknowledges the failure mode, and the methodology is clear and reproducible. The missing statistical details and the IOD/SPS confound in the overall SOTA table further justify a conditional rather than an unconditional accept. Since the reader already recommended conditional and my concern is the same one identified in its weakest_assumption, the verdict should remain unchanged pending the proposed diagnostic.","tokens_in":21347,"tokens_out":10503,"duration_ms":133964,"concrete_test":"On the released code and CLEVRER validation set, run CRCG_approx and full Algorithm 1 (Mes) with identical VRDP perception and simulation outputs. For each of the 20,558 options in Table 2, record the determined value, the CRCG_approx answer, the Algorithm 1 answer, and ground truth. Then separate the determined options where the two methods disagree. If the full Algorithm 1 is correct in most such disagreements, the cascade failure is a substantive source of error and the approximate-model gains should be reported with this caveat quantified. If disagreements are rare and not preferentially correct under Algorithm 1, the concern is peripheral and the claims stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the 'determined'/affected criterion in Definition 4 (Section 3.3). The criterion labels a counterfactual question as determined yes if the queried collision was perceived and the two queried object states at the collision frame are not descendants of the removed object in the causal graph. But the graph contains only collisions that occurred in the original video. If the removed object changes the trajectory of a third object and that third object then collides with one of the queried objects in the counterfactual world, the queried object's state is altered even though it is graph-unaffected. The paper's Figure 5 is exactly this case: removing blue lets green hit cyan, deflecting cyan and canceling the original purple-cyan collision. The Remark in Section 3.3 admits this can fail even with perfect perception. This is the load-bearing step for every CRCG_approx result (NS-DR on CLEVRER, GPT-3.5/GPT-4 on CRAFT), because those gains come from overriding the baseline prediction whenever the answer is determined. Table 2 reports 95.08% accuracy on 20,558 determined options, so roughly 4.9% of determined predictions are wrong, but there is no breakdown of how many errors are due to this cascade failure versus perception noise. The full Algorithm 1 includes the line-5 test for collisions with simulated objects, which is the intended corrective for exactly this failure; the approximation omits it. Thus the headline CLEVRER SOTA with VRDP, which uses Algorithm 1, is less exposed, but the CRAFT and NS-DR approximate gains are supported only by the unquantified rarity of this failure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRCG, a neuro-symbolic architecture for counterfactual question answering that uses Answer Set Programming (ASP) to build a causal graph from perceived collisions and temporal object states, then uses this graph to decide when to trust perception states over simulation states. For cases where only a black-box simulator (or none) is available, it introduces CRCG_approx, which marks a counterfactual query as 'determined' when the answer can be read directly from the perceived video under the causal graph, overriding the baseline prediction. Experiments on CLEVRER report state-of-the-art counterfactual question accuracy (CRCG_VRDP 87.8% vs. VRDP 84.3%) and on CRAFT show improved GPT-4 accuracy (from 77.93% to 79.68% on the easy split). The implementation is publicly available.","tokens_in":21654,"tokens_out":4415,"duration_ms":55469,"significance":"If the results hold, the core idea is valuable: using symbolic causal reasoning to orchestrate when to use perception versus simulation is a modular, interpretable enhancement that can be applied to existing neuro-symbolic systems and even to LLM-based baselines. The paper ships a public implementation and reports gains on two benchmarks, with ablations showing where the gains come from. The main caveat is that CRCG_approx's 'determined' criterion is acknowledged to be an over-approximation that can fail even with perfect perception; the empirical support would be stronger if this failure mode were quantified and separated from perception noise.","major_comments":[{"comment":"The load-bearing step for all CRCG_approx results (Tables 2, 4, 5) is the 'determined' criterion, yet the paper's own Remark admits that this criterion can be wrong even with perfect perception: a cascade collision invisible in the original video can change a queried object's state, as in Figure 5. The paper states such cases 'don't occur often in practice' but provides no quantitative estimate. Since Table 2 shows that determined predictions are only 95.08% accurate, the 4.9% error rate needs a breakdown: how many errors come from this cascade over-approximation versus from perception noise? Without this breakdown, the improvement attributed to causal-graph reasoning is not isolated from the baseline simulation errors it is meant to correct.","section":"Section 3.3, Definition 4 and Remark; Figure 5"},{"comment":"The accuracy gains for determined options are reported without error bars, confidence intervals, or significance tests. Given that the collision threshold (23.0) and temporal resolution are tuned on the validation set, and the CRAFT GPT experiments use a single temperature-0 run, the reader cannot assess whether the improvements are stable. The paper should report variance across multiple runs or bootstrap resampling, particularly for the 20,558-option CLEVRER subset and the 1,128-question CRAFT subset.","section":"Section 5.1, Table 2 and Section 6, Tables 4-5"},{"comment":"The text says GPT-x is used as a 'proxy for a simulator' (Ms in Figure 4), but in CRCG_approx the simulator is never invoked for determined cases; GPT-x is only used as the baseline predictor for undetermined cases. This is a conceptual mismatch that should be clarified. The CRCG-guided-prompt variant is a different mechanism (rephrasing the query as a perception question) and should be described as such rather than as simulation.","section":"Section 6 and Appendix C"},{"comment":"The state-of-the-art claim on CLEVRER is supported by CRCG together with additional IOD and SPS modules, not by CRCG alone. The ablation in Table 9 is helpful, but the paper should state more explicitly which components are responsible for the headline SOTA number and should report whether the SOTA gap over VRDP is statistically meaningful, especially since the gains on non-counterfactual question types come largely from the simple post-processing modules rather than from causal reasoning.","section":"Section 5.2, Table 3 and Table 9"}],"minor_comments":[{"comment":"The abstract attributes 'state-of-the-art performance' to the enhancement without mentioning that the CLEVRER SOTA result uses the additional IOD and SPS modules; please qualify the claim.","section":"Abstract and Section 5.2"},{"comment":"The prose for 'determined to be no' leaves the quantifier over t implicit: 'st_o1 and st_o2 are not affected' should read 'for all frames t, neither object's state is affected at t' (or equivalent), as the ASP code in Appendix B makes clear.","section":"Section 3.3, Definition 4"},{"comment":"The model name and version for GPT-x are inconsistent: Section 6 says 'gpt-3.5-turbo-0613' and 'gpt-4-0613' with temperature 0, while Appendix C says 'text-davinci-002'. Please reconcile.","section":"Section 6 versus Appendix C"},{"comment":"The column headers in Table 3 are misaligned; there should be separate 'opt.' and 'ques.' columns for each question type, and the current rendering makes several numbers ambiguous.","section":"Table 3"},{"comment":"The footnote says 'See Table 2 for illustration,' but Table 2 does not directly illustrate the rarity of cascade failures; it only reports determined-option accuracy. A direct example or a small case count would be more helpful.","section":"Section 3.3, footnote 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid contribution with a clear architecture and a public implementation, but the main advertised gains from CRCG_approx rest on an acknowledged approximation whose failure rate is not quantified. I would like to see the authors separate cascade-failure errors from perception errors, and add stability/statistical reporting, before I can recommend acceptance. The manuscript fits the journal's scope and is likely to be citable once these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nThe punchline: this paper has a genuinely new idea—using symbolic causal reasoning before simulation, not just after, to decide when to trust perception over simulation for counterfactual video QA—and it delivers real gains. On CLEVRER counterfactual questions, CRCG+VRDP goes from 84.3 to 87.8 question accuracy; on CRAFT, the approximation adds about 1.8 points with GPT-4 as a proxy simulator. The core mechanism is clean: build a causal graph from perceived collisions, mark states affected by the intervention, and start simulation only at the first affected frame. The ASP encoding is transparent and the code is public.\n\nThe paper is also honest about its main weakness. The approximate version—used when no frame-by-frame simulator exists—labels a question 'determined' if the queried collision happened in the video and neither object's state is marked affected. But the graph only contains collisions from the original video. If removing an object lets a third object hit one of the queried ones, the queried state changes even though it's 'unaffected'—exactly the case in Figure 5. The Remark admits this and says it's rare, but doesn't quantify it. Table 2 shows 95.08% accuracy on 20,558 determined options, so about 5% are wrong; there's no breakdown of how many errors come from cascade failures versus perception noise. That matters because in the approximate pipeline, the entire gain comes from overriding the baseline on determined cases. It's a caveat, not a fatal flaw—even with 5% error, the override beats the baseline by a wide margin—but the paper would be stronger with that breakdown.\n\nTwo smaller issues. First, there are no error bars or significance tests anywhere, and the CRAFT GPT-4 gain is modest. Second, the abstract's SOTA claim refers to the full pipeline including the IOD and SPS auxiliary modules, not CRCG alone; the counterfactual-only number in Table 1 is clean, but the overall claim is broader than the headline method.\n\nWho is this for? People in neuro-symbolic reasoning, video QA, or causal inference in hybrid systems. It deserves a serious referee: the idea is novel, the experiments are reproducible, and the limitations are acknowledged. I'd accept it into review and ask for the quantified failure-mode breakdown and error bars.","headline":"A genuinely new neuro-symbolic idea—using symbolic causality to decide when perception is safe—with real gains, but the approximate version's central shortcut has an acknowledged, unquantified failure mode.","tokens_in":22245,"tokens_out":5104,"would_cite":true,"duration_ms":52946,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that symbolic reasoning about a causal graph should decide when a video-question-answering system simulates and when it trusts perception, and shows this improves counterfactual answers on CLEVRER and CRAFT.","keywords":["counterfactual question answering","neuro-symbolic systems","causal graph","Answer Set Programming","video dynamics","CLEVRER","CRAFT","perception-simulation orchestration"],"falsifier":"The paper even supplies the falsifying scenario in Figure 5: remove a blue object and ask whether purple and cyan collide; the graph says the collision is unaffected and determines yes, but if the removed blue object would have collided with green, deflecting green into cyan, the true counterfactual has no purple-cyan collision. A reader could go look for such a case in a physical simulator or real video: whenever the determined verdict disagrees with a ground-truth simulation that models multi-step deflections, the over-approximation is exposed. A systematic version would build a small set of CLEVRER-like videos where the removed object's influence reaches the queried objects only through an intermediate object, and then compare CRCGapprox's determined answers to the actual physics.","tokens_in":21134,"feed_emoji":"🎬","tokens_out":10003,"duration_ms":106302,"temperature":0.7,"pith_summary":"The paper tries to establish that a neuro-symbolic video-question-answering system can be made substantially better at counterfactual questions by deciding, symbolically, when to trust what it saw in the video and when to run a physics simulation. Its central claim is that after an object is removed, an object's simulated trajectory should be replaced by its observed trajectory until the moment the removal could actually affect it, and that this moment can be read off a causal graph of collisions and temporal links. The paper implements the graph computation in Answer Set Programming and combines it with existing perception and simulation modules, reporting state-of-the-art counterfactual accuracy on CLEVRER and improved results on CRAFT when a large language model serves as the simulator. The method is designed to help exactly when perception is more reliable than simulation, which the paper argues is the typical case, and it is modular enough that better perception or better simulation can be plugged in for further gains.","feed_headline":"Causal graph lifts counterfactual video QA to a new best","feed_subtitle":"Symbolic reasoning decides when to trust perception over simulation, boosting CLEVRER SOTA and CRAFT GPT-4 answers.","key_machinery":"The central object is the causal graph, a directed graph whose nodes are object states at collision frames, with horizontal edges connecting two objects that collide at a frame and vertical edges connecting one object's state across consecutive collision frames. From this graph Answer Set Programming computes three derived relations: ancestor, affected, and sim node. A node is affected if an intervened object is one of its ancestors; the sim node for an unremoved object is its earliest affected node, which is exactly the frame where the enhanced simulator must switch from perception states to simulated states. In the black-box approximation, the same graph yields a determined verdict: a counterfactual question is determined yes when the queried collision already happened and both objects were unaffected at that frame, and determined no when a queried object is removed or no such collision happened and both were unaffected. The machinery's job is to avoid blind simulation from the first frame, trusting perception until the causal graph says the intervention could have reached the object.","core_discovery":"In the paper's own terms, the discovery is that symbolic reasoning about causal structure should orchestrate the use of neural perception and neural simulation, rather than merely aggregating their outputs at the end. The authors define a causal graph whose nodes are object states at the frames where collisions occur, with horizontal edges for collisions and vertical edges for the same object across time. A node is affected by the removed objects if an intervened object is one of its ancestors in this graph; the earliest affected node for each remaining object is its simulation node, and the enhanced simulator uses observed states up to that node and simulated states afterwards. For settings without a frame-by-frame simulator, the paper defines a counterfactual question as determined when the answer can be read directly from unaffected perception states, and it overrides the baseline prediction in those cases. This leads to the reported state-of-the-art counterfactual performance on CLEVRER and to gains on CRAFT, where GPT-3.5 and GPT-4 serve as proxy simulators and the causal graph decides whether a counterfactual question can be safely rephrased as a factual one.","pith_inferences":["A testable consequence the paper does not run: vary the quality of the perception and simulation modules across datasets and check whether the size of the CRCG gain tracks the perception-minus-simulation accuracy gap; the paper's own success condition predicts it should.","The determined predicate suggests a general recipe for LLM prompting: when a causal model shows a counterfactual question is insensitive to the intervention, re-ask it as a factual question; this could transfer beyond CRAFT to other LLM tasks with an external world model.","The paper's Figure 5 counterexample points to a stress test: construct videos where the removed object, though absent, would have caused a chain reaction through a third object before reaching the queried pair; CRCGapprox would answer from perception while true physics says otherwise, so a benchmark of such cases would quantify how often the over-approximation bites.","The causal-graph computation is cheap enough that the orchestration layer could plausibly run at inference time to decide whether to trust a simulator, a perception module, or an LLM in a larger video-reasoning system."],"forward_implications":["If the central claim holds, any neuro-symbolic video-QA system gains from CRCG whenever its perception module is more accurate than its simulator, which the paper says is the usual situation.","On CLEVRER counterfactual questions, CRCG with VRDP's perception and simulation modules reaches higher option and question accuracy than the VRDP baseline and the reported prior models.","Applying the approximation to NS-DR lifts accuracy on the roughly twenty thousand question-option pairs whose answers are determined by perception, while leaving the not-determined cases to the baseline.","On CRAFT, using a GPT-x language model as the simulator, CRCG improves counterfactual question accuracy, and on the subset whose answers are determined it reaches 97.96% and 99.00% accuracy on the easy and hard splits.","Because the architecture is modular, the same causal-graph layer can be combined with improved perception and improved simulation modules, which the paper demonstrates by adding its IOD and SPS modules for the other CLEVRER question types."],"supporting_citations":[{"why":"Introduces the CLEVRER benchmark and the NS-DR neuro-symbolic model whose counterfactual answers serve as the baseline that CRCG extends.","marker":"[32]"},{"why":"Provides VRDP, whose differentiable-physics perception and simulation modules CRCG wraps to achieve the new state of the art.","marker":"[9]"},{"why":"Contributes the CRAFT benchmark and its text-based LSTM-D and BERT-D baselines used in the comparison.","marker":"[2]"},{"why":"The end-to-end Aloe model sets a neural baseline for CLEVRER that ODDN-Aloe and VRDP are compared against.","marker":"[8]"},{"why":"ODDN-Aloe is the neural model whose overall CLEVRER performance is on par with VRDP and is used as a comparison point.","marker":"[28]"},{"why":"GPT-3.5, cited as a few-shot learner, is used as the proxy simulator for CRAFT counterfactual questions.","marker":"[5]"},{"why":"GPT-4, cited via its technical report, provides the stronger LLM proxy simulator for CRAFT.","marker":"[21]"},{"why":"Defines answer set programming, the declarative formalism in which the causal graph and determined relations are implemented.","marker":"[15]"}],"fun_headline_variants":["Causal reasoning decides when to trust perception over simulation","Orchestrating neural modules with causal reasoning improves video QA","Think before you simulate: causal logic guides counterfactual QA","Counterfactual video QA achieves SOTA via causal orchestration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the causal graph's affected relation, computed as ancestor reachability through collision and temporal edges, marks exactly which object states change under the intervention; the paper itself notes this is an over-approximation that can be wrong even with perfect perception, giving a case where a chain reaction through a third object would change the answer.","fun_headline_variants_meta":{"raw":{"variants":["Causal reasoning decides when to trust perception over simulation","Orchestrating neural modules with causal reasoning improves video QA","Think before you simulate: causal logic guides counterfactual QA","Counterfactual video QA achieves SOTA via causal orchestration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001322,"raw_usage":{"total_tokens":5392,"prompt_tokens":962,"completion_tokens":4430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":4370}},"tokens_in":578,"tokens_out":4430,"duration_ms":36459,"temperature":1.0,"reasoning_tokens":4370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:18:49.826137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The paper even supplies the falsifying scenario in Figure 5: remove a blue object and ask whether purple and cyan collide; the graph says the collision is unaffected and determines yes, but if the removed blue object would have collided with green, deflecting green into cyan, the true counterfactual has no purple-cyan collision. A reader could go look for such a case in a physical simulator or real video: whenever the determined verdict disagrees with a ground-truth simulation that models multi-step deflections, the over-approximation is exposed. A systematic version would build a small set of CLEVRER-like videos where the removed object's influence reaches the queried objects only through an intermediate object, and then compare CRCGapprox's determined answers to the actual physics.","supporting_citations":[{"cited_title":"Think before You Simu- late: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering","cited_arxiv_id":null,"evidence_quote":"Introduces the CLEVRER benchmark and the NS-DR neuro-symbolic model whose counterfactual answers serve as the baseline that CRCG extends."},{"cited_title":"Dynamic visual reasoning by learning differentiable physics models from video and lan- guage","cited_arxiv_id":null,"evidence_quote":"Provides VRDP, whose differentiable-physics perception and simulation modules CRCG wraps to achieve the new state of the art."},{"cited_title":"Attention over learned object embed- dings enables complex visual reasoning","cited_arxiv_id":null,"evidence_quote":"The end-to-end Aloe model sets a neural baseline for CLEVRER that ODDN-Aloe and VRDP are compared against."},{"cited_title":"Object dynamics distillation for scene decomposition and representation","cited_arxiv_id":null,"evidence_quote":"ODDN-Aloe is the neural model whose overall CLEVRER performance is on par with VRDP and is used as a comparison point."},{"cited_title":"Gpt-4 technical report, 2023","cited_arxiv_id":null,"evidence_quote":"GPT-4, cited via its technical report, provides the stronger LLM proxy simulator for CRAFT."},{"cited_title":"What is answer set programming? In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 1594–1597","cited_arxiv_id":null,"evidence_quote":"Defines answer set programming, the declarative formalism in which the causal graph and determined relations are implemented."}],"review_version":1}