{"id":"b615f0fc-7236-4fdf-b58b-65b59d11ba49","arxiv_id":"2505.05106","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes LTLZinc, a generator of relational-temporal sequence classification benchmarks, and shows that current neuro-symbolic methods degrade when temporal and relational reasoning are combined.","lead":"This paper introduces LTLZinc, a benchmark generator for sequence classification that combines relational constraints (how objects relate at one moment) with temporal rules (how those relations evolve over time). It finds that purely neural systems struggle on these tasks and that stacked neuro-symbolic components are prone to training instability, making the benchmark a useful stress test for the field.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Q1 never trains a neural model from sequence labels alone, so the claimed failure of 'purely neural pipelines' is unsupported until an end-to-end sequence-level baseline is tested.","rationale":"I considered the reader's representativeness concern and the best-of-many selection issue, but the more immediate, internal weakness is that Q1 does not test what its conclusion asserts. The text explicitly says supervision is provided at every level, and Appendix B.4 confirms all losses are active; hence the 'sequence-level labels alone' claim is not backed by any experiment in the paper. This is not an external generalizability issue but an internal mismatch between evidence and claim. The proposed check is inexpensive, uses the released code and data, and would settle whether the headline negative result about purely neural methods is an artifact of the modular pipeline. I also agree with the reader that best-of-many hyperparameter selection and the Task 3 bracketed single run should be reported more conservatively, but those are conditions on presentation; the missing neural baseline is the load-bearing gap.","tokens_in":22458,"tokens_out":7722,"duration_ms":73913,"concrete_test":"Using the released LTLZinc datasets and the same CNN backbone as Appendix B.1, train an end-to-end sequence classifier on the 320 training sequences per task: CNN features per timestep, then an LSTM (hidden size 128) or one-layer Transformer encoder with attention pooling, followed by a sigmoid; train with binary cross-entropy on the sequence-level labels only (no IC, CC, or NSP losses, no background-knowledge module), for 50 epochs with the same optimizer and seeds. Report test SC accuracy for all six tasks over three seeds, and compare to the neural-only rows of Table 1 and to chance. If any task is solved substantially above the neural-only pipeline (e.g., Task 1 above 0.90, Task 3 above 0.55), the claim that \"no task can be effectively learned with sequence-level labels alone\" is false for that setting; if all six remain near chance, the neural-failure claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is that the central negative result about neural methods is established only for the paper's own modular, densely supervised pipeline, not for a neural model trained from sequence labels. In Section 6 (Q1) the authors write that \"Supervision is provided at every level, weighting each loss by the same positive coefficient\", with Appendix B.4 setting lambda_IC = lambda_CC = lambda_NSP = lambda_SC = 1.0. Thus the IC, CC, and NSP modules all receive ground-truth image labels, constraint labels, and automaton-state labels produced by LTLZinc. Nevertheless, the same paragraph concludes that \"no task can be effectively learned with sequence-level labels alone\". That inference does not follow from this experiment, because no condition removes the intermediate supervision. The reader's summary of the paper's strongest claim repeats this: \"purely neural pipelines fail to learn the sequence-level decision rule from sequence labels\". This claim is currently an overreach: the failure could be an artifact of the hand-designed modular architecture, the auxiliary loss weighting, or the choice of MLP/GRU bottleneck, rather than a general limitation of neural sequence classifiers in this relational-temporal setting. Since the paper's motivation (Section 1) and conclusions (Section 7) both rely on this neural failure to motivate the benchmark and the neuro-symbolic comparison, the missing end-to-end baseline is load-bearing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LTLZinc, a benchmarking framework for sequence classification under relational and temporal background knowledge, where relational constraints are expressed as finite-domain constraints and temporal specifications as LTLf formulas. The authors define a multi-stage neuro-symbolic pipeline (IC, CC, NSP, SC), instantiate each stage with neural or symbolic components, and evaluate the resulting combinations on six synthetic tasks built from MNIST and Fashion-MNIST. The main empirical findings are that neural-only pipelines struggle on these tasks, that combining symbolic modules for both constraint classification and next-state prediction gives the best trade-off, and that stacking state-of-the-art neuro-symbolic components exposes training instabilities and failure modes. The paper also studies the effect of upstream label noise using oracle predictors.","tokens_in":22689,"tokens_out":5467,"duration_ms":54248,"significance":"The LTLZinc framework, the released generator, datasets, and code, and the extensive experimentation (2,232 runs) are concrete contributions that could serve as a useful testbed for neuro-symbolic temporal reasoning. The observation that symbolic modules are not automatically robust when stacked—temporal reasoners can fail even with full background knowledge, and relational reasoners show training instabilities—is a valuable reference for future research. However, the central negative claim about purely neural pipelines is not established by the experiments as reported, and the headline comparison in Table 1 is weakened by a validation-based selection across many hyperparameter configurations. These issues need to be addressed before the paper's main conclusions can be accepted.","major_comments":[{"comment":"The Q1 experiments provide dense supervision at every stage (λ_IC = λ_CC = λ_NSP = λ_SC = 1.0), so the conclusion that \"no task can be effectively learned with sequence-level labels alone\" is not supported by the evidence. No condition removes the intermediate IC, CC, and NSP supervision; therefore the poor sequence-level accuracy could be an artifact of the modular architecture, the auxiliary loss weighting, or the optimization, rather than a general limitation of neural sequence classifiers trained from sequence labels. Since the motivation in Section 1 and the conclusions in Section 7 rely on this neural failure, an end-to-end baseline trained only on the sequence-level loss (for example, a GRU or transformer over image-derived features) should be added before this claim can be made.","section":"Section 6 (Q1) and Appendix B.5"},{"comment":"The headline results select, for each task, the best configuration out of 96 hyperparameter combinations based on validation-set average accuracy across 3 seeds. This model-selection procedure can inflate the reported test accuracies, and the number of configurations is not balanced across the four compared families (for instance, Neural-Neural has more CC×NSP choices than Symbolic-Symbolic). The claim that the Symbolic-Symbolic family \"achieves the best trade-off for every task considered\" should be supported by reporting the full distribution of results across hyperparameters (as in Fig. 2) or by using a nested validation protocol; otherwise the apparent dominance may be an artifact of selection. Please provide mean and standard deviation across the selected models or apply a proper multiple-comparison correction.","section":"Table 1 and Section 6.1"}],"minor_comments":[{"comment":"Many cross-references are unresolved in the compiled manuscript (\"Appendix ??\", \"Figures ??, ??, ??\"), making it difficult to locate the supporting material; please fix these references before publication.","section":"Throughout (e.g., Section 5, Section 6, Appendix)"},{"comment":"The markers \"∗\", \"†\", and the bracketed sample run are explained only partially in the caption; add a complete legend that also explains what the combined markers (e.g., \"†*\") mean.","section":"Table 1"},{"comment":"The claim that the Scallop module performs \"near-exact inference\" is imprecise: with the default top-1 proof provenance, Scallop performs approximate inference. Please rephrase or justify, since the interpretation of the oracle experiments depends on this point.","section":"Section 6.2"},{"comment":"The total experiment count (2,232) is not obviously reconciled with the per-experiment counts (1,728 main, 36 Q1, 468 Q3); please make the arithmetic explicit to aid reproducibility.","section":"Appendix B.4-B.6"},{"comment":"The sentence \"Exploiting a symbolic component both for CC and NSP allows to achieve the best trade-off for every task considered\" is broader than the evidence; consider changing \"every task considered\" to \"all six tasks considered\" to avoid overgeneralization.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The skeptic's main concern is well-founded: the Q1 experiment does not support the claim that neural models fail when learning from sequence-level labels alone, because all intermediate supervision is provided. This is load-bearing for the paper's motivation and conclusions and must be fixed by adding a proper end-to-end sequence-level baseline. The validation-based selection in Table 1 is also a real threat to the empirical comparison and should be addressed. The benchmark framework and the released code are valuable contributions, and the paper is likely acceptable after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the thing to know about this paper is that the benchmark is real but the headline neural-failure claim is broader than the experiments justify. The authors never train a model from sequence labels alone; their neural-only pipeline receives dense supervision at every stage (image labels, constraint labels, automaton-state labels, and the sequence label, all losses active in Q1). So Q1 shows that their modular neural tower with hand-designed bottlenecks struggles, not that \"no task can be effectively learned with sequence-level labels alone,\" which is how they phrase it in Section 6 and echo in the conclusions. That is a load-bearing overreach because the motivation and the \"state-of-the-art methods fall short\" conclusion rest on it.\n\nWhat is actually new: LTLZinc, a generator that produces datasets where transition labels carry first-order constraints and the sequence label comes from an LTLf formula. That combination has not been covered by the existing neuro-symbolic temporal benchmarks (Umili et al., Manginas et al.), and the paper is explicit about the difference. The empirical study is extensive: 2,232 runs, code and data released, and the failure modes they document are useful open problems. In particular, showing that temporal reasoners (fuzzy and sd-DNNF automata) can break when fed relational constraints even with full knowledge, and that Scallop is unstable when stacked with a recurrent downstream task, is a service to the community.\n\nWhere it gets soft: besides the Q1 overreach, Table 1 selects the best hyperparameter combination per task on the validation set and reports the corresponding test accuracy. Across 96 configurations per task, that best-of-many selection inflates the headline numbers; the paper should at least disclose the distribution or apply a correction. Task 3 is also presented with a bracketed single successful run next to a mean with huge variance, which reads as cherry-picking even though the authors do call it a failure case. And the tasks are all MNIST/Fashion-MNIST with hand-written arithmetic and all-different constraints; the jump to autonomous driving and industrial control in the introduction is entirely speculative.\n\nWho is this for: people building or evaluating neuro-symbolic pipelines for knowledge-driven sequence classification. The benchmark is a stress test worth having, and the comparative results are a reference point even if the neural baseline needs to be redone.\n\nRecommendation: send it to peer review. The core artifact is valuable and the flaws are addressable: add an end-to-end sequence-level-only neural baseline, fix the best-of-many reporting, and characterize Task 3 across seeds. If those are fixed, the paper's claims would be much closer to what the experiments actually show.","headline":"A genuinely useful benchmark with an overstated neural-failure claim: Q1 never trains from sequence labels alone, so the paper's central negative result is broader than the evidence.","tokens_in":23224,"tokens_out":6021,"would_cite":true,"duration_ms":51702,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In knowledge-driven sequence classification that combines relational constraints with LTLf temporal specifications, purely neural pipelines fail to learn the sequence-level decision rule, and state-of-the-art neuro-symbolic components…","keywords":["neuro-symbolic AI","sequence classification","LTLf temporal logic","relational knowledge","automata-based reasoning","knowledge-driven learning","benchmarking framework"],"falsifier":"Take one of the six task specifications and train a purely neural pipeline on a much larger generated dataset, with thousands of sequences and sequence-level labels only; if its sequence accuracy exceeds the best symbolic-symbolic configuration, the claim that neural-only pipelines fail to learn the decision rule is false. Alternatively, construct a real-world relational-temporal sequence dataset with ground-truth labels and show that a neural-only model matches or beats the symbolic-symbolic pipeline, which would challenge the generality of the benchmark's conclusions.","tokens_in":22257,"feed_emoji":"🧩","tokens_out":5660,"duration_ms":53311,"temperature":0.7,"pith_summary":"This paper tries to establish that sequence classification driven by relational and temporal domain knowledge is a hard, under-explored setting in which neither purely neural nor naively stacked neuro-symbolic architectures succeed. It introduces LTLZinc, a benchmark generator that turns LTLf formulas and finite-domain constraints into labelled image sequences, and evaluates a modular pipeline with neural and symbolic components for constraint checking and next-state prediction. The authors claim that no neural-only pipeline learns the sequence rule from sequence labels, that symbolic components at both stages give the best accuracy trade-off, and that temporal reasoners extended to relational inputs can fail even with full background knowledge. A sympathetic reader would care because the setting matches safety-critical applications such as autonomous driving and industrial control, where decisions depend on relations at each step and on the order of events over time.","feed_headline":"Stacked neuro-symbolic reasoners fail on relational temporal tasks","feed_subtitle":"A new benchmark shows symbolic-symbolic pipelines win while neural-only models never learn the sequence rule.","key_machinery":"The carrying object is the constraint-annotated deterministic finite automaton: LTLZinc compiles the LTLf temporal formula into a DFA whose transition guards are finite-domain constraint satisfaction problems, then generates positive and negative sequences by random walks along the automaton. Inference is decoupled into a relational stage, which checks per-timestep constraints, and a temporal stage, which predicts the automaton state from the constraint-validity trace. Symbolic temporal variants evaluate the automaton's transition formulas by algebraic model counting over probability or log-probability semirings, implemented either with fuzzy-logic automata or with formulas compiled to sd-DNNF, a knowledge compilation target that supports polynomial-time model counting.","core_discovery":"On six generated tasks, the paper reports that purely neural pipelines, despite strong image classification, plateau near random or well below ceiling on sequence-level accuracy, so the sequence-level decision rule is not learned from sequence labels alone. When the pipeline is built from symbolic modules, the best configuration uses a symbolic constraint checker and a symbolic automaton-based next-state predictor, giving the best constraint-versus-temporal accuracy trade-off on every task. The paper also identifies two failure modes: temporal reasoners that excel on propositional inputs can fail when inputs are relational, even with perfect background knowledge and oracle labels, and first-order relational reasoners show training instabilities when recurrently stacked, with variance driven by architecture rather than initialization. Oracle experiments show that a confidently wrong classifier can beat a reluctantly correct one, meaning uncertainty can accumulate adversely over reasoning steps.","pith_inferences":["If these failure modes are general, practical neuro-symbolic systems for safety-critical temporal decision-making should keep symbolic constraint solvers and symbolic automata in the loop, using neural components only at perception, and should avoid end-to-end gradient flow through stacked soft reasoners.","A testable extension would vary the temporal horizon and the arity of constraints in generated tasks; the paper's results predict a sharp cliff where neural next-state prediction falls to chance as the DFA grows, while symbolic-stack variance increases.","The confidently-wrong-beats-reluctantly-correct result suggests that training objectives penalizing overconfidence on the sequence label, or calibration-aware losses, could improve downstream temporal accuracy.","LTLZinc's incremental generation mode could be used to stage curriculum learning over formulas of increasing temporal depth, potentially mitigating the training instability the paper observes."],"forward_implications":["Combining symbolic modules at both the constraint and temporal stages yields the best accuracy trade-off on all six tasks, so the paper's evidence favors decoupled symbolic-symbolic pipelines over neural-only or mixed stacks.","Neural-only methods cannot recover the sequence-level rule from sequence labels alone in this setting; the sequence label is predicted at or near chance for several tasks even when image classification accuracy exceeds 0.9.","Temporal reasoners known to work on propositional traces are not robust to relational inputs: they can fail even when given full background knowledge, so relational-temporal integration needs dedicated treatment rather than simple propositionalization.","Relational reasoners exhibit training instability in recurrent settings, and convergence depends on architectural choices rather than weight initialization, so multi-run reporting is essential for reliable conclusions.","The oracle experiments imply that wrong-but-confident upstream predictions can outperform correct-but-uncertain ones, making calibration and uncertainty handling first-order concerns in multi-stage neuro-symbolic reasoning."],"supporting_citations":[{"why":"Supplies the LTLf finite-trace semantics that defines the temporal specifications used throughout the benchmark.","marker":"[De Giacomo and Vardi, 2013]"},{"why":"Provides the fuzzy-logic automaton baseline for next-state prediction and the semantic sequence loss evaluated in the pipeline.","marker":"[Umili et al., 2023]"},{"why":"Provides the sd-DNNF probabilistic automaton baseline that the paper adapts for scalable temporal reasoning.","marker":"[Manginas et al., 2024]"},{"why":"Supplies the probabilistic Datalog engine used as the symbolic constraint-classification module.","marker":"[Li et al., 2023]"},{"why":"Defines the knowledge compilation target sd-DNNF used to compile automaton transition formulas.","marker":"[Darwiche and Marquis, 2002]"},{"why":"Provides the finite-domain constraint modelling language used to express relational knowledge and transition guards.","marker":"[Nethercote et al., 2007]"},{"why":"Defines the logic tensor network encoding used by one family of symbolic temporal reasoners.","marker":"[Badreddine et al., 2022]"},{"why":"Supplies the handwritten-digit image dataset that forms one perceptual domain in the generated tasks.","marker":"[LeCun, 1998]"},{"why":"Supplies the fashion-article image dataset that forms another perceptual domain in the generated tasks.","marker":"[Xiao et al., 2017]"}],"fun_headline_variants":["Symbolic pipeline beats neural on temporal sequence tasks","Neuro-symbolic stacks fail on relational time series","New benchmark: neural-only models plateau on relational sequences","Symbolic reasoners win temporal-sequence benchmark","Relational temporal tasks expose neuro-symbolic flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the six generated tasks, built from handwritten-digit and fashion-item images with hand-written constraints and temporal formulas, are representative of the general relational-temporal sequence classification problem; if these tasks are too easy, too contrived, or too sensitive to the random-walk generator, the reported failure modes may not transfer to real applications.","fun_headline_variants_meta":{"raw":{"variants":["Symbolic pipeline beats neural on temporal sequence tasks","Neuro-symbolic stacks fail on relational time series","New benchmark: neural-only models plateau on relational sequences","Symbolic reasoners win temporal-sequence benchmark","Relational temporal tasks expose neuro-symbolic flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1191,"prompt_tokens":823,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":439,"tokens_out":368,"duration_ms":3896,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:12:40.631738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the six task specifications and train a purely neural pipeline on a much larger generated dataset, with thousands of sequences and sequence-level labels only; if its sequence accuracy exceeds the best symbolic-symbolic configuration, the claim that neural-only pipelines fail to learn the decision rule is false. Alternatively, construct a real-world relational-temporal sequence dataset with ground-truth labels and show that a neural-only model matches or beats the symbolic-symbolic pipeline, which would challenge the generality of the benchmark's conclusions.","supporting_citations":[{"cited_title":"Linear temporal logic and linear dynamic logic on finite traces","cited_arxiv_id":null,"evidence_quote":"Supplies the LTLf finite-trace semantics that defines the temporal specifications used throughout the benchmark."},{"cited_title":"Minizinc: Towards a standard cp modelling language","cited_arxiv_id":null,"evidence_quote":"Provides the finite-domain constraint modelling language used to express relational knowledge and transition guards."}],"review_version":1}