{"id":"551bdeb9-873d-4ea7-9c67-0b7eb5c0ae1f","arxiv_id":"2501.11335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A few-shot neuro-symbolic pipeline decomposes policies into logic formulas and evaluates them with three-valued logic, reaching near state-of-the-art accuracy on ShARC without task-specific fine-tuning of its decomposition modules.","lead":"This paper builds a question-answering system that breaks written policies into yes/no questions, turns them into a logic formula, and evaluates the formula with a three-valued logic, using only a few examples and no fine-tuning of the decomposition step. The authors show it performs near the best published systems on the ShARC benchmark and use its transparent design to identify ambiguous reference answers in that dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The advertised 'no task-specific finetuning' is not literally true: the QA module in §4.5 is fine-tuned on ShARC NLI data, and every sub-question truth value flows through that task-trained component, so the headline result depends on task-specific training.","rationale":"I read the paper as a neuro-symbolic pipeline whose advertised property is that only 20 in-context examples are needed and no task-specific fine-tuning is required. For that property to hold, every learned component that touches ShARC data must be frozen. Section 4.5 violates this: RoBERTa-large is fine-tuned on ShARC NLI data. The QA model is load-bearing because it converts natural-language sub-questions into the three truth values on which the deterministic Kleene logic operates. The paper's own numbers make this concrete: QA errors (28%) are nearly as frequent as logic-formulation errors (29%), and a wrong truth value changes the final answer exactly when the formula's value depends on that variable. I therefore agree with the reader's conditional verdict, but I place more weight on the unsupported no-finetuning claim than on the NLI model's general accuracy. The proposed oracle/NLI-only ablation would settle both aspects: it isolates how much performance comes from the task-trained QA component and tests whether the advertised few-shot property holds. The paper is otherwise transparent, includes a useful error analysis, and makes a credible engineering contribution; it should be accepted after the claim is corrected and the ablation is reported.","tokens_in":12183,"tokens_out":5350,"duration_ms":54415,"concrete_test":"Run the full ShARC dev/test pipeline twice, keeping decomposition, relevance, logic formulation, and self-consistency fixed: (a) replace the §4.5 QA model with RoBERTa-large fine-tuned only on MNLI (no ShARC NLI), and (b) replace QA predictions with oracle truth values taken from the gold ShARC dialog trees for the generated sub-questions. Compare micro/macro accuracy to Table 2. If (a) drops materially, the no-finetuning claim fails; if (b) rises substantially above 79.0/70.2, the system is bottlenecked by the task-trained NLI module, confirming that the headline result is not attributable to few-shot decomposition alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims competitive ShARC performance 'with no task-specific finetuning.' Section 4.5, however, fine-tunes RoBERTa-large on MNLI and then 'further fine-tune[s] on the ShARC NLI data.' That is task-specific fine-tuning on data derived from the very benchmark being evaluated. This is not a cosmetic wording issue: the QA module is the component that assigns True/False/Maybe to every decomposed sub-question, and Algorithm 1 propagates those assignments through the deterministic logic evaluator. Section 5.4's error analysis attributes 28% of inspected errors to this QA step, second only to logic formulation. If the NLI model's entailment judgments are wrong, no amount of correct few-shot decomposition or correct logical form can recover, because the formula is evaluated on incorrect truth values. The paper's own Limitations section narrows the claim to 'without ShARC-specific fine-tuning for (de)composition,' which is accurate but much weaker than the abstract. Thus the central claim as stated is unsupported; what is actually demonstrated is that a frozen few-shot LLM for decomposition/logic can be coupled with a ShARC-NLI-finetuned QA model to approach SOTA.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LDPC, a neuro-symbolic pipeline for conversational policy compliance detection on the ShARC benchmark. An LLM, prompted with 20 in-context examples, decomposes a policy into yes/no sub-questions and then composes those questions into propositional logical formulas; a fine-tuned NLI model assigns each sub-question a truth value (True/False/Maybe) from the user scenario; and a deterministic Kleene three-valued logic evaluates the formula to produce Yes, No, Irrelevant, or a follow-up question. The authors report near-state-of-the-art micro accuracy on the ShARC dev set (79.0) and competitive macro accuracy, with a noticeable drop on the test set (70.2 micro). They also present an error analysis attributing failures to the logic formulation, question answering, decomposition, and relevance components, and claim that 36% of inspected errors are ambiguous reference-answer cases. The central advertised claim is that the approach achieves competitive ShARC performance with 'no task-specific finetuning.'","tokens_in":12443,"tokens_out":5508,"duration_ms":53724,"significance":"If the claims were fully supported, the paper would make a useful contribution: it combines few-shot LLM decomposition with a transparent, deterministic logical evaluator, and it explicitly separates the interpretable symbolic decision step from the neural component that assigns truth values. The use of logical equivalence classes for self-consistency, the model-selection protocol on QA4PC, and the component-level error analysis are all strengths. However, the headline 'no task-specific finetuning' claim is not literally supported, because the QA module is fine-tuned on ShARC-derived NLI data and every sub-question truth value flows through that module. The paper also contains a direct contradiction about the provenance of the in-context examples, and the self-consistency sample size is unreported. These issues are fixable with revised claims, clarified experimental setup, and additional analysis, so the work is worth further consideration rather than rejection.","major_comments":[{"comment":"The claim of 'no task-specific finetuning' is contradicted by the QA module. Section 4.5 fine-tunes RoBERTa-large on MNLI and then 'further fine-tune[s] on the ShARC NLI data,' and Algorithm 1 uses Q(s, question) to assign the truth value of every generated sub-question. Since the deterministic logic evaluator can only be as correct as these assignments, and §5.4 attributes 28% of inspected errors to this QA step, the headline result depends on ShARC-derived training. The Limitations section's narrower phrasing ('without ShARC-specific fine-tuning for (de)composition') is accurate, but the abstract and contribution 2 should be revised to match it, or the QA module should be replaced or ablated to substantiate the stronger claim.","section":"Abstract, §4.5, §5.4"},{"comment":"The provenance of the in-context examples is stated inconsistently: §4.2 says they are 'taken from the QA4PC dev set,' while Appendix Table 5 identifies the tree IDs as coming from the ShARC train set. This matters for evaluating the dev result in Table 2: if the QA4PC dev examples overlap with ShARC dev utterances, the 79.0 dev micro accuracy is inflated by leakage, and the drop to 70.2 on the test set would be partly explained. Please clarify the exact provenance of the 20 examples, report whether any overlap exists between those examples and the ShARC dev/test sets, and if necessary report test-only results or exclude overlapping examples.","section":"§4.2, Appendix Table 5, Table 2"},{"comment":"The self-consistency sample size k is never specified for the reported ShARC results. Section 4.3 describes sampling 'k alternative logical forms,' but Table 4 only reports behavior for sample size 3. Without k, the equivalence-class grouping procedure, and the tie-breaking rule, the numbers in Tables 2 and 3 are not reproducible. Please report k for all runs, state whether k was tuned, and include a sensitivity analysis over k.","section":"§4.3, §5.3, Table 4"},{"comment":"The claim that 36% of inspected errors are 'borderline cases that, although different from the expected answer in the ShARC dataset, could be considered correct' is based on a single pass of manual inspection of 160 errors, with no stated annotation protocol or inter-annotator agreement. Since this is the central evidence for the paper's dataset-ambiguity contribution, the authors should provide the criteria used, have at least two annotators label the sample, and report agreement.","section":"§5.4"}],"minor_comments":[{"comment":"The notation in Section 3 is malformed in places, e.g., 'P = {pi}D i=1' and 'C = {(qi, si, hi)}D i=1'; these should be written as indexed sets or cleaned up.","section":"Section 3"},{"comment":"The text says macro accuracy on the dev set dropped from 79.9 to 72.8, but Table 2 lists the dev macro score as 79.7; please reconcile the two numbers.","section":"Table 2, §5.3"},{"comment":"In Table 3, the BERTQA dev row reports BLEU-1 = 47.4 and BLEU-4 = 54.0; it is unusual for BLEU-4 to exceed BLEU-1, so please verify these values and the evaluation script.","section":"Table 3"},{"comment":"The logic formulation prompt asks for a 'python boolean expression,' but the illustrative formulas use symbols such as '¬' and '∧'; please clarify how these are converted into valid Python expressions and whether the LLM is expected to output Python syntax.","section":"§4.3"},{"comment":"Algorithm 1 deletes dictionary entries with 'del questions[ID]' and 'del answers[ID]' while iterating over 'questions'; in Python this would raise a RuntimeError. If this is intended as pseudocode, say so, or show the intended behavior explicitly.","section":"Algorithm 1"},{"comment":"Section 4.1 says the relevance threshold is 'predetermined from a sensitivity analysis,' but Algorithm 1 hardcodes a value of 0.25; the sensitivity analysis should be reported or referenced so the reader can assess how this free parameter was chosen.","section":"§4.1 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the abstract's 'no task-specific finetuning' is not accurate, and the contradiction between §4.2 and Appendix Table 5 about example provenance must be resolved before acceptance. The paper's core architecture is plausible and the final ShARC evaluation is independent of the decomposition/logic modules in a meaningful sense, but the authors need to either soften the claims or run the ablations that would support them. I recommend major revision, not rejection, because the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper does something real: it couples a frozen, few-shot LLM for policy decomposition and logic formulation with a deterministic Kleene three-valued evaluator, and shows that this gets within a few points of fully trained systems on ShARC. The QA step is a RoBERTa NLI model fine-tuned on ShARC NLI data, so the abstract's 'no task-specific finetuning' is overstated; the limitations section says it correctly, but the headline claim is not literally true. That is the biggest soft spot, and it is mostly a wording problem because Section 4.5 is transparent about the fine-tuning.\n\nCredit where due: the logic formulation is well-defined, the self-consistency over logically equivalent formulas is sensible, and the QA4PC model selection experiments are reproducible with clear numbers. The error analysis is the most valuable part. Attributing failures to stages and flagging that 36% of inspected errors were arguably correct due to ambiguous references is genuinely useful for the community. The paper also engages with prior work, including the spurious-pattern critique of ShARC.\n\nThe soft spots beyond the abstract: the relevance threshold, self-consistency sample size, and in-context example count are not fully disclosed; the dev-to-test drop from 79.0 to 70.2 micro accuracy is unexplained; there are no error bars on any number; and the manual error analysis is not released, so the 36% figure is unverifiable. The BLEU scores are poor, which the authors acknowledge, but it still weakens the practical claim.\n\nI think the central idea holds up: a frozen LLM for decomposition and logic, plus a task-fine-tuned NLI module for question answering, can approach SOTA with few examples. The paper is honest about where it fails. I would send it to a serious referee; the authors need to fix the abstract, disclose the missing hyperparameters, and ideally release the error-analysis annotations. For anyone working on interpretable QA or neuro-symbolic reasoning, this is worth reading.","headline":"Solid neuro-symbolic pipeline on ShARC that mostly delivers on its promises, but the abstract oversells the no-fine-tuning claim.","tokens_in":13013,"tokens_out":2386,"would_cite":true,"duration_ms":20999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","03B50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Few-shot LLM prompting can decompose policies into logic formulas that determine compliance answers.","keywords":["policy compliance detection","conversational machine reading","few-shot in-context learning","logic formulation","three-valued logic","ShARC dataset","neuro-symbolic reasoning","self-consistency"],"falsifier":"Run the full pipeline on the ShARC development set with the NLI module fine-tuned only on MNLI and not on ShARC entailment data, and compare end-to-end accuracy; if accuracy stays at roughly the reported 79%, the few-shot decomposition is carrying the result, while a large drop would show the ShARC-specific NLI fine-tuning is the actual source of the competitive numbers.","tokens_in":11981,"feed_emoji":"⚖️","tokens_out":7272,"duration_ms":65773,"temperature":0.7,"pith_summary":"The paper introduces Logical Decomposition for Policy Compliance (LDPC), a neuro-symbolic pipeline for conversational policy compliance detection that turns a written policy into explicit logical form before answering. An LLM prompted with 20 in-context examples decomposes the policy into yes/no sub-questions and composes those questions into a propositional formula, and a three-valued logic with Maybe is evaluated over truth values assigned by a natural-language-inference model, so the final yes/no/follow-up answer is computed deterministically rather than generated opaquely. On the ShARC benchmark the pipeline reaches 79.0% micro accuracy on the development set, within 1.5 points of the best trained system, while the decomposition and logic-formulation stages receive no task-specific fine-tuning. The explicit logic graphs also allow the authors to trace errors, and they report that roughly 36% of inspected errors are borderline cases where the reference answer relies on unsupported assumptions.","feed_headline":"Few-shot LLM prompts decompose policies into checkable logic","feed_subtitle":"On the ShARC benchmark, the pipeline lands within 1.5 points of the best trained system, with no task-specific tuning.","key_machinery":"The load-bearing object is a propositional formula over question variables, produced by an LLM in two prompted steps: first decomposing the policy into yes/no questions, then combining them into a Python boolean expression. The paper stabilizes formula selection with self-consistency, sampling several formulas, grouping logically equivalent ones, and returning a shortest member of the largest equivalence class. Evaluation uses Kleene's strong three-valued logic on values {False, Maybe, True}, where conjunction is min, disjunction is max, negation maps True to False and False to True and leaves Maybe fixed, and an outcome of Maybe triggers a follow-up question. Each variable's truth value comes either from the chat history or from a RoBERTa-large NLI model fine-tuned on ShARC entailment data, mapping 'entails', 'contradiction', and 'neutral' to True, False, and Maybe.","core_discovery":"The central claim is that the reasoning needed for policy compliance conversations can be made explicit and still work well: decompose the policy into basic questions, combine those questions into a logical expression, and evaluate the expression with Kleene's strong three-valued logic, where Maybe signals that a follow-up question must be asked. The paper demonstrates this on ShARC, achieving competitive accuracy against systems trained on thousands of examples while using only 20 in-context examples for the decomposition and logic steps. It further claims that the transparent architecture is a diagnostic tool, showing that a large share of its residual errors on ShARC are cases where the dataset's reference answer makes assumptions the policy text does not support.","pith_inferences":["A natural extension is to move from propositional formulas to deontic or temporal policy logic, which would let the system represent obligations, prohibitions, deadlines, and exceptions that a boolean expression cannot capture.","A testable generalization claim the paper does not make: the same few-shot decomposition should work on out-of-domain policies from a different regulatory domain, requiring only new in-context examples.","Since the error analysis points to the NLI module as a near-tie bottleneck with logic formulation, replacing it with a stronger or uncertainty-calibrated entailment model is a concrete path to higher end-to-end accuracy.","The closed-world versus open-world assumption is the deepest unresolved issue: the system asks a follow-up when evidence is missing, whereas many ShARC reference answers assume missing conditions are satisfied, and any deployment would need to choose one convention explicitly."],"forward_implications":["Policy compliance answers become auditable: every final yes/no/follow-up decision can be traced to a logical formula and to the truth value assigned to each sub-question.","The framework is portable to new policies without retraining, since the decomposition and logic-formation stages are few-shot prompted rather than fine-tuned on the target dataset.","Error localization is direct: a wrong final answer can be attributed to one of four stages, which the paper uses to show that logic formulation and question answering are the dominant failure points.","The Maybe outcome provides a principled trigger for asking a follow-up question whenever the formula is undecided under the current information.","The ambiguity findings imply that benchmark scores on ShARC should be interpreted cautiously, and that reference answers may encode a hidden closed-world assumption."],"supporting_citations":[{"why":"Defines ShARC, the conversational machine reading benchmark and dataset on which the pipeline is evaluated, including the yes/no/follow-up task and utterance structure.","marker":"Saeidi et al., 2018"},{"why":"Provides QA4PC, the annotated expression trees and decomposed policy questions that supply the 20 in-context examples and the test set for model selection.","marker":"Saeidi et al., 2021"},{"why":"Introduces the strong three-valued logic with Maybe that the paper uses to evaluate formulas and to decide when a follow-up question is needed.","marker":"Kleene, 1938"},{"why":"Contributes the self-consistency decoding technique, which the paper adapts to sample and select logical formulas by logical equivalence.","marker":"Wang et al., 2022"},{"why":"Presents E3, an entailment-driven trained baseline that the paper's few-shot results are compared against and that motivates rule-based policy handling.","marker":"Zhong and Zettlemoyer, 2019"},{"why":"Documents spurious patterns in ShARC that the paper's error analysis uses to interpret its borderline-answer findings.","marker":"Verma et al., 2020"}],"fun_headline_variants":["Few-shot prompts turn policies into checkable logic trees","Logic graphs make few-shot policy checks transparent","Three-valued logic cuts few-shot policy reasoning errors","No fine-tuning: few-shot LLM logic beats ShARC baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fine-tuned natural-language-inference model assigns correct truth values to the generated sub-questions from the user's scenario; if that judgment is wrong, the deterministic logic evaluation propagates the wrong value into the final answer.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot prompts turn policies into checkable logic trees","Logic graphs make few-shot policy checks transparent","Three-valued logic cuts few-shot policy reasoning errors","No fine-tuning: few-shot LLM logic beats ShARC baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3482,"prompt_tokens":906,"completion_tokens":2576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2512}},"tokens_in":522,"tokens_out":2576,"duration_ms":18815,"temperature":1.0,"reasoning_tokens":2512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:22:41.834819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on the ShARC development set with the NLI module fine-tuned only on MNLI and not on ShARC entailment data, and compare end-to-end accuracy; if accuracy stays at roughly the reported 79%, the few-shot decomposition is carrying the result, while a large drop would show the ShARC-specific NLI fine-tuning is the actual source of the competitive numbers.","supporting_citations":[{"cited_title":"Cross-Policy Compliance Detection via Question Answering","cited_arxiv_id":"2109.03731","evidence_quote":"Provides QA4PC, the annotated expression trees and decomposed policy questions that supply the 20 in-context examples and the test set for model selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents E3, an entailment-driven trained baseline that the paper's few-shot results are compared against and that motivates rule-based policy handling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents spurious patterns in ShARC that the paper's error analysis uses to interpret its borderline-answer findings."}],"review_version":1}