{"id":"d7b43cb0-3a6d-49a3-9033-ab12478aeb50","arxiv_id":"2608.09154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"UNSPECIFIC generates constraints common to two similar articles, hardens only too-easy constraints, and scores satisfaction after summarization, yielding a harder and more natural instruction-following benchmark.","lead":"This paper introduces UNSPECIFIC, a method for generating writing constraints that are harder to satisfy by copying text, along with a benchmark and a scoring metric that checks whether constraints survive summarization. It matters because current instruction-following tests often let models pass by pasting reference text, so better tests could change how LLM obedience is measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core satisfaction metric assumes 25% summaries preserve core constraint-relevant content; no human validation exists for this, so the paper's headline claims about superficial following and copy-paste mitigation rest on an untested operationalization.","rationale":"The paper makes a real contribution: it identifies a genuine loophole in back-translation, proposes concrete synthesis and revision changes, shows reduced constraint-reference similarity, and includes a length-control experiment (App. A.1). The reader's weakest-assumption identification matches my reading: the summary-based core metric is the linchpin for the 'superficial following' and 'copy-paste mitigation' conclusions, and it is the least validated component. My stress test does not find internal contradiction or fabrication; it finds a validation gap. The paper never shows that GPT-4.1 Mini's 25% summaries preferentially discard superficial constraint satisfactions while preserving core ones. Without this, the headline claim that 'a large portion of constraints are satisfied superficially' and the cross-model comparisons (e.g., Gemma 3n E4B's improvement being 'mostly superficial') are not yet supported. The proposed human annotation study would settle it. Because the concern is a missing validation rather than a demonstrated failure, the appropriate verdict remains conditional, not rejection. I therefore recommend no change to the reader's verdict.","tokens_in":28293,"tokens_out":4907,"duration_ms":46731,"concrete_test":"Recruit two expert annotators to judge 150 sampled constraints across the three domains (50 per domain) that were satisfied in the original article but not in its 25% summary: for each, decide whether the original satisfaction was (a) genuinely integrated into the core narrative, (b) superficial/detail-only, or (c) ambiguous. Also annotate 50 constraints satisfied in both original and summary for depth. Compute precision and recall of the Core metric's 'superficial' classification against the human labels. If recall or precision is below 0.8, or if any constraint rated (a) exceeds 10% of the failed-after-summarization set, the metric's validity is not established and the superficial-following conclusions of Secs. 3.3 and 4 need re-analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core metric (Sec. 2.3) is used as the operational definition of 'superficial instruction following': a constraint satisfied in the original article but not in its 25% summary is counted as superficially satisfied. This requires two unvalidated assumptions: (a) the summarizer reliably preserves every constraint that is genuinely integrated into the core narrative, and (b) every constraint lost in summarization was originally satisfied only in peripheral details. Neither is tested. Appendix A.1's short-article control addresses the trivial length-truncation hypothesis for one model, but it does not validate the summary as a core-narrative extractor, nor does it rule out systematic misclassification in the other direction (a superficial phrase surviving summarization). All headline 'superficial' rates (Tables 6-11) and the copy-paste behavior analyses in Sec. 3.3 inherit this untested operationalization, making the central claim that the summary-based metric reliably measures deep versus superficial instruction following unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper targets the copy-and-paste shortcut in back-translated instruction-following benchmarks. It proposes UNSPECIFIC with three components: (i) generating constraints common to two similar reference articles rather than one, (ii) revising only constraints that a constraint-blind base article already satisfies, and (iii) measuring satisfaction after summarizing the generated article to 25% of its length, interpreting failures after summarization as superficial following. The authors construct benchmarks in news, story, and blog domains and report that UNSPECIFIC constraints are more challenging and natural than single-article back-translation, reduce constraint-reference similarity, and expose differences in superficial satisfaction across LLMs. They release code and data.","tokens_in":28496,"tokens_out":8512,"duration_ms":80923,"significance":"If validated, the framework offers a useful recipe for building constraint-satisfaction benchmarks that resist verbatim copying, which is a recognized weakness of back-translation. The paper's strengths include the multi-domain benchmark, multiple baseline comparisons, transparent prompts in the appendix, public release, and honest limitation statements. However, the central metric, core satisfaction after 25% summarization, lacks direct validation, and the naturalness claim rests on a small two-annotator study with low agreement. The difficulty increase is partly a selection effect of the revision procedure. These issues make the current evidence suggestive rather than conclusive, though the direction is promising and the required additional analyses appear feasible within the paper's scope.","major_comments":[{"comment":"The core satisfaction metric assumes that a 25% summary preserves all genuinely integrated constraints and that constraints lost in summarization were satisfied only superficially. Neither assumption is tested. Appendix A.1's short-article control addresses the length-truncation hypothesis for one model, but it does not validate the summarizer as a core-narrative extractor, nor does it rule out a superficial phrase surviving summarization. Moreover, §4.2's own explanation that news has more facts, which tend to live in details, suggests the metric may partly measure domain-specific summarization loss rather than superficiality. Please provide human validation: for a sample of constraints, have annotators judge whether the summary removed genuinely core content and whether constraints that survive or fail after summarization are correctly classified.","section":"§2.3, Tables 6–11, Appendix A.1"},{"comment":"Because the Eval revision replaces exactly the constraints that the constraint-blind base article satisfies, the observed drop in Org and Core from Common+None to Common+Eval is in part the direct result of selecting harder-by-construction constraints. The paper should separate the selection effect from intrinsic difficulty, for example by reporting what fraction of the revised constraints would also be satisfied by a constraint-blind base article, or by comparing satisfaction on a held-out set of constraints matched for baseline satisfiability. Without this, the \"more challenging\" headline overstates the novelty of the difficulty gain.","section":"§2.2, Table 1"},{"comment":"The revision step abandons the reference grounding that guaranteed the original constraint set was jointly satisfiable. After replacing \"too easy\" constraints with new ones, there is no check that the 39 constraints are simultaneously satisfiable; the paper itself acknowledges that contradictions may exist. If some instructions contain an unsatisfiable subset, the lower satisfaction rates reflect infeasibility, not required depth. Please report the proportion of instructions with detectable contradictions, for example by prompting a strong LLM to produce a satisfying article or by checking pairwise consistency, and either exclude or analyze those cases.","section":"§2.2 and Limitations"},{"comment":"Naturalness is a headline claim, but it rests on two expert annotators with Cohen's kappa 0.310 and no confidence intervals, and only in the news domain. The abstract presents the naturalness result without the domain restriction. The study should be expanded with more annotators and per-condition agreement, and the win-rate numbers should be reported with uncertainty intervals. The claim should be restricted to news if story and blog naturalness are not human-evaluated.","section":"§3.1.3, Table 1"},{"comment":"Only 25 instructions per domain are used, and the method depends on several free parameters: the 25% summary ratio, the retrieval similarity threshold of 0.85, the 39-constraint count, and the ~500-word article length. No sensitivity analysis is given for any of these, even though the summary ratio directly controls the core rate and the retrieval threshold controls constraint generality. Please add a compact sensitivity analysis around the default values, or provide a justification for why the conclusions are stable to those choices.","section":"§3.1 and §4.1"}],"minor_comments":[{"comment":"The phrase \"LLM win-rate gap improves by 30%\" should say \"the human-versus-LLM preference gap in the news domain improves by 30 percentage points\"; the current phrasing is ambiguous.","section":"Abstract"},{"comment":"The retrieval criterion \"highest similarity less than 0.85\" does not specify the lower bound or whether the selected article is the top-1 below the threshold; please clarify the retrieval rule.","section":"§2.1"},{"comment":"Many comparative claims are made across differences that may be within the reported standard errors; the paper should state explicitly which differences are statistically meaningful or provide a consistent significance test.","section":"Tables 3, 6"},{"comment":"The appendix examples of the copy-paste problem are very helpful; consider moving one into the main text to illustrate the problem earlier and make the motivation more concrete.","section":"Tables 14–16"},{"comment":"There are minor formatting and terminology issues: Table 6 cells such as \"Claude Sonnet 4.589.6\" are missing spacing, and the embedding model is called \"sentence BERT\" in §2.1 but \"all-mpnet-base-v2\" in §3.1.1; unify the terminology.","section":"Throughout"},{"comment":"The paper reports only pairwise preference results in Table 1; it should also report the 1–5 naturalness scores collected in the MTurk task, since those scores would provide a less coarse view of the naturalness comparison.","section":"§3.1.3 and Appendix D.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honestly written and explicitly states its limitations, which is a positive signal. The main risk is the unvalidated core satisfaction metric; I would not accept the paper without additional validation of that metric and of the joint satisfiability of revised constraints. The naturalness study also needs strengthening before the claims are made as broadly as they are in the abstract. I see no integrity concerns, and the code/data release is a positive contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, genuinely useful paper. It identifies a real loophole in back-translation-based constraint synthesis — the synthesis LLM copies specific text from the reference, and the evaluated LLM can satisfy constraints by copying them back — and it proposes a three-part fix: generate constraints common to two similar articles, revise only the constraints that a base model satisfies without seeing them, and measure satisfaction on a 25% summary of the response. The combination is new, and the benchmark built on it (UNSPECIFIC, news/story/blog, 25 instructions × 39 constraints) is a credible addition to the instruction-following eval toolbox. Code and data are released. The LLM-judge comparison (Sonnet, Haiku, GPT-5 Nano) is a plus, and the appendix short-article control shows the core-satisfaction drop is not just a length artifact for the two models tested there.\n\nThe main soft spot is the core-satisfaction metric itself. The paper assumes a 25% summary preserves every constraint genuinely integrated into the core narrative and that a constraint lost in summarization was only superficially satisfied. Neither assumption is directly validated. The A.1 control tests the trivial length-truncation confound, not whether the summarizer is a reliable core-narrative extractor. So the headline 'superficial following' rates in Tables 6–11 and the copy-paste behavior analysis rest on an untested operationalization. A human study that checks (a) constraints satisfied in both original and summary, and (b) constraints lost in summary but still semantically present in the original, would fix this. I'd want that before treating core satisfaction as a definitive measure.\n\nThe difficulty increase is also partly by construction — the revision step replaces constraints a base model already satisfies — but the final benchmark is evaluated on other LLMs than the base generator, so this is a moderate caveat rather than a fatal flaw. The human naturalness study is small (100 comparisons, κ=0.310, no confidence intervals); the direction of the result is plausible but the estimate is rough.\n\nWho for: people building instruction-following benchmarks or analyzing shortcut behavior in LLMs. The method's value is real but it is not a revision to the foundations: it is a useful new tool. I would send it to peer review — likely conditional accept, with the core-metric validation as the main ask. Serious thinker: yes.","headline":"A useful new benchmark and method for constraint synthesis, with a real but fixable weakness in the summary-based 'core satisfaction' metric.","tokens_in":29000,"tokens_out":2593,"would_cite":true,"duration_ms":23183,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-part constraint recipe—common constraints from two articles, revision of only the trivially satisfied ones, and scoring on the summary—closes the copy-and-paste loophole that lets LLMs fake instruction following.","keywords":["instruction following","constraint synthesis","back-translation","copy-and-paste shortcut","superficial instruction following","summarization-based evaluation","LLM benchmark","naturalness evaluation"],"falsifier":"Take the generated articles whose constraints pass on the full text but fail on the 25% summary, and have human annotators read the full articles and judge, constraint by constraint, whether each flipped constraint was superficial decoration or genuinely integrated into the narrative; if many flipped constraints look genuinely integrated, the metric overstates superficiality. A cheaper companion check is to re-run the benchmark with summaries at 50%, 25%, and 10% of original length: if model rankings shift with the ratio, the metric is partly measuring summary-length sensitivity rather than integration depth.","tokens_in":28116,"feed_emoji":"🧩","tokens_out":9026,"duration_ms":77367,"temperature":0.7,"pith_summary":"Back-translation—generating test constraints by asking a model to extract them from a reference article—is undermined by a copy-and-paste shortcut: the generator copies specific text into a constraint, and the tested LLM copies it back out, earning credit without real compliance. UNSPECIFIC is the paper's attempt to close that loophole with three coordinated changes. Constraints are written against two similar reference articles at once, so they express general properties that no single article can leak; only the constraints a constraint-blind base article already satisfies are replaced with harder ones; and satisfaction is scored on the full response and again on its 25%-length summary, so a constraint that only appears in the details no longer counts. The paper shows this recipe produces harder and more natural constraints (GPT-5 Mini's satisfaction drops from about 90% to 78%, and human-judged naturalness against human-written constraints improves by 30 percentage points) and reveals that many satisfied constraints are followed only superficially.","feed_headline":"Two-source constraints beat the copy-paste cheat in LLM tests","feed_subtitle":"GPT-5 Mini's score falls from 90% to 78% once constraints span two articles and summaries expose shallow compliance.","key_machinery":"The load-bearing mechanism is the pairing of three operations that each close one escape hatch. Common-constraint synthesis (Section 2.1) presents the generator with two retrieved similar articles and demands constraints satisfied by both, removing the specific names, entities, and plot objects that enable copy-paste. Evaluation-based revision (Section 2.2) generates a base article blind to the constraints, runs the satisfaction judge on it, and hardens only the constraints the base article already satisfies—so roughly half the common constraints get replaced instead of all 39, preserving naturalness. Core satisfaction (Section 2.3) summarises the response with GPT-4.1 Mini to 25% of its length and re-runs the judge, defining a constraint as superficially satisfied if it passes on the full text but fails on the summary. The summary-length percentage is the tunable dial that sets how deep 'deep integration' must be.","core_discovery":"The paper's central claim is that the copy-and-paste loophole is not an accident of individual models but a structural flaw of single-reference constraint synthesis, and that three design choices undo it. Common-constraint synthesis presents the generator with a reference article and a second similar article retrieved by sentence embeddings, requiring every constraint to be satisfied by both; proper nouns and unique plot objects disappear from the constraint list because they exist only in one article. Evaluation-based revision generates a base article from the main task alone, then replaces only the constraints that base article trivially satisfies—about 50–60% of common constraints versus 20–30% of single-article ones—with harder constraints, preserving the naturalness that full-list revision destroys. Core satisfaction summarises the response to 25% of its length with an LLM and re-checks every constraint, counting a constraint as superficially followed when it passes on the full article but fails on the summary. On this combination the paper reports benchmark results across news, story, and blog domains showing lower satisfaction rates, a 30-point improvement in naturalness relative to human-written constraints, reduced similarity between constraints and both references and responses, and wide differences in original-versus-summary satisfaction gaps across models—smaller GPT models and larger Claude and Llama models copying more.","pith_inferences":["The 25% summary length is a dial the authors did not turn: measuring core satisfaction at 50%, 25%, and 10% of original length would yield a depth profile of each model's integration, separating length sensitivity from true superficiality.","Common-constraint synthesis transfers beyond prose: generating requirements from two similar code snippets or two similar technical documents would push models toward behavioural specifications instead of regurgitated identifiers, and the article-similarity knob gives benchmark builders a continuous difficulty control.","If the summary metric holds up, it becomes a cheap training reward: fine-tuning to maximise satisfaction on summaries would pressure models to weave constraints into the narrative spine—the authors state explicitly that they have not tested this use.","The lack of human validation of the summarisation assumption is the point to probe next: comparing core-satisfaction verdicts against annotators reading full articles would tell whether the metric measures depth or merely summary sensitivity."],"forward_implications":["Benchmarks built this way are harder without being weirder: satisfaction rates drop for every tested model while human-judged naturalness rises and coherence holds steady.","The original-versus-core gap separates shallow compliance from real integration: models that lean on copying show large gaps, and the paper shows that editing tasks and news-domain facts widen the gap.","Similarity-based copy detection is not a reliable reward: models with comparable constraint-response similarity can differ sharply in core satisfaction, so the summary metric is the usable signal.","Domain composition changes what a benchmark measures: only 28.4% of satisfied constraints in news survive summarisation versus 63.1% in blogs, so aggregate leaderboards can hide superficial following in fact-dense domains.","Model families differ in how extra parameters translate into instruction following: some families convert scale into genuine core satisfaction (the GPT series) while others spend it on surface copying (Claude and Llama families)."],"supporting_citations":[{"why":"Supplies the 39-constraint instruction format and the single-article back-translation baseline that UNSPECIFIC inherits and modifies.","marker":"Atmakuru et al. (2024)"},{"why":"Instruction back-translation, the synthesis paradigm whose copying loophole the paper identifies and targets.","marker":"Li et al. (2024)"},{"why":"Sentence-BERT embeddings retrieve the second similar article needed for common-constraint generation.","marker":"Reimers and Gurevych (2019)"},{"why":"3DLNews local-news corpus provides the news domain with references unlikely to be memorised by LLMs.","marker":"Ariyarathne and Nwala (2024)"},{"why":"Shows verbatim-overlap detectors can be evaded by paraphrasing, motivating the summary-based evaluation metric.","marker":"Ippolito et al. (2023)"},{"why":"HellaSWAG-style adversarial filtering is the direct-revision baseline that hardens every constraint, which the paper contrasts with selective revision.","marker":"Zellers et al. (2019)"},{"why":"Documents LLM judges favouring their own generations, motivating the paper's averaging of two judge models.","marker":"Panickssery et al. (2024)"},{"why":"WritingPrompt dataset supplies the story-domain articles for the benchmark.","marker":"Fan et al. (2018)"},{"why":"Dolma corpus supplies the blog-domain articles, filtered by URL heuristics to known blogging platforms.","marker":"Soldaini et al. (2024)"}],"fun_headline_variants":["Two-source constraints stop copy-paste cheat in LLM tests","LLM copy-paste shortcut broken by two-article constraints","UNSPECIFIC: dual-article constraints slash LLM cheating","Copy-paste no longer passes LLM constraint checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The core-satisfaction metric assumes that a 25%-length summary preserves the article's core narrative and strips away only peripheral detail, so a constraint that fails after summarisation must have been followed superficially—an assumption the paper never validates with human judges.","fun_headline_variants_meta":{"raw":{"variants":["Two-source constraints stop copy-paste cheat in LLM tests","LLM copy-paste shortcut broken by two-article constraints","UNSPECIFIC: dual-article constraints slash LLM cheating","Copy-paste no longer passes LLM constraint checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1497,"prompt_tokens":1088,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":704,"tokens_out":409,"duration_ms":4203,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:28:26.292491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the generated articles whose constraints pass on the full text but fail on the 25% summary, and have human annotators read the full articles and judge, constraint by constraint, whether each flipped constraint was superficial decoration or genuinely integrated into the narrative; if many flipped constraints look genuinely integrated, the metric overstates superficiality. A cheaper companion check is to re-run the benchmark with summaries at 50%, 25%, and 10% of original length: if model rankings shift with the ratio, the metric is partly measuring summary-length sensitivity rather than integration depth.","supporting_citations":[],"review_version":1}