{"id":"6bb13f9f-bf7c-4337-abf7-c3d060e8acb3","arxiv_id":"2506.05976","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A sentence-to-next-entity context extraction heuristic with XLM-RoBERTa outperforms supervised fine-tuning of 7-8B LLMs on multilingual narrative role classification.","lead":"This paper shows that a simple rule for picking the sentences around a mentioned entity can let a small multilingual model classify narrative roles in news articles. The approach matches or beats much larger fine-tuned language models on the shared task, at a fraction of the cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SFT comparison is not representative: LLMs receive one joint prompt per document with no context optimization, while XLM-R receives per-entity ent2ent segments. A fairer SFT baseline could overturn the abstract's comparative claim.","rationale":"The reader identified the SFT baseline as the weakest assumption, and the paper's own Section 4.3 confirms that the baseline was a deliberately resource-limited configuration. This is the single most load-bearing issue because the abstract makes a general comparative claim about supervised fine-tuning with larger generative models, yet the evidence only supports a comparison against one particular SFT recipe. The proposed test directly addresses whether the claimed advantage survives a more representative per-entity SFT setup. Other limitations, such as the absence of error bars and heuristic selection on the dev set, are real but secondary: the ent2ent result is plausibly robust in aggregate, and the within-XLM-R comparisons already support the narrower context-optimization claim. Since the reader's verdict is already conditional on rewording or supplementary evidence, my assessment does not change that verdict; it sharpens the specific test that would resolve the uncertainty.","tokens_in":6860,"tokens_out":3077,"duration_ms":31770,"concrete_test":"Re-run the Llama-3.1-8B-Instruct and Mistral-7B-Instruct SFT experiments on the same merged training set, but provide the same ent2ent context segment and prepended entity mention that XLM-R receives, with one entity per prompt rather than all entities in one document-level prompt. Report dev micro-F1 for the checkpoint selected by exact match ratio and by micro-F1. If either model exceeds 47.75, the abstract's comparison should be narrowed to the specific SFT configuration tested; if neither does, the central claim is robust to this confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline comparison is XLM-R with the ent2ent context heuristic (47.75 dev micro-F1) versus Supervised Fine-Tuning of Llama-3.1-8B (31.78) and Mistral-7B (29.52). The SFT baselines are implemented with a single prompt per document asking for all entities at once, with 4-bit LoRA and no context optimization. Section 4.3 explicitly concedes that one-entity-per-prompt prompting and applying context optimization to SFT 'might prove beneficial' but were not tested due to resource limits. This makes the comparison asymmetric: the XLM-R system sees one entity at a time with a distilled entity-to-entity segment, while the generative models are asked to solve a harder joint multi-entity decoding problem from the full document. The observed gap therefore conflates the benefit of per-entity decomposition and context selection with the choice of base model. If a reasonably tuned SFT setup with the same context optimization reaches or exceeds 47.75, the general claim that the approach 'outperforms Supervised Fine-Tuning with larger generative language models' would not hold. The paper's own caveat makes this the most load-bearing weakness in the central argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the LTG submission to SemEval-2025 Task 10, subtask 1, on entity framing in multilingual news articles. The authors compare several rule-based context-extraction strategies for fine-tuning XLM-RoBERTa-large, including single sentence, paragraph, full text, entity-to-entity (ent2ent) segments, and GPT-4o-extracted spans, with an entity-mention prefix prepended to the selected segment. They also report zero-shot ChatGPT-4o classification and supervised fine-tuning (SFT) of Llama-3.1-8B and Mistral-7B using 4-bit LoRA. On the dev set, the ent2ent-prefixed XLM-R obtains the best overall micro-F1 (47.75), outperforming the other XLM-R context variants and the SFT baselines (31.78 and 29.52). The paper argues that simple entity-oriented context optimization enables masked language models with limited context windows to compete with larger generative models.","tokens_in":7070,"tokens_out":4729,"duration_ms":42099,"significance":"The empirical demonstration that simple entity-oriented segment extraction improves performance for a masked language model is a useful practical contribution to shared-task system development. The ablation across context-extraction strategies (Table 2) is informative and supports the conclusion that context selection matters for XLM-R. However, the headline comparative claim against SFT is weakened by the acknowledged asymmetry in experimental setup: the XLM-R system classifies one entity at a time with a distilled context segment, while the SFT baselines use one joint prompt per document with no context optimization. The contribution is best read as a system description rather than a general finding that small MLMs outperform SFT of larger LLMs. Reproducibility is further limited by the lack of variance estimates and by dev-set-based method selection, but the core finding about context selection is likely robust.","major_comments":[{"comment":"The comparison with SFT is not controlled. The XLM-R system classifies each entity separately with an ent2ent context segment and an entity prefix, whereas the Llama and Mistral SFT baselines are given one joint prompt per document and must output roles for all entities at once, without context optimization. The paper explicitly acknowledges in Section 4.3 that per-entity prompting or applying context optimization to SFT \"might prove beneficial\" and was not tested due to resource limits. The gap between 47.75 and 31.78/29.52 therefore conflates the benefit of per-entity decomposition and context selection with the choice of base model. The abstract's claim that the approach \"outperforms Supervised Fine-Tuning with larger generative language models\" is not supported by the presented evidence; it should be qualified or supported by a more competitive SFT baseline.","section":"§3.3 and §4.3, Table 2"},{"comment":"The ranking of context-extraction strategies is based on single runs with no error bars or significance tests, and the best strategy (ent2ent) was selected by inspecting the same dev set used for the final reported numbers. With per-language dev sets as small as 30 (BG) and 91 (EN), micro-F1 differences on the order of 1–2 points (e.g., ent2ent 47.75 vs. sentence 46.06) are plausibly within noise. This selection effect inflates the reported performance of ent2ent and makes the \"best overall\" claim fragile. Reporting multiple seeds or confidence intervals, and ideally holding out a split for method selection, would strengthen the central claim.","section":"Table 2 and §4.1"}],"minor_comments":[{"comment":"\"has proven to to be\" is a typo; it should read \"has proven to be\".","section":"Section 1.2"},{"comment":"\"a simple entity-oriented heuristics\" should be \"a simple entity-oriented heuristic\".","section":"Abstract"},{"comment":"The definition of ent2ent should clarify what happens when no subsequent entity appears before the end of the document, and whether the sentence containing the entity is always included even if it also introduces a new entity.","section":"Section 2.1(d)"},{"comment":"The column headers \"language all in-lang samples\" are unclear, and the \"all\" row appears to be missing the in-lang value. Please reformat the table for readability.","section":"Table 3"},{"comment":"The caption refers to lines for Llama and Mistral, but the figure description does not include a legend or markers to distinguish the two models; please add them.","section":"Figure 1"},{"comment":"Stating that no hyperparameters were changed from the Transformers default is insufficient for reproducibility; please report the learning rate, batch size, and number of training steps or epochs used for the XLM-R runs.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a concise shared-task system paper with an interesting and well-scoped ablation of context-selection heuristics for XLM-R. The main technical concern is that the headline comparison against SFT of larger LLMs is not well controlled, and the paper's own Section 4.3 concedes the missing conditions. The lack of variance estimates and the dev-set-based method selection make the ranking among XLM-R variants less definitive than the text implies. If the authors soften the comparative claim and add a modest amount of reproducibility information, the paper could be acceptable after minor revision; as submitted, the central claim needs additional support or reframing. The authors also mention their official test-set rankings but do not report the test-set micro-F1 scores, which would be useful for context."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, plainly written system description with one genuinely reusable idea — the entity-to-entity context extraction heuristic — and one overstated headline claim. The stress-test note lands: the SFT comparison is set up at a disadvantage, and the paper itself says so in Section 4.3.\n\nWhat's actually new: the ent2ent rule (entity mention plus following sentences up to the next entity, prefixed with the mention) is simple, parameter-free, and reproducible. The ablations in Table 2 do real work: the prefix matters enormously (47.75 vs 30.11 without it), full text hurts (38.96), and the cheap heuristic beats a GPT-4o extraction baseline (43.14) at zero API cost. That's a practical result for low-resource multilingual classification, and the multilingual-vs-monolingual comparison (Table 3, Figure 3) is a nice bonus. I also credit the authors for flagging in Section 4.3 that their SFT setup — one joint prompt per document, all entities at once, no context optimization — could be improved. That is the main weakness staring out of the paper.\n\nThe soft spots, in proportion: the abstract's comparative claim rests on an asymmetric baseline. XLM-R sees one entity at a time with distilled context; Llama and Mistral are asked to solve a harder joint decoding problem from full documents. A reasonably tuned SFT baseline with per-entity prompts or the same context optimization might close or flip the gap. So the honest summary is: context selection makes a small MLM surprisingly strong, not that it beats a fair SFT setup. Second, all numbers are single runs without variance, and the winning heuristic was chosen on the dev split that is also used for reporting; that's a mild selection effect, not a fatal flaw, but the 47.75 should be read as a point estimate. Third, the official test-set results appear only as rankings, not scores, which weakens the paper as a standalone artifact.\n\nWho gets value: people working on entity-level classification in low-resource or multilingual settings, and anyone building shared-task baselines who wants a cheap, well-ablated context-selection recipe. It's not a conceptual breakthrough, but it doesn't need to be.\n\nRecommendation: yes, send it to peer review — it's a legitimate empirical comparison with an honest limitations section. A referee should ask for reworded claims, test-set numbers, and ideally variance estimates. The central context-selection finding holds; the SFT comparison needs a softer statement.","headline":"The ent2ent context recipe is genuinely useful and honestly reported, but the abstract's 'outperforms SFT' claim is stronger than the experiments support.","tokens_in":7581,"tokens_out":4972,"would_cite":true,"duration_ms":41121,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Entity-to-entity context segments let a fine-tuned masked model beat zero-shot GPT-4o and LoRA-fine-tuned LLMs on narrative role classification.","keywords":["entity framing","narrative roles","context optimization","XLM-RoBERTa-large","supervised fine-tuning","LoRA","multilingual classification","masked language models"],"falsifier":"The decisive check is to re-run the Llama/Mistral supervised fine-tuning with one prompt per entity (optionally with the same ent2ent context) and see whether either model's dev micro-F1 exceeds 47.75; if so, the paper's central comparison would not survive. A second check: replace the ent2ent boundary with a randomly chosen sentence boundary; if the random crop keeps most of the 47.75 score, the specific boundary rule is not what drives the result.","tokens_in":6659,"feed_emoji":"🎯","tokens_out":9187,"duration_ms":79939,"temperature":0.7,"pith_summary":"At issue is whether a model with a 512-token context window can classify the narrative role of an entity in news articles that are much longer than 512 tokens. The paper claims it can, if the input is pre-cropped with a simple rule: take the sentence where the entity appears plus all following sentences up to the next entity mention, and prefix it with the entity's name. Fine-tuned XLM-RoBERTa-large using these 'entity-to-entity' segments reaches 47.75 micro-F1 on the development split, above zero-shot GPT-4o (41.78), LoRA-tuned Llama-3.1-8B (31.78), and LoRA-tuned Mistral-7B (29.52). The paper concludes that context selection rather than model scale is what lets a small multilingual masked model match or exceed supervised fine-tuning of larger generative models.","feed_headline":"Cropped context lets a masked model outscore fine-tuned LLMs","feed_subtitle":"Cutting text at entity boundaries and naming the entity lifts micro-F1 above GPT-4o zero-shot and LoRA SFT baselines.","key_machinery":"The entity-to-entity (ent2ent) context segment. For each annotated entity, the input is the sentence containing the entity plus every later sentence until a new entity appears, prefixed by 'Regarding <entity>:\\n'. This gives the model a focused local window with the target entity named, and the ablation without the prefix (micro-F1 drops from 47.75 to 30.11) shows the prefix itself is load-bearing.","core_discovery":"On the SemEval-2025 Task 10 subtask 1 data, the paper's central discovery is that entity-oriented context extraction is the decisive design choice. Fine-tuning XLM-RoBERTa-large on ent2ent segments (the entity sentence plus all later sentences until the next entity) prefixed with 'Regarding <entity>:\\n' gives 47.75 micro-F1 on the dev set, the best of every approach the paper compares: full-document XLM-R (38.96), GPT-4o-extracted spans (43.14), single sentence (46.06), paragraph (40.79), a two-stage main-role-first pipeline (44.51), zero-shot ChatGPT-4o (41.78), LoRA-supervised Llama-3.1-8B (31.78), and LoRA-supervised Mistral-7B (29.52). The prefix is not decorative: dropping it lowers micro-F1 to 30.11. The authors therefore claim that a rule-based context-selection heuristic brings a small multilingual masked LM to parity with, or better than, supervised fine-tuning of 7-8B generative models for this task.","pith_inferences":["Beyond the paper: the entity prefix likely works as an attention anchor, so varying its wording would separate the contribution of identity signaling from the contribution of span selection.","A second extension: the ent2ent cropping rule should transfer to other multi-entity long-document classification tasks, such as event-role labeling or stance attribution, where a 512-token model must focus on local context.","A third extension: because the paper did not run SFT with one entity per prompt, a fair re-run of Llama/Mistral with ent2ent context is the immediate test that could overturn or confirm the headline comparison."],"forward_implications":["Fine-tuned masked language models remain competitive with much larger generative models on multilingual entity classification when the input is cropped to an entity-focused span.","A simple deterministic cropping rule can outperform both full-document input and spans extracted by GPT-4o for this task.","Merging all languages for fine-tuning helps every language, including low-resource ones, compared with training on the target language alone.","Removing the entity prefix drops micro-F1 from 47.75 to 30.11, so the identity signal is not optional in this architecture."],"supporting_citations":[{"why":"Defines SemEval-2025 Task 10 and the entity framing subtask, providing the task setup and evaluation the paper enters.","marker":"Piskorski et al., 2025"},{"why":"Supplies the annotation guidelines and the 22-role taxonomy that the classification labels and prompt rely on.","marker":"Stefanovitch et al., 2025"},{"why":"Introduces XLM-R and its multilingual pretraining, which the paper fine-tunes and uses to motivate the language-merging tradeoff.","marker":"Conneau et al., 2020"},{"why":"Shows that irrelevant context distracts large language models, providing the core motivation for context optimization.","marker":"Shi et al., 2023"},{"why":"Describes context rewriting for multi-fact retrieval, one of the strands the paper groups under the label 'context optimization'.","marker":"Wang et al., 2024"},{"why":"Provides evidence that focused context reduces distraction in long-context models, another motivation for cropping inputs.","marker":"Wu et al., 2024"},{"why":"Compares few-shot fine-tuning versus in-context learning, supporting the choice of supervised fine-tuning as a strong baseline.","marker":"Mosbach et al., 2023"}],"fun_headline_variants":["Entity-based context defeats fine-tuned LLMs","Simple heuristic outdoes 8B model fine-tuning","Context selection key: masked model tops GPT-4o","Rule-based context beats supervised LLMs","Entity prefix lifts micro-F1 over LoRA baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the assumption that the supervised fine-tuning recipe used for Llama and Mistral—one prompt per whole document asking for all entities at once, with LoRA on a causal LM objective—is a fair representative of SFT; the authors themselves note that one-entity-per-prompt or context-optimized SFT might do better but was not tested due to resource limits.","fun_headline_variants_meta":{"raw":{"variants":["Entity-based context defeats fine-tuned LLMs","Simple heuristic outdoes 8B model fine-tuning","Context selection key: masked model tops GPT-4o","Rule-based context beats supervised LLMs","Entity prefix lifts micro-F1 over LoRA baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1362,"prompt_tokens":875,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":491,"tokens_out":487,"duration_ms":5012,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:16.324430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The decisive check is to re-run the Llama/Mistral supervised fine-tuning with one prompt per entity (optionally with the same ent2ent context) and see whether either model's dev micro-F1 exceeds 47.75; if so, the paper's central comparison would not survive. A second check: replace the ent2ent boundary with a randomly chosen sentence boundary; if the random crop keeps most of the 47.75 score, the specific boundary rule is not what drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SemEval-2025 Task 10 and the entity framing subtask, providing the task setup and evaluation the paper enters."},{"cited_title":"FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval","cited_arxiv_id":"2410.21012","evidence_quote":"Describes context rewriting for multi-fact retrieval, one of the strands the paper groups under the label 'context optimization'."}],"review_version":1}