{"id":"c7a922a1-4ad7-44e1-8ecb-fd20794191b9","arxiv_id":"2507.05997","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A retrieval-based in-context learning pipeline built on synthetic demonstrations achieves only modest entity and relation extraction scores in zero-shot document-level information extraction.","lead":"This paper presents a fully automatic pipeline that uses a large language model to generate synthetic annotation examples from Wikipedia abstracts, then retrieves the most relevant example to help the model extract entities and relations from new documents. The authors release a 5,010-document synthetic dataset and report that in-context document-level relation extraction still yields very low F1 scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pipeline's central claim holds only if synthetic demonstrations improve over zero-shot without demonstrations; the paper never reports that baseline, so the headline contribution is unmeasured.","rationale":"I agree with the reader's conditional verdict and share the concern about unvalidated synthetic annotation quality, but I identify a more direct load-bearing issue: the paper never isolates the contribution of the synthetic demonstrations. The reader's weakest assumption (accuracy of synthetic annotations, Sections 4.2-4.3) is important, but it is downstream of a more basic experimental gap. Even if the synthetic data is perfectly verified, the method's central claim would still be unsupported without a no-demonstration baseline, because the entire system could be performing at the level of the underlying model's zero-shot ability. The missing baseline is cheap to run and directly tests the stated contribution. A fair reading of the paper is that it is an honest systems report with modest numbers, and the dataset release is a real contribution; the paper itself notes the modest scores and parse failures, which supports keeping the verdict CONDITIONAL rather than REJECT. The concrete test I propose would settle the question, and the recommended action is to require the authors to add the no-demonstration baseline (and ideally a manual-demonstration comparison) before final acceptance.","tokens_in":7971,"tokens_out":1468,"duration_ms":14155,"concrete_test":"Run the inference pipeline of Section 3.1 on the DocIE test set with the identical prompt, model, and temperature, but with the in-context example removed (a pure zero-shot call using the same JSON schema). Compare entity identification, entity classification, RE (General), and RE (Strict) F1 against Table 1. If the no-demonstration run matches or exceeds the demonstrated run, the paper's central claim fails; if the demonstrated run is clearly better, the claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that fully synthetic, automatically verified demonstrations can substitute for manual annotation in zero-shot document-level IE. What must be true for this claim is that the retrieved synthetic demonstrations actively help inference, i.e., the full pipeline beats a zero-shot inference run using the same model and prompt but with no demonstration. Nowhere in Section 5 is such a baseline reported: Tables 1-3 compare only against external shared-task systems and against the pipeline's own valid-output subset, neither of which isolates the effect of the synthetic demonstrations. The only internal control mentioned is the text-length experiment in Section 5.1, which concerns annotation yield, not inference quality. This is especially consequential because the inference prompt in Figure 4 includes a demonstration whose schema and relation types are explicitly constrained ('Do not add any entity or relation types! Use only the ones provided'); if the retrieved example is topically relevant but schema-mismatched, it could suppress recall (helping explain entity recall of 23.94% and relation recall of 2.44% on DocIE). Since the paper itself reports that only 63.91% of outputs parse and that relation F1 is below 4%, the contribution of the synthetic demonstration database is not established. The reader's weakest assumption (synthetic annotation quality) is real but downstream; even a perfectly clean synthetic database would not validate the method without a no-demonstration control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a fully automatic pipeline for synthetic data generation for document-level entity and relation extraction, and its use for retrieval-based in-context learning. The pipeline uses DeepSeek-R1-Distill-Qwen2.5-32B in a zero-shot prompt to annotate Wikipedia abstracts, followed by rule-based checks (entity spans appear in text, entity IDs match triples) and an LLM-based triple-directionality verification. The resulting dataset contains about 5k documents, 59k entities, and 30k relation triples. At inference, the most similar synthetic document is retrieved and used as a single in-context demonstration for the same model, with a two-call strategy over the first paragraph and the full document. The method is evaluated on the DocIE shared task and on Re-DocRED, reporting entity F1 around 32-38% and relation F1 below 7%. The paper concludes that zero-shot document-level IE remains challenging.","tokens_in":8318,"tokens_out":3274,"duration_ms":36994,"significance":"If the central claim is supported, the pipeline could lower annotation costs for schema-constrained IE by substituting fully synthetic, automatically verified demonstrations for manually annotated ones. The authors release their code and synthetic dataset, which is a contribution in itself. The evaluation is honestly reported, including parse rates and the distinction between full and valid-output results. However, the absence of a zero-shot baseline and the lack of validation of the synthetic data against a gold standard mean that the core claim that synthetic demonstrations actively help inference is not yet established. The reported relation extraction F1 scores are very low, so the practical strength of the method remains unclear. The paper is a reasonable shared-task system description but needs additional experiments to support its headline contribution.","major_comments":[{"comment":"The central claim of the paper is that fully synthetic, automatically verified demonstrations can substitute for manual annotation in zero-shot in-context learning. This claim is not measured: Tables 1-3 compare against external shared-task systems and against the pipeline's own valid-output subset, but never against zero-shot inference with the same model and prompt without any demonstration. Without such a baseline, the reader cannot tell whether the retrieved synthetic demonstrations help, hurt, or have no effect on extraction quality. Please add a zero-shot no-demonstration control on both DocIE and Re-DocRED, and ideally a control using a manually written demonstration. This is load-bearing for the abstract's claim and can be added with minimal effort.","section":"§3.1, §5.3"},{"comment":"The paper repeatedly calls the synthetic demonstration database 'high-quality', but this quality is never validated against any gold standard. The verification mechanisms in Sections 4.2 and 4.3 are internal consistency checks: entity spans must occur in the text, triple subject/object IDs must match extracted entities, and relation directionality is judged by the same model family that produced the annotations. These checks cannot detect systematic errors such as missing entities, incorrect type assignments, or relations that are plausible but not supported by the text. Since the demonstration examples are fed to the inference model with a prompt that forbids adding new entity or relation types (Figure 4), any noise or schema mismatch in the synthetic data can directly suppress recall and relation F1. To support the quality claim, please provide a human evaluation on a random sample of the synthetic dataset, or compare the synthetic annotations against gold labels on a subset of an existing benchmark (e.g., Re-DocRED training data).","section":"§4.2, §4.3"},{"comment":"Even when restricting to documents with valid outputs, relation F1 remains below 7% on Re-DocRED (general 6.87%, strict 4.26%) and the parse rate is only 38.8%. The paper notes these low numbers but does not analyze whether they stem from the inference model, the retrieval mechanism, or the synthetic demonstration quality. A concrete analysis of parse failures and their correlation with retrieval relevance or schema overlap would help interpret the results and would strengthen the paper's contribution, especially because the abstract claims that ICL with synthetic demonstrations is a viable alternative to manual annotation.","section":"§5.4, Table 3"}],"minor_comments":[{"comment":"The abstract says 'in-context joint entity and relation extraction at document-level remains a challenging task', which is consistent with the results; consider wording the contribution more cautiously in the abstract, e.g., 'we present a pipeline and dataset' rather than implying the method already achieves strong extraction performance.","section":"Abstract"},{"comment":"The sentence 'We find that this strategy drastically decreases failures of the model to adhere to the annotation format for long documents' is about the inference pipeline but appears in Section 4.2, which is about synthetic data generation; please move it to Section 3.1 or clarify.","section":"Section 4.2"},{"comment":"The inference prompt contains 'relation_types' + \": [...]\" with a stray plus sign; this looks like a LaTeX/typing artifact and should be fixed to 'relation_types': [...].","section":"Figure 4"},{"comment":"The text-length experiment is described only qualitatively and with a figure. Reporting the correlation coefficient or a small table of error rates across length buckets would make the claim 'text length and total verification failures are highly correlated' more precise.","section":"Section 5.1"},{"comment":"The sentence 'exhibits consistency and wide coverage of entity and relation types' is grammatically awkward; consider 'is internally consistent and covers a wide range of entity and relation types'.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description, and the authors are transparent about the weak relation extraction results, which is commendable. However, the missing zero-shot baseline is a critical omission for the paper's central claim. The suggested addition of a no-demonstration control is straightforward and would substantially increase the paper's value. The dataset release is a positive aspect, but the quality of the synthetic annotations should be validated beyond internal checks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on DocIE@XLLM25. It's a clean, honest shared-task systems report, and it ships a genuinely useful synthetic dataset. But the paper's headline claim—that fully synthetic demonstrations can substitute for manual annotation in zero-shot IE—is never actually tested, because the experiments never compare against a no-demonstration baseline.\n\nWhat's new: the dataset itself (5,010 Wikipedia abstracts with ~59k entity annotations and ~30k relation triples) and the automatic pipeline that builds it. The two-phase annotation with rule-based span checks and LLM-based triple verification is a thoughtful design; having the model write a natural-language description of each triple before emitting the structured form, then checking consistency, is a nice way to catch directionality errors. Reporting the unparseable-output rate on both benchmarks (63.9% valid on DocIE, 38.8% on Re-DocRED) is honest and useful.\n\nThe soft spots are real. Most importantly, there is no control isolating the effect of the synthetic demonstrations. We never see the same model without demonstrations, or with randomly retrieved ones, or with a manual one. The stress-test note is correct: the only internal control mentioned (Section 5.1) concerns annotation yield, not inference quality. Given the prompt instruction to use only the provided entity and relation types, a topically similar but schema-mismatched demonstration could actively suppress recall—which might help explain the sub-4% relation F1. Without a baseline, the central contribution is unmeasured.\n\nSecond, the 'high-quality' claim for the synthetic database is not validated against gold labels. The internal consistency checks are reasonable, but an LLM verifying an LLM's triples can miss systematic bias. A small human-annotated sample would settle it.\n\nThird, the unparseable-output problem is substantial. Two-thirds of outputs parse on DocIE; fewer on Re-DocRED. The paper's own analysis in Section 5.4 doesn't confirm the length hypothesis, so this remains a major practical weakness.\n\nThe citation pattern is fine, and the evaluation is against external benchmarks, so there's no circularity. The pipeline design is coherent and the results are reported without spin.\n\nWho gets value: anyone working on zero-shot IE, synthetic data generation, or in-context learning. The dataset is a plausible resource. The paper deserves a serious referee, but it needs a no-demonstration baseline and some gold validation before the central claim is credible. I'd send it to review expecting a major revision.","headline":"A useful synthetic dataset and a clean pipeline, but the central claim is unmeasured because the paper never runs the no-demonstration baseline.","tokens_in":8737,"tokens_out":5216,"would_cite":true,"duration_ms":47107,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automatic pipeline turns Wikipedia abstracts into synthetic demonstrations, letting zero-shot document-level entity and relation extraction run without manual annotation, although relation extraction still scores poorly.","keywords":["document-level information extraction","zero-shot relation extraction","in-context learning","synthetic data generation","retrieval-based prompting","LLM verification"],"falsifier":"Take the same retrieval-based in-context pipeline and swap the synthetic demonstrations for gold-annotated demonstrations from Re-DocRED (with the same schema and document type); if the gold demonstrations yield substantially higher entity and relation F1 than the synthetic ones on a held-out set, then the claim that synthetically verified demonstrations can replace manual annotation is falsified. Alternatively, sample a subset of the synthetic annotations and have human annotators judge their correctness; if a significant fraction of triples or entity labels are wrong, the demonstration database is likely noisy enough to mislead the model.","tokens_in":7759,"feed_emoji":"📄","tokens_out":2780,"duration_ms":29456,"temperature":0.7,"pith_summary":"The paper tries to establish that a fully synthetic, automatically verified demonstration database can replace manually annotated examples in zero-shot document-level entity and relation extraction. The pipeline uses a reasoning-optimized language model to annotate Wikipedia abstracts, applies rule-based and LLM-based checks to filter errors, and at inference time retrieves one similar example to prompt the same model. This removes the need for human annotation while keeping the setup zero-shot, since the demonstrations are generated and validated without using the task's training data. The authors report that the approach builds a usable dataset of 5,010 documents, but their evaluation shows that even with such demonstrations, in-context joint extraction remains hard, with relation F1 below 4% on the shared task.","feed_headline":"Synthetic examples replace manual annotation for zero-shot document extraction","feed_subtitle":"Paper builds a retrieval-based in-context pipeline on LLM-verified Wikipedia data, but relation F1 stays below 4 percent.","key_machinery":"The central object is the synthetic demonstration database: 5,010 Wikipedia abstracts annotated with roughly 59k entities and 30k relation triples, produced by an LLM and then filtered by rule-based span and ID checks plus an LLM-based triple verification step. The retrieval step uses sentence transformers (all-MiniLM-L6-v2) to select one demonstration most similar to the query document, and the inference prompt feeds that demonstration followed by a partially completed annotation JSON that the model must finish. The prompts are engineered to force machine-parseable JSON, inline HTML tags for mentions, entity IDs for coreference, and natural-language descriptions of triples, which together enable the verification hooks.","core_discovery":"The central claim is that a fully automatic pipeline combining synthetic data generation, two-phase verification, and retrieval-based in-context learning can supply high-quality demonstrations for zero-shot document-level IE. The pipeline first has DeepSeek-R1-Distill-Qwen2.5-32B produce JSON annotations with span-tagged text, entity lists, and relation triples described in natural language. Rule-based checks confirm that spans appear in the text and that triple subject and object ids match annotated entities; an LLM-based check then verifies each triple against its natural language description, discarding whole relation types if inconsistencies appear. On the DocIE test set, entity identification reaches 32.86 F1, entity classification 16.19, and relation extraction only 3.29 (general) and 3.01 (strict), with 63.91% of outputs parseable. On Re-DocRED, valid-output results are higher but still weak, indicating that the bottleneck is not just document length or output validity.","pith_inferences":["A direct test of the paper's central premise would be to run the same retrieval-ICL pipeline using gold annotations from Re-DocRED as demonstrations and compare against the synthetic ones; a large gap in favor of gold would indicate that synthetic data, despite verification, still misleads the model.","The parse-failure problem might be addressable with constrained decoding that forces the model to output valid JSON, which could raise F1 more than improving the demonstrations themselves.","The LLM-based triple verification step, which checks directionality against natural language descriptions, could be reused as a general post-verification module for any relation extraction pipeline.","The paper's synthetic dataset, although built from Wikipedia abstracts, may transfer poorly to other domains; an extension would measure retrieval quality and downstream F1 when the demonstration source domain differs from the query domain."],"forward_implications":["If the approach works as claimed, building demonstration databases for zero-shot IE no longer requires manual annotation, lowering the cost of adapting to new schemas and domains.","The released synthetic dataset of over 5k documents becomes a reusable resource for future few-shot and zero-shot document-level IE research.","The two-phase verification strategy can be applied to other schema-constrained generation tasks that need structured outputs.","The large share of unparseable outputs (63.91% valid on DocIE, 38.8% on Re-DocRED) identifies output validity as a key bottleneck separate from extraction accuracy.","The consistently low relation extraction scores suggest that in-context joint relation extraction at document level remains unsolved even with strong LLMs and demonstrations."],"supporting_citations":[{"why":"Supplies the reasoning-optimized LLM (DeepSeek-R1-Distill-Qwen2.5-32B) used for both synthetic annotation and inference.","marker":"DeepSeek-AI et al., 2025"},{"why":"Supplies the sentence-embedding model (all-MiniLM-L6-v2) used for retrieving the most similar demonstration.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides the Re-DocRED test set and ground-truth labels used for a second evaluation of the pipeline.","marker":"Tan et al., 2022"},{"why":"Grounds the idea of using synthetic training data for information extraction through the asymmetry-based SynthIE approach.","marker":"Josifoski et al., 2023"},{"why":"Supports the use of LLM-generated data for document-level relation extraction, a foundation for the synthetic augmentation strategy.","marker":"Li et al., 2023"},{"why":"Motivates the verification steps by studying how hallucinations in synthetic training data affect relation extraction.","marker":"Rogulsky et al., 2024"}],"fun_headline_variants":["Synthetic demos replace manual labels for zero-shot document IE","Zero-shot document IE still hard: synthetic demos give 3% relation F1","Synthetic demos, low F1: document IE remains unsolved","5k synthetic docs still lead to 3% relation F1 in zero-shot IE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic annotations produced by DeepSeek-R1-Distill-Qwen2.5-32B, after the automatic verification steps, are accurate enough to serve as effective demonstrations—yet the paper never validates that accuracy against human annotations or gold-standard labels, only against internal consistency checks.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic demos replace manual labels for zero-shot document IE","Zero-shot document IE still hard: synthetic demos give 3% relation F1","Synthetic demos, low F1: document IE remains unsolved","5k synthetic docs still lead to 3% relation F1 in zero-shot IE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":3989,"prompt_tokens":937,"completion_tokens":3052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2969}},"tokens_in":553,"tokens_out":3052,"duration_ms":26028,"temperature":1.0,"reasoning_tokens":2969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:12:34.318221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same retrieval-based in-context pipeline and swap the synthetic demonstrations for gold-annotated demonstrations from Re-DocRED (with the same schema and document type); if the gold demonstrations yield substantially higher entity and relation F1 than the synthetic ones on a held-out set, then the claim that synthetically verified demonstrations can replace manual annotation is falsified. Alternatively, sample a subset of the synthetic annotations and have human annotators judge their correctness; if a significant fraction of triples or entity labels are wrong, the demonstration database is likely noisy enough to mislead the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Re-DocRED test set and ground-truth labels used for a second evaluation of the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the verification steps by studying how hallucinations in synthetic training data affect relation extraction."}],"review_version":1}