{"id":"27109295-0f6b-4e31-99c4-1c98b30985e0","arxiv_id":"2412.04026","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"M3D is a new video-text, bilingual, four-task document-level information extraction dataset with a benchmark model, but the dataset and code are not yet released.","lead":"This paper introduces M3D, a bilingual English-Chinese dataset pairing document-level text with video, annotated for named entities, entity chains, relations, and visual grounding. It also presents a benchmark model that fuses text and video features and reports average F1 of 53.80 on English and 53.77 on Chinese across the four tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Split construction in §III does not prevent clips from the same source video from straddling train/test; if this leak exists, every reported F1 including the 1.15% margin over MDocRE-HN is inflated.","rationale":"The reader's conditional verdict identifies the correct central risk: dataset validity depends on preventing near-duplicate clip leakage between splits, and the paper does not demonstrate this safeguard. I read the construction pipeline in good faith: crawling biographical videos, splitting them into clips, generating subtitles, manual annotation, then an 8:1:1 split. The text nowhere states that the split respects source-video boundaries, and the default reading is a document-level random split. Since biographical videos are the domain and are split into many short clips, same-source leakage is a concrete mechanism, not a speculative one. The reported margins over the strongest baseline are modest, so leakage could reverse the headline comparison. This concern is more load-bearing than the reader's secondary point about evaluating entity chains and relations on gold entities, because the leakage issue threatens the numerical validity of the entire benchmark, whereas gold-entity evaluation primarily affects how the task results should be interpreted. I also credit the paper's genuine strengths: detailed annotation guidelines, inter-annotator agreement (Kappa 78.18), multiple baseline families including LLMs, ablations, and error analysis. None of these, however, substitutes for split verification. If the concrete test shows clean splits, the resource and benchmark are valuable; if it shows leakage, the results must be recomputed. Therefore the verdict remains CONDITIONAL, with the condition being a verifiable, source-video-aware split.","tokens_in":20122,"tokens_out":2988,"duration_ms":33461,"concrete_test":"Obtain the released dataset with the original source-video ID for each clip, then re-split at the source-video level so all clips from one video land in the same partition, and retrain the proposed model and MDocRE-HN under identical settings. If any source video appears in both train and test under the published split, or if the average F1 margin over MDocRE-HN falls below the reported 1.15%/1.35% after regrouping, the benchmark claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's validity as a benchmark depends on split hygiene, and the paper never establishes it. Section III Step 2 splits long biographical videos into one-to-two-minute clips, producing multiple documents from one source video. The subsequent statistics paragraph says only that 'we split the annotated English and Chinese datasets into training sets, development sets and testing sets at an 8:1:1 ratio,' with no statement that clips from the same original video are kept in the same split. If the split is at the document level, clips from one YouTube or bilibili video can easily appear in both train and test. Adjacent clips from the same biography share near-identical frames, audio-derived subtitles, and entity mentions, so the model can memorize or shortcut on identity-specific visual and textual cues. This would inflate scores on all four tasks, and the reported advantages over MDocRE-HN (1.15% on English, 1.35% on Chinese) are small enough that such leakage could account for them. Because the dataset is not released, readers cannot check this post hoc. The paper must either document a source-video-level split or provide evidence that no overlap exists.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces M3D, a new multimodal, multilingual, multitask dataset for document-level information extraction. The dataset contains 4,093 video-text documents in English and Chinese, annotated for named entity recognition, entity chain (coreference) extraction, relation extraction, and visual grounding. The authors also propose a hierarchical multimodal model with a denoised feature fusion module (DFFM) and a missing modality construction module (MMCM), reporting average F1 scores of 53.80 on English and 53.77 on Chinese, with ablations and error analyses.","tokens_in":20353,"tokens_out":4305,"duration_ms":43112,"significance":"If the dataset is released and its construction is validated, M3D would be a useful resource: it is among the first document-level video-text IE datasets, covers two languages and four tasks in one benchmark, and introduces a biography domain with fine-grained visual grounding annotations. The annotation specifications are detailed, inter-annotator agreement is reported, and the model baseline is accompanied by ablations and error analysis. However, the central artifact is not currently released, and the benchmark's validity depends on split hygiene and the evaluation protocol, both of which have unresolved issues.","major_comments":[{"comment":"The paper does not state whether clips cut from the same source video are kept within the same train/dev/test split. Step 2 splits 30+ minute videos into one-to-two-minute clips, so multiple documents originate from one source video; the statistics paragraph describes only an 8:1:1 document-level split. If clips from one biography video straddle splits, near-duplicate frames and subtitles leak across train and test, which would inflate all F1 scores in Table III, including the 1.15% and 1.35% margins over MDocRE-HN. Please document the split at the source-video level, report the number of source videos per split, and verify that no two documents from the same source video appear in different splits; if document-level splitting was used, re-run the experiments with a leakage-free split.","section":"Section III (Step 2 and Dataset Statistics; Table II)"},{"comment":"The sentence 'For entity chain extraction and relation extraction tasks, their entity (chain) pairs are both derived from golden entity (chain) pairs' means the reported Cha. and Rel. F1 scores are conditional on gold entities and chains, not end-to-end extraction performance. This is inconsistent with the joint task definitions in Eqs. (1)-(3), and it weakens the claim that Table III establishes a benchmark for the four tasks as defined. Please report end-to-end scores as well, and in any case disclose the conditional protocol in the table and text and discuss how each baseline is affected by the gold-pair assumption.","section":"Section VI.C (Implementation Details)"},{"comment":"The missing-modality setup is under-specified: 'Our main experiments will divide the train/dev/test sets into three equal parts, which are set to full modality, missing text modality, and missing visual modality.' Table III reports only a single average per task and language, with no per-condition breakdown, and the text does not state how each baseline encodes a missing modality (e.g., zeroing, blanking, or prompt construction). Because two-thirds of every split is corrupted by construction, the reported 'full modality' numbers are not full-modality numbers, and the comparison across systems depends on the unspecified corruption encoding. Please report condition-wise results and specify the missing-modality representation used for every model.","section":"Section VI.C (Implementation Details) and Table III"},{"comment":"The central artifact of the paper is the dataset, but the manuscript only says 'Our code will be released later' and gives no data URL, source-video identifiers, or release timeline. Without the data or at least a detailed datasheet and split metadata, the split-hygiene and evaluation-protocol concerns above cannot be checked by readers. For a dataset paper, releasing the dataset, along with annotation guidelines and source-video-level metadata, should be a condition of publication.","section":"Section I (Contributions) and general"}],"minor_comments":[{"comment":"The sentence 'establish a The model contains' is truncated and should be completed.","section":"Section VIII (Conclusion)"},{"comment":"Equation (14) contains a stray closing bracket: it reads 'ReLU(Convλ(Ĥgλ))]'.","section":"Equation (14)"},{"comment":"The horizontal axis of Figure 8 is labeled np, although the prompt length is denoted lp in Section V.E.","section":"Figure 8"},{"comment":"Reference [19] duplicates reference [15] (the same grounded multimodal NER paper appears twice); please deduplicate.","section":"References"},{"comment":"The text uses 'Blinear' where 'Bilinear' is intended.","section":"Equations (16)-(17)"},{"comment":"A single Cohen's Kappa of 78.18% is reported without specifying the annotation unit (document, mention, chain, or relation) or per-task agreement; please report per-task inter-annotator agreement.","section":"Section III (Step 6)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid dataset-and-benchmark paper conditional on data release. The split-level leakage risk is the central correctness issue; if the authors can document a source-video-level split and release the dataset, I would be satisfied. The journal should require a data availability statement with a working repository before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the dataset combination: video plus document-level text, English and Chinese, with NER, entity chains, relation extraction, and visual grounding all in one benchmark. Table I and the related work make the case that no existing resource does all of this, and the construction pipeline is described in enough detail that you could replicate the collection choices. The annotation specification is concrete, the kappa is reported, and the statistics are broken down per split. For a resource paper, that is the right kind of backbone.\n\nThe model is a secondary deliverable, and it is fine for a baseline: hierarchical fusion with a VAE-style denoising module and a missing-modality construction module, with ablations and an error analysis. The gains over MDocRE-HN are small (1.15% and 1.35% on the averages), and in NER the strongest baseline actually beats the proposed model. That is not disqualifying, but it means the empirical claim should be read as 'a usable benchmark standard' rather than 'a new state of the art.' Also, the implementation details state that entity chain and relation extraction use golden entity/chain pairs, so the RE and CR numbers are upper-bound-style evaluations, not end-to-end.\n\nThe load-bearing issue is the split construction. The paper splits long biographical videos into one-to-two-minute clips in Step 2, then says the annotated English and Chinese datasets are split 8:1:1. It never says the split is at the source-video level. Since adjacent clips from the same biography share frames, subtitles, and entity mentions, a document-level random split could leak near-duplicates into train and test. If that happened, every reported F1, including the small margin over MDocRE-HN, would be inflated. The dataset is not released and only 'code will be released later,' so there is no way to check. This is a correctable flaw, but it is exactly the kind of thing that determines whether the benchmark is trustworthy.\n\nThe reader's circularity score is right: training and evaluating on a new benchmark is standard, not circular. The stress-test concern is also on target, not overstated.\n\nWho is this for? Researchers working on multimodal IE, especially anyone who wants a video-grounded, bilingual, document-level testbed. It deserves a serious referee because the resource is novel and potentially reusable, but the verdict should be conditional: the authors need to release the dataset (or at least a documented subset), confirm the split is source-video-level, and rerun if any overlap exists. I would not cite it in my own work until the data and split verification are public.","headline":"The dataset is genuinely new and the construction is careful, but the benchmark numbers hinge on split hygiene that the paper never demonstrates, and the data are not released.","tokens_in":20870,"tokens_out":1753,"would_cite":false,"duration_ms":21038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper constructs M3D, a 4,093-document video-and-text dataset in English and Chinese annotated for entity recognition, coreference chains, relation extraction, and visual grounding, and reports benchmark F1 scores of 53.80 and 53.77.","keywords":["multimodal information extraction","document-level information extraction","visual grounding","coreference chains","relation extraction","multilingual dataset","video-text benchmark","missing modality reconstruction"],"falsifier":"Look at the released dataset's metadata for each clip's source video identifier and check whether clips from one source video appear in both training and test; alternatively, compute transcript n-gram overlap between splits and see whether any test document's text is nearly identical to a training document. If either check finds leakage, the reported numbers, including the advantage over MDocRE-HN, should be treated as inflated.","tokens_in":19932,"feed_emoji":"🎬","tokens_out":7453,"duration_ms":60237,"temperature":0.7,"pith_summary":"The paper constructs M3D, a dataset of 4,093 biography video-and-subtitle pairs in English and Chinese, annotated for four information extraction tasks: named entity recognition, coreference chain extraction, relation extraction, and visual grounding. It argues that existing multimodal IE datasets are mostly sentence-level, image-based, and English-only, whereas M3D offers document-level video-text grounding in two languages. To make the dataset usable as a benchmark, it also designs a hierarchical model that fuses denoised text and video features and that can reconstruct a missing modality; the model reaches average F1 of 53.80 on English and 53.77 on Chinese, slightly above the MDocRE-HN baseline. The intended contribution is a reusable resource and a reference standard for studying grounded, document-level multimodal IE.","feed_headline":"Video-based dataset benchmarks four text-and-vision IE tasks","feed_subtitle":"Sets baselines for entity, chain, relation, and grounding extraction from video plus subtitles.","key_machinery":"The central mechanism is a hierarchical cross-modal fusion pipeline. Encoders (Longformer for text, ViT for images) produce layer-wise features that are grouped into low, middle, and high levels; the DFFM passes each level through a VAE encoder, applies multi-head attention with the other modality's VAE-reduced features, and reconstructs the fused feature through the VAE decoder. The MMCM generates absent modality features by concatenating a randomly initialized prompt with the available modality's convolved features and reconstructing through a convolutional layer. For visual grounding, the YOLO-format bounding boxes are predicted by classification plus mean absolute error regression on the fused image features.","core_discovery":"On its own terms, the paper's central claim is that M3D is a valid multimodal, multilingual, multitask dataset and that its hierarchical model is a reliable benchmark, achieving average F1 53.80 on English and 53.77 on Chinese across the four tasks. The dataset contains two modalities (video frames and subtitle text), two languages (English and Chinese), four tasks, and a biography domain. The paper further claims that the Denoised Feature Fusion Module (DFFM), which uses a variational autoencoder to denoise low-, mid-, and high-level features before cross-modal attention, and the Missing Modality Construction Module (MMCM), which generates missing modality features from prompt and convolutional features, each contribute measurable gains in ablation experiments.","pith_inferences":["The paper does not state whether the 8:1:1 split was done on clips or on whole source videos; if a source video contributes clips to both training and test, the reported F1 numbers, including the margins over MDocRE-HN, could be inflated by near-duplicate footage and transcripts.","Because the multilingual component is two independently collected corpora rather than aligned translations, the dataset supports per-language benchmarking but not direct cross-lingual transfer experiments.","The visual grounding evaluation uses IoU > 0.5, a coarse localization standard; a stricter threshold might change the ranking of models on the grounding task."],"forward_implications":["Any model trained on M3D is evaluated on four tasks in the same document, so document-level cross-task interactions (e.g., chains feeding relations) can be studied jointly.","The reported baselines give later work concrete numbers to beat, including the observation that zero-shot large language models perform far below supervised baselines.","The ablation results imply that both denoised fusion and missing-modality construction are needed for the best robustness when video or text is absent.","The 8:1:1 splits and published annotation statistics support direct comparison of entity, chain, relation, and grounding performance in English and Chinese."],"supporting_citations":[{"why":"MDocRE-HN, the document-level multimodal relation extraction dataset and hierarchical network that serves as the primary baseline the benchmark must beat.","marker":"[18]"},{"why":"DocRED, the document-level relation extraction dataset whose entity-chain and relation annotation scheme M3D extends to the multimodal setting.","marker":"[23]"},{"why":"Grounded multimodal named entity recognition work supplying the visual grounding evaluation protocol (IoU threshold and type match).","marker":"[15]"},{"why":"Longformer, the pretrained text encoder used for both English and Chinese documents in the benchmark model.","marker":"[45]"},{"why":"ViT (visual transformer), the image encoder used to turn sampled video frames into patch embeddings.","marker":"[46]"},{"why":"Prompt-based missing modality construction work that directly inspires the MMCM module.","marker":"[44]"},{"why":"The average of MUC, B3 and CEAF metrics used to evaluate entity chain extraction.","marker":"[61]"},{"why":"YOLO format definition used for visual grounding bounding box representation.","marker":"[34]"}],"fun_headline_variants":["Four IE tasks benchmarked on bilingual video-text dataset","Dataset pairs video and text for four IE tasks in two languages","M3D: bilingual video-text benchmark for document-level IE","Video plus subtitles: new dataset for four IE tasks","English and Chinese video-text dataset for four extraction tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's reported scores assume that no two clips from the same original biography video end up in different splits, so near-duplicate footage and subtitles cannot leak between training and testing.","fun_headline_variants_meta":{"raw":{"variants":["Four IE tasks benchmarked on bilingual video-text dataset","Dataset pairs video and text for four IE tasks in two languages","M3D: bilingual video-text benchmark for document-level IE","Video plus subtitles: new dataset for four IE tasks","English and Chinese video-text dataset for four extraction tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001409,"raw_usage":{"total_tokens":5718,"prompt_tokens":993,"completion_tokens":4725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":4644}},"tokens_in":609,"tokens_out":4725,"duration_ms":28113,"temperature":1.0,"reasoning_tokens":4644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:50:17.348545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at the released dataset's metadata for each clip's source video identifier and check whether clips from one source video appear in both training and test; alternatively, compute transcript n-gram overlap between splits and see whether any test document's text is nearly identical to a training document. If either check finds leakage, the reported numbers, including the advantage over MDocRE-HN, should be treated as inflated.","supporting_citations":[{"cited_title":"A hierarchical network for multimodal document-level relation extraction,","cited_arxiv_id":null,"evidence_quote":"MDocRE-HN, the document-level multimodal relation extraction dataset and hierarchical network that serves as the primary baseline the benchmark must beat."},{"cited_title":"Docred: A large-scale document-level relation extraction dataset,","cited_arxiv_id":null,"evidence_quote":"DocRED, the document-level relation extraction dataset whose entity-chain and relation annotation scheme M3D extends to the multimodal setting."},{"cited_title":"Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,","cited_arxiv_id":null,"evidence_quote":"Prompt-based missing modality construction work that directly inspires the MMCM module."},{"cited_title":"Parallel data helps neural entity coreference resolution,","cited_arxiv_id":null,"evidence_quote":"The average of MUC, B3 and CEAF metrics used to evaluate entity chain extraction."}],"review_version":1}