{"id":"fa1cb872-da1b-43f5-9270-fa330f149953","arxiv_id":"2504.14321","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new Chinese multimodal coreference dataset from Douyin social-media dialogues annotates person mentions in comments and links them to head regions in videos, with benchmark results from a text coreference and visual alignment pipeline.","lead":"This paper introduces TikTalkCoref, a new Chinese dataset that pairs short videos from the Douyin platform with comment dialogues, where people named in the comments are linked to the corresponding faces in the video. It also runs a benchmark pipeline built from existing text coreference and face-tracking tools, reporting baseline numbers for this new task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Alignment benchmark relies on unvalidated automatic visual clusters as ground truth; R@K scores may not measure coreference against the manual gold standard.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing risk: the alignment benchmark's visual side is generated automatically and never validated against the manually annotated head regions. I agree with that assessment. The paper's strongest contribution is the dataset itself, and the annotation process described in Section 3.3 is plausible: two independent annotators, an expert adjudicator, an MUC-based inter-annotator agreement of 78.19, and a label-studio interface for simultaneously marking textual clusters and head boxes. These are real strengths. However, the benchmark results in Tables 5 and 7 are presented as reliable, and their reliability depends on the automatic visual clusters being faithful to the manual gold clusters. The paper never checks this. If the automatic pipeline merges or splits identities, the contrastive training pairs and the retrieval targets are corrupted, so the reported R@K numbers do not establish cross-modal coreference performance. Since the data and code are not released, this cannot be checked post hoc by reviewers. This is not a reason to reject the dataset claim, but it is a concrete condition on the benchmark claim. The reader already assigned CONDITIONAL, so the verdict should remain unchanged; the condition should explicitly include releasing data plus a validation of automatic visual clusters against manual head regions.","tokens_in":13007,"tokens_out":3680,"duration_ms":37013,"concrete_test":"On the dev and test splits, compute cluster-level agreement between the Section 4.2 automatic visual clusters and the Section 3.3 manual head-region clusters for the same videos, using pairwise precision/recall/F1 or B^3. If agreement is below a pre-specified threshold (e.g., pairwise F1 < 0.9), rerun the Section 5.2 alignment evaluation with manual head boxes and manual cluster IDs as candidates and targets, and compare R@1/R@2/R@3 with Table 5. A material shift would indicate that the reported benchmark scores conflate automatic clustering quality with multimodal coreference quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is in Sections 4.2-4.3: visual person identities are produced by the automatic pipeline (YOLOv5, DeepSORT, MTCNN, MobileFaceNet, cosine threshold 0.6), and these automatic clusters are used both to build the candidate image set I and to define positive pairs for contrastive fine-tuning of the alignment module. Tables 5 and 7 then report R@K against this automatically constructed visual side. The dataset annotation in Section 3.3 created manual head boxes and cross-modal cluster links using the same cluster numbers, but the paper never reports any agreement between the automatic visual clusters and these manual clusters. If the automatic clustering merges distinct people, splits one person across multiple clusters, or misassigns an identity - a real risk for stitched short videos with pose and segment changes - then the positive image paired with a text cluster can be wrong, and R@K measures retrieval among tracker identities rather than coreference with human gold. The claim of 'reliable benchmark results' therefore depends on an unvalidated premise, and the missing validation cannot be recovered from the released text because no data or code is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TikTalkCoref, a Chinese multimodal coreference resolution dataset built from 1,012 Douyin short-video dialogues sampled from the TikTalk dataset. The dataset provides manual annotations of person mentions and coreference clusters in the comment text, as well as head-region bounding boxes in video frames that are cross-modally linked to the textual clusters. The authors also propose a three-module benchmark pipeline: (1) Maverick, a DeBERTa-based textual coreference resolver; (2) an automatic visual person tracking module combining YOLOv5, DeepSORT, MTCNN, and MobileFaceNet with a 0.6 cosine-similarity threshold; and (3) a CN-CLIP-based contrastive text-visual alignment module. Experiments report a text coreference average F1 of 65.46 for Maverick and a fine-tuned CN-CLIP R@1 of 60.83 for text-visual alignment on a celebrity subset, along with analyses of data augmentation and mention-type-specific retrieval accuracy. The authors state that the dataset will be released.","tokens_in":13270,"tokens_out":5083,"duration_ms":49193,"significance":"If the benchmark results are validated, TikTalkCoref is a valuable resource: it fills a concrete gap as the first Chinese multimodal coreference dataset for real-world social media dialogues, and its annotation process is described with sufficient detail to be replicable, including an independent double-annotation workflow with a reported MUC agreement of 78.19. The pipeline is modular and comparably simple, and the paper provides honest limitations about dataset scale and domain. However, the benchmark claim currently rests on an unvalidated automatic visual clustering step, so the reported alignment numbers cannot yet be interpreted as measuring coreference against a human gold standard.","major_comments":[{"comment":"The visual side of the alignment benchmark is constructed entirely by the automatic pipeline (YOLOv5 head detection, DeepSORT tracking, MTCNN/MobileFaceNet face clustering with a hand-set cosine threshold of 0.6), and the paper never reports any agreement between these automatic person clusters and the manually annotated head regions described in §3.3. If the automatic clustering merges distinct people, splits one person into multiple clusters, or misassigns an identity—a real risk for stitched short videos with pose and segment changes—then the positive pairs used in contrastive learning and the candidate image set used for R@K evaluation are not aligned with the human gold standard. Consequently, Tables 5 and 7 may measure retrieval among tracker identities rather than cross-modal coreference. The authors should report visual clustering quality against the manual clusters (e.g., B³ or cluster purity on the head regions) and, ideally, also report alignment results using the manual visual clusters as the candidate set to isolate the contribution of the tracking module.","section":"§4.2–4.3, Tables 5 and 7"},{"comment":"The description of contrastive training for the alignment module is ambiguous: it is not specified whether the 'matching pairs of textual cluster Ck and their corresponding person head regions Ij' are derived from the manual cross-modal annotations (which link text clusters to manually drawn head boxes) or from the automatically generated visual clusters. When the visual tracking module produces incorrect clusters, the two choices lead to different models: one trained to align text to the human gold standard, the other trained to align text to tracker identities. This distinction is central to interpreting the fine-tuning gains in Table 5, and the paper must state precisely how positive and negative pairs are constructed in training.","section":"§4.3, Eq. (1) and L_align"},{"comment":"The central claim of the paper is an introduced dataset plus reliable benchmark results, yet no dataset URL, GitHub link, or code is actually provided in the manuscript (the text merely says 'at github'). For a resource and benchmark paper, independent verification of the statistics, the IAA, and the experimental results requires access to the data and the evaluation scripts, especially given the unresolved question of visual cluster validity. The authors should provide an anonymized or permanent link to the dataset and code in the revised manuscript.","section":"§1, last paragraph; §6"},{"comment":"The fixed 0.6 cosine-similarity threshold for grouping face trajectories is a free parameter that directly controls the visual cluster quality and therefore the alignment benchmark. No sensitivity analysis is reported for this threshold or for the choice of the representative image per cluster in §4.3. Since the paper claims reliable benchmark results, a brief threshold-sweep or an explicit justification of the 0.6 value is needed to show that the reported R@K numbers are not an artifact of this specific hand-set value.","section":"§4.2, face recognition and clustering paragraph"}],"minor_comments":[{"comment":"There are several typos: 'mutimodal' in the abstract, 'beter' in §5.2, 'TicktalkCoref' in §1, and 'base on' in §5.4. The paper should be proofread for these and for inconsistent capitalization of 'TikTalkCoref'.","section":"Abstract and throughout"},{"comment":"The paper reports p < 0.001 for comparisons between Maverick and e2e-coref and between R2D2 and CN-CLIP, but it runs experiments with only three random seeds. The randomized parsing evaluation comparator (Noreen, 1989) is not the standard tool for comparing the mean of three runs of a neural model, and the reported p-value is suspiciously small for three seeds. Please clarify the exact test procedure and report confidence intervals or the per-seed values.","section":"§5.1, significance testing"},{"comment":"Only the MUC score is reported for inter-annotator agreement. Since the annotation task involves both mention boundaries and cluster membership, it would be more informative to also report mention-level agreement (e.g., F1 on mention spans) or B³ agreement between annotators, so that the quality of the two annotation subtasks can be assessed separately.","section":"§3.4, Inter-Annotator Agreement"},{"comment":"The symbols for clusters in the visual tracking module are not consistently defined: the paper refers to 'head regions representing the same person are grouped into a same cluster' but does not define a formal cluster variable (e.g., V_j). Aligning the notation with that used in §4.3 for the candidate image set I would improve readability.","section":"§4.2, notation"},{"comment":"In the dataset comparison table, the 'Mention Type' column for TikTalkCoref is listed as 'PNs, CNs and PRs', but the descriptions in §3.2 mention proper names, common nouns, and pronouns; using the same terms in the table would avoid the unexplained abbreviations PNs/CNs/PRs.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is potentially strong and fills a real gap, but the benchmark claim is currently undermined by the unvalidated automatic visual clustering. I would like to see, in the revision, a direct evaluation of the visual clusters against the manual head-region annotations and a clarification of how the alignment training pairs are generated. The missing data/code link is also a concern for a resource paper. If the authors provide the validation and the data, I would be willing to reconsider for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: TikTalkCoref is a genuinely useful resource. It fills a concrete gap: no public Chinese multimodal coreference dataset exists for real social media dialogues, and this one pairs Douyin comment threads with manual person clusters and head-region boxes. The annotation workflow looks careful — double annotation with an expert adjudicator, MUC 78.19 agreement, and sensible guidelines for nested mentions and singletons. The statistics are internally consistent and the comparison with existing MCR datasets is accurate.\n\nWhere I land on the stress-test concern: it holds up on reading the paper. The alignment benchmark in Sections 4.2 and 4.3 uses visual person clusters produced by the YOLOv5/DeepSORT/MTCNN/MobileFaceNet pipeline with a cosine threshold of 0.6. Those automatic clusters are used both to pick candidate images for retrieval and to define positive pairs for contrastive fine-tuning. But the paper never reports agreement between these automatic clusters and the manually annotated head regions. For stitched social media videos, identity switches and over-merging are real risks, so the R@K scores in Tables 5 and 7 may measure retrieval against tracker identities rather than against the human gold standard. That is a load-bearing gap, not a cosmetic one.\n\nThe dataset itself stands independently of that flaw. If the authors release the data and code, as they say they will, the resource should be valuable despite the benchmark issue. The paper is also honest about its limitations: small scale, single platform, gender imbalance, and the fact that the method is an assembly of existing components. The novelty is the resource, not the approach.\n\nMinor aggravations: no data or code is currently available, so the annotation quality cannot be verified externally. And the abstract's phrase \"reliable benchmark results\" overstates what is shown.\n\nThis paper deserves a serious referee. The right referee will ask for automatic-vs-manual cluster agreement on a sample, for the dataset release, and for relabeling the benchmark numbers as provisional until that validation is done. The fix is straightforward and the core resource is worth engaging with.","headline":"TikTalkCoref is a real dataset contribution, but the alignment benchmark depends on unvalidated automatic visual clusters, so treat the R@K numbers as provisional rather than reliable.","tokens_in":13761,"tokens_out":2055,"would_cite":false,"duration_ms":20893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The authors introduce TikTalkCoref, an annotated Chinese multimodal coreference dataset drawn from Douyin short-video dialogues, and benchmark a text-plus-vision pipeline on it.","keywords":["multimodal coreference resolution","Chinese social media dialogues","TikTalkCoref dataset","cross-modal person alignment","text-video retrieval","contrastive learning","coreference annotation"],"falsifier":"Re-run the alignment evaluation with the manual head-region boxes substituted for the automatic visual person clusters; if the R@K scores move by a large margin, the automatic clustering is doing the work and the benchmark is not measuring coreference against the gold-standard annotations.","tokens_in":12817,"feed_emoji":"🎬","tokens_out":8898,"duration_ms":68337,"temperature":0.7,"pith_summary":"The paper claims that multimodal coreference resolution for real social media dialogue can be studied with a new resource, TikTalkCoref, which pairs Douyin short videos with user-comment dialogues and manually annotates which textual mentions point to which person's head regions in the video. It argues this is the first Chinese multimodal coreference dataset built from real-world short-video dialogue, and it supplies a benchmark pipeline plus results: textual coreference at 65.46 average F1 and fine-tuned text-to-video retrieval at 60.83 R@1. If the resource is sound, it gives researchers a way to measure how well models link comments to the people in the video they discuss.","feed_headline":"TikTalkCoref links people in short videos to names in comments","feed_subtitle":"A new resource pairs Douyin video with comments and labels who is mentioned, benchmarked at 65.46 text F1 and 60.83 retrieval R@1.","key_machinery":"The cross-modal coreference cluster is the central object: a textual cluster of mentions that all refer to one person, paired with head-region images of that same person from the video. To build these pairs, the paper's pipeline uses Maverick with a DeBERTa encoder to detect mention starts and ends and to cluster antecedents, YOLOv5-based head detection with DeepSORT tracking plus MTCNN and MobileFaceNet face recognition to group head regions into identity clusters, and CN-CLIP contrastive learning that retrieves the visual cluster for each textual cluster using a temperature-scaled cross-entropy loss. The mechanism turns coreference resolution into a retrieval problem: if the text cluster and the visual cluster lie close in the shared embedding space, the pair is accepted as cross-modal coreference.","core_discovery":"The paper's central claim is that TikTalkCoref is the first Chinese multimodal coreference dataset for social media in real-world scenarios, with manual annotations that cover both sides of the reference relation: textual mentions of people clustered into entities, and head-region boxes in the associated video frames for the same people. The dataset holds 1,012 dialogues from the Douyin platform, 519.65 minutes of video, 2,179 mentions, 1,435 textual clusters, and 958 bounding boxes, with 338 celebrity-focused dialogues set aside as a sub-dataset. The benchmark on the celebrity subset combines Maverick, a pipeline-style textual coreference model, with automated head detection, tracking and face clustering to form visual person clusters, and CN-CLIP, a Chinese contrastive image-text model, to align textual clusters to head-region images. Reported results are 65.46 average F1 for textual coreference and 60.83 R@1 for fine-tuned cross-modal alignment, with zero-shot and data-augmentation variants analysed.","pith_inferences":["If the automatic visual clusters are accurate enough, the benchmark implies that contrastive retrieval is a viable route to person-level grounding in social media, and the same approach could be extended to objects, locations, or brands mentioned in comments.","The annotation scheme's 'person not found' and 'face not identifiable' categories, described in Appendix A.3, could support a more honest evaluation that includes unanswerable alignment cases; the reported R@K numbers currently only score videos where a person was expected to appear.","A natural test of the benchmark's validity is to recompute alignment scores using the manually drawn head boxes instead of the automatic identity clusters; how much the numbers move would separate cross-modal alignment ability from visual clustering ability.","Because pronoun-central clusters align most easily in the fine-tuning results, a harder variant of the task might down-weight pronoun clusters and focus on noun- and name-central references, which are the ones that need visual grounding."],"forward_implications":["MCR on Chinese social media becomes a fixed benchmark with train, dev, and test splits, so future systems can be compared on the same real-world dialogues rather than on scripted or descriptive data.","The celebrity sub-dataset gives a clean evaluation setup: references to celebrities in comments can be verified against the video's visible faces.","The data-augmentation result implies that non-celebrity dialogues improve textual coreference on celebrity conversations, suggesting that broadening training data to generic person references transfers across domains.","Fine-tuned CN-CLIP outperforms R2D2 on R@1 (60.83 vs 56.25) and Mean (71.81 vs 70.00), suggesting that contrastive fine-tuning with negative images drawn from the same video is an effective alignment strategy for this task.","R2D2's better zero-shot and noun-central performance indicates strong pretraining transfer, so the choice between zero-shot retrieval and fine-tuning will matter for deployment scenarios."],"supporting_citations":[{"why":"Supplies the source TikTalk dataset of Douyin video-comment dialogues from which TikTalkCoref is sampled.","marker":"Lin et al. (2023)"},{"why":"Provides the Maverick textual coreference model used as the benchmark's text module.","marker":"Martinelli et al. (2024)"},{"why":"Supplies the coarse-to-fine mention-antecedent clustering used inside Maverick and the e2e-coref baseline.","marker":"Lee et al. (2018)"},{"why":"Provides DeepSORT tracking that links detected head regions across frames.","marker":"Wojke et al. (2017)"},{"why":"Supplies MTCNN for face detection before feature extraction from head regions.","marker":"Zhang et al. (2016)"},{"why":"Provides MobileFaceNet embeddings used to cluster head trajectories into identity groups.","marker":"Chen et al. (2018b)"},{"why":"Supplies Chinese CLIP, the contrastive image-text model used for cross-modal alignment.","marker":"Yang et al. (2022)"},{"why":"Provides the contrastive loss formulation and the CLIP architecture that alignment fine-tuning builds on.","marker":"Radford et al. (2021)"},{"why":"Supplies R2D2, the comparison model for zero-shot and fine-tuned text-to-image retrieval.","marker":"Xie et al. (2023)"}],"fun_headline_variants":["First Chinese multimodal coreference dataset for social dialogues","Douyin comments and faces matched in new coreference benchmark","TikTalkCoref pairs video faces with text mentions on Douyin","New dataset and benchmark for Chinese multimodal coreference","Linking people in Douyin videos to comment names: benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic person clusters built in Section 4.2 match the manually annotated head regions; the paper does not verify this, so if the automatic identity clusters are inaccurate, the R@K alignment scores in Section 4.3 are not measuring coreference against the human gold standard.","fun_headline_variants_meta":{"raw":{"variants":["First Chinese multimodal coreference dataset for social dialogues","Douyin comments and faces matched in new coreference benchmark","TikTalkCoref pairs video faces with text mentions on Douyin","New dataset and benchmark for Chinese multimodal coreference","Linking people in Douyin videos to comment names: benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2956,"prompt_tokens":974,"completion_tokens":1982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1901}},"tokens_in":590,"tokens_out":1982,"duration_ms":11899,"temperature":1.0,"reasoning_tokens":1901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:51:06.983436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the alignment evaluation with the manual head-region boxes substituted for the automatic visual person clusters; if the R@K scores move by a large margin, the automatic clustering is doing the work and the benchmark is not measuring coreference against the gold-standard annotations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Maverick textual coreference model used as the benchmark's text module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the coarse-to-fine mention-antecedent clustering used inside Maverick and the e2e-coref baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MTCNN for face detection before feature extraction from head regions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies R2D2, the comparison model for zero-shot and fine-tuned text-to-image retrieval."}],"review_version":1}