{"id":"95e10ea4-9d4b-477a-bfed-66204bfa3009","arxiv_id":"2510.08543","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A cross-cultural video benchmark shows VideoLLMs detect US norms better than Chinese ones and struggle more with nonverbal than verbal evidence.","lead":"VIDEONORMS is a new benchmark of over 1,000 (video clip, norm) pairs from US and Chinese TV shows, annotated for cultural norm adherence/violation and verbal/nonverbal evidence. Testing seven open-weight video-language models, it finds they are markedly worse on Chinese norms and on nonverbal evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CN gold labels rest on three annotators who disagree substantially (κ=0.27–0.45) and edit Gemini candidates; the CN-vs-US gap may reflect label noise/anchoring rather than cultural ability.","rationale":"The reader's weakest assumption — the reliability of the three trained monocultural annotators' edited labels — is the same load-bearing concern I identify. The central claim depends entirely on these labels, and the Chinese subset shows low inter-annotator agreement plus high edit rates, with annotators starting from Gemini suggestions. A fresh de novo re-annotation is the direct test: if new labels match the current golds, the concern is resolved; if not, the reported findings are not trustworthy. I also note the abstract promises hierarchical linear modeling and a video-modality ablation that do not appear in the body; those are additional evidential gaps, but the label-reliability issue is more fundamental. I do not think the verdict should change from CONDITIONAL: the dataset and framework are potentially valuable, but the empirical conclusions should not be relied upon until label reliability and the missing analyses are addressed. This is a condition on acceptance, not a rejection.","tokens_in":22912,"tokens_out":4072,"duration_ms":40573,"concrete_test":"Re-annotate a stratified random sample of 100 Chinese clips with three new monocultural Chinese annotators who are blind to the Gemini outputs and to existing labels, using the same instruction/interface but with no candidate suggestions. Aggregate by majority vote and compute agreement with the current gold labels (exact adherence/violation flip rate and κ). If κ < 0.5 or >15% of labels flip, the CN gold labels are too unstable to support the reported CN-vs-US performance gap. As a secondary check, give the same clips to another set of annotators with the Gemini candidates shown to estimate the anchor effect; if de novo and edit-based labels diverge, the benchmark's labels carry teacher-model bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — open-weight VideoLLMs are worse on Chinese than US norm understanding — is evaluated against the human-edited labels in VIDEONORMS. For the Chinese subset these labels are not convincingly reliable. Fleiss κ on the adherence/violation decision is only 0.27–0.45 (Table 2); annotator field-change rates are 42–53%, with 64.5% of verbal-evidence fields and 53.9% of nonverbal-evidence fields edited (Table 9). In addition, the annotation task starts from Gemini candidate outputs (Section 3.3), so edits may be anchored to teacher suggestions rather than being independent judgments; two of the four Chinese shows were unfamiliar to the annotators (Best Partner: 0/3, Amazing Night: 1/3, Appendix B.2). Section 4.2 aggregates labels by majority vote, but with three annotators and low agreement the majority is fragile; the CN violation class is only 20% of cases (Table 3), so the poor CN violation F1 scores may be driven by unstable minority labels rather than a real model deficit. Because every model comparison in Table 4 uses these labels, the headline CN-vs-US gap and the violation-vs-adherence gap are confounded with label reliability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VideoNorms, a benchmark of 1,015 (video clip, norm) pairs drawn from four US and four Chinese TV shows. Candidate annotations are generated by Gemini 2.0 with a speech-act-theory prompt, then edited by three trained monocultural annotators per item; each annotation includes norm category, specific norm, adherence/violation, and verbal/nonverbal evidence. The paper defines three tasks: binary adherence/violation classification, classification with evidence extraction, and specific-norm generation, and evaluates seven open-weight VideoLLMs with 95% confidence intervals. The headline empirical claims are that models perform worse on Chinese than US norms, worse on norm violation than adherence, extract nonverbal evidence less reliably than verbal evidence, perform worse in formal contexts, and show no classification benefit from scale. The abstract also claims that ablation studies confirm the necessity of video modality and that hierarchical linear modeling (HLM) was used, although these analyses are not reported in the manuscript. Appendices provide prompts, hyperparameters, per-show breakdowns, and annotation instructions.","tokens_in":23201,"tokens_out":5676,"duration_ms":49633,"significance":"If the gold labels are trustworthy, VideoNorms fills a real gap: it is among the first benchmarks to evaluate cultural norm understanding in video, and the human-AI annotation pipeline with three annotators per item, reported inter-annotator agreement, confidence intervals, and detailed appendices is a useful methodological contribution. The per-show and per-category breakdowns are valuable for diagnosing model failure modes. However, the benchmark's central empirical claims rest on a Chinese gold standard whose reliability is not established: Chinese inter-annotator agreement is low (κ=0.27–0.45), annotator edit rates on Chinese shows are high (42–53%), and two of the four Chinese shows were unfamiliar to the annotators. In addition, several headline claims (video-modality necessity, hierarchical linear modeling, scale invariance) are not supported by analyses present in the manuscript. The resource is potentially valuable, but the current version overclaims and requires substantial additional validation.","major_comments":[{"comment":"The Chinese gold standard is not established. Fleiss's κ for the adherence/violation decision is 0.27–0.45 for all four Chinese shows, and combined annotator change rates are 42–53%. Field-level edit rates are even higher: 64.5% for verbal evidence and 53.9% for nonverbal evidence (Table 9). Appendix B.2 reports that none of the three Chinese annotators had watched Best Partner and only one had watched Amazing Night. Because Section 4.2 aggregates by majority vote among three annotators, and because all model scores in Table 4 are computed against these labels, the reported Chinese-vs-US gap and the violation-vs-adherence gap may be artifacts of label noise or teacher-model anchoring rather than genuine model deficits. Please report per-annotator consistency, stratify by annotator familiarity, and include a de-anchoring check (e.g., a fresh annotation of a random subset without Gemini ca","section":"§3.3, Table 2, Table 9, Appendix B.2"},{"comment":"The abstract claims that 'ablation studies confirm video modality is indeed necessary' and that 'hierarchical linear modeling analysis' was used. Neither appears in the manuscript: there is no ablation study (e.g., audio-only, transcript-only, or image-only conditions), and no HLM model, equations, or results are presented in Sections 4–5 or the appendices. The reported analyses are descriptive F1 comparisons with confidence intervals. These claims should be removed or the analyses must be added and described in enough detail to be evaluated.","section":"Abstract, §4.4, §5"},{"comment":"The statement that 'scaling model size does not yield classification score improvements' is not supported by the experiments. All seven models are in the 7B–8B parameter range (Appendix C.2), so no controlled scale comparison is possible. The observation that models 'perform close to each other' confounds architecture, training data, and scale. Please either remove the scale claim or add a scale-controlled comparison (e.g., multiple sizes from the same model family).","section":"§4.3, Appendix C.2"},{"comment":"The LLM-as-judge protocol for evidence quality is validated on only 20 instances, with 2 of 20 reported as not fully adhering to the rubric; no inter-judge agreement or correlation with human scores is reported. Since the verbal-vs-nonverbal evidence gap is a headline finding, this validation is too thin to rule out judge-side bias. Please report agreement on a larger validation sample and, if possible, compare against human judgments on a random subset.","section":"§4.1, Task 2"}],"minor_comments":[{"comment":"The Chinese translations for several norm categories are inconsistent with the English labels: 'Granting request' is paired with 拒绝请求 (rejecting a request), 'Rejecting request' is paired with 商务谈判 (business negotiation), 'Apology' with 表示感谢 (expressing thanks), and 'Thanks' with 邀请 (invitation). Please align the translations.","section":"Table 3"},{"comment":"Typo: 'pormpt' should be 'prompt'.","section":"§3.2"},{"comment":"The abstract states 'over 3,000 human judgments'; the relationship between this number and the 1,015 (clip, norm) pairs and the 724/1,113 Task 1 cases in Table 3 should be clarified.","section":"Abstract"},{"comment":"The conclusion that models perform worse in formal contexts is based on per-show comparisons in Tables 12–13, but no statistical test is provided for the difference between workplace and informal shows; a simple mixed-effects model or paired test would strengthen the claim.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful benchmark idea and a transparent annotation pipeline, but the current version's central empirical findings are undermined by the low reliability of the Chinese gold labels and by several unsupported claims (ablation, HLM, scale). The paper can be made publishable if the authors either add the missing analyses, remove the overclaims, and demonstrate that the CN-vs-US and violation-vs-adherence gaps survive when label reliability is controlled. I would not reject outright, but the revisions are substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gets one big thing right: it delivers the first benchmark aimed at cultural norm adherence/violation in video, with paired US/Chinese shows and verbal/nonverbal evidence. The construction is transparent—teacher model candidates from Gemini, then three monocultural annotators per instance, with edit rates and inter-annotator agreement reported. Appendices include full prompts, hyperparameters, and per-show breakdowns, which is more than most benchmark papers bother to do. That resource is worth having, even if only as a diagnostic testbed.\n\nThe soft spots are real, and they cluster around the empirical claims rather than the dataset itself. The CN labels are fragile: Fleiss κ of 0.27–0.45 is low, annotator field-edit rates are 42–53%, and two of the four Chinese shows were unfamiliar to the annotators. Since every model comparison uses those labels, the CN-vs-US gap and the violation-vs-adherence gap are partly confounded with label reliability and teacher anchoring. The paper acknowledges teacher-model bias but never measures de-anchoring. I'd want to see either a subset of labels created from scratch by annotators, or at least a sensitivity analysis that treats the Gemini candidates as a baseline and asks how much the human edits actually change model rankings.\n\nThe abstract also promises two things the body does not deliver: a hierarchical linear modeling analysis and an ablation confirming video modality is necessary. Neither appears. And the claim that scaling does not help is unsupported, since all models are in the 7B–8B range—there is no scaling curve. The \"unlike humans\" formal-context finding also compares annotator agreement to model F1, which is apples to oranges.\n\nNone of this kills the dataset contribution, but it means the headline findings should be read as hypotheses, not results. The paper deserves a serious referee: the benchmark is novel and potentially influential, and the issues are fixable in revision. I'd suggest accepting the dataset, but asking the authors to either add the missing analyses or drop the claims, and to address CN label reliability head-on.","headline":"A genuinely new cross-cultural video norm benchmark with real human-verification effort, but the CN gold labels' low agreement and teacher anchoring, plus absent promised analyses, make the headline cultural-gap findings conditional.","tokens_in":23688,"tokens_out":1964,"would_cite":true,"duration_ms":19239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-weight video language models understand US cultural norms better than Chinese norms, extract verbal evidence more reliably than nonverbal evidence, and do not improve with scale, according to a new video benchmark built with human-AI c","keywords":["cultural norms","video language models","benchmark","cross-cultural evaluation","nonverbal evidence","norm adherence","speech act theory","TV shows"],"falsifier":"Re-annotate the Chinese subset from scratch without showing annotators the teacher model's candidate labels, using a larger pool that is familiar with all four shows; if the US/CN gap shrinks or disappears under the new gold labels, the measured cultural asymmetry was partly an artifact of label quality rather than model behavior. Alternatively, restrict evaluation to clips with unanimous annotator agreement and check whether the gap persists.","tokens_in":22824,"feed_emoji":"🎬","tokens_out":3788,"duration_ms":32080,"temperature":0.7,"pith_summary":"The paper introduces VideoNorms, a benchmark of over 1,000 short video clips from US and Chinese TV shows annotated for whether a stated social norm is adhered to or violated, with verbal and nonverbal evidence. Its central claim is that current open-weight VideoLLMs are culturally lopsided: they perform worse on Chinese norms than US norms, especially at flagging violations in Chinese contexts, and they struggle more to cite nonverbal than verbal cues. The paper also finds that larger models do not automatically score better, and that formal workplace settings are harder for models even though human annotators agree most there. If right, these results imply that cultural competence in video models cannot be assumed from scale or multilingual pretraining and needs targeted, culturally grounded evaluation and training.","feed_headline":"Video AI is worse at Chinese norms than US norms","feed_subtitle":"A 1,000-clip benchmark shows seven open-weight models also miss nonverbal cues and gain little from scale.","key_machinery":"The carrying mechanism is the VideoNorms construction pipeline: a teacher VideoLLM prompted with speech act theory generates candidate norm categories, adherence/violation labels, and verbal/nonverbal evidence from 15-second clips; three trained monocultural annotators from each culture then edit or confirm each field. The resulting human-edited labels define three evaluation tasks: binary adherence/violation classification, evidence extraction, and open norm generation. The speech-act-based category list (thanks, apology, greeting, requesting information, etc.) is what turns raw video into comparable norm judgments across the two cultures.","core_discovery":"The paper's central discovery is a consistent cultural asymmetry in seven open-weight VideoLLMs evaluated under identical protocols: on the VideoNorms benchmark, every model performed worse on Chinese norms than on US norms on the adherence/violation task (with gaps as large as 19.2 F1 points for one model family), and models found it harder to ground violations than adherences in Chinese clips. Models also produced weaker evidence for nonverbal cues than for verbal cues, and their evidence and norm-generation scores fell below 3 on a 5-point rubric in most cases. The authors attribute the pattern to over-alignment with Western, Educated, Industrialized, Rich, and Democratic (WEIRD) cultural","pith_inferences":["Beyond the paper: the Chinese label set may be noisier than the US set—inter-annotator agreement is low (κ 0.27–0.45) and annotator edits are frequent—so part of the reported US/CN gap could reflect gold-label difficulty rather than model cultural incompetence. A re-annotation study from scratch would separate these.","Beyond the paper: because Chinese annotators were unfamiliar with two of the four Chinese shows (none had watched Best Partner; only one had seen Amazing Night), the benchmark partly measures models against norms that even the human experts were learning on the fly; pooling more show-familiar annotators is a natural robustness test.","Beyond the paper: the finding that formal contexts are harder for models could be converted into a training signal: curating workplace and hierarchy-heavy video norm data and testing whether finetuning on it closes the gap without hurting informal-context scores.","Beyond the paper: the verbal/nonverbal evidence gap suggests a concrete diagnostic—probe models with muted or transcript-only inputs to measure how much of adherence/violation accuracy comes from speech versus visual social cues."],"forward_implications":["Cultural norm evaluation for video should separate adherence from violation, because models systematically underperform on violation detection, especially for Chinese norms.","Nonverbal grounding is a distinct bottleneck: even when labels are right, models cannot cite gaze, gesture, posture, or tone as evidence as well as they quote speech.","Scaling model size, at least across the 7B-8B open-weight models tested, does not reliably improve cultural norm classification.","Formal workplace contexts are a specific weakness for models, even though human raters agree most there, suggesting models lack the hierarchical and social sensitivity such settings demand.","Fully automatic norm annotation is unreliable for cultures underrepresented in training data; human verification changes a large fraction of Chinese annotations."],"fun_headline_variants":["Video AI worse on Chinese than US cultural norms","VideoLLMs miss nonverbal cues, lag on Chinese norms","Benchmark reveals video AI's cultural bias in norms","Seven open video models show Chinese norm gap vs US","Video AI underperforms on Chinese cultural norms"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's conclusions rest on the assumption that the three annotators' edited labels are a trustworthy gold standard for each clip—especially for Chinese clips, where annotator agreement is low (κ 0.27–0.45), edit rates are high (42–53%), and two shows were unfamiliar to the annotators.","fun_headline_variants_meta":{"raw":{"variants":["Video AI worse on Chinese than US cultural norms","VideoLLMs miss nonverbal cues, lag on Chinese norms","Benchmark reveals video AI's cultural bias in norms","Seven open video models show Chinese norm gap vs US","Video AI underperforms on Chinese cultural norms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1535,"prompt_tokens":747,"completion_tokens":788,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":491,"tokens_out":788,"duration_ms":7227,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:42:07.993551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the Chinese subset from scratch without showing annotators the teacher model's candidate labels, using a larger pool that is familiar with all four shows; if the US/CN gap shrinks or disappears under the new gold labels, the measured cultural asymmetry was partly an artifact of label quality rather than model behavior. Alternatively, restrict evaluation to clips with unanimous annotator agreement and check whether the gap persists.","supporting_citations":[],"review_version":1}