{"id":"c3fce16c-fa61-42f2-bcb5-a72e9c5ec752","arxiv_id":"2607.06468","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"EgoPolice introduces a 185-hour annotated police body-worn camera benchmark showing state-of-the-art video models fail on high-stakes actions due to motion, occlusion, and low inter-class visual separability.","lead":"EgoPolice is a new benchmark dataset of 185 hours of real police body-worn camera footage, annotated second-by-second for nine action types like 'Weapon Out' and 'Handcuffing.' It reveals that current video models, including Gemini 2.5 Pro, struggle on high-stakes egocentric video where motion blur, low light, and visual similarity between classes defeat standard shortcuts.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The real-world deployment claim (§6) is the weakest load-bearing point: it is stated in the abstract and listed as a contribution, but backed only by qualitative assertion with no quantitative results.","rationale":"The reader correctly identified the most load-bearing concern: the Section 6 deployment claim is prominently stated (abstract, contribution list) but unsupported by any quantitative evidence in the paper. This is the right concern to flag, and CONDITIONAL is the appropriate verdict — the benchmark contribution is solid, but the paper makes a transferability claim it does not back up. I considered whether a more fundamental concern exists with the benchmark itself (e.g., label noise from the per-second annotation protocol in §A.1, or the MCQ 5-way format potentially inflating VLM scores relative to the classification task). The per-second protocol does introduce boundary false negatives, but the paper is transparent about this tradeoff and provides multi-window evaluations (1s/10s/1min) that partially mitigate it. The MCQ format is a legitimate zero-shot evaluation paradigm, and the paper acknowledges format-error confounds for specific models (Table 17). Neither of these rises to a load-bearing concern for the central benchmark claim. The dataset construction is rigorous (multi-stage pipeline, objective definitions, 79.4% IAA, 25% manual verification), the domain-gap analysis is quantitatively grounded (optical flow, CLIP TVD), and the benchmark results clearly demonstrate model limitations. The deployment claim is the one place where the paper overreaches relative to its evidence, and the reader caught it precisely.","tokens_in":32641,"tokens_out":3633,"duration_ms":270992,"concrete_test":"Run the deployed model on a held-out subset of the uncurated BWC repository where ground-truth labels exist (or can be obtained via annotator review), and report recall, precision, and false-negative rate for 'BWC Wearer-Physical Interaction.' Compare against two baselines: (1) random sampling of the same number of segments, and (2) uniform temporal sampling. If the model-assisted approach does not achieve recall at least 2x higher than random sampling at the same precision, the transferability claim is substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper lists 'Demonstrating real-world transferability' as one of three primary contributions in §1 and states in the abstract that EgoPolice 'provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories.' Section 6 describes an ongoing deployment on a single action class ('BWC Wearer-Physical Interaction') in a human-in-the-loop setting, but provides no quantitative results: no recall, no precision, no false-negative rate, no comparison to a baseline without model assistance (e.g., random sampling or keyword filtering), and no measure of reviewer time savings. The section explicitly defers evidence: 'A full analysis of this deployment will be presented in a forthcoming companion paper.' This means the transferability claim — a stated contribution of the paper — is currently an assertion without supporting evidence in the paper itself. The benchmark contribution (dataset + evaluation) stands independently and is well-constructed, but the deployment claim is load-bearing for the paper's framing as more than a benchmark, and it is unsupported. The reader correctly identified this as the central weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces EgoPolice, a dataset of approximately 185 hours of real police body-worn camera (BWC) footage annotated with nine action classes at second-by-second granularity. The dataset is sourced from multiple U.S. police departments, with the primary data coming from Chicago's Civilian Office of Police Accountability (COPA). The annotation pipeline uses a two-stage process with objective, intent-free definitions, achieving a mean Krippendorff's alpha of 79.4%. The paper benchmarks both supervised linear-probing classifiers (using frozen features from CLIP, DINOv2, VideoMAE V2, etc.) and zero-shot video-language models (e.g., Gemini 2.5 Pro, GPT-4.1) on classification and multiple-choice question-answering tasks. Results show that even the best models struggle, particularly on high-stakes actions like 'Weapon Out' and 'Handcuffing.' The paper also includes a preliminary discussion of a real-world deployment in Section 6.","tokens_in":33482,"tokens_out":1185,"duration_ms":2016649,"significance":"The dataset fills a genuine gap: there is no existing public benchmark for video understanding in police BWC footage, despite commercial tools already being deployed in this domain. The annotation pipeline is well-designed, with objective definitions, case-level splits to prevent leakage, and 25% manual verification. The optical flow and CLIP-similarity analyses (Figure 3) quantitatively demonstrate that EgoPolice is harder than standard datasets due to severe camera motion and low inter-class visual separability (TVD 0.150 vs. 0.573 for Kinetics). The benchmarking is thorough, covering 6-fold cross-validation with OOD-time and OOD-location splits, multiple clip durations, and a broad set of open- and closed-source VLMs. The per-class breakdowns and failure mode analysis (Figure 4) are informative. The annotator management section, including vicarious traumatization mitigation, is a valuable contribution to best practices for high-stakes data collection.","major_comments":[{"comment":"§6 and Contributions (§1): The paper lists 'Demonstrating real-world transferability' as one of three primary contributions and the abstract states EgoPolice 'provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories.' However, Section 6 describes an ongoing deployment on a single action class ('BWC Wearer-Physical Interaction') with no quantitative results: no recall, precision, false-negative rate, baseline comparison, or measure of reviewer time savings. The section explicitly defers evidence to 'a forthcoming companion paper.' This makes the transferability claim a stated contribution that is currently unsupported by evidence in the manuscript. The benchmark contribution (dataset + evaluation) stands independently and is well-constructed. The authors should either (a) remove the deployment claim from the贡献","section":null}],"minor_comments":[{"comment":"Reference [6] (Attia et al.) appears to be about liver transplantation prioritization, not AI failures in high-stakes settings. Please verify.","section":null},{"comment":"§5.2 states '12,000 questions, with 500 questions per action ... on 1-second and 10-second-long clips and 200 questions per action on 1-minute-long clips.' With 10 actions (including 'None of the above'), this yields 500×10×2 + 200×10 = 12,000. Please clarify the counting in the text for readers.","section":null},{"comment":"Figure 3: The legend in panels (a-b) lists datasets including 'EPIC-KITCHENS' but the text does not discuss it. Consider adding a brief mention or adjusting the legend.","section":null},{"comment":"Table 4: The 'Random Baseline' description in §D.1 says it 'always predicts 1' with 'Recall is 100.' This is a reasonable baseline but the F1 values in Table 4 vary across splits (e.g., 7.9 for ID vs. 13.9 for OT at 1s). A footnote explaining why the random baseline F1 differs across splits (due to different class prevalence) would help readers.","section":null},{"comment":"§3.2: The 10-second buffer around Stage 1 interaction windows is mentioned without justification. A brief note on why 10 seconds was chosen would strengthen the methodology.","section":null},{"comment":"§D.11: The MCQ prompt example shows options A–E but the per-class accuracy table (Table 18) lists 9 action classes plus 'None of the above.' It would help to clarify how the 5-option MCQ maps to the 9-class taxonomy (i.e., one correct answer and four random distractors from the other 8 classes + 'None of the above').","section":null},{"comment":"The paper uses 'EgoPolice' and 'EgoPolice (Ours)' in figures. Consider defining the abbreviation 'EP' if used, or consistently use the full name.","section":null},{"comment":"§7: The ethical discussion is thoughtful but brief. Given that the dataset includes footage of civilians who died in police custody, a sentence on whether any faces were redacted or whether there are plans for controlled access would strengthen this section.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The deployment claim in §6 is the main concern. The benchmark itself is solid and would be acceptable on its own. The authors should either tone down the contribution language to match the evidence (i.e., frame §6 as a preliminary qualitative discussion, not a contribution) or provide at least minimal quantitative results. I lean toward minor revision because the core benchmark contribution is sound and the fix is straightforward (adjusting framing or adding a table). The dataset and code release appear to be planned via the project website, which is appropriate."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. We address the major comment below.","responses":[{"response":"The referee is correct. As the manuscript currently stands, Section 6 describes an ongoing deployment but provides no quantitative evidence—no precision, recall, false-negative rate, baseline comparison, or reviewer time-savings analysis. Listing 'Demonstrating real-world transferability' as a primary contribution in Section 1 and making the corresponding claim in the abstract overstates what the paper actually demonstrates. We agree that the benchmark contribution (dataset + evaluation) stands on its own and does not require the deployment claim to be compelling. In the revised manuscript, we will implement option (a): we will remove 'Demonstrating real-world transferability' from the list of primary contributions in Section 1 and will revise the abstract to remove the sentence 'Beyond serving as a benchmark, EgoPolice provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories, enabling more efficient downstream human review.' Section 6 will be retained but repositioned as a preliminary discussion of ongoing deployment work rather than a contribution claim, with the framing made explicit that no quantitative deployment results are presented in this paper and that a full evaluation is deferred to a forthcoming companion paper. We believe this accurately reflects the manuscript's actual contributions while preserving the value of the deployment discussion as motivation for future work.","revision_made":"yes","referee_comment":"§6 and Contributions (§1): The paper lists 'Demonstrating real-world transferability' as one of three primary contributions and the abstract states EgoPolice 'provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories.' However, Section 6 describes an ongoing deployment on a single action class ('BWC Wearer-Physical Interaction') with no quantitative results: no recall, precision, false-negative rate, baseline comparison, or measure of reviewer time savings. The section explicitly defers evidence to 'a forthcoming companion paper.' This makes the transferability claim a stated contribution that is currently unsupported by evidence in the manuscript. The benchmark contribution (dataset + evaluation) stands independently and is well-constructed. The authors should either (a) remove the deployment claim from the ["}],"tokens_in":32258,"tokens_out":745,"duration_ms":45898,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"EgoPolice fills a real gap. No prior public dataset provides second-by-second multi-label action annotations on real police body-worn camera footage, and the related work section confirms this convincingly. The annotation pipeline is solid: two-stage process, objective intent-free definitions, mean Krippendorff's alpha of 79.4%, 25% manual verification, case-level splits to prevent leakage. The domain-gap analysis is the paper's analytical highlight — the optical flow comparison showing heavy-tailed motion during action segments (vs. Ego4D's dropoff) and the CLIP TVD of 0.150 vs. 0.573 for Kinetics both quantitatively establish why this domain is hard. The benchmarking is thorough: linear probing across five embedding models with 6-fold CV and two OOD splits, plus zero-shot MCQ across 11 VLMs. The finding that Gemini 2.5 Pro plateaus at 76.9% MCQ and VideoMAE V2 reaches only ~48.5 mF1 on 1-second clips gives the community a concrete challenge target. The per-class breakdown and failure mode analysis (Figure 4) are useful and honest. The annotator management section, including vicarious traumatization safeguards, is a genuine contribution to methodology in sensitive-domain data collection and is rarely discussed this carefully in CV papers. Credit is earned here. The soft spot is Section 6. The paper lists 'demonstrating real-world transferability' as a primary contribution and the abstract claims EgoPolice 'provides a foundation for developing models capable of identifying events of interest in large-scale body-worn camera video repositories.' But Section 6 describes a deployment on a single action class with no quantitative results — no recall, no precision, no baseline comparison, no measure of reviewer time savings. It explicitly defers to a forthcoming companion paper. This is an assertion, not evidence. The fix is straightforward: either cut Section 6 and the corresponding contribution claim, or reframe it as future work. The benchmark stands on its own without it. Two minor issues: the MCQ format-error confound for LLaVA-Mini and GPT-4.1-Nano is acknowledged and handled reasonably via separate parsing-error reporting, so it doesn't undermine the main comparisons. The 9-class taxonomy is limited but the paper is transparent about why rarer classes were dropped. The dataset bias toward firearms-related incidents (COPA overrepresents critical incidents) is acknowledged in the limitations. None of these are load-bearing problems. This paper is for video understanding researchers who work on egocentric or action recognition benchmarks, and for computational social scientists interested in BWC analysis. It deserves a serious referee. The core contribution is real and well-executed; the deployment framing is the one thing that needs fixing before publication.","headline":"EgoPolice is a genuinely novel benchmark — first public BWC dataset with second-level action labels — with rigorous annotation and useful domain-gap analysis. The deployment claim in Section 6 is unsupported and should be cut or softened.","tokens_in":33334,"tokens_out":1077,"would_cite":true,"duration_ms":92576,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Even the best AI can't reliably flag weapons in police body-cam footage","keywords":[],"falsifier":"If a model trained on EgoPolice could not outperform a random or simple heuristic baseline at surfacing action segments in an uncurated BWC repository, or if the low inter-class TVD were an artifact of annotation noise rather than genuine visual similarity between classes, the paper's claim that BWC footage is qualitatively harder and that EgoPolice captures that difficulty would be undermined.","tokens_in":32899,"feed_emoji":"","tokens_out":893,"duration_ms":126805,"temperature":0.7,"pith_summary":"This paper introduces EgoPolice, the first curated dataset of real police body-worn camera (BWC) footage designed to test how well video-understanding models can identify critical police-civilian actions. The dataset spans roughly 185 hours of footage from multiple U.S. police departments, annotated second-by-second across nine action categories such as Weapon Out, Physical Interaction, and Handcuffing. The authors use EgoPolice to benchmark both trained classification models (via linear probing on frozen features) and zero-shot video-language models (via multiple-choice question answering), finding that even the strongest models fail to reach the reliability needed for autonomous deployment. The paper argues that BWC footage is fundamentally harder than standard video benchmarks because the most semantically important moments coincide with severe camera motion, and because different action classes are visually nearly indistinguishable using global appearance cues alone.","feed_headline":"Even the best AI can't reliably flag weapons in police body-cam footage","feed_subtitle":"A new 185-hour benchmark of real police encounters shows models confuse flashlights for guns and red shirts for injuries, with the top model","key_machinery":"The paper's argument rests on three constructed objects: (1) the EgoPolice dataset itself, built through a two-stage annotation pipeline with objective, intent-free action definitions and a mean inter-annotator agreement (Krippendorff's alpha) of 79.4%; (2) two evaluation tasks, a linear-probing classification protocol using frozen features from models like VideoMAE V2, CLIP, and DINOv2, and a zero-shot multiple-choice question-answering protocol for video-language models; and (3) two quantitative diagnostic measures that explain why the domain is hard, optical flow magnitude during action segments and CLIP-based inter-class TVD. Together these establish both the benchmark and the structural","core_discovery":"The central finding is that police body-worn camera footage poses a qualitatively different challenge from existing egocentric video datasets, and current state-of-the-art models are not adequate for it. The paper quantifies this in two ways. First, optical flow analysis shows that during annotated action segments, EgoPolice exhibits a heavy-tailed distribution of camera motion, unlike Ego4D or EPIC-KITCHENS where motion drops during actions. Second, CLIP embedding analysis reveals that inter-class visual separability in EgoPolice is extremely low, with a Total Variation Distance of 0.150 between same-class and different-class frame-pair similarity distributions, compared to 0.573 for Kin-50","pith_inferences":[],"forward_implications":["Commercial vendors selling BWC analysis tools can now be stress-tested against a public, independently curated benchmark, exposing whether their systems can distinguish a weapon from a flashlight or a red shirt from an injury.","The low inter-class TVD finding suggests that models relying on scene-level appearance shortcuts, which work well on Kinetics or ActivityNet, will systematically fail on BWC footage, motivating architectures that prioritize fine-grained temporal and motion reasoning.","The per-second annotation granularity and case-level data splits enable future work on temporal action localization and cross-jurisdictional generalization in a way that clip-level datasets cannot.","The human-in-the-loop deployment sketch, though preliminary, outlines a concrete path where model predictions surface candidate segments for human review rather than replacing human judgment, which may be the only viable deployment paradigm given current error rates."],"fun_headline_variants":["Top video models fail to reliably flag weapons in police body-cam footage","Police body-cam footage exposes blind spots in state-of-the-art AI models","New police body-cam benchmark shows AI struggles with high-risk actions","Rapid camera motion in police body-cams defeats current video AI models","AI models confuse flashlights for guns in new police body-cam benchmark"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper claims EgoPolice can serve as a foundation for scalable police oversight tools, but the only evidence for real-world transferability is a brief description of a preliminary deployment on a single action class with no quantitative results, no false-negative analysis, and no comparison to a baseline without model assistance.","fun_headline_variants_meta":{"raw":{"variants":["Top video models fail to reliably flag weapons in police body-cam footage","Police body-cam footage exposes blind spots in state-of-the-art AI models","New police body-cam benchmark shows AI struggles with high-risk actions","Rapid camera motion in police body-cams defeats current video AI models","AI models confuse flashlights for guns in new police body-cam benchmark","Even Gemini 2.5 Pro fails to reliably spot weapons in police body-cam video"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1217,"prompt_tokens":508,"completion_tokens":709,"prompt_tokens_details":null},"tokens_in":508,"tokens_out":709,"duration_ms":48735,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T04:40:23.135345+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a model trained on EgoPolice could not outperform a random or simple heuristic baseline at surfacing action segments in an uncurated BWC repository, or if the low inter-class TVD were an artifact of annotation noise rather than genuine visual similarity between classes, the paper's claim that BWC footage is qualitatively harder and that EgoPolice captures that difficulty would be undermined.","supporting_citations":[],"review_version":1}