{"id":"ac2208ad-aa56-4c1f-8652-70b0ce6a611b","arxiv_id":"2506.14854","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A YOLO-based key-frame selection method for retail videos lowers human annotation effort by auto-annotating high-confidence frames and interpolating the rest.","lead":"This paper uses a YOLO object detection model to pick which frames of a retail video need human annotation, auto-labeling the rest. The authors report annotators only need to check a few percent of frames, saving up to half of annotation cost while keeping bounding box quality near an IOU of 0.5.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 4 refutes the headline claim: 316/330 videos with IOU>0.5 is contradicted by 155/330 in Table 4, so the central '95% automatic annotations' claim rests on a conflation of two different counts.","rationale":"The reader's verdict is REJECT, and my stress-test finds the same conclusion but through a sharper route: the paper's own tables contradict the central claim. The strongest claim in Section 4.3.3 says 316 out of 330 videos had IOU exceeding 0.5, but Table 4 reports only 155 videos above IOU 0.5 for YOLOv5x. The 316 number comes from Table 2, which counts videos with detections above the probability threshold, not videos whose final interpolated annotations exceed the IOU threshold. This is an internal inconsistency, not merely a missing external validation, so it is more load-bearing than the reader's stated weakest assumption about unmeasured interpolation quality. The reader's rationale did mention the 316-vs-155 contradiction, which is why my agreement is 'partial' rather than 'disagree'. Because the reader already recommended REJECT and my concern only strengthens that recommendation, no verdict adjustment is needed; UNCHANGED is the appropriate final outcome. A single concrete check, recomputing the number of videos with IOU > 0.5 from the raw per-video data, would settle whether the headline should be 47% rather than 95%, or whether Table 4 is itself wrong. Either way, the paper as written cannot support the abstract's claim of human-comparable quality with 95% of videos automatically annotated and less than 5% requiring human verification.","tokens_in":11349,"tokens_out":3942,"duration_ms":36140,"concrete_test":"Recompute from the raw per-video IOU outputs (or, if unreleased, from Table 4) the number of videos with mean IOU > 0.5 for YOLOv5x at th1=0.5. If the count is 155 rather than 316, then the sentence in Section 4.3.3 should read 'approximately 47% of videos (155/330) had IOU>0.5', and the abstract's '95%' claim must be revised to '95% of videos had at least one high-confidence frame requiring no human verification', which is a materially weaker claim. If a recomputation instead yields at least 316, then Table 4 or the IOU definition is erroneous and must be corrected before the headline can be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3.3 asserts: 'Our proposed KFG method delivered automatic annotations for approximately 95% of retail videos (316 out of 330) with an IOU metric exceeding 0.5, completely eliminating the need for human verification in these cases.' The same section later states that 'approximately 46.96% of retail videos (155 out of 330) can be automatically annotated with an IOU exceeding 0.5' for YOLOv5x, and Table 4 confirms only 155 (YOLOv5x) and 156 (YOLOv8x) videos have IOU > 0.5. The 316 figure in Table 2 is the number of videos with at least one detection above the probability threshold th1=0.5, not the number with full-video IOU above 0.5. The central claim therefore conflates 'videos containing high-confidence detections' with 'videos whose automated annotations meet the quality threshold.' This is not a missing-validation issue: the paper's own reported numbers contradict its headline. The abstract and conclusions repeat the 95% and less-than-5%-human-verification claims, so the central result is unsupported by the evidence as reported. The later 155/330 figure still indicates a potentially useful annotation-assistance tool, but it does not support 'automating annotation for the remaining frames without reducing annotation quality' or '2 times cost savings' as claimed. Since the single most load-bearing number is refuted internally, the central claim cannot be accepted without corrected and clarified reporting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Key Frame Generation (KFG) method for retail video annotation: run off-the-shelf YOLOv5x/YOLOv8x detectors on video frames, divide detections by confidence thresholds into auto-accepted, human-verified, and interpolated frames, and compare the resulting annotations with human ground truth using mean IOU. The evaluation is performed on 330 retail videos containing at least one person. The paper claims that KFG automatically annotates about 95% of videos with IOU exceeding 0.5, requires human verification on less than 5% of frames, achieves accuracy comparable to human annotators, and yields about 2x cost savings.","tokens_in":11647,"tokens_out":4065,"duration_ms":44784,"significance":"If the headline claims were supported, the paper would describe a valuable practical tool for reducing retail video annotation cost. The proposed pipeline is simple and easy to deploy, and the authors report experiments on a substantial commercial dataset, including a concrete cost calculation in Appendix A. However, the central quantitative claim is internally contradicted by the paper's own Table 4, and the evaluation lacks a human-human baseline, error bars, and validation of interpolated boxes. With the corrected 155-out-of-330 figure, the contribution reduces to an annotation-assistance tool with a modest mean IOU of 0.51, which may still be useful but does not support the paper's stated claims of eliminating human verification or doubling cost savings.","major_comments":[{"comment":"The claim that \"Our proposed KFG method delivered automatic annotations for approximately 95% of retail videos (316 out of 330) with an IOU metric exceeding 0.5\" is directly contradicted by Table 4, which reports only 155 videos (YOLOv5x) and 156 videos (YOLOv8x) with IOU above 0.5. The 316 figure in Table 2 is the number of videos with at least one detection above the confidence threshold th1=0.5, not the number of videos whose full-video annotations satisfy the IOU quality threshold. The abstract, Table 3, and the conclusion repeat the 95% and less-than-5%-human-verification claims, so the central reported result is unsupported by the paper's own data and must be corrected.","section":"Section 4.3.3, Table 4"},{"comment":"The claim of \"accuracy comparable to human annotator labeling\" is not established because no human-human IOU baseline is measured. The reported mean IOU of 0.51 for YOLOv5x is modest, and without knowing the human-human agreement on the same videos, the comparison to human annotation quality is not interpretable. Additionally, Section 5 concludes that \"less than 5% of frames\" require human annotation, but Table 3 lists KFG annotated frames as 17.95% and human verification as 4.25% of videos, not frames; this conflation of frames and videos is a load-bearing error in the cost-savings argument.","section":"Section 4.2, Section 4.3.3"},{"comment":"The cost-savings claims are internally inconsistent. Table 3 reports that only 4.25% of videos need human annotations, which would suggest roughly 20x savings, while the text later states that 46.96% of videos (155/330) can be automatically annotated with IOU exceeding 0.5, \"translating to annotation cost savings of at least 2×.\" These two numbers cannot both describe the same operating point, and no explicit cost model connecting the IOU threshold to annotation cost is provided. The paper should clearly state the relationship between the confidence threshold, the IOU threshold, the fraction of auto-annotated videos, and the claimed cost savings.","section":"Section 4.3.3, Table 3"},{"comment":"The quality of automatically annotated frames is not directly measured. Table 2 counts frames with detections above a confidence threshold, but Table 4 reports IOU on full-video annotations \"including interpolation\" (Figure 6 caption). The paper never validates the interpolated bounding boxes against human labels, nor does it report per-frame YOLO accuracy against human annotations. Since the central claim is that the remaining frames can be automated without reducing annotation quality, the evaluation must include a per-frame or segment-level comparison and should report interpolation accuracy separately from detection accuracy.","section":"Section 3, Figure 6"}],"minor_comments":[{"comment":"The phrase \"achieving accuracy comparable to human annotator labeling\" should be replaced with the measured IOU values, since no human-human baseline is provided.","section":"Abstract"},{"comment":"The FFMPEG and K-means baselines do not produce bounding boxes, so their \"cost savings\" numbers are not directly comparable to KFG, which does produce boxes; this should be stated explicitly when comparing cost savings.","section":"Section 4.3.1, Section 4.3.2"},{"comment":"The test video example shows that increasing the detection threshold to 0.8 reduces the number of detected frames to 4 and lowers mean IOU to 0.29, which illustrates the threshold trade-off but is only briefly discussed in the main text; a fuller discussion would help the reader interpret Table 2.","section":"Appendix Table 5"},{"comment":"The dataset description reports 935 videos, but the experiments use 330 videos containing at least one person; the filtering criteria and the number of excluded videos should be stated explicitly.","section":"Section 4.1"},{"comment":"The column \"Human annotated frames (%)\" is listed as NA for KFG rows, but this number is needed to support the claim about the fraction of frames requiring human verification.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"The paper reads like an industry technical report and would benefit from a clear statement of what is new relative to standard detection-plus-threshold pipelines. The internal contradiction between the 95% claim and Table 4 is severe enough that I cannot recommend acceptance even under a revision-friendly policy, because correcting the claim would reduce the contribution to an annotation-assistance tool and would require new experiments (human-human baseline, per-frame validation, interpolation quality) that are not currently present. I also note that no code or data is released, which limits reproducibility for a systems-oriented contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is a dead end: the paper's central claim—that KFG auto-annotates 95% of retail videos (316/330) with IOU>0.5—is directly contradicted by its own Table 4, which counts only 155/330 (YOLOv5x) and 156/330 (YOLOv8x) at that threshold. The 316 number is the count of videos with at least one detection above the confidence threshold th1=0.5, not the number of videos whose full annotations clear the IOU bar. The abstract and conclusions repeat the 95% figure, so the main result as written is wrong.\n\nThat said, the paper has a real core: a sensible annotation workflow—run YOLOv5x/YOLOv8x, route high-confidence detections straight through, send medium-confidence frames to human verification, interpolate the rest. For retail person bounding boxes, that's a reasonable co-pilot design, and the authors are admirably specific about the trade-off between detection threshold and frame recall. Running this on 935 retail videos (330 with person annotations) is real work, and the comparison against FFMPEG and K-means clustering is fair in spirit.\n\nThe soft spots are serious. Beyond the headline conflation, there is no human-human IOU, so 'comparable to human annotator' is unsupported; the reported mean IOU of 0.51 is modest. The 2× cost savings is inferred from frame counts, not measured end-to-end throughput or cost. The interpolation step is unvalidated. No error bars or significance tests. Dataset and code are not available, so external checking is limited to what the tables reveal—which is enough to falsify the current claim.\n\nThe method is not novel, but the bigger issue is evidential. A corrected version, reporting the distribution of IOU across videos and honestly framing KFG as an annotation-assistance tool rather than full automation, could be a useful practice note for retail teams. As it stands, the central claim does not hold up.\n\nI'd send this to a serious referee rather than desk-reject, because the contradiction is fixable and the dataset is substantial. The referee will ask for the corrected numbers and a human baseline. If the authors can provide those, there's a useful paper in here; if not, it should not be published.","headline":"The central '95% auto-annotation' claim is internally contradicted by the paper's own IOU table; a serious referee would catch it, but the underlying workflow is worth a corrected look.","tokens_in":12176,"tokens_out":2533,"would_cite":false,"duration_ms":23267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Off-the-shelf YOLO detection can auto-annotate retail video, leaving under 5% of footage to humans.","keywords":["retail video annotation","key frame generation","object detection","YOLO","annotation cost reduction","bounding box interpolation","customer-product interaction","video labeling"],"falsifier":"Take the 316 videos the KFG method auto-annotated at threshold 0.5, have human annotators label every frame, and compute mean IOU per frame between the KFG/interpolated boxes and those dense labels. If mean per-frame IOU on the auto-annotated videos drops below 0.5, or if interpolated frames show systematic errors that the high-confidence key-frames do not, the claim that these videos need no human verification is refuted.","tokens_in":11118,"feed_emoji":"🛒","tokens_out":8298,"duration_ms":79945,"temperature":0.7,"pith_summary":"The paper sets out to show that retail video annotation can be automated by using object-detection confidence to decide which frames matter. Its key-frame generation (KFG) pipeline runs a YOLO (You Only Look Once) detector on every frame, treats high-confidence detections as final annotations, sends mid-confidence frames to a human for verification, and fills low-confidence frames by interpolation. On 330 retail videos containing at least one customer, the paper reports that about 95% of videos (316 of 330) were auto-annotated with intersection-over-union above 0.5 and no human verification, leaving roughly 4-5% of videos for traditional labeling. The payoff it claims is at least a 2-fold annotation cost saving, with a potential 20-fold saving before quality checks, which matters because manual retail annotation is expensive and sensitive to arbitrary frame selection.","feed_headline":"Key-frame AI auto-annotates 95% of retail videos in test set","feed_subtitle":"Retailers only verify under 5% of frames; bounding-box quality stays above IOU 0.5 for most auto-annotated videos.","key_machinery":"The load-bearing machinery is the KFG thresholding pipeline. A YOLOv5x or YOLOv8x detector is run over every frame, and each person detection is bucketed by confidence: high-confidence frames (e.g., $p_{\\mathrm{det}} > 0.5$) become final annotations, mid-confidence frames go to a human for verify-and-edit, and low-confidence frames are skipped and later filled by linear interpolation between neighboring key-frame boxes. Mean intersection-over-union (IOU) against human-annotated ground truth is the evaluation metric, and the fraction of videos whose high-confidence frames cover the whole clip is the source of the cost-saving figures.","core_discovery":"The central claim is that confidence-based frame selection turns an off-the-shelf detector into a complete annotation pipeline. With the detection threshold set at 0.5, YOLOv5x automatically annotated 316 of 330 retail videos with IOU exceeding 0.5, and only 14 videos still required the full human-annotation workflow; YOLOv8x left 16 videos requiring human attention. The paper also reports that at threshold 0.8, 284 of 330 videos needed no human verification, and that the per-video IOU distribution at threshold 0.5 puts 155 videos above IOU 0.5. In the authors' reading, these numbers establish that KFG meets human-level annotation quality while cutting cost and average annotation time.","pith_inferences":["The headline 95% automation counts videos, not frames: the paper's own Table 2 shows median auto-detected frame rates around 18%, so most frames in an 'automated' video are interpolation fill-ins; per-frame quality is the unmeasured quantity that really decides whether the cost savings survive.","A natural calibration rule follows from the paper's threshold trade-off: sweep $0.5$ to $0.8$ per store or per time-of-day and pick the threshold that maximizes mean IOU per annotation dollar, which is a testable extension the paper does not run.","Transferring the same confidence-threshold machinery to product or interaction labeling would require a detector with comparable precision for those classes; the paper only demonstrates person bounding boxes, so the generality claim is unsupported.","Validation on an open retail-like benchmark rather than a commercial dataset with a single annotation standard would clarify how much of the 95% figure is method versus dataset-specific."],"forward_implications":["At threshold 0.5, only 14 of 330 retail videos (YOLOv5x) or 16 (YOLOv8x) need the traditional full annotation process, giving a roughly 20-fold cost reduction before quality checks.","Raising the detection threshold reduces the number of frames that need human review, but the paper's test-video table shows IOU can drop to 0.29 when sparse detection leaves only 4 frames to interpolate across a 335-frame video.","The interpolation step lets the generated boxes feed downstream retail analytics, such as shopper journey analysis and product interaction detection, without per-frame human labels.","KFG can be combined with embedding-and-clustering key-frame selection, with clustering identifying behavioral phases and detector boxes supplying the object-level annotations those phases need."],"supporting_citations":[{"why":"Supplies the YOLOv5/YOLOv8 detectors whose confidence scores drive the KFG threshold bands.","marker":"[35]"},{"why":"Provides the IOU metric and key-frame detection framing used to compare KFG output against human ground truth.","marker":"[20]"},{"why":"Defines the deep semantic feature video summarization approach used as the clustering baseline for key-frame selection.","marker":"[5]"},{"why":"Introduces active-learning video annotation, the prior automated-annotation context KFG extends.","marker":"[16]"},{"why":"Provides a fast trajectory annotation baseline that also targets reducing human labeling effort in video.","marker":"[17]"}],"fun_headline_variants":["Key-frame AI auto-labels 95% of retail videos","Retail video AI cuts annotation cost by half","Key-frame detector matches human labeling quality","Retail video AI: 95% auto-annotated, 5% verified"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that high-confidence YOLO detections can be used as final annotations and that linear interpolation between them preserves quality; the paper provides no per-frame validation of interpolated boxes or detector accuracy against human labels, so the quality of the auto-annotated frames is unmeasured.","fun_headline_variants_meta":{"raw":{"variants":["Key-frame AI auto-labels 95% of retail videos","Retail video AI cuts annotation cost by half","Key-frame detector matches human labeling quality","Retail video AI: 95% auto-annotated, 5% verified"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1250,"prompt_tokens":938,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":554,"tokens_out":312,"duration_ms":3974,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:28.459292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 316 videos the KFG method auto-annotated at threshold 0.5, have human annotators label every frame, and compute mean IOU per frame between the KFG/interpolated boxes and those dense labels. If mean per-frame IOU on the auto-annotated videos drops below 0.5, or if interpolated frames show systematic errors that the high-confidence key-frames do not, the claim that these videos need no human verification is refuted.","supporting_citations":[{"cited_title":"Ultralytics/YOLOv5: v5.0","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv5/YOLOv8 detectors whose confidence scores drive the KFG threshold bands."},{"cited_title":"Deep learning approach to key frame detection in human action videos.Recent Trends in Computational Intelligence, 1:1–17, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the IOU metric and key-frame detection framing used to compare KFG output against human ground truth."},{"cited_title":"Video summarization using deep semantic features","cited_arxiv_id":null,"evidence_quote":"Defines the deep semantic feature video summarization approach used as the clustering baseline for key-frame selection."},{"cited_title":"Video annotation and tracking with active learning.Advances in Neural Information Processing Systems, 24, 2011","cited_arxiv_id":null,"evidence_quote":"Introduces active-learning video annotation, the prior automated-annotation context KFG extends."},{"cited_title":"Pathtrack: Fast trajectory annotation with path supervision","cited_arxiv_id":null,"evidence_quote":"Provides a fast trajectory annotation baseline that also targets reducing human labeling effort in video."}],"review_version":1}