{"id":"74534974-09d3-4ac6-a8b8-d06afb6f7676","arxiv_id":"2506.21903","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A YOLOv11 detector fine-tuned with auto-labeled lecture frames reaches about 95% AP50 on a new visual object detection benchmark for educational videos.","lead":"This paper compares six object detection models on lecture video frames and finds that YOLOv11 performs best, then boosts accuracy by training on automatically labeled frames. It also releases a new benchmark dataset of 4,000 lecture video frames, 1,000 manually annotated, to support future work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Enrichment gain is not independent: the auto-labeling teacher trains on a fixed 80% of LVVO_1k that overlaps four of the five 5-fold test sets, so Table II's 6.59-point AP50 gain can reflect test-fold leakage.","rationale":"The reader's conditional verdict is well founded. The fixed 80/20 split for the auto-labeler combined with 5-fold CV on the same 1,000 images is a genuine evaluation leak. The paper never states that the auto-labeler is retrained per fold or that LVVO_3k pseudo-labels are generated with a teacher excluding each fold's test set. The overlap analysis makes the mechanism concrete: with 800 train, 200 validation, and five 200-image folds, at least four folds have test images present in the auto-labeler's training set. The enrichment gain could therefore partly reflect the teacher having memorized the test images' visual distribution and annotation style, rather than the added pseudo-labeled frames contributing independent information. I agree with the reader's diagnosis. The proposed fold-aware re-run would settle it. The dataset release and general-model experiments are not affected by this specific flaw, so a conditional verdict with a required re-evaluation remains appropriate rather than an outright reject. If the wording 'entire extended dataset' is read literally as including all 1,000 manual labels in training, the leak would be even more direct, but the teacher-overlap argument is unambiguous regardless of that ambiguity.","tokens_in":8643,"tokens_out":8616,"duration_ms":95815,"concrete_test":"Using the released code and LVVO_1k, re-run the Section V-C enrichment protocol with fold-aware auto-labeling: for each of the five folds, train the YOLOv11 teacher on only that fold's 800 training images, generate pseudo-labels for LVVO_3k, then train the Progressive Fine-Tuned model and evaluate on the fold's 200 held-out images. If the mean AP50 gain over Baseline drops materially (e.g., below 2 percentage points), the 6.59-point gain in Table II is inflated by test-fold contamination; if the gain persists, the enrichment claim survives this specific check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption in Section V-C is that the LVVO_3k pseudo-labels used for enrichment are independent of the LVVO_1k images on which the enriched models are evaluated. This fails. The auto-labeling teacher is fine-tuned on a fixed 80% of LVVO_1k, and the enrichment experiments are then evaluated with standard 5-fold cross-validation on the same LVVO_1k set. Since the teacher's 800-image training set and the five 200-image test folds partition the same 1,000 images, at least four of the five folds necessarily overlap the teacher's training images (expected overlap is roughly 160 of 200 images per random 80/20 split). The teacher's weights therefore encode the very test frames used in Table II. The pseudo-labels themselves are on different images, but they are produced by a model whose decision boundary has been fitted to the held-out fold images and their annotation style. Training the Progressive model on those pseudo-labels and reporting AP50 rising from 90.75% to 95.32% on the same folds measures consistency with the teacher's already-seen data, not generalization to new lecture video content. This does not invalidate the dataset release or the cross-dataset comparisons, but it invalidates the headline enrichment claim as an independent measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses detection of visual objects (charts, tables, illustrations) in educational lecture videos. It introduces the LVVO dataset (4,000 frames, of which 1,000 are manually annotated), evaluates six COCO-pretrained object detectors on lecture video datasets, identifies YOLOv11 as the best performer, explores cross-dataset and joint-training strategies, and proposes a semi-supervised auto-labeling pipeline that enriches the training set with 3,000 automatically labeled LVVO frames. The authors report that progressive fine-tuning with the enriched dataset raises AP50 from 90.75% to 95.32% and AP from 77.6% to 84.19% on the LVVO_1k benchmark.","tokens_in":8913,"tokens_out":2760,"duration_ms":28025,"significance":"If the enrichment result were independent, the paper would offer a valuable practical recipe for leveraging unlabeled lecture video frames and a new benchmark (LVVO) for the community. The systematic comparison of multiple detectors on three datasets and the release of the dataset and code are concrete contributions. However, the central claim of dataset enrichment is compromised by an evaluation-leakage problem in Section V-C: the auto-labeling teacher is trained on 80% of LVVO_1k and the enriched models are then evaluated by 5-fold cross-validation on the same LVVO_1k, so the teacher has already seen the test-fold images and their annotation style. The reported gains therefore reflect consistency with the teacher's training data rather than genuine generalization to new lecture content. The cross-dataset and joint-training experiments are less affected and retain value, but the headline enrichment claim needs to be re-established under a leakage-free protocol.","major_comments":[{"comment":"The dataset enrichment evaluation has a structural leakage problem. The auto-labeling teacher is fine-tuned on an 80% split of LVVO_1k (Section V-C) and then used to generate pseudo-labels for LVVO_3k. The resulting enriched models are evaluated using 5-fold cross-validation on the same LVVO_1k set. Because the teacher's 800-image training set overlaps with four of the five 200-image test folds (expected overlap roughly 160 images per fold), the teacher's weights encode the appearance and annotation style of the validation folds. Training the Progressive or Comprehensive model on pseudo-labels produced by this teacher and then measuring AP50 on those same folds does not provide an independent estimate of the enrichment benefit. The 6.59-point AP50 gain reported in Table II is therefore not trustworthy as a measure of generalization to new lecture video content. The authors should redo the enrichment experiment with a held-out split that is disjoint from the teacher's training set, or with a teacher trained only on the per-fold training splits when generating pseudo-labels for enrichment, and report the resulting performance.","section":"Section V-C, Table II"},{"comment":"The pseudo-label quality is not assessed at all. The manuscript applies a confidence threshold of 0.5 to filter auto-labeled detections but provides no statistics on the resulting LVVO_3k labels: no distribution of detections per image, no agreement with manual labels on a subset, and no sensitivity analysis of the threshold. Since the whole enrichment claim rests on the assumption that these pseudo-labels are accurate enough to serve as training annotations, the paper should at least report a small-scale manual evaluation of the auto-labeled boxes or a comparison of models trained with different confidence thresholds. Without such evidence, the reader cannot separate the effect of added data from the effect of label noise.","section":"Section V-C, Figure 7"},{"comment":"The evaluation protocols are inconsistent across the paper's main claims. Section IV evaluates all models on a single 80%-20% train-validation split, while Section V (Figures 3 and 5, Table II) uses 5-fold cross-validation. The numbers in Figure 2 (e.g., YOLOv11 AP50 on LVVO_1k) are not directly comparable to the Baseline row in Table II, which is obtained with 5-fold CV. This makes it difficult to track how the model's performance evolves from the initial comparison to the enriched setting. Please state explicitly which protocol is used for each result and, where possible, report both metrics under the same protocol.","section":"Section IV versus Section V"}],"minor_comments":[{"comment":"There is a typo in the sentence \"We focus on locating visual elements not clasification.\" It should read \"We focus on locating visual elements, not classification.\"","section":"Section II"},{"comment":"The model name is written inconsistently as \"YOLO11\" (e.g., Figure 3 caption) and \"YOLOv11\" elsewhere. Please standardize to \"YOLOv11\" throughout.","section":"Throughout"},{"comment":"The terms \"Fine-tuning\" and \"Finetuning\" appear with inconsistent capitalization and hyphenation (e.g., \"Progressive Finetuning\" vs. \"Progressive Fine-Tuning\"). Please unify the terminology.","section":"Section V"},{"comment":"The y-axis labels in Figure 5 are missing; the reader cannot tell whether the bars represent AP50 or overall AP for the left and right panels. Please add axis labels and a legend explaining \"yellow\" and \"red\" bars in the caption.","section":"Figure 5"},{"comment":"The paper states that LVVO has four visual categories but also says category labels are not used. Please clarify whether the four categories are used for the detection task or only for the annotation process, and whether the reported metrics are class-agnostic or class-specific.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The LVVO dataset release is a useful contribution, and the cross-dataset analyses in Sections V-A and V-B are informative. However, the enrichment experiment in Section V-C is the paper's headline and it is compromised by test-fold leakage. This is fixable with a properly disjoint evaluation protocol, but as written the main quantitative claim does not stand. I would encourage the editor to request a re-analysis rather than reject, given the value of the dataset and the systematic detector comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: worth knowing for the LVVO dataset and the systematic detector comparison, but the headline enrichment gain is likely inflated by a test-fold leakage. The teacher that auto-labels the 3k unlabeled frames is fine-tuned on an 80% split of LVVO_1k, and the enrichment results are then reported with 5-fold CV on that same LVVO_1k. In expectation, four of the five folds overlap the teacher's training images (about 160 of 200 per fold), so the teacher has already seen the test frames. The 6.59-point AP50 gain in Table II is therefore not an independent measure of generalization; it may just reflect the model agreeing with a teacher that was fitted to the test distribution.\n\nWhat is actually new and solid: the LVVO benchmark is a real contribution—1,000 manually annotated frames (two annotators plus adjudication), 3,000 additional unlabeled frames, from 245 videos across 13 courses, publicly released with source code. The systematic comparison of Faster R-CNN, SSD, Mask R-CNN, RetinaNet, FCOS, and YOLOv11 under a fixed training protocol is useful; YOLOv11's advantage is consistent. The cross-dataset evaluation (Figure 4) and joint-training experiment (Figure 5) are well posed, and the finding that extra in-domain data helps mainly when the target set is small is sensible. Those parts don't rely on the auto-labeling pipeline.\n\nThe soft spots, in proportion: the enrichment evaluation is the main one and it impairs the paper's third contribution. The fix is straightforward: hold out a clean test set from LVVO_1k, train the teacher only on the remaining labeled frames, generate pseudo-labels on the 3k unlabeled frames, and report performance on the truly held-out set. The paper should also report variance across folds; with 1,000 images, fold-to-fold noise is non-trivial. Sample size is small but the authors are transparent about it.\n\nBottom line: solid empirical paper with one inflated claim. The dataset and detector comparison deserve a serious referee, and the enrichment experiment can be rerun cleanly. I'd bring it to a reading group for the dataset and the cross-dataset lessons.\n\nRecommendation: send to peer review, but the referee should push for the re-evaluation.","headline":"Useful dataset and detector comparison, but the auto-labeling gain is likely inflated by test-fold leakage.","tokens_in":9440,"tokens_out":4381,"would_cite":true,"duration_ms":40871,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning YOLOv11 on a dataset that combines 1,000 manually labeled lecture frames with 3,000 auto-labeled frames raises AP50 from 90.75% to 95.32% on the LVVO_1k benchmark, and the paper shows this semi-supervised enrichment makes a…","keywords":["educational video analysis","lecture video frames","visual object detection","transfer learning","semi-supervised auto-labeling","dataset enrichment","YOLOv11","LVVO benchmark"],"falsifier":"Run the auto-labeling pipeline inside each cross-validation fold, training the teacher only on that fold's training images and labeling only that fold's validation images, then evaluate the student on those same validation images; if the AP50 gain shrinks substantially from the reported 6.59-point increase, the improvement is an artifact of the teacher having seen the test fold.","tokens_in":8409,"feed_emoji":"🎓","tokens_out":6952,"duration_ms":63925,"temperature":0.7,"pith_summary":"This paper asks whether a detector for visual elements in lecture slides—charts, tables, graphs, and illustrations—can be built with very few manual labels. The authors evaluate six off-the-shelf object detectors fine-tuned on three lecture-video datasets and find that YOLOv11 is the strongest. They then grow a 1,000-frame manually labeled corpus (LVVO_1k) into a 4,000-frame corpus by having a YOLO teacher label 3,000 additional frames, and show that progressively fine-tuning the model on the combined data raises AP50 from 90.75% to 95.32% and average precision from 77.6% to 84.19% on the manually labeled benchmark. The paper also shows that jointly training on all three datasets yields a single model nearly as accurate as dataset-specific models. A new publicly released benchmark, LVVO, accompanies the claims.","feed_headline":"Auto-labeled frames boost lecture-video detection to 95.3 AP50","feed_subtitle":"Adding 3,000 pseudo-labeled slides to 1,000 manual ones raises AP50 from 90.75% to 95.32% and cuts annotation cost.","key_machinery":"The carrying mechanism is the combination of transfer learning and a semi-supervised auto-labeling pipeline. A COCO-pretrained YOLOv11 (a single-stage real-time object detector) is first fine-tuned on the manually annotated LVVO_1k; this teacher model then generates bounding boxes for the 3,000 unlabeled frames with a 0.5 confidence filter, producing LVVO_3k. The student is trained on the union by progressive fine-tuning, starting from the teacher's weights rather than from COCO weights, which the paper finds superior to training from COCO on the combined set. The evaluation is 5-fold cross-validation on LVVO_1k using COCO metrics AP, AP50, and AP75.","core_discovery":"The central claim is that the bottleneck for detecting visual objects in lecture videos is not the detector architecture but the scarcity of labeled frames, and that this scarcity can be mitigated by a semi-supervised auto-labeling loop. Starting from a COCO-pretrained YOLOv11, the authors fine-tune on 1,000 manually annotated LVVO frames using an 80/20 split, use that model to label 3,000 unlabeled frames at a confidence threshold of 0.5, and then continue fine-tuning the same model on the full 4,000-image set (progressive fine-tuning). On the 1,000 manually annotated frames, this raises AP50 from 90.75% to 95.32% and AP from 77.6% to 84.19%. The paper further claims that a model trained jointly on the LDD, LPM, and LVVO_1k datasets performs comparably to models trained on a single dataset, suggesting a general detector is achievable with a modest diverse sample.","pith_inferences":["If the auto-labeling result holds under nested cross-validation, the same teacher-labeler loop should transfer to other structured document images, such as scanned worksheets or whiteboard photos, where objects are also artificial and boundary-free.","The confidence threshold of 0.5 for pseudo-labels is fixed without a sensitivity study; sweeping this threshold would reveal the precision-recall trade-off of the synthesized labels and likely an optimal operating point.","The diminishing-returns curve for added auto-labeled data suggests an adaptive enrichment budget: stop adding pseudo-labeled frames when validation AP stops improving.","A direct comparison of progressive fine-tuning against training the student from COCO weights on the same combined data would isolate how much of the reported gain comes from the teacher's initialization rather than from the extra data alone."],"forward_implications":["Lecture-video search, navigation, and summarization tools can obtain usable detectors with 1,000 manual labels plus a few thousand unlabeled frames, avoiding large annotation budgets.","A single model trained jointly on LDD, LPM, and LVVO matches dataset-specific models, so a general-purpose lecture-video detector is within reach from a few hundred images per domain.","The biggest gain from auto-labeled data comes at the first 1,000 added frames, so enrichment campaigns can stop early without losing most of the benefit.","Fine-tuning on related in-domain datasets helps mainly when the target labeled set is small (below roughly 60% of 1,000 frames), giving a concrete recipe for low-resource lecture-video projects."],"supporting_citations":[{"why":"Supplies the COCO-pretrained YOLOv11 model that is fine-tuned on lecture frames.","marker":"[32]"},{"why":"Documents the YOLOv11 architecture whose modules (C3k2, SPPF, C2PSA) the paper credits for strong detection.","marker":"[29]"},{"why":"Provides the COCO source dataset whose transferred features seed all fine-tuned models.","marker":"[23]"},{"why":"Establishes that early network layers transfer across domains, the premise for fine-tuning with limited labeled data.","marker":"[30]"},{"why":"Supplies the LDD dataset used for cross-dataset evaluation and joint training.","marker":"[13]"},{"why":"Supplies the LPM dataset used for cross-dataset evaluation and joint training.","marker":"[21]"},{"why":"Provides the heuristic baseline (Logiform) whose lower performance motivates the deep-learning approach.","marker":"[16]"},{"why":"One of the compared detectors; the comparison establishes YOLOv11's superiority.","marker":"[24]"}],"fun_headline_variants":["3,000 auto-labeled frames lift lecture video AP50 to 95.3","Semi-supervised labeling turns 1k manual frames into 4k for video detection","Lecture video detection: adding auto-labeled slides beats model tweaking","Transfer learning plus auto-labeling: a general detector for lecture visuals","Pseudo-labels for slides: AP50 jumps from 90.75 to 95.32"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the auto-generated bounding boxes for the 3,000 unlabeled frames are accurate enough to teach the detector, and that measuring the gain on the same 1,000 manually labeled frames that helped create those pseudo-labels is a fair test of that gain.","fun_headline_variants_meta":{"raw":{"variants":["3,000 auto-labeled frames lift lecture video AP50 to 95.3","Semi-supervised labeling turns 1k manual frames into 4k for video detection","Lecture video detection: adding auto-labeled slides beats model tweaking","Transfer learning plus auto-labeling: a general detector for lecture visuals","Pseudo-labels for slides: AP50 jumps from 90.75 to 95.32"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002001,"raw_usage":{"total_tokens":7857,"prompt_tokens":1045,"completion_tokens":6812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":6707}},"tokens_in":661,"tokens_out":6812,"duration_ms":50296,"temperature":1.0,"reasoning_tokens":6707,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:15:25.289674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the auto-labeling pipeline inside each cross-validation fold, training the teacher only on that fold's training images and labeling only that fold's validation images, then evaluate the student on those same validation images; if the AP50 gain shrinks substantially from the reported 6.59-point increase, the improvement is an artifact of the teacher having seen the test fold.","supporting_citations":[{"cited_title":"YOLOv11 Documentation,","cited_arxiv_id":null,"evidence_quote":"Documents the YOLOv11 architecture whose modules (C3k2, SPPF, C2PSA) the paper credits for strong detection."},{"cited_title":"How transferable are features in deep neural networks?","cited_arxiv_id":null,"evidence_quote":"Establishes that early network layers transfer across domains, the premise for fine-tuning with limited labeled data."},{"cited_title":"Fitvid: Responsive and flexible video content adaptation,","cited_arxiv_id":null,"evidence_quote":"Supplies the LDD dataset used for cross-dataset evaluation and joint training."},{"cited_title":"Lecture presentations multimodal dataset: Towards understanding multimodality in educational videos,","cited_arxiv_id":null,"evidence_quote":"Supplies the LPM dataset used for cross-dataset evaluation and joint training."},{"cited_title":"Identification of visual objects in lecture videos with color and keypoints analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the heuristic baseline (Logiform) whose lower performance motivates the deep-learning approach."}],"review_version":1}