{"id":"7a4fbdc4-47f8-4ba7-a728-e4d1feded568","arxiv_id":"2506.13657","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark dataset of 4,000 lecture video frames, 1,000 manually annotated with four visual object categories and 3,000 auto-labeled via a fine-tuned YOLOv11 model.","lead":"This paper introduces LVVO, a dataset of 4,000 lecture video frames with bounding box labels for four kinds of visual content: tables, charts, photos, and illustrations. It is meant to help researchers build and test systems that find important visuals in educational videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central vulnerability is the unvalidated automatic labels in LVVO_3k; if they carry systematic error, the 4,000-frame benchmark claim is unsupported, though LVVO_1k may remain sound.","rationale":"I read the paper as primarily contributing a dataset; the manual annotation pipeline with dual independent annotation and third-expert conflict resolution is a genuine strength, and the public repository is a useful resource. The reader's weakest assumption—that the 3,000 automatically labeled frames are accurate enough to include in a benchmark—is also the most load-bearing concern for the central claim. The manual LVVO_1k subset can stand on its own, but the abstract's 4,000-frame claim and the 'valuable resource' framing depend on the auto-labels being reliable, and Section 4 provides no validation beyond an unanalyzed confidence threshold. I considered other issues such as the internal consistency of the annotation statistics and the absence of baseline detection results; these are worth fixing but do not threaten the core claim as directly as the unvalidated auto-label set. The proposed re-annotation of a stratified 300-frame sample is a feasible, decisive check: it would either confirm the auto-labels are usable or force the authors to restrict the benchmark claim to LVVO_1k. Since the reader already conditionalizes the verdict on this concern, I do not change the verdict.","tokens_in":4130,"tokens_out":3716,"duration_ms":38104,"concrete_test":"Take a random sample of 300 frames from LVVO_3k, stratified by subject area and instructor. Have two annotators label them with the same protocol and adjudicate disagreements with the third expert. Compare the auto-generated boxes to this consensus using COCO-style AP at IoU=0.5 and 0.75, plus per-class precision/recall for Table, Chart-Graph, Photographic-image, and Visual-illustration. If per-class AP falls substantially below the 83.41% inter-annotator F1 level, or if the rare Table class collapses to near-zero recall, then LVVO_3k should be released only as pseudo-labels and the 4,000-frame benchmark claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LVVO is a benchmark with 4,000 labeled frames. Only 1,000 frames are human-verified; the remaining 3,000 (LVVO_3k) are produced by fine-tuning a COCO-pretrained YOLOv11 on LVVO_1k and applying a 0.5 confidence threshold (Section 4). The text asserts this 'ensures the quality of the automatically generated annotations,' but no evidence is provided: there is no held-out human comparison, no per-class precision/recall on the 3k subset, and no analysis of failure modes such as the rare Table category or the Visual-illustration/Chart-Graph confusion noted in Section 3.3.1. A confidence threshold filters low-confidence boxes but does not correct systematic bias introduced by the training distribution or class imbalance; a model trained on 1k frames from these instructors may also transfer poorly to new slides or subjects. If the auto-labels are biased, the expanded 3,000-frame set is not a trustworthy benchmark, and the abstract's claim of a 4,000-frame dataset with 3,000 automatically annotated frames needs qualification or removal. The LVVO_1k manual subset would still be usable, so the concern is about the expanded dataset, not the entire contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Lecture Video Visual Objects (LVVO) dataset, a benchmark for detecting visual objects in educational video frames. It comprises 4,000 frames from 245 lecture videos across biology, computer science, and geosciences. A subset of 1,000 frames (LVVO_1k) is manually annotated with bounding boxes for four categories (Table, Chart-Graph, Photographic-image, Visual-illustration) by two annotators, with a third expert resolving disagreements; the reported inter-annotator F1 score is 83.41%. The remaining 3,000 frames (LVVO_3k) are automatically annotated by fine-tuning YOLOv11 on LVVO_1k and applying a confidence threshold. The paper also describes the annotation instructions, statistics, and dataset file structure, and makes the dataset publicly available.","tokens_in":4377,"tokens_out":4286,"duration_ms":39434,"significance":"If the dataset is reliable, it addresses a genuine gap: most object detection benchmarks target natural images, whereas lecture slides contain synthetic, semantically rich visual content. The manual annotation process with explicit guidelines and conflict resolution is a strength, and public release of the data supports reproducibility. The reported inter-annotator F1 of 83.41% indicates reasonable agreement, though the supporting counts need clarification. The automatically labeled LVVO_3k subset currently lacks any validation against human annotation, which is essential for a benchmark claiming to be a 4,000-frame resource. The LVVO_1k portion appears sound, but the expanded dataset's quality is unsubstantiated.","major_comments":[{"comment":"The assertion that applying a 0.5 confidence threshold to YOLOv11 predictions 'ensures the quality of the automatically generated annotations' is unsupported. No held-out human comparison, per-class precision/recall, or failure analysis is reported for the 3,000 LVVO_3k frames. Since the abstract and Section 1 present the complete 4,000-frame dataset as the benchmark, the quality of the auto-labeled subset is load-bearing. Please add a validation experiment, such as having human annotators label a random sample of LVVO_3k and computing F1/precision/recall relative to the automatic labels, or at minimum report the LVVO_1k validation performance and a qualitative discussion of failure modes such as the rare Table category and the Visual-illustration/Chart-Graph confusion noted in Section 3.3.1.","section":"Section 4"},{"comment":"The unmatched-box counts reported for different IoU thresholds are internally inconsistent. The text states that at the lowest IoU thresholds only 122 (version 1) and 152 (version 2) boxes remain unmatched, but at IoU 0.5 it reports 239 and 269 unmatched boxes. Because decreasing the IoU threshold can only increase the number of matched pairs (and decrease unmatched boxes), lower thresholds cannot yield fewer unmatched boxes than observed at 0.5. Additionally, the sentence 'As the IoU threshold decreases, the number of matched pairs gradually declines' reverses the expected relationship. Please clarify the exact thresholds used in Figure 1, correct the reported counts, and recompute the 83.41% F1 score if needed; this value is central to the paper's claim of annotator agreement.","section":"Section 3.3.1"},{"comment":"The abstract calls the 83.41% value an 'inter-annotator F1 score,' while Section 3.3.1 calls it '83.41% agreement.' If the metric is F1 computed as 2·matched / (2·matched + unmatched_v1 + unmatched_v2), state this definition explicitly and use consistent terminology throughout. Also specify that the greedy box matching is based on IoU >= 0.5 when computing the headline F1, since Figure 1 covers multiple thresholds and the choice of threshold directly affects the reported agreement.","section":"Section 3.3.1 and abstract"}],"minor_comments":[{"comment":"The caption of Figure 1 says 'Higher thresholds lead to fewer matches due to stricter overlap requirements,' which is correct, but the main text says 'Moving from right to left (i.e., from high to low IoU thresholds), there is a significant increase in the number of matched pairs at higher IoU values.' This is confusing; please rephrase to state that as the IoU threshold decreases, matched pairs increase.","section":"Section 3.3.1, Figure 1"},{"comment":"The text says the 1,000 frames are 'randomly selected,' but Table 1 shows subject-wise counts (408, 245, 347) that are not proportional to the total frames per subject (1,932, 786, 1,282). Clarify whether the selection was stratified by subject or instructor, and if so describe the stratification procedure.","section":"Section 2.2 / Table 1"},{"comment":"Provide sufficient training details for reproducibility: input resolution, number of epochs, learning rate, batch size, data augmentation, and the exact YOLOv11 variant (e.g., YOLOv11n/s/m/l/x). The current description is too brief for others to replicate the automatic labeling pipeline.","section":"Section 4"},{"comment":"The dataset file description mentions 'JSON annotation files' but does not specify the JSON schema (e.g., whether annotations are in COCO format, the fields for each box, or how the 'single category' variant is encoded). A brief example or reference to the schema would help users consume the dataset.","section":"Section 5"},{"comment":"Reference [6] is listed as 'to appear.' If the associated paper has been published by the time of submission, update the citation with volume, pages, or DOI; otherwise indicate the expected venue and year.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core manual annotation effort is valuable and the public dataset is a positive contribution. However, the unvalidated LVVO_3k auto-labels are a central part of the claimed 4,000-frame benchmark, and the internal inconsistency in the unmatched-box counts in Section 3.3.1 undermines the headline F1 until corrected. Both issues are fixable with additional experiments and careful revision, but they are load-bearing for the paper's main claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the LVVO paper delivers a genuinely useful resource—1,000 manually annotated lecture-video frames with four visual categories, a clear annotation protocol, and dual-annotation plus conflict resolution. The 3,000 auto-labeled frames, however, are not validated against human labels, so the full 4,000-frame benchmark claim is only as strong as the unvalidated model output.\n\nWhat's new: this is the first public benchmark I know of for detecting tables, charts/graphs, photographic images, and visual illustrations in lecture videos. The annotation instructions are concrete, the category definitions are usable, and the frame-extraction refinement that removes duplicates and text-only frames is a sensible contribution. The distribution stats across subjects and instructors give a fair picture of the dataset's scope.\n\nWhere it falls short: the biggest gap is the lack of any validation of LVVO_3k. Section 4 says a 0.5 confidence threshold 'ensures the quality' of the auto-labels, but a confidence threshold does nothing about systematic bias from the training distribution or class imbalance. No held-out human comparison, no per-class precision/recall, no error analysis. Until that's provided, the 3k set should be labeled 'unverified' rather than presented as part of a finished benchmark. Second, the paper reports no baseline detection results at all, which is odd for a paper calling itself a benchmark; even a YOLOv11 run on the test split of LVVO_1k would help calibrate expectations. Third, the relationship to the authors' prior work [4] and [6] is not stated. Reference [6] appears to describe the same transfer-learning pipeline, and the paper should say explicitly what this release adds beyond that. Finally, 'semi-supervised' is a stretch for 'fine-tune on 1k, infer on 3k'—it's just automatic labeling.\n\nOne correction to the reader's report: the matching statistics in Section 3.3.1 are internally consistent. At low IoU thresholds there are 122 and 152 unmatched boxes; at IoU 0.5 there are more unmatched (239 and 269) because the threshold is stricter. That's expected, not contradictory.\n\nBottom line: if you work on educational video analysis, LVVO_1k is worth having. The 3k extension needs validation or a clear downgrade to 'unverified auto-labels'. Despite these issues, the manual subset and annotation process deserve serious peer review, so I'd send it out—with a firm request for either validation of the auto-labels or a revised abstract that doesn't overclaim.","headline":"A useful 1,000-frame manually annotated lecture-video benchmark with an unvalidated 3,000-frame auto-labeled extension; the manual subset is sound, the expanded set needs qualification.","tokens_in":4938,"tokens_out":2094,"would_cite":false,"duration_ms":19903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a public benchmark for detecting visual objects in educational videos.","keywords":["lecture video dataset","visual object detection","bounding box annotation","benchmark","educational videos","semi-supervised labeling","chart and table detection","frame extraction"],"falsifier":"Have a fresh set of annotators label a random sample of, say, 100 frames from LVVO_3k under the same instructions, match their boxes to the automatic labels by greedy IoU, and compare category agreement; if the resulting F1 falls well below the 83.41% seen on the manually labeled subset, the claim that the expanded dataset is a reliable benchmark fails.","tokens_in":3902,"feed_emoji":"🎓","tokens_out":5665,"duration_ms":55762,"temperature":0.7,"pith_summary":"The paper introduces LVVO, a benchmark dataset for detecting visual objects inside educational lecture videos. The authors assembled 4,000 frames from 245 lecture videos in biology, computer science, and geosciences, manually labeled 1,000 of those frames with bounding boxes for tables, chart-graphs, photographic images, and visual illustrations, and report an inter-annotator F1 score of 83.41%. A third expert reconciled disagreements, and the remaining 3,000 frames were labeled automatically by a fine-tuned detection model. The goal is to give the research community a standard place to develop and compare supervised and semi-supervised methods for recognizing non-text visual content in lecture slides. If the dataset is sound, it closes a gap in benchmarks that mostly target natural images rather than the artificial, text-rich visuals typical of educational videos.","feed_headline":"New benchmark labels tables and charts in 4,000 lecture frames","feed_subtitle":"The 1,000 human-annotated frames hit 83.41% annotator agreement; 3,000 more are auto-labeled for expansion.","key_machinery":"The load-bearing mechanism is the dataset construction pipeline rather than a single algorithm. Frames are selected from 245 lectures by a slide-transition detector extended with duplicate removal and a filter that drops text-only frames. Annotation uses a two-pass human workflow: two annotators label each frame under shared instructions, boxes are matched with a greedy highest-IoU procedure to measure agreement, and a third expert resolves remaining conflicts. The automatically labeled expansion is produced by fine-tuning YOLOv11, a one-stage object detector, on LVVO_1k and thresholding predictions at 0.5 confidence. The four object categories—Table, Chart-Graph, Photographic-image, and Visual-illustration—are defined in the annotation instructions and form the label space of the benchmark.","core_discovery":"The central claim is that LVVO is a reliable public benchmark for visual object detection in educational videos, with four categories that cover the dominant types of non-text content on lecture slides. The manually annotated subset LVVO_1k consists of 1,000 frames, each independently labeled by two annotators and then adjudicated by a third expert, yielding an inter-annotator F1 of 83.41% at an IoU threshold of 0.5. The expanded subset LVVO_3k adds 3,000 frames automatically labeled by a YOLOv11 model fine-tuned on LVVO_1k with a 0.5 confidence cutoff. The paper argues that, together, these provide a resource for both fully supervised training and semi-supervised learning, and it releases category distributions showing that visual illustrations are most common, followed by chart-graphs, photographic images, and tables.","pith_inferences":["An implication the authors leave implicit is that the 3,000 automatically labeled frames need human validation before being treated as ground truth for training or evaluation, since the paper reports no such check.","The four categories could be refined into finer-grained classes, such as separating line charts, bar charts, scatter plots, and flow diagrams, which the current Chart-Graph and Visual-illustration labels partially merge.","Because lecture slides have consistent layouts across courses, LVVO may serve as a transfer-learning source for slide parsing, OCR layout analysis, and educational video navigation tasks beyond object detection.","One consequence not developed in the paper is that the benchmark's value depends on its source distribution, so models trained on these eight instructors may need domain adaptation before generalizing to other recording styles or disciplines."],"forward_implications":["Detectors trained and evaluated on LVVO can be compared for the first time on a common set of lecture-slide visuals spanning biology, computer science, and geosciences.","Semi-supervised object detection methods can use LVVO_1k as labeled data and LVVO_3k as unlabeled or automatically labeled data in the same evaluation setting.","The 83.41% inter-annotator F1 at 0.5 IoU gives downstream users a realistic upper bound on annotation consistency when measuring model performance.","The public release of both with-category and single-class variants lets researchers isolate whether errors come from localization or from category confusion.","Because the four categories dominate slide content, a detector that performs well on LVVO is likely to transfer to other slide-based educational video collections."],"supporting_citations":[{"why":"Supplies the source platform from which all 245 lecture videos were collected.","marker":"[1]"},{"why":"Provides the earlier lecture-video collection that establishes the video pool the dataset builds on.","marker":"[2]"},{"why":"Supplies the slide-transition method used to extract unique key frames from lecture videos.","marker":"[3]"},{"why":"Documents the prior visual-object identification work and the challenges of annotating artificial slide content.","marker":"[4]"},{"why":"Supplies the annotation tool used by annotators to draw bounding boxes and assign category labels.","marker":"[5]"},{"why":"Describes the transfer-learning and dataset-enrichment approach that produces the 3,000 automatically labeled frames.","marker":"[6]"},{"why":"Supplies the fine-tuned YOLOv11 detection model used for automatic annotation.","marker":"[7]"}],"fun_headline_variants":["Benchmark detects tables, charts, and images in lecture videos","New dataset labels visual objects in educational videos","1,000 human-annotated lecture frames for visual detection","LVVO: benchmark for visual objects in lecture slides","Detect tables, charts, and images in lecture videos with LVVO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 3,000 automatically labeled frames are accurate enough to count as benchmark-quality annotations, even though the paper reports no human verification of that subset.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark detects tables, charts, and images in lecture videos","New dataset labels visual objects in educational videos","1,000 human-annotated lecture frames for visual detection","LVVO: benchmark for visual objects in lecture slides","Detect tables, charts, and images in lecture videos with LVVO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1395,"prompt_tokens":932,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":548,"tokens_out":463,"duration_ms":5124,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:27:47.700345+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh set of annotators label a random sample of, say, 100 frames from LVVO_3k under the same instructions, match their boxes to the automatic labels by greedy IoU, and compare category agreement; if the resulting F1 falls well below the 83.41% seen on the manually labeled subset, the claim that the expanded dataset is a reliable benchmark fails.","supporting_citations":[{"cited_title":"Videopoints: Lecture video platform,","cited_arxiv_id":null,"evidence_quote":"Supplies the source platform from which all 245 lecture videos were collected."},{"cited_title":"Enhancing lecture video navigation with AI generated summaries,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier lecture-video collection that establishes the video pool the dataset builds on."},{"cited_title":"Indexed captioned searchable videos: A learning companion for STEM coursework,","cited_arxiv_id":null,"evidence_quote":"Supplies the slide-transition method used to extract unique key frames from lecture videos."},{"cited_title":"Identification of visual objects in lecture videos with color and keypoints analysis,","cited_arxiv_id":null,"evidence_quote":"Documents the prior visual-object identification work and the challenges of annotating artificial slide content."},{"cited_title":"Visual object tagging tool,","cited_arxiv_id":null,"evidence_quote":"Supplies the annotation tool used by annotators to draw bounding boxes and assign category labels."},{"cited_title":"Visual content detection in educational videos with transfer learn- ing and dataset enrichment,","cited_arxiv_id":null,"evidence_quote":"Describes the transfer-learning and dataset-enrichment approach that produces the 3,000 automatically labeled frames."},{"cited_title":"Ultralytics yolo11,","cited_arxiv_id":null,"evidence_quote":"Supplies the fine-tuned YOLOv11 detection model used for automatic annotation."}],"review_version":1}