{"id":"0104b694-f143-4216-9e2c-e79217db1317","arxiv_id":"2501.00958","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Converting 22,000 hours of instructional videos into a coherent image-text interleaved corpus improves VLM pretraining on knowledge and reasoning benchmarks.","lead":"The authors converted 22,000 hours of online instructional videos into a training corpus with 6.5 million still images paired with spoken and on-screen text. Vision-language models pretrained on this corpus scored higher on science and math reasoning tests and made better use of examples in their prompts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sample-matched control in Section 5.1 cannot separate data quality from data quantity: Textbook samples carry 2–4x more images and 1.6–3x more text tokens than MMC4/OBELICS (Table 1), so the headline quality attribution is unproven.","rationale":"The reader identified the sample-count-versus-content-budget confound as the weakest assumption, and I agree that this is the single most load-bearing issue for the paper's central claim. The paper is honest about the per-sample statistics in Table 1, which makes the concern concrete rather than speculative: 610K samples of Textbook expose the model to far more images and tokens than 610K samples of MMC4 or OBELICS. Since the loss is computed over all text tokens in the interleaved sequence, the training signal itself is larger for Textbook under the paper's protocol. This does not invalidate the dataset contribution: the curation pipeline, human evaluation, and ablations are genuinely useful, and the dataset is likely valuable regardless of the precise attribution. But the headline claim that textbook quality, coherence, or knowledge density drives the improvements cannot be separated from simple data-scale effects without matched-budget runs. The cheat test and shuffle experiment are interesting diagnostics, but they partly reward copying or brittle order sensitivity, so they do not settle the attribution question. A conditional verdict remains appropriate; the paper should be accepted for its dataset and pipeline, with the quality-vs-quantity claim explicitly conditional on matched-budget follow-ups, contamination analysis, and ideally repeated-seed error bars.","tokens_in":23038,"tokens_out":2108,"duration_ms":22278,"concrete_test":"Re-run the LLaVA-1.5 continual-pretraining comparison with matched content budgets, not matched sample counts. Specifically: (a) subsample and truncate Textbook samples so that the total number of images and text tokens matches the 610K MMC4 sample budget (roughly 325K Textbook samples if matching MMC4's 5.7 images/sample, and about 195K samples if matching 417 tokens/sample); (b) conversely, sample MMC4 and OBELICS with enough samples to match Textbook's total image and token count (roughly 1.15M MMC4 samples to match 10.7 images/sample). If the Textbook advantage persists when total images/tokens are equalized, the quality-attribution claim survives; if the advantage shrinks or inverts, the reported gains are substantially a data-volume effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states that for a fair comparison the authors sample an equivalent number of samples (610K) from MMC4, OBELICS, and Textbook, and apply identical training parameters. But Table 1 shows that the corpora are not comparable at the sample level: Textbook averages 10.7 images and 1,297 tokens per sample, MMC4 averages 5.7 images and 417 tokens, and OBELICS averages 2.5 images and 816 tokens. Consequently, a model trained on 610K Textbook samples sees roughly 1.9–4.3x more images and 1.6–3.1x more text tokens than the MMC4/OBELICS controls. Because the pretraining objective is next-token prediction on an interleaved sequence, the textbook's superiority on ScienceQA (+20% over MMC4 in Table 2), MathVista (+5.3% and +6.4% over OBELICS), and the SFT setting (Table 5) could be explained by greater effective data volume rather than by 'high quality' or 'coherent interleaving.' The shuffle experiment (Figure 3) and InSI-SIM metric demonstrate that Textbook has stronger image-order dependence and within-sample coherence, but they do not quantify how much of the downstream gain is attributable to that coherence versus the larger per-sample budget. The paper also does not report contamination checks against benchmarks that share school-subject domains, which is a secondary but real risk given the evaluation targets. The most load-bearing issue remains the confounded comparison: without matched image/token budgets, the central claim that video-derived textbook data transfers better because of its quality is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new vision-language pretraining corpus, the \"multimodal textbook,\" built from 75,000 instructional videos (22,697 hours) covering six school subjects. A pipeline uses an LLM-generated taxonomy to retrieve videos, then extracts and refines ASR text, keyframes, and OCR, and interleaves them chronologically into 610K samples containing 6.5M keyframes and 0.75B text tokens. The authors evaluate the corpus by continually pretraining LLaVA-1.5-7B and Idefics2-8B on either Textbook-6.5M or 610K-sample subsamples of MMC4 and OBELICS, reporting gains on ScienceQA, MathVista, OKVQA, and TextVQA, plus analyses of in-context awareness via a \"cheat test\" and image-order shuffling. The central claim is that video-derived interleaved textbook data transfers better to knowledge- and reasoning-intensive downstream tasks than web-crawled interleaved data under the same training protocol.","tokens_in":23332,"tokens_out":3167,"duration_ms":31300,"significance":"If the central comparison were clean, this would be a valuable contribution: it introduces a large, openly released interleaved dataset from an underexplored source (instructional video), with a reproducible multi-level curation pipeline, transparent statistics, human evaluation, and several ablations of pipeline components (ASR refinement, OCR, keyframe extraction). The cheat test is a clever diagnostic for whether pretraining improves use of in-context interleaved evidence. However, the main experimental evidence for data quality over data quantity is currently confounded, and the absence of a contamination check leaves the headline ScienceQA/MathVista gains incompletely supported. The dataset itself and the pipeline remain useful regardless of the outcome of the controlled comparison.","major_comments":[{"comment":"The sample-matched control does not match the amount of training signal. Table 1 reports that Textbook averages 10.7 images and 1,297 text tokens per sample, while MMC4 averages 5.7 images and 417 tokens and OBELICS averages 2.5 images and 816 tokens. Sampling 610K samples from each corpus therefore gives the Textbook model roughly 1.9x–4.3x more images and 1.6x–3.1x more text tokens than the MMC4/OBELICS controls. Because the pretraining objective is next-token prediction over the interleaved sequence, the gains in Tables 2, 3, and 5 could be driven by larger effective data volume per training step rather than by the asserted higher quality or coherence of the textbook data. The paper should add token- and image-matched controls, e.g., subsample Textbook to match MMC4/OBELICS in total images and text tokens, or report scaling curves that allow the reader to separate quantity from quality.","section":"§5.1, Table 1"},{"comment":"No contamination check is reported, although the corpus is deliberately curated from the same school-subject domains as the evaluation benchmarks. The taxonomy covers mathematics, physics, chemistry, earth science, engineering, and computer science, and the main gains appear on ScienceQA and math benchmarks. Since these benchmarks are public and contain images and questions that could overlap with instructional videos or their ASR/OCR text, the paper should report near-duplicate image detection and n-gram overlap between the pretraining corpus and benchmark samples, or evaluate on a held-out benchmark from a different distribution. Without this, the +20% ScienceQA gain over MMC4 in Table 2 is at risk of being inflated by contamination.","section":"§5.2, Table 2"},{"comment":"The shuffle experiment shows that Textbook is more sensitive to image order than MMC4/OBELICS, which is evidence of within-sample coherence, but it does not quantify how much of the downstream gain is attributable to that coherence. The figure reports average accuracy after shuffling 20%, 50%, and 100% of samples, yet the comparison still starts from the confounded budgets described above. A matched-budget shuffle comparison (e.g., equal total images and tokens) would directly estimate the contribution of coherent ordering, and should be reported together with the unshuffled matched-budget result.","section":"§5.3, Figure 3"},{"comment":"The ablations in Table 6 also use the full Textbook-6.5M corpus without matched budgets, so the conclusions about pipeline components (ASR refinement, OCR, SSIM vs. pixel/CLIP keyframe extraction) are internally valid for the proposed pipeline but do not by themselves establish superiority over web-crawled corpora. This is acceptable for component analysis, but the abstract and Section 5.2 should be reworded to avoid attributing all gains to 'data quality' until the quantity confound is resolved.","section":"§5.4, Table 6"}],"minor_comments":[{"comment":"The text has several typos and grammatical slips, e.g., 'totoaling' in Section 3.2 and 'asas' in the Table 4 caption; these should be corrected in a revision.","section":"Abstract / Section 3.2"},{"comment":"The list of benchmarks reads 'MathVista, MathVision, MathVision'; the second 'MathVision' should likely be 'MathVerse', matching Tables 2 and 3.","section":"§5.1 Evaluation Methods"},{"comment":"The header 'MathVison' is misspelled; it should be 'MathVision'.","section":"Table 3"},{"comment":"The sentence discussing refined ASR perplexity says refined ASR has PPL 13.9, which is still higher than MMC4-Core-ff (12.56) and OBELICS (11.27); the claim that it 'more closely aligns with standard training corpora' is relative to raw ASR (16.86) and should be phrased as such.","section":"§5.4"},{"comment":"The 0-shot ScienceQA column shows '-' for MMC4 and OBELICS; the paper should state whether this is a formatting omission or that these baselines were not evaluated in the 0-shot setting, since the text says improvements are observed in 'zero-shot' settings.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a potentially useful dataset and pipeline, and the cheat-test analysis is a nice diagnostic. The main experimental comparison, however, is confounded by unequal image/token budgets, and the contamination risk from domain-aligned benchmarks is unaddressed. I recommend major revision: the authors should add token/image-matched controls and contamination checks, and then temper the quality attributions accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nBottom line: the corpus is a genuine contribution and the pipeline is clearly described, but the headline comparison is confounded. Textbook samples carry roughly 2–4x more images and 1.6–3x more text tokens than the MMC4/OBELICS controls, so matching 610K samples does not match the training budget. The gains on ScienceQA and math benchmarks may reflect data quantity as much as 'textbook quality.' That said, this is a well-executed dataset paper, and the corpus itself is worth having.\n\nWhat's new: turning instructional videos into an interleaved keyframe-ASR-OCR textbook at this scale (75K videos, 22K hours, 6.5M keyframes, 0.75B tokens) is a new contribution. The LLM-based taxonomy, three-level filtering, and chronological interleaving are thoughtful. The ablations (ASR refinement, OCR inclusion, SSIM vs pixel vs CLIP keyframe extraction) are useful and directionally consistent. The shuffle experiment and InSI-SIM metric give evidence that the corpus has more coherent image ordering than web-crawled corpora. The paper's own limitations section acknowledges some residual redundancy, which is honest.\n\nSoft spots, in order of severity. First, the main results (Tables 2 and 3) cannot separate quality from quantity. A matched-token or matched-image control is needed before crediting 'quality.' Second, no contamination check is reported, and the evaluation benchmarks (ScienceQA especially) share school-subject content with the corpus. That is a real risk even if probably not the whole story. Third, the cheat test is an interesting probe of context awareness, but it partially rewards copying; a model that simply echoes the context gets high scores, so it should be interpreted as 'can attend to context,' not necessarily as 'understands better.' Minor: the paper reports no error bars or repeated seeds; given the confound, extra rigor would help.\n\nWho benefits: researchers building or using interleaved pretraining corpora, and anyone evaluating data quality vs. quantity in VLM training. I'd send it to peer review: the dataset is a resource, the pipeline is reproducible (code and data links are provided), and the confound is fixable in revision. Referees should push for matched-budget experiments and contamination analysis. I'd cite the corpus, with a caveat about the comparison.","headline":"Solid dataset, confounded comparison: Textbook's gains may reflect data quantity, not just quality.","tokens_in":23953,"tokens_out":3202,"would_cite":true,"duration_ms":26695,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining vision-language models on a video-derived 'textbook' corpus improves knowledge and reasoning benchmarks more than web-crawled interleaved data.","keywords":["vision-language pretraining","interleaved image-text data","instructional video","multimodal textbook","keyframe extraction","ASR refinement","OCR","in-context learning"],"falsifier":"Run a controlled pretraining experiment in which the Textbook-6.5M samples are truncated or downsampled so that mean images per sample and mean text tokens per sample match MMC4's and OBELICS's; if the ScienceQA and MathVista gains shrink to near zero, the paper's quality argument would not be supported.","tokens_in":22762,"feed_emoji":"📚","tokens_out":6238,"duration_ms":54703,"temperature":0.7,"pith_summary":"This paper tries to establish that instructional videos, such as recorded classroom lectures, can be converted into a high-quality 'multimodal textbook' for pretraining vision-language models (VLMs), and that this textbook teaches the models more than the web-crawled interleaved image-text datasets currently used. The authors assemble 22,000 hours of online courses, align keyframes with automatically transcribed and cleaned lecture speech plus on-screen text, and interleave them chronologically. Pretraining LLaVA-1.5 and Idefics2 on this corpus improves seven benchmarks, with the largest gains on knowledge- and reasoning-intensive tasks such as ScienceQA and MathVista. The same pretraining also makes the models far better at exploiting their few-shot context, as shown by a 'cheat test' in which the test question appears in the prompt.","feed_headline":"Class videos beat web pages as AI textbook data","feed_subtitle":"A 22,000-hour video corpus, turned into keyframes plus lecture text, lifts ScienceQA and MathVista scores.","key_machinery":"The load-bearing object is the video-to-textbook pipeline, which converts a lecture video into an ordered sequence of keyframes interleaved with two text streams: refined automatic speech recognition (ASR) of the instructor's narration and OCR of on-screen text, symbols, and formulas. The pipeline starts from an LLM-generated taxonomy of 3,915 knowledge points used to retrieve videos, then filters at video, clip, and keyframe levels: ASR transcripts are rewritten for fluency, clips are kept only when their caption resembles their ASR, keyframes are selected by structural similarity (SSIM) between consecutive frames, and OCR is deduplicated. This machinery creates the corpus's high in-sample image similarity (0.686 versus roughly 0.32 to 0.35 for web datasets), and the resulting sample-level coherence is what the paper credits for the downstream gains.","core_discovery":"The central claim is that a video-centric, textbook-style interleaved corpus is a better pretraining substrate than webpage-crawled interleaved corpora for knowledge and reasoning. From 75,000 instructional videos (22,697 hours) the pipeline produces 6.5 million keyframes and 0.75 billion text tokens, organized into 610K samples with an average of 10.7 images and 1,297 tokens per sample. On LLaVA-1.5, continual pretraining on this corpus outperforms equal-sample pretraining on MMC4 and OBELICS across seven VQA and math benchmarks, with ScienceQA improving by over 20 points and MathVista by roughly 5 to 6 points in few-shot settings; on Idefics2, the corpus brings additional gains on math benchmarks. The paper also argues that the corpus's coherent image sequences and tight image-text alignment teach VLMs to attend to interleaved context, demonstrated by near-perfect performance when the answer to the test question is hidden in the few-shot prompt, and by a large drop when image order is shuffled.","pith_inferences":["Beyond the paper: a testable extension is to ablate corpus quality versus quantity, because textbook samples contain roughly twice as many images and three times as many tokens as MMC4 and OBELICS samples; holding images and tokens per sample constant would separate the effect of content quality from the effect of more data per training example.","Beyond the paper: the same pipeline likely transfers to other modalities or languages, for instance non-English lecture videos or video paired with timed subtitles, and it would be useful to measure whether the interleaved-coherence benefit persists there.","Beyond the paper: the cheat-test result suggests a new evaluation protocol for interleaved-context awareness in VLMs, one that future model releases could report alongside standard benchmarks."],"forward_implications":["If the claim holds, video-derived textbook data can be used in place of or alongside web-crawled interleaved data in VLM pretraining, especially when target tasks are knowledge- or reasoning-heavy.","The observed ScienceQA and MathVista gains should transfer to instruction tuning, because the paper shows the pretraining benefit survives the supervised fine-tuning stage.","Interleaved-context awareness becomes a trainable capability: models pretrained on coherent frame sequences can exploit few-shot visual and textual evidence more reliably.","Image order in an interleaved sample carries information; shuffling the order during pretraining measurably hurts performance, so future corpus design should preserve chronological or pedagogical sequencing.","The recipe is reproducible: the paper reports ablations showing ASR refinement, OCR integration, and SSIM keyframe selection each contribute to the gains."],"supporting_citations":[{"why":"Web-crawled interleaved baseline; equal-sample pretraining on it is the primary control.","marker":"[60]"},{"why":"Web-crawled interleaved baseline; second control dataset used for sample-matched pretraining.","marker":"[22]"},{"why":"Provides the few-shot/RICES-style evaluation protocol and an additional multi-source interleaved comparison in Table 1.","marker":"[25]"},{"why":"LLaVA-1.5 is the base VLM for continual-pretraining experiments and downstream supervised fine-tuning.","marker":"[31]"},{"why":"Idefics2 supplies the multi-image VLM used for both scratch and continual-pretraining checks.","marker":"[21]"},{"why":"SSIM is the keyframe-selection algorithm whose choice is ablated; it determines the number and informativeness of frames.","marker":"[50]"},{"why":"The Qwen2 LLM is used to rewrite raw ASR transcriptions, an ablation item that strongly affects performance.","marker":"[51]"},{"why":"InternVL2 performs OCR and keyframe scoring, providing the on-screen text stream and filtering.","marker":"[11]"},{"why":"VideoLlama2 generates clip captions used to filter clips whose visuals do not match their ASR.","marker":"[12]"}],"fun_headline_variants":["Video lectures outdo web pages as AI training data","2.5 years of class videos beat web-crawled data for VLM pretraining","22,000 hours of lectures become a smarter AI textbook","Class videos outmatch web pages for training vision-language models","Lecture videos beat webpage crawls for AI knowledge acquisition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that sampling 610K examples from each corpus and training with identical hyperparameters is a fair control, even though textbook samples contain about twice as many images and three times as many text tokens on average.","fun_headline_variants_meta":{"raw":{"variants":["Video lectures outdo web pages as AI training data","2.5 years of class videos beat web-crawled data for VLM pretraining","22,000 hours of lectures become a smarter AI textbook","Class videos outmatch web pages for training vision-language models","Lecture videos beat webpage crawls for AI knowledge acquisition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3193,"prompt_tokens":1055,"completion_tokens":2138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":2052}},"tokens_in":671,"tokens_out":2138,"duration_ms":14418,"temperature":1.0,"reasoning_tokens":2052,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:38:36.892148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled pretraining experiment in which the Textbook-6.5M samples are truncated or downsampled so that mean images per sample and mean text tokens per sample match MMC4's and OBELICS's; if the ScienceQA and MathVista gains shrink to near zero, the paper's quality argument would not be supported.","supporting_citations":[{"cited_title":"Multimodal c4: An open, billion-scale corpus of images interleaved with text","cited_arxiv_id":null,"evidence_quote":"Web-crawled interleaved baseline; equal-sample pretraining on it is the primary control."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"LLaVA-1.5 is the base VLM for continual-pretraining experiments and downstream supervised fine-tuning."}],"review_version":1}