Pith. sign in

REVIEW 4 major objections 5 minor 65 references

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pretraining vision-language models on a video-derived 'textbook' corpus improves knowledge and reasoning benchmarks more than web-crawled interleaved data.

desk verdict Solid dataset, confounded comparison: Textbook's gains may reflect data quantity, not just quality. read the letter →

arxiv 2501.00958 v4 pith:J3UTHXGX submitted 2025-01-01 cs.CV cs.CLcs.LG

Wenqi Zhang , Hang Zhang , Xin Li , Jiashuo Sun , Yongliang Shen , Weiming Lu , Deli Zhao , Yueting Zhuang
show 1 more author
Lidong Bing
This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords vision-languagepretraininginterleavedimage-textdatainstructionalvideomultimodaltextbookkeyframeextractionASRrefinementOCRin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that instructional videos, such as recorded classroom lectures, can be converted into a high-quality 'multimodal textbook' for pretraining vision-language models (VLMs), and that this textbook teaches the models more than the web-crawled interleaved image-text datasets currently used. The authors assemble 22,000 hours of online courses, align keyframes with automatically transcribed and cleaned lecture speech plus on-screen text, and interleave them chronologically. Pretraining LLaVA-1.5 and Idefics2 on this corpus improves seven benchmarks, with the largest gains on knowledge- and reasoning-intensive tasks such as ScienceQA and MathVista. The same pretraining also makes the models far better at exploiting their few-shot context, as shown by a 'cheat test' in which the test question appears in the prompt.

What carries the argument

The load-bearing object is the video-to-textbook pipeline, which converts a lecture video into an ordered sequence of keyframes interleaved with two text streams: refined automatic speech recognition (ASR) of the instructor's narration and OCR of on-screen text, symbols, and formulas. The pipeline starts from an LLM-generated taxonomy of 3,915 knowledge points used to retrieve videos, then filters at video, clip, and keyframe levels: ASR transcripts are rewritten for fluency, clips are kept only when their caption resembles their ASR, keyframes are selected by structural similarity (SSIM) between consecutive frames, and OCR is deduplicated. This machinery creates the corpus's high in-sample image similarity (0.686 versus roughly 0.32 to 0.35 for web datasets), and the resulting sample-level coherence is what the paper credits for the downstream gains.

What would settle it

Run a controlled pretraining experiment in which the Textbook-6.5M samples are truncated or downsampled so that mean images per sample and mean text tokens per sample match MMC4's and OBELICS's; if the ScienceQA and MathVista gains shrink to near zero, the paper's quality argument would not be supported.

Watch

Extended reading notes

Core claim

The central claim is that a video-centric, textbook-style interleaved corpus is a better pretraining substrate than webpage-crawled interleaved corpora for knowledge and reasoning. From 75,000 instructional videos (22,697 hours) the pipeline produces 6.5 million keyframes and 0.75 billion text tokens, organized into 610K samples with an average of 10.7 images and 1,297 tokens per sample. On LLaVA-1.5, continual pretraining on this corpus outperforms equal-sample pretraining on MMC4 and OBELICS across seven VQA and math benchmarks, with ScienceQA improving by over 20 points and MathVista by roughly 5 to 6 points in few-shot settings; on Idefics2, the corpus brings additional gains on math benchmarks. The paper also argues that the corpus's coherent image sequences and tight image-text alignment teach VLMs to attend to interleaved context, demonstrated by near-perfect performance when the answer to the test question is hidden in the few-shot prompt, and by a large drop when image order is shuffled.

Load-bearing premise

The comparison assumes that sampling 610K examples from each corpus and training with identical hyperparameters is a fair control, even though textbook samples contain about twice as many images and three times as many text tokens on average.

Editorial extensions

If this is right

  • If the claim holds, video-derived textbook data can be used in place of or alongside web-crawled interleaved data in VLM pretraining, especially when target tasks are knowledge- or reasoning-heavy.
  • The observed ScienceQA and MathVista gains should transfer to instruction tuning, because the paper shows the pretraining benefit survives the supervised fine-tuning stage.
  • Interleaved-context awareness becomes a trainable capability: models pretrained on coherent frame sequences can exploit few-shot visual and textual evidence more reliably.
  • Image order in an interleaved sample carries information; shuffling the order during pretraining measurably hurts performance, so future corpus design should preserve chronological or pedagogical sequencing.
  • The recipe is reproducible: the paper reports ablations showing ASR refinement, OCR integration, and SSIM keyframe selection each contribute to the gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a testable extension is to ablate corpus quality versus quantity, because textbook samples contain roughly twice as many images and three times as many tokens as MMC4 and OBELICS samples; holding images and tokens per sample constant would separate the effect of content quality from the effect of more data per training example.
  • Beyond the paper: the same pipeline likely transfers to other modalities or languages, for instance non-English lecture videos or video paired with timed subtitles, and it would be useful to measure whether the interleaved-coherence benefit persists there.
  • Beyond the paper: the cheat-test result suggests a new evaluation protocol for interleaved-context awareness in VLMs, one that future model releases could report alongside standard benchmarks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new vision-language pretraining corpus, the "multimodal textbook," built from 75,000 instructional videos (22,697 hours) covering six school subjects. A pipeline uses an LLM-generated taxonomy to retrieve videos, then extracts and refines ASR text, keyframes, and OCR, and interleaves them chronologically into 610K samples containing 6.5M keyframes and 0.75B text tokens. The authors evaluate the corpus by continually pretraining LLaVA-1.5-7B and Idefics2-8B on either Textbook-6.5M or 610K-sample subsamples of MMC4 and OBELICS, reporting gains on ScienceQA, MathVista, OKVQA, and TextVQA, plus analyses of in-context awareness via a "cheat test" and image-order shuffling. The central claim is that video-derived interleaved textbook data transfers better to knowledge- and reasoning-intensive downstream tasks than web-crawled interleaved data under the same training protocol.

Significance. If the central comparison were clean, this would be a valuable contribution: it introduces a large, openly released interleaved dataset from an underexplored source (instructional video), with a reproducible multi-level curation pipeline, transparent statistics, human evaluation, and several ablations of pipeline components (ASR refinement, OCR, keyframe extraction). The cheat test is a clever diagnostic for whether pretraining improves use of in-context interleaved evidence. However, the main experimental evidence for data quality over data quantity is currently confounded, and the absence of a contamination check leaves the headline ScienceQA/MathVista gains incompletely supported. The dataset itself and the pipeline remain useful regardless of the outcome of the controlled comparison.

major comments (4)
  1. [§5.1, Table 1] The sample-matched control does not match the amount of training signal. Table 1 reports that Textbook averages 10.7 images and 1,297 text tokens per sample, while MMC4 averages 5.7 images and 417 tokens and OBELICS averages 2.5 images and 816 tokens. Sampling 610K samples from each corpus therefore gives the Textbook model roughly 1.9x–4.3x more images and 1.6x–3.1x more text tokens than the MMC4/OBELICS controls. Because the pretraining objective is next-token prediction over the interleaved sequence, the gains in Tables 2, 3, and 5 could be driven by larger effective data volume per training step rather than by the asserted higher quality or coherence of the textbook data. The paper should add token- and image-matched controls, e.g., subsample Textbook to match MMC4/OBELICS in total images and text tokens, or report scaling curves that allow the reader to separate quantity from quality.
  2. [§5.2, Table 2] No contamination check is reported, although the corpus is deliberately curated from the same school-subject domains as the evaluation benchmarks. The taxonomy covers mathematics, physics, chemistry, earth science, engineering, and computer science, and the main gains appear on ScienceQA and math benchmarks. Since these benchmarks are public and contain images and questions that could overlap with instructional videos or their ASR/OCR text, the paper should report near-duplicate image detection and n-gram overlap between the pretraining corpus and benchmark samples, or evaluate on a held-out benchmark from a different distribution. Without this, the +20% ScienceQA gain over MMC4 in Table 2 is at risk of being inflated by contamination.
  3. [§5.3, Figure 3] The shuffle experiment shows that Textbook is more sensitive to image order than MMC4/OBELICS, which is evidence of within-sample coherence, but it does not quantify how much of the downstream gain is attributable to that coherence. The figure reports average accuracy after shuffling 20%, 50%, and 100% of samples, yet the comparison still starts from the confounded budgets described above. A matched-budget shuffle comparison (e.g., equal total images and tokens) would directly estimate the contribution of coherent ordering, and should be reported together with the unshuffled matched-budget result.
  4. [§5.4, Table 6] The ablations in Table 6 also use the full Textbook-6.5M corpus without matched budgets, so the conclusions about pipeline components (ASR refinement, OCR, SSIM vs. pixel/CLIP keyframe extraction) are internally valid for the proposed pipeline but do not by themselves establish superiority over web-crawled corpora. This is acceptable for component analysis, but the abstract and Section 5.2 should be reworded to avoid attributing all gains to 'data quality' until the quantity confound is resolved.
minor comments (5)
  1. [Abstract / Section 3.2] The text has several typos and grammatical slips, e.g., 'totoaling' in Section 3.2 and 'asas' in the Table 4 caption; these should be corrected in a revision.
  2. [§5.1 Evaluation Methods] The list of benchmarks reads 'MathVista, MathVision, MathVision'; the second 'MathVision' should likely be 'MathVerse', matching Tables 2 and 3.
  3. [Table 3] The header 'MathVison' is misspelled; it should be 'MathVision'.
  4. [§5.4] The sentence discussing refined ASR perplexity says refined ASR has PPL 13.9, which is still higher than MMC4-Core-ff (12.56) and OBELICS (11.27); the claim that it 'more closely aligns with standard training corpora' is relative to raw ASR (16.86) and should be phrased as such.
  5. [Table 2] The 0-shot ScienceQA column shows '-' for MMC4 and OBELICS; the paper should state whether this is a formatting omission or that these baselines were not evaluated in the 0-shot setting, since the text says improvements are observed in 'zero-shot' settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central claim is an empirical, externally benchmarked comparison, not a derivation from fitted inputs or self-citations.

full rationale

The paper's central claim is that pretraining on the proposed multimodal Textbook-6.5M improves VLM performance on external benchmarks. The evidence is a set of controlled training runs evaluated on fixed public test sets (ScienceQA, MathVista, MathVision, MathVerse, OKVQA, TextVQA); none of these scores is derived by construction from the dataset's own statistics or from a parameter fitted to the benchmark. The InSI-SIM metric is an operational measurement of within-sample image similarity, not an assumption that the textbook is better; it is reported as data and then related to performance through separate experiments such as image-order shuffling. The cheat test is a diagnostic of in-context attention, not a target built into the training objective. Self-citations such as VideoLLaMA2 and Qwen2 are cited as tools used in the curation pipeline, not as evidence that the dataset must be superior; no load-bearing argument reduces to an unverified self-citation or a uniqueness theorem. The main validity concern is the comparison in Section 5.1: samples are matched in number (610K) but not in images or text tokens, and Table 1 shows Textbook samples contain about 10.7 images and 1,297 tokens versus 5.7 and 417 for MMC4 and 2.5 and 816 for OBELICS, so the headline quality attribution is partially confounded with data volume. This is an experimental-control weakness and a possible selection-bias issue given the school-subject overlap with ScienceQA and math benchmarks, but it is not circularity: the benchmark results are not definitionally forced by the dataset construction. The paper also acknowledges residual quality limitations in Section 9, which further shows the claims are treated as empirical rather than self-justifying. Accordingly, no circular step is exhibited and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on data curation choices and domain assumptions rather than fitted numerical parameters or new entities. The main free parameters are pipeline thresholds selected by hand. The key domain assumptions are that instructional videos are coherent and reliable, that ASR/OCR/LLM refinement preserves semantics, that equal sample counts are a fair baseline, and that evaluation benchmarks are not contaminated by the corpus.

free parameters (3)
  • SSIM keyframe threshold T = not reported
    Algorithm 1 uses a hand-chosen threshold T to decide when a frame becomes a keyframe; it directly controls the 6.5M keyframe count and all downstream training.
  • Top-50 videos per knowledge point = 50
    Chosen without sensitivity analysis in Section 3.1; determines which 159,565 videos enter the pipeline and therefore what knowledge the corpus contains.
  • Clip duration window = 10 to 20 seconds
    Used in Section 3.2 to segment long videos into clips; no ablation on this choice is reported.
assumptions (5)
  • domain assumption YouTube instructional videos are a high-quality, coherent source of foundational knowledge.
    Section 3.1 selects videos via a taxonomy and metadata filtering; the entire dataset value depends on this assumption.
  • domain assumption Whisper ASR, Qwen2-72B refinement, and InternVL2 OCR preserve the original semantics while improving fluency.
    Section 3.2 and the ablation in Table 6 assume refined ASR is faithful; if refinement changes meaning, the corpus text is corrupted.
  • ad hoc to paper Sampling 610K samples per corpus with identical hyperparameters is a fair comparison.
    Section 5.1 introduces this control, but Table 1 shows per-sample image and token counts differ by factors of 1.6 to 4, so the assumption is questionable.
  • ad hoc to paper Evaluation benchmarks are not contaminated by the pretraining videos.
    No decontamination or overlap analysis is reported, and ScienceQA plus math VQA cover the same subjects targeted by the Section 3.1 taxonomy.
  • domain assumption The LLM-generated knowledge taxonomy covers the target domains comprehensively.
    Section 3.1 uses GPT-4o to generate 3,915 knowledge points; coverage determines which videos are found and hence the corpus's scope.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining." pith.science (2026). https://pith.science/paper/J3UTHXGX

@misc{pith2026250100958,
  author       = {Pith},
  title        = {Pith review of: 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3UTHXGX}},
  note         = {Machine review of arXiv:2501.00958}
}
read the original abstract

Compared to image-text pair data, interleaved corpora enable Vision-Language Models (VLMs) to understand the world more naturally like humans. However, such existing datasets are crawled from webpage, facing challenges like low knowledge density, loose image-text relations, and poor logical coherence between images. On the other hand, the internet hosts vast instructional videos (e.g., online geometry courses) that are widely used by humans to learn foundational subjects, yet these valuable resources remain underexplored in VLM training. In this paper, we introduce a high-quality \textbf{multimodal textbook} corpus with richer foundational knowledge for VLM pretraining. It collects over 2.5 years of instructional videos, totaling 22,000 class hours. We first use an LLM-proposed taxonomy to systematically gather instructional videos. Then we progressively extract and refine visual (keyframes), audio (ASR), and textual knowledge (OCR) from the videos, and organize as an image-text interleaved corpus based on temporal order. Compared to its counterparts, our video-centric textbook offers more coherent context, richer knowledge, and better image-text alignment. Experiments demonstrate its superb pretraining performance, particularly in knowledge- and reasoning-intensive tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook exhibit outstanding interleaved context awareness, leveraging visual and textual cues in their few-shot context for task solving. Our code are available at https://github.com/DAMO-NLP-SG/multimodal_textbook.

Figures

Figures reproduced from arXiv: 2501.00958 by the authors.

Figure 1
Figure 1. Previous interleaved datasets, e.g., MMC4 and OBELICS, suffer from limitations like weak text-image relations, low knowledge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of constructing a multimodal textbook from instructional videos. We first instruct LLMs to construct a knowledge [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We randomly select 20%, 50%, and 100% samples from [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Top: We plot six subjects along with their corresponding sub-courses. Due to space constraints, we selectively visualized only [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: A case presented in our textbook illustrates the water cycle within the domain of earth science. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: A case presented in our textbook introducing the principles of mechanics within the domain of physics. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: A case presented in our textbook introducing the concepts of velocity and acceleration within the context of physics. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: A case presented in our textbook demonstrates how to solve a question about planar geometry in the domain of mathematics. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: A case presented in our textbook illustrates the concepts of molecules, atoms, and compounds in the domain of chemistry. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: A case presented in our textbook introduces a depth-first search algorithm. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 22 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 2

    arXiv 2024

  2. [2]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S ´ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. 2

    arXiv 2024

  3. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  5. [5]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint arXiv:2308.01390 , 2023. 1, 3, 6

    arXiv 2023

  6. [6]

    MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

    Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, et al. Mint-1t: Scaling open- source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv preprint arXiv:2406.11271, 2024. 1

    work page Pith review arXiv 2024

  7. [7]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 3

    arXiv 2023

  8. [8]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https : / / github . com / kakaobrain/coyo-dataset, 2022. 1

    work page 2022

Show all 65 references
  1. [9]

    Comm: A coherent inter- leaved image-text dataset for multimodal understanding and generation

    Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingt- ing Gao, Yu Wu, and Long Chen. Comm: A coherent inter- leaved image-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2406.10462, 2024. 1

    2024 arXiv

  2. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv prepri...

    2023 arXiv

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3, 5

    2024 arXiv

  4. [12]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 3, 5

    2024 arXiv

  5. [13]

    Infinity-mm: Scaling multimodal per- formance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal per- formance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024. 3

    arXiv

  6. [14]

    Textbooks are all you need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. 2

    2023 arXiv

  7. [15]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos. arXiv preprint arXiv:2501.13826, 2025. 2

    2025 arXiv

  8. [16]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36:72096–721...

    2023

  9. [17]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, S ´ebastien Bubeck, Marah Abdin, Jy- oti Aneja, Sebastien Bubeck, Caio C ´esar Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3, 2023. 2

    2023

  10. [18]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  11. [19]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024. 1

    2024 arXiv

  12. [20]

    Building and better understanding vision- language models: insights and future directions

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 1

    2024 arXiv

  13. [21]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 6, 7

    2024 arXiv

  14. [22]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web- scale filtered dataset of interleaved image-text documents,

  15. [23]

    Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training

    Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, pages 11336– 11344, 2020. 3

    2020

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 3

    2023

  17. [25]

    Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text

    Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shen- glong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhang- wei Gao, Erfei Cui, et al. Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.08418, 2024. 1, 6, 2

    2024 arXiv

  18. [26]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023. 2

    2023 arXiv

  19. [27]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023. 5

    2023 arXiv

  20. [28]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 1, 3

    2023

  21. [29]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 3

    2023

  22. [30]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 3

    2023

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 3, 6

    2024

  24. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  25. [33]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  26. [34]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  27. [35]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 6

    2019

  28. [36]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 1

    2024 arXiv

  29. [37]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision , pages...

    2019

  30. [38]

    True few- shot learning with language models

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few- shot learning with language models. Advances in neural in- formation processing systems, 34:11054–11070, 2021. 3

    2021

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    How2: a large-scale dataset for multimodal language under- standing

    Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Lo ¨ıc Barrault, Lucia Specia, and Florian Metze. How2: a large-scale dataset for multimodal language under- standing. arXiv preprint arXiv:1811.00347, 2018. 2

    2018 arXiv

  33. [41]

    LAION- 400M: open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION- 400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021. 3

    2021 arXiv

  34. [42]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  35. [43]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  36. [44]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun 10 Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  37. [45]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 1

    2024 arXiv

  38. [46]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur´elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langu...

    2023 arXiv

  39. [47]

    Mobile- clip: Fast image-text models through multi-modal reinforced training

    Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobile- clip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15963–...

    2024

  40. [48]

    Pin: A knowledge-intensive dataset for paired and interleaved multimodal documents

    Junjie Wang, Yin Zhang, Yatai Ji, Yuxiang Zhang, Chunyang Jiang, Yubo Wang, Kang Zhu, Zekun Wang, Tiezhen Wang, Wenhao Huang, et al. Pin: A knowledge-intensive dataset for paired and interleaved multimodal documents. arXiv preprint arXiv:2406.13923, 2024. 3

    2024 arXiv

  41. [49]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  42. [50]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5

    2004

  43. [51]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 3, 4

    2024 arXiv

  44. [52]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Pro- ceedings of the AAAI conference on artificial intelligence , pages 3081–3089, 2022. 6

    2022

  45. [53]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 3

    2024 arXiv

  46. [54]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 3

    2024 arXiv

  47. [55]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  48. [56]

    Merlot reserve: Neu- ral script knowledge through vision and language and sound

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yan- peng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neu- ral script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Compu...

    2022

  49. [57]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 3

    2023 arXiv

  50. [58]

    Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks

    Wenqi Zhang, Mengna Wang, Gangao Liu, Xu Huixin, Yi- wei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, et al. Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.21696, 2025. 1

    2025 arXiv

  51. [59]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1

    2023 arXiv

  52. [60]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems...

    2024

  53. [61]

    Implementation Details When synthesizing the Knowledge Taxonomy, we utilize GPT-4o to construct the taxonomy

    Detail of Video-to-Textbook Pipeline 7.1. Implementation Details When synthesizing the Knowledge Taxonomy, we utilize GPT-4o to construct the taxonomy. When filtering video at the metadata level, GPT-4o is also employed to review the metadata of the searched videos. During the...

  54. [62]

    Details of Experiments 8.1. Detail of Evaluation We evaluate the pre-trained VLMs on two VQA bench- marks (TextVQA, OKVQA), a knowledge-centric bench- mark (ScienceQA), and three math-related benchmarks (MathVista, MathVerse, MathVision) under few-shot set- tings. Following th...

  55. [63]

    We will continue to improve the quality and knowledge density of our textbook

    Limitations Although we already designed multiple levels of filtering, our textbook may still contain some redundant keyframes, low-quality texts, and so on. We will continue to improve the quality and knowledge density of our textbook. Besides, similar to prior multimodal mod...

  56. [64]

    Ethical discussion During the collection and release of our multimodal text- book dataset, We are very concerned about ethical consid- erations. In addition to following the established corpora (e.g., MMC4 [60], OBELICS [22] and Omnicorpus [25]), we make additional efforts to ...

  57. [65]

    License and Author Statement We release the dataset under a CC-BY license and Terms of Use that require disclosure of when the dataset is used for the purpose of training models. This license is not in- tended to replace the licenses of the source content, and any use of conte...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.