Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Everything is a Video: Unifying Modalities through Next-Frame Prediction

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single transformer can handle text, image, video, and audio by predicting the next rendered frame.

desk verdict A clean proof of concept that overclaims in the abstract: the experiments are per-task, so the 'single model' claim is untested. read the letter →

arxiv 2411.10503 v2 pith:6DPQ633Y submitted 2024-11-15 cs.CV cs.CLcs.LG

G. Thomas Hudson , Dean Slack , Thomas Winterbottom , Jamie Sterling , Chenghao Xiao , Junjie Shentu , Noura Al Moubayed This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords next-framepredictiontaskreformulationmultimodallearningvideorenderedtextspectrograminputpuretransformerunifiedmodalityinterface
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that every modality—text, images, audio, and video—can be converted into a sequence of 64x64 RGB frames, and that a single transformer trained to predict the next frame can then solve tasks across all of them without any modality-specific encoders. Text is rendered token-by-token in a fixed-width font, audio is collapsed into a spectrogram frame, and images or video are resized to the same frame size; outputs are also frames, with text answers later recovered by OCR. On seven datasets spanning sentiment classification, image classification, video action recognition, video QA, object tracking, colorization, and spoken-digit recognition, the model reaches accuracy comparable to single-task baselines trained without pretraining. The point is not to beat state-of-the-art systems but to show that a shared next-frame interface can carry real multimodal learning, a step toward foundation models that treat every input as video.

What carries the argument

The central object is the reformulation itself: every task becomes a video of 64x64 RGB frames, with inputs on the left of a separator frame and the target output as the frames to predict. The model is a pure transformer with 41M parameters, adapted from ViT and TimesFormer: input frames are split into non-overlapping patches, local space-time attention blocks operate at full resolution, patch-merge operations halve resolution, and a global space-time block runs at reduced resolution, then the process is reversed to produce the next frame; causal temporal masking ensures only past frames are visible. Training uses a multi-scale SSIM loss, and text answers are recovered at inference by tesseract OCR followed by nearest-vocabulary matching. What this machinery does is force all modalities through one pixel-level generative bottleneck, so knowledge transfer between tasks must happen in the shared frame space rather than through modality-specific features.

What would settle it

A concrete test: train the same model on AudioMNIST with 128x128 spectrograms instead of 64x64, keeping all other settings fixed; if accuracy improves materially, the fixed-resolution rendering—not the next-frame interface—is discarding acoustic information the task needs.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that next-frame prediction is a sufficient universal interface for multimodal tasks. The authors reformulate text-to-text, image-to-text, video-to-text, video QA, video-to-video, and audio-to-text tasks as sequences of 64x64 RGB frames: each input token, image, or video frame becomes one frame of a 'video', a separator frame marks the end of input, and the target (a label, an answer, a colored frame, a bounding box overlay) is the next frame or frames to predict. A single 41M-parameter pure transformer, with causal spatiotemporal attention, patch merging, and an SSIM training loss, learns each task from scratch without pretrained encoders or embeddings; text outputs are read off with OCR and matched to a fixed vocabulary. The strongest evidence is the accuracy table: 89.1 on CIFAR-10, 97.1 on AudioMNIST, 52.5 on CLEVRER descriptive QA, and IoU 0.63 on LaSOT tracking, each close to or above strong non-pretrained baselines. The authors frame this as establishing, for the first time, that a single visual-input paradigm can represent and solve these modalities, leaving joint multitask training and scale to future work.

Load-bearing premise

The load-bearing premise is that rendering is information-preserving: text in a fixed-width font, audio in a 64x64 spectrogram, and images resized to 64x64 all keep the information a task needs, and that OCR can read the model's generated text frames back accurately.

Editorial extensions

If this is right

  • A single model without modality-specific encoders can reach near-baseline accuracy on text, image, video, and audio tasks when each is reformulated as next-frame prediction.
  • New modalities or tasks can be added by defining a rendering that turns their inputs and outputs into frames, without changing the architecture or training objective.
  • Because the model is trained with a simple pixel loss, unstructured pretraining on raw video (or rendered text and spectrograms) could serve as a foundation-model pretraining stage for all modalities.
  • The approach removes the need for separate text tokenization and speech front-ends, replacing them with one visual interface, at the cost of an OCR step for reading outputs.
  • Task reformulation, already standard in NLP as prompt-based learning, is shown to extend to a non-linguistic supertask (next-frame prediction) spanning several modalities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 64x64 rendering resolution is likely the main capacity bottleneck: text longer than about 20 tokens is truncated, and audio is compressed into one spectrogram frame, so the method's ceiling on information-dense tasks may be set by the rendering, not by the next-frame interface itself.
  • A natural testable extension is joint multitask training: if the shared frame space truly enables transfer, training on several reformulated tasks together should improve per-task accuracy, especially on small datasets like TinyVIRAT.
  • The OCR readout is an external dependency; replacing it with a learned decoder that maps frames to text (or predicting character-level frames directly) would make the pipeline fully end-to-end and might remove vocabulary-matching errors.
  • Because everything is a frame, the paradigm could in principle absorb instruction-following by rendering prompts and answers as video, making a single next-frame model a multimodal analogue of a chat model; the paper does not demonstrate this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a multimodal task-reformulation framework in which text, image, video, and audio tasks are converted into a common 64x64 RGB-video next-frame prediction problem. Text is rendered token-by-token in a fixed-width font, images are resized, audio is presented as a spectrogram frame, and a separator frame marks the boundary between input and output. A pure transformer-based video prediction model with 41M parameters is trained with an SSIM loss on each dataset separately, and outputs are evaluated by OCR with vocabulary matching for text tasks, box extraction for tracking, and standard image metrics for colorization. Results are reported on SST-2, CIFAR-10, TinyVIRAT classification and colorization, AudioMNIST, CLEVRER, TGIF-QA, and LaSOT.

Significance. If the central 'single model' claim were supported, the paper would offer a notable simplification: one architecture, one training objective, and one interface across several modalities, with no modality-specific encoders and no external pretraining. The per-task feasibility results are real evidence that the next-frame interface can support a range of tasks at non-trivial accuracy, and the attention-map analysis is a useful qualitative check that the model is not merely copying input frames. The authors are also transparent about the scope of their aim (Section 5.2), explicitly disclaiming state-of-the-art ambitions. However, the headline claim of a single shared model is not tested: every Table 2 result comes from an independently trained checkpoint, and the paper leaves joint training to future work. For this reason, the contribution as written is a proof of concept for a shared architecture and task reformulation, not a demonstration of a unified model or of cross-task knowledge transfer.

major comments (3)
  1. [§3.3 and §4; Table 2] Section 3.3 states 'We train our model on each dataset independently,' and Section 4 repeats that joint training is left as future work. Every result in Table 2 therefore comes from a separately initialized and separately trained checkpoint, so the abstract's claim that the reformulation allows 'a single model' to solve text, image, video, and audio tasks is not tested. The architecture is shared, but the weights are not. The paper also does not specify how one shared set of weights would disambiguate tasks whose input formats overlap: for example, a TinyVIRAT sequence is used both for action classification and for colorization, and a CIFAR-10 frame could be mapped to a class label or a caption. The separator token only marks the input/output boundary. A concrete remedy is to train one model jointly on at least two tasks with identical input formats but different target formats (e.g., TinyVIRAT classification and colorization) and report per-task accuracy with and without an explicit task token; until then the central 'single model' claim should be softened to 'shared architecture.'
  2. [§3.1 and Table 2 (CLEVRER row)] The reported CLEVRER accuracy of 52.5 is obtained on a filtered 'descriptive' subset with videos subsampled to four 64x64 frames, whereas the LSTM and LSTM+CNN baselines cited from [37] are standard full-video results; the manuscript does not state whether those baselines were evaluated on the same subset or with the same subsampling. The claim that the model 'surpasses' the LSTM+CNN baseline is therefore not an apples-to-apples comparison. In addition, no error bars, standard deviations, or repeated-seed results are given for any row of Table 2, so the reported margins over several baselines (e.g., TGIF-QA 53.2 vs. 51.9) are not established as significant. Please report baseline numbers under the identical filtered/subsampled protocol and provide variance estimates.
  3. [§3.1 and §4 (OCR decoding)] The text-output numbers depend on tesseract OCR followed by matching to the closest vocabulary word, but the manuscript reports no analysis of how often OCR corrupts the rendered output or how much the vocabulary-matching step inflates accuracy. The Related Work claim that text and audio can be represented in images 'with no loss of information' is too strong given the actual renderings: space/punctuation tokenization rendered in a fixed-width font and audio collapsed to a single 64x64 spectrogram are lossy choices. Because information preservation is the load-bearing premise for treating next-frame prediction as a universal interface, please add a control experiment measuring OCR accuracy on rendered ground-truth labels and, for audio, classification accuracy from the spectrogram representation alone, or otherwise quantify the information retained by the rendering.
minor comments (5)
  1. [Figures 4 and 5] The Figure 4 caption uses 'Tinyvirat' instead of 'TinyVIRAT,' and there is a stray 'ch' just before the Section 5.1 heading; both should be corrected.
  2. [Related Work] There are typos in the Related Work section: 'modalitiy' should be 'modality' and 'spura-visual' should be 'supra-visual' or similar.
  3. [Section 4 (CDC equation)] In the CDC equation, the notation 'Pc(I I+t)' should be 'Pc(I_{i+t})', and the Jensen-Shannon divergence should be defined or cited explicitly rather than left implicit.
  4. [Section 5 (colorization)] The colorization paragraph reports CDC values of 0.0169 for the model and 0.00522 for the original dataset, but Table 2 lists 0.02 for the model and no original-dataset CDC row; these numbers should be reconciled.
  5. [Section 5 (SST-2)] The SST-2 discussion says that 'limiting the evaluation to shorter inputs 20 tokens or fewer increases the F1-score to 80.0,' but it is unclear whether 80.0 is computed on the truncated subset and whether Table 2's 76.8 is the full-test number; this should be stated explicitly.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reformulation is definitional but learned end-to-end; the untested single-model claim is a scope gap rather than a circular step.

full rationale

The paper does not derive its central results from a fitted parameter or a self-citation chain. The reformulation of tasks as next-frame prediction is definitional: inputs and outputs are rendered as 64x64 RGB frames, a separator token is inserted, and the model is trained end-to-end with an SSIM loss to predict the output frames. Evaluation is performed on held-out data by OCR or bounding-box extraction (Section 4), so reported accuracies reflect genuine generalization, not reconstruction of the training objective. The claim that a single transformer can handle all modalities is not actually tested, since Section 3.3 states 'We train our model on each dataset independently' and Section 4 states 'we leave training on all the tasks jointly in a multitask setting as future work.' That is an evidential scope gap, not circularity: the per-task experiments still support the weaker claim that the shared architecture can be trained across modalities. The only overlapping-author citation is [34] (Xiao et al.), used as background for rendering text as images; it is not load-bearing for the paper's main results and no uniqueness theorem is imported from it. Consequently, no circular step can be quoted or exhibited, and the appropriate finding is no significant circularity with a minor non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper's central claim rests on five domain assumptions: text and audio can be rendered into frames without losing task-relevant information, OCR can decode outputs, pixel-level SSIM training is sufficient, and the transformer can learn semantic mappings from rendered frames. These are not derived or validated independently; they are built into the method. No new physical or theoretical entities are introduced, and the only free parameters are architectural and optimization hyperparameters plus per-task sequence lengths.

free parameters (5)
  • Embedding dimension K = 512
    Set for all tasks; not derived from the task or data.
  • Patch size = 8x8 (implied by Figure 9 caption)
    Controls tokenization and the resolution at which global attention operates.
  • Learning rate = 3e-4
    Constant learning rate for all tasks, chosen by hand.
  • Dropout = 0.1
    Set for all layers except the final output layer.
  • Per-task input/target lengths and batch sizes = Table 1 values
    Hand-set for each task; these change the amount of context and the difficulty of next-frame prediction.
assumptions (6)
  • domain assumption Rendered fixed-width font text frames preserve all semantic content of the tokens
    Section 3.1 uses this to replace tokenization; if rendering loses word order or characters, text tasks fail.
  • domain assumption A 64x64 spectrogram preserves enough audio information for digit classification
    Section 3.1 AudioMNIST; spectrograms are lossy (phase is discarded) and downsampled to 64x64.
  • domain assumption OCR (tesseract) plus vocabulary matching recovers the model's intended text outputs
    Section 4; OCR errors are treated as postprocessing, so final metrics depend on OCR quality.
  • domain assumption SSIM pixel loss is a sufficient training signal for solving classification and QA tasks
    Section 3.3; the model is trained only to match pixels, with no task-level loss, assuming pixel fidelity aligns with semantic correctness.
  • domain assumption A pure transformer with local/global space-time attention can learn the input-output mapping from rendered frames
    Section 3.2; no pretraining, so the architecture must discover text reading and audio understanding from pixels alone.
  • standard math Standard transformer, backpropagation, and AdamW machinery
    Background assumed without proof; standard in deep learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Everything is a Video: Unifying Modalities through Next-Frame Prediction." pith.science (2026). https://pith.science/paper/6DPQ633Y

@misc{pith2026241110503,
  author       = {Pith},
  title        = {Pith review of: Everything is a Video: Unifying Modalities through Next-Frame Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DPQ633Y}},
  note         = {Machine review of arXiv:2411.10503}
}
read the original abstract

Multimodal learning, which involves integrating information from various modalities such as text, images, audio, and video, is pivotal for numerous complex tasks like visual question answering, cross-modal retrieval, and caption generation. Traditional approaches rely on modality-specific encoders and late fusion techniques, which can hinder scalability and flexibility when adapting to new tasks or modalities. To address these limitations, we introduce a novel framework that extends the concept of task reformulation beyond natural language processing (NLP) to multimodal learning. We propose to reformulate diverse multimodal tasks into a unified next-frame prediction problem, allowing a single model to handle different modalities without modality-specific components. This method treats all inputs and outputs as sequential frames in a video, enabling seamless integration of modalities and effective knowledge transfer across tasks. Our approach is evaluated on a range of tasks, including text-to-text, image-to-text, video-to-video, video-to-text, and audio-to-text, demonstrating the model's ability to generalize across modalities with minimal adaptation. We show that task reformulation can significantly simplify multimodal model design across various tasks, laying the groundwork for more generalized multimodal foundation models.

Figures

Figures reproduced from arXiv: 2411.10503 by the authors.

Figure 1
Figure 1. Sample outputs from our next frame prediction model across multiple modalities. The model receives frames to the left of the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Truncated example of the SST2 sentiment dataset ren [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Truncated example of the CIFAR10 image classification [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Truncated example of the CLEVRER task rendered as a [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Truncated example of the colorization task rendered as a [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Truncated example of the LaSOT object tracking task [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 8
Figure 8. Figure 8: Example of the AudioMNIST audio classification [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 9
Figure 9. Figure 9: Illustration of the Transformer-based video prediction model used in this work. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Attention visualisations showing where the model attends to both spatially (indicated by light/dark areas overlaid on the [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Spatio-temporal foundation models are organized into a pipeline of data harmonization, model design, training, and adaptation, with a data property taxonomy for model selection.

Reference graph

Works this paper leans on

41 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [37]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. 4, 6, 7

    arXiv 1910

  2. [1]

    From methods to datasets: A survey on image-caption generators

    Lakshita Agarwal and Bindu Verma. From methods to datasets: A survey on image-caption generators. Multime- dia Tools and Applications, 83:28077–28123, 2023. 1

    work page 2023

  3. [2]

    data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. ArXiv, abs/2202.03555, 2022. 2

    arXiv 2022

  4. [3]

    Audiomnist: Exploring explainable artificial intelli- gence for audio analysis on a simple benchmark

    S ¨oren Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert M ¨uller, Sebastian Lapuschkin, and Wojciech Samek. Audiomnist: Exploring explainable artificial intelli- gence for audio analysis on a simple benchmark. Journal of the Franklin Institute, 361(1):418–428, 2024. 4, 6, 7

    work page 2024

  5. [4]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 5

    work page 2021

  6. [5]

    Pcanet: A simple deep learning baseline for image classification? IEEE transactions on image pro- cessing, 24(12):5017–5032, 2015

    Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma. Pcanet: A simple deep learning baseline for image classification? IEEE transactions on image pro- cessing, 24(12):5017–5032, 2015. 6, 7

    work page 2015

  7. [6]

    Generative pre- training from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020. 3

    2020

  8. [7]

    Uniter: Learning universal image-text representations

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. ArXiv, abs/1909.11740, 2019. 2

    arXiv 1909

Show all 41 references
  1. [8]

    High-performance long- term tracking with meta-updater

    Kenan Dai, Yunhua Zhang, Dong Wang, Jianhua Li, Huchuan Lu, and Xiaoyun Yang. High-performance long- term tracking with meta-updater. In CVPR, 2020. 6

    2020

  2. [9]

    Tinyvirat: Low-resolution video action recognition

    Ugur Demir, Yogesh S Rawat, and Mubarak Shah. Tinyvirat: Low-resolution video action recognition. In 2020 25th in- ternational conference on pattern recognition (ICPR), pages 7387–7394. IEEE, 2021. 4, 7

    2020

  3. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  4. [11]

    Lasot: A high-quality benchmark for large-scale single ob- ject tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single ob- ject tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

  5. [12]

    Improving language understanding from screenshots

    Tianyu Gao, Zirui Wang, Adithya Bhaskar, and Danqi Chen. Improving language understanding from screenshots. arXiv preprint arXiv:2402.14073, 2024. 3

    2024 arXiv

  6. [13]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, 2023. 3

    2023

  7. [14]

    Unit: Multimodal mul- titask learning with a unified transformer

    Ronghang Hu and Amanpreet Singh. Unit: Multimodal mul- titask learning with a unified transformer. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1419–1429, 2021. 2

    2021

  8. [15]

    Video Question Answering with Spatio-Temporal Reasoning

    Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Video Question Answering with Spatio-Temporal Reasoning. IJCV, 2019. 4, 6

    2019

  9. [16]

    Unifying question answering, text classi- fication, and regression via span extraction

    Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering, text classi- fication, and regression via span extraction. arXiv preprint arXiv:1904.09286, 2019. 2

    1904 arXiv

  10. [17]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

    2009

  11. [18]

    Temporally consistent video colorization with deep feature propagation and self-regularization learning

    Yihao Liu, Hengyuan Zhao, Kelvin CK Chan, Xintao Wang, Chen Change Loy, Yu Qiao, and Chao Dong. Temporally consistent video colorization with deep feature propagation and self-regularization learning. Computational Visual Me- dia, 10(2):375–395, 2024. 6

    2024

  12. [19]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 5

    2017

  13. [20]

    The multi-modal fusion in vi- sual question answering: a review of attention mechanisms

    Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. The multi-modal fusion in vi- sual question answering: a review of attention mechanisms. PeerJ Computer Science, 9:e1400, 2023. 1

    2023

  14. [21]

    The natural language decathlon: Multitask learning as question answering

    Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. 2

    2018 arXiv

  15. [22]

    Transframer: Arbitrary frame prediction with generative models

    Charlie Nash, Joao Carreira, Jacob Walker, Iain Barr, An- drew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09494, 2022. 2, 5

    2022 arXiv

  16. [23]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. 2

    2024

  17. [24]

    Video (lan- guage) modeling: a baseline for generative models of natural videos

    Marc’Aurelio Ranzato, Arthur Szlam, Joan Bruna, Micha ¨el Mathieu, Ronan Collobert, and Sumit Chopra. Video (lan- guage) modeling: a baseline for generative models of natural videos. ArXiv, abs/1412.6604, 2014. 3

    2014 arXiv

  18. [25]

    Language modelling with pixels

    Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam de Lhoneux, and Desmond El- liott. Language modelling with pixels. arXiv preprint arXiv:2207.06991, 2022. 3

    2022 arXiv

  19. [26]

    Implicit stacked autoregressive model for video pre- diction

    Min seok Seo, Hakjin Lee, Do-Yeon Kim, and Junghoon Seo. Implicit stacked autoregressive model for video pre- diction. ArXiv, abs/2303.07849, 2023. 3

    2023 arXiv

  20. [27]

    FLA V A: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLA V A: A foundational language and vision alignment model. CoRR, abs/2112.04482, 2021. 2

    2021 arXiv

  21. [28]

    Recursive deep models for semantic compositional- ity over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositional- ity over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro...

    2013

  22. [29]

    Pixar: Auto-regressive language modeling in pixel space

    Yintao Tai, Xiyang Liao, Alessandro Suglia, and Antonio Vergari. Pixar: Auto-regressive language modeling in pixel space. arXiv preprint arXiv:2401.03321, 2024. 3

    2024 arXiv

  23. [30]

    van Amersfoort, Anitha Kannan, Marc’Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala

    Joost R. van Amersfoort, Anitha Kannan, Marc’Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala. Transformation-based models of video sequences. ArXiv, abs/1701.08435, 2017. 3

    2017 arXiv

  24. [31]

    Multimodal llm enhanced cross- lingual cross-modal retrieval

    Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Multimodal llm enhanced cross- lingual cross-modal retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8296– 8305, 2024. 1

    2024

  25. [32]

    Bovik, H.R

    Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4): 600–612, 2004. 5

    2004

  26. [33]

    Visual question answer- ing: A survey of methods and datasets

    Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Visual question answer- ing: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21–40, 2017. 1

    2017

  27. [34]

    Pixel sentence repre- sentation learning

    Chenghao Xiao, Zhuoxu Huang, Danlu Chen, G Thomas Hudson, Yizhi Li, Haoran Duan, Chenghua Lin, Jie Fu, Jun- gong Han, and Noura Al Moubayed. Pixel sentence repre- sentation learning. arXiv preprint arXiv:2402.08183, 2024. 3

    2024 arXiv

  28. [35]

    Videogpt: Video generation using vq-vae and trans- formers, 2021

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers, 2021. 3

    2021

  29. [36]

    Colormnet: A memory-based deep spatial-temporal feature propagation network for video colorization

    Yixin Yang, Jiangxin Dong, Jinhui Tang, and Pan Jinshan. Colormnet: A memory-based deep spatial-temporal feature propagation network for video colorization. In ECCV, 2024. 6

    2024

  30. [38]

    Akin Yilmaz and Ahmet Murat Tekalp

    M. Akin Yilmaz and Ahmet Murat Tekalp. Dfpn: De- formable frame prediction network. 2021 IEEE Interna- tional Conference on Image Processing (ICIP), pages 1944– 1948, 2021. 3

    2021

  31. [39]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. 6

    2016

  32. [40]

    Bag of tricks for ef- fective language model pretraining and downstream adapta- tion: A case study on glue.arXiv preprint arXiv:2302.09268,

    Qihuang Zhong, Liang Ding, Keqin Peng, Juhua Liu, Bo Du, Li Shen, Yibing Zhan, and Dacheng Tao. Bag of tricks for ef- fective language model pretraining and downstream adapta- tion: A case study on glue.arXiv preprint arXiv:2302.09268,

    arXiv

  33. [41]

    A survey on vqa: Datasets and approaches

    Yeyun Zou and Qiyu Xie. A survey on vqa: Datasets and approaches. In 2020 2nd International Conference on Infor- mation Technology and Computer Application (ITCA), pages 289–297. IEEE, 2020. 1

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.