Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Inference Compute-Optimal Video Vision Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives a recipe for spending a fixed inference budget in video VLMs, showing that the optimal mix of model size, frame count, and tokens per frame shifts as finetuning data grows.

desk verdict Solid empirical scaling study for video VLMs whose joint-scaling conclusion is directly supported by the sweeps, but whose headline data-size elasticity is a model-based prediction that still needs direct validation. read the letter →

arxiv 2505.18855 v1 pith:ISWKVKJO submitted 2025-05-24 cs.CV cs.CL

Peiqi Wang , ShengYun Peng , Xuewen Zhang , Hanchao Yu , Yibo Yang , Lifu Huang , Fujun Liu , Qifan Wang This is my paper · ORCID
classification cs.CVcs.CL
keywords videovisionlanguagemodelsinferencecomputescalinglawscompute-optimalfrontierfinetuningdatasizeframesamplingvisualtokensparametricmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to allocate a fixed per-video inference compute budget among three scaling factors in video vision language models: language model size, number of frames, and visual tokens per frame. Through training sweeps and parametric fits, it claims the compute-optimal configuration scales all three factors jointly, at task-dependent rates, and that increasing finetuning data shifts the optimum toward smaller language models and richer video representations. This matters because inference, not finetuning, dominates cost in large-scale deployment, so a quantitative allocation rule directly guides practical model selection.

What carries the argument

The load-bearing object is the add-interact parametric model of task error, $f(x,n) = \sum_k \alpha_k x_k^{-a_k} + \sum_k \beta_k x_k^{b_k} n^{-d} + \xi n^{-d} + \varepsilon$, fitted in log space to star and isoFLOP sweep data. It couples each scaling factor to finetuning data size through the interaction term $\beta_k x_k^{b_k} n^{-d}$, which is what makes the optimal allocation depend on $n$. The optimization problem is $x^*(c;n) = \arg\min_{x: c(x) \le c} f(x,n)$ with the compute cost model $c(x) = 2x_T(x_M x_W + x_N x_V)$, and the frontier is obtained by brute-force search over the discrete set of feasible configurations.

What would settle it

Train and evaluate video VLMs at configurations on and off the predicted frontier at higher budgets, for example at 60 and 100 TFLOPs with finetuning data sizes of 5M and 10M, and check whether the observed errors match the add-interact extrapolation and whether the optimal allocation still shifts away from language model size as the fitted model predicts.

Watch

Extended reading notes

Core claim

The central claim is that the inference compute-optimal frontier for video VLMs is not achieved by scaling any single factor but by jointly scaling language model size, frame count, and tokens per frame, with the optimal mix depending on the finetuning data size. The paper fits an additive power-law model with interaction terms, the add-interact form, to roughly one hundred training runs from star and isoFLOP sweeps, and then solves a constrained discrete optimization over the allowed configurations. The fitted frontier predicts that as finetuning data grows, the optimal language model size decreases while the optimal number of frames and tokens per frame increase, an elasticity trend that holds on average across eight video tasks.

Load-bearing premise

The fitted add-interact model, trained on roughly one hundred runs within budgets up to 30 TFLOPs and data sizes up to 2M, is assumed to extrapolate reliably to budgets up to 100 TFLOPs and data sizes up to 10M, where the headline frontier and elasticity trends are predicted.

Editorial extensions

If this is right

  • A fixed inference budget should be spent on joint scaling of language model size, frame count, and tokens per frame, not on scaling a single factor in isolation.
  • The compute cost of the vision encoder must be included in the budget; ignoring it overestimates the benefit of adding frames.
  • As finetuning data size increases, the optimal allocation shifts toward more frames and more tokens per frame and away from a larger language model.
  • The optimal allocation is task-dependent, so deployment choices should be tuned to the target video task rather than following a universal recipe.
  • After establishing a strong baseline, further compute is better spent exploring joint scaling configurations than varying one factor at a time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fitted frontier extrapolates reliably, deployment engineers could select model configurations directly from the parametric model, avoiding costly per-task grid searches.
  • The elasticity results suggest that vision-side capacity (frames and tokens) is more data-hungry than language-model capacity, which may generalize to other multimodal inputs such as audio or higher-resolution images.
  • The predictions beyond the tested range, such as budgets up to 100 TFLOPs and data sizes up to 10M, are direct extrapolations of the fitted add-interact model and should be verified with targeted runs at those scales.
  • Because the compute cost model covers only prefill-stage FLOPs and not decoding bandwidth, the frontier may shift for latency-bound or memory-bound serving scenarios.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how to allocate a fixed per-example inference compute budget among three scaling factors of a video VLM—language model size xN, frame count xT, and visual tokens per frame xV—as a function of the finetuning data size n. The authors run a star sweep at n in {0.25, 0.5, 1}M and an isoFLOP sweep at n = 2M for budgets 2–30 TFLOPs, compare several parametric forms for downstream task error, select an add-interact power-law model with bootstrap aggregation, and then solve the constrained discrete optimization problem x*(c;n) = argmin f(x,n) s.t. c(x) <= c by brute force. The main empirical findings are that performance exhibits diminishing returns in each factor, that jointly scaling all three factors is needed for optimality, and that the inferred compute-optimal frontier shifts with n: as finetuning data grows, the model predicts smaller optimal LM size and larger optimal frame count and tokens per frame. The paper also derives task-specific elasticities to quantify this shift and discusses practical implications for deployment.

Significance. The paper addresses a genuinely important and underexplored question: how to choose deployment configurations of video VLMs under inference-compute constraints, accounting for the cost of the vision encoder. Its strengths are substantial: the training sweeps are large (about 100k A100 hours), the model-selection procedure is careful, the isoFLOP sweep serves as a held-out set for extrapolation, bootstrap bagging is used to stabilize the fit, and the code is promised in the abstract. The qualitative joint-scaling result is grounded in directly measured isoFLOP curves and is a useful contribution that goes beyond one-factor-at-a-time ablation studies. The main quantitative novelty—the dependence of the optimal allocation on finetuning data size—is derived from an extrapolating parametric fit rather than from direct measurements, which makes that specific claim less secure than the framing suggests. If validated, the recipe would be practically useful for deployment decisions, and the model-selection methodology is transferable to other scaling settings.

major comments (3)
  1. [§5.3, Eq. (4) and Eq. (3)] The headline data-size result—that x_N* decreases while x_T* and x_V* increase with n—is not directly measured anywhere. The isoFLOP sweep is conducted only at n = 2M (Table 6), and the star sweep varies one factor at a time without holding inference compute fixed. The sign pattern therefore comes entirely from the interaction terms beta_k * x_k^b_k * n^{-d} in Eq. (3), fit to roughly 100 runs. Because x* is a discrete argmin over a small grid, small changes in the fitted exponents can flip the optimal configuration, and Figure 3 (left) shows high bootstrap variance in the parameters. The paper does not propagate this uncertainty to x*(c;n) or to the elasticities in Eqs. (5)–(11); Table 10 reports only point estimates. This is load-bearing for the claim that finetuning data size reshapes the compute-optimal frontier. I request a direct validation: either run isoFLOP sweeps at one or two additional n values and check whether the predicted argmin shifts are observed, or provide bootstrap confidence intervals on e_k and on the argmin and show that the sign pattern is robust.
  2. [Table 2 and Figure 3] The per-task extrapolation is poor precisely for LongVideoBench and Next-QA, with E% >= 5 corresponding to an average deviation exceeding 3 points, as the text states. Nevertheless, Figure 5 reports task-specific elasticities for those same benchmarks. Task-level guidance derived from the fitted model is not credible for tasks whose held-out fit fails this badly. The task-specific elasticity claims should either be restricted to tasks with acceptable extrapolation accuracy or accompanied by explicit error bars and a caveat that the fitted model is not reliable for those tasks.
  3. [§5.2 and Eq. (4)] The validation of the parametric model only checks the predictive accuracy of f on held-out isoFLOP data; it does not check whether the predicted argmin x*(c;n) matches the empirically best configuration in the isoFLOP sweep. Since the practical recipe is the optimal allocation, the relevant diagnostic is the argmin: for each isoFLOP budget at n = 2M, compare the predicted optimal configuration with the best observed configuration and report the performance gap. This would directly test the operational value of the model and would be cheap to compute from already-collected data.
minor comments (5)
  1. [Abstract] The sentence 'While prior works typically focuses on optimizing model efficiency' contains a subject-verb agreement error ('works' should be followed by 'focus').
  2. [§3, after Eq. (2)] The text says 'ak, bk, c∈R are exponent parameters' but the exponent of the data term is d, not c; this appears to be a typo.
  3. [Figure 4] The subplots of the predicted frontier would be easier to interpret if the y-axis scales and the discrete grid points were annotated, since the claimed non-monotonicity stems from the discreteness of X.
  4. [Appendix E] Using a forward difference of a discrete argmin is not a derivative in the usual sense; the paper acknowledges this in the ablation but should state more prominently that e_k is a finite-difference summary over a chosen step size, not an elasticity of a smooth function.
  5. [Limitations] The Limitations section is honest about extrapolation risk to 10x FLOPs but does not mention that the data-size frontier shift is itself an extrapolation beyond the measured range of joint (x, n) variation; adding this caveat would better calibrate reader expectations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the frontier and data-size elasticities are estimated outputs of an empirically fitted performance model, not inputs redefined as predictions.

full rationale

The paper's derivation chain is: collect empirical (x, n, f) points from star and isoFLOP sweeps; select a parametric form f(x, n) by comparing candidate functions on held-out isoFLOP data, with add-interact winning; fit parameters by minimizing log-space MSE; solve the constrained argmin x*(c; n) in Eq. 4 by brute-force search; and compute elasticities e_k from x* via forward differences. Each step is a genuine estimation/derivation step. The headline data-size shift (x_N down, x_T/x_V up) is a consequence of the fitted interaction exponents b_k and d in Eq. 3, but those parameters are estimated from data, and the interaction-containing model was selected because it outperformed additive and multiplicative alternatives on a held-out isoFLOP set (Table 2). The prediction is therefore not equivalent to the model form by construction. The paper does not define f in terms of x*, nor fit parameters to the elasticity itself, nor rely on a self-citation for a load-bearing premise. The skeptic's attack concerns extrapolation beyond the tested region and parameter variance; these are external-validity and robustness risks, not circularity. The Limitations section itself acknowledges the extrapolation uncertainty. No circular step can be exhibited with the required specificity, so the score is 0.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central quantitative claims rest on a fitted additive power-law model (Eq. 3) whose coefficients and exponents are estimated from about 100 training runs, on a simplified FLOP cost model (Eq. 1) that counts only prefill compute, and on hand-chosen analysis settings for elasticity. The model does not introduce new physical entities; all free parameters are statistical or analysis choices. The empirical trends (diminishing returns, joint scaling, task variation) are direct observations, but the headline frontier-shift result is an output of the fitted model.

free parameters (4)
  • add-interact coefficients alpha_k, beta_k, xi, epsilon = estimated from star and isoFLOP sweeps, not reported in text
    Fitted by minimizing log-space MSE (Eq. 7) over about 100 runs; these coefficients define the error surface from which the optimal frontier is derived.
  • add-interact exponents a_k, b_k, d = estimated from sweeps, high bootstrap variance, exact numbers not reported
    Exponents control diminishing returns and the interaction between each scaling factor and data size; b_k is precisely what makes the optimal x depend on n.
  • elasticity averaging grid and step size = |C|=300, |N|=100, max(N)=10M, forward-difference step delta_n=5
    Hand-selected in Appendix E to stabilize elasticity estimates; the reported e_N, e_T, e_V values change with these choices.
  • bagging ensemble size and aggregation method = 100 bootstrap resamples, median aggregation
    Selected in Table 9; improves stability of f and therefore of x*. It is a design choice rather than a physical parameter.
assumptions (7)
  • domain assumption Task error follows an additive power-law with interaction terms: f(x,n) = sum_k alpha_k x_k^-a_k + sum_k beta_k x_k^b_k n^-d + xi n^-d + epsilon.
    Equation (3). This functional form is assumed a priori (from Alabdulmohsin et al. 2023) and then selected among alternatives by fit; if the true error surface is not of this form, the derived frontier is wrong.
  • domain assumption The data exponent d is independent of the scaling factors and shared across them.
    Stated in Section 3: 'we assume d is independent of the scaling factors.' This makes the interaction term separable.
  • standard math Inference FLOPs scale as 2 times the number of parameters per token, and only prefill-stage compute for the vision encoder and LM is counted.
    Equation (1), based on Kaplan et al. (2020). The Limitations section explicitly notes decoding is memory-bandwidth-bound and ignored. If true cost differs materially, x* shifts.
  • domain assumption Visual token length dominates instruction and output token lengths, so their compute is ignored.
    Section 3: 'the length of visual tokens dominate that of the input instructions or output generations.' This may fail for captioning tasks with long outputs.
  • domain assumption The three Llama-3.2 sizes {1B, 2.8B, 7.5B} are sufficient to estimate the LM-size scaling exponent.
    Limitations section: lack of pretrained LM sizes between 8B and 70B may limit the estimated relationship between xN and task performance.
  • domain assumption The balanced average of 8 standardized benchmark metrics is a meaningful optimization target.
    Section 4: metrics are standardized and averaged as Metrics/Avg; the paper acknowledges task-specific variation, so the average is one target among several.
  • domain assumption Finetuning compute cost is negligible relative to inference compute in the target deployment.
    Appendix A.1 computes a cost ratio of about 340x for a TikTok-scale example; this justifies treating n as not part of the compute budget.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inference Compute-Optimal Video Vision Language Models." pith.science (2026). https://pith.science/paper/ISWKVKJO

@misc{pith2026250518855,
  author       = {Pith},
  title        = {Pith review of: Inference Compute-Optimal Video Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ISWKVKJO}},
  note         = {Machine review of arXiv:2505.18855}
}
read the original abstract

This work investigates the optimal allocation of inference compute across three key scaling factors in video vision language models: language model size, frame count, and the number of visual tokens per frame. While prior works typically focuses on optimizing model efficiency or improving performance without considering resource constraints, we instead identify optimal model configuration under fixed inference compute budgets. We conduct large-scale training sweeps and careful parametric modeling of task performance to identify the inference compute-optimal frontier. Our experiments reveal how task performance depends on scaling factors and finetuning data size, as well as how changes in data size shift the compute-optimal frontier. These findings translate to practical tips for selecting these scaling factors.

Figures

Figures reproduced from arXiv: 2505.18855 by the authors.

Figure 1
Figure 1. IsoPerformance Contours. Contours show average task performance as a function of a scaling factor (e.g., xN , xT , or xV ) and finetuning data size n, derived from the star sweep. As detailed in Section 5.1, we construct the star sweep by starting with a inference compute-intensive “center” x ⋆ = (7.5B, 32, 196), varying one factor at a time while keeping the others fixed, and finetuning on different data sizes. For… view at source ↗
Figure 2
Figure 2. IsoFLOP Curves and Compute-Optimal Frontier. IsoFLOP curves (dotted lines) show task performance (color-coded) for models with fixed inference compute cost c(x) across four TFLOP budgets: 2, 5, 15, and 30. The compute-optimal frontier (solid line) connects models with the best average task performance. Both are derived from the isoFLOP sweep described in Section 5.1. The compute-optimal frontier reveals that optimal… view at source ↗
Figure 3
Figure 3. Parametric Fitting of Task Performance. (Left) Box plot of bootstrap-resampled parameter estimates (100 resamples) for the add-interact model (defined in Equation 3) highlights the challenge with fitting a model with just ∼100 examples. (Center) Scatter plot comparing the predicted average task performance (“Metrics/Avg”) with the actual performance for each run in the star and isoFLOP sweeps. add-interact achieves … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Predicted Compute-Optimal Frontier for Video VLMs. The left three subplots show the predicted inference compute-optimal frontier x ∗ (c; n) for key scaling factors x of video VLMs, across varying fine-tuning data sizes n (shades of blue). The blue text indicates the in…
Figure 5
Figure 5. Figure 5: Elasticity Across Tasks. Bar plot showing the elasticity (defined in Equation 11) for scaling fac￾tors k ∈ {N, T, V } across video tasks. This measures the sensitivity of optimal scaling factors x ∗ k to changes in data size n. While there is significant task-specific …
Figure 6
Figure 6. Figure 6: % FLOPs Increase From Adding Vision Model Compute. Contours illustrates the percentage increase in inference FLOPs, when accounting for the compute cost of the vision model, as a function of language model size xN and the number of tokens per frame xV . The vision enco…
Figure 7
Figure 7. Figure 7: Empirical Data from Star and IsoFLOP Sweeps. We show the average task performance across different runs of the star and isoFLOP sweeps. The data gathered from sweeps is used for visualization and parametric fitting of scaling curves. In this paper, we conduct two types…
Figure 8
Figure 8. Figure 8: Task-Specific IsoPerformance Contours [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Task-Specific isoFLOPs Curves and Compute-Optimal Frontier [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: Task-Specific Predicted Compute-Optimal Frontier [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 23 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2023. Getting ViT in Shape : Scaling Laws for Compute-Optimal Model Design . In Thirty-Seventh Conference on Neural Information Processing Systems

    work page 2023

  4. [4]

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2024. https://doi.org/10.1073/pnas.2311878121 Explaining neural scaling laws . Proceedings of the National Academy of Sciences

    doi:10.1073/pnas.2311878121 2024

  5. [5]

    Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. 2021. https://doi.org/10.1109/ICCV48922.2021.00175 Frozen in Time : A Joint Video and Image Encoder for End-to-End Retrieval . In 2021 IEEE / CVF International Conference on Computer Vision ( ICCV )

    arXiv 2021

  6. [6]

    Le, Christopher R \'e , and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \'e , and Azalia Mirhoseini. 2024. https://doi.org/10.48550/arXiv.2407.21787 Large Language Monkeys : Scaling Inference Compute with Repeated Sampling . Preprint, arXiv:2407.21787

    arXiv doi:10.48550/arxiv.2407.21787 2024

  7. [7]

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal , Jenq-Neng Hwang, Saining Xie, and Christopher D. Manning. 2024. AuroraCap : Efficient , Performant Video Detailed Captioning and a New Benchmark . In The Thirteenth International Conference on Learning Representations

    work page 2024

  8. [8]

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, Aj Piergiovanni, Matthias Mi...

    arXiv 2024

Show all 47 references
  1. [9]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3 .int8(): 8-bit Matrix Multiplication for Transformers at Scale . In Advances in Neural Information Processing Systems

    2022

  2. [10]

    Yifan Du, Yuqi Huo, Kun Zhou, Zijia Zhao, Haoyu Lu, Han Huang, Xin Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. 2024. Exploring the Design Space of Visual Context Representation in Video MLLMs . In The Thirteenth International Conference on Learning Representations

    2024

  3. [11]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. https://doi.org/10.48550/a...

    2024 doi arXiv

  4. [12]

    Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang W...

    2024 doi arXiv

  5. [13]

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag , Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. https://doi.org/10.1109/ICCV.2...

    2017 doi

  6. [14]

    Aaron Grattafiori and et al. 2024. https://doi.org/10.48550/arXiv.2407.21783 The Llama 3 Herd of Models . Preprint, arXiv:2407.21783

    2024 doi arXiv

  7. [15]

    Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish...

    2020 doi arXiv

  8. [16]

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. https://doi.org/10.48550/arXiv.2102.01293 Scaling Laws for Transfer . Preprint, arXiv:2102.01293

    2021 doi arXiv

  9. [17]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas , Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche , Bogdan Damoc, Aurelia Guy, Simon Os...

    2022

  10. [18]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://doi.org/10.48550/arXiv.2001.08361 Scaling Laws for Neural Language Models . Preprint, arXiv:2001.08361

    2020 doi arXiv

  11. [19]

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. https://doi.org/10.48550/arXiv.1705.06950 The Kinetics Human Action Video Dataset ....

    2017 doi arXiv

  12. [20]

    Li Kunchang, He Yinan, Wang Yi, Li Yizhuo, Wang Wenhai, Luo Ping, Wang Yali, Wang Limin, and Qiao Yu. 2025. https://doi.org/10.1007/s11432-024-4321-9 VideoChat : Chat-Centric Video Understanding . SCIENCE CHINA Information Sciences

    2025 doi

  13. [21]

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Advances in Neural Information Processing Systems

    2024

  14. [22]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding . In Proceedings of the 40th International Conference on Machine Learning

    2023

  15. [23]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . LLaVA-OneVision : Easy Visual Task Transfer . Transactions on Machine Learning Research

    2024

  16. [24]

    Semedo, and J

    Kevin Li, Sachin Goyal, Jo \ a o D. Semedo, and J. Zico Kolter. 2024 b . Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters . In The Thirteenth International Conference on Learning Representations

    2024

  17. [25]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024 c . MVBench : A Comprehensive Multi-modal Video Understanding Benchmark . In Proceedings of the IEEE / CVF Conference on Computer Vision and Patt...

    2024

  18. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning . Advances in Neural Information Processing Systems

    2023

  19. [27]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.679 Video- ChatGPT : Towards Detailed Video Understanding via Large Vision and Language Models . In Proceedings of the 62nd Annual Meeting of the Association for Com...

    2024 doi

  20. [28]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024 b . https://arxiv.org/abs/2406.09418 VideoGPT +: Integrating Image and Video Encoders for Enhanced Video Understanding . Preprint, arXiv:2406.09418

    2024 arXiv

  21. [29]

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Hongyu H \`e , Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang...

    2024 doi

  22. [30]

    David Owen. 2024. https://doi.org/10.48550/arXiv.2401.04757 How predictable is language model benchmark performance? Preprint, arXiv:2401.04757

    2024 doi arXiv

  23. [31]

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens Continente, Larisa Markeeva, Dylan Sunil Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fr \'e chette, Hanna Klimczak...

    2023

  24. [32]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision . In Proceeding...

    2021

  25. [33]

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. Beyond Chinchilla-optimal : Accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning

    2024

  26. [34]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning . In The Thirteenth International Conference on Learning Representations

    2024

  27. [35]

    Mingxing Tan and Quoc Le. 2019. EfficientNet : Rethinking Model Scaling for Convolutional Neural Networks . In Proceedings of the 36th International Conference on Machine Learning

    2019

  28. [36]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. https://doi.org/10.48550/arXiv.2409.1219...

    2024 doi arXiv

  29. [37]

    Dai, and Quoc V

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models are Zero-Shot Learners . In International Conference on Learning Representations

    2022

  30. [38]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024 a . LongVideoBench : A Benchmark for Long-context Interleaved Video-Language Understanding . In 38th Conference on Neural Information Processing Systems ( NeurIPS 2024) Track on Datasets and Benchmarks

    2024

  31. [39]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024 b . Inference Scaling Laws : An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving . In The Thirteenth International Conference on Learning Representations

    2024

  32. [40]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. https://doi.org/10.1109/CVPR46437.2021.00965 NExT-QA : Next Phase of Question-Answering to Explaining Temporal Actions . In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )

    2021

  33. [41]

    Tenenbaum

    Kexin Yi*, Chuang Gan*, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2019. CLEVRER : Collision Events for Video Representation and Reasoning . In International Conference on Learning Representations

    2019

  34. [42]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. https://doi.org/10.1609/aaai.v33i01.33019127 ActivityNet-QA : A Dataset for Understanding Complex Web Videos via Question Answering . Proceedings of the AAAI Conference on Artificial Intelligence

    2019 doi

  35. [43]

    Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2024. Inference Scaling for Long-Context Retrieval Augmented Generation . In The Thirteenth International Conference on Learning Representations

    2024

  36. [44]

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2023. When Scaling Meets LLM Finetuning : The Effect of Data , Model and Finetuning Method . In The Twelfth International Conference on Learning Representations

    2023

  37. [45]

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2024 a . https://doi.org/10.48550/arXiv.2407.12772 LMMs-Eval : Reality Check on the Evaluation of Large Multimodal Models . Pre...

    2024 doi arXiv

  38. [46]

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander G Hauptmann, Yonatan Bisk, and Yiming Yang. 2025. Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward . In Proceedings of the 202...

    2025

  39. [47]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024 b . https://arxiv.org/abs/2410.02713 Video Instruction Tuning With Synthetic Data . Preprint, arXiv:2410.02713

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.