Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Optimizing GPT for Video Understanding: Zero-Shot Performance and Prompt Engineering

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt design alone—compressing a 4,023-word moderation policy to 96 words and splitting clickbait into eight scored subcategories—raises zero-shot GPT-4o video classification to near-production level without fine-tuning.

desk verdict Useful applied benchmark with one solid descriptive finding, but the two headline prompt-engineering gains are not yet supported because the regression weights are fit on the evaluation set and the shortened policy is undisclosed. read the letter →

arxiv 2502.09573 v3 pith:IJ7K5J56 submitted 2025-02-13 cs.CV cs.CLcs.LG

Mark Beliaev , Victor Yang , Madhura Raju , Jiachen Sun , Xinghai Hu This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords zero-shotvideoclassificationGPT-4opromptengineeringpolicycompressiondecomposition-aggregationcontentmoderationfeedqualitymultimodalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This is an empirical study of whether GPT-4o, used as a zero-shot vision-language classifier, can handle real video content moderation across seven feed-quality categories defined by written policies. The paper's central claim is that prompt design alone, not fine-tuning, can substantially improve the model's accuracy. Two results carry the claim: compressing the 4,023-word Sensitive & Mature policy to 96 words reduced false negatives by 0.07 at only 0.01 more false positives, raising AUC from 0.73 to 0.79, and splitting clickbait detection into eight subcategories with separate scores, aggregated by mean, max, or linear regression, beat the single-prompt method, with the fitted aggregate outperforming the production model at high precision. The stakes for a reader are practical: if these effects are real, a platform can get meaningfully better moderation by rewriting instructions more tersely, at no training cost.

What carries the argument

The mechanism is the prompt itself: a system-level task description, the category policy, and an output contract that demands a JSON reply with reasoning and a 0-100 score, plus two prompt transformations applied to it. Policy compression rewrites a long, detailed moderation rule into a short statement of intent (4,023 words to 96 in the Sensitive case), which the analysis argues changes GPT-4o from conservative low-score behavior, with few false positives but many false negatives, to aggressive high-coverage behavior. Decomposition-aggregation splits one complex category into the subcategories named in its policy, collects an independent score for each, and combines them by mean, max, or fitted linear regression to form the final prediction. Both transformations operate on the instruction text only; the model weights, frames, and input features are untouched.

What would settle it

Run a preregistered replication in which the 96-word Sensitive policy is fixed before any evaluation and the clickbait regression weights are fit on a training split, then measure AUC and precision-recall on a held-out split of videos never used in prompt or weight selection; if the 0.73-to-0.79 AUC gain and the high-precision clickbait improvement disappear out of sample, the effects are fitting artifacts rather than transferable prompt-design gains.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes three things. First, GPT-4o, given video frames and text features plus a written policy, matches production baselines on the simpler categories (Static Frame, Non-Interactive, Watermark) and trails them by no more than 0.1 AUC on Shocking, Useful, and Clickbait, with its worst performance on Sensitive & Mature (0.73 AUC, driven by a 0.41 false-negative rate). Second, policy length is a lever: shortening the Sensitive & Mature policy to 96 words raised AUC to 0.79, with false negatives dropping by 0.07 and false positives rising by only 0.01, a change the authors explain as a shift from conservative fine-brush scoring to aggressive broad-brush scoring. Third, for the 713-word Clickbait policy, decomposing the category into eight subcategories, asking GPT-4o to score each, and aggregating by mean, max, or linear regression outperforms single-prompt scoring, and the linear-regression aggregate beats the production baseline at high precision. The unifying claim is that application-specific prompt engineering is a cost-effective substitute for additional training.

Load-bearing premise

The reported gains stand or fall on the assumption that the shortened policy and the clickbait aggregation weights were not chosen by looking at the evaluation data: the 96-word policy is not reproduced in the paper, and the best linear regression weights are fit to the same dataset whose precision-recall curves are used to measure the gain.

Editorial extensions

If this is right

  • If the results hold, a video platform can improve zero-shot classification accuracy by rewriting its moderation policies more tersely, without retraining models or collecting more labeled data.
  • Decomposition-aggregation gives a recipe for hard categories: break the policy into its natural subcategories, score each separately, and combine, with linear regression the strongest aggregator in the high-precision regime.
  • The 0.07 false-negative reduction at only 0.01 more false positives for Sensitive & Mature quantifies the practical trade between coverage and precision that prompt length controls.
  • GPT-4o's zero-shot parity with production models on several categories suggests it could serve as a fast, general fallback classifier for categories too small to train dedicated models.
  • The authors' observation that long policies push GPT-4o to be conservative yields a design rule: match prompt specificity to the error cost structure of the task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension of the paper's logic is that long policies act as a prior over the score distribution, so policy compression should shift score entropy; one could test this by measuring GPT-4o's score spread as a function of policy length across several categories, which the paper does not report.
  • The threshold analysis suggests a calibration reading: the 96-word policy may not improve understanding of Sensitive content but rather shift the decision boundary, meaning a similar AUC gain could also be reached by re-ranking scores on a validation set.
  • The decomposition result points to a cheaper alternative to fine-tuning for other hard categories: if a complex policy's subcomponents can be enumerated, per-component scoring plus a learned aggregator may generalize better than a single prompt, but only if the aggregator is trained on separate data.
  • Because the authors attribute the effect to prompt properties rather than model-specific training, the same compression benefit should appear in other instruction-following vision-language models, a prediction the paper leaves untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper evaluates GPT-4o as a zero-shot classifier for seven TikTok feed-quality categories using balanced industry datasets. It reports AUC, false-positive and false-negative rates against production baselines, and proposes two prompt-engineering improvements: shortening a 4,023-word policy for Sensitive & Mature content to 96 words, and splitting the Clickbait policy into 8 subcategories whose scores are aggregated by mean, max, or linear regression. The paper claims that the shortened policy reduces false negatives and that the decomposition-aggregation approach, especially with linear regression, outperforms the single-prompt method and the production model at high precision.

Significance. The paper addresses a practical, high-impact problem (content moderation) and provides an evaluation framework based on real industry data. If the improvements are validated out-of-sample, the results would show that prompt design alone can yield meaningful gains without fine-tuning. The paper also has strengths: the experimental protocol is clearly described (temperature, sampling, balanced datasets), and the mean/max aggregators are parameter-free. However, the two headline claims are currently not supported by the reported experiments because both rely on choices that may be fitted to the evaluation data.

major comments (4)
  1. [Section 4.3 Exp. 3 / Figure 4] The precision-recall curve labeled 'Reg. GPT-4o-multi' is computed on the same balanced Clickbait dataset that was used to fit the linear regression weights. Since no train/test split or cross-validation is described, the curve is an in-sample fit, and the claim that this aggregator outperforms the production model at high precision is not an out-of-sample prediction. Please report cross-validated or held-out PR curves, or restrict the claim to the parameter-free mean and max aggregators.
  2. [Section 4.3 Exp. 2 / Q3] The shortened Sensitive & Mature policy is listed as 'XXX words' in the experimental setup and the actual 96-word prompt is not disclosed. The paper reports a reduction in FN of 0.07 and an AUC increase from 0.73 to 0.79, but without knowing the prompt or the procedure used to derive it, one cannot exclude the possibility that it was selected after inspecting the test-set results. Please disclose the full prompt and describe its construction as independent of the evaluation data.
  3. [Section 4.3 / Table 1] No error bars or significance tests are reported for any AUC or FP/FN comparisons in Table 1 and Figures 3–4. Given the dataset sizes of 'at least 500', differences such as 0.73 vs 0.79 AUC could be within sampling noise. Provide confidence intervals or a significance test for the key comparisons.
  4. [Section 4.4 Q3–Q5] The observed correlation between policy length and error type is based on only seven categories and is confounded by category difficulty. The paper itself acknowledges this in Q5, but the earlier phrasing in Q3 ('this pattern corresponds directly to the number of words in the policy') overstates the evidence. Please temper the causal language or provide a controlled analysis.
minor comments (6)
  1. [Section 4.3 Exp. 2] The placeholder 'XXX words' should be replaced with the actual word count; in Q3 the number 96 appears without explanation.
  2. [Section 3.2 / Abstract] The terminology is inconsistent: the abstract says 'zero-shot' while Exp. 3 and Q6 use 'one-shot'. Please clarify the intended meaning and use it consistently.
  3. [Section 1] There is a typo: 'mutliple' should be 'multiple'.
  4. [Section 4.2 and Q2] 'SD/SF/Watermark' appears to be a typo; 'SD' should likely be 'SF' (Static Frame).
  5. [Figure 3 caption] The formatting 'GP T− 4o' should be corrected to 'GPT-4o'.
  6. [Section 4.3 Exp. 2] The phrase 'test the affect of shortening' should be 'test the effect of shortening'.

Circularity Check

1 steps flagged · score 6.0 of 10

Exp. 3's 'best linear regression' aggregation is fit on the same PR-evaluation data, so the claimed high-precision win over the production model is a fitted in-sample result presented as a zero-shot prediction.

  1. fitted input called prediction [Section 4.3 Exp. 3; Section 4.4 Q6; Figure 4 caption]
    "To produce a final score for GPT-4o-multi, we consider the mean, max, as well as the best linear regression of the category scores. ... Interestingly, we found that performing linear regression on the scores allowed us to outperform the production model for higher precision values (which is the typical, more conservative, regime of interest.)"

    No train/test split is reported for Exp. 3. The paper describes one balanced Clickbait dataset (Section 4.1) and uses that same dataset for the PR curves in Figure 4; the 'best' linear-regression weights are therefore selected using the labels that also define the curve being reported. The Reg curve's advantage at high precision is an in-sample fit, not an out-of-sample zero-shot prediction, so the load-bearing 'outperform the production model' claim reduces to a fitted value. The mean and max aggregators are parameter-free and still show improvement, which leaves the decomposition idea partially independent, but the specifically claimed production-model win rests on the fitted regression.

full rationale

Most of the paper is self-contained against external baselines: Exp. 1 directly compares GPT-4o scores with production-model scores, and there is no load-bearing self-citation chain. The clear circular step is in Exp. 3, where the linear-regression aggregation is fit to the same Clickbait labels used to draw the precision-recall curves, with no held-out split described; its reported high-precision superiority is thus a fitted result rather than a prediction. The 96-word Sensitive policy is undisclosed and no search procedure is reported, which is a reproducibility gap and a possible test-set-selection risk, but the paper provides no explicit evidence that the policy was tuned on the test data, so I do not count it as a demonstrated circular step. The abstract's broader 'decomposition-aggregation outperforms single-prompt' claim is independently supported by the parameter-free mean/max aggregators, which is why the score is partial rather than total.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central results rely on trusting human annotations as ground truth, balanced datasets as representative, and prompt choices that are not disclosed. Two free parameters (the shortened policy and the clickbait regression weights) are effectively fit to the evaluation data, which undermines the predictive claims.

free parameters (2)
  • shortened Sensitive & Mature policy = 96 words (text not disclosed)
    Hand-crafted summary replacing the 4,023-word policy; chosen to reduce false negatives on the same test set, so its effect is partly a fit. Section 4.3 Exp. 2 and Figure 3.
  • clickbait linear-regression aggregation weights = not reported
    Combination weights for the 8 subcategory scores fit to the same dataset used for evaluation; reported improvement over production model at high precision is therefore fitted, not predicted. Section 4.3 Exp. 3 and Figure 4.
assumptions (4)
  • domain assumption Human annotations and production-model baseline scores are correct ground truth.
    Used to compute AUC and FP/FN; Section 3.1 says human annotations serve as labels; no inter-annotator agreement reported.
  • domain assumption Balanced datasets (equal positives and negatives) are representative enough for comparing classifiers.
    Section 4.1 notes categories are rare in the real feed but datasets are balanced 'for fair comparison'; FP/FN numbers therefore do not reflect production prevalence.
  • domain assumption Sampled frames (0.5 fps, max 30) plus text features give GPT-4o sufficient information to classify.
    Section 4.1; no ablation of frame sampling density or feature set.
  • domain assumption GPT-4o scores are treated as calibrated likelihoods for threshold-based FP/FN analysis.
    Section 4.4 uses a single threshold (dashed line) to report FP/FN; no calibration analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing GPT for Video Understanding: Zero-Shot Performance and Prompt Engineering." pith.science (2026). https://pith.science/paper/IJ7K5J56

@misc{pith2026250209573,
  author       = {Pith},
  title        = {Pith review of: Optimizing GPT for Video Understanding: Zero-Shot Performance and Prompt Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJ7K5J56}},
  note         = {Machine review of arXiv:2502.09573}
}
read the original abstract

In this study, we tackle industry challenges in video content classification by exploring and optimizing GPT-based models for zero-shot classification across seven critical categories of video quality. We contribute a novel approach to improving GPT's performance through prompt optimization and policy refinement, demonstrating that simplifying complex policies significantly reduces false negatives. Additionally, we introduce a new decomposition-aggregation-based prompt engineering technique, which outperforms traditional single-prompt methods. These experiments, conducted on real industry problems, show that thoughtful prompt design can substantially enhance GPT's performance without additional finetuning, offering an effective and scalable solution for improving video classification.

Figures

Figures reproduced from arXiv: 2502.09573 by the authors.

Figure 1
Figure 1. To have a fair comparison across categories, we design the experiment such that the item [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This figure shows the precision-recall curves when using the score provided by GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. These charts plot the normalized (0-1) distribution of scores (x-axis) provided by GPT-4o, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: This figure shows the precision-recall curves for the Clickbait category when using both the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 6 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  3. [3]

    Openflamingo: An open-source framework for training large autoregressive vision-language models

    Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  4. [4]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.06500, 2, 2023

    arXiv 2023

  5. [5]

    A Multimodal CNN-based Tool to Censure Inappropriate Video Scenes

    de Freitas, P. V., Mendes, P. R., dos Santos, G. N., Busson, A. J. G., Guedes, \'A . L., Colcher, S., and Milidi \'u , R. L. A multimodal cnn-based tool to censure inappropriate video scenes. arXiv preprint arXiv:1911.03974, 2019

    work page Pith review arXiv 1911

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  8. [8]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

Show all 24 references
  1. [9]

    H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al

    Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10965--10975, 2022

    2022

  2. [10]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  3. [11]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  4. [12]

    Openai official api documentation, 2025

    OpenAI. Openai official api documentation, 2025. URL https://openai.com/index/openai-api/

    2025

  5. [13]

    J., Mahmud, A., Sobuj, M

    Prottasha, N. J., Mahmud, A., Sobuj, M. S. I., Bhat, P., Kowsher, M., Yousefi, N., and Garibay, O. O. Parameter-efficient fine-tuning of large language models using semantic knowledge tuning. Scientific Reports, 14 0 (1): 0 30667, 2024

    2024

  6. [14]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  7. [15]

    Will the \ 1 trillion of generative ai investment pay off?, 2024

    Sachs, G. Will the \ 1 trillion of generative ai investment pay off?, 2024. URL https://www.goldmansachs.com/insights/articles/will-the-1-trillion-of-generative-ai-investment-pay-off. Accessed: 2025-02-20

    2024

  8. [16]

    Video understanding with large language models: A survey

    Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432, 2023

    2023

  9. [17]

    Peer: Expertizing domain-specific tasks with a multi-agent framework and tuning methods

    Wang, Y., Li, X., Wang, B., Zhou, Y., Lin, Y., Ji, H., Chen, H., Zhang, J., Yu, F., Zhao, Z., et al. Peer: Expertizing domain-specific tasks with a multi-agent framework and tuning methods. arXiv preprint arXiv:2407.06985, 2024

    2024 arXiv

  10. [18]

    F., Tuan, L

    Wang, Z., Kuan, K., Ravaut, M., Manek, G., Song, S., Fang, Y., Kim, S., Chen, N., D'Haro, L. F., Tuan, L. A., et al. Truly multi-modal youtube-8m video classification with video, audio, and text. arXiv preprint arXiv:1706.05461, 2017

    2017 arXiv

  11. [19]

    and Nawaz, T

    Yousaf, K. and Nawaz, T. A deep learning-based approach for inappropriate content detection and classification of youtube videos. IEEE Access, 10: 0 16283--16298, 2022

    2022

  12. [20]

    Fine-tuning large language models for domain-specific machine translation

    Zheng, J., Hong, H., Liu, F., Wang, X., Su, J., Liang, Y., and Wu, S. Fine-tuning large language models for domain-specific machine translation. arXiv preprint arXiv:2402.15061, 2024

    2024 arXiv

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  14. [22]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  15. [23]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  16. [24]

    world knowledge

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.