{"id":"9e23bb89-a208-4c8d-942c-d45396a2628d","arxiv_id":"2508.19295","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A two-level fine-tuned LVLM pipeline generates stylized football captions, claiming improved F1 and BERT scores over zero-shot and directly fine-tuned models, but the F1 evidence is absent and evaluation is limited to one event.","lead":"This paper describes a two-stage fine-tuned vision-language system that captions American football photos in a wire-service style, and reports using it live during Super Bowl LIX. It is a compact industry report whose headline accuracy claim is not backed by the numbers shown.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central F1 improvement claim appears nowhere in the results, and the only reported metric (BERTScore) shows a 1.4-point difference within one standard deviation, so the headline quantitative result is unsupported.","rationale":"The reader's weakest assumption identifies evaluation on the deployment event and possible memorization; I agree that is a serious validity threat. My concern is more immediate: the central F1 claim is not presented in the results at all, and the one reported metric, BERTScore, shows a difference that overlaps within one standard deviation. This is an internal evidential gap rather than a dispute with external consensus. The paper reads as an industry proof-of-concept, and the authors are transparent about production context and about variability in fine-tuning, but the scientific claim as written is unsupported. I therefore agree with the REJECT verdict, though my primary reason is the missing F1 result and the statistically weak BERTScore difference rather than the same-event evaluation alone. If the authors provided an F1 table with error bars and a held-out game evaluation, the verdict could change to CONDITIONAL or ACCEPT; until then, the central quantitative claim remains unverified.","tokens_in":3555,"tokens_out":3406,"duration_ms":35354,"concrete_test":"Run the Level-1 entity-recognition evaluation on a holdout game (e.g., a 2024 regular-season game) that was not used for SFT, comparing direct fine-tuning against the two-level pipeline and reporting F1 with standard deviations or confidence intervals. If the F1 improvement on that holdout is not at least 8 points, the paper's headline claim '>8–10% F1' fails; if the authors cannot supply such a held-out evaluation, the claim is unsubstantiated as written.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract and the 'Our Approach' section claim an F1 improvement of >8–10%, but no F1 value is reported anywhere in the paper, including in Table 1, which only lists BERTScore. The Level-2 sentence 'provided better accuracyF1score >8%' is an unnumbered assertion with no supporting table or error bars. The only reported comparison against direct fine-tuning is BERTScore: 89.8±2.1 versus 91.2±2.8, a 1.4-point difference within one standard deviation and below the claimed '>2–10%' gain over the strongest baseline. The evaluation is confined to Super Bowl LIX, the same event on which the system was deployed and from which fine-tuning data were likely drawn, with no disclosed train/test split. The paper's own Future Research paragraph concedes 'high variability in fine-tuned performance,' so missing variance or confidence intervals around F1 are not a minor omission. Without an F1 table or a held-out game evaluation, the central quantitative claim is not verifiable from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a two-level fine-tuned LVLM pipeline for generating stylized sports captions in the Imagn wire-service format. Level 1 identifies players through team and jersey-number labels with HIGH/LOW confidence annotations, and Level 2 generates captions using Level 1 outputs together with metadata and outputs from additional vision models. The authors claim more than 8-10% F1 improvement and 2-10% BERT-score improvement over alternative approaches, and report live deployment during Super Bowl LIX with over 1000 captions generated at a rate of 6 images per 3-5 seconds. The only quantitative evidence in the paper is Table 1, which reports BERT scores for zero-shot, few-shot, direct fine-tuning, and the proposed two-level pipeline.","tokens_in":3771,"tokens_out":4460,"duration_ms":43029,"significance":"If the reported accuracy and efficiency claims were fully substantiated, this would be a useful engineering contribution to automated sports-media captioning: the paper addresses a real production task, demonstrates a concrete deployment at Super Bowl LIX, and reports substantial efficiency gains from 4-bit quantization and a latency margin of over 72x. However, the central scientific claims are not verifiable from the manuscript as written. No F1 values are reported anywhere, the single reported BERT-score comparison shows a 1.4-point difference within one standard deviation, and the evaluation is confined to the same event from which the fine-tuning data were likely drawn, with no disclosed train/test split. No code, dataset, annotation protocol, or hyperparameter details are provided, so the results are not reproducible from the manuscript.","major_comments":[{"comment":"The headline quantitative claim of an F1 improvement greater than 8-10% appears in the abstract and is repeated in the Level 1 and Level 2 descriptions, but no F1 value is reported anywhere in the paper. Table 1 lists only BERT scores, so the paper's central accuracy claim is unsupported by the presented evidence. The authors should either report the F1 values with their evaluation protocol and error bars or remove the claim.","section":"Abstract; Our Approach (Level 1 and Level 2)"},{"comment":"The only quantitative comparison against direct fine-tuning is a BERT score of 89.8 ± 2.1 for Direct Fine-Tune versus 91.2 ± 2.8 for the Two-Level pipeline. This 1.4-point difference is within one standard deviation of both means and does not support the abstract's claim of a BERT-score improvement of 2-10%. The paper should report effect sizes, confidence intervals or bootstrap tests, the number of test images or captions, and the exact BERT-score variant used.","section":"Table 1"},{"comment":"The evaluation is in-domain: both the fine-tuning data and the evaluation data appear to come from the same Imagn/Super Bowl LIX source, and no train/validation split, held-out game, date-based split, or pre-registered protocol is disclosed. Because the Level 2 training captions are filtered by the Level 1 model's HIGH-confidence labels, the pipeline could in principle memorize the style and entities of that single game. The authors should evaluate on a held-out game or season and report the split; otherwise the claimed gains cannot be distinguished from memorization.","section":"Problem Statement; Table 1; Summary"},{"comment":"The paper attributes the reported gains to specific design choices, including HIGH/LOW confidence labels, a decoder-based architecture, and the use of Level 1 outputs in Level 2 training, but none of these components is evaluated in isolation. No ablation results are provided that compare the proposed labeling with plain labels, decoder-based with encoder-based backbones, or the two-level pipeline with a direct fine-tuned model on the same data. These ablations are necessary to support the design claims that are load-bearing for the proposed method.","section":"Level 1 - LVLM Player Model; Level 2 - LVLM Caption Model"}],"minor_comments":[{"comment":"The phrase 'better accuracyF1score > 8%' appears to be a typographical error and should be corrected to a clear and grammatical statement, such as 'better accuracy, with F1 score improved by more than 8%.'","section":"Level 2 - LVLM Caption Model"},{"comment":"Figure 3 is labeled as the SFT template, but the text says 'see Fig. 3' when referring to the 4-bit quantization of the final models. The figure numbering or references should be corrected so that each figure is referenced consistently.","section":"Figure 3 and quantization paragraph"},{"comment":"Table 1 should state the number of evaluation images, the unit over which the standard deviation is computed (images, captions, or bootstrap resamples), and the BERT-score aggregation mode (F1, precision, or recall).","section":"Table 1"},{"comment":"The paper does not report training data sizes, annotation counts, hyperparameters, or the specific base model versions used for fine-tuning. The Future Research paragraph acknowledges 'high variability in fine-tuned performance,' so the manuscript should disclose the number of fine-tuning runs and the seed-to-seed variance for the reported results.","section":"Our Approach; Future Research"},{"comment":"The phrase 'over 1000 usable captions' is not operationalized. The paper should define what makes a caption 'usable' and, ideally, provide a human-evaluation protocol or a sample of generated captions with expert assessments.","section":"Summary"}],"recommendation":"reject","confidential_remarks":"This is a very short paper that reads more like a product deployment note than a journal article. The central accuracy claim is absent from the results, the only reported metric shows a small and statistically non-robust difference, and the evaluation is in-domain with no disclosed split. Even if the missing F1 table were added, the in-domain evaluation would not support a claim of generalizable improvement. The paper also lacks the reproducibility details expected of a serious journal submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fairly short end. This is an industry proof-of-concept, and it reads like one. The two-level design — a Level-1 player identification model that outputs jersey numbers with HIGH/LOW confidence, mapped to names via the roster, and a Level-2 caption model that takes those outputs plus OCR and face detections — is a reasonable, practical engineering combination. The roster-grounded mapping is a nice trick for reducing name hallucination, and the confidence labeling has a plausible effect on focusing attention. I believe the deployment at Super Bowl LIX happened; the press references support it. The paper is also honest about dead ends: ORPO gave no significant gain, and the Future Research section admits high variability in fine-tuned performance. That honesty counts.\n\nThe problem is the headline claim. The abstract and the Level-1 section say \">8–10% improvement in F1,\" and the Level-2 text repeats \"better accuracyF1score >8%,\" but no F1 number appears anywhere in the paper. Table 1 shows only BERTScore, and the difference between the two-level fine-tune (91.2 ± 2.8) and the direct fine-tune (89.8 ± 2.1) is 1.4 points, within one standard deviation. So the strongest baseline comparison in the reported table does not support the magnitude claimed, and the actual F1 evidence is absent. The evaluation is also entirely on Super Bowl LIX, the same event the system was built for and deployed on, with no disclosed train/test split and no held-out game. The Level-2 training captions were filtered by Level-1 HIGH-confidence labels, so there's a real circularity risk: the two stages were tuned in-domain and then scored in-domain. No code, no data, no human judge on the \"1000 usable captions\" claim. These are not minor quibbles; the paper's central quantitative assertion is unverifiable from the manuscript as written.\n\nThat said, the architecture is not nonsense, and the latency/VRAM claims (4-bit quantization, >72× margin) are concrete and plausible. This paper is useful for practitioners in sports media NLP who want a template for a production captioning pipeline. It is not, as it stands, a scientific validation of that pipeline. I would send it to peer review only because a referee could demand the F1 table, error bars, and a held-out game; if those don't materialize, reject. But I would not recommend desk reject outright — the deployment and the design are worth a look.","headline":"A sensible two-level captioning pipeline with a real deployment behind it, but the central F1 claim is missing from the results and the only reported metric shows a statistically weak gain.","tokens_in":4298,"tokens_out":2589,"would_cite":false,"duration_ms":24148,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-level, fine-tuned large vision-language model pipeline generates stylized sports captions that beat zero-shot, few-shot, and direct fine-tuning baselines in F1 and BERT score, and it ran in production at Super Bowl LIX.","keywords":["sports captioning","large vision-language model","supervised fine-tuning","entity recognition","football","Super Bowl","BERT score","low-latency inference"],"falsifier":"Evaluate the pipeline on a fully held-out game (different teams, different venue) and compare entity F1 and BERT score with the numbers in Table 1; if the F1 margin over zero-shot falls below the reported 8-10%, the Super Bowl LIX result is not evidence of generalization.","tokens_in":3314,"feed_emoji":"🏈","tokens_out":5485,"duration_ms":46527,"temperature":0.7,"pith_summary":"The paper argues that off-the-shelf large language and vision-language models cannot produce production-grade sports captions: they lack sport-specific jargon and make entity mistakes. To fix this, the authors build a two-level pipeline in which one fine-tuned LVLM identifies players from jersey numbers and team context, and a second fine-tuned LVLM assigns actions and writes the caption in a required wire-service style. They report that this division of labor improves entity F1 by more than 8-10 percent and BERT score by 2-10 percent over baselines, including direct fine-tuning. The pipeline was used live at Super Bowl LIX, where it generated over 1,000 captions at a rate of six images per three to five seconds with a small memory footprint. The central claim is that a purpose-built, two-level fine-tuned LVLM can meet real-time sports journalism requirements that generic LVLMs cannot.","feed_headline":"Two-level AI pipeline writes Super Bowl captions with 8-10% F1 gain","feed_subtitle":"The system runs at 6 images per 3-5 seconds with over 1000 captions generated at Super Bowl LIX.","key_machinery":"The load-bearing object is the two-level fine-tuned LVLM pipeline (Fig. 2). Level 1, the Player Model, is a decoder-based LVLM trained with a supervised fine-tuning template that takes the image plus metadata and team roster, and is asked to output only team and jersey numbers with HIGH or LOW confidence; this limits generation to deterministic roster mapping and cuts hallucination. Level 2, the Caption Model, is a second fine-tuned LVLM that receives the image, metadata, roster, the Level 1 output, and outputs from auxiliary vision models (celebrity-face detection, logo detection, OCR, and a generic image-captioning model), and is trained on ground-truth captions re-annotated to keep only HIGH-confidence players. The auxiliary signals and the confidence-based filtering are what let the system associate the correct entity with the correct action and hold the required wire-service style. The pipeline is quantized to 4 bits, yielding a >10x reduction in inference memory versus the 11B baseline and >90x versus the 90B baseline.","core_discovery":"The central discovery is a decomposition: instead of asking one model to recognize players, infer actions, and write a stylized caption, the authors split the task into Level 1 player recognition and Level 2 caption generation. Level 1 is a decoder-based LVLM fine-tuned to output only team and jersey numbers with HIGH/LOW confidence labels, which reduces hallucinated player names because names are mapped deterministically from the roster; the confidence labels let the model attend to in-focus players. Level 2 is a separately fine-tuned LVLM that takes the image, metadata, roster, Level 1 output, and auxiliary vision signals (celebrity faces, logos, OCR, generic description) and produces the caption in Imagn's style. The authors report that this two-level approach yields >8-10% F1 and >2-10% BERT-score improvements over direct fine-tuning and zero-shot/few-shot alternatives, and that 4-bit quantization gives a 10x memory reduction versus an 11B model and 90x versus a 90B model, while inference latency exceeds the operational requirement by 72x.","pith_inferences":["A stronger test of generalization would hold out one game for evaluation and another for training; without that split, the Super Bowl LIX numbers may reflect memorization of that game's rosters and events rather than a generalizable capability.","The deterministic roster mapping hints at a broader design principle: constraining an LVLM's output to a closed set of identifiers (numbers, codes) before generating free text can reduce hallucination more cheaply than preference optimization.","The 72x latency margin suggests the pipeline could be extended to video or live broadcast captioning, though the paper does not test temporal consistency across frames.","BERT score is a token-overlap proxy; a human rater study or a downstream engagement metric would be needed to confirm that the generated captions meet wire-service readability standards."],"forward_implications":["Sports image wire services can automate game-day captioning with turnaround times of about 1.5 to 2 seconds per image, covering 1,000+ images in a single game.","The same two-level decomposition can be transferred to other sports that have jersey numbers and roster files, such as basketball, baseball, or soccer, by re-annotating data and retraining the two LVLMs.","The confidence-label trick (HIGH/LOW) is a reusable recipe for any fine-grained entity-recognition task where a model tends to hallucinate names.","Using team and jersey numbers instead of names means the caption generator adapts to roster changes simply by swapping the roster context at inference time, avoiding retraining for personnel changes.","Quantization to 4 bits makes on-site, low-memory deployment feasible for news organizations without high-end GPU infrastructure."],"supporting_citations":[{"why":"Defines the Imagn caption style and provides an example of the target format that the system must reproduce.","marker":"[5]"},{"why":"BERTScore, the evaluation metric used to measure caption quality in Table 1.","marker":"[13]"},{"why":"GIT, the generic image-to-text model whose description is one of the auxiliary vision-modality inputs to the Level 2 caption model.","marker":"[10]"}],"fun_headline_variants":["Two-level LVLM pipeline improves sports captions by 8-10% F1","Super Bowl-tested AI captions: 1000+ images, 8-10% F1 gain","AI sports captions: 8-10% F1 gain, 6 images per 3-5 sec","Two-stage fine-tuned LVLM beats SoTA on sports captioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains are measured on captions from the same game (Super Bowl LIX) that the system was built for, with no disclosed train/test split, so the numbers may reflect memorization of that game's rosters and events rather than a generalizable ability to caption unseen football.","fun_headline_variants_meta":{"raw":{"variants":["Two-level LVLM pipeline improves sports captions by 8-10% F1","Super Bowl-tested AI captions: 1000+ images, 8-10% F1 gain","AI sports captions: 8-10% F1 gain, 6 images per 3-5 sec","Two-stage fine-tuned LVLM beats SoTA on sports captioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3441,"prompt_tokens":1004,"completion_tokens":2437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2350}},"tokens_in":620,"tokens_out":2437,"duration_ms":18724,"temperature":1.0,"reasoning_tokens":2350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:56:51.911783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the pipeline on a fully held-out game (different teams, different venue) and compare entity F1 and BERT score with the numbers in Table 1; if the F1 margin over zero-shot falls below the reported 8-10%, the Super Bowl LIX result is not evidence of generalization.","supporting_citations":[{"cited_title":"Imagn caption example","cited_arxiv_id":null,"evidence_quote":"Defines the Imagn caption style and provides an example of the target format that the system must reproduce."},{"cited_title":"A survey on large language models for recommendation, 2024","cited_arxiv_id":null,"evidence_quote":"BERTScore, the evaluation metric used to measure caption quality in Table 1."}],"review_version":1}