{"id":"95584358-bea6-4d6e-bb91-74fcd6a47883","arxiv_id":"2505.02693","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs ranking new movies by popularity from metadata can outperform random and embedding-similarity baselines, with the largest model and richest prompt working best.","lead":"This paper tests whether large language models can predict which newly released movies will become popular before they gain audience attention. The authors report that a large Llama model given rich metadata outperformed simple embedding-based baselines on proprietary production data from a streaming platform.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'before they happen' claim hinges on an unverified 6–12 month cutoff margin; no release dates, checkpoint dates, or metadata as-of timestamps are provided, so training-data contamination cannot be ruled out.","rationale":"The reader's weakest assumption identifies the same load-bearing premise: the 6–12 month knowledge-cutoff margin must prevent the model from having seen post-release popularity information. My stress-test confirms this is the most critical point because the paper's headline claim is about prediction 'before they happen,' not correlation with known outcomes. The paper provides no dataset, no release-date distribution, no exact cutoff date, and no metadata timestamp policy, so the premise is unverified rather than established. I found no internal contradiction in the reported experiments; the LLM-vs-BERT comparisons are internally consistent, but they cannot discriminate between forecasting and memorization without provenance data. A concrete audit, as proposed, could settle the issue. Since the reader's verdict is already CONDITIONAL and the concern does not by itself prove the claim false, the appropriate recommendation is to keep the verdict unchanged until the required evidence is supplied.","tokens_in":5745,"tokens_out":5847,"duration_ms":68535,"concrete_test":"Independent audit: obtain the exact Llama-3.1/3.3 checkpoint identifiers and documented knowledge-cutoff dates plus release dates for every test movie; then rerun the Llama-3.1-405B V4 evaluation on the subset of movies released at least 12 months after the cutoff, with all metadata fields (especially awards) truncated to records available 30 days before release. Also run the same prompt on a matched set of movies released before the cutoff and compare ACC@1. If the pre-cutoff accuracy is substantially higher than the post-cutoff accuracy, or if the post-cutoff-only improvement over BERT V4 falls below the reported 28.33% ACC@1 gain, training-data contamination or metadata leakage is present.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 asserts: 'There is a margin of 6-12 months between LLMs' knowledge cutoff and movie release dates.' This single sentence is the only barrier between forecasting and memorization for the central claim that LLMs identify hits 'before they happen.' The paper reports no dataset, no release-date distribution, no exact checkpoint or knowledge-cutoff date, and no check that test movies or their popularity outcomes are absent from training data. The margin is silently assumed to hold for every movie, while the model family (Llama 3.1/3.3) has multiple released checkpoints with different instruction-tuning data. In addition, the metadata inputs (Section 2.1: genre, synopsis, content ratings, era, cast, crew, mood, awards, character types) are not stated to be timestamped to what was known at release; if any field (e.g., awards, mood, character types) is populated from a post-release database, then future information enters through the prompt even with a correct cutoff. The observed Llama-405B V4 gains (ACC@1 +28.33%, RR +22.46%, NDCG@3 +12.90%, RC@3 +31.42% over BERT V4, Table 2) are therefore consistent both with genuine forecasting and with memorization or retrospective leakage. The paper even mentions 'awareness of prior data points used' in prompt design (Section 2.2.1), so the provenance question is decisive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes using large language models (LLMs) in generative mode to rank newly released movies by their expected popularity, as a cold-start solution for an entertainment platform. The authors construct a proprietary dataset of newly released movies with popularity labels defined by later user interactions, build a Popular Embedding baseline by comparing metadata embeddings to the average embedding of top popular items, and then measure the percentage improvement of Llama 3.1/3.3 models under four metadata prompt variants (V1 through V4). The central empirical result is in Table 2: Llama 3.1 405B with the richest prompt (V4) improves ACC@1 by 28.33%, RR by 22.46%, NDCG@3 by 12.90%, and RC@3 by 31.42% relative to the BERT V4 baseline. The authors conclude that LLMs can identify hit movies before they gain significant audience attention and could support editorial teams or retrieval systems.","tokens_in":6084,"tokens_out":3428,"duration_ms":39475,"significance":"If the result holds, the paper would provide a practical, scalable tool for cold-start movie promotion, a genuinely important problem for large content platforms. The study has several strengths: it uses real production data, compares multiple model sizes (8B, 70B, 405B), systematically varies prompt complexity, and attempts a forecasting setup with a stated knowledge-cutoff margin. The prompt engineering details, including structured JSON output and post-processing, are useful practical contributions. However, the central claim that LLMs predict hits 'before they happen' rests on an unverified temporal-separation assumption, and the reported evidence is purely relative to a weak self-developed baseline without absolute numbers, error bars, or a supervised comparison. As presented, the paper is an interesting exploratory study but not yet a validated demonstration of the headline claim.","major_comments":[{"comment":"The only safeguard against training-data contamination is the sentence 'There is a margin of 6-12 months between LLMs' knowledge cutoff and movie release dates.' No release dates, no exact checkpoint or knowledge-cutoff dates, and no verification that the test movies or their popularity outcomes are absent from the models' training data are provided. Because the title and conclusion claim prediction 'before they happen,' this is load-bearing evidence. The authors should report the release-date distribution, specify the exact checkpoint of each Llama model (including instruction-tuning data dates), and demonstrate that the margin holds for every movie in the evaluation; otherwise the Table 2 gains are equally consistent with memorization or retrospective leakage.","section":"Section 3 (LLM Evaluation)"},{"comment":"Label construction is underspecified: the text says the authors examined 'various time windows and popularity list sizes' but does not give the chosen values, the interaction threshold for popularity, or the candidate list size. More importantly, the metadata fields in V4 include awards and top cast awards; if these fields are populated from post-release databases, future information enters the prompt even if the model cutoff is correctly earlier than the release date. The authors should state the exact label definition and provide as-of timestamps for each metadata field, showing that all fields were known at release time.","section":"Section 2.1 (Dataset)"},{"comment":"Only percentage improvements relative to the BERT V4 baseline are reported. The paper gives no absolute metric values, no standard deviations or confidence intervals, and the claim that 'the average results fall within the expected statistical range' is not backed by any displayed analysis. With only 10 experiments per condition, the headline +28.33% ACC@1 improvement could be within run-to-run noise. The authors should report the underlying distributions or error bars for the key comparisons.","section":"Section 3.2 (Results) and Table 2"},{"comment":"The comparison baseline is a self-developed embedding cosine-similarity model plus a random ordering; no supervised popularity-prediction baseline (e.g., a content-based regression, a collaborative-filtering model, or a standard recommender baseline) is tested. The abstract's phrase 'established baselines' overstates what is compared. A stronger baseline is needed before concluding that LLMs 'significantly outperform' existing approaches rather than a particular embedding heuristic.","section":"Section 3 (Baseline Evaluation)"}],"minor_comments":[{"comment":"The prompt instructs the model to return 'awareness of prior data points used.' Clarify whether this refers to the provided metadata or to the model's internal knowledge, and explain why this phrasing does not invite the model to draw on post-release information.","section":"Section 2.2.1 (Prompt engineering)"},{"comment":"The section states that pairwise ranking did not improve key metrics and sometimes underperformed, but no table or metric values are provided. Add the supporting numbers or remove the section.","section":"Section 3.3 (Pairwise vs Listwise)"},{"comment":"The candidate list size and the number of lists/movies in the dataset are never stated, although Section 3.3 refers to 'fewer items per list.' Reporting these numbers is necessary for reproducibility and for interpreting the metrics.","section":"Section 2.1 (Dataset)"},{"comment":"The text says the highest and second highest values are in bold, but the formatting is not visible in the manuscript text; ensure the table rendering makes these values distinguishable.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a four-page workshop-style UMAP contribution, but the claims in the title and abstract are strong. The core issue is not the absence of a derivation—the evaluation is externally labeled—but the absence of verifiable temporal separation between the models' knowledge and the movies' release/popularity data. The authors can likely address this with additional dataset documentation and, if necessary, a cutoff-filtered rerun. I would not recommend acceptance in the current form because the central 'before they happen' claim is unsubstantiated without those details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a 4-page UMAP paper with a real empirical result—LLMs ranking cold-start movies on production metadata beat a BERT-embedding baseline—but the headline claim (“before they happen”) hinges on a data-provenance assumption the paper never verifies. I’d send it to review, not desk-reject, and push hard on the leakage question.\n\nWhat’s new: they built a proprietary dataset of newly released movies with rich metadata, tracked popularity over time windows, and evaluated Llama 3.1/3.3 under several prompt variants. The prompt ablation is clean: richer metadata helps larger models, hurts the 8B, and the best 405B V4 prompt beats BERT V4 by roughly +28% ACC@1, +22% RR, +13% NDCG@3, +31% RC@3. The pairwise-vs-listwise negative result in Sec. 3.3 is a genuinely useful observation for anyone building these systems. I also credit them for averaging ten runs per variant and for using full, non-quantized models in the main LLM evaluation.\n\nThe soft spots are proportional, and one is load-bearing. The entire “before they happen” framing rests on one sentence in Sec. 3: “There is a margin of 6-12 months between LLMs’ knowledge cutoff and movie release dates.” No release-date distribution, checkpoint dates, or cutoff dates are given, and there is no check that the test movies or their popularity outcomes are absent from training data. With Llama 3.1/3.3 there are multiple checkpoints and instruction-tuning data, so the margin has to be demonstrated, not asserted. The same problem applies to the metadata itself: if awards, mood, or character types are populated from a post-release database, future information enters through the prompt even with a correct cutoff. On top of that, no absolute metrics or error bars are reported, the label construction (time window, list size) is described only as “various,” and the main comparison is against a single embedding baseline with no supervised baseline. These are fixable, but as submitted the evidence does not rule out memorization or retrospective leakage. The citation pattern is fine; the Linq-Embed-Mistral self-citation is natural since it is used as a baseline.\n\nWho this is for: applied recommender systems and editorial-decision researchers, especially people building LLM-based cold-start tools. A serious referee will get value from it. My recommendation: accept as a conditional/revise if the authors add leakage checks and metadata timestamping, and ideally release a small public benchmark with absolute numbers and confidence intervals; otherwise the claim should be downgraded to “LLMs rank movies consistently with later popularity on internal data.”","headline":"Plausible, useful cold-start LLM ranking result on real production data, but the central 'before release' claim rests on an unverified leakage guard and should be treated as conditional.","tokens_in":6577,"tokens_out":2069,"would_cite":false,"duration_ms":23374,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can forecast movie hits from metadata alone, before any audience data exists.","keywords":["Large Language Model","Personalization","Movie Recommendation","LLM Ranking","Cold-start","popularity prediction","prompt engineering","metadata"],"falsifier":"Build a matched test set of movies released before the model's knowledge cutoff and ask the model to rank them as if they were new, then compare performance to the original after removing release dates and award names from the prompt; if performance collapses, the reported forecasting edge was memorization of post-release information.","tokens_in":1351,"feed_emoji":"🎬","tokens_out":4012,"duration_ms":99488,"temperature":0.7,"pith_summary":"Large language models, prompted only with movie metadata, can forecast which newly released movies will become popular before any user interaction data exists. This matters for the cold-start problem in recommendation, where new content lacks the behavior signals collaborative filters rely on. Using a production dataset of new releases, the paper shows that Llama 3.1 405B with the richest metadata prompt outperforms a BERT embedding baseline on all four reported metrics. The intended use is an assisted tool for editorial teams and an early retrieval-stage signal in personalization pipelines.","feed_headline":"LLM ranks movie hits from metadata alone, beating embeddings","feed_subtitle":"Llama 3.1 405B plus full metadata lifts top-1 hit accuracy 28% over the BERT baseline on cold-start movies.","key_machinery":"The central mechanism is listwise generative ranking: an LLM is prompted as a movie-popularity expert, given a list of new releases, instructed to reorder the list without adding items, and asked to return JSON with a ranked list, popularity scores, reasoning, and data awareness. Metadata is layered in four versions: V1 uses genre, V2 adds synopsis, V3 adds content ratings, character types, mood, and era, and V4 adds cast, crew, and top cast awards. The comparison baseline is a popular-embedding model that ranks movies by cosine similarity between each movie's embedding and the average embedding of the top-100 popular items in the weeks before release. The argument runs on the difference between these two mechanisms: explicit language reasoning over metadata versus latent similarity in embedding space.","core_discovery":"The paper's claim is that a large language model can act as a cold-start popularity forecaster: given a list of newly released movies and structured metadata, it returns a ranked list with popularity scores and reasoning that identifies hit movies before they accumulate significant audience attention. In the strongest reported setting, Llama 3.1 405B with the V4 prompt (genre, synopsis, content ratings, character types, mood, era, cast, crew, and awards) improves ACC@1 by 28.33%, reciprocal rank by 22.46%, NDCG@3 by 12.90%, and recall@3 by 31.42% relative to the BERT V4 embedding baseline. The paper also finds that richer prompts help larger models but hurt the smallest 8B model, and that listwise ranking works better than pairwise ranking for this open-ended cold-start task. These results are presented as evidence that LLM-based scoring can complement human editorial review and reduce the exposure bias against new content.","pith_inferences":["Editor's inference: the reported edge would evaporate if the model's training data contained post-release chatter, so a masked-release-date control is the key validity test.","Editor's inference: the method likely transfers to other content domains with rich metadata, but movie-specific effects such as cast drawing power may not generalize to books or music.","Editor's inference: since awards are part of the best prompt, the model may be forecasting industry-backed visibility rather than intrinsic quality, which could bias promotion toward already-privileged titles.","Editor's inference: a hybrid system that switches from LLM metadata scores to collaborative signals once early interactions arrive would likely outperform either approach alone."],"forward_implications":["Production cold-start pipelines can use LLM-generated scores as an early ranking signal before collaborative filters have user interaction data.","Richer metadata improves large-model forecasts: adding cast, crew, and awards to the prompt produces the largest gains, so data enrichment of new titles pays off.","The prompt strategy must be matched to model scale, because the 8B model loses accuracy when complex metadata is added while the 405B model gains.","Structured JSON instructions and post-processing make LLM ranking outputs reliable enough for a practical retrieval or editorial workflow.","Pairwise ranking appears less suited than listwise ranking for open-ended cold-start lists where candidates share no common query."],"supporting_citations":[{"why":"Supplies the Llama 3.1 and 3.3 model family used for generative ranking in the main experiments.","marker":"[5]"},{"why":"Supplies BERT, the embedding baseline against which LLM improvements are measured.","marker":"[4]"},{"why":"Supplies the Linq-Embed-Mistral embedding model used as a strong baseline candidate.","marker":"[3]"},{"why":"Provides the embedding benchmark used to select the strong dense baseline.","marker":"[6]"},{"why":"Motivates the pairwise ranking strategy that the paper tests and finds less effective in the cold-start setting.","marker":"[10]"},{"why":"Supplies the setwise/listwise ranking prompting approach adapted in the main experiments.","marker":"[14]"}],"fun_headline_variants":["LLM predicts movie hits before any audience data","Cold-start movie hits predicted by LLMs","LLM ranks movie hits from metadata alone","LLM beats BERT for cold-start movie hits","LLM forecasts movie hits before release"],"cache_read_input_tokens":8704,"weakest_assumption_plain":"The argument depends on the 6-12 month gap between the LLM's training-data cutoff and each movie's release date: if the model has already seen publicity, reviews, or popularity signals for a movie, its predictions are memorization, not forecasting.","fun_headline_variants_meta":{"raw":{"variants":["LLM predicts movie hits before any audience data","Cold-start movie hits predicted by LLMs","LLM ranks movie hits from metadata alone","LLM beats BERT for cold-start movie hits","LLM forecasts movie hits before release"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2341,"prompt_tokens":841,"completion_tokens":1500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1432}},"tokens_in":457,"tokens_out":1500,"duration_ms":11710,"temperature":1.0,"reasoning_tokens":1432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:43:28.839292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a matched test set of movies released before the model's knowledge cutoff and ask the model to rank them as if they were new, then compare performance to the original after removing release dates and award names from the prompt; if performance collapses, the reported forecasting edge was memorization of post-release information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the embedding benchmark used to select the strong dense baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies BERT, the embedding baseline against which LLM improvements are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the setwise/listwise ranking prompting approach adapted in the main experiments."}],"review_version":1}