{"id":"4e9b1e51-b6ca-4245-966d-697d5477c981","arxiv_id":"1908.03180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Moviescope provides a 5,000-movie multimodal dataset and shows that simple average-pooled frame and word features (fastVideo, fastText) outperform LSTMs and action-recognition models for movie genre and budget prediction.","lead":"This paper introduces Moviescope, a dataset of about 5,000 movies with trailers, audio, posters, plots, and metadata, and benchmarks simple pooled features against recurrent and 3D networks for predicting genre and budget. A generalist should read it for the finding that simple average-pooling of pretrained features can beat more complex temporal models on movie-level prediction tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset alignment is unverified: Section 3's 'first YouTube result' and regex-parsed plots lack manual or automated quality control despite a claim of manual curation, and a nontrivial mismatch rate would undermine the fastVideo comparison and all content-based results.","rationale":"The reader's weakest assumption—that trailer/plot alignment is accurate—is directly supported by the paper's own text. Section 3 describes a fully automatic pipeline for both modalities and offers no verification step; the Introduction's promise of 'manually curated mappings' is not reflected in the methodology. This is a missing support that should be flagged. Because the central claim about fastVideo is an empirical comparison on Moviescope, any misalignment introduces measurement error; if the error is correlated with genre or budget (likely for less-popular films), the ranking of methods could change. A random mismatch would dilute every method equally and could hide real differences, while a systematic mismatch could create spurious edges. Therefore the dataset's integrity is the single most load-bearing link. The paper does have strengths: a clear held-out split, a reasonable set of baselines, and a new multimodal resource. However, without verification of the data, the results are conditional on that verification. Hence the reader's CONDITIONAL verdict remains appropriate, and we recommend no change. The concrete test—a human audit of a stratified sample—would settle whether the concern actually lands.","tokens_in":14157,"tokens_out":7872,"duration_ms":79135,"concrete_test":"Randomly sample 100 movies from the final 4,927 used in experiments, stratified across genres and budget tiers. Have at least two independent annotators (or one plus automated checks) confirm that the downloaded trailer is for the same film as the IMDb record (title, year, and if possible cast/scenes), and that the parsed plot belongs to that same film. Compute the mismatch rate; if it exceeds, say, 5% or is significantly higher in a particular stratum, the reported mAP comparisons should be re-run after removing or correcting mismatches to see if fastVideo's margin over C3D/LSTM persists. Also check whether the claimed 'manually curated mappings' exist by asking for a sample of curation logs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fastVideo outperforms LSTM/C3D/I3D rests entirely on the Moviescope dataset. Section 3 ('Video trailers') states that for each movie the pipeline 'fetches the first search result' of a YouTube query that appends 'trailer', and that this 'guarantee[s] downloading the best possible video trailer'—a guarantee with no support. No manual verification, agreement metric, or quality-control threshold is described for the 4,996 automatically fetched trailers; the remaining 47 were manually downloaded but not verified either. Similarly, 'Text plots' relies on regex parsing of Wikipedia HTML, with fallback to CMU Movie Summary Corpus or IMDb storyline, but no disambiguation step (e.g., checking title/year against the IMDb record) is specified. Yet the Introduction claims the release includes 'manually curated mappings between these two modalities'—a statement not corroborated in the data-collection section. If even a modest fraction of trailers or plots belong to different movies (or to clips/reviews rather than trailers), then every content-based result in Tables 3 and 5 is computed on mislabeled inputs. The effect need not be random: first-search-result crawls are likely to fail more for older, obscure, or non-English films, which are precisely the classes that may be under-represented in genres like 'Biography' or budget tiers, potentially changing the relative ranking of methods. Consequently, the dataset's integrity is the linchpin of the paper's empirical contributions, and it is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Moviescope, a multimodal dataset of about 5,000 movies with YouTube trailers (video and audio), Wikipedia-derived plots, posters, and IMDb-based metadata, and uses it to study multi-label genre classification and budget estimation. It proposes fastVideo, which average-pools VGG-16 frame features over a subsampled sequence, as a video analog of fastText for text. Experiments compare fastVideo against LSTM, bidirectional LSTM, C3D, and I3D for video; fastText and LSTM for text; a CRNN for audio; VGG-16 for posters; and random forest/XGBoost for metadata, as well as score fusion. The central empirical claim is that simple pooling of pretrained features outperforms recurrent and 3D-convolutional baselines for both tasks, and that text and audio are strong predictors. The paper also reports a human study for genre prediction and comparisons on UCF101.","tokens_in":14456,"tokens_out":7300,"duration_ms":68957,"significance":"If the empirical comparisons hold, the paper makes a useful contribution: it provides a new multimodal benchmark, demonstrates that average-pooled pretrained features are a strong and cheap baseline for movie-level semantics, and identifies several non-obvious findings (e.g., audio predicts budget better than video). The planned release of the dataset, pretrained embeddings, and code would increase reproducibility. However, the value of the benchmark and the validity of the central comparison depend critically on the integrity of the crawled trailer-plot-movie alignments and on fair, controlled baselines, both of which are currently insufficiently documented.","major_comments":[{"comment":"The paper claims that the pipeline 'guarantee[s] downloading the best possible video trailer' and that the release includes 'manually curated mappings between these two modalities,' but Section 3 describes only fetching the first YouTube search result for each query and regex-parsing Wikipedia HTML, with no manual verification, agreement metric, or quality-control threshold. The 4,996 automatically downloaded trailers and the parsed plots are not checked against the IMDb records (e.g., for title/year match). This is load-bearing because every content-based result in Tables 3 and 5 (including the fastVideo vs. LSTM/C3D/I3D comparison) is computed on these pairings; if even a modest fraction are wrong, the rankings can change. The authors should report a validation of the alignments, for instance a manual audit on a random sample with an estimated mismatch rate.","section":"Section 3 and Introduction"},{"comment":"The paper states that 'choosing random clips of 16 or 49 continuous frames worked better than using 200 frames at once' and that at test time the LSTM sees 12 clips of 16 frames and 4 clips of 49 frames, which are then averaged. This protocol prevents the LSTM from ever seeing the full temporal extent of the trailer, while fastVideo averages over 200 frames drawn from the whole trailer. The comparison therefore conflates the aggregation method with the amount of temporal context available. The authors should either provide LSTM results on the full 200-frame sequence (with comparable capacity) or justify why the clip-based LSTM is the appropriate strong baseline; otherwise the central claim that fastVideo 'provides higher levels of accuracy' is not fully supported.","section":"Section 6, Table 3 and LSTM baseline paragraph"},{"comment":"The table is titled 'Mean Average Precision Scores on UCF101,' but the baseline numbers cited from [16,34,37,28,5] are top-1 classification accuracies reported in the original papers (UCF101 is a single-label dataset). Comparing these published accuracy numbers to the mAP numbers reported for fastVideo is an uncontrolled, apples-to-oranges comparison. The authors should reimplement the baselines under the same evaluation protocol (same metric, same input frames, same downstream classifier) or at minimum disclose the metric mismatch and limit conclusions accordingly. As written, the statement that fastVideo 'does outperform similar competing approaches such as LSTM encodings or C3D' on UCF101 is not justified.","section":"Section 6, Table 4 (UCF101)"},{"comment":"All results are reported as single mAP values from one random 70/10/20 split, with no error bars, confidence intervals, or significance tests. Given that several comparisons are close (e.g., fastVideo vs. C3D on several genres in Table 3, and I3D RGB vs. fastVideo on budget tiers in Table 5), the reader cannot determine whether the observed differences are statistically reliable. The authors should report results over multiple seeds and/or multiple splits, and perform a significance test (e.g., paired bootstrap) for the headline fastVideo vs. LSTM/C3D/I3D differences.","section":"Tables 3 and 5, Section 6"}],"minor_comments":[{"comment":"The number of movies is stated as 5,000 in the abstract, 5,043 in Section 3 ('IMDb5000 dataset consisting of 5,043 movie records'), 4,996 after trailer download, and 4,927 in training after discarding those without trailer/plot; please make the dataset size consistent and clarify the exact splits.","section":"Section 3 and Table 1"},{"comment":"The caption contains the typo 'previoulsy' and should read 'previously.'","section":"Figure 1 caption"},{"comment":"The word 'enconding' should be 'encoding.'","section":"Section 4.1"},{"comment":"The text contains the typos 'Morever' (should be 'Moreover') and 'cleary' (should be 'clearly').","section":"Section 6"},{"comment":"The sentence about supplementing frames ('starting at the 200th frame by taking one every 6 frames') is confusing; specify whether the second pass starts after the first 200 sampled frames or at the 200th frame of the trailer.","section":"Section 4.2"},{"comment":"The summation indices start at j=0 and i=0, but the text refers to word/frame embeddings starting from the first element; use j=1 and i=1 or define the zero-index convention.","section":"Equations (1) and (2)"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful dataset and an interesting empirical study, but the central comparisons currently rest on unverified alignments and a mismatched UCF101 baseline. The revision should prioritize release of the alignment audit and controlled baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: the paper's real contribution is the dataset concept—5k movies with trailers, plots, posters, and metadata in a 13-way multi-label genre scheme—and the empirical claim that average-pooled pretrained features (fastText/fastVideo) beat LSTM and 3D models on movie-level tasks. That claim is plausible and the paper gives it a fair initial test, including held-out evaluation and a human study. It deserves a serious referee, but not acceptance in this form.\n\nWhat it does well: the experimental structure is clear. They compare five modalities, build a fusion with attention weights, and check against human workers. The finding that text and audio are better for budget than video is non-obvious. The budget tiers are sensible. The UCF101 table is labeled honestly: fastVideo is not SOTA, and they say so.\n\nThe soft spots are real. The data collection in Section 3 is under-specified: 'first YouTube result' plus regex-parsed Wikipedia plots, with no manual verification or quality metric. The intro and related work claim manual curation, but the collection section never describes that manual step. That discrepancy matters for a dataset paper. If a nontrivial share of trailers or plots are misaligned, the benchmark numbers are tainted. The effect may not be random—older and obscure films are more likely to fetch wrong videos. The authors should add a verification step or soften the guarantee.\n\nThe other issues are more moderate. There are no error bars or significance tests, so the margin of fastVideo over C3D (56.5 vs 53.4 mAP) may be noise. The LSTM clip lengths (16/49 frames) were chosen on validation, which is fine, but they don't report variance across splits. The UCF101 comparison uses published numbers without a controlled rerun, so it's context, not evidence. And the dataset is only promised, not released—the abstract and intro say 'plan to release,' so the reproducibility claims are forward-looking.\n\nBottom line: the core idea is sound and the paper is a useful benchmark contribution for anyone working on movie understanding or multimodal analysis. But the data integrity concern is the linchpin, and the lack of release and error bars make the result conditional. I'd send it to review, asking for verification of the alignments, error bars, and a clear release commitment.","headline":"Moviescope is a genuinely useful multimodal movie benchmark with a plausible finding that simple pooling beats recurrent models, but the data alignment is under-verified and the dataset is only promised for release, so the paper needs revision before the claims are fully acceptable.","tokens_in":15021,"tokens_out":3048,"would_cite":false,"duration_ms":32255,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that simple average-pooled pretrained frame features outperform LSTM, C3D, and I3D models for predicting movie genres and budgets.","keywords":["movie understanding","multimodal learning","video representation","average pooling","genre classification","budget estimation","trailer analysis","dataset"],"falsifier":"Take a random sample of about 200 Moviescope entries, manually check whether the downloaded trailer and parsed plot actually correspond to the IMDb movie, and record the mismatch rate; then rerun the fastVideo versus LSTM genre comparison on only verified pairs and see whether the reported advantage disappears.","tokens_in":13952,"feed_emoji":"🎬","tokens_out":3802,"duration_ms":39669,"temperature":0.7,"pith_summary":"This paper sets out to show that for understanding movies at the level of genre and budget, simple content-based representations can beat more complex temporal models, and that different modalities carry complementary signals. It introduces Moviescope, a dataset of more than 5,000 movies with trailers, plots, posters, and metadata, and benchmarks several encodings. The central finding is that fastVideo and fastText, which average frame and word features, outperform LSTM, C3D, and I3D baselines on these holistic movie tasks. If the finding holds, average pooling with pretrained features is a strong, cheap baseline for other long-form video classification tasks.","feed_headline":"Simple frame averaging beats deep video models for movie labels","feed_subtitle":"On a 5,000-movie benchmark, pooled VGG features outperform LSTMs, C3D, and I3D for genre and budget.","key_machinery":"The central objects are fastText and its video analogue fastVideo: representations built by feeding pretrained per-word or per-frame features through an affine layer after a global average pooling operation over the sequence. For video, frames are subsampled to 200 per trailer and encoded with the penultimate layer of VGG-16; for text, plots up to 3,000 words are encoded with GloVe embeddings. This pooling operation replaces explicit temporal modeling, and it is what allows the paper to compare cheap holistic encodings against LSTM, C3D, and I3D on holistic movie-level tasks.","core_discovery":"Moviescope is a new multimodal dataset of 5,027 movies, with YouTube trailers (video plus audio), Wikipedia plot summaries, posters, and IMDb-derived metadata. On this data, for multilabel genre prediction over 13 non-exclusive genres and for 5-tier budget estimation, the paper's fastVideo encoding—average pooling over VGG-16 frame activations—achieves higher mean average precision than LSTM, bidirectional LSTM, C3D, and Two-Stream I3D baselines. fastText, the analogous average pooling over GloVe word embeddings, is the strongest single modality for genre, and audio and text outperform video for budget. Fusing all modalities with learned per-genre attention weights yields the best overall scores.","pith_inferences":["Editorial inference: the same average-pooling recipe may transfer to other long-form media tasks such as TV episode classification, because the paper's evidence suggests temporal order is not crucial for holistic labels.","The dataset's alignment quality is the key risk; a small manual audit of trailer–plot–movie matches would quantify the mismatch rate and could be reported alongside the benchmark.","The modality attention weights suggest a content-based recommender could emphasize trailers for animation, plots for drama, and audio for high-budget films—an application the paper does not build out."],"forward_implications":["Simple pooling with pretrained features should be the default baseline for movie-level semantic tasks, since it beats recurrent and 3D convolutional models at lower computational cost.","Trailers carry enough information about a full movie to predict genre, so trailer-level content can support recommendation and retrieval without full-length video analysis.","Audio is a stronger budget signal than video, suggesting production scale is audible in sound design and mixing.","Multimodal fusion with per-genre attention improves over any single modality, and the learned attention weights reveal which signal matters most for each genre.","The dataset provides a new benchmark for joint video, text, audio, and image understanding, with aligned modalities for the same movies."],"supporting_citations":[{"why":"Supplies the fastText pooling-based text classification method that fastVideo adapts to video frames.","marker":"[14]"},{"why":"Provides the GloVe word embeddings used to encode movie plot text.","marker":"[20]"},{"why":"Provides the pretrained VGG-16 network whose penultimate activations form frame and poster features.","marker":"[29]"},{"why":"Defines the C3D 3D convolutional baseline that fastVideo is compared against and outperforms.","marker":"[37]"},{"why":"Defines the Two-Stream I3D baseline for video representation and the UCF101 comparison.","marker":"[5]"},{"why":"Defines the LSTM temporal aggregation baseline for both text and video.","marker":"[12]"},{"why":"Serves as the fallback source of plot summaries from the CMU Movie Summary Corpus.","marker":"[2]"},{"why":"Provides the convolutional recurrent neural network architecture used for audio representation.","marker":"[7]"},{"why":"Introduces the prior LTMD trailer dataset with four mutually exclusive genres, the comparison point for genre-task design.","marker":"[27]"},{"why":"Introduces an earlier movie-trailer genre classification dataset that motivates the canonical task.","marker":"[38]"}],"fun_headline_variants":["Simple pooling beats deep nets for movie genre and budget","Moviescope: 5,000 movies reveal shallow beats deep","Average pooling outdoes LSTM, C3D, I3D on movie labels","Plain averaging wins for movie prediction in new dataset","Multimodal fusion best for movie genre and budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatically downloaded first YouTube result for each movie-title-plus-trailer query and the regex-parsed Wikipedia plot are correctly matched to the corresponding IMDb movie record, with no manual verification of the alignment for the full dataset.","fun_headline_variants_meta":{"raw":{"variants":["Simple pooling beats deep nets for movie genre and budget","Moviescope: 5,000 movies reveal shallow beats deep","Average pooling outdoes LSTM, C3D, I3D on movie labels","Plain averaging wins for movie prediction in new dataset","Multimodal fusion best for movie genre and budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1108,"prompt_tokens":856,"completion_tokens":252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":167}},"tokens_in":472,"tokens_out":252,"duration_ms":3142,"temperature":1.0,"reasoning_tokens":167,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:20:40.496616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 Moviescope entries, manually check whether the downloaded trailer and parsed plot actually correspond to the IMDb movie, and record the mismatch rate; then rerun the fastVideo versus LSTM genre comparison on only verified pairs and see whether the reported advantage disappears.","supporting_citations":[{"cited_title":"Pennington, R","cited_arxiv_id":null,"evidence_quote":"Provides the GloVe word embeddings used to encode movie plot text."},{"cited_title":"Bamman, B","cited_arxiv_id":null,"evidence_quote":"Serves as the fallback source of plot summaries from the CMU Movie Summary Corpus."},{"cited_title":"Convolutional Recurrent Neural Networks for Music Classification","cited_arxiv_id":"1609.04243","evidence_quote":"Provides the convolutional recurrent neural network architecture used for audio representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the prior LTMD trailer dataset with four mutually exclusive genres, the comparison point for genre-task design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces an earlier movie-trailer genre classification dataset that motivates the canonical task."}],"review_version":1}