{"id":"936c38c0-81b3-4a89-9c0b-0f856ec8e1bb","arxiv_id":"2508.11801","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"VideoAVE is a new publicly available video-to-text e-commerce attribute value extraction dataset spanning 14 domains and 172 attributes, with benchmark results showing video-to-text AVE remains challenging.","lead":"This paper introduces VideoAVE, a video-to-text dataset for extracting product attributes from e-commerce videos, the first public dataset of its kind across 14 domains and 172 attributes. A benchmark of video language models shows the task is still hard, suggesting room for models that use temporal information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset quality rests on the CLIP-MoE filtering step, whose accuracy is unreported; benchmark conclusions are conditional on this filter being reliable.","rationale":"The reader identified the CLIP-MoE filter as the weakest assumption, and I agree. The abstract provides no methodological detail about the filter, so the central quality claim cannot be assessed from the available material. This is not an internal inconsistency; it is a missing validation step. The appropriate verdict remains UNVERDICTED because the full paper and artifacts are needed to verify the filter's accuracy and the resulting dataset quality. The concrete test would directly evaluate the filter's precision and recall, which is the single most informative check for whether the dataset and benchmark conclusions are trustworthy. I did not find a more load-bearing concern: the claims about domain coverage, attribute count, and public availability are concrete and easily checked, while the benchmark conclusions are explicitly framed as challenging, which is a soft and defensible claim.","tokens_in":675,"tokens_out":1977,"duration_ms":21313,"concrete_test":"Sample 1,000 video-product pairs retained by CLIP-MoE and 1,000 pairs removed by it. Have three independent annotators judge whether the video actually depicts or matches the product listing, using a majority vote. Compute filter precision (retained pairs that are truly matched) and recall (true matches among removed pairs). If precision on retained pairs is below 95%, or if recall on removed pairs shows a high rate of discarded true matches, the dataset quality claim and benchmark conclusions need revision. Report these numbers alongside the dataset release.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—VideoAVE is a publicly available, high-quality video-to-text AVE dataset whose benchmark reliably shows that video AVE remains challenging—depends on the post-hoc CLIP-MoE filtering system correctly removing mismatched video-product pairs. The abstract reports no evaluation of this filter: no precision, recall, human agreement, or error analysis. CLIP-style video-text matching is known to be imperfect for fine-grained product attributes and can be biased by superficial cues. If the filter lets through a substantial fraction of mismatched pairs, the 224k training and 25k evaluation splits are noisy, and the reported VLM performance on open attribute-value extraction becomes an uninterpretable mixture of attribute-extraction errors and video-product matching errors. If the filter instead discards many true matches, the benchmark underestimates model capability or at least the dataset distribution is distorted. No internal inconsistency is apparent from the abstract, but this unvalidated dependency means the dataset-quality and benchmark conclusions are unverified rather than demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VideoAVE, a publicly available video-to-text e-commerce attribute value extraction dataset spanning 14 domains and 172 unique attributes, with 224k training and 25k evaluation samples after filtering. The construction pipeline relies on a CLIP-based Mixture of Experts (CLIP-MoE) system to remove mismatched video-product pairs. The authors benchmark several state-of-the-art video vision-language models on attribute-conditioned value prediction and open attribute-value pair extraction, concluding that video-to-text AVE remains challenging, especially in open settings. The dataset and code are provided in a GitHub repository.","tokens_in":985,"tokens_out":2350,"duration_ms":24719,"significance":"If the dataset is indeed high-quality and publicly released, it fills a clear gap: existing AVE datasets are limited to text or image inputs, and video-to-text AVE is underexplored. The benchmark provides a reusable evaluation protocol for a practical e-commerce task and highlights the limitations of current video VLMs in fine-grained attribute extraction. However, the significance is conditional on the reliability of the CLIP-MoE filtering step and on the quality of the gold labels, neither of which is evidenced in the abstract. The paper's contribution would be strengthened by explicit validation of the filtering system and by transparent annotation-quality statistics.","major_comments":[{"comment":"The CLIP-MoE filter is the central data-quality mechanism, yet the abstract reports no evaluation of its accuracy, precision, recall, or agreement with human judgment. Without such validation, the 224k/25k splits may contain mismatched video-product pairs, making the downstream benchmark scores an uninterpretable mixture of attribute-extraction errors and video-product matching errors. The full manuscript must include filter validation, such as human-evaluated precision and recall on a random sample, an error analysis, and a comparison of benchmark results against an unfiltered baseline.","section":"Abstract (CLIP-MoE filtering claim)"},{"comment":"The abstract mentions two tasks, attribute-conditioned value prediction and open attribute-value pair extraction, but provides no details on evaluation metrics, input/output formats, how VLMs are adapted to sparse extraction, or how the 25k evaluation samples are split across domains and attributes. These details are essential for reproducibility and for interpreting the claim that video-to-text AVE remains challenging; without them, the benchmark conclusions cannot be verified from the abstract.","section":"Abstract (benchmark protocol)"},{"comment":"The abstract does not describe the annotation process, the number of annotators, inter-annotator agreement, or quality-control measures for the attribute-value labels themselves. If the gold labels contain substantial noise, the benchmark conclusions would be affected independently of the filtering step. The full paper should report annotation statistics and, ideally, a human evaluation of label correctness on a subset.","section":"Abstract (annotation quality)"}],"minor_comments":[{"comment":"The term 'post-hoc' is used without clarifying what it means in this context; please specify whether the filter is applied after automatic video-product pairing and before human annotation or after human annotation.","section":"Abstract"},{"comment":"The abstract uses '224k training data and 25k evaluation data'; consider using 'training samples' and 'evaluation samples' for grammatical consistency.","section":"Abstract"},{"comment":"The GitHub URL is provided, but the abstract does not state the dataset license or access terms; please include a license statement (e.g., CC-BY or a research-only license) in the final manuscript.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based on the abstract only, as the full text was not available. The stress-test concern about the unvalidated CLIP-MoE filter is legitimate and should be resolved by examining the full manuscript. If the full paper lacks filter validation and annotation-quality statistics, the verdict should be major_revision; if those details are present, the paper may be acceptable. I recommend requesting the full text before making a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the abstract for VideoAVE. The genuinely new thing is the resource: the first public video-to-text e-commerce AVE dataset, 14 domains, 172 attributes, 224k train / 25k eval splits. That fills a real gap—existing AVE data is text- or image-based, and e-commerce video understanding has been short on public benchmarks. If the dataset is clean, it's a useful contribution.\n\nWhat the paper does well on the face of it: it targets a specific practical problem, proposes a filtering method (CLIP-MoE) to remove mismatched video-product pairs, and benchmarks several video VLMs under two task settings. The task framing (attribute-conditioned value prediction vs. open extraction) is sensible and likely to expose real difficulties.\n\nThe soft spot is exactly where the stress-test note lands: dataset quality rests on the CLIP-MoE filter, and the abstract offers no evidence about its accuracy—no precision, recall, human agreement, or error analysis. CLIP-style video-text matching is known to be brittle on fine-grained attributes and can be biased by superficial scene cues. If the filter passes mismatched pairs, the training and evaluation splits are noisy and the benchmark numbers become an uninterpretable mix of attribute-extraction errors and video-product matching errors. If it discards true matches, the distribution is distorted and benchmark results understate model capability. Either way, the central quality claim is unverified rather than demonstrated. That's not a fatal flaw—dataset papers often validate filters in the body—but it's the first thing I'd check.\n\nAlso worth noting: the benchmark uses the same data the paper introduces, which is normal for dataset papers, but it means the reported \"challenging\" results are not independent validation of the filter; they're conditional on it.\n\nI can't tell from the abstract whether the filter evaluation exists in the full paper. If it does, and the human-agreement numbers are reasonable, this is a solid dataset contribution. If it doesn't, the dataset should be treated as preliminary until independent verification.\n\nMy take: this deserves a serious referee. The resource is novel and likely useful, and the filtering question is exactly what peer review should probe. I'd read the full paper before citing it, and I'd want to see the released data and filter code before using it in my own work.\n\nRecommendation: send to peer review, with referees asked to focus on filter evaluation and dataset quality checks.","headline":"A genuinely new public dataset for video-to-text AVE, but the unverified CLIP-MoE filter is the load-bearing assumption that needs fuller evidence before the dataset's quality claims land.","tokens_in":1325,"tokens_out":1444,"would_cite":false,"duration_ms":13013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video-to-text product-attribute extraction needs its own dataset, and this one proves the point.","keywords":["video-to-text","attribute value extraction","e-commerce","video language models","benchmark dataset","CLIP","mixture of experts","multimodal understanding"],"falsifier":"Take a random sample of the 224k training pairs and the rejected pairs, have human annotators label whether the video actually shows the product named in the paired text, and compare precision of kept pairs against recall of the filter. If a large share of kept pairs are mismatched, say above a few percent, or if the filter silently discards valid pairs, the dataset-quality claim and the benchmark conclusions are directly undermined.","tokens_in":517,"feed_emoji":"🎬","tokens_out":4299,"duration_ms":45707,"temperature":0.7,"pith_summary":"VideoAVE is a new, publicly released e-commerce resource for attribute value extraction (AVE) that works from product videos rather than text or still images. The paper's claim is that this video-to-text setting is currently unsupported by public data, and that a large curated collection—14 domains, 172 unique attributes, roughly 224k training and 25k evaluation video-text pairs—can expose where existing video language models fall short. A CLIP-based mixture-of-experts filter is introduced to remove mismatched video-product pairs and is presented as the mechanism that makes the dataset trustworthy. Benchmarking several current video vision language models on attribute-conditioned value prediction and open attribute-value pair extraction shows that the task remains hard, especially when attribute-value pairs must be discovered without being named in advance. A sympathetic reader would take away that video AVE remains an open problem with a reusable testbed.","feed_headline":"Video dataset shows e-commerce attribute extraction is still hard","feed_subtitle":"New public benchmark covering 14 domains and 172 attributes leaves open-set video-to-text extraction unsolved.","key_machinery":"The load-bearing object is the CLIP-based Mixture of Experts (CLIP-MoE) filtering system: a set of CLIP-derived experts whose votes are combined to decide whether a product video and its paired product record actually match. It carries the dataset-quality argument, because the benchmark conclusions are only meaningful if the filtered corpus contains few mismatched video-product pairs. The second piece of machinery is the dataset itself, organized into 14 domains with 172 attributes and split into 224k training and 25k evaluation examples, which supplies the common ground on which the video vision language models are compared.","core_discovery":"The central discovery is that video-to-text attribute value extraction is a distinct and still-unmet e-commerce task, and that a purpose-built public dataset can measure that gap. The paper constructs VideoAVE from product videos paired with structured attribute-value annotations, spanning 14 domains and 172 unique attributes. To keep the corpus clean, it introduces CLIP-MoE, a post-hoc filtering system that scores each video-product match through a mixture of CLIP-based experts and discards mismatched pairs, leaving 224k training and 25k evaluation items. On this data, state-of-the-art video vision language models are evaluated under two task formats: attribute-conditioned value prediction, where the attribute is supplied, and open attribute-value pair extraction, where the model must identify both. The results show current models handle closed, conditioned settings better than open extraction, and that temporal information in videos is not yet being exploited effectively.","pith_inferences":["Beyond the paper: because the dataset's quality rests entirely on CLIP-MoE, an independent re-annotation of a random sample of kept and rejected pairs would tell whether the benchmark numbers hold. This is a direct consequence the paper does not fully develop.","Beyond the paper: video adds temporal cues that text and images lack—object motion, packaging change, or product use over time—so a natural next step is to test whether video-only attributes are extracted better by models that attend to frame sequences.","Beyond the paper: the open-extraction failure may indicate a need for models that first propose candidate attributes from the video and then ground values in text; a pipeline that separates attribute discovery from value grounding could be evaluated directly on this benchmark."],"forward_implications":["Researchers gain a public benchmark on which video-to-text attribute value extraction can be measured and compared across methods.","Current video vision language models are shown to be insufficient for open attribute-value pair discovery, so future work has a concrete target: find pairs without being told the attribute first.","The two task formats give a controlled comparison, letting future systems separate progress in recognizing attributes from progress in value extraction.","The 14-domain coverage suggests that domain-agnostic methods are still an open problem; a model trained on this set is tested on breadth as well as accuracy."],"supporting_citations":[],"fun_headline_variants":["VideoAVE: E-commerce video attribute extraction still falls short","Open video AVE unsolved in new 14-domain benchmark","New dataset shows video-to-text AVE lags on open extraction","VideoAVE benchmark: temporal cues not yet leveraged for AVE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole corpus depends on the automatic filter that decides whether a product video and its text description actually match; if that filter is systematically wrong, every conclusion drawn from the dataset inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["VideoAVE: E-commerce video attribute extraction still falls short","Open video AVE unsolved in new 14-domain benchmark","New dataset shows video-to-text AVE lags on open extraction","VideoAVE benchmark: temporal cues not yet leveraged for AVE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3282,"prompt_tokens":941,"completion_tokens":2341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2270}},"tokens_in":557,"tokens_out":2341,"duration_ms":19437,"temperature":1.0,"reasoning_tokens":2270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:25:36.995807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 224k training pairs and the rejected pairs, have human annotators label whether the video actually shows the product named in the paired text, and compare precision of kept pairs against recall of the filter. If a large share of kept pairs are mismatched, say above a few percent, or if the filter silently discards valid pairs, the dataset-quality claim and the benchmark conclusions are directly undermined.","supporting_citations":[],"review_version":1}