{"id":"930f41f0-4a55-4337-802c-5ce7e819f3f1","arxiv_id":"1908.11310","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"By filtering noisy web comments, the authors built AVA-Captions, a 230,000-image aesthetic captioning dataset, and showed a weakly supervised CNN can match ImageNet-pretrained features for this task.","lead":"Researchers built a large dataset of aesthetic photo critiques by automatically cleaning noisy comments from a photography website. They also trained the image-understanding network without expensive labels, using topic patterns found in the comments, and show the cleaned data improves caption quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed CWS/CS parity in Table 1(a) is not isolated from the LSTM's language prior; without a no-visual-feature ablation or variance estimates, the weak-supervision contribution may be an artifact.","rationale":"The reader's weakest assumption concerns the visual-comment correlation underlying the weakly supervised CNN. I agree that this is the fragile point of the central claim, but I would sharpen the concern: the paper does not ablate the visual feature contribution, so even if the correlation holds, the reported parity between CWS and CS could be produced by the language model alone. The proposed concrete test (constant-feature baseline) directly addresses whether the CNN features matter at all. This concern does not overturn the paper's plausible contributions, especially the cleaning strategy and dataset; however, it does mean the weak-supervision conclusion needs stronger evidence. Since the reader already assigned CONDITIONAL, the verdict should remain unchanged, with the ablation and variance reporting as explicit conditions for acceptance.","tokens_in":15405,"tokens_out":7295,"duration_ms":71260,"concrete_test":"Train the same NeuralTalk2 LSTM on the same AVA-Captions train/validation split with three feature extractors: (1) ImageNet-ResNet101 (CS), (2) the proposed LDA-supervised ResNet101 (CWS), and (3) a constant feature vector (e.g., all zeros) or a randomly initialized, fixed CNN, keeping all LSTM hyperparameters identical. Compare CIDEr, SPICE, and S-1 on the validation set. If condition (3) performs within a few hundredths of CWS and CS, the visual features are not the cause of the gains and the weak-supervision claim fails. Additionally, run each condition with at least 3 random seeds and report standard deviations to check whether the CS/CWS differences are within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central weak-supervision claim in Section 6.3.1 ('the performance of CWS and CS is comparable') rests on Table 1(a), where CWS and CS differ by at most 0.002 on CIDEr and SPICE, yet no error bars or significance tests are reported. More fundamentally, the comparison does not isolate the contribution of the visual features. For both CS and CWS, the LSTM is trained on the same cleaned captions, so the language model can dominate the scores. If the LSTM generates similar captions regardless of the image feature vector, then CWS and CS would appear comparable even if the weakly supervised CNN (Section 4.3) learned nothing useful about aesthetics. The paper provides no baseline with a constant or randomly initialized CNN feature extractor, leaving open the possibility that the reported accuracy and diversity gains over NS are driven primarily by the caption cleaning and the language prior rather than by the learned visual representation. The assumption in Section 4.3 that visual aesthetic attributes correlate with comment topics is plausible but never directly validated; the qualitative topic-image examples in Figure 3 are selected and do not quantify this correlation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses aesthetic image captioning by (1) building AVA-Captions, a cleaned large-scale dataset from noisy AVA user comments via a probabilistic n-gram informativeness score (Eq. 2), and (2) proposing a weakly supervised CNN that is trained to predict LDA topic distributions of the cleaned captions. The authors compare three configurations: NS (noisy captions + ImageNet CNN), CS (cleaned captions + ImageNet CNN), and CWS (cleaned captions + weakly supervised CNN), using automatic metrics, diversity measures, transfer to PCCD, and human studies. The central claims are that cleaning improves accuracy and diversity, and that CWS performs comparably to CS.","tokens_in":15585,"tokens_out":3719,"duration_ms":33351,"significance":"If the results hold, the paper contributes the first large-scale benchmark for aesthetic image captioning and demonstrates that a CNN trained with weak topic-level supervision can substitute for ImageNet pretraining in this task, which is practically important when clean annotations are unavailable. The strengths include the human evaluation of the filtering strategy and a consistent set of automatic metrics showing the cleaning benefit. The dataset itself is a valuable resource for the community. However, the parity claim for weak supervision is not yet statistically supported.","major_comments":[{"comment":"The claim that 'the performance of CWS and CS is comparable' is not supported by significance tests or error bars; the CIDEr and SPICE differences are at most 0.002. Moreover, because both CS and CWS share the same LSTM trained on the same cleaned captions, the comparison does not isolate the visual features; a baseline with random or constant CNN features is needed to show that the weak supervision contributes beyond the language model. Please report multiple runs with standard deviations and add such a no-visual-feature control.","section":"Section 6.3.1, Table 1(a)"},{"comment":"The weak supervision pipeline is self-referential in part: the cleaned captions are used to infer LDA topics, to train the CNN, and as references for evaluating the final captions. The assumption that visual aesthetic attributes are correlated with comment topics is only illustrated with selected qualitative examples in Figure 3. To establish that the CNN learns meaningful aesthetic features, please validate the topic-image correlation quantitatively, for example by reporting topic prediction accuracy on held-out images or by training with permuted topic labels as a control.","section":"Section 4.3"},{"comment":"The informativeness threshold (set at 20) is chosen 'experimentally' but no sensitivity analysis is provided. Since the cleaning contribution rests on the separation of informative and uninformative captions, the paper should report automatic metrics and human consistency for a range of thresholds, or justify the chosen value with a validation-based criterion.","section":"Section 3, Eq. (2)"}],"minor_comments":[{"comment":"The word 'meanigful' is a typo for 'meaningful'.","section":"Section 3"},{"comment":"The category 'Com' should be spelled out as 'Average' to match the description in the text.","section":"Section 6.3.4, Table 2"},{"comment":"The notation 'AVA-CP' should be defined in the caption, and the comparison with AF (which remains the best) should be discussed explicitly to temper the 'impressive generalizability' claim.","section":"Table 1(b)"},{"comment":"The choices of K=200 and the 10% frequency cutoff are introduced without supporting analysis; a brief justification or reference to a sensitivity check would help.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution and the cleaning-versus-noisy comparison are likely to be of interest to the community. The main risk is the weak supervision parity claim; adding the suggested control experiments would substantially strengthen the paper. The authors may also consider releasing the cleaned dataset and code to facilitate reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Alice,\n\nYou should know two things. First, this is the first large-scale aesthetic image captioning (AIC) benchmark: AVA-Captions, ~230k images with ~5 filtered comments each, roughly 60x the size of PCCD. Second, the cleaning strategy is the best-supported contribution; the weakly supervised CNN claim is plausible but not nailed down.\n\nThe new stuff: a probabilistic n-gram informativeness filter that discards ~55% of noisy AVA comments, an LDA topic model over the cleaned comments to produce 200 pseudo-labels for training a ResNet101, and an evaluation of the full CNN-LSTM pipeline against noisy-data and clean-data baselines. The contrast between NS (noisy) and CS/CWS (clean) is consistent across BLEU, METEOR, CIDEr, SPICE, diversity plots, and a small human study. That makes the dataset curation defensible.\n\nThe soft spots are in the weak supervision story. The headline claim \"CWS and CS are comparable\" rests on Table 1(a), where differences are at most 0.002 in CIDEr/SPICE, but there are no error bars or significance tests. More importantly, there is no baseline with a constant or randomly initialized CNN, so the language model alone could be driving the scores. The LSTM is trained on the same cleaned captions for both CS and CWS; if it ignores the image features, you would see the same apparent parity even if the weakly supervised CNN learned nothing about aesthetics. A no-visual-feature ablation would settle that. Also, the filtering threshold and LDA topic count are tuned on the same corpus, with no sensitivity analysis, and the dataset and code are not released, which limits the benchmark's near-term utility. The assumption that comment topics correlate with visual aesthetics is plausible but only supported by selected qualitative examples.\n\nNone of this kills the paper. The cleaning contribution stands, and the human evaluation adds support. But the weak supervision claim needs the missing control before it is cited as evidence.\n\nWho is this for? Researchers working on captioning, computational aesthetics, or weakly supervised representation learning. It deserves a serious referee and a major-revision path.\n\nBest,\n[You]","headline":"AVA-Captions is a genuinely useful large-scale AIC benchmark and the cleaning result is solid, but the claimed parity between weakly supervised and ImageNet features needs error bars and a no-visual-feature control.","tokens_in":16150,"tokens_out":2154,"would_cite":true,"duration_ms":20241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Filtering noisy photography comments yields a 230,000-image aesthetic-captioning dataset, and weakly supervised topic labels match ImageNet-trained features.","keywords":["aesthetic image captioning","weakly supervised learning","Latent Dirichlet Allocation","caption filtering","AVA-Captions","CNN-LSTM","image aesthetics","noisy web data"],"falsifier":"Train a copy of the weakly supervised CNN on the same AVA-Captions data but with the image-comment pairs randomly reshuffled, so each image is paired with topic labels drawn from another image; if the captioning metrics on the validation set are essentially unchanged from the properly paired model, then the topic labels are not teaching image-specific visual features and the reported parity with ImageNet would be an artifact of the language model.","tokens_in":15176,"feed_emoji":"📷","tokens_out":12872,"duration_ms":112516,"temperature":0.7,"pith_summary":"The paper sets out to make aesthetic image captioning trainable at scale without hand-curated labels. It claims that a probabilistic n-gram informativeness score can filter the noisy user comments in the AVA dataset down to a clean corpus of about 230,000 images with roughly five informative critiques each, called AVA-Captions. It also claims that topic labels inferred from those comments with Latent Dirichlet Allocation can train a ResNet to extract aesthetic visual features that match an ImageNet-pretrained network, so the captioning pipeline no longer depends on expensive manual aesthetic annotations. The experiments support the claim by showing that a captioning model trained on the cleaned captions outperforms one trained on the raw comments on all reported metrics, and that the weakly supervised visual extractor performs comparably to the supervised one.","feed_headline":"Filtered photo critiques yield a 230K-image captioning dataset","feed_subtitle":"A probabilistic score drops filler comments, and topic-model labels match ImageNet-pretrained features.","key_machinery":"Two mechanisms carry the argument. The first is the informativeness score $\\rho_s = -\\frac{1}{2}\\left[\\log \\prod_i P(u_i) + \\log \\prod_j P(b_j)\\right]$, computed for each comment as the union of its noun unigrams and descriptor-object bigrams, with corpus probabilities $P(\\omega)=C_\\omega / \\sum_i C_i$; a low score marks a safe, uninformative comment, and a threshold of 20 discards about 55% of the raw corpus. The second is Latent Dirichlet Allocation (LDA), a generative topic model that represents each document as a mixture of topics and each topic as a mixture of words, run on all captions attached to an image as a single document; it discovers $K=200$ latent aesthetic topics, and the predicted topic distribution of an image's comments becomes the weak label that trains the final 200-way layer of a ResNet101 by cross-entropy. The LDA topics are the bridge that turns free-form text into visual supervision, on the assumption that images described with similar words share aesthetic properties.","core_discovery":"The core discovery is that the same web comments that look too noisy for supervised aesthetic captioning carry a usable signal once they are filtered by an informativeness score and summarized by topic modelling. The paper defines the informativeness of a comment as the average negative log probability of its noun unigrams and descriptor-object bigrams, so rare phrases such as \"post processing\" or \"top half\" count as informative while frequent fillers such as \"nice shot\" are discarded; thresholding this score removes about 55% of the corpus and leaves AVA-Captions. From the cleaned comments, Latent Dirichlet Allocation with 200 topics produces per-image topic distributions that serve as weak labels for training a ResNet101 with cross-entropy loss. In the final CNN-LSTM framework, the model trained this way (CWS) is comparable, on BLEU, METEOR, ROUGE, CIDEr, SPICE, and diversity, to the same framework with an ImageNet-trained CNN (CS), and both clearly outperform the version trained on raw noisy comments (NS). The paper presents this as evidence that the weak labels transfer real aesthetic content to the visual features, not just to the language model.","pith_inferences":["The 200 discovered topics could serve as an interpretable vocabulary of aesthetic attributes, which would allow generation to be steered toward or away from specific qualities such as motion blur or black-and-white treatment.","A length-calibrated version of the informativeness score is a natural next step, since the paper's own subjective study found that experts judged some long captions as false positives.","The two-stage pipeline trains the CNN and the LSTM separately, so an end-to-end variant that lets the captioning loss back-propagate into the weakly supervised CNN might close the small remaining gap between the weakly supervised and supervised models."],"forward_implications":["A captioning model trained on AVA-Captions scores higher than the same model trained on raw AVA comments on every automatic metric reported, so filtering alone drives a significant accuracy gain.","The weakly supervised CNN matches the ImageNet-supervised CNN on accuracy and diversity, so aesthetic visual features can be learned without manual labels whenever weakly labelled photo collections are available.","Models trained on AVA-Captions transfer to the independently collected PCCD validation set, so the cleaned data generalize beyond the source website.","Since the recipe is generic, the cleaning-plus-topic-label strategy can be applied to other weakly labelled captioning tasks such as visual storytelling and stylized captioning.","AVA-Captions is roughly sixty times the size of the previous PCCD benchmark, allowing deeper architectures and more complex aesthetic representations than before."],"supporting_citations":[{"why":"Supplies the AVA photographs and raw user comments that form the input for the cleaning and weak-supervision pipeline.","marker":"[58]"},{"why":"Introduces the prior aesthetic-captioning task, the PCCD benchmark, and the baseline models and human-evaluation categories used for comparison.","marker":"[11]"},{"why":"Latent Dirichlet Allocation, the topic model used to turn cleaned comment collections into weak visual labels.","marker":"[10]"},{"why":"ImageNet, the dataset used to train the supervised CNN features that the weakly supervised features are compared against.","marker":"[15]"},{"why":"ResNet101, the CNN architecture trained with weak topic labels and used for the supervised ImageNet features.","marker":"[26]"},{"why":"Provides the CNN-LSTM captioning pipeline on which the noisy, clean, and weakly supervised baselines are all built.","marker":"[46]"},{"why":"CIDEr, the consensus-based metric used to measure caption quality and show that cleaned data beats noisy data.","marker":"[78]"},{"why":"SPICE, the semantic scene-graph metric whose standard and best-reference forms are both reported in the accuracy comparisons.","marker":"[2]"},{"why":"Defines the per-position unique n-gram diversity measure used to show that cleaned training data produce more varied captions.","marker":"[3]"}],"fun_headline_variants":["Why noisy photo comments suffice for aesthetic AI","230K image captions from web critique cleanup","Weak labels from comments match pretrained CNNs","Aesthetic captioning gets a big data boost from web noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the words people write about a photograph reliably reflect visual aesthetic properties that are actually present in the image, so topic distributions inferred from comments can stand in for visual labels when training the CNN.","fun_headline_variants_meta":{"raw":{"variants":["Why noisy photo comments suffice for aesthetic AI","230K image captions from web critique cleanup","Weak labels from comments match pretrained CNNs","Aesthetic captioning gets a big data boost from web noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1620,"prompt_tokens":973,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":586}},"tokens_in":589,"tokens_out":647,"duration_ms":6688,"temperature":1.0,"reasoning_tokens":586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:19:01.110684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a copy of the weakly supervised CNN on the same AVA-Captions data but with the image-comment pairs randomly reshuffled, so each image is paired with topic labels drawn from another image; if the captioning metrics on the validation set are essentially unchanged from the properly paired model, then the topic labels are not teaching image-specific visual features and the reported parity with ImageNet would be an artifact of the language model.","supporting_citations":[{"cited_title":"Ava: A large-scale database for aesthetic visual analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the AVA photographs and raw user comments that form the input for the cleaning and weak-supervision pipeline."},{"cited_title":"Aesthetic critiques generation for photos","cited_arxiv_id":null,"evidence_quote":"Introduces the prior aesthetic-captioning task, the PCCD benchmark, and the baseline models and human-evaluation categories used for comparison."},{"cited_title":"Imagenet: A large-scale hierarchical im- age database","cited_arxiv_id":null,"evidence_quote":"ImageNet, the dataset used to train the supervised CNN features that the weakly supervised features are compared against."},{"cited_title":"An image captioning codebase in pytorch, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the CNN-LSTM captioning pipeline on which the noisy, clean, and weakly supervised baselines are all built."}],"review_version":1}