Aesthetic Image Captioning From Weakly-Labelled Photographs

Aakanksha Rana; Aljosa Smolic; Koustav Ghosal

REVIEW 3 major objections 4 minor 86 references

Aesthetic Image Captioning From Weakly-Labelled Photographs

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Filtering noisy photography comments yields a 230,000-image aesthetic-captioning dataset, and weakly supervised topic labels match ImageNet-trained features.

desk verdict AVA-Captions is a genuinely useful large-scale AIC benchmark and the cleaning result is solid, but the claimed parity between weakly supervised and ImageNet features needs error bars and a no-visual-feature control. read the letter →

arxiv 1908.11310 v1 pith:S5M6XT4U submitted 2019-08-29 cs.CV cs.CL

Koustav Ghosal , Aakanksha Rana , Aljosa Smolic This is my paper · ORCID

classification cs.CVcs.CL

keywords aestheticimagecaptioningweaklysupervisedlearningLatentDirichletAllocationcaptionfilteringAVA-CaptionsCNN-LSTMaestheticsnoisywebdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make aesthetic image captioning trainable at scale without hand-curated labels. It claims that a probabilistic n-gram informativeness score can filter the noisy user comments in the AVA dataset down to a clean corpus of about 230,000 images with roughly five informative critiques each, called AVA-Captions. It also claims that topic labels inferred from those comments with Latent Dirichlet Allocation can train a ResNet to extract aesthetic visual features that match an ImageNet-pretrained network, so the captioning pipeline no longer depends on expensive manual aesthetic annotations. The experiments support the claim by showing that a captioning model trained on the cleaned captions outperforms one trained on the raw comments on all reported metrics, and that the weakly supervised visual extractor performs comparably to the supervised one.

What carries the argument

Two mechanisms carry the argument. The first is the informativeness score $\rho_s = -\frac{1}{2}\left[\log \prod_i P(u_i) + \log \prod_j P(b_j)\right]$, computed for each comment as the union of its noun unigrams and descriptor-object bigrams, with corpus probabilities $P(\omega)=C_\omega / \sum_i C_i$; a low score marks a safe, uninformative comment, and a threshold of 20 discards about 55% of the raw corpus. The second is Latent Dirichlet Allocation (LDA), a generative topic model that represents each document as a mixture of topics and each topic as a mixture of words, run on all captions attached to an image as a single document; it discovers $K=200$ latent aesthetic topics, and the predicted topic distribution of an image's comments becomes the weak label that trains the final 200-way layer of a ResNet101 by cross-entropy. The LDA topics are the bridge that turns free-form text into visual supervision, on the assumption that images described with similar words share aesthetic properties.

What would settle it

Train a copy of the weakly supervised CNN on the same AVA-Captions data but with the image-comment pairs randomly reshuffled, so each image is paired with topic labels drawn from another image; if the captioning metrics on the validation set are essentially unchanged from the properly paired model, then the topic labels are not teaching image-specific visual features and the reported parity with ImageNet would be an artifact of the language model.

Watch

Extended reading notes

Core claim

The core discovery is that the same web comments that look too noisy for supervised aesthetic captioning carry a usable signal once they are filtered by an informativeness score and summarized by topic modelling. The paper defines the informativeness of a comment as the average negative log probability of its noun unigrams and descriptor-object bigrams, so rare phrases such as "post processing" or "top half" count as informative while frequent fillers such as "nice shot" are discarded; thresholding this score removes about 55% of the corpus and leaves AVA-Captions. From the cleaned comments, Latent Dirichlet Allocation with 200 topics produces per-image topic distributions that serve as weak labels for training a ResNet101 with cross-entropy loss. In the final CNN-LSTM framework, the model trained this way (CWS) is comparable, on BLEU, METEOR, ROUGE, CIDEr, SPICE, and diversity, to the same framework with an ImageNet-trained CNN (CS), and both clearly outperform the version trained on raw noisy comments (NS). The paper presents this as evidence that the weak labels transfer real aesthetic content to the visual features, not just to the language model.

Load-bearing premise

The load-bearing premise is that the words people write about a photograph reliably reflect visual aesthetic properties that are actually present in the image, so topic distributions inferred from comments can stand in for visual labels when training the CNN.

Editorial extensions

If this is right

A captioning model trained on AVA-Captions scores higher than the same model trained on raw AVA comments on every automatic metric reported, so filtering alone drives a significant accuracy gain.
The weakly supervised CNN matches the ImageNet-supervised CNN on accuracy and diversity, so aesthetic visual features can be learned without manual labels whenever weakly labelled photo collections are available.
Models trained on AVA-Captions transfer to the independently collected PCCD validation set, so the cleaned data generalize beyond the source website.
Since the recipe is generic, the cleaning-plus-topic-label strategy can be applied to other weakly labelled captioning tasks such as visual storytelling and stylized captioning.
AVA-Captions is roughly sixty times the size of the previous PCCD benchmark, allowing deeper architectures and more complex aesthetic representations than before.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 200 discovered topics could serve as an interpretable vocabulary of aesthetic attributes, which would allow generation to be steered toward or away from specific qualities such as motion blur or black-and-white treatment.
A length-calibrated version of the informativeness score is a natural next step, since the paper's own subjective study found that experts judged some long captions as false positives.
The two-stage pipeline trains the CNN and the LSTM separately, so an end-to-end variant that lets the captioning loss back-propagate into the weakly supervised CNN might close the small remaining gap between the weakly supervised and supervised models.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Alice,

You should know two things. First, this is the first large-scale aesthetic image captioning (AIC) benchmark: AVA-Captions, ~230k images with ~5 filtered comments each, roughly 60x the size of PCCD. Second, the cleaning strategy is the best-supported contribution; the weakly supervised CNN claim is plausible but not nailed down.

The new stuff: a probabilistic n-gram informativeness filter that discards ~55% of noisy AVA comments, an LDA topic model over the cleaned comments to produce 200 pseudo-labels for training a ResNet101, and an evaluation of the full CNN-LSTM pipeline against noisy-data and clean-data baselines. The contrast between NS (noisy) and CS/CWS (clean) is consistent across BLEU, METEOR, CIDEr, SPICE, diversity plots, and a small human study. That makes the dataset curation defensible.

The soft spots are in the weak supervision story. The headline claim "CWS and CS are comparable" rests on Table 1(a), where differences are at most 0.002 in CIDEr/SPICE, but there are no error bars or significance tests. More importantly, there is no baseline with a constant or randomly initialized CNN, so the language model alone could be driving the scores. The LSTM is trained on the same cleaned captions for both CS and CWS; if it ignores the image features, you would see the same apparent parity even if the weakly supervised CNN learned nothing about aesthetics. A no-visual-feature ablation would settle that. Also, the filtering threshold and LDA topic count are tuned on the same corpus, with no sensitivity analysis, and the dataset and code are not released, which limits the benchmark's near-term utility. The assumption that comment topics correlate with visual aesthetics is plausible but only supported by selected qualitative examples.

None of this kills the paper. The cleaning contribution stands, and the human evaluation adds support. But the weak supervision claim needs the missing control before it is cited as evidence.

Who is this for? Researchers working on captioning, computational aesthetics, or weakly supervised representation learning. It deserves a serious referee and a major-revision path.

Best, [You]

Referee Report

3 major / 4 minor

Summary. The paper addresses aesthetic image captioning by (1) building AVA-Captions, a cleaned large-scale dataset from noisy AVA user comments via a probabilistic n-gram informativeness score (Eq. 2), and (2) proposing a weakly supervised CNN that is trained to predict LDA topic distributions of the cleaned captions. The authors compare three configurations: NS (noisy captions + ImageNet CNN), CS (cleaned captions + ImageNet CNN), and CWS (cleaned captions + weakly supervised CNN), using automatic metrics, diversity measures, transfer to PCCD, and human studies. The central claims are that cleaning improves accuracy and diversity, and that CWS performs comparably to CS.

Significance. If the results hold, the paper contributes the first large-scale benchmark for aesthetic image captioning and demonstrates that a CNN trained with weak topic-level supervision can substitute for ImageNet pretraining in this task, which is practically important when clean annotations are unavailable. The strengths include the human evaluation of the filtering strategy and a consistent set of automatic metrics showing the cleaning benefit. The dataset itself is a valuable resource for the community. However, the parity claim for weak supervision is not yet statistically supported.

major comments (3)

[Section 6.3.1, Table 1(a)] The claim that 'the performance of CWS and CS is comparable' is not supported by significance tests or error bars; the CIDEr and SPICE differences are at most 0.002. Moreover, because both CS and CWS share the same LSTM trained on the same cleaned captions, the comparison does not isolate the visual features; a baseline with random or constant CNN features is needed to show that the weak supervision contributes beyond the language model. Please report multiple runs with standard deviations and add such a no-visual-feature control.
[Section 4.3] The weak supervision pipeline is self-referential in part: the cleaned captions are used to infer LDA topics, to train the CNN, and as references for evaluating the final captions. The assumption that visual aesthetic attributes are correlated with comment topics is only illustrated with selected qualitative examples in Figure 3. To establish that the CNN learns meaningful aesthetic features, please validate the topic-image correlation quantitatively, for example by reporting topic prediction accuracy on held-out images or by training with permuted topic labels as a control.
[Section 3, Eq. (2)] The informativeness threshold (set at 20) is chosen 'experimentally' but no sensitivity analysis is provided. Since the cleaning contribution rests on the separation of informative and uninformative captions, the paper should report automatic metrics and human consistency for a range of thresholds, or justify the chosen value with a validation-based criterion.

minor comments (4)

[Section 3] The word 'meanigful' is a typo for 'meaningful'.
[Section 6.3.4, Table 2] The category 'Com' should be spelled out as 'Average' to match the description in the text.
[Table 1(b)] The notation 'AVA-CP' should be defined in the caption, and the comparison with AF (which remains the best) should be discussed explicitly to temper the 'impressive generalizability' claim.
[Section 4.3] The choices of K=200 and the 10% frequency cutoff are introduced without supporting analysis; a brief justification or reference to a sensitivity check would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cleaning and weak-supervision claims are supported by held-out metrics, human evaluation, and cross-dataset generalization, not by algebraic identity or self-citation.

full rationale

The derivation chain is empirical rather than definitional. The caption-filtering score in Eq. 2 is a corpus-statistics heuristic used to build AVA-Captions, and the paper then evaluates trained captioning models on held-out references, including human ratings and PCCD generalization; no fitted value is renamed as a prediction. The weakly supervised CNN is trained to predict LDA topic distributions derived from comments, while the LSTM is trained on the same cleaned captions, but this is a standard self-supervised pretraining setup: the CNN never directly optimizes captioning metrics, and the comparison CWS vs. CS isolates the visual feature extractor by holding the language model fixed. The absence of a no-visual-feature baseline is a legitimate experimental limitation, not circularity, because the reported parity is an empirical outcome rather than a constructional identity. Self-citations to the authors' other papers are incidental and not load-bearing. Therefore the paper receives a 0 circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the quality of the filtered captions and on the assumption that comment topic distributions are meaningful visual supervision; both are empirical premises rather than derived facts. The key thresholds and topic count are hand-set free parameters, and no new theoretical entities are introduced.

free parameters (3)

Informativeness threshold for caption filtering = 20
Comments with score below 20 are discarded. The threshold is described as 'experimentally kept 20' with no sensitivity analysis; it directly determines the composition of AVA-Captions and all downstream results.
LDA topic count K = 200
Used to derive weak labels for the CNN. 'K is experimentally fixed to 200' in Section 4.3; different K would change the label space and likely the captioning results.
LDA vocabulary frequency cutoff = 10% of comments
N-grams appearing in more than 10% of comments are removed from the LDA vocabulary (Section 4.3) to avoid generic topics; this hand-set cutoff affects which topics are discovered and thus the weak labels.

assumptions (4)

domain assumption Comments in AVA are genuine aesthetic feedback tied to the corresponding photographs.
The entire dataset construction relies on user comments from dpchallenge.com describing what makes each photo appealing or weak (Section 1).
domain assumption LDA topic distributions over cleaned comments are meaningful visual aesthetic attributes.
Section 4.3 assumes visual attributes are correlated with comment words and that similar photos are described with similar words; this justifies using topic distributions as CNN labels.
standard math N-gram independence in the informativeness score (Equation 2).
The score is derived under the simplest Markov assumption that all n-grams are independent, stated in Section 3; this is a modeling simplification, not a proven property of comments.
domain assumption Standard captioning metrics and the small human evaluation are valid proxies for aesthetic caption quality.
Section 6.3 uses BLEU, CIDEr, SPICE, and 15-participant ratings as evidence; their validity for aesthetic critique is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Aesthetic Image Captioning From Weakly-Labelled Photographs." pith.science (2026). https://pith.science/paper/S5M6XT4U

@misc{pith2026190811310,
  author       = {Pith},
  title        = {Pith review of: Aesthetic Image Captioning From Weakly-Labelled Photographs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5M6XT4U}},
  note         = {Machine review of arXiv:1908.11310}
}

read the original abstract

Aesthetic image captioning (AIC) refers to the multi-modal task of generating critical textual feedbacks for photographs. While in natural image captioning (NIC), deep models are trained in an end-to-end manner using large curated datasets such as MS-COCO, no such large-scale, clean dataset exists for AIC. Towards this goal, we propose an automatic cleaning strategy to create a benchmarking AIC dataset, by exploiting the images and noisy comments easily available from photography websites. We propose a probabilistic caption-filtering method for cleaning the noisy web-data, and compile a large-scale, clean dataset "AVA-Captions", (230, 000 images with 5 captions per image). Additionally, by exploiting the latent associations between aesthetic attributes, we propose a strategy for training the convolutional neural network (CNN) based visual feature extractor, the first component of the AIC framework. The strategy is weakly supervised and can be effectively used to learn rich aesthetic representations, without requiring expensive ground-truth annotations. We finally show-case a thorough analysis of the proposed contributions using automatic metrics and subjective evaluations.

Figures

Figures reproduced from arXiv: 1908.11310 by the authors.

**Figure 2.** Informativeness of captions. We suggest the readers to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Some topics / labels discovered from AVA-Captions us [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Proposed pipeline trained using the clean and informative comments from the AVA-Captions dataset (See Figure 4b). 6. Experiments The experiments are designed to evaluate the two primary contributions: First, the caption cleaning strategy and second, the weakly-supervi…

**Figure 5.** Figure 5: Diversity: Figures (a) - (c) report diversity of captions following [3]. The x-axes correspond to n-gram positions in a sentence. The y-axes correspond to the number of unique n-grams at each position, for the entire validation set. Figure (d) plots the overall diversi…

**Figure 6.** Figure 6: Subjective evaluation of caption filtering: The matrix compares our scoring strategy and human judgement for distinguishing a good and a bad caption. The rows stand for our output, and the columns represent what humans thought. We observe that the proposed caption fi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 71 canonical work pages

[1]

Dont just assume; look and answer: Over- coming priors for visual question answering

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Anirud- dha Kembhavi. Dont just assume; look and answer: Over- coming priors for visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971–4980, 2018

2018
[2]

Spice: Semantic propositional image cap- tion evaluation

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382–398. Springer, 2016

2016
[3]

Convolutional image captioning

Jyoti Aneja, Aditya Deshpande, and Alexander G Schwing. Convolutional image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5561–5570, 2018

2018
[4]

Deep compositional captioning: Describing novel ob- ject categories without paired training data

Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Dar- rell. Deep compositional captioning: Describing novel ob- ject categories without paired training data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–10, 2016

2016
[5]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425– 2433, 2015

2015
[6]

Automated aesthetic analysis of photographic images

Tunc ¸ Ozan Aydın, Aljoscha Smolic, and Markus Gross. Automated aesthetic analysis of photographic images. IEEE transactions on visualization and computer graphics , 21(1):31–42, 2015

2015
[7]

Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005

2005
[8]

Au- tomatic attribute discovery and characterization from noisy web data

Tamara L Berg, Alexander C Berg, and Jonathan Shih. Au- tomatic attribute discovery and characterization from noisy web data. In European Conference on Computer Vision , pages 663–676. Springer, 2010

work page 2010

Show all 86 references

[9]

Vizwiz: nearly real-time answers to visual questions

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User inte...

2010
[10]

Latent dirichlet allocation

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003

2003
[11]

Aesthetic critiques generation for photos

Kuang-Yu Chang, Kung-Hung Lu, and Chu-Song Chen. Aesthetic critiques generation for photos. In Proceedings of the IEEE International Conference on Computer Vision , pages 3514–3523, 2017

2017
[12]

Neil: Extracting visual knowledge from web data

Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. Neil: Extracting visual knowledge from web data. In Proceedings of the IEEE International Conference on Computer Vision , pages 1409–1416, 2013

2013
[13]

Learning to evaluate image captioning

Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. Learning to evaluate image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5804–5812, 2018

2018
[14]

Study- ing aesthetics in photographic images using a computational approach

Ritendra Datta, Dhiraj Joshi, Jia Li, and James Wang. Study- ing aesthetics in photographic images using a computational approach. Computer Vision–ECCV 2006 , pages 288–301, 2006

2006
[15]

Imagenet: A large-scale hierarchical im- age database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition,
[16]

High level describable attributes for predicting aesthetics and in- terestingness

Sagnik Dhar, Vicente Ordonez, and Tamara L Berg. High level describable attributes for predicting aesthetics and in- terestingness. In Computer Vision and Pattern Recogni- tion (CVPR), 2011 IEEE Conference on , pages 1657–1664. IEEE, 2011

2011
[17]

Learn- ing everything about anything: Webly-supervised visual con- cept learning

Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learn- ing everything about anything: Webly-supervised visual con- cept learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3270– 3277, 2014

2014
[18]

Multi-task self- supervised visual learning

Carl Doersch and Andrew Zisserman. Multi-task self- supervised visual learning. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 2051–2060, 2017

2017
[19]

Long-term recurrent convolutional net- works for visual recognition and description

Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional net- works for visual recognition and description. In Proceed- ings of the IEEE conference on computer vision an...

2015
[20]

From captions to vi- sual concepts and back

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Sri- vastava, Li Deng, Piotr Doll´ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to vi- sual concepts and back. In Proceedings of the IEEE con- ference on computer vision and pattern recog...

2015
[21]

Every picture tells a story: Generating sentences from images

Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Pe- ter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every picture tells a story: Generating sentences from images. In European conference on computer vision , pages 15–29. Springer, 2010

2010
[22]

Im- age style transfer using convolutional neural networks

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Im- age style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016

2016
[23]

Visual turing test for computer vision sys- tems

Donald Geman, Stuart Geman, Neil Hallonquist, and Lau- rent Younes. Visual turing test for computer vision sys- tems. Proceedings of the National Academy of Sciences , page 201422953, 2015

2015
[24]

The data that transformed ai researchand possibly the world, 2017

Dave Gershgorn. The data that transformed ai researchand possibly the world, 2017

2017
[25]

A geometry-sensitive approach for photographic style classiﬁ- cation

Koustav Ghosal, Mukta Prasad, and Aljosa Smolic. A geometry-sensitive approach for photographic style classiﬁ- cation. 2018

2018
[26]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

2016
[27]

Fram- ing image description as a ranking task: Data, models and evaluation metrics

Micah Hodosh, Peter Young, and Julia Hockenmaier. Fram- ing image description as a ranking task: Data, models and evaluation metrics. Journal of Artiﬁcial Intelligence Re- search, 47:853–899, 2013

2013
[28]

Learning hypergraph-regularized attribute predic- tors

Sheng Huang, Mohamed Elhoseiny, Ahmed Elgammal, and Dan Yang. Learning hypergraph-regularized attribute predic- tors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 409–417, 2015

2015
[29]

Learn- ing cross-modality similarity for multinomial data

Yangqing Jia, Mathieu Salzmann, and Trevor Darrell. Learn- ing cross-modality similarity for multinomial data. In Com- puter Vision (ICCV), 2011 IEEE International Conference on, pages 2407–2414. IEEE, 2011

2011
[30]

Aligning where to see and what to tell: image caption with region-based attention and scene factorization

Junqi Jin, Kun Fu, Runpeng Cui, Fei Sha, and Changshui Zhang. Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv preprint arXiv:1506.06272, 2015

2015 arXiv
[31]

Densecap: Fully convolutional localization networks for dense caption- ing

Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense caption- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016

2016
[32]

Aesthet- ics and emotions in images

Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang- Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo. Aesthet- ics and emotions in images. IEEE Signal Processing Maga- zine, 28(5):94–115, 2011

2011
[33]

Speech & language processing

Dan Jurafsky. Speech & language processing. Pearson Edu- cation India, 2000

2000
[34]

Recognizing image style

Sergey Karayev, Aaron Hertzmann, Holger Winnemoeller, Aseem Agarwala, and Trevor Darrell. Recognizing image style. In BMVC 2014, 2014

2014
[35]

Deep visual-semantic align- ments for generating image descriptions

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015

2015
[36]

The design of high- level features for photo quality assessment

Yan Ke, Xiaoou Tang, and Feng Jing. The design of high- level features for photo quality assessment. In Computer Vi- sion and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 1, pages 419–426. IEEE, 2006

2006
[37]

Where does data come from?, 2018

Adam Kell. Where does data come from?, 2018

2018
[38]

neural-storyteller, 2015

Ryan Kiros. neural-storyteller, 2015

2015
[39]

Topic modeling and latent dirichlet allocation (lda) in python, 2018

Susan Li. Topic modeling and latent dirichlet allocation (lda) in python, 2018

2018
[40]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004

2004
[41]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014

2014
[42]

Fully convolutional networks for semantic segmentation

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3431–3440, 2015

2015
[43]

Nltk: The natural language toolkit

Edward Loper and Steven Bird. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Ef- fective Tools and Methodologies for Teaching Natural Lan- guage Processing and Computational Linguistics - Volume 1, ETMTNLP ’02, pages 63–70, Stroudsburg, PA, USA, 2...

2002
[44]

Rapid: Rating pictorial aesthetics using deep learn- ing

Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z Wang. Rapid: Rating pictorial aesthetics using deep learn- ing. In Proceedings of the 22nd ACM international confer- ence on Multimedia, pages 457–466. ACM, 2014

2014
[45]

Deep multi-patch aggregation network for image style, aesthetics, and quality estimation

Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z Wang. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In Proceedings of the IEEE International Conference on Computer Vision , pages 990–998, 2015

2015
[46]

An image captioning codebase in pytorch, 2017

Ruotian Luo. An image captioning codebase in pytorch, 2017

2017
[47]

Photo and video quality eval- uation: Focusing on the subject

Yiwen Luo and Xiaoou Tang. Photo and video quality eval- uation: Focusing on the subject. Computer Vision–ECCV 2008, pages 386–399, 2008

2008
[48]

A-lamp: Adap- tive layout-aware multi-patch deep convolutional neural net- work for photo aesthetic assessment

Shuang Ma, Jing Liu, and Chang Wen Chen. A-lamp: Adap- tive layout-aware multi-patch deep convolutional neural net- work for photo aesthetic assessment. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) , July 2017

2017
[49]

Composition-preserving deep photo aesthetics assessment

Long Mai, Hailin Jin, and Feng Liu. Composition-preserving deep photo aesthetics assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 497–506, 2016

2016
[50]

Ask your neurons: A neural-based approach to answering questions about images

Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE interna- tional conference on computer vision, pages 1–9, 2015

2015
[51]

Bapi, and Bipin Indurkhya

Gautam Malu, Raju S. Bapi, and Bipin Indurkhya. Learning photography aesthetics with deep cnns, 2017

2017
[52]

Generation and comprehension of unambiguous object descriptions

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016

2016
[53]

Learning like a child: Fast novel visual concept learning from sentence descriptions of images

Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L Yuille. Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In Proceedings of the IEEE International Conference on Com- puter Vision, pages 2533–2541, 2015

2015
[54]

Deep captioning with multimodal recurrent neural networks (m-rnn)

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 , 2014

2014 arXiv
[55]

Dis- covering beautiful attributes for aesthetic image analysis

Luca Marchesotti, Naila Murray, and Florent Perronnin. Dis- covering beautiful attributes for aesthetic image analysis. International journal of computer vision , 113(3):246–266, 2015

2015
[56]

Semstyle: Learning to generate stylised image captions using unaligned text

Alexander Mathews, Lexing Xie, and Xuming He. Semstyle: Learning to generate stylised image captions using unaligned text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8591–8600, 2018

2018
[57]

From red wine to red tomato: Composition with context

Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1792–1801, 2017

2017
[58]

Ava: A large-scale database for aesthetic visual analysis

Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2408–2415. IEEE, 2012

2012
[59]

Attributes as op- erators: factorizing unseen attribute-object compositions

Tushar Nagarajan and Kristen Grauman. Attributes as op- erators: factorizing unseen attribute-object compositions. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 169–185, 2018

2018
[60]

Towards category-based aesthetic models of photographs

Pere Obrador, Michele A Saad, Poonam Suryanarayan, and Nuria Oliver. Towards category-based aesthetic models of photographs. In International Conference on Multimedia Modeling, pages 63–76. Springer, 2012

2012
[61]

Im2text: Describing images using 1 million captioned pho- tographs

Vicente Ordonez, Girish Kulkarni, and Tamara L Berg. Im2text: Describing images using 1 million captioned pho- tographs. In Advances in neural information processing sys- tems, pages 1143–1151, 2011

2011
[62]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on as- sociation for computational linguistics, pages 311–318. As- sociation for Computational Linguistics, 2002

2002
[63]

Relative attributes

Devi Parikh and Kristen Grauman. Relative attributes. In 2011 International Conference on Computer Vision , pages 503–510. IEEE, 2011

2011
[64]

Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

2015
[65]

Fast collapsed gibbs sampling for latent dirichlet allocation

Ian Porteous, David Newman, Alexander Ihler, Arthur Asun- cion, Padhraic Smyth, and Max Welling. Fast collapsed gibbs sampling for latent dirichlet allocation. In Proceed- ings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 569–5...

2008
[66]

Using tf-idf to determine word relevance in document queries

Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the ﬁrst instructional conference on machine learning , volume 242, pages 133– 142, 2003

2003
[67]

Deep tone mapping operator for high dynamic range images

Aakanksha Rana, Praveer Singh, Giuseppe Valenzise, Fred- eric Dufaux, Nikos Komodakis, and Aljosa Smolic. Deep tone mapping operator for high dynamic range images. Transaction of Image Processing, 2019

2019
[68]

Fea- ture learning for the image retrieval task

Aakanksha Rana, Joaquin Zepeda, and Patrick Perez. Fea- ture learning for the image retrieval task. In Asian Confer- ence on Computer Vision (ACCV), pages 152–165. Springer, 2014

2014
[69]

Cross-domain self- supervised multi-task feature learning using synthetic im- agery

Zhongzheng Ren and Yong Jae Lee. Cross-domain self- supervised multi-task feature learning using synthetic im- agery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 762–771, 2018

2018
[70]

The cross-entropy method: a uniﬁed approach to combinatorial optimization, Monte-Carlo simulation and machine learning

Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: a uniﬁed approach to combinatorial optimization, Monte-Carlo simulation and machine learning . Springer Science & Business Media, 2013

2013
[71]

Recognition us- ing visual phrases

Mohammad Amin Sadeghi and Ali Farhadi. Recognition us- ing visual phrases. In CVPR 2011, pages 1745–1752. IEEE, 2011

2011
[72]

Leveraging user comments for aesthetic aware image search reranking

Jose San Pedro, Tom Yeh, and Nuria Oliver. Leveraging user comments for aesthetic aware image search reranking. In Proceedings of the 21st international conference on World Wide Web, pages 439–448. ACM, 2012

2012
[73]

Neural algebra of classiﬁers

Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Neural algebra of classiﬁers. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 729–737. IEEE, 2018

2018
[74]

Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

2018
[75]

Grounded compositional se- mantics for ﬁnding and describing images with sentences

Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional se- mantics for ﬁnding and describing images with sentences. Transactions of the Association of Computational Linguis- tics, 2(1):207–218, 2014

2014
[76]

Automatic con- cept discovery from parallel text and visual corpora

Chen Sun, Chuang Gan, and Ram Nevatia. Automatic con- cept discovery from parallel text and visual corpora. In Pro- ceedings of the IEEE international conference on computer vision, pages 2596–2604, 2015

2015
[77]

Joint video and text parsing for under- standing events and answering queries

Kewei Tu, Meng Meng, Mun Wai Lee, Tae Eun Choe, and Song-Chun Zhu. Joint video and text parsing for under- standing events and answering queries. IEEE MultiMedia, 21(2):42–70, 2014

2014
[78]

Cider: Consensus-based image description evalua- tion

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

2015
[79]

Au- tomatic attribute discovery with neural activations

Sirion Vittayakorn, Takayuki Umeda, Kazuhiko Murasaki, Kyoko Sudo, Takayuki Okatani, and Kota Yamaguchi. Au- tomatic attribute discovery with neural activations. In Eu- ropean Conference on Computer Vision , pages 252–268. Springer, 2016

2016
[80]

Walk and learn: Facial attribute representation learning from ego- centric video and contextual data

Jing Wang, Yu Cheng, and Rogerio Schmidt Feris. Walk and learn: Facial attribute representation learning from ego- centric video and contextual data. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2295–2304, 2016

2016
[81]

Show, attend and tell: Neural image caption gen- eration with visual attention

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In International conference on machine learning, pages 2048–2057, 2015

2015
[82]

Learning to describe e- commerce images from noisy online data

Takuya Yashima, Naoaki Okazaki, Kentaro Inui, Kota Ya- maguchi, and Takayuki Okatani. Learning to describe e- commerce images from noisy online data. In Asian Con- ference on Computer Vision, pages 85–100. Springer, 2016

2016
[83]

Color- net - estimating colorfulness in natural images

Emin Zerman, Aakanksha Rana, and Aljosa Smolic. Color- net - estimating colorfulness in natural images. In The Inter- national Conference on Image Processing (ICIP), 2019

2019
[84]

Visual translation embedding network for visual relation detection

Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat- Seng Chua. Visual translation embedding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5532–5540, 2017

2017
[85]

Panda: Pose aligned net- works for deep attribute modeling

Ning Zhang, Manohar Paluri, Marc’Aurelio Ranzato, Trevor Darrell, and Lubomir Bourdev. Panda: Pose aligned net- works for deep attribute modeling. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1637–1644, 2014

2014
[2009]

IEEE Conference on , pages 248–255

CVPR 2009. IEEE Conference on , pages 248–255. IEEE, 2009

2009

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Dont just assume; look and answer: Over- coming priors for visual question answering

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Anirud- dha Kembhavi. Dont just assume; look and answer: Over- coming priors for visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971–4980, 2018

2018

[2] [2]

Spice: Semantic propositional image cap- tion evaluation

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382–398. Springer, 2016

2016

[3] [3]

Convolutional image captioning

Jyoti Aneja, Aditya Deshpande, and Alexander G Schwing. Convolutional image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5561–5570, 2018

2018

[4] [4]

Deep compositional captioning: Describing novel ob- ject categories without paired training data

Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Dar- rell. Deep compositional captioning: Describing novel ob- ject categories without paired training data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–10, 2016

2016

[5] [5]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425– 2433, 2015

2015

[6] [6]

Automated aesthetic analysis of photographic images

Tunc ¸ Ozan Aydın, Aljoscha Smolic, and Markus Gross. Automated aesthetic analysis of photographic images. IEEE transactions on visualization and computer graphics , 21(1):31–42, 2015

2015

[7] [7]

Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005

2005

[8] [8]

Au- tomatic attribute discovery and characterization from noisy web data

Tamara L Berg, Alexander C Berg, and Jonathan Shih. Au- tomatic attribute discovery and characterization from noisy web data. In European Conference on Computer Vision , pages 663–676. Springer, 2010

work page 2010