Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Towards Unsupervised Image Captioning with Shared Multimodal Embeddings

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that unsupervised image captioning is achievable by projecting images and text into a shared, visually structured embedding space, and reports state-of-the-art results on the unpaired COCO benchmark.

desk verdict Solid unpaired captioning contribution: the SOTA claim is plausible, the ablation is the best part, and the main concerns are a WGAN typo, missing error bars, and an under-analyzed min-of-K loss. read the letter →

arxiv 1908.09317 v1 pith:XUTFLNM2 submitted 2019-08-25 cs.CV

Iro Laina , Christian Rupprecht , Nassir Navab This is my paper · ORCID
classification cs.CV
keywords unsupervisedimagecaptioningsharedmultimodalembeddingsvisualconceptsdomainalignmentadversarialtrainingrobustlosssentenceembeddingweaksupervision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that image captioning can be done without any image-caption pairs: a language model trained only on sentences provides a shared embedding space, and a translator maps image features into that space so the same decoder can caption the image. The key move is to structure the sentence embedding space by visual concepts (words with visual grounding and their co-occurrence) before aligning the two domains. Alignment is learned from a noisy bipartite matching induced by object-detector labels, using a robust loss that pulls a translated image embedding toward the nearest of several sampled sentence embeddings, plus a concept-conditioned adversarial loss. On the unpaired COCO benchmark the method reports BLEU-4 19.3, METEOR 20.2, ROUGE 45.0, CIDEr 61.8 and SPICE 12.9, which the paper presents as a new state of the art and nearly matching early supervised captioners. A sympathetic reader would care because it suggests captions can be generated from the vast supply of unpaired text and weakly labeled images rather than expensive paired annotations.

What carries the argument

The load-bearing mechanism is the visually structured shared embedding space together with the two alignment losses. The language model's triplet loss (Eq. 4) pushes sentence embeddings so that sentences sharing at least two visual concepts are closer than sentences sharing none, creating a manifold ordered by visual semantics rather than by surface grammar. The robust alignment loss (Eq. 10) selects the closest of $K$ sampled sentence embeddings as the training target for a translated image feature, which the paper argues avoids the mode-averaging behavior of plain $\ell^2$ alignment; the conditional adversarial loss (Eq. 11) then forces the translated feature to encode the image's detected concepts. The same decoder used for sentences decodes the translated features, and image and sentence embeddings end up mixed together in the same clusters.

What would settle it

Train the model with $K=1$ instead of $K=10$ in the robust alignment loss, and with $K=100$, on the same unpaired COCO split; if captioning quality does not improve with $K$, or if removing all sampled sentences that share no visual concepts with the image leaves performance unchanged, then the min-over-samples assumption is not carrying the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that a shared latent space, organized around visual concepts, makes image and language features indistinguishable, and that a decoder trained for language can therefore decode translated image features into fluent captions. The paper argues that a sequence-to-sequence language model trained with a triplet loss over visual-concept overlap creates an embedding in which sentences about the same visual content are close; then a multilayer perceptron translates CNN image features into this space. The translation is learned with a robust alignment loss, $\min_{\varphi_j \sim p(s_j | I_i)} \|h(\psi_i) - \varphi_j\|_2^2$, which samples candidate sentence embeddings from a noisy assignment distribution and keeps the closest one, and with a WGAN-GP discriminator (a Wasserstein GAN with gradient penalty) conditioned on the image's detected concepts. The paper reports that these components each improve the unpaired COCO result, and that the full model outperforms the previous unsupervised method [18] on all reported metrics while approaching the level of early supervised models.

Load-bearing premise

The load-bearing premise is that a sentence sampled (and then minimized over) from a list built by matching detected objects to words is a valid description of the image; if the closest sampled sentence is often semantically unrelated, the alignment has no reliable teacher.

Editorial extensions

If this is right

  • Because the language model can be pretrained on any text corpus, caption generation can draw on external text sources, including stylized ones such as questions from VQA, without any paired image-caption data (the paper demonstrates this by generating questions from VQA-v2 sentences).
  • The captioner can name concepts that are not in the detector's label set, such as beach, tracks, or tower, by exploiting co-occurrence between detected concepts and words in the language corpus.
  • The unpaired COCO results (BLEU-4 19.3, CIDEr 61.8) imply that a captioner trained without correspondences can approach the level of early supervised captioning models, making large-scale captioning from weakly labeled images and text corpora plausible.
  • Each component of the model contributes: removing the triplet-structured embedding, replacing the robust loss with plain $\ell^2$ alignment, or removing the adversarial term each lowers scores, so the full combination is what carries the reported state-of-the-art result.
  • The same recipe transfers across image and language sources (COCO images with Conceptual Captions, Flickr30k images with COCO captions), so the method is not tied to one particular dataset distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the min-over-$K$ estimator's bias could be hiding a shortcut: if the nearest sampled sentence is usually a generic caption shared by many images, the robust loss may reward blandness rather than semantic correctness; a direct test would be measuring performance as $K$ varies or as the nearest-sentence overlap with the image's ground-truth concepts v
  • A testable extension would be to apply the same shared-space recipe to other modality pairs, such as video frames and narrations, since the method never requires paired exemplars; success there would suggest the concept-structured alignment, not the image-caption task, is the general principle.
  • The model's ability to generate questions when trained on VQA sentences suggests the same architecture could control output style by choosing the text corpus, which the paper notes but does not develop into a systematic style-transfer evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses unsupervised image captioning, where images and sentences come from disjoint sources and no image-caption pairs are used. The method first trains a sequence-to-sequence language model on a text corpus with a concept-based triplet loss that structures the embedding space by visual concepts. It then learns a translator h from image features to that embedding space, trained with a robust alignment loss (the L2 distance to the closest of K sentence embeddings sampled from a noisy concept-overlap assignment graph), a concept-conditioned adversarial loss in feature space, and an MLE loss for decoding. Experiments on unpaired COCO captioning and on cross-domain settings (Flickr30k images with COCO captions; COCO images with Conceptual Captions) report state-of-the-art unpaired captioning results and ablations showing each loss term contributes. The paper also shows qualitative examples and a t-SNE visualization of the joint embedding.

Significance. If the reported results hold, the paper is a meaningful step in unpaired image captioning: it improves consistently over the only prior unpaired-captioning baseline and approaches early supervised captioners. The problem formulation is clear, the idea of structuring the shared embedding space with visual concepts is sensible, and the experiments include both ablations and cross-domain evaluations. The main weakness is that the largest ablation gain is attributed to the robust alignment loss, whose min-over-K behavior is not analyzed for semantic correctness, and the reported state-of-the-art margin is small without significance tests or variance estimates. These issues are addressable and do not invalidate the approach, but they need to be resolved before the central claim can be fully accepted.

major comments (3)
  1. [3.3 (Eq. 10), Table 1] The robust alignment loss in Eq. (10) uses the minimum over K sampled sentence embeddings from the noisy assignment distribution p(s_j|I_i). This min is a downward-biased estimate of the distance to the true matching sentence, and because the assignment distribution is built from detector labels (many images share generic concepts such as 'person'), the selected candidate can be a generic or unrelated caption. The paper provides no quantitative check that the chosen phi_j is semantically relevant to the image: the t-SNE in Fig. 4 is qualitative, and the oracle row of Table 1 shows that even the highest-overlap ground-truth candidate reaches only BLEU-4 16.0, below the robust model's 18.8. Since Table 1 attributes the largest ablation gain to this loss (BLEU-4 16.6 to 18.8; CIDEr 53.8 to 62.4), and Section 5 reports only about 20% unique and 16% novel generated captions, the central claim that the alignment is semantically meaningful is not fully supported. Please add a diagnostic, e.g., concept recall of the selected phi_j or a comparison of min-selection with random selection, and an analysis of the bias of the min estimator.
  2. [Table 2] The reported improvements over Feng et al. are modest in some metrics (BLEU-4 19.3 vs 18.6, ROUGE 45.0 vs 43.1) and no variance or significance tests are reported. Because the training procedure involves stochastic sampling of K candidate sentences, the differences could be within run-to-run noise. Please report standard deviations over multiple training runs or bootstrap significance tests (e.g., paired bootstrap over test captions) to support the 'new state of the art' claim.
  3. [Section 5] The paper states that approximately 20% of generated captions are unique and 16% are novel (not found in the training set). For an unsupervised method this level of mode collapse is a significant concern, especially because the min-over-K objective may actively encourage selection of generic sentence embeddings. The comparison with the supervised baseline [60] does not fully address this, since the baseline already has access to paired data. Please discuss the relationship between the robust alignment loss and caption diversity, and report diversity-aware metrics (e.g., number of unique n-grams, self-CIDEr) for the proposed model and the baselines.
minor comments (5)
  1. [3.3 (WGAN-GP paragraph)] The sentence 'we use a WGAN-GP formulation to train the discriminator D to maximize its output for fake examples and minimize it for real' appears to invert the standard WGAN-GP objective. As written, the subsequent generator loss Ladv = -D(h(psi_i), V_i) would move h toward features that the discriminator scores as fake. Please correct the direction to 'maximize for real and minimize for fake' or clarify the intended convention.
  2. [Table 1 caption] The table caption says 'COCO test set [27]' but reference [27] is the Karpathy and Li paper that defines the split, not the COCO dataset paper [38]. Please cite the dataset and the split precisely.
  3. [4.1, Oracle] The oracle row selects the highest-overlap ground-truth caption per image and reports the best over 100 random tie-breaks; this is a different selection procedure from the min-over-K used by the model. Please state explicitly whether the oracle is intended to estimate the ceiling of the assignment graph, and if so, note that the comparison is indirect.
  4. [Implementation details] The text says the translator h is a single-layer MLP of size 512, but it is unclear whether 512 is the hidden size or the output size, given that psi is in R^2048 and phi is in R^256. Please clarify the architecture dimensions.
  5. [Table 3] The WMD metric is reported in Table 3 but not defined in the main text. Please add a one-sentence definition or a reference to the metric's description.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core captioning result is evaluated on held-out COCO test captions and does not reduce by construction to the weak training assignments.

full rationale

The paper's derivation chain is self-contained with respect to the target claim of unsupervised captioning on COCO. The language model is trained on text alone (Eq. 7), the image-to-embedding translator h is trained with Eq. (10) and Eq. (11) using weak concept-overlap assignments p(s_j|I_i) built from detector labels, and the final captions are decoded from h(psi_i) and scored against held-out COCO test captions using standard metrics (Tables 1-3). No test-set caption is used as a training target, and no fitted parameter is renamed as a prediction. The robust alignment loss (Eq. 10) is the training objective, not the evaluation quantity; the min-over-K candidate is a training target drawn from the unpaired training corpus, and the oracle row in Table 1 explicitly shows that even the best concept-overlap candidate among ground-truth captions scores 16.0 BLEU-4, below the full model's 19.3, so the reported test result is not simply a retrieval of the nearest assigned caption. There are no load-bearing self-citations, uniqueness theorems, or ansatz-via-citation steps: the comparison to Feng et al. [18] is an external baseline, and the detector supervision is acknowledged as a weak training signal rather than presented as the derived result. The paper's own limitation statement in Section 5 about caption collapse and low novelty is a diversity concern, not a circularity concern. Any concern about the bias of the min estimator is a statistical validity issue, not circularity: it does not make the evaluation equivalent to the method's inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on no new physical or mathematical entities. It rests on a chain of modeling assumptions: domain overlap, detector-derived concepts, the triplet threshold, the min-of-K alignment target, and the adversarial objective. These are all disclosed in the text, but several are not independently validated beyond the ablation study.

free parameters (6)
  • Triplet margin m = not reported
    Appears in Eq. (4); chosen by hand and not specified in the experimental section.
  • Triplet loss weight lambda_t = 0.1
    Used in Eq. (7); set in Section 4, Training details.
  • MLE loss weight lambda_CE = 1
    Eq. (12) weight; set to 1 in Section 4, Training details.
  • Robust alignment loss weight lambda_R = 1
    Eq. (12) weight; set to 1 in Section 4, Training details.
  • Adversarial loss weight lambda_adv = 0.1
    Eq. (12) weight; set to 0.1 in Section 4, Training details.
  • Candidate sentence count K = 10
    K=10 sentences sampled per image in the robust alignment loss; reported in Section 4.1 ablation.
assumptions (5)
  • domain assumption There exists a non-empty universal set of visual concepts Omega = V intersect W shared by the image and language domains.
    Section 3 states this explicitly: 'we assume a universal set of concepts Omega = V intersect W that language and images have in common.' Without overlap, the weak assignment graph G is empty.
  • domain assumption A pretrained object detector's class labels, expanded with hyponyms, provide a usable approximation of the visual concepts in an image.
    Section 3.2 uses OpenImages detector labels and hyponym queries to build the bipartite graph; the paper notes the detector is fixed and not fine-tuned.
  • ad hoc to paper Sentences sharing at least two visual concepts are semantically close, and sentences sharing none are far; pairs with exactly one shared concept are ignored.
    Eqs. (5)-(6) define positive and negative pairs for the triplet loss; the one-overlap exclusion is a design choice to avoid human-centered bias.
  • ad hoc to paper The closest of K sampled candidate sentence embeddings is a valid target for the image translation.
    Eq. (10) replaces the conditional average with a min over K samples; the paper provides no proof that this target avoids semantic mismatches.
  • standard math WGAN-GP yields a stable adversarial update for feature-space alignment.
    The method relies on Gulrajani et al. [21]; the paper's own description of the update direction is internally inconsistent, as noted in the red flags.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Unsupervised Image Captioning with Shared Multimodal Embeddings." pith.science (2026). https://pith.science/paper/XUTFLNM2

@misc{pith2026190809317,
  author       = {Pith},
  title        = {Pith review of: Towards Unsupervised Image Captioning with Shared Multimodal Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUTFLNM2}},
  note         = {Machine review of arXiv:1908.09317}
}
read the original abstract

Understanding images without explicit supervision has become an important problem in computer vision. In this paper, we address image captioning by generating language descriptions of scenes without learning from annotated pairs of images and their captions. The core component of our approach is a shared latent space that is structured by visual concepts. In this space, the two modalities should be indistinguishable. A language model is first trained to encode sentences into semantically structured embeddings. Image features that are translated into this embedding space can be decoded into descriptions through the same language model, similarly to sentence embeddings. This translation is learned from weakly paired images and text using a loss robust to noisy assignments and a conditional adversarial component. Our approach allows to exploit large text corpora outside the annotated distributions of image/caption data. Our experiments show that the proposed domain alignment learns a semantically meaningful representation which outperforms previous work.

Figures

Figures reproduced from arXiv: 1908.09317 by the authors.

Figure 1
Figure 1. Method overview. Our model learns a joint embedding space of language and image features that is structured by visual concepts and their co-occurrence. Images and text come from dis￾joint sources. During inference, model embeds images into the shared space from which a caption can be decoded. limited number of object categories and are focused on per￾formance under imperfect evaluation metrics. Thus, meth￾ods develo… view at source ↗
Figure 2
Figure 2. Unsupervised image captioning architecture. We first learn a language model with a triplet loss formulation that structures the embedding φ using visual concepts from the sentences. We then learn a mapping from images to the embedding space using a robust alignment scheme and adversarial training in feature space. external emphasis on structure is important, since uncon￾strained language models are more likely to gr… view at source ↗
Figure 3
Figure 3. Qualitative Results. We show caption predictions on images from the COCO dataset. COCO and OID are results from our unpaired model trained with weak pairs coming from a detector trained on the respective dataset. GCC and VQA refer to the unsupervised model trained on COCO images using the Conceptual Captions and VQA-v2 datasets respectively. that both the COCO model and the OID model capture the image contents well,… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: t-SNE Embedding. We show a projection of the learned joint embedding of our model and zoom into a cluster to visualize that sentences from the text corpus (denoted by [L]) lie in visual￾semantic groups together with image embeddings [I]. Colors are generated by groups …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 60 canonical work pages

  1. [60]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3156–3164, 2015. 2, 5, 6, 8

    work page 2015

  2. [1]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382–398. Springer, 2016. 5

    work page 2016

  3. [2]

    Guided open vocabulary image captioning with constrained beam search

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. In EMNLP, 2017. 2

    work page 2017

  4. [3]

    Partially-supervised image captioning

    Peter Anderson, Stephen Gould, and Mark Johnson. Partially-supervised image captioning. In Advances in Neural Information Processing Systems , pages 1879–1890,

    work page

  5. [4]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018. 2

    work page 2018

  6. [5]

    Women also snowboard: Over- coming bias in captioning models

    Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Over- coming bias in captioning models. In Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), pages 771– 787, 2018. 2

    work page 2018

  7. [6]

    Deep compositional captioning: Describing novel ob- ject categories without paired training data

    Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Dar- rell. Deep compositional captioning: Describing novel ob- ject categories without paired training data. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2016. 2

    work page 2016

  8. [7]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015. 7

    work page 2015

Show all 70 references
  1. [8]

    Adversarial text generation via feature-mover’s distance

    Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. Adversarial text generation via feature-mover’s distance. InAdvances in Neural Information Processing Systems, pages 4671–4682, 2018. 5

    2018

  2. [9]

    Show, adapt and tell: Adversarial training of cross-domain image cap- tioner

    Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun. Show, adapt and tell: Adversarial training of cross-domain image cap- tioner. In Proceedings of the IEEE International Conference on Computer Vision, pages 521–530, 2017. 1, 2

    2017

  3. [10]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. 5

    2014 arXiv

  4. [11]

    To- wards diverse and natural image descriptions via a condi- tional GAN

    Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. To- wards diverse and natural image descriptions via a condi- tional GAN. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 2970–2979, 2017. 2

    2017

  5. [12]

    Visual dialog

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos´e MF Moura, Devi Parikh, and Dhruv Ba- tra. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 326–335,

  6. [13]

    Meteor universal: Lan- guage specific translation evaluation for any target language

    Michael Denkowski and Alon Lavie. Meteor universal: Lan- guage specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014. 5

    2014

  7. [14]

    Long-term recurrent convolutional net- works for visual recognition and description

    Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional net- works for visual recognition and description. In Proceed- ings of the IEEE conference on computer vision an...

    2015

  8. [15]

    Ad- versarial feature learning

    Jeff Donahue, Philipp Kr ¨ahenb¨uhl, and Trevor Darrell. Ad- versarial feature learning. arXiv preprint arXiv:1605.09782,

    arXiv

  9. [16]

    VSE++: Improving visual-semantic embeddings with hard negatives

    Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612, 2017. 3

    2017 arXiv

  10. [17]

    From captions to vi- sual concepts and back

    Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Sri- vastava, Li Deng, Piotr Doll´ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to vi- sual concepts and back. In Proceedings of the IEEE con- ference on computer vision and pattern recog...

    2015

  11. [18]

    Unsupervised image captioning

    Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. arXiv preprint arXiv:1811.10787, 2018. 2, 5, 6, 8

    2018 arXiv

  12. [19]

    StyleNet: Generating attractive visual captions with styles

    Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. StyleNet: Generating attractive visual captions with styles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3137–3146, 2017. 2

    2017

  13. [20]

    Un- paired image captioning by language pivoting

    Jiuxiang Gu, Shafiq Joty, Jianfei Cai, and Gang Wang. Un- paired image captioning by language pivoting. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 503–519, 2018. 2, 8

    2018

  14. [21]

    Improved training of wasserstein gans

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Pro- cessing Systems, pages 5767–5777, 2017. 5

    2017

  15. [22]

    MSCap: Multi-style image captioning with un- paired stylized text

    Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Han- qing Lu. MSCap: Multi-style image captioning with un- paired stylized text. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 1, 2

    2019

  16. [23]

    VizWiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. VizWiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3608– ...

    2018

  17. [24]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  18. [25]

    Speed/accuracy trade-offs for modern convolutional object detectors

    Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wo- jna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. InPro- ceedings of the IEEE conference on computer v...

    2017

  19. [26]

    DenseCap: Fully convolutional localization networks for dense caption- ing

    Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully convolutional localization networks for dense caption- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016. 2

    2016

  20. [27]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015. 2, 6, 8

    2015

  21. [28]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  22. [29]

    Unifying visual-semantic embeddings with multimodal neu- ral language models

    Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neu- ral language models. arXiv preprint arXiv:1411.2539, 2014. 3

    2014 arXiv

  23. [30]

    OpenImages: A public dataset for large-scale multi-label and multi-class image classification

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai...

  24. [31]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Compute...

    2017

  25. [32]

    From word embeddings to document distances

    Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. From word embeddings to document distances. In International Conference on Machine Learning, pages 957– 966, 2015. 5

    2015

  26. [33]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detect...

    2018 arXiv

  27. [34]

    Unsupervised machine translation using monolingual corpora only

    Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. InInternational Conference on Learning Representations (ICLR), 2018. 2, 3

    2018

  28. [35]

    Phrase-based & neural unsupervised machine translation

    Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic De- noyer, and Marc’Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 2018. 3

    2018

  29. [36]

    Generating diverse and accurate visual cap- tions by comparative adversarial learning

    Dianqi Li, Xiaodong He, Qiuyuan Huang, Ming-Ting Sun, and Lei Zhang. Generating diverse and accurate visual cap- tions by comparative adversarial learning. arXiv preprint arXiv:1804.00861, 2018. 2

    2018 arXiv

  30. [37]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004. 5

    2004

  31. [38]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014. 1, 5

    2014

  32. [39]

    Teaching machines to describe images via natural language feedback

    Huan Ling and Sanja Fidler. Teaching machines to describe images via natural language feedback. In Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, pages 5075–5085. Curran Associates Inc.,

  33. [40]

    Improved image captioning via policy gra- dient optimization of spider

    Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gra- dient optimization of spider. In Proceedings of the IEEE in- ternational conference on computer vision , pages 873–881,

  34. [41]

    Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning

    Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 375–383, 2017. 2

    2017

  35. [42]

    Neural baby talk

    Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7219– 7228, 2018. 2

    2018

  36. [43]

    Visualizing data using t-SNE

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research , 9(Nov):2579–2605, 2008. 7

    2008

  37. [44]

    The stan- ford CoreNLP natural language processing toolkit

    Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The stan- ford CoreNLP natural language processing toolkit. In Pro- ceedings of 52nd annual meeting of the association for com- putational linguistics: system demonstrations , pa...

  38. [45]

    Learning like a child: Fast novel visual concept learning from sentence descriptions of images

    Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L Yuille. Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In Proceedings of the IEEE international conference on com- puter vision, pages 2533–2541, 2015. 2

    2015

  39. [46]

    Sem- Style: Learning to generate stylised image captions using unaligned text

    Alexander Mathews, Lexing Xie, and Xuming He. Sem- Style: Learning to generate stylised image captions using unaligned text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8591– 8600, 2018. 2

    2018

  40. [47]

    Jointly modeling embedding and translation to bridge video and language

    Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4594– 4602, 2016. 3

    2016

  41. [48]

    GloVe: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Man- ning. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532–1543,

    2014

  42. [49]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  43. [50]

    Self-critical sequence training for image captioning

    Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7008– 7024, 2017. 2

    2017

  44. [51]

    ImageNet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015. 2

    2015

  45. [52]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  46. [53]

    Speaking the same language: Matching machine to human captions by adversarial train- ing

    Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial train- ing. In Proceedings of the IEEE International Conference on Computer Vision, pages 4135–4144, 2017. 2

    2017

  47. [54]

    Deforming autoencoders: Unsupervised disentangling of shape and ap- pearance

    Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and ap- pearance. In The European Conference on Computer Vision (ECCV), September 2018. 2

    2018

  48. [55]

    Engaging image captioning via per- sonality

    Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bor- des, and Jason Weston. Engaging image captioning via per- sonality. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition , pages 12516–12526,

  49. [56]

    Towards text generation with adversarially learned neural outlines

    Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C Courville, and Chris Pal. Towards text generation with adversarially learned neural outlines. In Advances in Neural Information Processing Sys- tems, pages 7562–7574, 2018. 5

    2018

  50. [57]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. InAdvances in neu- ral information processing systems, pages 3104–3112, 2014. 3

    2014

  51. [58]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 5

    2015

  52. [59]

    Captioning images with diverse objects

    Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko. Captioning images with diverse objects. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5753–5761, 2017. 2

    2017

  53. [61]

    Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space

    Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In Advances in Neural Information Processing Systems , pages 5756–5766, 2017. 2

    2017

  54. [62]

    Automatic alt-text: Computer-generated image de- scriptions for blind users on a social network service

    Shaomei Wu, Jeffrey Wieland, Omid Farivar, and Julie Schiller. Automatic alt-text: Computer-generated image de- scriptions for blind users on a social network service. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing , pages...

    2017

  55. [63]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In International conference on machine learning, pages 2048–2057, 2015. 2

    2015

  56. [64]

    Review networks for caption gen- eration

    Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, and Ruslan R Salakhutdinov. Review networks for caption gen- eration. In Advances in Neural Information Processing Sys- tems, pages 2361–2369, 2016. 2

    2016

  57. [65]

    Incorpo- rating copying mechanism in image captioning for learn- ing novel objects

    Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Incorpo- rating copying mechanism in image captioning for learn- ing novel objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6580– 6588, 2017. 2

    2017

  58. [66]

    Explor- ing visual relationship for image captioning

    Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Explor- ing visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 684–699, 2018. 2

    2018

  59. [67]

    Boosting image captioning with attributes

    Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting image captioning with attributes. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 4894–4902, 2017. 2

    2017

  60. [68]

    Image captioning with semantic attention

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016. 2

    2016

  61. [69]

    Dual learning for cross-domain image captioning

    Wei Zhao, Wei Xu, Min Yang, Jianbo Ye, Zhou Zhao, Yabing Feng, and Yu Qiao. Dual learning for cross-domain image captioning. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 29–38. ACM, 2017. 1, 2

    2017

  62. [70]

    Unpaired image-to-image translation using cycle- consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2223– 2232, 2017. 2

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.