Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Scene-based Factored Attention for Image Captioning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scene concepts embedded in factored attention weights improve image captioning, raising CIDEr to 1.268 on the MS COCO benchmark.

desk verdict Plausible scene-conditioned attention variant, but gamma-on-test and a weak baseline make the reported gains unreliable; the ablation is the only solid evidence. read the letter →

arxiv 1908.02632 v3 pith:BHB3PKEF submitted 2019-08-07 cs.CV

Chen Shen , Rongrong Ji , Fuhai Chen , Xiaoshuai Sun , Xiangming Li This is my paper · ORCID
classification cs.CV
keywords imagecaptioningsceneconceptsfactoredattentiontensorfactorizationvisualLSTMMSCOCOrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that image captioning models ignore scene concepts—the global context of an image, such as indoor or outdoor, kitchen or beach—and that attending with those concepts produces more accurate captions. It proposes a scene-based factored attention module: the scene vector from a Places-pretrained CNN is turned into a diagonal matrix that factorizes the attention weight matrix, so the LSTM hidden state is conditioned on scene context before it attends regional features and object-centered visual concepts. On the MS COCO Karpathy split, the method improves over the Up-Down baseline and reports CIDEr 1.268 under CIDEr-optimized training, above prior state-of-the-art results.

What carries the argument

The central object is the factored attention weight matrix $W_h = U_h S V_h$, where $S = \operatorname{diag}(v_{\text{scene}})$ is the diagonal matrix built from the scene concept vector extracted by a CNN pretrained on the Places dataset. Factoring the usual attention transform this way lets the same learned matrices $U_h$ and $V_h$ be shared across all images while the scene vector rescales the intermediate space, so scene identity modulates which visual features are attended at each word-generation step. A second LSTM then consumes the weighted regional features and object-centered visual concepts to emit the next word.

What would settle it

Run the same two-LSTM captioning model with identical scene features but replace the factored diagonal conditioning $W_h = U_h \operatorname{diag}(v_{\text{scene}}) V_h$ with an additive or gated conditioning; if the CIDEr gains vanish or reverse, the claimed mechanism is not the cause. Also measure the Places classifier's top-1 accuracy on MS COCO validation images; if the scene predictions are mostly wrong, the scene vector is too noisy to carry the claimed semantics.

Watch

Extended reading notes

Core claim

The central claim is that scene concepts carry higher-level visual semantics that should guide attention: the same object can be described differently depending on whether the scene is a beach or a bedroom. Concretely, the paper embeds the scene concept vector $v_{\text{scene}}$ into the attention computation by writing the hidden-state transformation as $W_h = U_h \operatorname{diag}(v_{\text{scene}}) V_h$, so the attention scores over regional features and object-centered visual concepts are computed under scene context. This factored conditioning, combined with a two-LSTM generator, yields captions that mention scene-appropriate keywords and attend to more relevant regions. The authors report consistent gains over their baseline across BLEU, METEOR, ROUGE, CIDEr, and SPICE, and state-of-the-art or near-state-of-the-art results on MS COCO.

Load-bearing premise

The improvements depend on the scene vector extracted by a Places-pretrained CNN being informative for MS COCO images, and on the diagonal embedding of that vector into the attention weights being the actual cause of the gains rather than added model capacity or other training differences.

Editorial extensions

If this is right

  • If scene concepts genuinely guide attention, captioning models that add a scene-conditioning pathway should improve on datasets where scene context is predictive of caption wording.
  • The factored form $W_h = U_h \operatorname{diag}(v_{\text{scene}}) V_h$ gives a parameter-efficient way to inject side information into attention, applicable beyond scene recognition to other global attributes of an image.
  • Because the module plugs into the standard two-LSTM encoder-decoder, it can be combined with other attention mechanisms and with reinforcement-learning training schemes, as the paper notes.
  • Scene-conditioned captioning produces more scene-specific keywords, as shown in the paper's qualitative examples where the model says 'kitchen counter' instead of 'counter top' or 'in the ocean' instead of 'on a surfboard'.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The claimed mechanism could be tested by ablating against additive or gated scene conditioning using identical scene features; if the diagonal factorization is the active ingredient, it should beat those alternatives by a similar margin.
  • Because the scene vector comes from a Places-pretrained classifier, COCO images that fall outside typical Places categories may receive noisy scene labels; the method's robustness to that noise is not measured and could be probed by corrupting $v_{\text{scene}}$ at test time.
  • Scene-conditioned attention may also help captioning in specialized domains such as news photography, medical imaging, or egocentric video, where the scene type strongly constrains the vocabulary.
  • The paper reports single-model results that sometimes exceed published ensembles on the online COCO server, suggesting scene conditioning and ensembling may be additive, though this is not tested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a scene-based factored attention module for image captioning. The model extracts three levels of visual features - regional features from Faster R-CNN, object-centered visual concepts, and scene concepts from a Places-pretrained CNN - and embeds the scene vector vscene as a diagonal matrix S into the attention-weight factorization Wh = Uh S Vh (Eqs. 15-16). The resulting scene-conditioned attention weights are used to compute weighted sums of regional features and visual concepts, which are fed into a second LSTM for word generation. The model is trained with a two-term MLE objective (Eq. 25) and optionally with self-critical sequence training (Eqs. 26-27). Experiments on MS COCO Karpathy split and the online server report improvements over a re-implemented Up-Down baseline (Table 1) and over published state-of-the-art methods (Tables 2-3).

Significance. The core idea - explicitly conditioning attention on scene-level semantics through a factored weight matrix - is clean and potentially useful for hierarchical visual attention in captioning. If the reported gains were robust and correctly attributed, the paper would constitute a solid empirical contribution to the field. The manuscript also provides a clear formulation, MLE and RL training regimes, offline and online evaluation, and qualitative attention visualizations. However, the current experimental evidence does not yet establish that the scene mechanism itself causes the improvements: hyperparameter selection is performed on the test split, the re-implemented baseline appears much weaker than the published Up-Down model it is supposed to reproduce, and no ablations isolate scene semantics from added model capacity. These issues materially weaken the central claim as presented.

major comments (4)
  1. [Sec. 4.2 / Fig. 3] The scalar gamma in Eq. 25 is selected by evaluating on the Karpathy test split: the paper states, 'The objective results on the Karpathy test split with different gamma values are shown in Fig. 3' and chooses gamma = 3. Tuning any hyperparameter on the test split inflates all subsequent reported numbers and invalidates the comparisons in Tables 2 and 3, because the test split no longer provides an unbiased estimate of generalization. Please rerun the gamma selection on the validation split (or a nested validation procedure) and re-report all tables, or explicitly justify why the test-split tuning is acceptable in this case.
  2. [Table 1 / Sec. 4.1.4] The paper calls the baseline 'a re-implementation of Up-Down method proposed in [2]', yet the baseline achieves CIDEr 1.088 in Table 1, whereas the published Up-Down result in Table 2 (top) is CIDEr 1.135. This 0.047 gap is larger than the paper's entire reported gain of the full model over the re-implemented baseline (1.147 vs 1.088). It is therefore possible that the reported improvements reflect a weak re-implementation rather than the proposed scene module. Please report the re-implementation details, compare against the official Up-Down numbers, and provide error bars or significance tests for the Table 1 abations.
  3. [Table 1 / Eq. (15)-(16)] The improvement attributed to scene semantics is not isolated. Adding scene alone ('Baseline + Scene') gives CIDEr 1.124, which is lower than 'Baseline + VC' (1.131), and the full model adds not only the scene vector but also the new learned matrices Uh and Vh and the per-image diagonal conditioner. A control with an uninformative (e.g., shuffled or zero) scene vector, or an alternative fusion such as additive or gated conditioning of the same scene features, is needed to show that the scene semantics, rather than the added parameters or the factorization itself, cause the improvement. Without such controls, the central claim that 'scene concepts capture higher-level visual semantics and serve as an important cue' is not established.
  4. [Table 2 (bottom)] The RL results report the full model only (Ours, CIDEr 1.268) with no RL ablation of the scene module. The MLE ablation in Table 1 does not necessarily transfer to the RL setting, especially since the RL objective and training dynamics differ. For the state-of-the-art claim in the RL regime, please provide an RL baseline (e.g., Up-Down re-implemented with the same training scheme) and an RL ablation without the scene module, so that the RL gain can be attributed to the proposed mechanism.
minor comments (5)
  1. [Sec. 4.1.2] The sentence 'All the sentences in the training set are truncated to 16 characters' should read 'truncated to 16 words' (or 'tokens'), since truncating to 16 characters would severely cut the captions.
  2. [Sec. 4.3] The text reads 'BLUE, ROUGE, CIDEr scores'; 'BLUE' should be 'BLEU'.
  3. [Eq. (20)] In Eq. (20), 'WvbVobj' uses an uppercase 'V' for the object-centered visual concepts; for consistency with Eq. (17) and the text, this should be 'Wvb vobj' (lowercase bold).
  4. [Table 3] The table caption and the text note that SCST:Att2all and Up-Down are ensembles of 4 models while Ours is a single model; this is helpful, but the comparison would be cleaner if the table also listed the corresponding single-model results from those works for direct comparison.
  5. [References] Reference [16] (Gao et al., Deliberate Attention Networks) appears to lack publication venue/arXiv ID and year in the reference list; please complete the bibliographic entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: scene embeddings come from an external Places-trained model and the captioning gains are empirical, not derived from the evaluation targets.

full rationale

The paper's central claim—that embedding scene concepts into factored attention weights improves captioning—is an empirical architecture claim, not a derivation. S=diag(vscene) (Eq. 15) and W_h=U_h S V_h (Eq. 16) are design choices; vscene is produced by a CNN pretrained on the Places dataset (Sec. 4.1.2), external to MS COCO, so the scene signal is not constructed from the captions or metrics being predicted. No output quantity is defined in terms of the reported metrics, and no fitted parameter is renamed as a prediction. The ablation (Table 1) compares model variants, and the state-of-the-art comparisons are benchmark results rather than consequences of the equations. The main methodological caveat is that gamma is tuned on the Karpathy test split (Sec. 4.2), which biases the reported numbers upward; however, this is test-set selection bias, not circularity by construction, and it does not collapse the scene-attention claim into its inputs. Self-citations ([4], [5]) appear only as related work and are not load-bearing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the transferability of pretrained scene and object concept extractors to COCO, on a re-implementation of the Up-Down baseline being faithful, and on the diagonal factorization being a sufficient way to condition attention. The only free parameter explicitly tuned on the evaluation data is gamma. No invented entities are introduced.

free parameters (2)
  • Loss weighting gamma (γ) = 3
    Chosen by a variable-controlling experiment on the Karpathy test split (Figure 3, Sec 4.2). The paper states the optimal scores occur at gamma=3, so the reported numbers are selected on the test data.
  • Model dimension hyperparameters (LSTM hidden size, attention hidden, embedding size, beam size) = hidden=1000, attn=512, embed=1000, beam=2
    Stated in Sec 4.1.4; standard choices, but no sensitivity analysis is provided, so the reported scores depend on them.
assumptions (4)
  • domain assumption Places-pretrained scene classifier features transfer to MS COCO images.
    Sec 4.1.2: scene concepts Vscene come from a CNN pretrained on the Place dataset. The method conditions attention on these features; if scene predictions are noisy or domain-mismatched, the module may not help.
  • domain assumption Object concept extractor trained on COCO caption nouns gives useful middle-level concepts.
    Sec 4.1.2: object-centered visual concepts Vobj are extracted from a CNN trained on COCO caption nouns with multi-label loss; the approach depends on the quality of these concepts.
  • domain assumption The re-implemented baseline equals the published Up-Down model.
    Sec 4.2 calls the baseline 'a re-implementation of Up-Down method'. The comparisons in Tables 2 and 3 assume this re-implementation matches the original Up-Down numbers; if not, the reported gains are inflated.
  • ad hoc to paper The factorization Wh = Uh S Vh with diagonal S is sufficient to capture scene-conditioned attention.
    Eq. 16 introduces this factorization without a study of alternative interaction forms (e.g., full matrix S or additive conditioning); the method's performance rests on this design choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scene-based Factored Attention for Image Captioning." pith.science (2026). https://pith.science/paper/BHB3PKEF

@misc{pith2026190802632,
  author       = {Pith},
  title        = {Pith review of: Scene-based Factored Attention for Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BHB3PKEF}},
  note         = {Machine review of arXiv:1908.02632}
}
read the original abstract

Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored attention module for image captioning. Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image. Then, an adaptive LSTM is used to generate captions for specific scene types. Experimental results on Microsoft COCO benchmark show that the proposed scene-based attention module improves model performance a lot, which outperforms the state-of-the-art approaches under various evaluation metrics.

Figures

Figures reproduced from arXiv: 1908.02632 by the authors.

Figure 1
Figure 1. Top: Scene concepts affect word chosen in caption gen￾eration. Middle: Words probability distribution of leveraging scene concepts as semantic concepts. Bottom: Words probabil￾ity distribution of our scene-based factored attention method. tion [35], some captioning methods [49, 53, 15] also prefer to selectively attend a set of detected object-centered visual concepts. These concepts are then combined into the hid￾d… view at source ↗
Figure 2
Figure 2. The overview of the proposed model. Given a set of visual information, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A variable-controlling experiment for γ selection sults demonstrate that our proposed scene-based factored at￾tention module can significantly boost the scores compared with the existing state-of-the-arts We also compare our model to the recent results on the official MS COCO evaluation by uploading results to the online MS COCO test server. The online server provides ”C5” and ”C40” metrics which denote 5 reference … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative analysis. The notation of ”Detected” denote the scene concepts detected from the image. And the notations of ”Ours [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of attention regions with/without scene. The notations of ”Ours w scene” and ”Ours wo scene” denote our proposed [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 45 canonical work pages

  1. [2]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Con- 8 ference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018

    work page 2018

  2. [1]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382–398. Springer, 2016

    work page 2016

  3. [3]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  4. [4]

    Structcap: Structured semantic embedding for image captioning

    Fuhai Chen, Rongrong Ji, Jinsong Su, Yongjian Wu, and Yunsheng Wu. Structcap: Structured semantic embedding for image captioning. In Proceedings of the 25th ACM in- ternational conference on Multimedia , pages 46–54. ACM, 2017

    work page 2017

  5. [5]

    Groupcap: Group-based image captioning with structured relevance and diversity constraints

    Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. Groupcap: Group-based image captioning with structured relevance and diversity constraints. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1345–1353, 2018

    work page 2018

  6. [6]

    Boosted attention: Leveraging hu- man attention for image captioning

    Shi Chen and Qi Zhao. Boosted attention: Leveraging hu- man attention for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 68–84, 2018

    work page 2018

  7. [7]

    Show, adapt and tell: Adversarial training of cross-domain image cap- tioner

    Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun. Show, adapt and tell: Adversarial training of cross-domain image cap- tioner. In Proceedings of the IEEE International Conference on Computer Vision, pages 521–530, 2017

    work page 2017

  8. [8]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

Show all 56 references
  1. [9]

    Mind’s eye: A recur- rent visual representation for image caption generation

    Xinlei Chen and C Lawrence Zitnick. Mind’s eye: A recur- rent visual representation for image caption generation. In CVPR, pages 2422–2431, 2015

    2015

  2. [10]

    Regularizing rnns for caption generation by reconstruct- ing the past with the present

    Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu. Regularizing rnns for caption generation by reconstruct- ing the past with the present. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7995–8003, 2018

    2018

  3. [11]

    To- wards diverse and natural image descriptions via a condi- tional gan

    Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. To- wards diverse and natural image descriptions via a condi- tional gan. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 2970–2979, 2017

    2017

  4. [12]

    Meteor universal: Lan- guage specific translation evaluation for any target language

    Michael Denkowski and Alon Lavie. Meteor universal: Lan- guage specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014

    2014

  5. [13]

    Aligning where to see and what to tell: Image cap- tioning with region-based attention and scene-specific con- texts

    Kun Fu, Junqi Jin, Runpeng Cui, Fei Sha, and Changshui Zhang. Aligning where to see and what to tell: Image cap- tioning with region-based attention and scene-specific con- texts. IEEE transactions on pattern analysis and machine intelligence, 39(12):2321–2334, 2017

    2017

  6. [14]

    Stylenet: Generating attractive visual captions with styles

    Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3137–3146, 2017

    2017

  7. [15]

    Semantic compositional networks for visual captioning

    Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5630–5639, 2017

    2017

  8. [16]

    Deliberate attention networks for image captioning

    Lianli Gao, Kaixuan Fan, Jingkuan Song, Xianglong Liu, Xing Xu, and Heng Tao Shen. Deliberate attention networks for image captioning. 2019

    2019

  9. [17]

    Locally supervised deep hybrid model for scene recogni- tion

    Sheng Guo, Weilin Huang, Limin Wang, and Yu Qiao. Locally supervised deep hybrid model for scene recogni- tion. IEEE transactions on image processing , 26(2):808– 820, 2017

    2017

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016

    2016

  11. [19]

    Maximum expected bleu train- ing of phrase and lexicon translation models

    Xiaodong He and Li Deng. Maximum expected bleu train- ing of phrase and lexicon translation models. InProceedings of the 50th Annual Meeting of the Association for Computa- tional Linguistics: Long Papers-Volume 1 , pages 292–301. Association for Computational Linguistics, 2012

    2012

  12. [20]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  13. [21]

    Learning to guide decoding for image captioning

    Wenhao Jiang, Lin Ma, Xinpeng Chen, Hanwang Zhang, and Wei Liu. Learning to guide decoding for image captioning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018

    2018

  14. [22]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In CVPR, pages 3128–3137, 2015

    2015

  15. [23]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015

    2015

  16. [24]

    Multi- modal neural language models

    Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. Multi- modal neural language models. In International Conference on Machine Learning, pages 595–603, 2014

    2014

  17. [25]

    Learn- ing hierarchical semantic description via mixed-norm reg- ularization for image understanding

    Liang Li, Shuqiang Jiang, and Qingming Huang. Learn- ing hierarchical semantic description via mixed-norm reg- ularization for image understanding. IEEE Transactions on Multimedia, 14(5):1401–1413, 2012

    2012

  18. [26]

    Object bank: A high-level image representation for scene classification & semantic feature sparsification

    Li-Jia Li, Hao Su, Li Fei-Fei, and Eric P Xing. Object bank: A high-level image representation for scene classification & semantic feature sparsification. In Advances in neural infor- mation processing systems, pages 1378–1386, 2010

    2010

  19. [27]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004

    2004

  20. [28]

    Improved image captioning via policy gra- dient optimization of spider

    Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gra- dient optimization of spider. In Proceedings of the IEEE in- ternational conference on computer vision , pages 873–881, 2017

    2017

  21. [29]

    Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning

    Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 375–383, 2017

    2017

  22. [30]

    Discriminability objective for training de- scriptive captions

    Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training de- scriptive captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6964– 6974, 2018

    2018

  23. [31]

    Deep captioning with multimodal recurrent neural networks (m-rnn)

    Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 , 2014. 9

    2014 arXiv

  24. [32]

    Unsupervised learning of image transformations

    Roland Memisevic and Geoffrey Hinton. Unsupervised learning of image transformations. In 2007 IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1–

    2007

  25. [33]

    Modeling the shape of the scene: A holistic representation of the spatial envelope

    Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision , 42(3):145–175, 2001

    2001

  26. [34]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318. Association for Com- putational Linguistics, 2002

    2002

  27. [35]

    Relative attributes

    Devi Parikh and Kristen Grauman. Relative attributes. In 2011 International Conference on Computer Vision , pages 503–510. IEEE, 2011

    2011

  28. [36]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91–99, 2015

    2015

  29. [37]

    Self-critical sequence training for image captioning

    Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7008– 7024, 2017

    2017

  30. [38]

    Biologically inspired fea- ture manifold for scene classification

    Dongjin Song and Dacheng Tao. Biologically inspired fea- ture manifold for scene classification. IEEE Transactions on Image Processing, 19(1):174–184, 2010

    2010

  31. [39]

    Factored tem- poral sigmoid belief networks for sequence learning

    Jiaming Song, Zhe Gan, and Lawrence Carin. Factored tem- poral sigmoid belief networks for sequence learning. In In- ternational Conference on Machine Learning , pages 1272– 1281, 2016

    2016

  32. [40]

    Gen- erating text with recurrent neural networks

    Ilya Sutskever, James Martens, and Geoffrey E Hinton. Gen- erating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024, 2011

    2011

  33. [41]

    Sequence to sequence learning with neural networks

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112, 2014

    2014

  34. [42]

    Rethinking the inception archi- tecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2818–2826, 2016

    2016

  35. [43]

    Factored con- ditional restricted boltzmann machines for modeling motion style

    Graham W Taylor and Geoffrey E Hinton. Factored con- ditional restricted boltzmann machines for modeling motion style. In Proceedings of the 26th annual international con- ference on machine learning, pages 1025–1032. ACM, 2009

    2009

  36. [44]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

    2015

  37. [45]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In CVPR, pages 3156–3164, 2015

    2015

  38. [46]

    Show and tell: Lessons learned from the 2015 mscoco image captioning challenge

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence , 39(4):652–663, 2017

    2015

  39. [47]

    Knowledge guided disambiguation for large- scale scene classification with multi-resolution cnns

    Limin Wang, Sheng Guo, Weilin Huang, Yuanjun Xiong, and Yu Qiao. Knowledge guided disambiguation for large- scale scene classification with multi-resolution cnns. IEEE Transactions on Image Processing, 26(4):2055–2068, 2017

    2017

  40. [48]

    Semantics- preserving bag-of-words models and applications

    Lei Wu, Steven CH Hoi, and Nenghai Yu. Semantics- preserving bag-of-words models and applications. IEEE Transactions on Image Processing, 19(7):1908–1920, 2010

    1908

  41. [49]

    Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel. What value do explicit high level concepts have in vision to language problems? In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 203–212, 2016

    2016

  42. [50]

    On multiplicative integration with recurrent neural networks

    Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov. On multiplicative integration with recurrent neural networks. In Advances in neural infor- mation processing systems, pages 2856–2864, 2016

    2016

  43. [51]

    Show, attend and tell: Neural im- age caption generation with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural im- age caption generation with visual attention. In ICML, vol- ume 14, pages 77–81, 2015

    2015

  44. [52]

    Review networks for caption gen- eration

    Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, and Ruslan R Salakhutdinov. Review networks for caption gen- eration. In Advances in Neural Information Processing Sys- tems, pages 2361–2369, 2016

    2016

  45. [53]

    Boosting image captioning with attributes

    Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting image captioning with attributes. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 4894–4902, 2017

    2017

  46. [54]

    Image captioning with semantic attention

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651–4659, 2016

    2016

  47. [55]

    Pairwise constraints based multiview features fusion for scene classi- fication

    Jun Yu, Dacheng Tao, Yong Rui, and Jun Cheng. Pairwise constraints based multiview features fusion for scene classi- fication. Pattern Recognition, 46(2):483–496, 2013

    2013

  48. [56]

    Learning deep features for scene recognition using places database

    Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Tor- ralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages 487–495, 2014. 10

    2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.