Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Towards Generating Stylized Image Captions via Adversarial Training

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that adding adversarial training to an attention-based caption generator produces stylized image captions that are more visually correlated and more lexically varied than previous two-stage systems.

desk verdict Workmanlike combination of attention and adversarial training for stylized captions; the ablation is solid, but the paper's own qualitative examples contradict its content-correlation claim. read the letter →

arxiv 1908.02943 v1 pith:UE53XZKJ submitted 2019-08-08 cs.CV cs.CL

classification cs.CVcs.CL
keywords imagecaptioningstylizedcaptionssentimentadversarialtrainingattentionmechanismpolicygradientSentiCapdatasetWassersteinGAN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stylized image captioning usually trains a model first on large factual image-caption data, then fine-tunes it on a small sentiment-labeled set. The paper argues that this two-stage recipe sacrifices visual accuracy for style and repeats a narrow set of sentiment words. ATTEND-GAN is proposed as a fix: an attention-based generator that ties each generated word to an image region, plus an adversarial discriminator that rewards captions that look human-written. The paper reports that this combination outperforms prior stylized captioning systems on every standard metric on both splits of the SentiCap dataset, and that it uses a wider range of sentiment adjectives. If correct, the result would make stylized captioning practical without needing a large aligned stylistic corpus.

What carries the argument

The mechanism that carries the argument is the adversarial fine-tuning loop between a soft-attention LSTM caption generator and a Wasserstein GAN caption discriminator. The discriminator scores whole captions; because language is discrete, Monte Carlo search completes every partially generated sentence so the discriminator can reward each prefix, and the generator is updated by policy gradient using those rewards. The attention module is what keeps the caption tied to the image: at each time step it weights image regions and conditions the next word on the weighted visual features. This is what lets the adversarial reward push the caption toward sentiment style while preserving, in principle, the image-caption correlation.

What would settle it

A human evaluation in which raters judge whether ATTEND-GAN captions name the correct objects, actions, people, and scene in each image, compared side-by-side with the non-adversarial variant and the prior state-of-the-art, would settle the central claim. If the adversarial model's content-accuracy ratings are equal to or worse than the non-adversarial model even though its n-gram scores are higher, the claim that the full model produces captions 'strongly correlated with visual content' would be undercut.

Watch

Extended reading notes

Core claim

The central claim is that the full ATTEND-GAN model achieves the best results on all image captioning metrics on both the positive and negative parts of SentiCap, and that the gain comes specifically from joining attention with adversarial fine-tuning. In the architecture, an LSTM caption generator uses soft attention over spatial image regions so each generated word is conditioned on a relevant visual region, and a Wasserstein GAN discriminator scores how human-like a complete caption is. Because captions are discrete sequences, Monte Carlo search rolls out the unfinished sentence at every word position and the generator receives a policy-gradient reward from the discriminator. The paper reports that the adversarial component lifts the model above its non-adversarial variant on every metric, and that the full model produces sentiment adjectives with higher entropy and lower concentration in the most frequent words, indicating greater stylistic variety.

Load-bearing premise

The load-bearing premise is that adversarial fine-tuning on the small SentiCap dataset improves, or at least does not degrade, how accurately the generated captions describe what is actually in the image—not just how human-like the wording sounds.

Editorial extensions

If this is right

  • On the SentiCap positive and negative splits, ATTEND-GAN reports the highest scores for BLEU-1 through BLEU-4, ROUGE-L, METEOR, CIDEr, and SPICE, with the largest average gains over the prior best model in BLEU-1, ROUGE-L, CIDEr, and BLEU-2.
  • The discriminator is doing real work: ATTEND-GAN beats its non-adversarial counterpart ATTEND-GAN−A on every metric on both splits, showing adversarial training largely removes the content-correlation penalty of two-stage fine-tuning.
  • The full model generates more stylistically varied captions, measured by higher entropy and lower Top-4 probability mass over sentiment adjectives than the non-adversarial variant.
  • Training only the attention-based generator on factual data produces almost no sentiment adjectives; adding the sentiment fine-tuning and the discriminator is what introduces stylistic vocabulary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same attention-plus-adversarial recipe could be applied to other small stylistic caption datasets—humor, formality, or narrative voice—where large aligned corpora do not exist.
  • The reported n-gram and diversity gains do not by themselves prove semantic grounding; the paper's own qualitative examples include content errors, so a human rating of object and action accuracy would be a sharper test of the 'strongly correlated with visual content' claim.
  • Future versions might replace or augment the human-likeness reward with a visual-semantic consistency reward that penalizes generated captions for naming objects or actions absent from the image.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ATTEND-GAN, a stylized image captioning model that combines an attention-based caption generator with a Wasserstein GAN discriminator trained through policy gradient. The model is first pretrained on MSCOCO and then adversarially fine-tuned on the SentiCap dataset. The authors claim state-of-the-art results on SentiCap for BLEU, METEOR, CIDEr, ROUGE-L, and SPICE, and report qualitative and quantitative (adjective entropy) analyses showing greater stylistic diversity. The code and trained model are publicly released.

Significance. Generating sentiment-bearing captions without degrading visual content is a timely and practically relevant problem, and the adversarial mechanism is a reasonable extension of sequence GANs to image captioning. The paper's strengths include a clean two-stage training design, an internal ablation (ATTEND-GAN-SA vs. ATTEND-GAN-A vs. ATTEND-GAN) that isolates the contribution of the discriminator on most metrics, a publicly available implementation, and a linguistic diversity analysis. If the content-accuracy concern raised in this report is addressed, the approach would be a solid contribution to stylized captioning.

major comments (3)
  1. [Section 4.8, Fig. 3] The claim that ATTEND-GAN produces captions 'strongly correlated with visual content' is not supported by the paper's own examples: for the soccer image the model generates 'a group of stupid people are playing frisbee on a field,' and for the woman-cutting-cake image it generates 'a man is making a bad picture of a sandwich.' The facts that 'stupid people' and 'bad picture' appear in SentiCap ground truths do not repair the object/activity errors. Because BLEU, METEOR, CIDEr, and ROUGE-L are n-gram overlap metrics that can reward stylistic phrasing at the expense of content, and SPICE is reported only for the authors' models, the quantitative improvements do not establish content fidelity. Please add a systematic content-accuracy evaluation (e.g., human judgments of correctness, object/action accuracy, or SPICE computed for all compared models) before claiming strong visual-content correlation.
  2. [Section 4.5, Table 1] The comparison with prior work is confounded by the choice of backbone. ATTEND-GAN uses a ResNet-152 encoder with spatial attention, whereas the reported numbers for SentiCap, RNN-Transfer, and SF-LSTM+Adap come from architectures with weaker encoders and no attention. The large BLEU-1 and ROUGE-L gains may therefore reflect the backbone rather than the adversarial component. The internal ablation controls for this only partially. Please either reimplement the baselines with the same backbone/attention or add an ablation without attention to isolate the contribution of the adversarial mechanism.
  3. [Section 3.1, Eq. (1)] The generator objective is written as minimizing L1(θ) = Σ_t Gθ(x_t | x1:t-1, â_t) · Z(⋅), where Z is a reward. Under the standard REINFORCE/SeqGAN formulation, the policy-gradient objective to maximize expected reward is J(θ) = E[R], so the loss to minimize is -J(θ). As written, minimizing L1 would decrease the probability of high-reward tokens. Please correct Eq. (1) (or Eq. (8)) and clarify the sign convention so that the described loss matches the implemented algorithm.
minor comments (6)
  1. [Section 4.5] The phrase 'significantly better' is used without significance tests or variance estimates; please report standard deviations across runs or a significance test, especially given the small SentiCap training set and GAN training variability.
  2. [Section 4.4] The text mentions 'a single gradient descent phase on the generator (g steps)' but the number of generator steps g is never specified; please define it explicitly.
  3. [Section 4.4] The Monte Carlo search count N=5 is quite small; please discuss the variance of the reward estimates or provide a sensitivity analysis for this hyperparameter.
  4. [Table 3] In the top-10 adjectives for ATTEND-GAN-SA, empty positions appear for both positive and negative sections; replace them with a placeholder such as '—' or state that fewer than ten sentiment-bearing adjectives were generated.
  5. [Fig. 2] The notation for the attention mechanism is not defined in the figure caption; consider adding a brief description of e_j,t, â_t, and the Attend module for readers unfamiliar with soft attention.
  6. [Section 3.3] The stopping criterion 'until ATTEND-GAN converges' is vague for a GAN; specify the actual convergence rule or the number of training epochs used in practice.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ATTEND-GAN's reported gains are evaluated against external benchmarks on held-out SentiCap test folds, not defined in terms of training objectives or fitted parameters.

full rationale

I walked the derivation chain of the paper and found no step in which a predicted quantity reduces by construction to an input, a fitted constant, or a self-citation. ATTEND-GAN is a two-stage architecture: the caption generator is pretrained on MSCOCO and then adversarially fine-tuned on the SentiCap dataset using Eqs. (1)-(10). The reported results in Table 1 are standard external captioning metrics (BLEU, METEOR, CIDEr, ROUGE-L, SPICE) computed on the held-out SentiCap test splits, with baseline numbers taken from prior published work by other authors. The training objectives (Eqs. 8 and 10) contain only reward signals from the discriminator and maximum-likelihood losses; they do not contain the evaluation metrics, so there is no fitted-input-called-prediction pattern. Model selection on validation METEOR is standard practice and does not make test-set numbers circular because the test folds are separate and the validation choice is not a fitted parameter entering the reported test values. The qualitative claims about content correlation rest on examples and analysis, not on a definitional equivalence or an imported uniqueness result. The paper does not invoke load-bearing self-citations; its references to SentiCap, SF-LSTM, SeqGAN, and other prior work are external contributions by different research groups. The skeptical concern about hallucinated content (e.g., 'frisbee' instead of soccer) is a legitimate correctness and robustness risk, but it is not a circularity concern: the derivation does not assume the conclusion that content is strongly correlated. Overall, the evaluation is self-contained against external benchmarks and no circular step was identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new conceptual entities. The central results depend on several hand-chosen hyperparameters (lambda1, lambda2, MC search count, weight clip, update ratio) and on domain assumptions about evaluation metrics and two-stage training. These are typical for an empirical deep learning paper but should be stated explicitly.

free parameters (5)
  • lambda1 (attention regularization weight) = 1.0
    In Eq. (3), balances the attention double-stochastic penalty; no sensitivity analysis is given.
  • lambda2 (adversarial loss weight) = 0.1
    In Eq. (8), weights the GAN reward against the MLE objective; no sensitivity analysis is given.
  • Monte Carlo search count N = 5
    Number of rollouts used to estimate rewards in Eq. (2); chosen without explicit justification.
  • Discriminator weight clipping bound = [-0.01, 0.01]
    WGAN weight clipping bound stated in Section 3.2 and 4.4; standard for WGAN but still a hand-chosen hyperparameter.
  • Generator-to-discriminator update ratio = 1:3
    One generator gradient step per three discriminator steps during adversarial training (Section 4.4).
assumptions (4)
  • domain assumption Standard automatic captioning metrics (BLEU, METEOR, CIDEr, ROUGE-L) are valid proxies for human judgment of caption quality, including style appropriateness.
    Used in Sections 4.2 and 4.5 to evaluate all models; no human evaluation is reported.
  • domain assumption Pretraining on MSCOCO and then fine-tuning on SentiCap preserves factual content while adding sentiment.
    The two-stage training in Section 4.3 assumes the small SentiCap fine-tuning does not destroy the image-caption correlation learned from MSCOCO.
  • domain assumption The WGAN-style discriminator with weight clipping provides stable and meaningful reward signals for caption generation.
    The adversarial training in Section 3.2 relies on the Wasserstein critic's score as a reward for the generator; no theoretical guarantee is given for discrete text.
  • standard math REINFORCE with Monte Carlo rollouts yields unbiased gradient estimates for discrete word sampling.
    Used in Eq. (1)-(2) following Yu et al.; treated as a standard result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Generating Stylized Image Captions via Adversarial Training." pith.science (2026). https://pith.science/paper/UE53XZKJ

@misc{pith2026190802943,
  author       = {Pith},
  title        = {Pith review of: Towards Generating Stylized Image Captions via Adversarial Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE53XZKJ}},
  note         = {Machine review of arXiv:1908.02943}
}
read the original abstract

While most image captioning aims to generate objective descriptions of images, the last few years have seen work on generating visually grounded image captions which have a specific style (e.g., incorporating positive or negative sentiment). However, because the stylistic component is typically the last part of training, current models usually pay more attention to the style at the expense of accurate content description. In addition, there is a lack of variability in terms of the stylistic aspects. To address these issues, we propose an image captioning model called ATTEND-GAN which has two core components: first, an attention-based caption generator to strongly correlate different parts of an image with different parts of a caption; and second, an adversarial training mechanism to assist the caption generator to add diverse stylistic components to the generated captions. Because of these components, ATTEND-GAN can generate correlated captions as well as more human-like variability of stylistic patterns. Our system outperforms the state-of-the-art as well as a collection of our baseline models. A linguistic analysis of the generated captions demonstrates that captions generated using ATTEND-GAN have a wider range of stylistic adjectives and adjective-noun pairs.

Figures

Figures reproduced from arXiv: 1908.02943 by the authors.

Figure 1
Figure 1. Examples of positive (green) and negative (red) captions. In stylistically enhanced descriptions, the content of images should still be reflected correctly. Moreover, the descriptions should fluently include stylistic words or phrases. To meet these criteria, previous models have used two-stage training: first, training on a large factual dataset to describe the content of an image; and then training on a small styl… view at source ↗
Figure 2
Figure 2. The architecture of the ATTEND-GAN model. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples on the positive (first 3) and negative (last 3) datasets (AS for ATTEND [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    In: ECCV

    Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: ECCV . pp. 382–398. Springer (2016)

  2. [2]

    In: CVPR

    Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom- up and top-down attention for image captioning and visual question answering. In: CVPR. vol. 3, p. 6 (2018) Towards Generating Stylized Image Captions via Adversarial Training 13

  3. [3]

    arXiv preprint arXiv:1701.07875 (2017)

    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)

  4. [4]

    In: 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR)

    Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T.S.: Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In: 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 6298–6306. IEEE (2017)

  5. [5]

    factual” or “emo- tional

    Chen, T., Zhang, Z., You, Q., Fang, C., Wang, Z., Jin, H., Luo, J.: “factual” or “emo- tional”: Stylized image captioning with adaptive learning and attention. arXiv preprint arXiv:1807.03871 (2018)

  6. [6]

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierar- chical image database (2009)

  7. [7]

    Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: WMT. pp. 376–380 (2014)

  8. [8]

    In: CVPR

    Gan, C., Gan, Z., He, X., Gao, J., Deng, L.: Stylenet: Generating attractive visual captions with styles. In: CVPR. IEEE (2017)

Show all 31 references
  1. [9]

    In: Advances in neural information processing systems

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y .: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)

  2. [10]

    In: CVPR

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)

  3. [11]

    arXiv preprint arXiv:1502.03167 (2015)

    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

  4. [12]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Isola, P., Zhu, J.Y ., Zhou, T., Efros, A.A.: Image-to-image translation with conditional ad- versarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)

  5. [13]

    arXiv preprint arXiv:1412.6980 (2014)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  6. [14]

    In: Proceedings of the 24th ACM inter- national conference on Multimedia

    Li, Y ., Yao, T., Mei, T., Chao, H., Rui, Y .: Share-and-chat: Achieving human-level video commenting by search and multi-view embedding. In: Proceedings of the 24th ACM inter- national conference on Multimedia. pp. 928–937. ACM (2016)

  7. [15]

    arXiv preprint arXiv:1703.07022 (2017)

    Liang, X., Hu, Z., Zhang, H., Gan, C., Xing, E.P.: Recurrent topic-transition gan for visual paragraph generation. arXiv preprint arXiv:1703.07022 (2017)

  8. [16]

    Text Summarization Branches Out (2004)

    Lin, C.Y .: Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out (2004)

  9. [17]

    In: European conference on computer vision

    Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ´ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

  10. [18]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Mathews, A., Xie, L., He, X.: Semstyle: Learning to generate stylised image captions using unaligned text. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8591–8600 (2018)

  11. [19]

    In: AAAI

    Mathews, A.P., Xie, L., He, X.: Senticap: Generating image descriptions with sentiments. In: AAAI. pp. 3574–3580 (2016)

  12. [20]

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of ma- chine translation. In: ACL. pp. 311–318. Association for Computational Linguistics (2002)

  13. [21]

    arXiv preprint arXiv:1511.06434 (2015)

    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convo- lutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  14. [22]

    In: CVPR

    Rennie, S.J., Marcheret, E., Mroueh, Y ., Ross, J., Goel, V .: Self-critical sequence training for image captioning. In: CVPR. vol. 1, p. 3 (2017)

  15. [23]

    nature 529(7587), 484 (2016) 14 Nezami et al

    Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrit- twieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. nature 529(7587), 484 (2016) 14 Nezami et al

  16. [24]

    COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)

    Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)

  17. [25]

    In: NAACL HLT

    Toutanova, K., Klein, D., Manning, C.D., Singer, Y .: Feature-rich part-of-speech tagging with a cyclic dependency network. In: NAACL HLT. pp. 173–180. Association for Compu- tational Linguistics (2003)

  18. [26]

    In: CVPR

    Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: CVPR. pp. 4566–4575. IEEE (2015)

  19. [27]

    In: CVPR

    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption gener- ator. In: CVPR. pp. 3156–3164. IEEE (2015)

  20. [28]

    In: IJCAI

    Wang, K., Wan, X.: Sentigan: Generating sentimental texts via mixture adversarial networks. In: IJCAI. pp. 4446–4452 (2018)

  21. [29]

    In: ICML

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y .: Show, attend and tell: Neural image caption generation with visual attention. In: ICML. pp. 2048–2057 (2015)

  22. [30]

    In: CVPR

    You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR. pp. 4651–4659. IEEE (2016)

  23. [31]

    In: AAAI

    Yu, L., Zhang, W., Wang, J., Yu, Y .: Seqgan: Sequence generative adversarial nets with policy gradient. In: AAAI. pp. 2852–2858 (2017)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.