CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning

Atsushi Hashimoto; Donghyun Kim; Kuniaki Saito; Kwanyong Park; Yoshitaka Ushiku

REVIEW 3 major objections 7 minor 45 references

CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single image captioning model can smoothly dial the length, descriptiveness, and word uniqueness of its captions by conditioning on continuous scalar values encoded as linear interpolation between two learned endpoint vectors.

desk verdict A solid, honest empirical paper on continuous caption conditioning; the core idea is simple and disclosed as such, but per-image smoothness and a lopsided evaluation protocol need referee attention. read the letter →

arxiv 2507.01409 v1 pith:2QYTNKJE submitted 2025-07-02 cs.CV

Kuniaki Saito , Donghyun Kim , Kwanyong Park , Atsushi Hashimoto , Yoshitaka Ushiku This is my paper · ORCID

classification cs.CV

keywords imagecaptioningcontrollablegenerationcontinuousconditioningtokenembeddinginterpolationcaptionlengthdescriptivenessvocabularyuniquenesslanguagepatterncontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CaptionSmiths, a way to make one image captioning model produce captions whose length, descriptiveness, and word uniqueness can be dialed continuously, like sliders, without training a separate model for each style. The key idea is to compute each of the three properties as a normalized scalar per caption, then represent the condition as a linear interpolation between two learned endpoint token embeddings, one for the low extreme and one for the high extreme. Trained on a mix of captioning datasets with this conditioning, the model is claimed to smoothly change its output as the scalar moves, to give higher lexical alignment than discrete-cluster conditioning, and to reduce length-control error by 506% relative to the baseline. If correct, this gives users a single model that can switch between terse and elaborate descriptions and between general and fine-grained vocabulary purely by changing continuous inputs.

What carries the argument

The central mechanism is the interpolation identity for each property: the condition embedding is $E^c = \bar{c} E_1 + (1-\bar{c}) E_0$, a linear combination of two learned endpoint vectors representing extreme states such as very short and very long. This is equivalent to a single linear layer applied to the scalar and to merging two parameter vectors, and it adds only $2d$ parameters while allowing the scalar to move continuously. A condition calculator derives the three scalars from a caption, optionally decorrelates them with linear regression, and normalizes them to $[0,1]$; the interpolated embeddings are prepended to the language model's input sequence.

What would settle it

Train CaptionSmiths while feeding only the extreme conditioning values 0 and 1 (or a few discrete values), then generate captions at intermediate scalars such as 0.25, 0.5, and 0.75 on a fixed set of images and plot the measured property (e.g., token count) against the scalar; if the curve is flat, jumps, or is non-monotonic, the central claim of smooth interpolation fails.

Watch

Extended reading notes

Core claim

CaptionSmiths establishes that a single captioning model can smoothly and controllably vary three language properties, length, descriptiveness, and word uniqueness, by conditioning on continuous scalars computed per caption without human annotation. Length is the token count; descriptiveness is the fraction of nouns and adjectives; uniqueness is the inverse corpus frequency of the words used. The conditioning is encoded by linearly interpolating between two learned endpoint vectors, and the paper shows that this continuous parameterization gives smoother control and higher lexical alignment than discrete cluster indices, improves self-retrieval of the source image, and generalizes across datasets and to a larger backbone.

Load-bearing premise

Linear interpolation between two learned endpoint token embeddings makes the output caption's properties change smoothly and monotonically with the scalar, even for intermediate values the model was never trained on.

Editorial extensions

If this is right

A single trained model can replace separate models for short, medium, and long captions, and for general versus fine-grained vocabulary.
Continuous conditioning reduces length mismatch to 1.6 tokens compared with 9.7–13.8 tokens for discrete grouping, while improving or matching lexical alignment metrics.
The three controls are mostly disentangled: varying one condition changes the other two only slightly, allowing independent adjustment.
Raising descriptiveness increases the alignment between the generated caption and the image, and raising uniqueness increases recall of fine-grained category names on bird, dog, and car classification benchmarks.
The conditioning scheme transfers to a larger vision-language model with lightweight tuning and gives zero-shot captioning gains over open baselines on unseen datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same interpolation conditioning could control other scalar text properties, such as sentiment, formality, or inferred specificity, because the mechanism does not depend on how the scalar is computed.
One could extrapolate the interpolation weights beyond the $[0,1]$ range to push properties further than the training extremes; the paper only tests interpolation, so extrapolation is a natural testable extension.
The word-frequency uniqueness metric is corpus-relative; in deployment, a user could renormalize it to their own vocabulary distribution without retraining the endpoint vectors.
Measuring the target property at many intermediate scalars on fixed images would directly quantify smoothness and monotonicity, which the paper supports qualitatively in figures rather than with a formal test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a worthwhile empirical paper with an honest, modest core idea; the biggest problems are in the evaluation protocol and one overstated number, not in the mechanism itself. The genuinely useful contribution is the Condition Calculator, which turns length, descriptiveness, and word uniqueness into normalized scalars, plus the demonstration that linearly interpolating between two learned endpoint embeddings gives a single LLaVA model slider-like control across COCO, LN-COCO, and Docci. The interpolation trick is not deep—the authors themselves show in Eq. (7) that it is equivalent to one linear layer—but that equivalence is disclosed, and the empirical payoff is real: Figure 4 and Tables 3–4 show wide length coverage and lower length mismatch than discrete cluster conditioning, and the fine-grained recall curves for uniqueness are convincing. I also credit the zero-shot results, the hallucination note in the appendix, and the Qwen LoRA experiment; they make the claim more portable. Reader who gets value: people building controllable captioning or assistive tech, who will find the recipe and ablations useful.

Soft spots, in decreasing order. First, per-image smoothness is not actually demonstrated. The curves in Figures 4, 5, and 7 are aggregate, and the training objective in Eq. (6) never penalizes a mismatch between the conditioning scalar and the measured property of the generated caption. For a specific image, the scalar-to-property response could be non-monotonic or high-variance, so the user-facing claim of smooth dialing is only supported at the population level. This is the load-bearing gap, but I would call it a missing experiment rather than evidence of failure. Second, the target-aware evaluation gives CaptionSmiths the ground-truth-derived scalars while the baselines get only dataset-level prompts or cluster indices, so the lexical alignment advantage in Table 1 is not a like-for-like comparison. It is a fair way to test controllability, but not a fair way to sell better-than-baseline alignment. Third, the abstract's 506% error reduction is wrong: Table 4 shows length mismatch dropping from 9.7 to 1.6, about an 84% reduction, and even the 100-cluster baseline is at 9.7. Fourth, code is not available despite the repo link.

Desk decision: send to review. The central claim is plausible, the empirical footprint is substantial, and the flaws are addressable with per-image trajectory plots and a rebalanced evaluation. A serious referee should ask for those, not for a rewrite of the idea.

Referee Report

3 major / 7 minor

Summary. The paper introduces CaptionSmiths, a single image-captioning model conditioned on three continuous scalar properties of captions: length, descriptiveness, and word uniqueness. The scalars are computed automatically from captions using token counts, POS-based noun/adjective ratios, and inverse word frequencies, optionally decorrelated by linear regression. A condition embedding is formed by linearly interpolating between two learned endpoint vectors (Eq. 5), and the model is trained with a standard autoregressive loss given the interpolated tokens (Eq. 6). Experiments on a mixed training set of about 1.3M image-caption pairs report lexical alignment against ground-truth captions, controllability curves for each property, self-retrieval, zero-shot results on Nocaps and Vizwiz, an ablation of discrete versus continuous conditioning, and a Qwen-backbone fine-tuning study. The central claim is that a user can smoothly and independently dial the three properties of generated captions without retraining.

Significance. If the control claim holds, CaptionSmiths is a practically useful and parameter-efficient alternative to discrete condition tokens: it adds only 2d parameters per property, avoids cluster-count tuning, and exposes an interpretable scalar interface. The paper has genuine strengths: it evaluates on a diverse mixture of caption datasets, reports zero-shot generalization, validates the descriptiveness criterion against ChatGPT, includes a larger-backbone test (Qwen2.5-VL-32B), and explicitly acknowledges long-caption hallucination as a limitation. The main gap is that the headline 'smooth control' claim is supported only by aggregate curves, while per-image monotonicity, which is what a user actually needs, is not demonstrated. The reported lexical-alignment advantage is also weakened by the target-aware evaluation protocol. With additional per-image evidence and a fairer baseline comparison, the contribution would be solid.

major comments (3)

[Sec. 3.3, Eq. (5); Sec. 4.2, Figs. 4, 5, 7; Sec. 4.3, Fig. A] The central claim that a user can smoothly dial a property for a given image is not established by the reported evidence. Equation (5) defines the conditioning embedding as an affine function of the scalar, and the training loss in Eq. (6) never compares the scalar to the measured property of the generated caption; it only trains the model to produce a caption given the scalar derived from that same caption. Figures 4, 5, and 7 report aggregate curves, and Fig. 4 explicitly shows only the baselines' mean +/- std ranges. Such averages can hide per-image non-monotonic or high-variance responses, and Fig. A reports only small aggregate changes when fixing other conditions. I recommend reporting per-image trajectories or at least the distribution of per-image Spearman correlations between each conditioning scalar and the corresponding measured property (e.g., for 10-20 images across the scalar grid), together with per-condition mean absolute errors. Without this, 'smoothly change the properties of the output captions' is a parameterization, not a demonstrated behavioral guarantee.
[Abstract; Table 4] The abstract's statement that 'CaptionSmiths reduces the error in controlling caption length by 506%' is numerically inconsistent with Table 4. The length mismatch falls from 9.7 (100 clusters) to 1.6, which is a relative reduction of about 83.5%, not 506%; equivalently, the error is about 6 times smaller. A percentage reduction larger than 100% is not meaningful for a non-negative error metric. The wording should be corrected to either a relative reduction (83.5%) or a ratio/factor (about 6x), and the phrase 'despite better lexical alignment' should be rephrased if the intended meaning is 'in addition to' or 'while also achieving' better lexical alignment.
[Sec. 4.1; Tables 1 and 3] The target-aware evaluation protocol gives CaptionSmiths an information advantage over most baselines. The conditioning scalars for CaptionSmiths are computed from the ground-truth caption (Sec. 4.1), so at test time the model receives per-caption information about the target length, descriptiveness, and uniqueness, while Concap receives only a dataset-level prompt and open-source models receive only a concise/detailed instruction. Tables 1 and 3 then measure lexical alignment against those same ground-truth captions. This protocol may inflate the reported alignment advantage, and it does not reflect a user who specifies a desired style without knowing the reference caption. I recommend at least one of the following: (i) give baselines the same scalar information through an equivalent continuous conditioning module; (ii) report controllability under user-chosen scalars independent of any GT caption; or (iii) report alignment separately for each condition value so the reader can see whether the effect is due to the scalar information rather than to the model.

minor comments (7)

[Sec. 3.3, Eq. (6)] The displayed loss in Eq. (6) is inconsistent with Eqs. (1) and (2): it lacks the 1/T_c normalization and the negative logarithm, and it sums probabilities rather than log-probabilities. Please correct the equation or the surrounding notation.
[Sec. 1 and Appendix C] There is a typo 'ofsmoothly' in Sec. 1, and the caption of Fig. D spells 'Hallunination' instead of 'Hallucination'.
[Fig. 4] The caption says that the range of length covered by each baseline is highlighted, but it does not state whether the CaptionSmiths curve is a mean, median, or per-condition aggregate, nor whether any error bars or quartiles are shown. Please specify this so the reader can interpret the aggregate nature of the curve.
[Sec. 4.2 and Fig. A] The text refers to a 'black dashed array' where the figure appears to show a range or interval; please rephrase as 'black dashed range' or 'dashed line indicating mean +/- std' for clarity.
[Table 4] The row label 'Length mismatch' is not defined in the main text; please define it explicitly (e.g., mean absolute difference in token counts between generated and ground-truth captions) and report the standard deviation or distribution, since a single mean may hide the variance relevant to the controllability claim.
[Sec. 4.3 bullet list; Appendix C] The hallucination limitation currently appears only as a bullet in the 'List of the empirical results in the appendix' and in the appendix itself. Because the paper emphasizes flexible control over long captions, this limitation should be stated in the main body and weighed in the conclusion, rather than left as an appendix remark.
[References] References [2] and [3] are duplicates of the same Qwen-VL paper, and the journal field of Reference [3] contains a typo ('1(2):3'); please deduplicate and verify all bibliographic entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CaptionSmiths' conditioning is a learned interpolation and the control claim is empirical, not derived from its inputs.

full rationale

The derivation chain is self-contained. Equation (5) defines the length-conditioning embedding as an affine interpolation E_L^c = bar L_c * E_L^1 + (1 - bar L_c) * E_L^0 between two learned endpoint vectors, and Eq. (6) is a standard conditional autoregressive language-modeling loss. The paper explicitly notes in Eq. (7) that this is equivalent to a single linear layer, so no result is being renamed or smuggled in. The central claim that varying the scalar changes caption properties smoothly is supported by empirical curves (Figs. 4, 5, 7), not derived tautologically from the interpolation formula. The only concern is the target-aware evaluation protocol, where the paper states: 'We compute the values of these properties using ground-truth captions and feed them to our model. Then, we assess the alignment between the target and generated captions.' This gives CaptionSmiths access to ground-truth-derived condition values that some baselines do not receive, which is a fair-comparison limitation that may inflate lexical-alignment numbers. However, it is not circularity under the required reduction test: the condition scalars encode only length, descriptiveness, and uniqueness, not the target caption's words, so the model must still generate the lexical content from the image. The output is not equal to the input by construction, and no fitted parameter is renamed as a prediction. The appendix's acknowledged hallucination issue is a correctness risk, not a circular step. Overall, no load-bearing self-citation or definitional equivalence was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical or ontological entities are introduced. The only new objects are learned condition vectors and regression coefficients, which are counted as free parameters. The main hidden assumptions are the validity of the property metrics and the smooth-interpolation transfer.

free parameters (4)

Endpoint conditioning vectors for length (E_L^0, E_L^1) = learned during training
Equation (5) defines the length condition as a linear interpolation of two d-dimensional learnable vectors; these are the parameters that carry the conditioning signal.
Endpoint conditioning vectors for descriptiveness (E_D^0, E_D^1) = learned during training
Analogous to length endpoints; used in Eq. (6) to condition generated captions on descriptiveness.
Endpoint conditioning vectors for uniqueness (E_U^0, E_U^1) = learned during training
Analogous to length endpoints; used in Eq. (6) to condition on vocabulary uniqueness.
Linear regression coefficients f_L and f_{L,U} for decorrelation = fitted to caption statistics in the training mixture
Appendix A uses linear regression to remove correlations among length, uniqueness, and descriptiveness; these coefficients shape the conditioning scalars used in all main experiments.

assumptions (4)

domain assumption Linear interpolation between two endpoint token embeddings produces smoothly varying, interpretable language patterns for intermediate scalars.
Sec. 3.3, Eq. (5) and Figures 4, 5, and 7 rely on this; the paper provides no formal argument that interpolation in token-embedding space transfers to generation behavior.
domain assumption Descriptiveness as the ratio of nouns and adjectives and uniqueness as inverse word frequency correspond to human-interpretable caption properties.
Sec. 3.2 defines these metrics; validation is only via ChatGPT in Figure F, and the authors acknowledge there is no human study.
domain assumption Normalization by maximum value and linear decorrelation preserve independent, [0,1]-valued control signals.
Sec. 3.2 Steps 2 and 3; disentanglement is only partially verified in Figure A.
domain assumption Word-frequency statistics and max values computed from the training mixture generalize to inference distributions, including the fine-grained datasets CUB, Stanford Dogs, and Stanford Cars.
Sec. 3.2 uses dataset-wide frequency F(w) for uniqueness; Figure 7 evaluates on fine-grained datasets not in the training mixture.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning." pith.science (2026). https://pith.science/paper/2QYTNKJE

@misc{pith2026250701409,
  author       = {Pith},
  title        = {Pith review of: CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QYTNKJE}},
  note         = {Machine review of arXiv:2507.01409}
}

read the original abstract

An image captioning model flexibly switching its language pattern, e.g., descriptiveness and length, should be useful since it can be applied to diverse applications. However, despite the dramatic improvement in generative vision-language models, fine-grained control over the properties of generated captions is not easy due to two reasons: (i) existing models are not given the properties as a condition during training and (ii) existing models cannot smoothly transition its language pattern from one state to the other. Given this challenge, we propose a new approach, CaptionSmiths, to acquire a single captioning model that can handle diverse language patterns. First, our approach quantifies three properties of each caption, length, descriptiveness, and uniqueness of a word, as continuous scalar values, without human annotation. Given the values, we represent the conditioning via interpolation between two endpoint vectors corresponding to the extreme states, e.g., one for a very short caption and one for a very long caption. Empirical results demonstrate that the resulting model can smoothly change the properties of the output captions and show higher lexical alignment than baselines. For instance, CaptionSmiths reduces the error in controlling caption length by 506\% despite better lexical alignment. Code will be available on https://github.com/omron-sinicx/captionsmiths.

Figures

Figures reproduced from arXiv: 2507.01409 by the authors.

**Figure 2.** Overview during training (left) and inference (right). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Condition Calculator (Sec. 3.2) consists of three stages: (i) property assessment computes quantifies each caption’s three properties, (ii) descriptiveness and uniqueness values are decorrelated, and (iii) values are normalized by the stats of all captions. 3.2. Condition Calculator We aim to control three key properties of captions: length, descriptiveness, and uniqueness. The optimal length of a caption varies ac… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Results of varying length conditioning. The Y-axis indi [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Generated captions by increasing descriptiveness conditioning value while fixing the other two. More objects are described. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: The effect of varying uniqueness values. We assess if the output captions include fine-grained category names as recall, using [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Generated captions by increasing a uniqueness conditioning value while fixing the other two conditions. Increasing the value can [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Comparison in the ratio of unique words in generated [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 10.** Figure 10: Increasing the uniqueness achieves richer vocabulary [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

[1]

nocaps: novel object caption- ing at scale

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. nocaps: novel object caption- ing at scale. In ICCV, pages 8948–8957, 2019. 8, 12

work page 2019
[3]

Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 1, 2

arXiv 2023
[4]

Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 5

2005
[5]

Clair: Evaluating image captions with large language models

David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971, 2023. 14

arXiv 2023
[6]

Sharegpt4v: Improving large multi-modal models with better captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, pages 370–387. Springer, 2025. 1, 2

work page 2025
[7]

Learning distinct and representative modes for image captioning

Qi Chen, Chaorui Deng, and Qi Wu. Learning distinct and representative modes for image captioning. NeurIPS, 35: 9472–9485, 2022. 1, 2, 5

work page 2022
[8]

Say as you wish: Fine-grained control of image caption generation with abstract scene graphs

Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In CVPR, pages 9962–9971, 2020. 2, 3

work page 2020
[9]

Length- controllable image captioning

Chaorui Deng, Ning Ding, Mingkui Tan, and Qi Wu. Length- controllable image captioning. In ECCV, pages 712–729. Springer, 2020. 1, 2, 4, 5, 7, 8

work page 2020

Show all 45 references

[10]

Flexcap: Describe anything in images in controllable detail

Debidatta Dwibedi, Vidhi Jain, Jonathan Tompson, Andrew Zisserman, and Yusuf Aytar. Flexcap: Describe anything in images in controllable detail. In NeurIPS, 2024. 2

2024
[11]

Captioning images taken by people who are blind

Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhat- tacharya. Captioning images taken by people who are blind. In ECCV, pages 417–434. Springer, 2020. 8, 12

2020
[12]

Clipscore: A reference-free evaluation met- ric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

arXiv
[13]

From descriptive richness to bias: Unveil- ing the dark side of generative image caption enrichment

Yusuke Hirota, Ryo Hachiuma, Chao-Han Huck Yang, and Yuta Nakashima. From descriptive richness to bias: Unveil- ing the dark side of generative image caption enrichment. arXiv preprint arXiv:2406.13912, 2024. 8, 13

2024 arXiv
[14]

spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing

Matthew Honnibal and Ines Montani. spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017. 4

2017
[15]

Scaling up vision-language pre-training for image captioning

Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. InCVPR, pages 17980–17989, 2022. 1

2022
[16]

Noise-aware learning from web-crawled image-text data for image captioning

Wooyoung Kang, Jonghwan Mun, Sungjun Lee, and Byungseok Roh. Noise-aware learning from web-crawled image-text data for image captioning. In ICCV, pages 2942– 2952, 2023. 2, 3, 4, 7

2023
[17]

Imageability-and length-controllable image caption- ing

Marc A Kastner, Kazuki Umemura, Ichiro Ide, Ya- sutomo Kawanishi, Takatsugu Hirayama, Keisuke Do- man, Daisuke Deguchi, Hiroshi Murase, and Shin’Ichi Satoh. Imageability-and length-controllable image caption- ing. IEEE Access, 9:162951–162961, 2021. 2

2021
[18]

Novel dataset for fine-grained image categorization: Stanford dogs

Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on fine-grained visual categorization (FGVC), 2011. 6

2011
[19]

3d object representations for fine-grained categoriza- tion

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- Fei. 3d object representations for fine-grained categoriza- tion. In 2013 IEEE international conference on computer vision workshops, pages 554–561. IEEE, 2013. 6

2013
[20]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 1

2023
[21]

Mon- key: Image resolution and text label are important things for large multi-modal models

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763–26773,
[22]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 5

2004
[23]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 5, 7, 11

2014
[24]

Improved baselines with visual instruction tuning

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 1, 2

2024
[25]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 1, 2, 3, 5, 6, 11 9

2024
[26]

Quark: Controllable text generation with reinforced unlearn- ing

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearn- ing. NeurIPS, 35:27591–27609, 2022. 2

2022
[27]

Analysis of diversity-accuracy tradeoff in image captioning

Ruotian Luo and Gregory Shakhnarovich. Analysis of diversity-accuracy tradeoff in image captioning. arXiv preprint arXiv:2002.11848, 2020. 1

2002 arXiv
[28]

Wordnet: a lexical database for english

George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995. 4

1995
[29]

Docci: De- scriptions of connected and contrasting images

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: De- scriptions of connected and contrasting images. In ECCV, pages 291–309. Springer, 2025. 5, 11, 15, 16

2025
[30]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318,
[31]

Adapterfusion: Non- destructive task composition for transfer learning

Jonas Pfeiffer, Aishwarya Kamath, Andreas R ¨uckl´e, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non- destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020. 2

2005 arXiv
[32]

Connecting vision and lan- guage with localized narratives

Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. In ECCV, pages 647–664. Springer, 2020. 5, 11

2020
[33]

Learn- ing transferable visual models from natural language super- vision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 1, 5

2021
[34]

Laion coco: 600m syn- thetic captions from laion2b-en, 2022

Christoph Schuhmann, Andreas K ¨opf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m syn- thetic captions from laion2b-en, 2022. 5, 11

2022
[35]

Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1, 5

2023 arXiv
[36]

Cider: Consensus-based image description evalua- tion

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In CVPR, pages 4566–4575, 2015. 5

2015
[37]

Spot: Better frozen model adaptation through soft prompt transfer.arXiv preprint arXiv:2110.07904, 2021

Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer.arXiv preprint arXiv:2110.07904, 2021. 2

2021 arXiv
[38]

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds-200-2011. Technical Report CNS-TR- 2011-001, California Institute of Technology, 2011. 6

2011
[39]

Controllable image captioning via prompting

Ning Wang, Jiahao Xie, Jihao Wu, Mingbo Jia, and Linlin Li. Controllable image captioning via prompting. In AAAI, pages 2617–2625, 2023. 1, 2, 5, 6, 7

2023
[40]

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

2024 arXiv
[41]

On diversity in image captioning: Metrics and methods

Qingzhong Wang, Jia Wan, and Antoni B Chan. On diversity in image captioning: Metrics and methods. IEEE Transac- tions on Pattern Analysis and Machine Intelligence , 44(2): 1035–1049, 2020. 1, 3

2020
[42]

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Ko- rnblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing ...

2022
[43]

xgen-mm (blip-3): A family of open large multimodal models

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 5, 6, 7, 12

2024
[44]

Minigpt-4: Enhancing vision-language understanding with advanced large language models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2

2023 arXiv
[45]

Regularization and variable se- lection via the elastic net

Hui Zou and Trevor Hastie. Regularization and variable se- lection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301–320,
[2005]

image”, “side

4 10 A. Addition Details on Method Details on decorrelation. We describe the details on decorrelating properties using linear regression. To decorrelate the properties, we apply linear regression to model the dependency between properties and remove the correlation. First, we ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

nocaps: novel object caption- ing at scale

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. nocaps: novel object caption- ing at scale. In ICCV, pages 8948–8957, 2019. 8, 12

work page 2019

[2] [3]

Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 1, 2

arXiv 2023

[3] [4]

Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 5

2005

[4] [5]

Clair: Evaluating image captions with large language models

David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971, 2023. 14

arXiv 2023

[5] [6]

Sharegpt4v: Improving large multi-modal models with better captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, pages 370–387. Springer, 2025. 1, 2

work page 2025

[6] [7]

Learning distinct and representative modes for image captioning

Qi Chen, Chaorui Deng, and Qi Wu. Learning distinct and representative modes for image captioning. NeurIPS, 35: 9472–9485, 2022. 1, 2, 5

work page 2022

[7] [8]

Say as you wish: Fine-grained control of image caption generation with abstract scene graphs

Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In CVPR, pages 9962–9971, 2020. 2, 3

work page 2020

[8] [9]

Length- controllable image captioning

Chaorui Deng, Ning Ding, Mingkui Tan, and Qi Wu. Length- controllable image captioning. In ECCV, pages 712–729. Springer, 2020. 1, 2, 4, 5, 7, 8

work page 2020