Pith. sign in

REVIEW 4 major objections 7 minor 67 references

Language Features Matter: Effective Language Representations for Vision-Language Tasks

T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that the choice of language representation matters as much as visual features in vision-language tasks, and introduces a multi-task-trained graph embedding, GrOVLE, that outperforms standard embeddings on four of five…

desk verdict A genuinely useful empirical survey whose headline 'four of five' claim does not survive close reading; worth refereeing with re-analysis. read the letter →

arxiv 1908.06327 v1 pith:F7VGCPE2 submitted 2019-08-17 cs.CV cs.CL

Andrea Burns , Reuben Tan , Kate Saenko , Stan Sclaroff , Bryan A. Plummer This is my paper · ORCID
classification cs.CVcs.CL
keywords vision-languagetaskswordembeddingsmulti-tasktrainingGrOVLEretrofittingVisualGenomeimage-sentenceretrievallanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the language side of vision-language models has been under-engineered, and that the choice of word embedding and language model changes task performance as much as visual features do. It compares ten embeddings and three language-model architectures across five tasks, finding that simple average-pooled embeddings beat LSTMs on retrieval-style tasks, and that text-only state-of-the-art embeddings such as BERT do not lead on any of the five tasks. The paper then introduces GrOVLE, a 300-dimensional embedding made by retrofitting Word2Vec with WordNet and a co-occurrence graph built from Visual Genome, and trains this embedding across all five tasks using a masking strategy borrowed from PackNet. The paper's central empirical claim is that the multi-task trained GrOVLE is the best embedding for four of the five tasks when paired with the best corresponding language model, which motivates its release as a ready-to-use feature.

What carries the argument

GrOVLE (Graph Oriented Vision-Language Embedding) is the object that carries the argument. It is a 300-dimensional word embedding produced by retrofitting Word2Vec: the vector for each word is pulled toward its neighbors in a joint graph while a squared-distance penalty keeps it near the original pretrained vector. The graph combines WordNet synonym/hypernym/hyponym links with a novel Visual Genome relational graph, in which words are connected if they co-occur often in the same dense image description, ranked by pointwise mutual information. Finally, multi-task training adapts GrOVLE across all five tasks by freezing the $K$ most informative dimensions per task ($K=60$ with $D=300$ and $T=5$), following the pruning idea of PackNet, before fine-tuning on the next task; the released version is this multi-task embedding.

What would settle it

Re-run the same five-task comparison with the best language model chosen on a separate validation split and report standard deviations over multiple random seeds; the central claim fails if multi-task GrOVLE with fine-tuning does not lead on at least four tasks under that protocol. A direct spot-check is to evaluate the released GrOVLE on an additional vision-language task it was not trained on and compare against Word2Vec with the same language model.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that language representations for vision-language tasks can be systematically improved along three axes: using average-pooled or self-attention language models for retrieval and grounding rather than LSTMs, fine-tuning the embedding on each task, and injecting visual and relational knowledge through retrofitting plus multi-task training. The concrete embodiment of this recipe is GrOVLE, which updates Word2Vec vectors so that words linked in WordNet or co-occurring in Visual Genome descriptions move closer together, and then selects a per-task subset of embedding dimensions to keep frozen during multi-task training. In the reported experiments, multi-task GrOVLE with fine-tuning outperforms Word2Vec, FastText, Visual Word2Vec, HGLMM Fisher vectors, InferSent, and BERT on image-sentence retrieval, phrase grounding, text-to-clip retrieval, and VQA, while being a compact 300-dimensional representation. The paper reads these results as showing that language features deserve the same design attention as visual features.

Load-bearing premise

The paper selects the best language model for each task using test-set performance and reports small gains without error bars, so the multi-task advantage could partly reflect overfitting to those test sets rather than genuine transfer.

Editorial extensions

If this is right

  • Swapping in multi-task GrOVLE with the recommended language model should improve retrieval and grounding accuracy in existing pipelines without changing the rest of the architecture.
  • The strong performance of average-pooled embeddings implies that word order is not essential for matching sentences to images and video clips, so simpler and cheaper language models are viable for those tasks.
  • State-of-the-art text-only representations like BERT should not be assumed to transfer to vision-language tasks; task-specific visual grounding of the embedding matters more.
  • Multi-task pretraining of word embeddings transfers knowledge across vision-language tasks and does not hurt any single task, with the largest gains appearing in image-sentence retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GrOVLE is a static 300-D embedding, the comparison does not include fine-tuned contextual representations at comparable compute; a modern re-run would likely narrow the gap, so the lead may be specific to the static-embedding regime.
  • The Visual Genome co-occurrence graph is task-agnostic, so the same retrofitting recipe could be applied to dense annotations in specialized domains, such as medical imaging or satellite imagery, to build domain-specific vision-language embeddings.
  • The average-embedding result may depend on the short, concrete sentences in Flickr30K and ReferIt; tasks with longer or more compositional language might favor order-sensitive models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper conducts an extensive empirical comparison of ten word and sentence embeddings (from-scratch, Word2Vec, FastText, InferSent, BERT, visual variants, Fisher vectors, and a new GrOVLE) across three language models (Average Embedding, Self-Attention, LSTM) on five vision-language tasks: image-sentence retrieval, phrase grounding, text-to-clip retrieval, image captioning, and VQA. It introduces GrOVLE, a Word2Vec embedding retrofitted with WordNet and a novel Visual Genome co-occurrence graph, and evaluates multi-task pretraining of GrOVLE with and without fine-tuning on the target task. The paper claims five findings, the strongest being that multi-task trained GrOVLE is the leading embedding for four of the five tasks, and releases the embedding for public use.

Significance. If the claims held up, this paper would be a valuable practical resource: it would provide a systematic comparison of language feature choices across common VL tasks and a ready-to-use 300-D vision-language embedding. The study's strengths include the breadth of the evaluation, the use of publicly available code for most task models, the inclusion of held-out 'w/o target task' experiments for transfer, and the release of the GrOVLE artifact. However, the headline 'four of five' claim is not stable across the two task-model families (original models vs. additional models), and the model selection procedure relies on test-set performance without variance estimates. The central empirical claims therefore need substantial additional support before the conclusions can be accepted as stated.

major comments (4)
  1. [Section 7.1, Tables 1-3] Section 7.1 states, 'We use the best performing language model in our comparisons for each task.' This selection is made by comparing test-set metrics from Tables 1 and 2, and the multi-task variant chosen for the headline claim is the one that performs best on the test set. This is a test-set-based model selection, which can inflate reported gains and makes the 'leading embedding' ranking optimistic. A held-out validation split should be used for choosing both the language model and the multi-task variant, or the paper should report all variants and a pre-specified selection rule.
  2. [Section 8, Finding 5; Tables 3, 4, 14] Finding 5 ('Multi-task trained GrOVLE is the leading embedding option for four of the five vision-language tasks') is not stable across the two task-model families. With the original models (Table 3), the '+ multi-task pretraining w/ target task + ft' row is not leading on VQA (61.46 vs. 61.53 for the non-fine-tuned variant). With the additional models, Table 14 shows multi-task GrOVLE + ft on TGN (43.62) below GrOVLE without multi-task pretraining + ft (44.59) for text-to-clip. The excluded task therefore changes depending on which task models are consulted, so the 'four of five' claim is a post-hoc selection rather than a stable empirical generalization.
  3. [Table 3; Abstract] The released GrOVLE is the multi-task-pretrained embedding, and the best row in Table 3 always includes fine-tuning on the target task (row 4). This conflates the embedding itself with a per-task fine-tuning procedure; the 'ready-to-use' claim in the abstract is not supported by the honest held-out row (line 2, 'w/o target task'), where gains over the baseline are at most about 1.2 points and GrOVLE is not the leading embedding. The paper should separate the fixed-embedding transfer result from the fine-tuned result and state which artifact is released.
  4. [Tables 2-4] Many reported differences are smaller than one metric point (e.g., VQA 61.46 vs. 61.53 in Table 3; BLEU-4 differences of 0.1-0.3 in Table 2), yet no error bars, multiple seeds, or significance tests are reported. This is not sufficient support for claims that one embedding 'is the leading' option when the margin is below plausible run-to-run noise. At minimum, report multiple runs with standard deviation for the headline comparisons.
minor comments (7)
  1. [Section 3, Figure 3] The notation for the Self-Attention model ('context vector C', 'W') is introduced without formal definitions; please define these symbols or refer to the figure more explicitly.
  2. [Section 5.3, Table 1] The statement 'Average Embedding and Self-Attention are not used in generation tasks for Image Captioning and VQA as they are known to show worse performance' is plausible but gives no citation or experiment; either provide a reference or report the numbers.
  3. [Section 9.4] The appendix notes that baseline results differ from published values because ensembles are used in the original papers; this is important for comparability and should be stated in the main text, not only in the appendix.
  4. [Section 5.3, Table 1(d)] The claim that BERT 'surprisingly outperforms InferSent by 11.55%' on Flickr30K Entities should be phrased as an absolute accuracy difference, not a percentage, and the BERT representation (which layer and pooling) should be specified.
  5. [Section 6.3.2] The Visual Genome co-occurrence thresholds (more than 50 occurrences, top-10 PMI neighbors) are presented without sensitivity analysis; a brief ablation would strengthen the GrOVLE construction.
  6. [Table 3 caption] The caption uses 'w/o target task' and 'w/ target task' without defining that the target task is the task on which the embedding is evaluated; please define explicitly.
  7. [Section 7] In Section 7, 'K = D/T' is stated, but the definition of K as the number of 'most informative features' depends on a variance ranking that is only described in prose; please formalize the feature-freezing procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical benchmark comparison, and the multi-task result is accompanied by a held-out 'w/o target task' control rather than being predicted from its own training objective.

full rationale

The paper's contributions are measurements: it compares embeddings and language models on five fixed benchmarks, and it introduces GrOVLE by retrofitting Word2Vec with WordNet and a Visual Genome co-occurrence graph. There is no derivation chain in which an output quantity is defined in terms of the quantity it is claimed to predict. The closest potential concern is the 'multi-task pretraining w/ target task + ft' row in Table 3, since that embedding was trained on the target task before evaluation; however, the paper explicitly labels this row and also provides a held-out 'w/o target task' row, so the transfer claim is not a fitted input disguised as a prediction. The authors' own prior systems (CITE, n2nmn) are used as task models, but they are published, code-reproduced baselines and are cross-checked with additional models (SCAN, TGN, BUTD, BAN), so the central comparisons do not reduce to self-citation. Issues such as using the test set to choose the best language model and the changing identity of the excluded task across model families are robustness concerns, but they are not circularity under the specified criteria. Therefore, no circular steps are identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claims rest on several hand-set choices: the Visual Genome co-occurrence threshold and per-word edge count, the multi-task feature budget K, and per-task regularization coefficients. The paper also assumes that the selected tasks, metrics, and task models are representative, and that BERT used as a fixed sentence embedding is a fair comparison point. No new physical entities are introduced; GrOVLE is a new trained artifact.

free parameters (4)
  • Visual Genome co-occurrence threshold = 50
    Word pairs with fewer than 50 co-occurrences in Visual Genome are excluded from the relational graph (Section 6.3.2).
  • PMI top-k neighbors per word = 10
    For each word, links are created to the top 10 co-occurring words by PMI (Section 6.3.2).
  • Multi-task feature budget K = 60
    K = D/T = 300/5 = 60 features frozen per task during multi-task training, following PackNet (Section 7).
  • Fine-tuning L2 regularization coefficients = 1e-4 to 5e-2 (per task)
    Alpha values for embedding fine-tuning are set separately for each task and language model (Appendix 9.2).
assumptions (5)
  • standard math Retrofitting objective from Faruqui et al. (2015) improves embeddings
    The paper adopts the retrofitting algorithm and its regularization parameters wholesale (Section 6.3.1), assuming the objective behaves as claimed.
  • domain assumption The five selected VL tasks and metrics are representative
    Best practices are generalized from results on image-sentence retrieval, phrase grounding, text-to-clip, captioning, and VQA (Section 4.1).
  • domain assumption Visual Genome co-occurrence and PMI capture visual semantic relations
    GrOVLE's graph is built from word co-occurrences in Visual Genome annotations (Section 6.3.2), assuming these links encode task-relevant semantics.
  • domain assumption PackNet-style feature freezing transfers to word embeddings
    The multi-task training freezes K=60 features per task (Section 7), assuming variance-based freezing prevents catastrophic forgetting in embeddings as it does in networks.
  • domain assumption BERT as a fixed, non-fine-tuned sentence embedding is a valid comparison
    The paper compares BERT without fine-tuning against fine-tuned baselines (Section 5.3), which is an assumption about the fairness of the setup.
invented entities (1)
  • GrOVLE embedding independent evidence
    purpose: Provide a 300-D vision-language word embedding for use in VL tasks
    Not a physical entity, but a new trained artifact introduced by the paper. The embedding is publicly released and can be benchmarked on other tasks, providing a falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Features Matter: Effective Language Representations for Vision-Language Tasks." pith.science (2026). https://pith.science/paper/F7VGCPE2

@misc{pith2026190806327,
  author       = {Pith},
  title        = {Pith review of: Language Features Matter: Effective Language Representations for Vision-Language Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7VGCPE2}},
  note         = {Machine review of arXiv:1908.06327}
}
read the original abstract

Shouldn't language and vision features be treated equally in vision-language (VL) tasks? Many VL approaches treat the language component as an afterthought, using simple language models that are either built upon fixed word embeddings trained on text-only data or are learned from scratch. We believe that language features deserve more attention, and conduct experiments which compare different word embeddings, language models, and embedding augmentation steps on five common VL tasks: image-sentence retrieval, image captioning, visual question answering, phrase grounding, and text-to-clip retrieval. Our experiments provide some striking results; an average embedding language model outperforms an LSTM on retrieval-style tasks; state-of-the-art representations such as BERT perform relatively poorly on vision-language tasks. From this comprehensive set of experiments we propose a set of best practices for incorporating the language component of VL tasks. To further elevate language features, we also show that knowledge in vision-language problems can be transferred across tasks to gain performance with multi-task training. This multi-task training is applied to a new Graph Oriented Vision-Language Embedding (GrOVLE), which we adapt from Word2Vec using WordNet and an original visual-language graph built from Visual Genome, providing a ready-to-use vision-language embedding: http://ai.bu.edu/grovle.

Figures

Figures reproduced from arXiv: 1908.06327 by the authors.

Figure 1
Figure 1. How should language features be constructed for a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average rank is defined using each tasks’ best perform [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The language model variants used in our experiments [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 58 canonical work pages

  1. [1]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. 1, 3, 8, 12, 13, 18

    work page 2018

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. 1, 3

    work page 2015

  3. [3]

    A neural probabilistic language model

    Yoshua Bengio, Rjean Ducharme, Pascal Vincent, and Chris- tian Jauvin. A neural probabilistic language model. In Jour- nal of Machine Learning Research, 3:1137-1155, 2003. 4

    work page 2003

  4. [4]

    Enriching word vectors with subword infor- mation

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword infor- mation. Transactions of the Association for Computational Linguistics, 5:135–146, 2017. 1, 4, 5

    work page 2017

  5. [5]

    Temporally grounding natural sentence in video

    Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat- Seng Chua. Temporally grounding natural sentence in video. In EMNLP, 2018. 1, 8, 11, 12, 16

    work page 2018

  6. [6]

    Query-guided regression network with context policy for phrase grounding

    Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In ICCV, 2017. 3

    work page 2017

  7. [7]

    Regularizing rnns for caption generation by reconstruct- ing the past with the present

    Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu. Regularizing rnns for caption generation by reconstruct- ing the past with the present. In arXiv:1803.11439v2, 2018. 5, 11, 17

    arXiv 2018

  8. [8]

    Supervised learning of univer- sal sentence representations from natural language inference data

    Alexis Conneau, Douwe Kiela, Holger Schwenk, and Loc Barraultand Antoine Bordes. Supervised learning of univer- sal sentence representations from natural language inference data. 2017. 1, 4, 5

    work page 2017

Show all 67 references
  1. [9]

    ImageNet: A Large-Scale Hierarchical Image Database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. 11

    2009

  2. [10]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InarXiv:1810.04805v1,

    arXiv

  3. [11]

    Fleet, Jamie Ryan Kiros, and Sanja Fidler

    Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, 2018. 2

    2018

  4. [12]

    Image caption- ing with word level attention

    Fang Fang, Hanli Wang, and Pengjie Tang. Image caption- ing with word level attention. 2018 25th IEEE International Conference on Image Processing (ICIP), pages 1278–1282,

    2018

  5. [13]

    From captions to visual concepts and back

    Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivas- tava, Li Deng, Piotr Doll ´ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John Platt, et al. From captions to visual concepts and back. arXiv:1411.4952, 2014. 1

    2014 arXiv

  6. [14]

    Jauhar, Chris Dyer, Eduard Hovy, and Noah A

    Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting word vectors to semantic lexicons. In NAACL, 2015. 1, 2, 6, 7

    2015

  7. [15]

    Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding

    Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding. In EMNLP, 2016. 1

    2016

  8. [16]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 5, 11

    2017

  9. [17]

    The IAPR TC-12 benchmark – a new evaluation resource for visual information systems, 2006

    Michael Grubinger, Paul Clough, Henning Mller, and Thomas Deselaers. The IAPR TC-12 benchmark – a new evaluation resource for visual information systems, 2006. 11

    2006

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015. 11

    2015 arXiv

  11. [19]

    Localizing mo- ments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language. In ICCV, 2017. 1, 3, 5, 11

    2017

  12. [20]

    Discriminative learning of open-vocabulary object retrieval and localization by neg- ative phrase augmentation

    Ryota Hinami and Shin’ichi Satoh. Discriminative learning of open-vocabulary object retrieval and localization by neg- ative phrase augmentation. In EMNLP, 2018. 1, 8, 12, 15

    2018

  13. [21]

    Learning to reason: End-to- end module networks for visual question answering

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to- end module networks for visual question answering. CoRR, abs/1704.05526, 3, 2017. 5, 12

    2017 arXiv

  14. [22]

    Natural language object retrieval

    Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, 2016. 1, 11

    2016

  15. [23]

    Learning semantic concepts and order for image and sentence matching

    Yan Huang, Qi Wu, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In CVPR, 2018. 1

    2018

  16. [24]

    ReferItGame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 5, 11

    2014

  17. [25]

    Learning image embeddings using convolutional neural networks for improved multi- modal semantics

    Douwe Kiela and Lon Bottou. Learning image embeddings using convolutional neural networks for improved multi- modal semantics. In EMNLP, 2014. 2

    2014

  18. [26]

    Bilin- ear attention networks

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilin- ear attention networks. In NeurIPS, 2018. 8, 13, 18

    2018

  19. [27]

    Fisher vectors derived from hybrid gaussian-laplacian mixture mod- els for image annotation

    Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Fisher vectors derived from hybrid gaussian-laplacian mixture mod- els for image annotation. In CVPR, 2015. 1, 2, 6, 7

    2015

  20. [28]

    What are you talking about? text-to-image coreference

    Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. 3

    2014

  21. [29]

    Satwik Kottur, Ramakrishna Vedantam, Jose M. F. Moura, and Devi Parikh. Visual word2vec (vis-w2v): Learning vi- sually grounded word embeddings using abstract scenes. In CVPR, 2016. 1, 2, 5, 7

    2016

  22. [30]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017. 1

    2017

  23. [31]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016...

    2016

  24. [32]

    Combining language and vision with a multimodal skip- gram model

    Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. Combining language and vision with a multimodal skip- gram model. In NAACL, 2015. 2

    2015

  25. [33]

    Stacked cross attention for image-text matching

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xi- aodong He. Stacked cross attention for image-text matching. In ECCV, 2018. 1, 8, 12, 13, 14

    2018

  26. [34]

    RNN fisher vectors for action recognition and image annotation

    Guy Lev, Gil Sadeh, Benjamin Klein, and Lior Wolf. RNN fisher vectors for action recognition and image annotation. In ECCV, 2016. 2

    2016

  27. [35]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 5, 11

    2014

  28. [36]

    Temporal modular networks for retrieving complex compositional activities in videos

    Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Temporal modular networks for retrieving complex compositional activities in videos. In ECCV, 2018. 11

    2018

  29. [37]

    Hierarchical question-image co-attention for visual question answering

    Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Proceedings of the 30th International Confer- ence on Neural Information Processing Systems , NIPS’16, pages 289–297, USA, 2016. Curran Associates Inc. 1

    2016

  30. [38]

    Packnet: Adding mul- tiple tasks to a single network by iterative pruning

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding mul- tiple tasks to a single network by iterative pruning. InCVPR,

  31. [39]

    Linguis- tic regularities in continuous space word representations

    Tom Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguis- tic regularities in continuous space word representations. In NAACL, 2013. 1, 4, 5

    2013

  32. [40]

    George A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995. 6

    1995

  33. [41]

    Dual attention networks for multimodal reasoning and matching

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In CVPR, 2017. 1, 3, 6, 11

    2017

  34. [42]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014. 1

    2014

  35. [43]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018. 1

    2018

  36. [44]

    Plummer, Paige Kordas, M

    Bryan A. Plummer, Paige Kordas, M. Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, and Svetlana Lazebnik. Condi- tional image-text embedding networks. In ECCV, 2018. 1, 5, 6, 11, 13, 15, 16

    2018

  37. [45]

    Plummer, Arun Mallya, Christopher M

    Bryan A. Plummer, Arun Mallya, Christopher M. Cervantes, Julia Hockenmaier, and Svetlana Lazebnik. Phrase local- ization and visual relationship detection with comprehensive image-language cues. In ICCV, 2017. 1, 3

    2017

  38. [46]

    Plummer, Kevin J

    Bryan A. Plummer, Kevin J. Shih, Yichen Li, Ke Xu, Svet- lana Lazebnik, Stan Sclaroff, and Kate Saenko. Revisiting image-language embeddings for open-ended phrase detec- tion. arXiv:1811.07212, 2018. 11, 12

    2018 arXiv

  39. [47]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30K Entities: Collecting region-to-phrase cor- respondences for richer image-to-sentence models. IJCV, 123(1):74–93, May 2017. 1, 5, 11

    2017

  40. [48]

    Faster R-CNN: Towards real-time object detection with re- gion proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with re- gion proposal networks. In NIPS, 2015. 12

    2015

  41. [49]

    Grounding of textual phrases in images by reconstruction

    Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In ECCV, 2016. 1, 11

    2016

  42. [50]

    Training region-based object detectors with online hard ex- ample mining

    Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard ex- ample mining. In CVPR, 2016. 12

    2016

  43. [51]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. 11

    2014 arXiv

  44. [52]

    Plummer, Svetlana Lazebnik, Alex C

    Tatiana Tommasi, Arun Mallya, Bryan A. Plummer, Svetlana Lazebnik, Alex C. Berg, and Tamara L. Berg. Solving Visual Madlibs with Multiple Cues. In BMVC, 2016. 1

    2016

  45. [53]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, 2017. 4

    2017

  46. [54]

    Order embeddings of images and language

    Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order embeddings of images and language. In ICLR, 2016. 1, 2

    2016

  47. [55]

    Se- quence to sequence – video to text

    Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Se- quence to sequence – video to text. In ICCV, 2015. 1

    2015

  48. [56]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In CVPR, 2015. 1, 11, 12, 13, 17

    2015

  49. [57]

    Learning two-branch neural networks for image-text match- ing tasks

    Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text match- ing tasks. arXiv:1704.03470, 2017. 1, 5, 6, 11, 13, 14

    2017 arXiv

  50. [58]

    Structured matching for phrase lo- calization

    Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. Structured matching for phrase lo- calization. In ECCV, 2016. 3

    2016

  51. [59]

    R-C3D: Region convolutional 3d network for temporal activity detection

    Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region convolutional 3d network for temporal activity detection. In ICCV, 2017. 11

    2017

  52. [60]

    Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko

    Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In AAAI, 2019. 1, 6

    2019

  53. [61]

    Show, attend and tell: Neural image caption generation with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044, 2015. 1, 3

    2015 arXiv

  54. [62]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. TACL, 2:67–78, 2014. 5, 11

    2014

  55. [63]

    Berg, and Tamara L

    Licheng Yu, Eunbyung Park, Alexander C. Berg, and Tamara L. Berg. Visual Madlibs: Fill in the blank Image Generation and Question Answering. ICCV, 2015. 1

    2015

  56. [64]

    Improving lexical embeddings with semantic knowledge

    Mo Yu and Mark Dredze. Improving lexical embeddings with semantic knowledge. In ACL, 2014. 2

    2014

  57. [65]

    Yin and Yang: Balancing and an- swering binary visual questions

    Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and an- swering binary visual questions. In CVPR, 2016. 1

    2016

  58. [66]

    Deep cross-modal projection learning for image-text matching

    Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In ECCV, 2018. 11

    2018

  59. [67]

    Datasets Flickr30K [62]

    Appendix 9.1. Datasets Flickr30K [62]. This dataset consists of 32K images obtained from the Flickr website, each of which has been anno- tated with five descriptive captions. We use the splits of Plummer et al. [47], which separate the dataset into 30K/1K/1K train/test/validat...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.