Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Multi-modality Latent Interaction Network for Visual Question Answering

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A VQA network reasons over latent modality summaries instead of every region-word pair, matching state-of-the-art accuracy with fewer message passings.

desk verdict A real but unproven centerpiece: MLIN's learned latent summarization is a plausible efficiency trick with competitive VQA numbers, but the paper never demonstrates that the learned summarizations are non-degenerate. read the letter →

arxiv 1908.04289 v1 pith:U3K76FVN submitted 2019-08-10 cs.CV cs.SDeess.ASeess.IV

Peng Gao , Haoxuan You , Zhanpeng Zhang , Xiaogang Wang , Hongsheng Li This is my paper · ORCID
classification cs.CVcs.SDeess.ASeess.IV
keywords visualquestionansweringmulti-modalityfusionlatentsummarizationvectorsrelationalreasoningcross-modalattentionmessagepassingcomplexityVQAv2.0TDIUC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that visual question answering should not reason over the large set of individual region–word relations, as most relational VQA models do, but over a small number of learned summaries of each modality. It introduces the Multi-modality Latent Interaction (MLI) module, which pools visual regions and question words into $k$ latent vectors per modality, creates $k\times k$ cross-modal pairs from those vectors, propagates information among the pairs, and lets each original region and word re-aggregate the updated summaries through attention. The authors claim this design captures global, high-level interactions, avoids modeling uninformative low-level relations, and reduces message passing complexity from $O((M+N)^2)$ to $O(k^2(M+N))$. On VQA v2.0 the best model with BERT reaches 71.27 test-std, and on TDIUC it reaches 87.60, outperforming the DFAF baseline while using far fewer message passings.

What carries the argument

The core mechanism is the MLI module with four stages: Summarization, Interaction, Propagation, and Aggregation. Summarization uses learned softmax weights (Eqs. 2–5) to pool $M$ visual regions and $N$ question words into $k$ latent vectors per modality; Interaction forms a $k\times k\times 512$ tensor by element-wise multiplying each visual summary with each language summary (Eq. 6); Propagation applies a per-pair linear transformation and a cross-pair linear transformation over the $k^2$ pairs (Eqs. 7–9); Aggregation uses Transformer-style key-query attention to let each original region and word gather weighted information from the $k^2$ updated summaries (Eqs. 10–15). The module is residual and stackable, so the network refines features over several stages.

What would settle it

On the VQA v2.0 validation set, replace the learned summarization weights in Eqs. (2)–(3) with fixed uniform weights (pure average pooling) while keeping everything else identical; if accuracy does not drop materially, or if the learned weights have near-uniform entropy across most questions, the paper's central claim about learned latent summarizations is not supported.

Watch

Extended reading notes

Core claim

The central claim is that compressing each modality into a small number of latent summarization vectors, then performing relational reasoning on those summaries, is both more efficient and more accurate than reasoning over individual visual regions and question words. Each summarization vector is a learned softmax-weighted pooling of the modality's features, so it can represent a global aspect of the image or question, such as background context or a group of interacting objects. The MLI module builds a $k\times k$ relation tensor by element-wise multiplying visual and language summaries, applies two complementary linear transformations (one per-pair, one across pairs) to propagate cross-modal information, and finally uses key-query attention to redistribute the fused summaries back to the original features. The authors demonstrate the design through ablations and report competitive accuracy on VQA v2.0 and TDIUC, with the BERT-enhanced variant setting the best results.

Load-bearing premise

The learned softmax pooling is assumed to produce latent summaries that keep the answer-relevant visual and textual information; if the pooling weights become close to uniform or redundant, the module effectively performs average pooling and loses its claimed advantage.

Editorial extensions

If this is right

  • If the central claim holds, relational VQA models can replace dense region–word attention with a small set of latent summaries, cutting memory use from quadratic in $M+N$ to quadratic only in $k$.
  • The competitive results with fewer message passings suggest that much of the useful cross-modal signal in VQA is global and reusable, rather than tied to individual region-word pairs.
  • Stacking multiple MLI modules gives steady accuracy gains (66.04 at one stage to 66.53 at eight stages), indicating the latent interaction refinement is complementary across depth.
  • Combining MLI with a fine-tuned BERT question encoder yields the best results, showing that the fusion module and a strong language encoder contribute independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the summarization-then-interact design to other vision-and-language tasks such as image captioning or visual grounding, where the same argument about uninformative low-level pairs applies.
  • The reliance on learned softmax pooling suggests a testable risk: if the $k$ summaries degenerate toward the same or near-uniform attention, the module would collapse toward average pooling; measuring pooling entropy across validation questions would quantify this.
  • An ablation holding every component identical except replacing the learned summarization pooling with average pooling would directly test whether the learned pooling, rather than the interaction and propagation structure, is the essential ingredient.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes a Multi-modality Latent Interaction Network (MLIN) for Visual Question Answering. The core module, MLI, summarizes visual-region features and question-word features into a small number k of latent summarization vectors per modality via learned row-wise softmax pooling (Eqs. 2–5), builds all k×k cross-modal paired features (Eq. 6), applies two linear propagation operations on those pairs (Eqs. 7–9), and then lets each original region/word feature attend to the resulting k² latent interaction features with a key-query attention mechanism (Eqs. 10–15). The module is stacked with residual connections, and the final answer is produced by classifying the elementwise product of mean-pooled visual and word features (Eqs. 16–18). The method is evaluated on VQA v2.0 and TDIUC, with extensive ablations in Table 1 and state-of-the-art comparisons in Tables 2–3; the best configuration, MLIN-BERT, reports 71.27 test-std on VQA v2.0 and 87.60 on TDIUC.

Significance. If the proposed mechanism is genuine, the paper makes a practically useful contribution: it reduces cross-modal relational message passing from O((M+N)²) to O(k²(M+N)) per stage while remaining competitive with relation-based VQA models such as DFAF. The equations are clearly specified, the ablation study is unusually thorough, and the reported benchmark numbers are plausible and consistent with contemporaneous results. The BERT integration experiment is also informative. The main weakness is at the level of mechanism validation: the paper's central claim that the learned latent summarizations capture distinct, non-redundant, answer-relevant aspects is supported only by two qualitative visualizations, with no quantitative check of degeneracy and no ablation replacing the learned pooling with a fixed averaging baseline. No code is released, and all numbers are single-run, which makes the small headline gains over DFAF-BERT difficult to assess.

major comments (2)
  1. [§3.2, Eqs. (2)–(5); §4.5, Fig. 3] The central mechanism of the paper is the claim in §3.2 that each of the k latent summarization vectors 'captures a different aspect' of a modality and the claim in §4.5 that different summarization vectors have distinct functions. Equations (2)–(5) only impose row-wise softmax on learned scores, and the training signal is the final cross-entropy loss; nothing prevents L_R and L_E from collapsing to near-uniform rows, which would reduce the summarization to average pooling, or to mutually redundant rows. The only evidence offered, Figure 3, shows three attention weight maps for two examples and reports no entropy, effective rank, pairwise overlap, or any ablation that replaces the learned pooling with uniform pooling. Because the claimed advantage over relation models such as DFAF depends on the latent summarization mechanism, please add a quantitative non-degeneracy analysis and a uniform-pooling (or no-summarization) baseline. Without this, the reported accuracy could be attributable instead to the BERT encoder, the residual stacking, or the key-query aggregation stage.
  2. [§4.4, Table 2 and §4.3, Table 1] The headline comparison shows MLIN-BERT at 71.27 test-std versus DFAF-BERT at 70.81, a gain of 0.46, and the validation comparison shows 67.83 versus 67.23. No error bars, repeated runs, or significance tests are reported, and no code is released, so it is not possible to rule out run-to-run variation as the source of the reported differences. Given that the paper's main claim is that the method is competitive with or better than DFAF, please report results over multiple seeds with mean and variance, or otherwise provide a clear statement of training stochasticity; code release would also substantially strengthen the reproducibility of the comparison.
minor comments (6)
  1. [Eq. (10)] In Eq. (10), the bias terms b_qr and b_qe are declared as being in R^{512×128}; since Q_R and Q_E are M×128 and N×128 respectively, these biases should be in R^{128}. This appears to be a dimension typo.
  2. [Table 2] The rows for DFAF and DFAF-BERT are cited with reference [54], which is the 'Learning to count objects in natural images for visual question answering' paper; DFAF is reference [8]/[36]. The citation should be corrected.
  3. [§3.2] The symbols R and E are reused for both the raw input features and the summarization matrices in Eqs. (2)–(5); this reuse makes the derivation harder to follow. Please use distinct symbols for the summarization vectors, such as R' and E'.
  4. [§4.2] The sentence 'decay the learning rate 0.0005 and fix it for the following epochs' is ambiguous; state clearly whether the learning rate is decayed to 0.0005 or multiplied by a factor of 0.0005.
  5. [§4.5] The description of Figure 3, 'the colors ranging from clear to white in bounding boxes denote the attention weights from 0 to 1,' is unclear; specify the colormap and state which end corresponds to zero and which to one.
  6. [§4.1] The name 'Kafke et al.' is a typo for 'Kafle et al.' in the discussion of TDIUC.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MLI component is trained and evaluated on external VQA benchmarks, and self-citations to DFAF are used only as baselines, not as load-bearing evidence.

full rationale

The paper's central claim is that latent summarization vectors, formed by learned softmax pooling over visual and word features (Eqs. 2-5), enable efficient cross-modal interaction that improves VQA accuracy. This claim is evaluated against external benchmarks, VQA v2.0 and TDIUC, using standard train/validation/test splits and published baselines. Hyperparameters such as the number of summarization vectors, stacking depth, and attention heads are selected on the validation set, which is ordinary model selection rather than fitting the reported test result. The self-citations to DFAF appear in the related-work discussion and complexity comparison, but the DFAF accuracy numbers used in Tables 2 and 3 are external published results, and no correctness claim in the paper reduces to those citations. No equation defines the output in terms of the target answer, no fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The weakest point, that the learned softmax pooling might degenerate to near-uniform or redundant distributions, is a soundness or robustness concern about whether the claimed mechanism is responsible for the observed accuracy; it is not a circularity, because the mechanism is not assumed to be effective by construction but is rather tested empirically. Therefore the derivation chain is self-contained with respect to external evidence, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard neural network operations and benchmark assumptions. The key free parameters are hyperparameters selected on the validation set, which is normal for empirical machine learning. The ad hoc assumption about the message-passing operator is an architectural choice without independent evidence beyond the final accuracy. No new physical or conceptual entities are introduced.

free parameters (5)
  • Number of summarization vectors per modality (k) = 6
    Ablation in Table 1 compares 3x3, 6x6, 6x12, and 12x12; 6x6 is chosen as a trade-off between accuracy and computation.
  • Number of stacked MLI blocks = 8
    Ablation shows accuracy increasing from MLI-1 (66.04) to MLI-5 (66.32) and MLI-8 (66.53); no blocks beyond 8 are tested.
  • Embedding dimension = 512
    Ablation shows 1024 dimensions gives 66.18 vs 66.04 for 512, but 512 is chosen for default due to memory and speed.
  • Number of parallel attention heads in feature gathering = 12
    Ablation shows 16 heads gives 66.19 vs 66.04 for 12; 12 is chosen as default, presumably for efficiency.
  • BERT fine-tuning learning rate multiplier = 1/10 of main LR
    Ablation shows 1/10 gives 67.83, versus freezing (65.51) and 1/100 (66.99); this setting is used in the final model.
assumptions (4)
  • standard math Softmax normalization and dot-product attention are differentiable and behave as expected
    Equations (2), (12), and (13) rely on standard softmax and attention operations from the Transformer literature.
  • domain assumption Faster R-CNN object detection features provide useful region representations for VQA
    Section 4.2 uses Faster R-CNN features following Anderson et al. [1]; the method inherits their strengths and biases.
  • domain assumption VQA v2.0 and TDIUC annotations are reliable and evaluation protocols are consistent
    Accuracy numbers are reported without verification of annotation quality or any human agreement checks.
  • ad hoc to paper The linear message passing over all k^2 paired features (Eq. 8) captures higher-order cross-modal relations
    The paper asserts this models 'second-order' information, but provides no theoretical justification or ablation that isolates this operation from others.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-modality Latent Interaction Network for Visual Question Answering." pith.science (2026). https://pith.science/paper/U3K76FVN

@misc{pith2026190804289,
  author       = {Pith},
  title        = {Pith review of: Multi-modality Latent Interaction Network for Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3K76FVN}},
  note         = {Machine review of arXiv:1908.04289}
}
read the original abstract

Exploiting relationships between visual regions and question words have achieved great success in learning multi-modality features for Visual Question Answering (VQA). However, we argue that existing methods mostly model relations between individual visual regions and words, which are not enough to correctly answer the question. From humans' perspective, answering a visual question requires understanding the summarizations of visual and language information. In this paper, we proposed the Multi-modality Latent Interaction module (MLI) to tackle this problem. The proposed module learns the cross-modality relationships between latent visual and language summarizations, which summarize visual regions and question into a small number of latent representations to avoid modeling uninformative individual region-word relations. The cross-modality information between the latent summarizations are propagated to fuse valuable information from both modalities and are used to update the visual and word features. Such MLI modules can be stacked for several stages to model complex and latent relations between the two modalities and achieves highly competitive performance on public VQA benchmarks, VQA v2.0 and TDIUC . In addition, we show that the performance of our methods could be significantly improved by combining with pre-trained language model BERT.

Figures

Figures reproduced from arXiv: 1908.04289 by the authors.

Figure 1
Figure 1. Illustration of the information flow in our pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of our proposed stack Multi-modality Latent Interaction Network. Multi-modality reasoning is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We visualize the first three visual attention weights for creating visual summarization vectors. Bounding boxes [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LXMERT: Learning Cross-Modality Encoder Representations from Transformers

    cs.CL 2019-08 accept novelty 7.0 of 10

    LXMERT pretrains a three-encoder Transformer on image-sentence pairs with five tasks and achieves state-of-the-art VQA, GQA, and NLVR2 results after fine-tuning.

Reference graph

Works this paper leans on

56 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [54]

    Learning to count objects in natural images for visual ques- tion answering

    Yan Zhang, Jonathon Hare, and Adam Pr ¨ugel-Bennett. Learning to count objects in natural images for visual ques- tion answering. arXiv preprint arXiv:1802.05766, 2018. 7, 8

    arXiv 2018

  2. [36]

    Dynamic fusion with intra-and inter-modality attention flow for visual ques- tion answering

    Gao Peng, Hongsheng Li, Haoxuan You, Zhengkai Jiang, Pan Lu, Steven Hoi, and Xiaogang Wang. Dynamic fusion with intra-and inter-modality attention flow for visual ques- tion answering. arXiv preprint arXiv:1812.05252, 2018. 1, 2, 3, 5, 6, 8

    arXiv 2018

  3. [8]

    Dy- namic fusion with intra-and inter-modality attention flow for visual question answering

    Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dy- namic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 6639–6648, 2019. 5

    work page 2019

  4. [1]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018. 1, 2, 3, 5, 6, 7

    work page 2018

  5. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1, 6

    work page 2015

  6. [3]

    Mutan: Multimodal tucker fusion for visual question answering

    Hedi Ben-Younes, R ´emi Cadene, Matthieu Cord, and Nico- las Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE Interna- tional Conference on Computer Vision , pages 2612–2620,

    work page

  7. [4]

    Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning

    Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition , pages 5659–5667,

  8. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009. 2

    work page 2009

Show all 56 references
  1. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1, 2, 6, 7

    2018 arXiv

  2. [7]

    Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding

    Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding. arXiv preprint arXiv:1606.01847, 2016. 1, 8

    2016 arXiv

  3. [9]

    Question-guided hy- brid convolution for visual question answering

    Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang. Question-guided hy- brid convolution for visual question answering. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 469–485, 2018. 3

    2018

  4. [10]

    Compact bilinear pooling

    Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 317–326, 2016. 1

    2016

  5. [11]

    2nd place solution to the gqa challenge

    Shijie Geng, Ji Zhang, Hang Zhang, Ahmed Elgammal, and Dimitris N Metaxas. 2nd place solution to the gqa challenge

  6. [12]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2, 7

    2017

  7. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1, 2, 7

    2016

  8. [14]

    Relation networks for object detection

    Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3588–3597, 2018. 1, 2

    2018

  9. [15]

    Weakly-supervised compositional featureaggregation for few-shot recognition

    Ping Hu, Ximeng Sun, Kate Saenko, and Stan Sclaroff. Weakly-supervised compositional featureaggregation for few-shot recognition. arXiv preprint arXiv:1906.04833 ,

    1906 arXiv

  10. [16]

    Learning to segment every thing

    Ronghang Hu, Piotr Doll ´ar, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4233–4241, 2018. 3

    2018

  11. [17]

    Densely connected convolutional net- works

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- ian Q Weinberger. Densely connected convolutional net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017. 1, 2

    2017

  12. [18]

    Video object detection with locally-weighted deformable neighbors

    Zhengkai Jiang, Peng Gao, Chaoxu Guo, Qian Zhang, Shim- ing Xiang, and Chunhong Pan. Video object detection with locally-weighted deformable neighbors. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):8529– 8536, Jul. 2019. 3

    2019

  13. [19]

    Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recogn...

    2017

  14. [20]

    An analysis of visual question answering algorithms

    Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE International Conference on Computer Vision, pages 1965– 1973, 2017. 1, 2, 6

    1965

  15. [21]

    Bilin- ear attention networks

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilin- ear attention networks. In Advances in Neural Information Processing Systems, pages 1571–1581, 2018. 2, 3, 5, 6, 7

    2018

  16. [22]

    Hadamard product for low-rank bilinear pooling

    Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325, 2016. 1

    2016 arXiv

  17. [23]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  18. [24]

    Skip-thought vectors

    Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302, 2015. 2, 6

    2015

  19. [25]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. In Advances in neural information processing sys- tems, pages 1097–1105, 2012. 1, 2

    2012

  20. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014. 6

    2014

  21. [27]

    Improving referring expression grounding with cross-modal attention-guided erasing

    Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring expression grounding with cross-modal attention-guided erasing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1950–1959, 2019. 3

    1950

  22. [28]

    Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. 2

    1908 arXiv

  23. [29]

    Hierarchical question-image co-attention for visual question answering

    Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pages 289–297, 2016. 1, 2, 5

    2016

  24. [30]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013. 2, 7

    2013

  25. [31]

    Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering

    Duy-Kien Nguyen and Takayuki Okatani. Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6087–6096, 2018. 2, 7

    2018

  26. [32]

    Training recurrent an- swering units with joint loss minimization for vqa

    Hyeonwoo Noh and Bohyung Han. Training recurrent an- swering units with joint loss minimization for vqa. arXiv preprint arXiv:1606.03647, 2016. 8

    2016 arXiv

  27. [33]

    Im- age question answering using convolutional neural network with dynamic parameter prediction

    Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han. Im- age question answering using convolutional neural network with dynamic parameter prediction. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 30–38, 2016. 3

    2016

  28. [34]

    Learning conditioned graph structures for interpretable vi- sual question answering

    Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable vi- sual question answering. In Advances in Neural Information Processing Systems, pages 8344–8353, 2018. 7

    2018

  29. [35]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 6

    2017

  30. [37]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Man- ning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532–1543,

    2014

  31. [38]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018. 3

    2018

  32. [39]

    Deep contextualized word representations

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2018

  33. [40]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. 2019. 2

    2019

  34. [41]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information pro- cessing systems, pages 91–99, 2015. 2, 3, 6

    2015

  35. [42]

    A simple neural network module for relational rea- soning

    Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational rea- soning. In Advances in neural information processing sys- tems, pages 4967–4976, 2017. 2, 3, 7

    2017

  36. [43]

    Question type guided attention in visual ques- tion answering

    Yang Shi, Tommaso Furlanello, Sheng Zha, and Animashree Anandkumar. Question type guided attention in visual ques- tion answering. In Proceedings of the European Conference on Computer Vision (ECCV), pages 151–166, 2018. 8

    2018

  37. [44]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 1, 2

    2014 arXiv

  38. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. 1, 2, 3, 5, 6, 7

    2017

  39. [46]

    Non-local neural networks

    Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 7794–7803, 2018. 2

    2018

  40. [47]

    Pay less attention with lightweight and dy- namic convolutions

    Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dy- namic convolutions. arXiv preprint arXiv:1901.10430, 2019. 7

    1901 arXiv

  41. [48]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In International conference on machine learning, pages 2048–2057, 2015. 1, 2

    2015

  42. [49]

    Stacked attention networks for image question answering

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 21–29, 2016. 1, 2

    2016

  43. [50]

    Multi-modal learning with prior visual relation reasoning

    Zhuoqian Yang, Jing Yu, Chenghao Yang, Zengchang Qin, and Yue Hu. Multi-modal learning with prior visual relation reasoning. arXiv preprint arXiv:1812.09681, 2018. 7

    2018 arXiv

  44. [51]

    Explor- ing visual relationship for image captioning

    Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Explor- ing visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 684–699, 2018. 1, 2

    2018

  45. [52]

    Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering

    Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering. IEEE transactions on neural networks and learning systems, (99):1–13, 2018. 1, 7

    2018

  46. [53]

    Yin and Yang: Balancing and an- swering binary visual questions

    Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and an- swering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1

    2016

  47. [55]

    Structured attentions for visual question answering

    Chen Zhu, Yanpeng Zhao, Shuaiyi Huang, Kewei Tu, and Yi Ma. Structured attentions for visual question answering. In Proceedings of the IEEE International Conference on Com- puter Vision, pages 1291–1300, 2017. 2

    2017

  48. [2019]

    arXiv preprint arXiv:1907.06794, 2019. 1

    1907 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.