Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Multimodal Unified Attention Networks for Vision-and-Language Interactions

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single attention stage over concatenated image and text features outperforms separated co-attention models on five benchmarks.

desk verdict A simple, plausible unified-attention block that performs well on grounding and VQA, but the paper's central mechanistic claim—that unified beats sequential—is not isolated by a matched ablation. read the letter →

arxiv 1908.04107 v2 pith:UFCUGBEV submitted 2019-08-12 cs.CV

Zhou Yu , Yuhao Cui , Jun Yu , Dacheng Tao , Qi Tian This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringgroundingunifiedattentiongatedself-attentionco-attentionmultimodallearningreferringexpressioncomprehension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MUAN claims that the cleanest way to fuse an image and a sentence is to treat them as one sequence and run self-attention over the whole thing, rather than alternating between self-attention inside each modality and co-attention across them. Its unified attention block builds a single attention map whose four corners handle word-to-word, region-to-region, word-to-region, and region-to-word interactions at once, and stacking these blocks yields end-to-end models for visual question answering and visual grounding. On that recipe the paper reports state-of-the-art accuracy on VQA-v2, CLEVR, RefCOCO, RefCOCO+, and RefCOCOg, including what it identifies as the first single model to pass 71 percent on VQA-v2 test-std with standard bottom-up features. The reason to care is architectural: if correct, it shows that separate co-attention stages are unnecessary and that one unified stage plus depth is enough for strong multimodal reasoning.

What carries the argument

The unified attention (UA) block is the load-bearing component. It takes textual features $X\in\mathbb{R}^{m\times d_x}$ and visual features $Y\in\mathbb{R}^{n\times d_y}$, projects them into a common $d_z$-dimensional space with a fully-connected layer (identity for text in the implementation), concatenates them row-wise into $Z=[X;\,FC_y(Y)]$, and runs gated self-attention over $Z$. The gating mechanism uses low-rank bilinear pooling to compute masks $M_q$ and $M_k$ from $\sigma(FC_g(FC_q(Q)\odot FC_k(K)))$, reweighting queries and keys before the scaled dot-product softmax, so that individual features' salience is factored into the attention weights. A feed-forward network, residual connection, and layer normalization complete the block, and stacking $L$ such blocks forms MUAN-$L$ with the first block interleaving the modalities and later blocks refining the joined sequence.

What would settle it

Train MUAN with the learned common-space projection $FC_y$ replaced by a fixed random projection while keeping all other capacity the same; if test accuracy stays near 71 percent on VQA-v2, the learned alignment is not the mechanism. Alternatively, probe the final block's cross-modal attention on controlled VQA examples and check whether a question word's attention to the correct image region reliably exceeds its attention to same-category distractors; if not, the cross-modal attention maps are not semantically grounded.

Watch

Extended reading notes

Core claim

The central discovery is that intra-modal and inter-modal attention need not be computed by separate modules. By concatenating question features and image-region features along the sequence dimension and applying gated self-attention, the attention matrix decomposes into four blocks, $A_{TT}$, $A_{VV}$, $A_{VT}$, and $A_{TV}$, and all four are learned jointly in every block. The paper argues that prior co-attention models use only the cross-modal blocks and therefore discard contextual information carried by the intra-modal blocks; ablations that mask either the self or the cross blocks lower VQA accuracy, with the cross blocks mattering more in shallow networks. When the blocks are stacked, best at depth 10, the attended feature of the answer token progressively aggregates question words and image objects, which the authors read as a visual reasoning process.

Load-bearing premise

The load-bearing premise is that one learned linear layer is enough to put image-region features and word features in a common space where their dot products express real semantic relatedness; if the two modalities stay misaligned, the four attention sub-blocks could be encoding modality-specific priors rather than genuine vision-language interactions.

Editorial extensions

If this is right

  • On VQA-v2, MUAN-10 reports 71.10 percent test-std accuracy with the same bottom-up visual features as prior models, and it beats a dedicated counting module on number questions without using bounding-box coordinates.
  • On CLEVR, MUAN reaches 98.7 percent accuracy, comparable to the best published compositional-reasoning models, without program labels or data augmentation.
  • On RefCOCO, RefCOCO+, and RefCOCOg, MUAN improves over prior grounding models by several points under both COCO-trained and Visual Genome-trained proposal features.
  • Ablation studies show that both the intra-modal blocks and the cross-modal blocks contribute to accuracy, so the unified interaction is not carried by either alone.
  • Depth helps: accuracy rises as the number of UA blocks increases from 2 to 10 and saturates at 10, consistent with deep joint refinement rather than a single fusion layer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same unified attention block could serve as a drop-in fusion layer for other vision-and-language tasks such as captioning, visual entailment, or text-to-image retrieval, though the paper itself only demonstrates VQA and grounding.
  • Because the gating mask is computed from queries and keys before attention, the model is implicitly learning a per-feature salience signal; exposing and analyzing these masks could give an interpretability handle the paper only touches qualitatively.
  • A natural stress test would be to replace the learned common-space projection $FC_y$ with a fixed random projection; if accuracy stays high, the projection is not the source of the gains, and if it collapses, the alignment assumption is load-bearing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Multimodal Unified Attention Networks (MUAN), a stack of unified attention (UA) blocks that apply a gated self-attention operation to a sequence formed by concatenating visual and textual features. Because the attention is computed over the concatenated sequence, each attention matrix contains both intra-modal quadrants (image-to-image and word-to-word) and inter-modal quadrants (word-to-image and image-to-word), so intra- and inter-modal interactions are modeled simultaneously rather than in separate sequential stages. The authors apply MUAN to VQA-v2 and CLEVR for visual question answering and to RefCOCO, RefCOCO+, and RefCOCOg for visual grounding, reporting a VQA-v2 test-std accuracy of 71.10, CLEVR accuracy of 98.7, and strong results on the grounding datasets. They also report ablations on VQA-v2 val showing that both the gating mechanism and the presence of the four attention quadrants contribute to accuracy, that depth improves accuracy up to 10 blocks, and that the choice of dimensionality and number of heads matters. The central claim is that a single unified attention stage is better than the sequential self-attention plus co-attention stacks used in earlier models such as MCAN and DDPN.

Significance. If the central claim is substantiated, the unified attention formulation is a clean and general design: it replaces the two-stage modular stacks of MCAN and DDPN with one attention operation over concatenated modalities, which is conceptually simpler and could transfer to other vision-and-language tasks. The paper is also strong in scope: it evaluates on five established benchmarks, uses standard bottom-up features, and reports careful ablations of the gating mechanism and hyper-parameters. The qualitative attention visualizations are informative and provide some evidence that the four quadrants play distinct roles. However, the load-bearing comparative claim—that simultaneous unified attention outperforms sequential self- and co-attention—is not isolated by a controlled experiment, and the magnitude of the main VQA gain over the closest sequential baseline is small. The contribution is therefore plausible but not yet convincingly established.

major comments (3)
  1. [Section V-C, Figure 5b] The central claim that simultaneous intra- and inter-modal attention in a single block is superior to sequential self- and co-attention is never tested against a matched sequential baseline. The ablation in Fig. 5b masks quadrants of the unified attention matrix, which removes information flow but does not reproduce the architecture of MCAN or DDPN, where self-attention and guided attention are separate modules with different parameterizations, normalizations, and residual paths. To support the paper's main thesis, the authors need a baseline that replaces the UA block with a sequential self-attention + guided-attention block of matched depth, width, gating, and training schedule. Without such a control, the reported differences in Table II (e.g., 70.82 vs. 70.63 on test-dev) cannot be attributed to the unified formulation rather than to capacity, effective depth, or the gating mechanism.
  2. [Section V-C] The description of the masking ablation is internally inconsistent. The text says "By masking the values in the self-attention part (i.e., AT T and AV V ) or the co-attention part (i.e., AT T and AV V )"—the two parentheticals name the same blocks. The co-attention variant must presumably mask ATV and AVT instead. As written, the experiment is ambiguous and the reader cannot determine which quadrants were actually ablated. Please correct the notation and state the exact mask used for each variant.
  3. [Table II] The claim that MUAN "significantly outperforms" MCAN rests on a 0.19-point gap on test-std (71.10 vs. 70.90) and a 0.19-point gap on test-dev (70.82 vs. 70.63), reported without standard deviations, confidence intervals, or multiple seeds. Given that VQA test accuracies typically vary by more than 0.2 points across training runs, this gap is not sufficient to support the word "significantly." The authors should report results over at least three seeds for MUAN and for the comparison baseline under identical training conditions, or otherwise provide an appropriate significance estimate.
minor comments (5)
  1. [Section III-B, Eq. (7)] The notation is slightly confusing because X denotes both the textual feature matrix and (via Z = [X; FCy(Y)]) the concatenated input to the first UA block; please distinguish the two roles, for example by writing the textual input as X_t and the unified matrix as Z^(0).
  2. [Section IV-A] The paper does not state whether positional encodings are used inside the UA blocks. The question features likely already encode order through the LSTM, and the visual features include spatial coordinates, but this should be stated explicitly for reproducibility.
  3. [Section V-E, Table III] The abstract and introduction say MUAN achieves "top level performance" on the two VQA datasets, but on CLEVR MUAN (98.7) is below MAC (98.9). The conclusion phrase "at least comparable" is appropriate; please adjust the wording in the abstract and contributions to avoid overstating the CLEVR result.
  4. [Section V-C, Figure 5] In Fig. 5b, the caption and the text discuss results for L in {2, 6, 10}, but the figure shows curves over a wider range; please make the range of L consistent between the text and the figure.
  5. [Section V-C, Table I] Row (A) of Table I varies d_g, but the three variants report accuracies within 0.12 points of the reference, which is within the likely run-to-run variance given that no multiple seeds are reported; please temper the claim that the reference setting is clearly better.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MUAN's benchmark performance is an empirical result evaluated on held-out test splits, not a quantity derived from its own fitted inputs.

full rationale

This is an empirical systems paper: MUAN is trained with standard losses (BCE/softmax for VQA, ranking plus regression for grounding) on public benchmarks and evaluated on held-out test splits. The central claim is architectural — that a unified attention block jointly captures intra- and inter-modal interactions — and it is supported by validation-set ablations in Section V-C and benchmark comparisons in Tables II–IV. There is no fitted parameter later relabeled as a prediction: hyperparameters are selected on the validation split, and the reported test-std/test numbers come from the respective evaluation servers or dataset splits. Self-citations are present ([39], [61]), but they are prior baselines (DDPN, MCAN) rather than the source of MUAN's correctness; nothing in the architecture is justified by invoking a uniqueness theorem or a self-cited theoretical guarantee. The ablation text has an internal typo (Section V-C says both masked variants remove 'ATT and AVV'), and the lack of a matched sequential-baseline ablation weakens the comparative claim, but that is a correctness/experimental-design risk, not circularity. No equation in the paper reduces to its own inputs by construction, and no external benchmark result is imported from the authors' own prior work as evidence for the present model. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The architecture introduces no new physical or mathematical entities. The gated self-attention block is a learned function, not an invented entity. The main free parameters are model hyperparameters chosen on validation sets. The axioms are standard deep-learning assumptions: the transformer inductive bias, the sufficiency of pre-extracted object features, and the common-space linear projection. These assumptions are not derivations, but they are tested indirectly by the benchmark results.

free parameters (8)
  • latent dimensionality d = 768
    Hidden size of each UA block; varied in Table I row C, 768 chosen as trade-off between accuracy and cost.
  • number of attention heads h = 8
    Number of parallel heads; varied in Table I row B, h=8 best.
  • gating dimensionality d_g = 96
    Latent size of the low-rank bilinear gating; varied in Table I row A, effect is small (67.16 to 67.28).
  • number of UA blocks L = 10
    Depth of MUAN-L; accuracy saturates at L=10 in Fig. 5.
  • grounding loss weight lambda = 0.5
    Balances ranking and regression losses; no ablation supplied.
  • learning rate schedule = 1.5e-2/sqrt(dL) for VQA/grounding, 3.5e-3/sqrt(dL) for CLEVR
    Chosen per task; warmup and decay as described in Section V-B.
  • max question length m = 14 (VQA-v2), 43 (CLEVR), 15 (grounding)
    Truncation/padding length per dataset.
  • number of proposals n = 100 per image for grounding; 10-100 for VQA with confidence threshold
    Detection confidence threshold controls proposal count.
assumptions (5)
  • domain assumption Scaled dot-product attention is a sufficient mechanism for relating tokens within and across modalities.
    The whole model is a stack of gated self-attention blocks; no proof, supported only by empirical results.
  • domain assumption A single linear projection into a common space aligns the visual and textual feature distributions (Eq. 7).
    X and FCy(Y) are concatenated and compared with dot-product attention; if the projection were insufficient, cross-modal attention would be uninformative.
  • domain assumption Pre-trained object detector features (Faster R-CNN on Visual Genome, Mask R-CNN on COCO) provide sufficient representations for both tasks.
    The model never sees raw pixels; performance is bounded by the quality of region proposals and features.
  • ad hoc to paper The dummy [ans] token's final attended feature is sufficient to predict the VQA answer.
    This token is prepended to the question in VQA; the classifier receives only its output feature.
  • standard math Softmax, layer normalization, residual connections, and backpropagation are reliable building blocks.
    Standard deep learning background assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Unified Attention Networks for Vision-and-Language Interactions." pith.science (2026). https://pith.science/paper/UFCUGBEV

@misc{pith2026190804107,
  author       = {Pith},
  title        = {Pith review of: Multimodal Unified Attention Networks for Vision-and-Language Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFCUGBEV}},
  note         = {Machine review of arXiv:1908.04107}
}
read the original abstract

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.

Figures

Figures reproduced from arXiv: 1908.04107 by the authors.

Figure 1
Figure 1. Schematic of the proposed unified attention, which [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Flowcharts of the Gated Self-Attention (GSA) model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architectures of the Multimodal Unified Attention Networks (MUAN) for visual question answer (left) and visual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Typical examples from VQA-v2, CLEVR, RefCOCO, RefCOCO+, and RefCOCOg. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Ablation of the MUAN models with the number of UA [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualizations of the learned unified attention maps (Eq.(5)) for VQA. The attention maps come from the 1st, 3rd, 6th [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Visualizations of the prediction and the learned visual attention for visual grounding. The groundtruth (red), top-ranked [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LXMERT: Learning Cross-Modality Encoder Representations from Transformers

    cs.CL 2019-08 accept novelty 7.0 of 10

    LXMERT pretrains a three-encoder Transformer on image-sentence pairs with five tasks and achieves state-of-the-art VQA, GQA, and NLVR2 results after fine-tuning.

Reference graph

Works this paper leans on

70 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Multimodal deep network embedding with integrated structure and attribute information,

    C. Zheng, L. Pan, and P. Wu, “Multimodal deep network embedding with integrated structure and attribute information,” IEEE transactions on neural networks and learning systems , 2019

    work page 2019

  2. [2]

    Discrim- inative coupled dictionary hashing for fast cross-media retrieval,

    Z. Yu, F. Wu, Y . Yang, Q. Tian, J. Luo, and Y . Zhuang, “Discrim- inative coupled dictionary hashing for fast cross-media retrieval,” in International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , 2014, pp. 395–404

    work page 2014

  3. [3]

    Shared predictive cross-modal deep quantization,

    E. Yang, C. Deng, C. Li, W. Liu, J. Li, and D. Tao, “Shared predictive cross-modal deep quantization,” IEEE transactions on neural networks and learning systems , vol. 29, no. 11, pp. 5292–5303, 2018

    work page 2018

  4. [4]

    Show, attend and tell: Neural image caption generation with visual attention

    K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention.” in International Conference on Machine Learning (ICML) , vol. 14, 2015, pp. 77–81. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 10

    work page 2015

  5. [5]

    From deterministic to generative: multi-modal stochastic rnns for video cap- tioning,

    J. Song, Y . Guo, L. Gao, X. Li, A. Hanjalic, and H. T. Shen, “From deterministic to generative: multi-modal stochastic rnns for video cap- tioning,” IEEE transactions on neural networks and learning systems , 2018

    work page 2018

  6. [6]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “Vqa: Visual question answering,” in IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 2425–2433

    work page 2015

  7. [7]

    Ground- ing of textual phrases in images by reconstruction,

    A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele, “Ground- ing of textual phrases in images by reconstruction,” in European Conference on Computer Vision (ECCV) , 2016, pp. 817–834

    work page 2016

  8. [8]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y . Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

Show all 70 references
  1. [9]

    Recurrent models of visual attention,

    V . Mnih, N. Heess, A. Graves et al. , “Recurrent models of visual attention,” in NIPS, 2014, pp. 2204–2212

    2014

  2. [10]

    Draw: A recurrent neural network for image generation,

    K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, “Draw: A recurrent neural network for image generation,” in Interna- tional Conference on Machine Learning (ICML) , 2015, pp. 1462–1471

    2015

  3. [11]

    Attention to scale: Scale-aware semantic image segmentation,

    L.-C. Chen, Y . Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 3640– 3649

    2016

  4. [12]

    Effective ap- proaches to attention-based neural machine translation,

    M.-T. Luong, H. Pham, and C. D. Manning, “Effective ap- proaches to attention-based neural machine translation,” arXiv preprint arXiv:1508.04025, 2015

    2015 arXiv

  5. [13]

    Deep biaffine attention for neural dependency parsing,

    T. Dozat and C. D. Manning, “Deep biaffine attention for neural dependency parsing,” in International Conference on Learning Representations (ICLR), 2017. [Online]. Available: https://nlp.stanford. edu/pubs/dozat2017deep.pdf

    2017

  6. [14]

    A neural attention model for abstractive sentence summarization,

    A. M. Rush, S. Chopra, and J. Weston, “A neural attention model for abstractive sentence summarization,” arXiv preprint arXiv:1509.00685 , 2015

    2015 arXiv

  7. [15]

    Stacked attention net- works for image question answering,

    Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, “Stacked attention net- works for image question answering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 21–29

    2016

  8. [16]

    Multimodal compact bilinear pooling for visual question answering and visual grounding,

    A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach, “Multimodal compact bilinear pooling for visual question answering and visual grounding,” Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2016

    2016

  9. [17]

    Hierarchical question-image co-attention for visual question answering,

    J. Lu, J. Yang, D. Batra, and D. Parikh, “Hierarchical question-image co-attention for visual question answering,” in NIPS, 2016, pp. 289–297

    2016

  10. [18]

    Multi-modal factorized bilinear pooling with co-attention learning for visual question answering,

    Z. Yu, J. Yu, J. Fan, and D. Tao, “Multi-modal factorized bilinear pooling with co-attention learning for visual question answering,” IEEE International Conference on Computer Vision (ICCV) , pp. 1839–1848, 2017

    2017

  11. [19]

    Bilinear attention networks,

    J.-H. Kim, J. Jun, and B.-T. Zhang, “Bilinear attention networks,” NIPS, 2018

    2018

  12. [20]

    Improved fusion of visual and language representations by dense symmetric co-attention for visual question an- swering,

    D.-K. Nguyen and T. Okatani, “Improved fusion of visual and language representations by dense symmetric co-attention for visual question an- swering,”IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  13. [21]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 6000–6010

    2017

  14. [22]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  15. [23]

    Non-local neural net- works,

    X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural net- works,” in IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2018, pp. 7794–7803

    2018

  16. [24]

    Relation networks for object detection,

    H. Hu, J. Gu, Z. Zhang, J. Dai, and Y . Wei, “Relation networks for object detection,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  17. [25]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  18. [26]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,

    J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick, “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 2901–2910

    2017

  19. [27]

    Referitgame: Referring to objects in photographs of natural scenes,

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 787–798

    2014

  20. [28]

    Generation and comprehension of unambiguous object descriptions,

    J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in IEEE International Conference on Computer Vision (ICCV) , 2016, pp. 11–20

    2016

  21. [29]

    Simple base- line for visual question answering,

    B. Zhou, Y . Tian, S. Sukhbaatar, A. Szlam, and R. Fergus, “Simple base- line for visual question answering,” arXiv preprint arXiv:1512.02167 , 2015

    2015 arXiv

  22. [30]

    Hadamard Product for Low-rank Bilinear Pooling,

    J.-H. Kim, K. W. On, W. Lim, J. Kim, J.-W. Ha, and B.-T. Zhang, “Hadamard Product for Low-rank Bilinear Pooling,” in International Conference on Learning Representation (ICLR) , 2017

    2017

  23. [31]

    Mutan: Multi- modal tucker fusion for visual question answering,

    H. Ben-Younes, R. Cadene, M. Cord, and N. Thome, “Mutan: Multi- modal tucker fusion for visual question answering,” in IEEE Interna- tional Conference on Computer Vision (ICCV) , 2017

    2017

  24. [32]

    Abc- cnn: An attention based convolutional neural network for visual question answering,

    K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and R. Nevatia, “Abc- cnn: An attention based convolutional neural network for visual question answering,” arXiv preprint arXiv:1511.05960 , 2015

    2015 arXiv

  25. [33]

    A focused dynamic attention model for visual question answering,

    I. Ilievski, S. Yan, and J. Feng, “A focused dynamic attention model for visual question answering,” arXiv preprint arXiv:1604.01485 , 2016

    2016 arXiv

  26. [34]

    Where to look: Focus regions for visual question answering,

    K. J. Shih, S. Singh, and D. Hoiem, “Where to look: Focus regions for visual question answering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 4613–4621

    2016

  27. [35]

    Beyond bilinear: Generalized multi-modal factorized high-order pooling for visual question answer- ing,

    Z. Yu, J. Yu, C. Xiang, J. Fan, and D. Tao, “Beyond bilinear: Generalized multi-modal factorized high-order pooling for visual question answer- ing,” IEEE Transactions on Neural Networks and Learning Systems , 2018

    2018

  28. [36]

    A joint speaker-listener- reinforcer model for referring expressions,

    L. Yu, H. Tan, M. Bansal, and T. L. Berg, “A joint speaker-listener- reinforcer model for referring expressions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 7282–7290

    2017

  29. [37]

    Edge boxes: Locating object proposals from edges,

    C. L. Zitnick and P. Doll ´ar, “Edge boxes: Locating object proposals from edges,” in European Conference on Computer Vision (ECCV), 2014, pp. 391–405

    2014

  30. [38]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  31. [39]

    Rethinking diversified and discriminative proposal generation for visual grounding,

    Z. Yu, J. Yu, C. Xiang, Z. Zhao, Q. Tian, and D. Tao, “Rethinking diversified and discriminative proposal generation for visual grounding,” International Joint Conference on Artificial Intelligence (IJCAI) , 2018

    2018

  32. [40]

    Mattnet: Modular attention network for referring expression comprehension,

    L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg, “Mattnet: Modular attention network for referring expression comprehension,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 1307–1315

    2018

  33. [41]

    Parallel attention: A unified framework for visual object discovery through dialogs and queries,

    B. Zhuang, Q. Wu, C. Shen, I. Reid, and A. van den Hengel, “Parallel attention: A unified framework for visual object discovery through dialogs and queries,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4252–4261

    2018

  34. [42]

    Visual grounding via accumulated attention,

    C. Deng, Q. Wu, Q. Wu, F. Hu, F. Lyu, and M. Tan, “Visual grounding via accumulated attention,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7746–7755

    2018

  35. [43]

    Be- yond rnns: Positional self-attention with co-attention for video question answering,

    X. Li, J. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan, “Be- yond rnns: Positional self-attention with co-attention for video question answering,” in AAAI, 2019

    2019

  36. [44]

    Dynamic fusion with intra-and inter-modality attention flow for visual question answering,

    G. Peng, H. Li, H. You, Z. Jiang, P. Lu, S. Hoi, and X. Wang, “Dynamic fusion with intra-and inter-modality attention flow for visual question answering,” arXiv preprint arXiv:1812.05252 , 2018

    2018 arXiv

  37. [45]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” URL https://s3-us-west-2. amazonaws. com/openai-assets/research- covers/languageunsupervised/language understanding paper. pdf, 2018

    2018

  38. [46]

    Factorized bilinear models for image recognition,

    Y . Li, N. Wang, J. Liu, and X. Hou, “Factorized bilinear models for image recognition,” IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  39. [47]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  40. [48]

    Glove: Global vectors for word representation

    J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation.” in Conference on Empirical Methods in Natural Language Processing (EMNLP) , vol. 14, 2014, pp. 1532–1543

    2014

  41. [49]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  42. [50]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  43. [51]

    Tips and tricks for visual question answering: Learnings from the 2017 challenge,

    D. Teney, P. Anderson, X. He, and A. v. d. Hengel, “Tips and tricks for visual question answering: Learnings from the 2017 challenge,” arXiv preprint arXiv:1708.02711, 2017. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 11

    2017 arXiv

  44. [52]

    Fast r-cnn,

    R. Girshick, “Fast r-cnn,” in IEEE International Conference on Com- puter Vision (ICCV) , 2015, pp. 1440–1448

    2015

  45. [53]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision (ECCV) , 2014, pp. 740–755

    2014

  46. [54]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  47. [55]

    Faster r-cnn: Towards real- time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real- time object detection with region proposal networks,” in NIPS, 2015, pp. 91–99

    2015

  48. [56]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shammaet al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” arXiv preprint arXiv:1602.07332 , 2016

    2016 arXiv

  49. [57]

    Compositional attention networks for machine reasoning,

    D. A. Hudson and C. D. Manning, “Compositional attention networks for machine reasoning,” arXiv preprint arXiv:1803.03067 , 2018

    2018 arXiv

  50. [58]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask r-cnn,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2961–2969

    2017

  51. [59]

    Modeling context in referring expressions,

    L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” in European Conference on Computer Vision (ECCV). Springer, 2016, pp. 69–85

    2016

  52. [60]

    Learning to count objects in natural images for visual question answering,

    Y . Zhang, J. Hare, and A. Pr ¨ugel-Bennett, “Learning to count objects in natural images for visual question answering,” International Conference on Learning Representation (ICLR) , 2018

    2018

  53. [61]

    Deep modular co- attention networks for visual question answering,

    Z. Yu, J. Yu, Y . Cui, D. Tao, and Q. Tian, “Deep modular co- attention networks for visual question answering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 6281– 6290

    2019

  54. [62]

    Learning to reason: End-to-end module networks for visual question answering,

    R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko, “Learning to reason: End-to-end module networks for visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  55. [63]

    A simple neural network module for relational reasoning,

    A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, “A simple neural network module for relational reasoning,” in Advances in neural information processing systems, 2017, pp. 4967–4976

    2017

  56. [64]

    Inferring and executing programs for visual reasoning,

    J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick, “Inferring and executing programs for visual reasoning,” in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2989–2998

    2017

  57. [65]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Thirty-Second AAAI Conference on Artificial Intelligence , 2018

    2018

  58. [66]

    Ssd: Single shot multibox detector,

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y . Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European Conference on Computer Vision (ECCV) . Springer, 2016, pp. 21–37

    2016

  59. [67]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    2014 arXiv

  60. [68]

    Referring expression generation and comprehension via attributes,

    J. Liu, L. Wang, and M.-H. Yang, “Referring expression generation and comprehension via attributes,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 4856–4864

    2017

  61. [69]

    Modeling relationships in referential expressions with compositional modular networks,

    R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko, “Modeling relationships in referential expressions with compositional modular networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1115–1124

    2017

  62. [70]

    Grounding referring expressions in images by variational context,

    H. Zhang, Y . Niu, and S.-F. Chang, “Grounding referring expressions in images by variational context,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.