Pith. sign in

REVIEW 3 major objections 4 minor 26 references

Question-Agnostic Attention for Visual Question Answering

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a fixed, question-agnostic binary object map, applied as a mask to convolutional visual features and combined with any VQA model, improves accuracy and lifts simple fusion models to near state-of-the-art levels.

desk verdict A simple and portable object-map pre-processing for VQA that shows real gains on simple fusion models, but the headline causal claim is under-supported by a missing two-branch control. read the letter →

arxiv 1908.03289 v2 pith:QP2OG3GZ submitted 2019-08-09 cs.CV

Moshiur R Farazi , Salman H Khan , Nick Barnes This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringquestion-agnosticattentionobjectmapinstancesegmentationmultimodalfusionspatialcenterpriorraretypes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a Visual Question Answering system can compute a useful attention map without ever looking at the question. Its proposed Question-Agnostic Attention (QAA) runs an instance-segmentation network once per image, builds a binary map of where objects are, and masks the convolutional feature map with that map before fusion. Because the mask is fixed and lightweight, it can be added as a preprocessing step to almost any existing VQA model at nearly no extra training cost. The paper reports consistent accuracy gains across VQAv1, VQAv2, and TDIUC, with the largest gains (roughly +18 points on simple linear-sum models) bringing those simple models up to the level of much more expensive tensor-fusion models. If the claim holds, object-location priors carry a large share of what learned question-dependent attention is usually credited with.

What carries the argument

The load-bearing mechanism is the binary object map $M \in \mathbb{R}^g$, generated by an off-the-shelf instance-segmentation network and aligned one-to-one with the $g$ spatial cells of the CNN feature map. Applying it as an element-wise mask produces question-agnostic features $\mathbf{v}_M = \mathbf{v} \odot M$, which select object-occupied locations without any ROI pooling or learned attention. The other component is the Multiple Prediction Embedding: predictions from the QAA branch and from any existing VQA model are concatenated and passed through a learned layer to produce a final answer prediction, which is how QAA is combined with the baselines in the experiments.

What would settle it

Train the same two-branch architecture with the second branch fed unmasked spatial-grid features instead of object-masked ones, and separately with a random binary mask of the same occupancy density; if either matches the reported accuracy of the object-mask model on VQAv2 validation, the object map itself contributes nothing.

Watch

Extended reading notes

Core claim

The paper's central claim is that object locations alone are a powerful, complementary attention signal for VQA. The object map is a binary grid $M \in \mathbb{R}^g$ marking which coarse spatial cells contain detected object instances; multiplying the CNN feature map by this mask yields QAA features that a VQA model can fuse with the original visual features. Empirically, on the VQAv1 validation set a linear-sum model jumps from 39.7 to 57.9 accuracy when a QAA branch is added, reaching the level of the strongest tested tensor-decomposition fusion, while smaller but consistent gains appear for the stronger fusion baselines and for difficult question types measured by Harmonic MPT on TDIUC. The paper further shows that an image- and question-independent global map, built by thresholding the training-set count of object presence per grid cell, still yields competitive accuracy, revealing a strong center bias in object locations. The authors conclude that question-agnostic object location information is complementary to learned question-aware attention and can be supplied at almost no training cost.

Load-bearing premise

The claimed gains come from the object map itself rather than from the extra branch and learned prediction-fusion layer used in the combined model, since the paper never tests a control with a second spatial-grid branch that has no object mask.

Editorial extensions

If this is right

  • On the paper's evidence, any CNN-based VQA model can be upgraded by adding a QAA pre-processing branch, with only a small increase in parameters and training cost.
  • A linear-sum or concatenation-MLP model with QAA approaches or matches the accuracy of much more parameter-heavy tensor-fusion models on VQAv1 and VQAv2 validation sets.
  • QAA improves performance on rare and reasoning-heavy question types, as shown by higher Harmonic MPT and normalized MPT scores on TDIUC.
  • Even a fixed, dataset-level center-biased object map (IQAA) gives competitive VQA accuracy, suggesting that model capacity spent on learning attention could be partly redirected.
  • When used with object-proposal features, QAA still gives a gain, though smaller, mainly on counting questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if the object map itself causes the gains, then the expensive question-dependent attention modules in current VQA systems may be over-engineered for object localization; a cheap segmentation prior could substitute for part of them.
  • My inference: the strong IQAA center-prior result predicts that performance will degrade on deliberately off-center object images, which would be a clean out-of-distribution test of the mechanism.
  • My inference: the same binary object-map masking could be transplanted to other vision-and-language tasks, such as image captioning or referring expression comprehension, where object locations matter independently of the text.
  • My inference: to isolate the mechanism, one could compare the binary object map against a random mask with the same occupancy density and against a confidence-weighted segmentation map; equal gains from random masks would falsify the object-semantics story.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Question-Agnostic Attention (QAA), a fixed object map derived from Mask R-CNN instance segmentation, which is applied as a binary mask to ResNet spatial-grid features before multimodal fusion. The masked branch is added alongside the original visual branch, and the two prediction vectors are combined through a learned Multiple Prediction Embedding. Experiments on VQAv1, VQAv2, and TDIUC report that this complementary branch gives large gains for linear and concatenation-based fusion models, smaller gains for Mutan and Block, and that even a fixed center-weighted mask (IQAA) helps. The paper claims that the object map itself, not question-specific training or added capacity, is responsible for the improvement.

Significance. If the causal claim holds, the contribution is practically valuable: any CNN-based VQA model could be improved by a cheap, task-agnostic object prior, and simple fusion models could approach the accuracy of more expensive ones. The IQAA center-prior experiment is a compelling, falsifiable observation, and the paper's modular design is a strength. However, the main attribution is not isolated by the reported experiments: the Table I comparisons confound the object map with an extra branch and a learned prediction-embedding layer, and the validation-set object maps are produced by a detector that has seen a large fraction of the VQA validation images. These issues are fixable with additional controls, but they currently prevent the paper from establishing its central claim.

major comments (3)
  1. [Sec. III-B, Table I] Table I compares row (1), Spatial Grid (SG), with row (3), Ours(SG+QAA), and similarly rows (4) with (6). In both comparisons, the baseline is a single-branch model, while Ours(SG+QAA) is a two-branch model whose prediction vectors are concatenated and re-embedded by the Multiple Prediction Embedding described in Sec. III-B. The linear-fusion gain of 18.2 points on VQAv1 therefore changes two things at once: it adds a second visual branch and it adds a learned fusion-of-predictions layer. A control with two spatial-grid branches (SG+SG) under the same Multiple Prediction Embedding is necessary to attribute the gain to the object map; without it, the reported improvement is consistent with the extra model capacity alone. This control is load-bearing for the abstract's claim that QAA, rather than the wider architecture, produces the boost.
  2. [Sec. IV, Instance Segmentation] The instance-segmentation paragraph states that Mask R-CNN was trained on 'COCO train and the val-minus-minival split.' The paper's main ablations in Table I and Fig. 3 are on VQAv1 and VQAv2 validation sets, whose images are sourced from COCO val2014. Because val-minus-minival is a large subset of COCO val2014, the object maps used in these central validation experiments come from a segmentation model that has seen those exact images during training. The paper's reassurance that 'none of the test images have been previously seen' therefore does not cover the validation set used for the headline results. This is a data-hygiene issue that could inflate QAA's apparent benefit; it should be addressed by re-running the ablation with a detector trained only on COCO train2014, or by reporting the corresponding test-dev numbers.
  3. [Sec. IV-A, Tables I and III] The text claims a 'consistent boost for all fusion mechanisms' (Introduction and Sec. IV-A), but for the sophisticated fusion models the gains in Table I are only 0.2–0.6 points, and Table III shows that several TDIUC categories decrease for the Mutan and Block variants, for example Color Attributes falls from 68.6 to 64.5 and Sentiment Understanding from 66.0 to 63.5 for the Block variant. No variance or significance information is reported, so gains of this size are not distinguished from training noise. At minimum, the paper should report multiple seeds and per-category results that allow the reader to verify the 'all cases' claim.
minor comments (4)
  1. [Sec. III-B, Eq. (2)] In Eq. (2), the spatial attention weight is written as alpha_i = softmax(Psi(q, v_i)), but the text describes the similarity between the question and each question-agnostic feature grid location v_i^M; please correct the notation to v_i^M for consistency.
  2. [Throughout] There are several typos: 'pre-prcoessing' in Sec. III, 'Acuuracy' in the y-axis label of Fig. 3, 'TUDIC' in the Model Architecture paragraph of Sec. IV, 'liner summation' in Sec. IV, and 'preform' in the Conclusion.
  3. [Table III] The header of Table III is difficult to parse because baseline and 'Ours' columns are interleaved without clear separators; please restructure the header so each baseline and its QAA variant are explicitly labeled.
  4. [Code availability] No code or pretrained QAA models are released; given the paper's stated goal of being a generic light-weight pre-processing step, releasing the object-map generation and the Multiple Prediction Embedding implementation would substantially aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QAA is an externally evaluated preprocessing method, and the paper's benchmark comparisons are self-contained.

full rationale

The paper makes an empirical claim that a question-agnostic object map derived from instance segmentation improves VQA accuracy when used as complementary attention. This claim is tested against fixed external benchmarks (VQAv1, VQAv2, TDIUC) with reported accuracies; it does not derive a result from a fitted parameter that is then renamed as a prediction. The object map itself is generated by Mask R-CNN trained on COCO, independent of the VQA training objective, and is applied as a fixed binary mask. The Multiple Prediction Embedding module is an architectural component, and while the paper lacks an SG+SG control, that is a possible confound in attribution rather than a circularity: no equation or fitted value reduces the reported gains to the method's own inputs. The IQAA threshold sweep in Sec. IV-B is an analysis of a global center prior, not a fitted parameter used to produce the central QAA results. Self-citations [6] and [10] appear only in related-work context and are not load-bearing for the proposed method or its evaluation. No self-definitional step, renamed known result, or imported uniqueness claim is present. The derivation chain is therefore self-contained with respect to circularity, and any concerns about the causal role of the object map belong to experimental control, not circular reasoning.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the transferability of instance segmentation to VQA, on the object-centric hypothesis about questions, and on an untested assumption that the observed gains are not produced by the extra prediction branch. The only explicit numeric free parameter is the IQAA threshold; other hyperparameters such as fusion-layer sizes are not reported.

free parameters (1)
  • IQAA global representation threshold = swept over 0.1 to 0.9; optimal value not stated
    In Sec. IV-B and Fig. 3, the threshold selects which spatial grid locations form the fixed object map. The accuracy curve is used to compare thresholds, so the choice is data-dependent.
assumptions (4)
  • domain assumption Object instances in an image provide the visual cues needed to answer most VQA questions.
    Sec. I motivates QAA from the assumption that questions relate to the state, number, type, actions of objects and their mutual relationships.
  • domain assumption A Mask-RCNN trained on COCO produces object masks that transfer to VQA images and have not seen VQA test images.
    Sec. III-A and Sec. IV invoke a pretrained Mask-RCNN to generate the object map; if the segmentation fails or has dataset bias, the attention mask is wrong.
  • ad hoc to paper The accuracy gain of SG+QAA is caused by the object map rather than by the second branch and learned prediction embedding.
    The Multiple Prediction Embedding in Sec. III-B adds a parallel prediction path; the paper does not report an SG+SG control, so this causal attribution is an unverified premise.
  • domain assumption The center bias measured on training images generalizes to test images.
    Sec. IV-B constructs IQAA from a global object-presence prior over training data and assumes the prior holds at inference time.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Question-Agnostic Attention for Visual Question Answering." pith.science (2026). https://pith.science/paper/QP2OG3GZ

@misc{pith2026190803289,
  author       = {Pith},
  title        = {Pith review of: Question-Agnostic Attention for Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QP2OG3GZ}},
  note         = {Machine review of arXiv:1908.03289}
}
read the original abstract

Visual Question Answering (VQA) models employ attention mechanisms to discover image locations that are most relevant for answering a specific question. For this purpose, several multimodal fusion strategies have been proposed, ranging from relatively simple operations (e.g., linear sum) to more complex ones (e.g., Block). The resulting multimodal representations define an intermediate feature space for capturing the interplay between visual and semantic features, that is helpful in selectively focusing on image content. In this paper, we propose a question-agnostic attention mechanism that is complementary to the existing question-dependent attention mechanisms. Our proposed model parses object instances to obtain an `object map' and applies this map on the visual features to generate Question-Agnostic Attention (QAA) features. In contrast to question-dependent attention approaches that are learned end-to-end, the proposed QAA does not involve question-specific training, and can be easily included in almost any existing VQA model as a generic light-weight pre-processing step, thereby adding minimal computation overhead for training. Further, when used in complement with the question-dependent attention, the QAA allows the model to focus on the regions containing objects that might have been overlooked by the learned attention representation. Through extensive evaluation on VQAv1, VQAv2 and TDIUC datasets, we show that incorporating complementary QAA allows state-of-the-art VQA models to perform better, and provides significant boost to simplistic VQA models, enabling them to performance on par with highly sophisticated fusion strategies.

Figures

Figures reproduced from arXiv: 1908.03289 by the authors.

Figure 1
Figure 1. A comparison of various multimodal fusion schemes for VQA [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture of our Question-Agnostic Attention (QAA) based VQA model. QAA features are generated using instance segmentation (generated [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. VQA accuracy (right y-axis) using complementary Image-Question [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Global Representation of object maps: Count of object presence at each of the 196 (14×14) spatial grid locations generated from the training images of VQA dataset. the dataset. In [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

  1. [1]

    Block: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Rela- tionship Detection,

    H. Ben-Younes, R. Cadene, N. Thome, and M. Cord, “Block: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Rela- tionship Detection,” in The Thirty-Third AAAI Conference on Artificial Intelligence, 2019

    work page 2019

  2. [2]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 2425– 2433

    work page 2015

  3. [3]

    Multimodal compact bilinear pooling for visual question answering and visual grounding,

    A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach, “Multimodal compact bilinear pooling for visual question answering and visual grounding,” arXiv preprint arXiv:1606.01847 , 2016

    arXiv 2016

  4. [4]

    Stacked attention networks for image question answering,

    Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, “Stacked attention networks for image question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 21– 29

    2016

  5. [5]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in CVPR, 2018

    2018

  6. [6]

    Reciprocal attention fusion for visual question answering,

    M. R. Farazi and S. Khan, “Reciprocal attention fusion for visual question answering,” inThe British Machine Vision Conference (BMVC), September 2018

    work page 2018

  7. [7]

    Co-attending free-form regions and detections with multi-modal multiplicative feature embed- ding for visual question answering,

    P. Lu, H. Li, W. Zhang, J. Wang, and X. Wang, “Co-attending free-form regions and detections with multi-modal multiplicative feature embed- ding for visual question answering,” in Thirty-Second AAAI Conference on Artificial Intelligence , 2018

    work page 2018

  8. [8]

    Mutan: Multi- modal tucker fusion for visual question answering,

    H. Ben-Younes, R. Cadene, M. Cord, and N. Thome, “Mutan: Multi- modal tucker fusion for visual question answering,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2612– 2620

    work page 2017

Show all 26 references
  1. [9]

    Hierarchical question-image co-attention for visual question answering,

    J. Lu, J. Yang, D. Batra, and D. Parikh, “Hierarchical question-image co-attention for visual question answering,” in Advances In Neural Information Processing Systems , 2016, pp. 289–297

    2016

  2. [10]

    From known to the unknown: Transferring knowledge to answer questions about novel visual and semantic concepts,

    M. R. Farazi, S. H. Khan, and N. Barnes, “From known to the unknown: Transferring knowledge to answer questions about novel visual and semantic concepts,” Image and Vision Computing , vol. 103, p. 103985, 2020

    2020

  3. [11]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778

    2016

  4. [12]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99

    2015

  5. [13]

    Learning to count objects in natural images for visual question answering,

    Y . Zhang, J. Hare, and A. Pr ¨ugel-Bennett, “Learning to count objects in natural images for visual question answering,” in International Conference on Learning Representations , 2018

    2018

  6. [14]

    Where to look: Focus regions for visual question answering,

    K. J. Shih, S. Singh, and D. Hoiem, “Where to look: Focus regions for visual question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 4613–4621

    2016

  7. [15]

    Human attention in visual question answering: Do humans and deep networks look at the same regions?

    A. Das, H. Agrawal, L. Zitnick, D. Parikh, and D. Batra, “Human attention in visual question answering: Do humans and deep networks look at the same regions?” Computer Vision and Image Understanding , vol. 163, pp. 90–100, 2017

    2017

  8. [16]

    Learning to predict where humans look,

    T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning to predict where humans look,” in 2009 IEEE 12th international conference on computer vision. IEEE, 2009, pp. 2106–2113

    2009

  9. [17]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on . IEEE, 2009, pp. 248–255

    2009

  10. [18]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543

    2014

  11. [19]

    Skip-thought vectors,

    R. Kiros, Y . Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Tor- ralba, and S. Fidler, “Skip-thought vectors,” in Advances in neural information processing systems , 2015, pp. 3294–3302

    2015

  12. [20]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  13. [21]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision. Springer, 2014, pp. 740–755

    2014

  14. [22]

    An analysis of visual question answering algorithms,

    K. Kafle and C. Kanan, “An analysis of visual question answering algorithms,” in ICCV, 2017

    2017

  15. [23]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2961–2969

    2017

  16. [24]

    Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering,

    Z. Yu, J. Yu, C. Xiang, J. Fan, and D. Tao, “Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering,” IEEE Transactions on Neural Networks and Learning Systems , 2018

    2018

  17. [25]

    Neural module networks,

    J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Neural module networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 39–48

    2016

  18. [26]

    Training recurrent answering units with joint loss minimization for vqa,

    H. Noh and B. Han, “Training recurrent answering units with joint loss minimization for vqa,” arXiv preprint arXiv:1606.03647 , 2016

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.