REVIEW 2 major objections 5 minor 43 references
Answering Questions about Data Visualizations using Efficient Bimodal Fusion
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read PReFIL, a simple parallel fusion model, answers chart questions more accurately than previous systems and, in most settings, more accurately than humans.
desk verdict Solid empirical paper on chart QA, but the 'beats humans' claim only holds with oracle OCR — the abstract oversells what the body honestly qualifies. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the parallel recurrent fusion of image and language. Two fusion branches take low-level features (DenseNet layer 14) and high-level features (layer 40), concatenate the LSTM question vector to each spatial location, and apply four 1x1 convolutions with ReLU to produce bimodal embeddings. A bidirectional GRU then scans the spatial locations of each embedded feature map in sequence, producing a fixed-size vector for the classifier. The design deliberately replaces attention and relational modules with dense spatial fusion plus ordered aggregation. For DVQA, the model adds SANDY's dynamic OCR encoding: chart words detected by OCR are placed in an image-specific dictionary keyed by spatial position, so out-of-vocabulary words can appear in questions and answers.
What would settle it
Run PReFIL with oracle OCR on a chart where a legend word is placed closer to the bar of a different label than to its own bar; correct answers require spatial pairing, so a drop on such examples would show the assumption. Likewise, on real-world charts with annotations, the authors themselves predict a large accuracy drop, which would confirm the limitation.
Extended reading notes
Core claim
On its own terms, PReFIL is a claim that chart question answering can be solved by joint bimodal fusion plus an ordered aggregator. It processes the chart image through a 40-layer DenseNet, takes features from an early layer and the final layer, concatenates the LSTM question representation to every spatial cell of both feature maps, and passes each map through 1x1 convolutions to create question-specific embeddings. A bi-GRU then reads the embeddings in spatial order and the pooled result is classified. On FigureQA this reaches 93.16% overall on Test 2 (unseen color combinations), compared with 72.18% for the relational-network baseline and 91.21% for the human baseline on the shared subset. On DVQA's Test-Novel split, PReFIL with oracle OCR scores 96.53% overall versus 56.62% for SANDY and 88.18% for humans; with a real OCR pipeline, scores fall to about 80%, confirming that the remaining bottleneck is reading text, not reasoning.
Load-bearing premise
The DVQA results rest on the assumption that OCR can correctly pair every chart word with its spatial location; if a chart has extra text, overlapping labels, or human annotations, the dynamic encoding used by PReFIL is expected to fail.
Editorial extensions
If this is right
- On FigureQA, PReFIL surpasses the prior relational-network baseline by about 21 points on the color-alternated test set and also exceeds the human baseline on the same subset.
- On DVQA, the gap between real OCR and oracle OCR (about 69.5% and 80.0% for two real pipelines versus 96.5% for oracle) shows that chart-reading systems gain directly from better text recognition.
- The recurrent spatial aggregator and low-plus-high feature fusion each contribute: ablations show removal of bimodal embeddings costs over 12 points, removal of high-level features costs about 6 points, while low-level features and recurrent aggregation add smaller but consistent gains.
- A chart's underlying table can be reconstructed at about 90-91% overall accuracy by iteratively asking template questions, with value prediction (around 84-85%) as the weakest link.
- Because both benchmarks are synthetic and templated, the authors conclude that future datasets should move to real-world charts, human-generated questions, and document-level context.
Reading between the lines
- A testable extension is to run PReFIL on unmodified human-drawn charts: the authors predict that extra text and annotations will break the positional OCR encoding, which would separate reading robustness from reasoning ability.
- The architecture's strong results suggest that explicit relational or attention modules may be unnecessary for structured graphics; one could replace the bi-GRU with a simpler learned pooling to see whether the advantage really comes from preserving spatial order.
- Table reconstruction via question asking turns the QA model into a data-extraction tool; a natural next benchmark would measure reconstruction errors on charts whose values are non-integer or whose labels overlap, where value prediction is already the bottleneck.
- Because the largest gap on DVQA is between real and oracle OCR, combining the fusion network with a modern layout-aware OCR model is a more direct route to gains than further architectural innovation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PReFIL, a chart question answering (CQA) model that fuses question embeddings with low- and high-level image features in two parallel branches, then aggregates the resulting bimodal embeddings with a bi-GRU. The authors evaluate PReFIL on the DVQA and FigureQA datasets, reporting large improvements over prior state-of-the-art systems and, in several configurations, over crowd-sourced human baselines. They also contribute the first human accuracy estimate for DVQA, an ablation study of PReFIL's components, an analysis of OCR integration via SANDY's dynamic encoding scheme, and a demonstration of chart-to-table reconstruction through iterative question asking.
Significance. If the results hold, the paper is significant in two respects. First, it shows that a relatively simple parallel-fusion architecture without explicit attention or relation modules can outperform substantially more complex systems on two CQA benchmarks, which is a useful contribution to the growing literature on reasoning about synthetic visual data. Second, it provides the first human-performance baseline on DVQA and a careful study of how OCR quality affects end-to-end CQA accuracy. The chart-reconstruction application is a nice practical demonstration. These contributions are concrete and reproducible in principle, though the absence of released code and of variance estimates tempers the strength of the headline claims. The main caveat is that the DVQA human-surpassing result depends on oracle OCR, which the paper itself acknowledges; the unqualified abstract claim is therefore stronger than the reported evidence supports.
major comments (2)
- [Abstract; Section 1; Section 4.2; Table 4] The abstract claims that PReFIL 'greatly surpasses ... human baselines' on DVQA, but Table 4 shows this is true only for the oracle-OCR variant (96.53% vs 88.18% on Test-Novel). With the paper's improved real OCR pipeline, PReFIL reaches 80.04% on Test-Novel, which is below the same human baseline of 88.18%. Since Section 3.4 describes oracle OCR as 'emulated by providing the correct text-annotations for all the elements in the images,' this is an idealized condition rather than an end-to-end system. Furthermore, Section 5 explicitly notes that real-world chart text or human annotations would likely cause the dynamic encoding method to fail. Please qualify the abstract and the contributions list to state that the human comparison on DVQA uses oracle OCR, and explicitly note that with the current real OCR pipeline humans outperform PReFIL.
- [Section 4.1; Table 3] The claim that PReFIL surpasses human performance on FigureQA rests on an aggregate accuracy of 92.79% vs 91.21% on the 16,876-question Test 2 subset. A category-wise breakdown shows PReFIL is below humans on line charts (87.79% vs 90.55%) and the overall margin is only 1.58 points. Because no error bars, multiple-seed results, or significance tests are reported, this headline claim is stronger than the evidence supports. I request variance estimates over at least three training runs for the main comparisons, or a softened claim such as 'aggregate accuracy comparable to humans.'
minor comments (5)
- [Table 1] The FigureQA entry under 'Num. QA Pairs' reads '2,38,8698' and should presumably be '2,388,698'; please correct this typographical error.
- [Section 4.4 caption] The caption of Table 6 refers to 'PreFIL' but the model name is 'PReFIL'; use the consistent spelling throughout the manuscript.
- [Section 1 contributions list] The model name is given as 'parallel recurrent early fusion of image and language' in the contributions list but as 'parallel recurrent fusion of image and language' in the title and abstract; please unify the name.
- [Section 4.2] The phrases 'about a 24% improvement' and 'about an 11%' would be more informative if exact numbers from Table 4 were provided, since the table contains all the relevant values.
- [Section 1; Section 3.5] The contributions list states that code and pre-trained models will be publicly released, but no repository URL or availability statement is provided. Please include a link or a statement of when the code will be available.
Circularity Check
No circularity: PReFIL is evaluated on held-out public benchmarks against external baselines; oracle OCR is an experimental condition, not a fitted target.
full rationale
This is an empirical benchmark paper rather than a derivation or fitting exercise. PReFIL is a neural architecture trained with standard cross-entropy losses on public training splits and evaluated on held-out test splits (FigureQA Test 1/2 and DVQA Test-Familiar/Test-Novel) against published baselines and independently crowdsourced human performance. The DVQA OCR module reuses SANDY's published dynamic encoding scheme [16], which is a self-citation but not a load-bearing circular step: it is an input preprocessing method shared by both PReFIL and the SANDY baseline, and PReFIL's contribution is the parallel fusion and recurrent aggregation architecture plus an improved OCR pipeline. The oracle-OCR condition supplies ground-truth text annotations, which is an experimental setup limitation affecting external validity rather than a circular derivation, because the target accuracy is not encoded in the model parameters or in the evaluation protocol. No equation in the paper defines one claimed result in terms of another, and no reported prediction is a renamed fit of the data used to define the method. Therefore no significant circularity is present.
Assumptions & free parameters
free parameters (6)
- DenseNet depth and growth rate =
40 layers, growth rate 32
- Number of Q+I fusion 1x1 conv layers =
four 1x1 convolutions with 256 channels
- Bi-GRU hidden dimension =
256
- Classifier hidden layer size and dropout =
1024 units, dropout 0.5
- Learning rate schedule =
Adamax, base LR 7e-4, warmup 4 epochs, decay 0.7 at epochs 15-25
- OCR pipeline settings =
EAST at 0, 45, 90 degrees; 200% resize; Tesseract
assumptions (4)
- domain assumption The FigureQA and DVQA datasets provide valid, bias-controlled benchmarks for chart question answering.
- domain assumption The dynamic encoding scheme from SANDY correctly maps OOV words to chart positions when OCR is accurate.
- domain assumption Human baselines collected for DVQA are representative and were post-processed fairly.
- domain assumption No information leakage or scoring error for FigureQA's non-public test set.
Cite this review
Pith. "Pith review of Answering Questions about Data Visualizations using Efficient Bimodal Fusion." pith.science (2026). https://pith.science/paper/D7GTAPZX
@misc{pith2026190801801,
author = {Pith},
title = {Pith review of: Answering Questions about Data Visualizations using Efficient Bimodal Fusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/D7GTAPZX}},
note = {Machine review of arXiv:1908.01801}
}
read the original abstract
Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g. bar charts, pie charts, and line graphs. CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers. Without modifications, state-of-the-art VQA algorithms perform poorly on this task. Here, we propose a novel CQA algorithm called parallel recurrent fusion of image and language (PReFIL). PReFIL first learns bimodal embeddings by fusing question and image features and then intelligently aggregates these learned embeddings to answer the given question. Despite its simplicity, PReFIL greatly surpasses state-of-the art systems and human baselines on both the FigureQA and DVQA datasets. Additionally, we demonstrate that PReFIL can be used to reconstruct tables by asking a series of questions about a chart.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
M. Acharya, K. Jariwala, and C. Kanan. VQD: Visual query detection in natural scenes. In NAACL, 2019. 2
work page 2019
-
[2]
M. Acharya, K. Kafle, and C. Kanan. TallyQA: Answering complex counting questions. In AAAI, 2019. 1, 2
work page 2019
-
[3]
A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. In EMNLP, 2016. 2
work page 2016
-
[4]
A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2018. 2
work page 2018
-
[5]
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down atten- tion for image captioning and visual question answering. In CVPR, 2018. 4
work page 2018
-
[6]
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep compositional question answering with neural module net- works. In CVPR, 2016. 2
work page 2016
- [7]
-
[8]
C. Clark and M. Gardner. Simple and effective multi- paragraph reading comprehension. In ACL, 2018. 8
work page 2018
Show all 43 references
-
[9]
Cliche, D
M. Cliche, D. Rosenberg, D. Madeka, and C. Yee. Scatteract: Automated extraction of data from scatter plots. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2017. 2
2017
-
[10]
D. He, S. Cohen, B. Price, D. Kifer, and C. L. Giles. Multi- scale multi-task fcn for semantic page segmentation and ta- ble detection. In ICDAR, 2017. 8
2017
-
[11]
Huang, S
G. Huang, S. Liu, L. van der Maaten, and K. Q. Wein- berger. CondenseNet: An efficient densenet using learned group convolutions. In CVPR. 2018. 4
2018
-
[12]
Huang, Z
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 4
2017
-
[13]
Y . Jang, Y . Song, Y . Yu, Y . Kim, and G. Kim. Tgif-qa: To- ward spatio-temporal reasoning in visual question answer- ing. In CVPR, 2017. 1
2017
-
[14]
Johnson, B
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 1, 2
2017
-
[15]
Johnson, B
J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick. Inferring and executing programs for visual reasoning. In ICCV. 8
-
[16]
K. Kafle, S. Cohen, B. Price, and C. Kanan. Dvqa: Un- derstanding data visualizations via question answering. In CVPR, 2018. 1, 2, 4, 5, 6, 8, 10
2018
-
[17]
Kafle and C
K. Kafle and C. Kanan. An analysis of visual question an- swering algorithms. In ICCV, 2017. 2
2017
-
[18]
Kafle and C
K. Kafle and C. Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Im- age Understanding, 2017. 1, 2
2017
-
[19]
K. Kafle, R. Shrestha, and C. Kanan. Challenges and prospects in vision and language research. arXiv preprint arXiv:1904.09317, 2019. 2, 8
1904 arXiv
-
[20]
K. Kafle, M. Yousefhussien, and C. Kanan. Data augmenta- tion for visual question answering. In INLG, 2017. 1
2017
-
[21]
S. E. Kahou, A. Atkinson, V . Michalski, A. Kadar, A. Trischler, and Y . Bengio. FigureQA: An anno- tated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 1, 2, 7, 10, 11
2017 arXiv
-
[22]
J. S. Kallimani, K. Srinivasa, and R. B. Eswara. Extrac- tion and interpretation of charts in technical documents. In Advances in Computing, Communications and Informatics (ICACCI), 2013 International Conference on , pages 382–
2013
-
[23]
Kazemi and A
V . Kazemi and A. Elqursh. Show, ask, attend, and answer: A strong baseline for visual question answering.arXiv preprint arXiv:1704.03162, 2017. 4
2017 arXiv
-
[24]
Kembhavi, M
A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 2, 5
2016
-
[25]
Kembhavi, M
A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, 2017. 2
2017
-
[26]
J.-H. Kim, J. Jun, and B.-T. Zhang. Bilinear attention net- works. In NeurIPS, 2018. 6
2018
-
[27]
Krishna, Y
R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, M. Bern- stein, and L. Fei-Fei. Visual Genome: Connecting lan- guage and vision using crowdsourced dense image annota- tions. IJCV, 2017. 2
2017
-
[28]
J. J. Lau, S. Gayen, A. B. Abacha, and D. Demner-Fushman. A dataset of clinically generated visual questions and an- swers about radiology images. Scientific data , 5:180251,
-
[29]
Malinowski and C
M. Malinowski and C. Doersch. The visual QA devil in the details: The impact of early fusion and batch norm on clevr. arXiv preprint arXiv:1809.04482, 2018. 4
2018 arXiv
-
[30]
Malinowski and M
M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncer- tain input. In NeurIPS, 2014. 1, 2
2014
-
[31]
Perez, F
E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville. FiLM: Visual Reasoning with a General Con- ditioning Layer. In AAAI, 2018. 4
2018
-
[32]
Poco and J
J. Poco and J. Heer. Reverse-engineering visualizations: Re- covering visual encodings from chart images. In Computer Graphics Forum, volume 36, pages 353–363. Wiley Online Library, 2017. 2
2017
-
[33]
Reddy, R
R. Reddy, R. Ramesh, A. Deshpande, and M. M. Khapra. A question-answering framework for plots using deep learning. arXiv preprint arXiv:1806.04655, 2018. 4, 7
2018 arXiv
-
[34]
M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NeurIPS, 2015. 2
2015
-
[35]
Santoro, D
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. In NeurIPS, 2017. 4, 10, 11
2017
-
[36]
Savva, N
M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 393–402. ACM, 2011. 2
2011
-
[37]
Shrestha, K
R. Shrestha, K. Kafle, and C. Kanan. Answer them all! to- ward universal visual question answering models. In CVPR,
-
[38]
Singh, V
A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In CVPR, 2019. 1
2019
-
[39]
Tsutsui and D
S. Tsutsui and D. J. Crandall. A data driven approach for compound figure separation using convolutional neural net- works. In ICDAR, volume 1, pages 533–540. IEEE, 2017. 2
2017
-
[40]
Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola. Stacked attention networks for image question answering. In CVPR,
-
[41]
Yosinski, J
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015. 4
2015 arXiv
-
[42]
X. Zhou, C. Yao, H. Wen, Y . Wang, S. Zhou, W. He, and J. Liang. East: an efficient and accurate scene text detector. In CVPR, 2017. 5
2017
-
[43]
Is X the smoothest/roughest?
Y . Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 2 A. Analysis per FigureQA Question Template Table 7 shows results for PReFIL compared to RN [35, 21] and human baselines [21] for different question tem- plates. T...
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.