Pith. sign in

REVIEW 2 major objections 5 minor 43 references

Answering Questions about Data Visualizations using Efficient Bimodal Fusion

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read PReFIL, a simple parallel fusion model, answers chart questions more accurately than previous systems and, in most settings, more accurately than humans.

desk verdict Solid empirical paper on chart QA, but the 'beats humans' claim only holds with oracle OCR — the abstract oversells what the body honestly qualifies. read the letter →

arxiv 1908.01801 v2 pith:D7GTAPZX submitted 2019-08-05 cs.CV cs.AIcs.CLcs.LG

classification cs.CVcs.AIcs.CLcs.LG
keywords chartquestionansweringvisualbimodalfusiondatavisualizationOCRdynamicencodingDenseNettablereconstructionrecurrentaggregation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that answering questions about data visualizations does not need a purpose-built reasoning module: a straightforward image-and-language fusion network can outperform both previous systems and human annotators. The proposed model, PReFIL, combines low- and high-level image features with a question embedding at every spatial position, then aggregates those bimodal embeddings with a recurrent layer before classifying the answer. On the FigureQA benchmark it reaches 93.16% overall on the harder color-alternated test set, and on DVQA with perfect OCR it reaches 96.53% on the novel-word test split, beating the prior SANDY method by roughly 40 points and the human baseline by about 8 points. The paper also demonstrates that by asking itself a fixed sequence of questions, PReFIL can reconstruct the data table underlying a bar chart. The wider suggestion, made by the authors, is that the field is ready for harder, real-world chart datasets.

What carries the argument

The central mechanism is the parallel recurrent fusion of image and language. Two fusion branches take low-level features (DenseNet layer 14) and high-level features (layer 40), concatenate the LSTM question vector to each spatial location, and apply four 1x1 convolutions with ReLU to produce bimodal embeddings. A bidirectional GRU then scans the spatial locations of each embedded feature map in sequence, producing a fixed-size vector for the classifier. The design deliberately replaces attention and relational modules with dense spatial fusion plus ordered aggregation. For DVQA, the model adds SANDY's dynamic OCR encoding: chart words detected by OCR are placed in an image-specific dictionary keyed by spatial position, so out-of-vocabulary words can appear in questions and answers.

What would settle it

Run PReFIL with oracle OCR on a chart where a legend word is placed closer to the bar of a different label than to its own bar; correct answers require spatial pairing, so a drop on such examples would show the assumption. Likewise, on real-world charts with annotations, the authors themselves predict a large accuracy drop, which would confirm the limitation.

Watch

Extended reading notes

Core claim

On its own terms, PReFIL is a claim that chart question answering can be solved by joint bimodal fusion plus an ordered aggregator. It processes the chart image through a 40-layer DenseNet, takes features from an early layer and the final layer, concatenates the LSTM question representation to every spatial cell of both feature maps, and passes each map through 1x1 convolutions to create question-specific embeddings. A bi-GRU then reads the embeddings in spatial order and the pooled result is classified. On FigureQA this reaches 93.16% overall on Test 2 (unseen color combinations), compared with 72.18% for the relational-network baseline and 91.21% for the human baseline on the shared subset. On DVQA's Test-Novel split, PReFIL with oracle OCR scores 96.53% overall versus 56.62% for SANDY and 88.18% for humans; with a real OCR pipeline, scores fall to about 80%, confirming that the remaining bottleneck is reading text, not reasoning.

Load-bearing premise

The DVQA results rest on the assumption that OCR can correctly pair every chart word with its spatial location; if a chart has extra text, overlapping labels, or human annotations, the dynamic encoding used by PReFIL is expected to fail.

Editorial extensions

If this is right

  • On FigureQA, PReFIL surpasses the prior relational-network baseline by about 21 points on the color-alternated test set and also exceeds the human baseline on the same subset.
  • On DVQA, the gap between real OCR and oracle OCR (about 69.5% and 80.0% for two real pipelines versus 96.5% for oracle) shows that chart-reading systems gain directly from better text recognition.
  • The recurrent spatial aggregator and low-plus-high feature fusion each contribute: ablations show removal of bimodal embeddings costs over 12 points, removal of high-level features costs about 6 points, while low-level features and recurrent aggregation add smaller but consistent gains.
  • A chart's underlying table can be reconstructed at about 90-91% overall accuracy by iteratively asking template questions, with value prediction (around 84-85%) as the weakest link.
  • Because both benchmarks are synthetic and templated, the authors conclude that future datasets should move to real-world charts, human-generated questions, and document-level context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to run PReFIL on unmodified human-drawn charts: the authors predict that extra text and annotations will break the positional OCR encoding, which would separate reading robustness from reasoning ability.
  • The architecture's strong results suggest that explicit relational or attention modules may be unnecessary for structured graphics; one could replace the bi-GRU with a simpler learned pooling to see whether the advantage really comes from preserving spatial order.
  • Table reconstruction via question asking turns the QA model into a data-extraction tool; a natural next benchmark would measure reconstruction errors on charts whose values are non-integer or whose labels overlap, where value prediction is already the bottleneck.
  • Because the largest gap on DVQA is between real and oracle OCR, combining the fusion network with a modern layout-aware OCR model is a more direct route to gains than further architectural innovation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces PReFIL, a chart question answering (CQA) model that fuses question embeddings with low- and high-level image features in two parallel branches, then aggregates the resulting bimodal embeddings with a bi-GRU. The authors evaluate PReFIL on the DVQA and FigureQA datasets, reporting large improvements over prior state-of-the-art systems and, in several configurations, over crowd-sourced human baselines. They also contribute the first human accuracy estimate for DVQA, an ablation study of PReFIL's components, an analysis of OCR integration via SANDY's dynamic encoding scheme, and a demonstration of chart-to-table reconstruction through iterative question asking.

Significance. If the results hold, the paper is significant in two respects. First, it shows that a relatively simple parallel-fusion architecture without explicit attention or relation modules can outperform substantially more complex systems on two CQA benchmarks, which is a useful contribution to the growing literature on reasoning about synthetic visual data. Second, it provides the first human-performance baseline on DVQA and a careful study of how OCR quality affects end-to-end CQA accuracy. The chart-reconstruction application is a nice practical demonstration. These contributions are concrete and reproducible in principle, though the absence of released code and of variance estimates tempers the strength of the headline claims. The main caveat is that the DVQA human-surpassing result depends on oracle OCR, which the paper itself acknowledges; the unqualified abstract claim is therefore stronger than the reported evidence supports.

major comments (2)
  1. [Abstract; Section 1; Section 4.2; Table 4] The abstract claims that PReFIL 'greatly surpasses ... human baselines' on DVQA, but Table 4 shows this is true only for the oracle-OCR variant (96.53% vs 88.18% on Test-Novel). With the paper's improved real OCR pipeline, PReFIL reaches 80.04% on Test-Novel, which is below the same human baseline of 88.18%. Since Section 3.4 describes oracle OCR as 'emulated by providing the correct text-annotations for all the elements in the images,' this is an idealized condition rather than an end-to-end system. Furthermore, Section 5 explicitly notes that real-world chart text or human annotations would likely cause the dynamic encoding method to fail. Please qualify the abstract and the contributions list to state that the human comparison on DVQA uses oracle OCR, and explicitly note that with the current real OCR pipeline humans outperform PReFIL.
  2. [Section 4.1; Table 3] The claim that PReFIL surpasses human performance on FigureQA rests on an aggregate accuracy of 92.79% vs 91.21% on the 16,876-question Test 2 subset. A category-wise breakdown shows PReFIL is below humans on line charts (87.79% vs 90.55%) and the overall margin is only 1.58 points. Because no error bars, multiple-seed results, or significance tests are reported, this headline claim is stronger than the evidence supports. I request variance estimates over at least three training runs for the main comparisons, or a softened claim such as 'aggregate accuracy comparable to humans.'
minor comments (5)
  1. [Table 1] The FigureQA entry under 'Num. QA Pairs' reads '2,38,8698' and should presumably be '2,388,698'; please correct this typographical error.
  2. [Section 4.4 caption] The caption of Table 6 refers to 'PreFIL' but the model name is 'PReFIL'; use the consistent spelling throughout the manuscript.
  3. [Section 1 contributions list] The model name is given as 'parallel recurrent early fusion of image and language' in the contributions list but as 'parallel recurrent fusion of image and language' in the title and abstract; please unify the name.
  4. [Section 4.2] The phrases 'about a 24% improvement' and 'about an 11%' would be more informative if exact numbers from Table 4 were provided, since the table contains all the relevant values.
  5. [Section 1; Section 3.5] The contributions list states that code and pre-trained models will be publicly released, but no repository URL or availability statement is provided. Please include a link or a statement of when the code will be available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PReFIL is evaluated on held-out public benchmarks against external baselines; oracle OCR is an experimental condition, not a fitted target.

full rationale

This is an empirical benchmark paper rather than a derivation or fitting exercise. PReFIL is a neural architecture trained with standard cross-entropy losses on public training splits and evaluated on held-out test splits (FigureQA Test 1/2 and DVQA Test-Familiar/Test-Novel) against published baselines and independently crowdsourced human performance. The DVQA OCR module reuses SANDY's published dynamic encoding scheme [16], which is a self-citation but not a load-bearing circular step: it is an input preprocessing method shared by both PReFIL and the SANDY baseline, and PReFIL's contribution is the parallel fusion and recurrent aggregation architecture plus an improved OCR pipeline. The oracle-OCR condition supplies ground-truth text annotations, which is an experimental setup limitation affecting external validity rather than a circular derivation, because the target accuracy is not encoded in the model parameters or in the evaluation protocol. No equation in the paper defines one claimed result in terms of another, and no reported prediction is a renamed fit of the data used to define the method. Therefore no significant circularity is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical deep learning contribution. Its central claims rest on the validity of two synthetic benchmarks, the correctness of the dynamic OCR encoding, and a set of hand-selected hyperparameters. No physical constants or invented entities are introduced.

free parameters (6)
  • DenseNet depth and growth rate = 40 layers, growth rate 32
    Chosen by hand; no derivation given. Affects the learned visual feature hierarchy.
  • Number of Q+I fusion 1x1 conv layers = four 1x1 convolutions with 256 channels
    Chosen by hand; ablations show bimodal fusion matters, but exact depth is not derived.
  • Bi-GRU hidden dimension = 256
    Chosen by hand for recurrent spatial aggregation.
  • Classifier hidden layer size and dropout = 1024 units, dropout 0.5
    Chosen by hand; regularizes the final classifier.
  • Learning rate schedule = Adamax, base LR 7e-4, warmup 4 epochs, decay 0.7 at epochs 15-25
    Chosen by hand following prior work; affects convergence.
  • OCR pipeline settings = EAST at 0, 45, 90 degrees; 200% resize; Tesseract
    Chosen by hand for the improved OCR variant; impacts DVQA results.
assumptions (4)
  • domain assumption The FigureQA and DVQA datasets provide valid, bias-controlled benchmarks for chart question answering.
    The paper's central claim is defined as accuracy on these datasets; if they contain artifacts, the generality of the claim weakens.
  • domain assumption The dynamic encoding scheme from SANDY correctly maps OOV words to chart positions when OCR is accurate.
    Used in Section 3.4; the oracle-OCR variant presumes perfect text detection and localization.
  • domain assumption Human baselines collected for DVQA are representative and were post-processed fairly.
    Section 4.2; typo tolerance and numeric conversion affect the human comparison.
  • domain assumption No information leakage or scoring error for FigureQA's non-public test set.
    FigureQA Test 1/2 results were obtained by sending predictions to dataset creators; the process assumes correct scoring.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Answering Questions about Data Visualizations using Efficient Bimodal Fusion." pith.science (2026). https://pith.science/paper/D7GTAPZX

@misc{pith2026190801801,
  author       = {Pith},
  title        = {Pith review of: Answering Questions about Data Visualizations using Efficient Bimodal Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D7GTAPZX}},
  note         = {Machine review of arXiv:1908.01801}
}
read the original abstract

Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g. bar charts, pie charts, and line graphs. CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers. Without modifications, state-of-the-art VQA algorithms perform poorly on this task. Here, we propose a novel CQA algorithm called parallel recurrent fusion of image and language (PReFIL). PReFIL first learns bimodal embeddings by fusing question and image features and then intelligently aggregates these learned embeddings to answer the given question. Despite its simplicity, PReFIL greatly surpasses state-of-the art systems and human baselines on both the FigureQA and DVQA datasets. Additionally, we demonstrate that PReFIL can be used to reconstruct tables by asking a series of questions about a chart.

Figures

Figures reproduced from arXiv: 1908.01801 by the authors.

Figure 1
Figure 1. We propose the PReFIL algorithm for chart question [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example images and PReFIL outputs for FigureQA (top) and DVQA (bottom). Red denotes incorrect predictions. For incorrect [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. An example output of the chart to table algorithm. Red [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Some example predictions for PReFIL on the DVQA dataset. Red denotes incorrect predictions. For incorrect predictions, correct [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Some example predictions for PReFIL on the FigureQA dataset. Bottom row shows some incorrect predictions made by PReFIL. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 40 canonical work pages

  1. [1]

    Acharya, K

    M. Acharya, K. Jariwala, and C. Kanan. VQD: Visual query detection in natural scenes. In NAACL, 2019. 2

  2. [2]

    Acharya, K

    M. Acharya, K. Kafle, and C. Kanan. TallyQA: Answering complex counting questions. In AAAI, 2019. 1, 2

  3. [3]

    Agrawal, D

    A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. In EMNLP, 2016. 2

  4. [4]

    Agrawal, D

    A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2018. 2

  5. [5]

    Anderson, X

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down atten- tion for image captioning and visual question answering. In CVPR, 2018. 4

  6. [6]

    Andreas, M

    J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep compositional question answering with neural module net- works. In CVPR, 2016. 2

  7. [7]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. 1, 2

  8. [8]

    Clark and M

    C. Clark and M. Gardner. Simple and effective multi- paragraph reading comprehension. In ACL, 2018. 8

Show all 43 references
  1. [9]

    Cliche, D

    M. Cliche, D. Rosenberg, D. Madeka, and C. Yee. Scatteract: Automated extraction of data from scatter plots. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2017. 2

  2. [10]

    D. He, S. Cohen, B. Price, D. Kifer, and C. L. Giles. Multi- scale multi-task fcn for semantic page segmentation and ta- ble detection. In ICDAR, 2017. 8

  3. [11]

    Huang, S

    G. Huang, S. Liu, L. van der Maaten, and K. Q. Wein- berger. CondenseNet: An efficient densenet using learned group convolutions. In CVPR. 2018. 4

  4. [12]

    Huang, Z

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 4

  5. [13]

    Y . Jang, Y . Song, Y . Yu, Y . Kim, and G. Kim. Tgif-qa: To- ward spatio-temporal reasoning in visual question answer- ing. In CVPR, 2017. 1

  6. [14]

    Johnson, B

    J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 1, 2

  7. [15]

    Johnson, B

    J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick. Inferring and executing programs for visual reasoning. In ICCV. 8

  8. [16]

    K. Kafle, S. Cohen, B. Price, and C. Kanan. Dvqa: Un- derstanding data visualizations via question answering. In CVPR, 2018. 1, 2, 4, 5, 6, 8, 10

  9. [17]

    Kafle and C

    K. Kafle and C. Kanan. An analysis of visual question an- swering algorithms. In ICCV, 2017. 2

  10. [18]

    Kafle and C

    K. Kafle and C. Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Im- age Understanding, 2017. 1, 2

  11. [19]

    K. Kafle, R. Shrestha, and C. Kanan. Challenges and prospects in vision and language research. arXiv preprint arXiv:1904.09317, 2019. 2, 8

  12. [20]

    K. Kafle, M. Yousefhussien, and C. Kanan. Data augmenta- tion for visual question answering. In INLG, 2017. 1

  13. [21]

    S. E. Kahou, A. Atkinson, V . Michalski, A. Kadar, A. Trischler, and Y . Bengio. FigureQA: An anno- tated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 1, 2, 7, 10, 11

  14. [22]

    J. S. Kallimani, K. Srinivasa, and R. B. Eswara. Extrac- tion and interpretation of charts in technical documents. In Advances in Computing, Communications and Informatics (ICACCI), 2013 International Conference on , pages 382–

  15. [23]

    Kazemi and A

    V . Kazemi and A. Elqursh. Show, ask, attend, and answer: A strong baseline for visual question answering.arXiv preprint arXiv:1704.03162, 2017. 4

  16. [24]

    Kembhavi, M

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 2, 5

  17. [25]

    Kembhavi, M

    A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, 2017. 2

  18. [26]

    J.-H. Kim, J. Jun, and B.-T. Zhang. Bilinear attention net- works. In NeurIPS, 2018. 6

  19. [27]

    Krishna, Y

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, M. Bern- stein, and L. Fei-Fei. Visual Genome: Connecting lan- guage and vision using crowdsourced dense image annota- tions. IJCV, 2017. 2

  20. [28]

    J. J. Lau, S. Gayen, A. B. Abacha, and D. Demner-Fushman. A dataset of clinically generated visual questions and an- swers about radiology images. Scientific data , 5:180251,

  21. [29]

    Malinowski and C

    M. Malinowski and C. Doersch. The visual QA devil in the details: The impact of early fusion and batch norm on clevr. arXiv preprint arXiv:1809.04482, 2018. 4

  22. [30]

    Malinowski and M

    M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncer- tain input. In NeurIPS, 2014. 1, 2

  23. [31]

    Perez, F

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville. FiLM: Visual Reasoning with a General Con- ditioning Layer. In AAAI, 2018. 4

  24. [32]

    Poco and J

    J. Poco and J. Heer. Reverse-engineering visualizations: Re- covering visual encodings from chart images. In Computer Graphics Forum, volume 36, pages 353–363. Wiley Online Library, 2017. 2

  25. [33]

    Reddy, R

    R. Reddy, R. Ramesh, A. Deshpande, and M. M. Khapra. A question-answering framework for plots using deep learning. arXiv preprint arXiv:1806.04655, 2018. 4, 7

  26. [34]

    M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NeurIPS, 2015. 2

  27. [35]

    Santoro, D

    A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. In NeurIPS, 2017. 4, 10, 11

  28. [36]

    Savva, N

    M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 393–402. ACM, 2011. 2

  29. [37]

    Shrestha, K

    R. Shrestha, K. Kafle, and C. Kanan. Answer them all! to- ward universal visual question answering models. In CVPR,

  30. [38]

    Singh, V

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In CVPR, 2019. 1

  31. [39]

    Tsutsui and D

    S. Tsutsui and D. J. Crandall. A data driven approach for compound figure separation using convolutional neural net- works. In ICDAR, volume 1, pages 533–540. IEEE, 2017. 2

  32. [40]

    Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola. Stacked attention networks for image question answering. In CVPR,

  33. [41]

    Yosinski, J

    J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015. 4

  34. [42]

    X. Zhou, C. Yao, H. Wen, Y . Wang, S. Zhou, W. He, and J. Liang. East: an efficient and accurate scene text detector. In CVPR, 2017. 5

  35. [43]

    Is X the smoothest/roughest?

    Y . Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 2 A. Analysis per FigureQA Question Template Table 7 shows results for PReFIL compared to RN [35, 21] and human baselines [21] for different question tem- plates. T...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.