Pith. sign in

REVIEW 4 major objections 8 minor 24 references

DocVXQA: Context-Aware Visual Explanations for Document Question Answering

T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DocVXQA is the first self-explainable DocVQA framework: it learns answer-justifying masks during training, and people rate those masks clearer and more context-aware than attention heatmaps.

desk verdict A credible mask-based self-explainable DocVQA system with a real human-preference signal, but the 'context-aware' and IB claims outrun the evidence. read the letter →

arxiv 2505.07496 v1 pith:QKPGHH2E submitted 2025-05-12 cs.CV cs.LG

Mohamed Ali Souibgui , Changkyu Choi , Andrey Barsky , Kangsoo Jung , Ernest Valveny , Dimosthenis Karatzas This is my paper · ORCID
classification cs.CVcs.LG
keywords documentvisualquestionansweringself-explainablemodelexplanationsrelevancemaskinformationbottleneckexplainabilityvision-languagemodelscontext-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DocVXQA claims that a document question answering (DocVQA) system can be made self-explainable: together with each answer, the model learns a mask over the document that highlights the regions justifying that answer, and the masked document is literally what the model reads when producing the answer. The paper's central thesis is that this explainability can be trained directly — sufficiency by requiring the answer to stay correct on the masked image, minimality by penalizing how much of the image is highlighted — and that a retrieval model's heatmaps can be used as a prior to make the masks context-aware rather than mere reproductions of the answer text. This matters because current DocVQA explanations are post-hoc attention maps that often highlight every occurrence of the answer without showing why it was selected, which is an obstacle to trusting automated document reading in finance, healthcare, and administration. The reported experiments, including human ratings, support the claim that the learned masks carry enough context for people to answer from the masked image alone and that people prefer them to attention-based explanations.

What carries the argument

The load-bearing machinery is a learnable relevance mask M treated as an information-bottleneck representation: the masked image T = X⊙M is the only input the answering network — Pix2Struct, a pretrained OCR-free visual-language model — reads, so explanation and prediction are inseparable. Three losses carry the argument: cross-entropy with the ground-truth answer enforces sufficiency (the mask must retain what is needed to answer), an L1 term with an anisotropic total-variation continuity loss enforces minimality (sparse, coherent highlighted regions), and an MSE term pulls the mask toward a retrieval model's heatmaps to add context. The mask head is a lightweight MLP over encoder tokens, decoder attentions, and positional embeddings, so the whole apparatus attaches to an already-fine-tuned DocVQA model with only a new head and three loss terms, and the framework ports to other backbones such as Donut.

What would settle it

Build a set of questions where the answer string appears in several semantically different places in one document (for example, the same number under 'Total' and 'Subtotal', or the same name in two contract clauses) and ask which one the question means. A mask that merely encircles any occurrence of the answer will still let the model answer correctly, as the paper itself notes for its 'total amount' example, so the discriminating test is whether the learned mask consistently includes the disambiguating label or layout context. Running this probe with the released code would settle whether the masks carry reasoning context or track the retrieval prior regardless of where the true justification lies.

Watch

Extended reading notes

Core claim

The paper's claim is that a self-explainable DocVQA model can be built with small changes to a pretrained OCR-free vision-language model: a mask head turns encoded image tokens, decoder attention, and positional embeddings into a relevance mask M, and the masked image X⊙M is the bottleneck representation that the network reads to answer. The training objective unites three terms: a cross-entropy loss that keeps answers correct on masked inputs (sufficiency), an L1 plus anisotropic total-variation loss that keeps highlighted regions sparse and coherent (minimality), and an MSE loss that pulls the mask toward the question-document heatmaps of ColPali, a pretrained retrieval model whose late-interaction matching localizes question-relevant patches (context-awareness). The paper reports that without the third term the mask degenerates into a copy of the answer text, and that with it the method beats raw attention, attention rollout, Grad-CAM, and the retrieval-heatmap baseline in masked-input accuracy at comparable pixel ratio, while scoring highest in human context and clarity ratings; in a forced-choice study 163 of 252 trials preferred DocVXQA's masks.

Load-bearing premise

The method takes a pretrained retrieval model's heatmaps as the ground truth for 'context,' so the whole context-aware property rests on the assumption that what retrieval deems relevant for finding a document is also what a human needs to see in order to trust an answer, and that a pixel-level pull toward those heatmaps imports context rather than the retrieval model's own biases.

Editorial extensions

If this is right

  • If the central claim holds, DocVQA deployments can provide built-in justifications without post-hoc saliency tools or datasets annotated with answer locations.
  • The masked-image protocol turns 'does the explanation explain?' into a testable property: a mask is sufficient if the model — or a human — can still answer correctly from only the masked document.
  • The same mask head and losses attach to different DocVQA backbones, so the recipe is not bound to one architecture.
  • Threshold and postprocessing choices trade utility against minimality, giving practitioners a dial between revealing more of the document (higher accuracy) and revealing only the key evidence (higher interpretability).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test than Table 1's aggregates: build questions where the answer string appears in several places with different meanings (for instance, the same number under 'Total' and 'Subtotal') and check whether the mask includes the disambiguating header or layout; the paper's own 'total amount' example shows that highlighting any number can pass a sufficiency check without being a real justificat
  • Because the context prior is a single retrieval model, the explanations' ceiling is set by what retrieval relevance considers important; swapping in priors with different grounding, or an ensemble of priors, is an untested lever that could make masks better aligned with human reasoning.
  • The mask-then-answer loop suggests an iterative training direction the paper does not pursue: feeding the learned mask back as a hard attention constraint during further fine-tuning could push the underlying DocVQA model to rely on context it otherwise ignores.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes DocVXQA, a self-explainable framework for document visual question answering. A mask head on a frozen Pix2Struct backbone produces a relevance map; the masked image is fed to the same model to answer the question. The training objective (Eq. 3) combines a cross-entropy sufficiency term, an L1 plus total-variation minimality term, and an MSE term that aligns the learned mask with a ColPali-derived prior. The authors claim that this is the first self-explainable DocVQA model producing contextually sufficient and representation-efficient explanations, and they support this with quantitative comparisons against attention, gradient, and retrieval baselines, ablations, a human rating study, and a human preference study. Code is released.

Significance. If the central claims are sustained, the paper would contribute a practical way to attach learned visual explanations to an existing DocVQA model with modest architectural changes, and the release of code and use of an external pretrained prior are commendable. The human preference result (64.7% preference over ColPali+Pix2Struct) is a useful signal. However, the significance is currently bounded by three load-bearing issues: the actual objective (Eq. 3) is not the information-bottleneck objective announced in Eq. (1); the sufficiency evaluation partly measures the model's own training objective; and the 'context-aware' property is imported from ColPali rather than learned independently. These issues affect the core claims, not just the presentation.

major comments (4)
  1. [Sec. 3.1 and Sec. 3.3, Eqs. (1) and (3)] The paper announces an information-bottleneck objective in Eq. (1) and lists 'formalize explainability as an explicit learning objective with the information bottleneck principle' as a contribution, but the implemented objective in Eq. (3) replaces I(X;X⊙M) with L1 sparsity plus total-variation smoothness and adds an MSE term to a ColPali prior. Section 3.3 explicitly states that the kernel-density-estimation approach to mutual information 'did not yield the expected results.' No derivation is given to show that L1 plus TV approximates the mutual information term I(X;X⊙M), and the objective in Eq. (3) is not an information bottleneck in any formal sense. Please either provide a formal or empirical justification that the surrogate losses implement the IB trade-off, or revise the contribution and abstract to describe the actual objective (sparsity, smoothness, prior alignment) without claiming the IB principle.
  2. [Sec. 4.1 and Table 1] The sufficiency evaluation is partly circular. In the protocol, all explanation masks are applied to the input and the resulting image is passed to 'our fine-tuned Pix2Struct'—the model that was trained with L_CE on images masked by our own learned mask. Thus our mask is co-optimized with the evaluator model, while baseline masks (raw attention, Grad-CAM, ColPali) are not. The high accuracy of Ours at a given pixel ratio therefore reflects not only mask quality but also that the model has specialized to its own mask distribution. Please evaluate all methods with a fixed evaluator that has not been trained on masked inputs (e.g., the original pretrained Pix2Struct), or at least report the accuracy of random masks and of S-only masks under the same evaluator to bound the effect.
  3. [Sec. 3.4, Eq. (3), and Table 2] The central 'context-aware' claim is inherited from ColPali rather than learned. The ablation in Table 2 shows that without the token-interactions term (S+M only) the mask collapses to a tiny answer-only region (pixel ratio 0.02) with degraded accuracy, and the only source of context in Eq. (3) is L_MSE pulling the mask toward ColPali's retrieval heatmaps. Section 3.4 itself states that ColPali visualizations 'often fail to align with human intuition or provide comprehensive explanations.' The paper therefore needs evidence that ColPali's retrieval relevance coincides with the context needed to justify a DocVQA answer. Please provide (i) an evaluation against human-annotated context regions, or (ii) a comparison replacing the ColPali prior with a different prior (or a random fixed prior) to show the method's context-awareness is not just a copy of ColPali's biases.
  4. [Sec. 4.3, Table 3] The human evaluation of context-awareness is confounded by showing participants the ground-truth answer before they rate the masked image. The question 'How confident are you (1-5) to answer the question using only this masked image?' becomes, in effect, 'How easy is it to locate the answer string in this mask?' when the GT answer is displayed, so a mask that highlights any occurrence of the answer token can score highly even if it omits reasoning context. Please re-run the study with the GT answer hidden, or ask participants to produce the answer from the mask and then assess correctness separately.
minor comments (8)
  1. [Sec. 4.1 and Table 1] The threshold values for each baseline are chosen per method (e.g., Ours at 0.7, ColPali+Pix2Struct at 0.5), but no criterion for selecting these thresholds is stated. Please report the full threshold sweeps for all methods so the reader can see whether the reported operating points are comparably chosen.
  2. [Appendix A.1] The appendix says the background-removal and connected-region postprocessing steps are applied to all methods in Table 1, but the main text (Sec. 3.6) says postprocessing is applied only at inference. Please state clearly which steps are applied to which methods.
  3. [Sec. 2.1] There is a typo: 'text is extrtacted with OCR systems' should be 'extracted'.
  4. [Appendix A.1] Typos: 'envolve' and 'throught' should be 'involves' and 'through'.
  5. [Sec. 3.2, Eq. (2)] Strictly, cross-entropy CE(Y;Y_hat) equals H(Y|X⊙M) only in the ideal case; in general it is an upper bound. This is a standard surrogate, but the equality in Eq. (2) is written as if exact.
  6. [Sec. 3.4 and Fig. 2] The paper does not describe how the ColPali heatmap is extracted, resized, and aligned to the mask resolution for the L_MSE term. Please add these implementation details to the appendix.
  7. [Sec. 4.3] The human preference result is reported as 'p<< 0.001'; please report the actual p-value or exact binomial test statistic.
  8. [Sec. 4.3 and Appendix B.4] The model-agnostic claim is supported only by qualitative examples with Donut. Please include quantitative results or state explicitly that the model-agnostic demonstration is qualitative.

Circularity Check

1 steps flagged · score 5.0 of 10

Masked-input accuracy is the model's own L_CE training objective, so the main sufficiency evidence is partly by construction; the external ColPali prior and human study keep the central claim from being fully self-referential.

  1. fitted input called prediction [Sec. 3.2 Eq. (2); Sec. 3.5 Eq. (3); Sec. 4.1 evaluation protocol]
    "maxI(X⊙M ;Y ) = H(Y )| {z } constant −H(Y |X⊙M ) ⇐⇒ minH(Y |X⊙M )≃CE (Y ; ˆY) ... Predicted answer is compared with the ground truth (GT) using accuracy and average normalized Levenshtein similarity (ANLS) to assess the sufficiency. ... However, this does not necessarily indicate good relevance maps. In such scenarios, highlighting only any number will lead the DocVQA to use it as an answer, even without evidence that it corresponds to the “total”."

    The mask M is trained end-to-end by backpropagating L_CE on the masked input (Eq. 3), so the accuracy of answering from the masked image is the training objective itself. Reporting that same accuracy/ANLS on masked inputs as evidence of explanation 'sufficiency' (Table 1) is therefore a self-evaluation: the model has been explicitly optimized to answer from its own mask, and the metric does not independently test whether the mask contains the context required to justify the answer. The paper's own caveat that a mask can succeed by highlighting a single number without evidence it corresponds to the question shows why this objective alone cannot validate the context-sufficiency claim.

full rationale

The paper's derivation is not globally circular: the context prior (ColPali) is an external pretrained model, and the L_MSE alignment term in Eq. (3) imports relevance information from outside the paper's own fitted values. The citations to Choi et al. (2024), one of whose authors overlaps with this paper, are used to ground the minimality/sufficiency framing and the mask-learning objective, but the same principles are also attributed to external sources (Sokol & Flach, 2020; Tishby & Zaslavsky, 2015), and the cited DIB-X work has its own published experiments, so this self-citation is not the sole load-bearing support. The main circularity is in the sufficiency evaluation: Sec. 3.2 equates sufficiency with minimizing L_CE on the masked input, Eq. (3) trains with that L_CE, and Sec. 4.1 then uses masked-input accuracy/ANLS to 'assess the sufficiency.' This is a fitted-input-called-prediction pattern, though only partial because the paper adds a human study, a ColPali+Pix2Struct baseline, and a Donut backbone experiment. The minimality term is also a hand-chosen L1/total-variation proxy rather than the stated mutual information I(X;X⊙M); the paper admits the MI estimator 'did not yield the expected results,' so the information-bottleneck framing is asserted rather than demonstrated, but this is an approximation issue rather than a circular reduction. Overall: one partial circularity in the central sufficiency metric, with enough external evidence to avoid a high circularity score.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The ledger shows the framework rests on two borrowed components: a sparsity heuristic standing in for the information bottleneck, and a pretrained retrieval model standing in for ground-truth context. The hyperparameters are set once and not swept.

free parameters (4)
  • beta (minimality weight) = 5
    Weight on the L1 sparsity term in Eq (3); chosen without sensitivity analysis, values not justified.
  • gamma (ColPali prior weight) = 0.5
    Weight on the MSE mask-alignment loss in Eq (3); chosen without sensitivity analysis.
  • k (postprocessing boxes) = 3
    Top-k bounding boxes kept in postprocessing; selected via the ablation in Table 5 as a trade-off.
  • Mask evaluation threshold = 0.7
    Threshold used to binarize our masks in Table 1; different methods are compared at different thresholds, for example Raw Attention at 0.05, ColPali at 0.5, Ours at 0.7.
assumptions (3)
  • ad hoc to paper The information bottleneck objective in Eq (1) can be implemented by L1 sparsity plus total-variation smoothness and an MSE alignment to an external prior.
    Sec 3.3 reports that direct MI estimation did not work; Eq (3) is presented as the objective, but no derivation connects Eq (1) to Eq (3).
  • domain assumption ColPali's retrieval heatmaps provide a suitable prior for context-aware explanation masks.
    Sec 3.4 uses a pretrained ColPali as the source of context; if retrieval relevance and answer justification diverge, the explanations inherit ColPali's biases.
  • domain assumption The pretrained Pix2Struct can be fine-tuned to answer from masked images without losing its original document understanding.
    The unmasked accuracy of the fine-tuned model drops from 0.56 to 0.51 on DocVQA and from 0.80 to 0.57 on PFL-DocVQA (Table 1), which questions this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DocVXQA: Context-Aware Visual Explanations for Document Question Answering." pith.science (2026). https://pith.science/paper/QKPGHH2E

@misc{pith2026250507496,
  author       = {Pith},
  title        = {Pith review of: DocVXQA: Context-Aware Visual Explanations for Document Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKPGHH2E}},
  note         = {Machine review of arXiv:2505.07496}
}
read the original abstract

We propose DocVXQA, a novel framework for visually self-explainable document question answering. The framework is designed not only to produce accurate answers to questions but also to learn visual heatmaps that highlight contextually critical regions, thereby offering interpretable justifications for the model's decisions. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning objectives. Unlike conventional methods that emphasize only the regions pertinent to the answer, our framework delivers explanations that are \textit{contextually sufficient} while remaining \textit{representation-efficient}. This fosters user trust while achieving a balance between predictive performance and interpretability in DocVQA applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method. The code is available at https://github.com/dali92002/DocVXQA.

Figures

Figures reproduced from arXiv: 2505.07496 by the authors.

Figure 1
Figure 1. An illustration of the relevant regions in a DocVQA model (highlighted zones), produced by extracting the raw at￾tention maps from the last layer (top) and by using our method (bottom) for the question “ ‘Pleasure to burn since 1913’, Which cigarette’s tagline is this?”. Here, the answer given correctly by the model is “Camel”. Despite these promising advances in terms of utility, DocVQA relies on large, opaque neur… view at source ↗
Figure 2
Figure 2. Overview of the proposed DocVXQA framework: ① The input question and the full document image are sent to both the pretrained Pix2Struct ENC-DEC and a pretrained ColPali model. ② The mask head (MASK) generates a learnable mask based on the decoder output and positional embeddings. ③ ColPali provides a mask prior, highlighting the relevant regions of the input document in relation to the question. ④ The learnable mask… view at source ↗
Figure 3
Figure 3. A comparison of explanations generated by different methods for the question “What is the total amount?” with the model’s answer being “$180,000”. Relevance maps with different (thresholds) are applied to the input image to keep only the relevant regions. Best viewed at high zoom. Additional qualitative results across diverse contextual scenarios are provided in Appendix B.3. gles to maintain performance when forced… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Performance of our method with and without postpro￾cessing, under different thresholds applied to the relevance masks. evidence that it corresponds to the “total”. Our method lever￾ages the most clear and context-aware explanation. This results in an accurate visualiza…
Figure 5
Figure 5. Figure 5: Visualization of the background removal step in our methods. Question: “What is the estimated budget of ‘conduct analysis of decision makers/ information targets’ in research and development?”. Model answer “$ 7,000.00”. Bounding connected regions. After filtering out …
Figure 6
Figure 6. Figure 6: Visualization of connecting the relevance regions in the heatmap and keeping only 1 region with top confidence score (k = 1). Question: “What is the estimated budget of ‘conduct analysis of decision makers/ information targets’ in research and development?”. Model answ…
Figure 7
Figure 7. Figure 7: Visualization of the effect of using token interactions in our method. Left: Original Image. Middle: Masked image with relevance map learned without token interactions loss. Right: Masked image with relevance map learned with token interactions loss. 16 [PITH_FULL_IMA…
Figure 8
Figure 8. Figure 8: Question: What is the total percentage of Palestinians residing at places other than West Bank and Arab countries? Answer: 32.6 % (correct) [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Question: What is the percentage of women with disabilities in low income countries? Answer: 22.1 % (correct). 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Question: Under Private service how many patients were discharged in Neurology? Answer: 696 (correct) [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Question: What is the estimated cost of Space costs and Rentals? Answer: 700 (correct). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Question: What is the page number? Answer: 9 (correct) [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Question: How many cases of artworks are there in the shipment? Answer: 1 (correct) [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Question: Which bank’s check is this? Answer: The chase manhattan bank (correct). 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Question: Who is this question addressed to? Answer: bob bexon (correct) [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Question: What is the amount for publising one poster / abstract? Answer: $8,500 (correct). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Question: what type of communication is issued by Biomet, inc.? Answer: royalty payment form (correct). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

  1. [4]

    Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers

    Chefer, H., Gur, S., and Wolf, L. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 397– 406, 2021a. Chefer, H., Gur, S., and Wolf, L. Transformer interpretabil- ity beyond attention visualization. In Proceedings of the IEEE/CV...

    work page 2024

  2. [6]

    Boundingdocs: a unified dataset for document ques- tion answering with spatial annotations

    Giovannini, S., Coppini, F., Gemelli, A., and Marinai, S. Boundingdocs: a unified dataset for document ques- tion answering with spatial annotations. arXiv preprint arXiv:2501.03403,

    arXiv

  3. [7]

    Perceptual losses for real-time style transfer and super-resolution

    Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Com- puter Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II 14, pp. 694–711. Springer,

    work page 2016

  4. [8]

    Donut: Document understanding transformer without ocr

    Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664,

    arXiv

  5. [9]

    Mohammadshirazi, A., Neogi, P. P. G., Lim, S.-N., and Ramnath, R. Dlava: Document language and vision assis- tant for answer localization with enhanced interpretability and trustworthiness. arXiv preprint arXiv:2412.00151,

    arXiv

  6. [10]

    Rise: Randomized input sampling for explanation of black-box models

    Petsiuk, V . Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421,

    arXiv

  7. [12]

    Re- stricting the flow: Information bottlenecks for attribution

    Schulz, K., Sixt, L., Tombari, F., and Landgraf, T. Re- stricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396,

    arXiv 2001

  8. [14]

    and Zaslavsky, N

    Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1–5. IEEE,

    2015

Show all 24 references
  1. [15]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191,

    arXiv

  2. [16]

    Bloomberggpt: A large language model for finance

    Wu, S., Irsoy, O., Lu, S., Dabravolski, V ., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564,

    arXiv

  3. [17]

    Ureader: Universal ocr- free visually-situated language understanding with mul- timodal large language model

    Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. Ureader: Universal ocr- free visually-situated language understanding with mul- timodal large language model. In The 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  4. [18]

    Visrag: Vision- based retrieval-augmented generation on multi-modality documents

    Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y ., Liu, Z., Wang, S., Han, X., Liu, Z., et al. Visrag: Vision- based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024a. Yu, S., Yu, X., Løkse, S., Jenssen, R., and Principe, J. ...

    arXiv

  5. [19]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Zhang, Y ., Zhang, R., Gu, J., Zhou, Y ., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107,

    arXiv

  6. [20]

    Information- bottleneck approach to salient region discovery

    Zhmoginov, A., Fischer, I., and Sandler, M. Information- bottleneck approach to salient region discovery. In Ma- chine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Bel- gium, September 14–18, 2020, Proceedings, Part III, pp. 531–54...

    2020

  7. [22]

    What is the estimated budget of ‘conduct analysis of decision makers/ information targets’ in research and development?

    Visualization of connecting the relevance regions in the heatmap and keeping only 1 region with top confidence score (k = 1). Question: “What is the estimated budget of ‘conduct analysis of decision makers/ information targets’ in research and development?”. Model answer “$ 7,...

    2021

  8. [23]

    Left: Original Image

    Visualization of the effect of using token interactions in our method. Left: Original Image. Middle: Masked image with relevance map learned without token interactions loss. Right: Masked image with relevance map learned with token interactions loss. 16 DocVXQA: Context-Aware ...

    2022

  9. [24]

    19 DocVXQA: Context-Aware Visual Explanations for Document Question Answering B.4

    Question: Which bank’s check is this? Answer: The chase manhattan bank (correct). 19 DocVXQA: Context-Aware Visual Explanations for Document Question Answering B.4. Results with a different backbone In this section, we show the results of DocVXQA using the Donut backbone (Kim ...

    2021

  10. [2019]

    Going full-tilt boogie on document understanding with text-image-layout trans- former

    Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., and Pałka, G. Going full-tilt boogie on document understanding with text-image-layout trans- former. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzer- land...

    2021

  11. [2020]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  12. [2021]

    Doge: Towards versatile vi- sual document grounding and referring

    Zhou, Y ., Chen, Y ., Lin, H., Yang, S., Zhu, L., Qi, Z., Ma, C., and Shan, Y . Doge: Towards versatile vi- sual document grounding and referring. arXiv preprint arXiv:2411.17125,

    arXiv

  13. [2022]

    Colpali: Efficient document retrieval with vision language models

    Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., and Colombo, P. Colpali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.01449,

    arXiv

  14. [2023]

    M., 9 DocVXQA: Context-Aware Visual Explanations for Document Question Answering Liu, Q., Choudhury, M., Chauhan, H

    Aggarwal, K., Khandelwal, A., Tanmay, K., Khan, O. M., 9 DocVXQA: Context-Aware Visual Explanations for Document Question Answering Liu, Q., Choudhury, M., Chauhan, H. H., Som, S., Chaudhary, V ., and Tiwary, S. Dublin–document un- derstanding by language-image network. arXiv ...

    arXiv

  15. [2024]

    Weighted anisotropic– isotropic total variation for poisson denoising

    Bui, K., Lou, Y ., Park, F., and Xin, J. Weighted anisotropic– isotropic total variation for poisson denoising. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 1020–1024. IEEE,

    2023

  16. [2025]

    and Flach, P

    Sokol, K. and Flach, P. Explainability fact sheets: a framework for systematic assessment of explainable ap- proaches. In Proceedings of the 2020 conference on fairness, accountability, and transparency , pp. 56–67,

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.