Pith. sign in

REVIEW 2 major objections 5 minor 58 references

Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Expert routing in multimodal continual instruction tuning is a soft class-incremental learning problem; reusing continual-learning classifiers as routers improves strict LoRA matching by up to 16.3 percentage points.

desk verdict Solid benchmark and a useful reformulation, but the headline CIL-router gains are undercut by the paper's own logit-scale calibration: on PureLoRA, simply raising tau from 28 to 80 beats every transferred CIL router. read the letter →

arxiv 2608.01437 v1 pith:SQOXRBKA submitted 2026-08-02 cs.AI

Huiyu Yi , Yongqi Xu , Bogang Zhang , Dunwei Tu , Xu Zhiming , Zhen-Hao Xie , Baile Xu , Furao Shen This is my paper · ORCID
classification cs.AI
keywords multimodalcontinualinstructiontuningexpertroutingLoRAclass-incrementallearningtask-as-classbenchmarktextualfingerprintssaturation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that expert routing in multimodal continual instruction tuning is not actually solved: standard benchmarks let routers guess the task from textual fingerprints and short task sequences, so routing looks saturated. To expose this, it builds FLEX, a 34-task benchmark that shares answer-format templates across confusable tasks and weakens identity leakage. It then reframes progressive-LoRA routing as soft task-as-class multimodal class-incremental learning, where each task is a class and the full score distribution sets the LoRA mixture weights. With that interface, four continual-learning classifiers can be plugged into existing routers without touching trained experts, improving strict LoRA matching by up to 16.3 percentage points and MacroScore by up to 4.6 points.

What carries the argument

The load-bearing object is the task-as-class MCIL reformulation. In a progressive LoRA pool, task $k$ becomes routing class $k$: the target response is discarded, and every input $x_k^i=(v_k^i,q_k^i)$ is relabelled $c_k^i=k$, turning routing into an expanding $t$-way classification problem. The router output $s_t(x)\in\mathbb{R}^t$ is converted into LoRA mixture weights by the temperature-scaled softmax $\alpha^{\mathrm{soft}}_{t,k}(x)=\exp(\tau s_{t,k}(x))/\sum_{j=1}^t \exp(\tau s_{t,j}(x))$, with hard routing as the $\arg\max$ special case. This identity makes 'which LoRA should process this input' the same decision as 'which class does this input belong to', and lets any CIL/MCIL scorer t

What would settle it

Take FLEX and randomly permute the shared answer-format suffixes across tasks, then train and evaluate a text-only router on the same split; if strict matching stays above roughly 90%, residual textual cues rather than the normalized templates are carrying task identity, and the benchmark would not be testing what the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that progressive-LoRA expert routing is an expanding task-identification problem, not a solved subproblem. Concretely, each incoming task $k$ defines a routing class $\widetilde{D}_k=\{(x_k^i, c_k^i)\}$ with $c_k^i=k$ (Eq. 3), so after $t$ tasks the router faces a $t$-way classification problem over an expanding class set. The router's score vector is converted into LoRA mixture weights by $\alpha^{\mathrm{soft}}_{t,k}(x)=\exp(\tau s_{t,k}(x))/\sum_{j=1}^t \exp(\tau s_{t,j}(x))$ (Eq. 7), with hard routing as the $\arg\max$ special case. Existing frozen-CLIP prototype routers are, under this view, just nearest-class-mean classifiers over frozen features. On the 34

Load-bearing premise

FLEX's ability to expose routing difficulty rests on the assumption that normalizing answer-format templates and grouping by interface removes textual task-identity shortcuts without stripping task-necessary content, leaving no residual cues that still leak task identity.

Editorial extensions

If this is right

  • Existing 4–10-task MCIT benchmarks overstate router quality: once textual fingerprints are weakened and the expert pool grows, task-agnostic routing falls far short of oracle routing, so routing should be evaluated on long, interface-shared sequences.
  • Any CIL/MCIL method that yields comparable scores over an expanding task set can be dropped into an MCIT router; the transferred classifiers here recover 28.3–50.2% of the Base-to-oracle MacroScore gap on the routing-sensitive frameworks.
  • Soft routing is not just a compromise: with fixed experts and prototypes, a calibrated soft mixture (logit scale ×80) outperforms hard top-1 selection by 2.01 MacroScore points at the same top-1 accuracy, so composition matters after identification.
  • Routing-quality metrics and downstream scores co-move mainly through top-1 identification (Spearman ρ=0.71), but the cost of a wrong expert depends on cross-task LoRA compatibility; forced transfer shows most confusions hurt, while one (GQA→VQAv2) helps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if routing is task-as-class MCIL, benchmark difficulty can be engineered independently of task content---adding near-duplicate interface groups or adversarial suffix permutations could make a routing stress test even sharper than FLEX.
  • The saturation finding implies that some previously reported MCIT gains may reflect better shortcut recovery rather than better continual learning; re-running established methods on fingerprint-reduced sequences could re-rank them.
  • The task-as-class mapping treats each task as a flat class; a natural next step is hierarchical routing---first choose an interface group (classification, captioning, multiple-choice, VQA), then a task inside it---which would shrink the effective expert pool and may scale better than a 34-way softmax.
  • Because soft routing weights every expert, routing calibration and cross-task LoRA compatibility could be disentangled by training deliberately confusable task pairs and testing whether a calibrated router learns to share compatible LoRAs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that expert routing in multimodal continual instruction tuning (MCIT) is saturated on existing benchmarks because of textual fingerprints that leak task identity and short 4–10-task sequences with few competing LoRAs. To address this, it introduces FLEX, a 34-task long-horizon benchmark with weakened textual fingerprints, and reformulates progressive-LoRA routing as soft task-as-class Multimodal Class-Incremental Learning (MCIL). Using a controlled PureLoRA baseline, the authors adapt four CIL methods (HC, HC-SOINN, RanPAC, DDAS) as plug-in routers into four MCIT frameworks, reporting improved strict LoRA matching and MacroScore. The appendix contains extensive ablations: fingerprint injection, pool scaling, task order, seed robustness, backbone transfer, and modality fusion.

Significance. If the quantitative claims hold, FLEX is a valuable diagnostic benchmark and the task-as-class MCIL interface is a useful conceptual bridge between expert routing and continual learning. The paper's strengths include a controlled router-replacement protocol, complete per-task results, a public code release, and a thorough supplementary analysis. The routing-quality improvements (R and G) are substantial and not easily explained by confounds. However, the downstream MacroScore improvements attributed to MCIL transfer are currently confounded with logit-scale sharpening, and the main table lacks repeated runs for most configurations. The practical benefit of the MCIL formulation is therefore plausible but not yet established at the claimed level.

major comments (2)
  1. [Fig. 5 and S2.4] The MacroScore gains attributed to CIL routers are not separated from logit-scale sharpening. For PureLoRA, raising only the logit scale τ from 28 to 80 with fixed experts and prototypes gives M=56.27, higher than the best PureLoRA+CIL variant in Table 2 (RanPAC, M=56.14), while keeping R=83.77 unchanged. S2.4 similarly shows that lowering τ from 28 to 20 moves PureLoRA toward DISCO's Base score. Since RanPAC and DDAS are applied with temperatures 0.03 and 0.2 (S2.5), Table 2 mixes router identity with output sharpness. Please add a calibrated logit-scale Base to Table 2 (e.g., PureLoRA at τ=80) and, for DISCO and SAME, sweep τ/temperature to bound how much of the +1.54–+4.61 MacroScore gains is attributable to the CIL classifier rather than to sharpening. Without this control, the headline downstream claim overstates the role of the MCIL formulation.
  2. [Table 2 and S3.4] The main results table reports single-run scores for most of the 20 configurations, with no error bars. Gains as small as +0.09 (HiDe-LLaVA+HC-SOINN) and +0.11 (HiDe-LLaVA+HC) are within plausible random variation. The seed studies in S3.4 cover only HiDe-LLaVA training seeds and RanPAC random-projection seeds; they do not cover DISCO, SAME, HC-SOINN, or DDAS, nor the larger gains on those frameworks. Given that the headline claims depend on small MacroScore differences, the paper should provide at least three seeds for each configuration in Table 2 (or a principled subset with justified selection), and report mean±std for M, G, and R. This is needed to determine whether the observed improvements are stable or partly noise.
minor comments (5)
  1. [S3.1] The text refers to 'FLEXv2' in the t-SNE visualization caption, although the benchmark is called FLEX. Please correct this inconsistency.
  2. [Table 2] The '(fixed)' tag for DISCO and HiDe-LLaVA rows is not explained until S2.1–S2.2. Since these configurations differ from the official released protocols, add a brief footnote at the first occurrence in the main table.
  3. [Experimental Setup] The hardware description '8 T-Head Zhenwu 810E PPUs' is not standard; please specify the GPU/accelerator type and, if possible, include runtime per task for reproducibility.
  4. [Fig. 1] The statement that tied UCIT tasks under DISCO are 'shown by their 50/50 soft weights' is unclear. Does this mean the softmax weights are exactly 0.5/0.5, or that the matching metric uses a tie-breaking rule? Please clarify.
  5. [Eq. (2)] The definition of g_s uses the subscript notation 'α^{(T)}_{s,i,s}' which is easy to misread. Consider rewriting as g_s = (1/N_s) Σ_i α^{(T)}_{s,i,s} to make the indexing clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are empirical and the MCIL reformulation is a definitional interface, not a derived prediction; self-citations are not load-bearing.

full rationale

The paper's contributions — the FLEX benchmark, the saturation diagnosis, and the transfer of CIL classifiers to routing — are empirical. Equation (3) assigns each input the routing label c_i^k = k; it is a labeling convention, not a derived result. Equation (7) converts task scores into soft LoRA weights via softmax; it is a design choice, and hard routing is explicitly treated as a discrete special case. No reported metric (R, G, M, Oracle-gap recovery) is forced by either equation. FLEX hardness is validated by ablation controls: Strong-vs-Weak fingerprint contrasts, prefix-matched expert-pool scaling (Table 3.9), and the modality-fusion optimum shifting from text-only to balanced (S3.10). The paper also acknowledges the benchmark's limits: 'FLEX is not entirely fingerprint-free' and S3.8 states that necessary content is retained. Self-citations to HC/HC-SOINN are method implementations, but the main gains also come from externally published RanPAC and DDAS, and all experts are fixed across comparisons, so self-citation is not load-bearing. The skeptic's observation that a simple logit-scale change (tau=80) exceeds PureLoRA+RanPAC, and that tau and lambda explain PureLoRA's Base advantage (S2.4), is a legitimate confound for causal attribution but is not circularity: no parameter was fitted to the target outcome and no equation is equivalent by construction. Thus the derivation chain is self-contained and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim is empirical, resting on the design of FLEX and the transfer of CIL classifiers. The free parameters are global hyperparameters that affect the quantitative comparisons. The axioms are the main assumptions about feature informativeness, the one-task-one-class labeling, and the validity of softmax composition.

free parameters (3)
  • Routing logit scale tau = 28 for PureLoRA/SAME/HiDe; DISCO uses softmax temperature T=0.05 (equiv tau=20)
    Hand-chosen hyperparameter that converts routing scores to LoRA weights (Eq. 7). S2.4 shows changing tau from 28 to 20 lowers PureLoRA MacroScore by 1.86 points, so quantitative results depend on this choice.
  • Modality fusion weight lambda = 0.5 (PureLoRA), 0.2 (SAME), 0 (DISCO)
    Balances visual and textual routing features (Eq. 10). S2.4 shows changing lambda from 0.5 to 0.2 lowers MacroScore by 1.88 points.
  • DDAS score scaling kappa = 1e5
    Scales negative MSE before softmax (Eq. 14); arbitrary since softmax temperature can absorb it, but chosen by hand.
assumptions (3)
  • domain assumption Frozen CLIP ViT-L/14-336 encoders provide routing features that remain informative across 34 FLEX tasks after fingerprint weakening.
    All routers (Base and CIL) operate on these features; if they were uninformative, routing gains would be impossible. The paper's confusion matrices and gains support this, but it is an unproven assumption about feature quality.
  • ad hoc to paper Each MCIT task can be treated as a single routing class, assigning all samples of task k the label k (Eq. 3).
    This is the MCIL reformulation premise. It holds by construction for progressive LoRA pools (one LoRA per task), and is argued to hold for SAME via training-time slot binding (S2.3).
  • domain assumption Softmax over task scores with a shared temperature yields valid LoRA mixture weights, i.e., the composition in Eq. (1) improves generation.
    The evaluation assumes calibrated soft weights are better than hard selection; the calibration experiment (Figure 5) tests this empirically but the general validity is assumed.
invented entities (1)
  • FLEX benchmark (34-task fingerprint-reduced long-horizon MCIT benchmark) independent evidence
    purpose: To evaluate expert routing under weakened textual fingerprints and a larger expert pool
    The benchmark is released with code and data, so other groups can run it and check whether routing is indeed unsaturated. The paper provides ablations (fingerprint injection, pool scaling) as internal validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning." pith.science (2026). https://pith.science/paper/SQOXRBKA

@misc{pith2026260801437,
  author       = {Pith},
  title        = {Pith review of: Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQOXRBKA}},
  note         = {Machine review of arXiv:2608.01437}
}
read the original abstract

Multimodal Continual Instruction Tuning (MCIT) enables multimodal large language models to acquire new tasks sequentially while retaining previously learned capabilities. Many recent methods maintain task-specific LoRA experts and route each input to one or more experts at inference. Yet the task-identification problem underlying expert routing remains under-explored. We show that routing is nearly saturated on widely used MCIT benchmarks. Textual fingerprints that leak task identity and short 4--10-task sequences with few competing experts jointly obscure the long-horizon routing problem. To expose this challenge, we introduce FLEX (Fingerprint-reduced Long-horizon Expert eXamination), a 34-task long-horizon MCIT benchmark with weakened textual fingerprints. FLEX groups tasks with similar instruction and answer formats but diverse visual and knowledge domains, normalizes their outer templates, and evaluates routing over a substantially larger expert pool. Crucially, we formulate progressive-LoRA routing as soft task-as-class Multimodal Class-Incremental Learning (MCIL): each task defines an incremental routing class, whose complete score distribution supplies the LoRA mixture weights, with hard routing as a discrete special case. FLEX exposes this expanding task-identification challenge, while the MCIL formulation provides a principled interface for transferring CIL methods to expert routing. We instantiate PureLoRA as a controlled baseline and adapt four CIL methods to four MCIT frameworks without modifying their LoRA experts or generation pipelines. Our plug-in routers improve strict LoRA matching by up to 16.3 percentage points and overall MacroScore by up to 4.6 points. Code is available at: https://github.com/RINC-CL/FLEX

Figures

Figures reproduced from arXiv: 2608.01437 by the authors.

Figure 1
Figure 1. Task–LoRA matching on CoIN, UCIT, and TriGap. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Routing quality across established MCIT bench [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of FLEX and the proposed task-as-class MCIL formulation for multimodal LoRA routing. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Routing quality and downstream performance. (a,b) Each point is one CIL router minus its Base: [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: PureLoRA routing-scale calibration with fixed ex [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 2.1
Figure 2.1. Figure 2.1: Task-level correlations between MacroScore changes and changes in strict matching. [PITH_FULL_IMAGE:figures/full_fig_p014_2_1.png]
Figure 2.1
Figure 2.1. Figure 2.1: Task-level correlations between MacroScore changes and changes in ground-truth LoRA weight. [PITH_FULL_IMAGE:figures/full_fig_p015_2_1.png]
Figure 2.2
Figure 2.2. Figure 2.2: PureLoRA task–expert confusion matrix on [PITH_FULL_IMAGE:figures/full_fig_p015_2_2.png]
Figure 3.1
Figure 3.1. Figure 3.1: CLIP image and deduplicated text representations used by the routing analysis. [PITH_FULL_IMAGE:figures/full_fig_p018_3_1.png]
Figure 3.2
Figure 3.2. Figure 3.2: Controlled router-only and end-to-end inference [PITH_FULL_IMAGE:figures/full_fig_p019_3_2.png]
Figure 3.3
Figure 3.3. Figure 3.3: Base and CIL routing on UCIT (top) and TriGap (bottom). [PITH_FULL_IMAGE:figures/full_fig_p020_3_3.png]
Figure 3.4
Figure 3.4. Figure 3.4: Performance variation across RanPAC projection [PITH_FULL_IMAGE:figures/full_fig_p020_3_4.png]
Figure 3.5
Figure 3.5. Figure 3.5: Four representative PureLoRA routing failures. Semantically related but task-incorrect experts receive high weight [PITH_FULL_IMAGE:figures/full_fig_p022_3_5.png]
Figure 3.6
Figure 3.6. Figure 3.6: Effect of task-specific answer-suffix fingerprints [PITH_FULL_IMAGE:figures/full_fig_p025_3_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 15 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Coin: A benchmark of continual instruction tuning for multimodel large language models , author=. Advances in neural information processing systems , volume=

  2. [2]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  3. [3]

    Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning

    Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning , author=. arXiv preprint arXiv:2605.26110 , year=

    work page Pith review arXiv

  4. [4]

    International Conference on Machine Learning (ICML) , year=

    SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning , author=. International Conference on Machine Learning (ICML) , year=

    work page

  5. [5]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Progressive lora for multimodal continual instruction tuning , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  6. [6]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Federated continual instruction tuning , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    work page

  7. [7]

    Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages=

    icarl: Incremental classifier and representation learning , author=. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages=

  8. [8]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Continual prototype evolution: Learning online from non-stationary data streams , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

Show all 58 references
  1. [9]

    arXiv preprint arXiv:2207.09248 , year=

    Don't stop learning: Towards continual learning for the clip model , author=. arXiv preprint arXiv:2207.09248 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2605.11904 , year=

    Beyond Point-wise Neural Collapse: A Topology-Aware Hierarchical Classifier for Class-Incremental Learning , author=. arXiv preprint arXiv:2605.11904 , year=

    arXiv

  3. [11]

    Advances in Neural Information Processing Systems , volume=

    Ranpac: Random projections and pre-trained models for continual learning , author=. Advances in Neural Information Processing Systems , volume=

  4. [12]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Boosting continual learning of vision-language models via mixture-of-experts adapters , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  5. [13]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  6. [14]

    International conference on machine learning , pages=

    The flan collection: Designing data and methods for effective instruction tuning , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  7. [15]

    ACM Computing Surveys , volume=

    Instruction tuning for large language models: A survey , author=. ACM Computing Surveys , volume=. 2026 , publisher=

    2026

  8. [16]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    arXiv

  9. [17]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  10. [18]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  11. [19]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  12. [20]

    , author=

    Visualizing data using t-SNE. , author=. Journal of machine learning research , volume=

  13. [21]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  14. [22]

    ImageNet: A large-scale hierarchical image database , url=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Kai Li and Li Fei-Fei , year=. ImageNet: A large-scale hierarchical image database , url=. doi:10.1109/cvpr.2009.5206848 , booktitle=

    2009

  15. [23]

    A Dataset and Baselines for Visual Question Answering on Art , url=

    Garcia, Noa and Ye, Chentao and Liu, Zihua and Hu, Qingtao and Otani, Mayu and Chu, Chenhui and Nakashima, Yuta and Mitamura, Teruko , year=. A Dataset and Baselines for Visual Question Answering on Art , url=. doi:10.1007/978-3-030-66096-3_8 , booktitle=

    doi

  16. [24]

    Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering , url=

    Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi , year=. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering , url=. doi:10.1109/cvpr.2017.670 , booktitle=

    2017 doi

  17. [25]

    RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System , volume=

    Guan, Runwei and Hu, Rongsheng and Chen, Shangshu and Xiao, Ningyuan and Xia, Xue and Liu, Jiayang and Chen, Beibei and Tang, Ziren and Ouyang, Ningwei and Liang, Shaofeng and Fan, Yuxuan and Sun, Wanjie and Yue, Yutao , year=. RoadSceneVQA: Benchmarking Visual Question Answer...

    doi

  18. [26]

    Captioning Images Taken by People Who Are Blind , url=

    Gurari, Danna and Zhao, Yinan and Zhang, Meng and Bhattacharya, Nilavra , year=. Captioning Images Taken by People Who Are Blind , url=. doi:10.1007/978-3-030-58520-4_25 , booktitle=

    doi

  19. [27]

    and Manning, Christopher D

    Hudson, Drew A. and Manning, Christopher D. , year=. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering , url=. doi:10.1109/cvpr.2019.00686 , booktitle=

    2019

  20. [28]

    Lawrence and Girshick, Ross , year=

    Johnson, Justin and Hariharan, Bharath and van der Maaten, Laurens and Fei-Fei, Li and Zitnick, C. Lawrence and Girshick, Ross , year=. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning , url=. doi:10.1109/cvpr.2017.215 , booktitle=

    2017 doi

  21. [29]

    A Diagram is Worth a Dozen Images , url=

    Kembhavi, Aniruddha and Salvato, Mike and Kolve, Eric and Seo, Minjoon and Hajishirzi, Hannaneh and Farhadi, Ali , year=. A Diagram is Worth a Dozen Images , url=. doi:10.1007/978-3-319-46493-0_15 , booktitle=

    doi

  22. [30]

    Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension , url=

    Kembhavi, Aniruddha and Seo, Minjoon and Schwenk, Dustin and Choi, Jonghyun and Farhadi, Ali and Hajishirzi, Hannaneh , year=. Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension , url=. doi:10.1109/cvpr.2017.571 , booktitle=

    2017 doi

  23. [31]

    and Gayen, Soumya and Ben Abacha, Asma and Demner-Fushman, Dina , year=

    Lau, Jason J. and Gayen, Soumya and Ben Abacha, Asma and Demner-Fushman, Dina , year=. A dataset of clinically generated visual questions and answers about radiology images , volume=. Scientific Data , publisher=. doi:10.1038/sdata.2018.251 , number=

    2018 doi

  24. [32]

    Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering , url=

    Liu, Bo and Zhan, Li-Ming and Xu, Li and Ma, Lin and Yang, Yan and Wu, Xiao-Ming , year=. Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering , url=. doi:10.1109/isbi48211.2021.9434010 , booktitle=

    2021

  25. [33]

    OCRBench: on the hidden mystery of OCR in large multimodal models , volume=

    Liu, Yuliang and Li, Zhang and Huang, Mingxin and Yang, Biao and Yu, Wenwen and Li, Chunyuan and Yin, Xu-Cheng and Liu, Cheng-Lin and Jin, Lianwen and Bai, Xiang , year=. OCRBench: on the hidden mystery of OCR in large multimodal models , volume=. Science China Information Sci...

    doi

  26. [34]

    Learn to Explain: Multimodal Reasoning Via Thought Chains for Science Question Answering , url=

    Lu, Pan and Mishra, Swaroop and Xia, Tanglin and Qiu, Liang and Chang, Kai-Wei and Zhu, Song-Chun and Tafjord, Oyvind and Clark, Peter and Kalyan, Ashwin , year=. Learn to Explain: Multimodal Reasoning Via Thought Chains for Science Question Answering , url=. doi:10.52202/0684...

    doi

  27. [35]

    OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge , url=

    Marino, Kenneth and Rastegari, Mohammad and Farhadi, Ali and Mottaghi, Roozbeh , year=. OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge , url=. doi:10.1109/cvpr.2019.00331 , booktitle=

    2019

  28. [36]

    Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, C. V. , year=. DocVQA: A Dataset for VQA on Document Images , url=. doi:10.1109/wacv48630.2021.00225 , booktitle=

    2021

  29. [37]

    Mathew, Minesh and Bagal, Viraj and Tito, Ruben and Karatzas, Dimosthenis and Valveny, Ernest and Jawahar, C. V. , year=. InfographicVQA , url=. doi:10.1109/wacv51458.2022.00264 , booktitle=

    2022

  30. [38]

    OCR-VQA: Visual Question Answering by Reading Text in Images , url=

    Mishra, Anand and Shekhar, Shashank and Singh, Ajeet Kumar and Chakraborty, Anirban , year=. OCR-VQA: Visual Question Answering by Reading Text in Images , url=. doi:10.1109/icdar.2019.00156 , booktitle=

    2019

  31. [39]

    FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding , volume=

    Rahnemoonfar, Maryam and Chowdhury, Tashnim and Sarkar, Argho and Varshney, Debvrat and Yari, Masoud and Murphy, Robin Roberson , year=. FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding , volume=. doi:10.1109/access.2021.3090981 , journal=

    2021

  32. [40]

    A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge , url=

    Schwenk, Dustin and Khandelwal, Apoorv and Clark, Christopher and Marino, Kenneth and Mottaghi, Roozbeh , year=. A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge , url=. doi:10.1007/978-3-031-20074-8_9 , booktitle=

    doi

  33. [41]

    Towards VQA Models That Can Read , url=

    Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus , year=. Towards VQA Models That Can Read , url=. doi:10.1109/cvpr.2019.00851 , booktitle=

    2019

  34. [42]

    Chandra and Kumar, P

    Mohan, B. Chandra and Kumar, P. Sumanth and Madhavarao, V. and Srinivasarao, N. , year =

  35. [43]

    2020 , eprint=

    PathVQA: 30000+ Questions for Medical Visual Question Answering , author=. 2020 , eprint=

    2020

  36. [44]

    Multimodal A r X iv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models

    Li, Lei and Wang, Yuqi and Xu, Runxin and Wang, Peiyi and Feng, Xiachong and Kong, Lingpeng and Liu, Qi. Multimodal A r X iv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  37. [45]

    The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , year =

    IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning , author =. The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , year =

  38. [46]

    International Conference on Learning Representations (ICLR) , year =

    Lu, Pan and Bansal, Hritik and Xia, Tony and Liu, Jiacheng and Li, Chunyuan and Hajishirzi, Hannaneh and Cheng, Hao and Chang, Kai-Wei and Galley, Michel and Gao, Jianfeng , title =. International Conference on Learning Representations (ICLR) , year =

  39. [47]

    C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

    Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul. C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. Findings of the Association for Computational Linguistics: ACL 2022. 2022. doi:10.18653/v1/2022.f...

    2022 doi

  40. [48]

    2026 , eprint=

    PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science , author=. 2026 , eprint=

    2026

  41. [49]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics. 2014. doi:10.1162/tacl_a_00166

    2014 doi

  42. [50]

    2024 , eprint=

    PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering , author=. 2024 , eprint=

    2024

  43. [51]

    arXiv preprint arXiv:2506.05453 , year=

    Mllm-cl: Continual learning for multimodal large language models , author=. arXiv preprint arXiv:2506.05453 , year=

  44. [52]

    Nature Machine Intelligence , volume=

    Shortcut learning in deep neural networks , author=. Nature Machine Intelligence , volume=. 2020 , publisher=

    2020

  45. [53]

    International conference on machine learning , pages=

    On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  46. [54]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    arXiv

  47. [55]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Distance-based image classification: Generalizing to new classes at near-zero cost , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2013 , publisher=

    2013

  48. [56]

    Psychometrika , volume=

    Hierarchical clustering schemes , author=. Psychometrika , volume=. 1967 , publisher=

    1967

  49. [57]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Multiple Queries with Multiple Keys: A Precise Prompt Matching Paradigm for Prompt-based Continual Learning , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  50. [58]

    Neural Networks , pages=

    Embedding Space Allocation with Angle-Norm Joint Classifiers for few-shot class-incremental learning , author=. Neural Networks , pages=. 2025 , publisher=

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.