Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ArtSeek identifies, retrieves, and explains artworks from the image alone.

desk verdict Good engineering, credible classification gains, but the headline captioning result may be an artifact of retrieving the reference Wikipedia page—needs a leakage-controlled rerun. read the letter →

arxiv 2507.21917 v1 pith:BYKOYOND submitted 2025-07-29 cs.CV

Nicola Fanelli , Gennaro Vessio , Giovanna Castellano This is my paper · ORCID
classification cs.CV
keywords artworkunderstandingretrieval-augmentedgenerationmultimodallargelanguagemodelslateinteractionretrievalmultitaskclassificationvisualquestionansweringimage-textfragmentsWiki
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ArtSeek is a framework that takes only a digitized artwork image and returns attribute labels, retrieved background fragments, and a written explanation, without relying on metadata or links to external knowledge bases. The paper claims this works because a late-interaction multimodal retriever searches a large collection of art-related image-text fragments, a contrastive multitask classifier produces an artwork card, and a multimodal language model uses one in-context example to plan retrieval calls and reason over the retrieved evidence. On standard benchmarks the framework reports better scores than prior art-specific systems, including an 8.4-point F1 gain in style classification over GraphCLIP and a 7.1-point BLEU@1 gain on ArtPedia. The reason a reader should care is that image-only operation makes the method applicable to the many digitized artworks that have no knowledge-base entry, and retrieved fragments make the model's reasoning inspectable.

What carries the argument

The load-bearing machinery is late interaction retrieval, a scoring rule that compares every query token embedding with every document token embedding through a MaxSim sum, applied to multimodal fragments. Three adaptations make it work at scale: multimodal query encoding that stitches the user's text into the image prompt and filters out non-text token embeddings; token pooling that collapses each document's many content embeddings into a fixed set of cluster centroids plus one special-token centroid; and a two-stage search that prefetches with binarized pooled embeddings, then reranks with full embeddings. On the classification side, LICN prepends five learned task tokens to each image and one task token to each label text, passes them through a shared transformer, and trains with a sigmoid contrastive loss per task. The agentic loop is steered with a single in-context example that shows the model when to think, when to call the retriever, and how to consume tool output.

What would settle it

Evaluate retrieval on a held-out set of human-written or independently collected questions about artworks, with no access to the target fragment during question writing, and check whether NDCG, recall, and the end-to-end answer gain over the base multimodal model survive; if they do not, the framework's central advantage is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a multimodal large language model can do credible art-historical interpretation when it is grounded in retrieval rather than left to parametric memory. ArtSeek encodes each query as a modified ColQwen2 input in which the default prompt is replaced by the user's question, keeps only the text-token embeddings to make retrieval efficient, and searches over WikiFragments, a five-million-fragment collection built from encyclopedia pages in the visual arts category. The same encoder feeds LICN, a contrastive multitask classifier that predicts artist, genre, style, media, and tags for any class in the label set. The agentic reasoning loop then lets Qwen2.5-VL decide when and what to retrieve, and the final answer can be traced to the retrieved fragments. The paper reports state-of-the-art classification and captioning results and qualitative cases where the model identifies figures or contexts that the base multimodal model cannot.

Load-bearing premise

The load-bearing assumption is that retrieval scores measured on questions a model wrote while looking at the correct answer will hold for real users' questions, which are less aligned with the stored fragments.

Editorial extensions

If this is right

  • Because the pipeline needs only an image, any digitized artwork can be analyzed even when it has no knowledge-base entry, Wikipedia page, or other metadata.
  • Answers become auditable: a user can inspect exactly which fragments were retrieved and used, which helps distinguish grounded reasoning from hallucination.
  • The fragment-knowledge-base construction and its category-recursive filtering can be reapplied to other knowledge-intensive domains beyond visual art.
  • Classification over an open class set means new artists, genres, or styles can be added by supplying label text, without retraining the architecture.
  • A single in-context example is enough to make the 32B multimodal model plan retrieval calls, suggesting that similar agentic retrieval behavior may be obtainable without task-specific fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the retrieval benchmark uses questions generated by a model that was shown the target paragraph, so the measured NDCG and recall may overstate performance on natural user queries; an unbiased test would need independently written questions.
  • Editorial inference: the image-only advantage still depends on the visual-arts encyclopedia category having a fragment that resembles the query artwork; for genuinely obscure works with no near-duplicate text or imagery, retrieval may return irrelevant context and explanation quality will fall.
  • Editorial inference: the same late-interaction architecture could be tested in other image-plus-question domains, such as medical imaging or natural history, where the key question is whether token-pooled retrieval preserves enough fine-grained visual detail.
  • Editorial inference: the claim that one in-context example suffices likely depends on model scale; smaller multimodal models may need more demonstrations or fine-tuning to reproduce the agentic loop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents ArtSeek, a multimodal framework for artwork understanding that combines (i) WikiFragments, a large-scale dataset of image-text fragments drawn from English Wikipedia's Visual Arts category; (ii) a late-interaction retrieval pipeline built on ColQwen2 with a two-stage pooled-then-rerank search; (iii) a contrastive multitask classifier (LICN) for artist, genre, style, media, and tags; and (iv) an agentic Qwen2.5-VL reasoning loop, steered by a single in-context example, that calls the retriever to ground answers. The authors report state-of-the-art classification gains (e.g., +8.4 F1 style over GraphCLIP) and captioning gains (+7.1 BLEU@1 over KALE on ArtPedia), and they provide qualitative examples of retrieval-grounded VQA. A central design principle is that the pipeline needs only an image as input, avoiding reliance on metadata or Wikipedia links at inference time.

Significance. If the reported evaluations hold up, ArtSeek is a substantial practical contribution: it demonstrates that image-only, retrieval-augmented reasoning can beat art-specific models on several benchmarks, and the WikiFragments dataset plus code release would benefit the community. The classification results are benchmarked against external datasets (ArtGraph/WikiArt, ArtPedia, SemArt, PaintingForm), and the ablations consistently show gains from adding retrieval and classification. The main caveats are in the evaluation design: the retrieval benchmark is built from model-generated queries conditioned on the target fragment, and the ArtPedia captioning benchmark may be contaminated because the reference captions are Wikipedia-derived while the retrieval index contains Wikipedia pages. These issues are fixable and do not by themselves invalidate the system, but they currently overstate the strength of the headline evidence.

major comments (3)
  1. [Sec. 5.1.1 and Table 2] The retrieval evaluation is self-referential: the 10,000 queries are generated by prompting Qwen2.5-VL-32B with the target paragraph and asking for a question answerable by that paragraph. This measures the system's ability to retrieve a fragment when the query is designed to match it, not its ability to handle natural user queries such as those shown in the qualitative VQA examples. The reported NDCG@5 of 27.61 and R@1 of 23.87 for the recommended configuration are therefore not a reliable estimate of retrieval quality in actual use. Please re-evaluate with independently written queries (e.g., from human annotators or a held-out set of real user questions) and report retrieval quality on those queries, ideally along with a comparison of the synthetic-query results.
  2. [Sec. 5.3.1 and Table 5] The ArtPedia captioning comparison is vulnerable to index leakage. ArtPedia descriptions are Wikipedia-based, and WikiFragments indexes English Wikipedia pages in the Visual Arts category; for a given ArtPedia painting, the page from which the reference caption is drawn is very likely present in the index. ArtSeek retrieves up to 10 fragments and feeds them to the MLLM, so it can copy or paraphrase text that overlaps the ground-truth reference, whereas the compared captioning systems (KALE, Bai et al., Wu) cannot access that source at inference. The paper reports no overlap analysis between retrieved fragments and reference captions and no ablation that removes the source page from the index. Please rerun the generation experiments with the source pages (or pages overlapping the reference captions beyond a threshold) excluded from the index, and report the resulting BLEU/SPICE/ROUGE numbers. The +7.1 BLEU@1 and SPICE gains are the headline evidence and need to be shown robust to this leakage.
  3. [Sec. 4.1, Eqs. (2)-(3)] The central retrieval mechanism replaces ColQwen2's training-time text prompt with the user query inside the image input sequence. The decomposition into functional subsequences (Ipref, Icontent, Itext, Isuff) is plausible, but no evidence is given that substituting an arbitrary user query for the model's training prompt preserves a valid input distribution. Table 2 compares this strategy only against CLIP and against using all query embeddings, not against a standard alternative such as concatenating an image caption with the query for text-only retrieval, or against a retriever fine-tuned for multimodal queries. Since the final pipeline's RAG advantage depends on this untrained modification, please add such a baseline or provide an analysis of where the filtered query embeddings derive their signal.
minor comments (5)
  1. [Sec. 4.1, Eq. (5)] The text says 'we concatenate the special token centroid with the eight cluster centroids', while Eq. (5) uses K cluster centroids and K is later set to 8. Please make the notation consistent by defining K generically and then stating K=8 in the experiments.
  2. [Sec. 5.3.2] The claim of 'state-of-the-art results on multiple benchmarks' is too broad: on PaintingForm, GalleryGPT [7] outperforms ArtSeek on BLEU and ROUGE, and on SemArt v2.0, Bai et al. [26] and Jiang et al. [8] achieve higher METEOR/ROUGE. Please restrict the SOTA claim to the specific datasets and metrics where it actually holds.
  3. [Table 5] The captioning results are reported as point estimates without confidence intervals or significance tests; the differences between ArtSeek and ArtSeek (w/o classification) are small (e.g., BLEU@1 39.7 vs 39.64) and may be within noise. Please add error bars or a statistical test.
  4. [Sec. 4.1 and Table 2] The hyperparameters K, N1, N2, and the number of clusters in token pooling are fixed without a sensitivity analysis. Since the efficiency and effectiveness trade-off is a stated contribution, please include an ablation varying K and N1 (or at least justify the chosen values).
  5. [Figure 5] The caption mentions that 'the second input to the retriever function specifies whether the retrieval should be multimodal, using both image and text, or purely textual', but this interface is not described in the text. Please define the retriever tool signature and how the MLLM decides between multimodal and text-only retrieval.

Circularity Check

2 steps flagged · score 6.0 of 10

Retrieval evaluation is self-conditioned on the target fragments, and the ArtPedia captioning benchmark may index the reference text; classification comparisons remain externally grounded.

  1. self definitional [Section 5.1.1 (experimental setting) and Section 5.1.2 (metrics and discussion)]
    "To build multimodal queries, we used the removed images and prompted Qwen-2.5-VL-32B to generate a question about each image that could be answered using the corresponding paragraph. ... Recall measures whether the original fragment (from which the query was derived) was successfully retrieved."

    The query is generated specifically so that one particular fragment contains the answer, and the retrieval score is computed against that same fragment as the relevant item. NDCG@5 and R@1 therefore measure how well the retriever finds the paragraph that was used to synthesize the query, not how well it answers natural user queries. The conclusion that the results 'confirm the effectiveness of our multimodal query encoding strategy' is a self-consistency check: the evaluation target is defined from the retrieval target by construction. This does not invalidate the component's engineering trade-offs, but it cannot support a claim of general retrieval quality.

  2. other [Section 5.3.1 (experimental setting) and Table 5 (ArtPedia rows)]
    "We evaluated on three datasets: ArtPedia [10], with Wikipedia-based artwork descriptions; ... Retrieval was based on ColQwen2 with our multi-stage strategy, applied over the full WikiFragments dataset (5,651,060 visual arts-related text fragments)."

    ArtPedia's reference captions are Wikipedia-based, while WikiFragments is the English Wikipedia Visual Arts subset used as the retrieval index. The pipeline retrieves N2=10 fragments and passes them to Qwen2.5-VL before the generated text is scored against the ArtPedia reference. If the source article behind a reference is present in the index, the model can copy or paraphrase the answer key at inference, something the prior captioning baselines could not do. The paper reports no exclusion or overlap analysis, so the headline +7.1 BLEU@1 and SPICE gains are not shown to measure image-grounded generation rather than retrieval of the reference text.

full rationale

The classification contribution is not circular: LICN is trained and evaluated on the external WikiArt/ArtGraph label set, and the comparison with GraphCLIP, although a same-group baseline, is a legitimate benchmark with independent content. No uniqueness theorem is imported, and the self-citations to ArtGraph and GraphCLIP are not load-bearing arguments. However, two evaluation steps are self-referential. First, the retrieval benchmark in Sec. 5.1.1 constructs every query from the target fragment and then scores retrieval against that same fragment, so Table 2 measures a self-consistency property of the query-generation loop rather than retrieval quality for unconstrained queries. Second, the ArtPedia captioning evaluation uses a Wikipedia-derived retrieval index together with a Wikipedia-derived reference set; without leakage control, the top-10 retrieved fragments can contain the reference text itself, making the reported captioning gains partly a measure of copying the answer key. These issues affect the headline captioning and retrieval claims, but not the externally grounded classification results, so the overall circularity score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims rest on several hand-chosen hyperparameters, the internal-structure assumption of ColQwen2, the sufficiency of Wikipedia as a knowledge base, the accuracy of ArtGraph labels, and the reliability of a single in-context example for steering the agentic loop. None of these are derived from first principles, and each is validated only through the paper's own experiments.

free parameters (5)
  • K (number of clusters in token pooling) = 8
    Chosen by hand in Sec 4.1 for document embedding pooling; retrieval quality and runtime trade-off tested only at this value.
  • N1 (prefetch candidates) = 100
    First-stage candidate count in two-stage retrieval, set in Secs 5.1.1 and 5.3.1.
  • N2 (reranked results) = 5 or 10
    Number of final retrieved documents: 5 in Sec 5.1.1 and 10 in Sec 5.3.1.
  • SigLIP temperature c and bias b = learned during training
    Learnable scaling and bias in the contrastive loss (Eq. 7), fitted on the classification training split.
  • Category depth for WikiFragments selection = 5 levels
    Depth of Wikipedia category traversal used to subset fragments to Visual Arts, reported in Sec 3.
assumptions (4)
  • ad hoc to paper The ColQwen2 input embeddings decompose into functional subsequences and replacing the training-time prompt text with the user query preserves a valid multimodal query representation.
    Invoked in Sec 4.1, Eqs. 2-3; the method depends on this internal-structure assumption, which is only tested indirectly through retrieval metrics.
  • domain assumption Wikipedia fragments are a sufficient and reliable knowledge source for art interpretation.
    The entire RAG design assumes Wikipedia's visual-arts pages contain the contextual knowledge needed for the target question types; no external knowledge-grounded evaluation against other sources is provided.
  • domain assumption ArtGraph labels (artist, genre, style, media, tags) are accurate and complete for the 116,475 WikiArt images.
    Classification training and evaluation in Sec 5.2.1 rely on these labels as ground truth, and ArtGraph is the authors' own prior dataset.
  • ad hoc to paper A single manually curated in-context example is sufficient for Qwen2.5-VL-32B to acquire the agentic retrieval-and-reasoning loop.
    Sec 4.3 states that system-prompt steering failed and one example worked; this is a behavioral assumption about the model, with no systematic evaluation of prompt robustness.
invented entities (1)
  • Multimodal fragment independent evidence
    purpose: Atomic retrievable unit combining one Wikipedia paragraph with images appearing above it, rendered as a grid image with captions and text for ColQwen2 encoding.
    This is a new data representation introduced in Sec 3. Its effectiveness is supported by the retrieval experiments in Sec 5.1, but those experiments use Qwen-generated queries conditioned on the target fragments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval." pith.science (2026). https://pith.science/paper/BYKOYOND

@misc{pith2026250721917,
  author       = {Pith},
  title        = {Pith review of: ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYKOYOND}},
  note         = {Machine review of arXiv:2507.21917}
}
read the original abstract

Analyzing digitized artworks presents unique challenges, requiring not only visual interpretation but also a deep understanding of rich artistic, contextual, and historical knowledge. We introduce ArtSeek, a multimodal framework for art analysis that combines multimodal large language models with retrieval-augmented generation. Unlike prior work, our pipeline relies only on image input, enabling applicability to artworks without links to Wikidata or Wikipedia-common in most digitized collections. ArtSeek integrates three key components: an intelligent multimodal retrieval module based on late interaction retrieval, a contrastive multitask classification network for predicting artist, genre, style, media, and tags, and an agentic reasoning strategy enabled through in-context examples for complex visual question answering and artwork explanation via Qwen2.5-VL. Central to this approach is WikiFragments, a Wikipedia-scale dataset of image-text fragments curated to support knowledge-grounded multimodal reasoning. Our framework achieves state-of-the-art results on multiple benchmarks, including a +8.4% F1 improvement in style classification over GraphCLIP and a +7.1 BLEU@1 gain in captioning on ArtPedia. Qualitative analyses show that ArtSeek can interpret visual motifs, infer historical context, and retrieve relevant knowledge, even for obscure works. Though focused on visual arts, our approach generalizes to other domains requiring external knowledge, supporting scalable multimodal AI research. Both the dataset and the source code will be made publicly available at https://github.com/cilabuniba/artseek.

Figures

Figures reproduced from arXiv: 2507.21917 by the authors.

Figure 1
Figure 1. Examples from the WikiFragments dataset. (a) A fragment that includes four images with captions and the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ArtSeek pipeline. We construct a knowledge base of image-text fragments from Wikipedia [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Our multi-stage retrieval pipeline. We design an efficient retrieval method over a large-scale knowledge base [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of the contrastive training process for LICN. Given a batch of images [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Example conversation given to the MLLM. Our manually annotated example contains a visual-only question [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: First qualitative example. ArtSeek correctly recognized the author of this painting and searched for information [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Second qualitative example. ArtSeek correctly identifies the author of the painting using the LICN module [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Question answering examples. ArtSeek is capable of answering user queries involving the identification and [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

    cs.HC 2026-08 conditional novelty 6.0 of 10

    An LLM-agent artwork annotation system that combines proactive label suggestions with interaction-driven skill learning reported roughly 50% faster annotation and higher label agreement in a 12-participant study.

Reference graph

Works this paper leans on

49 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    D. G. Stork, Pixels & paintings: Foundations of computer-assisted connoisseurship. John Wiley & Sons, 2023

    work page 2023

  2. [2]

    Leveraging knowledge graphs and deep learning for automatic art analysis,

    G. Castellano, V . Digeno, G. Sansaro, and G. Vessio, “Leveraging knowledge graphs and deep learning for automatic art analysis,” Knowledge-Based Systems, vol. 248, p. 108859, 2022

    work page 2022

  3. [3]

    GraphCLIP: Image-graph contrastive learning for multimodal artwork classification,

    R. Scaringi, G. Fiameni, G. Vessio, and G. Castellano, “GraphCLIP: Image-graph contrastive learning for multimodal artwork classification,” Knowledge-Based Systems, vol. 310, p. 112857, 2025

    work page 2025

  4. [4]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763

    work page 2021

  5. [5]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    ArtGPT-4: Towards artistic-understanding large vision-language models with enhanced adapter,

    Z. Yuan, Y . He, K. Wang, Y . Ye, and L. Sun, “ArtGPT-4: Towards artistic-understanding large vision-language models with enhanced adapter,” arXiv preprint arXiv:2305.07490, 2023

    arXiv 2023

  7. [7]

    Gallerygpt: Analyzing paintings with large multimodal models,

    Y . Bin, W. Shi, Y . Ding, Z. Hu, Z. Wang, Y . Yang, S.-K. Ng, and H. T. Shen, “Gallerygpt: Analyzing paintings with large multimodal models,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 7734–7743

    work page 2024

  8. [8]

    KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

    Y . Jiang, K. A. Ehinger, and J. H. Lau, “KALE: an artwork image captioning system augmented with heterogeneous graph,” arXiv preprint arXiv:2409.10921, 2024

    work page Pith review arXiv 2024

Show all 49 references
  1. [9]

    Colbert: Efficient and effective passage search via contextualized late interaction over bert,

    O. Khattab and M. Zaharia, “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, 2020, pp. 39–48

    2020

  2. [10]

    Artpedia: A new visual-semantic dataset with visual and contextual sentences in the artistic domain,

    M. Stefanini, M. Cornia, L. Baraldi, M. Corsini, and R. Cucchiara, “Artpedia: A new visual-semantic dataset with visual and contextual sentences in the artistic domain,” in Image Analysis and Processing–ICIAP 2019: 20th International Conference, Trento, Italy, September 9–13, ...

    2019

  3. [11]

    Deep learning approaches to pattern extraction and recognition in paintings and drawings: An overview,

    G. Castellano and G. Vessio, “Deep learning approaches to pattern extraction and recognition in paintings and drawings: An overview,” Neural Computing and Applications, vol. 33, no. 19, pp. 12 263–12 282, 2021

    2021

  4. [12]

    Machine learning for cultural heritage: A survey,

    M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James, “Machine learning for cultural heritage: A survey,” Pattern Recognition Letters, vol. 133, pp. 102–108, 2020

    2020

  5. [13]

    Fine-tuning convolutional neural networks for fine art classification,

    E. Cetinic, T. Lipic, and S. Grgic, “Fine-tuning convolutional neural networks for fine art classification,”Expert Systems with Applications, vol. 114, pp. 107–118, 2018

    2018

  6. [14]

    Recognizing image style,

    S. Karayev, M. Trentacoste, H. Han, A. Agarwala, T. Darrell, A. Hertzmann, and H. Winnemoeller, “Recognizing image style,” arXiv preprint arXiv:1311.3715, 2013

    2013 arXiv

  7. [15]

    Large-scale classification of fine-art paintings: Learning the right metric on the right feature,

    B. Saleh and A. Elgammal, “Large-scale classification of fine-art paintings: Learning the right metric on the right feature,” arXiv preprint arXiv:1505.00855, 2015. 17

    2015 arXiv

  8. [16]

    Toward Discovery of the Artist’s Style: Learning to recognize artists by their artworks,

    N. Van Noord, E. Hendriks, and E. Postma, “Toward Discovery of the Artist’s Style: Learning to recognize artists by their artworks,” IEEE Signal Processing Magazine, vol. 32, no. 4, pp. 46–54, 2015

    2015

  9. [17]

    A deep learning approach to clustering visual arts,

    G. Castellano and G. Vessio, “A deep learning approach to clustering visual arts,” International Journal of Computer Vision, vol. 130, no. 11, pp. 2590–2605, 2022

    2022

  10. [18]

    Toward automated discovery of artistic influence,

    B. Saleh, K. Abe, R. S. Arora, and A. Elgammal, “Toward automated discovery of artistic influence,”Multimedia Tools and Applications, vol. 75, pp. 3565–3591, 2016

    2016

  11. [19]

    Wasielewski, Computational formalism: Art history and machine learning

    A. Wasielewski, Computational formalism: Art history and machine learning. MIT Press, 2023

    2023

  12. [20]

    ContextNet: representation and exploration for painting classification and retrieval in context,

    N. Garcia, B. Renoust, and Y . Nakashima, “ContextNet: representation and exploration for painting classification and retrieval in context,” International Journal of Multimedia Information Retrieval , vol. 9, no. 1, pp. 17–30, 2020

    2020

  13. [21]

    How to read paintings: semantic art understanding with multi-modal retrieval,

    N. Garcia and G. V ogiatzis, “How to read paintings: semantic art understanding with multi-modal retrieval,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0

    2018

  14. [22]

    Generating captions for images of ancient artworks,

    S. Sheng and M.-F. Moens, “Generating captions for images of ancient artworks,” inProceedings of the 27th ACM international conference on multimedia, 2019, pp. 2478–2486

    2019

  15. [23]

    A dataset and baselines for visual question answering on art,

    N. Garcia, C. Ye, Z. Liu, Q. Hu, M. Otani, C. Chu, Y . Nakashima, and T. Mitamura, “A dataset and baselines for visual question answering on art,” in Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 2020, pp. 92–108

    2020

  16. [24]

    Iconographic image captioning for artworks,

    E. Cetinic, “Iconographic image captioning for artworks,” in Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part III. Springer, 2021, pp. 502–516

    2021

  17. [25]

    Iconclass: an iconographic classification system,

    L. D. Couprie, “Iconclass: an iconographic classification system,” Art libraries journal, vol. 8, no. 2, pp. 32–49, 1983

    1983

  18. [26]

    Explain me the painting: Multi-topic knowledgeable art description generation,

    Z. Bai, Y . Nakashima, and N. Garcia, “Explain me the painting: Multi-topic knowledgeable art description generation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 5422–5432

    2021

  19. [27]

    Reading wikipedia to answer open-domain questions,

    D. Chen, A. Fisch, J. Weston, and A. Bordes, “Reading wikipedia to answer open-domain questions,” arXiv preprint arXiv:1704.00051, 2017

    2017 arXiv

  20. [28]

    Is GPT-3 all you need for visual question answering in cultural heritage?

    P. Bongini, F. Becattini, and A. Del Bimbo, “Is GPT-3 all you need for visual question answering in cultural heritage?” in European Conference on Computer Vision. Springer, 2022, pp. 268–281

    2022

  21. [29]

    Exploring the Synergy Between Vision-Language Pretrain- ing and ChatGPT for Artwork Captioning: A Preliminary Study,

    G. Castellano, N. Fanelli, R. Scaringi, and G. Vessio, “Exploring the Synergy Between Vision-Language Pretrain- ing and ChatGPT for Artwork Captioning: A Preliminary Study,” in International Conference on Image Analysis and Processing. Springer, 2023, pp. 309–321

    2023

  22. [30]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in International Conference on Learning...

    2021

  23. [31]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., “Flamingo: a visual language model for few-shot learning,” Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  24. [32]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    2023

  25. [33]

    InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning,

    W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, ...

    2023

  26. [34]

    Reveal: Retrieval- augmented visual-language pre-training with multi-source multimodal knowledge memory,

    Z. Hu, A. Iscen, C. Sun, Z. Wang, K.-W. Chang, Y . Sun, C. Schmid, D. A. Ross, and A. Fathi, “Reveal: Retrieval- augmented visual-language pre-training with multi-source multimodal knowledge memory,” in Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2023

  27. [35]

    EchoSight: Advancing Visual-Language Models with Wiki Knowledge,

    Y . Yan and W. Xie, “EchoSight: Advancing Visual-Language Models with Wiki Knowledge,” inFindings of the Association for Computational Linguistics: EMNLP 2024 , Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov...

    2024

  28. [36]

    Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms,

    D. Caffagni, F. Cocchi, N. Moratelli, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara, “Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 1818–1826. 18

    2024

  29. [37]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang et al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  30. [38]

    Toolformer: Language Models Can Teach Themselves to Use Tools,

    T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” inThirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available:...

    2023

  31. [39]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  32. [40]

    Colpali: Efficient document retrieval with vision language models,

    M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo, “Colpali: Efficient document retrieval with vision language models,” in The Thirteenth International Conference on Learning Representations, 2024

    2024

  33. [41]

    Wikiextractor,

    G. Attardi et al., “Wikiextractor,” 2012

    2012

  34. [42]

    Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling,

    B. Clavié, A. Chaffin, and G. Adams, “Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling,”arXiv preprint arXiv:2409.14683, 2024

    2024 arXiv

  35. [43]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  36. [44]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,

    A. Kendall, Y . Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7482–7491

    2018

  37. [45]

    Art History: A Preliminary Handbook (1996)

    R. J. Belton, “Art History: A Preliminary Handbook (1996).”

    1996

  38. [46]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  39. [47]

    Composed image retrieval using contrastive learning and task-oriented CLIP-based features,

    A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, “Composed image retrieval using contrastive learning and task-oriented CLIP-based features,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 20, no. 3, pp. 1–24, 2023

    2023

  40. [48]

    Artwork interpretation,

    Z. Wu, “Artwork interpretation,” Master’s thesis, University of Melbourne, 2022

    2022

  41. [49]

    Artquest: Countering hidden language biases in artvqa,

    T. Bleidt, S. Eslami, and G. De Melo, “Artquest: Countering hidden language biases in artvqa,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 7326–7335. 19

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.