Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Patent Figure Classification using Large Vision-language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A series of multiple-choice questions lets a vision-language model classify patent figures by type, projection, object, and USPC class, with few-shot fine-tuning and a tournament-style strategy that scales to thousands of labels.

desk verdict Useful new datasets and a pragmatic LVLM tournament classifier, but the headline advantage over CNNs may come from seeing class names in prompts rather than from visual understanding. read the letter →

arxiv 2501.12751 v1 pith:4NDXQ6UM submitted 2025-01-22 cs.IR cs.CVcs.LG

Sushil Awale , Eric Müller-Budack , Ralph Ewerth This is my paper · ORCID
classification cs.IRcs.CVcs.LG
keywords patentfigureclassificationlargevision-languagemodelsvisualquestionansweringfew-shotlearningtournament-styleretrievalPatFigVQACLS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large vision-language models (LVLMs), previously untested on patent figures, can be adapted to this domain with modest amounts of training data and can compete with, or beat, traditional CNN-based classifiers on several classification aspects. The authors introduce two new resources, PatFigVQA for visual question answering and PatFigCLS for classification, and propose a tournament-style multiple-choice strategy that keeps the number of model queries logarithmic in the number of candidate classes. If the approach is correct, patent retrieval systems could add faceted figure search over type, projection, patent class, and depicted objects using a single fine-tuned LVLM instead of separate supervised classifiers.

What carries the argument

The central object is the tournament-style multiple-choice classification strategy (MC-TS), which partitions a large concept set $\mathcal{C}$ into smaller subsets of size $k$, asks the LVLM a multiple-choice question over each subset, then iteratively pits the winning concepts against each other in subsequent rounds until one concept remains. The number of rounds is $R = \lceil \log_k |\mathcal{C}| \rceil$, so a dataset with thousands of classes requires only a handful of queries per figure instead of one query per class. This mechanism is what makes LVLM-based classification computationally feasible for the 1,447-object and 32-class settings.

What would settle it

Re-annotate a random sample of test figures from PatFigCLS with patent experts, then recompute Top-1 accuracy for MC-TS and the CNN baselines on the clean labels; if the reported LVLM advantages over CNNs largely vanish on clean labels, the published numbers would be measuring agreement with noisy references rather than true classification ability.

Watch

Extended reading notes

Core claim

The paper claims that a fine-tuned InstructBLIP model, queried through a tournament of multiple-choice questions, outperforms CNN-based supervised classifiers for two of four patent-figure classification aspects (Type and USPC) and consistently outperforms the binary-question classification (BC) baseline across all aspects. The same model, fine-tuned on the newly introduced PatFigVQA dataset with as few as 150 samples per concept, improves monotonically with training data and starts to saturate around 80 samples per concept. The authors also report that multiple-choice questions degrade as the number of options grows, and that open-ended classification remains competitive on Projection and Object, but that the tournament strategy offers the best balance of accuracy and query efficiency for large concept sets.

Load-bearing premise

The ground-truth labels for Projection, Object, and USPC come from DeepPatent2, where concepts were extracted automatically from figure references and were not normalized; the authors normalize them with rule-based keyword mapping and embedding-based clustering, then treat those normalized labels as ground truth for both training and evaluation.

Editorial extensions

If this is right

  • Patent examiners could use faceted figure search to filter prior art by figure type, projection, USPC class, and depicted objects without training a separate model for each aspect.
  • Few-shot fine-tuning with just dozens of samples per concept is enough to close much of the domain gap between natural-image-pretrained LVLMs and patent drawings.
  • The tournament-style querying scheme extends to any LVLM classification task with a large label space, since the number of model calls grows logarithmically rather than linearly with class count.
  • The reported saturation around 80 samples per concept suggests that further gains will come from better visual discrimination or label quality rather than from simply adding more training images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the DeepPatent2-derived labels are as noisy as the authors suspect, a human-cleaned version of PatFigCLS might reveal larger LVLM advantages over CNNs on Projection and Object, because the SemEq metric already shows that LVLM predictions are often semantically equivalent to the reference.
  • The tournament mechanism is label-order agnostic only if the LVLM treats option order fairly; a useful stress test would be to shuffle option order across runs and check variance, since long-context sensitivity is known to affect multiple-choice performance.
  • The same querying scheme could be applied to hierarchical patent classification (e.g., IPC subclasses) by running tournaments within sibling groups first and then a final tournament across group winners, mirroring the hierarchy of the label space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces two new resources, PatFigVQA and PatFigCLS, for patent figure visual question answering and classification, and studies three LVLM-based classification strategies (binary, multiple-choice tournament, and open-ended) using InstructBLIP with FlanT5-XL. The central empirical claim is that the proposed tournament-style MC-TS approach outperforms CNN baselines (ResNet50, ResNext101) for Type and USPC classification in few-shot settings, while remaining competitive or superior to other LVLM-based approaches. The authors also propose a semantic equivalence metric (SemEq) to mitigate label noise inherited from the DeepPatent2 corpus.

Significance. If the claims hold, the paper provides a useful step toward applying LVLMs to patent figure understanding: it contributes publicly available datasets, a practical tournament-style strategy for handling large label sets, and a comparative evaluation against established CNN baselines. The strength of the paper is its empirical grounding on held-out data, with multiple aspects (Type, Projection, USPC, Object) and both zero-shot and few-shot settings. The main significance is relatively incremental, however, because the headline result ('outperforms CNNs for Type and USPC') is central and needs to be demonstrated under comparable information conditions before the LVLM advantage can be attributed to visual understanding rather than to the presence of class names in the prompt.

major comments (4)
  1. [Section 4.3, Table 2] The comparison between MC-TS and the CNN baselines is confounded by asymmetric access to class labels. MC-TS is prompted with multiple-choice options that literally contain the candidate class names (e.g., 'edible products', 'apparel and haberdashery'), whereas the CNN baselines are trained as image-to-index classifiers that never see class names or any semantic description. For USPC, whose class definitions are broad and linguistically informative, the LVLM can exploit pretrained language priors even when visual evidence is weak. Without an ablation that removes this asymmetry (e.g., giving CNNs access to text embeddings via CLIP or a text-initialized classifier head, or removing class-name semantics from the MC-TS prompt), the claim that MC-TS outperforms CNNs for Type and USPC is not established as a claim about visual figure classification. This is load-bearing for the central contribution and should be addressed experimentally.
  2. [Section 3.2, Table 1] The ground-truth labels for Projection, Object, and USPC derive from DeepPatent2, in which concepts were automatically extracted from figure references and are not normalized. The authors normalize them using rule-based keyword mapping and embedding-based clustering, but they do not report validation of the resulting labels, such as human agreement on a sample or comparison with the original references. This is especially relevant for the Object test set, which contains exactly one figure per concept (1,447 test samples). With one item per class, any label noise or normalization error directly becomes measured accuracy error, and the reported accuracies may reflect agreement with noisy references rather than true classification ability. The paper should either provide evidence that the normalized labels are reliable or treat the labels as a weak/soft supervision setting and temper the accuracy-based claims accordingly.
  3. [Section 4.1, Table 2] All reported accuracies appear to come from a single training run per configuration, with no error bars, multiple seeds, or statistical significance tests. Several comparisons in Table 2 are close (e.g., few-shot MC-TS (10) vs MC-TS (5) for Type: 87.12 vs 87.98; zero-shot MC-TS (10) vs MC-TS (20) for USPC SemEq: 25.70 vs 19.50), and without variance estimates it is unclear which differences are reliable. The paper's central claims about outperforming CNNs and about the tournament size k would be stronger with at least 3-5 runs and standard deviation reporting.
  4. [Section 4.3, SemEq metric] The SemEq metric uses FlanT5-XXL, which belongs to the same FlanT5 family as the backbone (FlanT5-XL) used in InstructBLIP, and it is validated on only 200 samples with a Cohen's Kappa of 0.59 (moderate agreement). Because SemEq is used to reinterpret the accuracy results for USPC and Object, the paper should report whether the SemEq conclusions are robust to the choice of the LLM evaluator or to the threshold for semantic equivalence. As written, the SemEq improvements could partly reflect evaluator bias rather than genuine semantic agreement with the reference labels.
minor comments (5)
  1. [Section 3.2] The text states that DeepPatent2 covers '33 USPC classes' and later says the 'Miscellaneous' concept was excluded, resulting in '32 USPC' in Table 1; this transition should be stated explicitly when the final concept count is introduced.
  2. [Section 3.2] The clustering procedure for Objects is underspecified: the embedding model (PatentBERT is mentioned later, but the exact checkpoint and pooling) and the number of clusters or the similarity threshold are not given. This makes the normalization step hard to reproduce.
  3. [Section 4.1] There are several typos and wording issues, e.g., 'we postpend short instructions' (should be 'append'), 'the LLM bakbone FlanT5' (should be 'backbone'), and 'using both both traditional and deep learning methods' in Section 2. These should be corrected in a revision.
  4. [Figure 3] The y-axis label says 'Accuracy (%)' but the values are plotted as fractions (0.1 to 0.9). Either change the axis to percent (10-90) or change the label to 'Accuracy'.
  5. [Section 3.3] The notation for the number of queries in MC-TS appears to have a typo: the formula reads 'N = PR r=1 l |C| kr m' which is not rendered correctly; it should be a summation of ceil(|C|/k^r). Please clarify the formula.

Circularity Check

0 steps flagged · score 2.0 of 10

Empirical study with no reduction-by-construction; minor same-group dataset citation is not load-bearing.

full rationale

This paper does not present a derivation chain; its claims are empirical accuracies on held-out test splits. The PatFigCLS/PatFigVQA datasets are constructed from external corpora (CLEF-IP 2011 and DeepPatent2), and the reported numbers for MC-TS, BC, OC, and CNN baselines are measured on test samples not used for fine-tuning. No fitted parameter is renamed as a prediction, and no equation reduces a reported quantity to its input. The use of Ghauri et al. [14] for the Type validation/test splits and the Extended CLEF-IP labels is a same-group citation, but it supplies the benchmark split, not the conclusion; the Type comparison is still measured on held-out data. The SemEq metric is adapted from LAVE and validated against human annotation (kappa = 0.59, IAA = 77%), so it is not merely an internal consistency check. The acknowledged noise in DeepPatent2's automatically extracted labels is a data-quality limitation, not a circular step. Overall, no central claim is equivalent to its inputs by construction; the only shading is minor non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper rests on the reliability of automatically extracted patent labels, on embedding-based label normalization, and on a hand-built projection schema. These are data-construction choices rather than theoretical axioms, but they carry the load: every accuracy number inherits their errors.

free parameters (5)
  • Minimum figures per concept = 150
    Concepts with fewer than 150 figures are removed from the PatFigCLS train set; this shapes the label space and the few-shot setup (Section 3.2).
  • Object concept clustering output size = 1,447 canonical object classes from 132,890 raw concepts
    Raw DeepPatent2 object labels are clustered via PatentBERT embeddings; the final number of classes is an algorithmic choice that defines the Object label space and the one-figure-per-concept test set (Section 3.2).
  • Projection schema classes = 7 classes (axonometric, detail, elevational, oblique, perspective, plan, sectional)
    Hand-designed extension of Carlbom et al. based on Radford's architectural drawing categories; used to map noisy projection labels (Section 3.2).
  • Tournament subset size k = 5, 10, or 20
    MC-TS subset size is chosen by hand; results vary with k, for example Type accuracy of 87.98 for k=5 versus 87.12 for k=10 (Section 4.3).
  • Option sampling pool size for Object = 100 most similar concepts per concept
    Multiple-choice options for Object questions are sampled from the 100 nearest PatentBERT neighbors of the correct concept, which affects question difficulty (Section 3.2).
assumptions (5)
  • domain assumption DeepPatent2 auto-extracted figure-reference labels are accurate enough to serve as ground truth after normalization.
    Section 3.2 builds PatFigCLS labels from these extractions; the paper acknowledges they are not normalized and may be noisy.
  • domain assumption PatentBERT embeddings capture patent-relevant semantics for object concepts.
    Used both for clustering object labels and for selecting multiple-choice options; no external validation of the clustering quality is provided (Section 3.2).
  • ad hoc to paper The seven-class projection schema is an appropriate taxonomy for design patent figures.
    The schema is assembled from Carlbom et al. and Radford's architectural drawing book; it is introduced for this paper and not validated against an external standard (Section 3.2).
  • domain assumption InstructBLIP is a representative LVLM for this task.
    All LVLM results use InstructBLIP with FlanT5-XL; no comparison with other LVLMs such as LLaVA is performed (Section 4.1).
  • domain assumption LLM-based SemEq scores reflect semantic correctness.
    FlanT5-XXL equivalence judgments agree with a human annotator on only a 200-sample subset, with kappa 0.59 and IAA 77% (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Patent Figure Classification using Large Vision-language Models." pith.science (2026). https://pith.science/paper/4NDXQ6UM

@misc{pith2026250112751,
  author       = {Pith},
  title        = {Pith review of: Patent Figure Classification using Large Vision-language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NDXQ6UM}},
  note         = {Machine review of arXiv:2501.12751}
}
read the original abstract

Patent figure classification facilitates faceted search in patent retrieval systems, enabling efficient prior art search. Existing approaches have explored patent figure classification for only a single aspect and for aspects with a limited number of concepts. In recent years, large vision-language models (LVLMs) have shown tremendous performance across numerous computer vision downstream tasks, however, they remain unexplored for patent figure classification. Our work explores the efficacy of LVLMs in patent figure visual question answering (VQA) and classification, focusing on zero-shot and few-shot learning scenarios. For this purpose, we introduce new datasets, PatFigVQA and PatFigCLS, for fine-tuning and evaluation regarding multiple aspects of patent figures~(i.e., type, projection, patent class, and objects). For a computational-effective handling of a large number of classes using LVLM, we propose a novel tournament-style classification strategy that leverages a series of multiple-choice questions. Experimental results and comparisons of multiple classification approaches based on LVLMs and Convolutional Neural Networks (CNNs) in few-shot settings show the feasibility of the proposed approaches.

Figures

Figures reproduced from arXiv: 2501.12751 by the authors.

Figure 1
Figure 1. A figure from patent USD534354S1 showing a drawing of a modular tool storage drawer in cross-sectional projection, and three different questions asking about various aspects of the figure: 1) Binary question asking about projection 2) Multiple-choice question asking about figure type, and 3) Open-ended question asking about object depicted. The figure also shows the response (and token probability) generated from an… view at source ↗
Figure 2
Figure 2. Workflow of patent figure classification using different LVLM-based classifica￾tion approaches. On the left, question templates for different question types used to create PatFigVQA dataset are shown. On the right, three different approaches to figure classification using a fine-tuned LVLM is shown, which include Binary Classifi￾cation [BC], Multiple-choice Classification - Tournament-style Strategy [MC-TS], and Ope… view at source ↗
Figure 3
Figure 3. Average exact string matching accuracy of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Confusion matrix on results produced by fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Qualitative examples for aspects Object (rows 1 and 2) and USPC (rows 3 and 4) comparing classification results of MC-TS and ResNext101 ing Type and Projection for patent figures. Similar to the performance in the VQA setting, the LVLM struggles to distinguish between …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages

  1. [1]

    Scientific Data 10(1), 772 (Nov 2023)

    Ajayi, K., Wei, X., Gryder, M., Shields, W., Wu, J., Jones, S.M., Kucer, M., Oyen, D.: DeepPatent2: A Large-Scale Benchmarking Corpus for Tech- nical Drawing Understanding. Scientific Data 10(1), 772 (Nov 2023). https://doi.org/10.1038/s41597-023-02653-7

    work page doi:10.1038/s41597-023-02653-7 2023

  2. [2]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVRW 2020, Online Event, June 14-19, 2020

    Bhattarai, M., Oyen, D., Castorena, J., Yang, L., Wohlberg, B.: Diagram image retrieval using sketch-based deep learning and transfer learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVRW 2020, Online Event, June 14-19, 2020. pp. 663–672. Computer Vision Foundation / IEEE (2020). https://doi.org/10.1109/CVPRW50498.2020.00095

    arXiv 2020

  3. [3]

    ACM Computing Surveys 10(4), 465–502 (Dec 1978)

    Carlbom, I., Paciorek, J.: Planar geometric projections and viewing transformations. ACM Computing Surveys 10(4), 465–502 (Dec 1978). https://doi.org/10.1145/356744.356750

    work page arXiv 1978

  4. [4]

    Educational Psychology Review 14(1), 5–26 (2002), http://www.jstor.org/stable/23363486

    Carney, R.N., Levin, J.R.: Pictorial illustrations still improve students’ learning from text. Educational Psychology Review 14(1), 5–26 (2002), http://www.jstor.org/stable/23363486

    work page arXiv 2002

  5. [7]

    Journal of Machine Learning Research25(70), 1–53 (2024), http://jmlr.org/papers/v25/23-0870.html

    Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E.H., Dean, J., Devlin, J., Roberts, A....

    work page 2024

  6. [8]

    In: CLEF 2011 Labs and Workshop, Notebook Papers, Amsterdam, The Netherlands, Septe- meber 19-22, 2011

    Csurka, G., Renders, J., Jacquet, G.: Xrce’s participation at patent image clas- sification and image-based patent retrieval tasks of the clef-ip 2011. In: CLEF 2011 Labs and Workshop, Notebook Papers, Amsterdam, The Netherlands, Septe- meber 19-22, 2011. vol. 1177. CEUR-WS.org (2011), https://ceur-ws.org/Vol- 1177/CLEF2011wn-CLEF-IP-CsurkaEt2011.pdf

    work page 2011

  7. [9]

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.C.H.: Instructblip: Toward and relative positios general-purpose vision-language models with instruction tuning. In: Advances in Neural Information Process- ing Systems, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023 (2023), http://papers.nips.cc/paper_files/pap...

    work page 2023

  8. [10]

    A Survey on Figure Classification Techniques in Scientific Documents

    Dhote, A., Javed, M., Doermann, D.S.: A survey on figure classifica- tion techniques in scientific documents. CoRR abs/2307.05694 (2023). https://doi.org/10.48550/ARXIV.2307.05694

    work page Pith review arXiv doi:10.48550/arxiv.2307.05694 2023

Show all 48 references
  1. [11]

    In: IEEE Interna- tional Conference on Software Quality, Reliability, and Security, QRS 2023, Chiang Mai, Thailand, October 22-26, 2023

    Ding, J., Chen, H., Kolapudi, S., Pobbathi, L., Nguyen, H.: Quality eval- uation of summarization models for patent documents. In: IEEE Interna- tional Conference on Software Quality, Reliability, and Security, QRS 2023, Chiang Mai, Thailand, October 22-26, 2023. pp. 250–259. ...

    2023

  2. [12]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023

    Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: Eva: Exploring the limits of masked visual representation learning at scale. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, ...

    2023

  3. [13]

    IEEE Access 11, 76202–76221 (2023)

    Farahani, A.M., Adibi, P., Ehsani, M.S., Hutter, H.P., Darvishy, A.: Auto- matic chart understanding: A review. IEEE Access 11, 76202–76221 (2023). https://doi.org/10.1109/ACCESS.2023.3298050

    2023

  4. [14]

    In: International Conference on Theory and Practice of Digital Libraries, TPDL 2023, Zadar, Croatia, September 26-29, 2023

    Ghauri, J.A., Müller-Budack, E., Ewerth, R.: Classification of visualization types and perspectives in patents. In: International Conference on Theory and Practice of Digital Libraries, TPDL 2023, Zadar, Croatia, September 26-29, 2023. pp. 182–

    2023

  5. [15]

    In: International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 (2024), https://arxiv.org/abs/2402.07270

    Ging, S., Bravo, M.A., Brox, T.: Open-ended VQA benchmarking of vision- language models by exploiting classification datasets and their semantic hierarchy. In: International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 (2024), https://arxi...

    2024 arXiv

  6. [16]

    In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recog- nition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

    2016 doi

  7. [17]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, USA, June 21-24, 2022

    Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., Wang, L.: Scal- ing up vision-language pre-training for image captioning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, USA, June 21-24, 2022. pp. 17980–17989 (June 2022). https...

    2022

  8. [18]

    In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, ASME 2020, Virtual Event, August 17-19, 2020

    Jiang, S., Luo, J., Pava, G.R., Hu, J., Magee, C.L.: A convolutional neural network- based patent image retrieval method for design ideation. In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, ASME 2020, Virtual ...

    2020 doi

  9. [19]

    Jobin, K.V., Mondal, A., Jawahar, C.V.: Docfigure: A dataset for scientific doc- ument figure classification. In: International Workshop on Graphics Recognition co-located with International Journal on Document Analysis and Recognition, GREC@ICDAR 2019, Sydney, Australia, Sept...

    2019

  10. [20]

    Information Technologies in Science, Management, Social sphere and Medicine

    Kravets, A., Lebedev, N., Legenchenko, M.: Patents images retrieval and convo- lutional neural network training dataset quality improvement. In: International research conference on "Information Technologies in Science, Management, Social sphere and Medicine", ITSMSSM 2017, To...

    2017 doi

  11. [21]

    World Patent Information 65, 102035 (2021)

    Krestel, R., Chikkamath, R., Hewel, C., Risch, J.: A survey on deep learning for patent analysis. World Patent Information 65, 102035 (2021). https://doi.org/https://doi.org/10.1016/j.wpi.2021.102035 Patent Figure Classification using LVLMs 15

    2021

  12. [22]

    In: IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022

    Kucer,M.,Oyen,D.,Castorena,J.,Wu,J.:Deeppatent:Largescalepatentdrawing recognition and retrieval. In: IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022. pp. 557–

    2022

  13. [23]

    In: Bollegala, D., Huang, R., Ritter, A

    Li, D., Li, J., Le, H., Wang, G., Savarese, S., Hoi, S.C.H.: LAVIS: A one-stop li- brary for language-vision intelligence. In: Bollegala, D., Huang, R., Ritter, A. (eds.) Annual Meeting of the Association for Computational Linguistics: System Demon- strations, ACL 2023, Toront...

    2023 doi

  14. [24]

    In: International Conference on Machine Learning, ICML 2023, Hon- olulu, Hawaii, USA, July 23-29, 2023

    Li, J., Li, D., Savarese, S., Hoi, S.C.H.: BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language mod- els. In: International Conference on Machine Learning, ICML 2023, Hon- olulu, Hawaii, USA, July 23-29, 2023. pp. 19730–19742. PMLR (...

    2023

  15. [25]

    https://doi.org/10.48550/ARXIV.2308.03349

    Li, S., Tajbakhsh, N.: Scigraphqa: A large-scale synthetic multi- turn question-answering dataset for scientific graphs (2023). https://doi.org/10.48550/ARXIV.2308.03349

    2023 doi arXiv

  16. [26]

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023 (2023), http://papers.nips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0- Abstract-C...

    2023

  17. [27]

    Trans- actions of the Association for Computational Linguistics 12, 157–173 (2024)

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Trans- actions of the Association for Computational Linguistics 12, 157–173 (2024). https://doi.org/10.1162/TACL_A_00638

    2024 doi

  18. [28]

    In: International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net (2019), https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  19. [29]

    In: AAAI Conference on Artificial Intelligence, AAAI 2024, Vancouver, Canada, February 20-27, 2024

    Mañas, O., Krojer, B., Agrawal, A.: Improving automatic VQA evaluation using large language models. In: AAAI Conference on Artificial Intelligence, AAAI 2024, Vancouver, Canada, February 20-27, 2024. pp. 4171–4179. AAAI Press (2024). https://doi.org/10.1609/AAAI.V38I5.28212

    2024 doi

  20. [30]

    In: Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ire- land, May 22-27, 2022

    Masry, A., Long, D.X., Tan, J.Q., Joty, S.R., Hoque, E.: Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In: Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ire- land, May 22-27, 2022. pp. 2263–2279...

    2022 doi

  21. [31]

    In: IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020

    Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over sci- entific plots. In: IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020. pp. 1516–1525. IEEE (2020). https://doi.org/10.1109/WACV45572.202...

    2020

  22. [32]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2007, Minneapolis, Minnesota, USA, June 18-23, 2007

    Perronnin, F., Dance, C.R.: Fisher kernels on visual vocabularies for image catego- rization. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2007, Minneapolis, Minnesota, USA, June 18-23, 2007. IEEE Computer So- ciety (2007). https://doi.org/10.1109/C...

    2007

  23. [33]

    In: CLEF 2011 Labs and Workshop, Notebook Papers, Am- sterdam, The Netherlands, September 19-22, 2011

    Piroi, F., Lupu, M., Hanbury, A., Zenz, V.: CLEF-IP 2011: Retrieval in the intellec- tual property domain. In: CLEF 2011 Labs and Workshop, Notebook Papers, Am- sterdam, The Netherlands, September 19-22, 2011. CEUR Workshop Proceedings, 16 Awale et al. vol. 1177. CEUR-WS.org (...

    2011

  24. [34]

    Pustu-Iren, K., Bruns, G., Ewerth, R.: A multimodal approach for se- mantic patent image retrieval. In: Patent Text Mining and Seman- tic Technologies co-located with ACM SIGIR Conference on Research and Development in Information Retrieval, PatentSemTech@SIGIR 2021, Aachen, G...

    2021

  25. [35]

    In: Associa- tion for Computational Linguistics, NAACL 2024, Mexico City, Mexico, June 16-21, 2024

    Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., Bendersky, M.: Large language mod- els are effective text rankers with pairwise ranking prompting. In: Associa- tion for Computational Linguistics, NAACL 2024, Mexic...

    2024 doi

  26. [36]

    In: International Conference on Machine Learning, ICML 2021, Virtual Event, July 18-24, 2021

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, ICML 2021...

    2021

  27. [37]

    The Radford architectural company, Chicago, Ill

    Radford, W.A., Johnson, A.S., Provine, L.H.: Radford’s architectural drawing; complete guide to work of architect’s office, drawing to scale–tracing–detailing– lettering–rendering–designing– classic orders of architecture; a complete and thor- ough course. The Radford architec...

    1912

  28. [38]

    Sciento- metrics 127(1), 207–231 (2022)

    Roudsari, A.H., Afshar, J., Lee, W., Lee, S.: Patentnet: multi-label classification of patent documents using deep learning based language understanding. Sciento- metrics 127(1), 207–231 (2022). https://doi.org/10.1007/S11192-021-04179-4

    2022 doi

  29. [39]

    Seebacher,D.,Stein,M.,Janetzko,H.,Keim,D.A.:Patentretrieval:Amulti-modal visualanalyticsapproach.In:InternationalEuroVisWorkshoponVisualAnalytics, EuroVA@EuroVis 2016, Groningen, The Netherlands, June 6-7, 2016. pp. 13–17. Eurographics Association (2016). https://doi.org/10.23...

    2016 doi

  30. [40]

    In: SIAM Inter- national Conference on Data Mining, SDM 2018, San Diego, CA, USA, May 3-5, 2018

    Shalaby, M., Stutzki, J., Schubert, M., Günnemann, S.: An LSTM approach to patent classification based on fixed hierarchy vectors. In: SIAM Inter- national Conference on Data Mining, SDM 2018, San Diego, CA, USA, May 3-5, 2018. Society for Industrial and Applied Mathematics (2...

    2018 doi

  31. [41]

    In: Bengio, Y., LeCun, Y

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015 (2015), http://arxiv.org/abs/1409.1556

    2015 arXiv

  32. [42]

    White paper (2020), https://services.google.com/fh/files/blogs/bert_for_patents_white_paper.pdf

    Srebrovic, R., Yonamine, J.: Leveraging the bert algorithm for patents with tensorflow and bigquery. White paper (2020), https://services.google.com/fh/files/blogs/bert_for_patents_white_paper.pdf

    2020

  33. [43]

    World Patent Information73, 102192 (2023)

    Vowinckel, K., Hähnke, V.D.: Searchformer: Semantic patent embeddings by siamese transformers for prior art search. World Patent Information73, 102192 (2023). https://doi.org/https://doi.org/10.1016/j.wpi.2023.102192

    2023

  34. [44]

    World Patent Information 34(4), 292–303 (2012)

    Vrochidis, S., Moumtzidou, A., Kompatsiaris, I.: Concept-based patent image retrieval. World Patent Information 34(4), 292–303 (2012). https://doi.org/https://doi.org/10.1016/j.wpi.2012.07.002

    2012 doi

  35. [45]

    World Patent Information 32(2), 94–106 (2010)

    Vrochidis, S., Papadopoulos, S., Moumtzidou, A., Sidiropoulos, P., Pi- anta, E., Kompatsiaris, I.: Towards content-based patent image retrieval: Patent Figure Classification using LVLMs 17 A framework perspective. World Patent Information 32(2), 94–106 (2010). https://doi.org/...

    2010 doi

  36. [46]

    In: Aizawa, A., Mandl, T., Carevic, Z., Hinze, A., Mayr, P., Schaer, P

    Wei, X., Wu, J., Ajayi, K., Oyen, D.: Visual descriptor extraction from patent figure captions: a case study of data efficiency between bilstm and transformer. In: Aizawa, A., Mandl, T., Carevic, Z., Hinze, A., Mayr, P., Schaer, P. (eds.) Joint Conference on Digital Libraries,...

    2022

  37. [47]

    In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017

    Xie,S.,Girshick,R.,Dollár,P.,Tu,Z.,He,K.:Aggregatedresidualtransformations for deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pp. 5987–5995 (2017). https://doi.org/10.1109/CVPR.2017.634

    2017 doi

  38. [48]

    In: International ACM SIGIR Conference on Research and Development in In- formation Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024

    Zhuang, S., Zhuang, H., Koopman, B., Zuccon, G.: A setwise approach for ef- fective and highly efficient zero-shot ranking with large language models. In: International ACM SIGIR Conference on Research and Development in In- formation Retrieval, SIGIR 2024, Washington DC, USA,...

    2024

  39. [191]

    https://doi.org/10.1007/978-3-031-43849-3_16

    Springer (2023). https://doi.org/10.1007/978-3-031-43849-3_16

    2023 doi

  40. [566]

    https://doi.org/10.1109/WACV51458.2022.00063

    IEEE (2022). https://doi.org/10.1109/WACV51458.2022.00063

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.