{"id":"2ea9ed51-122e-4d67-8360-20d93318c65a","arxiv_id":"2501.12751","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new dataset pair and a tournament-style multiple-choice query strategy let a fine-tuned InstructBLIP model classify patent figures by type, projection, object, and USPC class, beating CNN baselines on type and USPC.","lead":"This paper tests whether large vision-language models can classify patent drawings by type, projection, object, and USPC class, and introduces two new datasets for that purpose. It also proposes a tournament-style multiple-choice query strategy to handle large label sets, and reports that the approach beats CNN baselines on two of the four classification aspects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MC-TS's reported edge over CNNs for Type/USPC may stem from asymmetric access to class-name semantics, not superior visual classification.","rationale":"The reader's weakest assumption concerned label noise from DeepPatent2, which is indeed a real limitation and is acknowledged in the paper (Section 3.2 and Section 4.3). That concern affects absolute accuracy numbers and the validity of the dataset as ground truth, but it applies roughly symmetrically to both MC-TS and the CNNs, so it does not directly explain the relative advantage of MC-TS. The more load-bearing issue for the central comparative claim is the asymmetric availability of class-name semantics: MC-TS sees the candidate labels as natural language, while the CNN sees only indices. This is not acknowledged or controlled in the paper, and it can systematically inflate MC-TS performance on aspects like USPC where class labels are semantically rich and visual cues are ambiguous. The paper's contribution of the PatFigVQA/PatFigCLS datasets and the tournament-style strategy remains valuable independent of this comparison, and the reader's CONDITIONAL verdict already calls for additional scrutiny. Adding a controlled comparison would either substantiate or weaken the strongest claim; the current evidence does not require overturning the verdict, only reinforcing the conditions.","tokens_in":14254,"tokens_out":5627,"duration_ms":61522,"concrete_test":"Recompute Table 2 using a CNN baseline that receives the same semantic label information, for example by fine-tuning a CLIP model or replacing the CNN's final layer with a linear probe on class-name text embeddings (e.g., CLIP text embeddings of the USPC and Type labels). If the MC-TS advantage over the CNN for Type and USPC disappears or drops below significance, then the reported outperformance is an artifact of class-name leakage. As a complementary check, rerun MC-TS with all class names replaced by arbitrary tokens (e.g., 'class 1', 'class 2') and measure the accuracy drop on USPC; a large drop would confirm that the model relies on label semantics rather than visual content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim rests on Table 2, where MC-TS outperforms ResNet/ResNext for Type and USPC. However, the two model families operate under different information conditions. In MC-TS (Section 3.3), the LVLM is prompted with multiple-choice questions that literally list candidate class names (e.g., 'edible products', 'apparel and haberdashery'), so the model can exploit its pretrained language priors to select a plausible label even when the visual evidence is weak or the ground truth is noisy. The CNN baselines (Section 4.1) are trained with standard image-to-index classification: they never see class names or any semantic description, only numeric labels. For USPC especially, whose classes are broad and linguistically informative, this asymmetry can fully explain the LVLM's advantage. The paper does not ablate the contribution of label text, nor does it give CNNs access to equivalent semantic information (e.g., via CLIP text embeddings or a text-initialized classifier head). Consequently, the headline claim that the proposed method 'outperforms the CNN-based classifiers for two out of four aspects' is not yet established as a claim about visual figure classification; it may simply reflect an unfair comparison between a vision-language model and a pure vision model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two new resources, PatFigVQA and PatFigCLS, for patent figure visual question answering and classification, and studies three LVLM-based classification strategies (binary, multiple-choice tournament, and open-ended) using InstructBLIP with FlanT5-XL. The central empirical claim is that the proposed tournament-style MC-TS approach outperforms CNN baselines (ResNet50, ResNext101) for Type and USPC classification in few-shot settings, while remaining competitive or superior to other LVLM-based approaches. The authors also propose a semantic equivalence metric (SemEq) to mitigate label noise inherited from the DeepPatent2 corpus.","tokens_in":14493,"tokens_out":2271,"duration_ms":24237,"significance":"If the claims hold, the paper provides a useful step toward applying LVLMs to patent figure understanding: it contributes publicly available datasets, a practical tournament-style strategy for handling large label sets, and a comparative evaluation against established CNN baselines. The strength of the paper is its empirical grounding on held-out data, with multiple aspects (Type, Projection, USPC, Object) and both zero-shot and few-shot settings. The main significance is relatively incremental, however, because the headline result ('outperforms CNNs for Type and USPC') is central and needs to be demonstrated under comparable information conditions before the LVLM advantage can be attributed to visual understanding rather than to the presence of class names in the prompt.","major_comments":[{"comment":"The comparison between MC-TS and the CNN baselines is confounded by asymmetric access to class labels. MC-TS is prompted with multiple-choice options that literally contain the candidate class names (e.g., 'edible products', 'apparel and haberdashery'), whereas the CNN baselines are trained as image-to-index classifiers that never see class names or any semantic description. For USPC, whose class definitions are broad and linguistically informative, the LVLM can exploit pretrained language priors even when visual evidence is weak. Without an ablation that removes this asymmetry (e.g., giving CNNs access to text embeddings via CLIP or a text-initialized classifier head, or removing class-name semantics from the MC-TS prompt), the claim that MC-TS outperforms CNNs for Type and USPC is not established as a claim about visual figure classification. This is load-bearing for the central contribution and should be addressed experimentally.","section":"Section 4.3, Table 2"},{"comment":"The ground-truth labels for Projection, Object, and USPC derive from DeepPatent2, in which concepts were automatically extracted from figure references and are not normalized. The authors normalize them using rule-based keyword mapping and embedding-based clustering, but they do not report validation of the resulting labels, such as human agreement on a sample or comparison with the original references. This is especially relevant for the Object test set, which contains exactly one figure per concept (1,447 test samples). With one item per class, any label noise or normalization error directly becomes measured accuracy error, and the reported accuracies may reflect agreement with noisy references rather than true classification ability. The paper should either provide evidence that the normalized labels are reliable or treat the labels as a weak/soft supervision setting and temper the accuracy-based claims accordingly.","section":"Section 3.2, Table 1"},{"comment":"All reported accuracies appear to come from a single training run per configuration, with no error bars, multiple seeds, or statistical significance tests. Several comparisons in Table 2 are close (e.g., few-shot MC-TS (10) vs MC-TS (5) for Type: 87.12 vs 87.98; zero-shot MC-TS (10) vs MC-TS (20) for USPC SemEq: 25.70 vs 19.50), and without variance estimates it is unclear which differences are reliable. The paper's central claims about outperforming CNNs and about the tournament size k would be stronger with at least 3-5 runs and standard deviation reporting.","section":"Section 4.1, Table 2"},{"comment":"The SemEq metric uses FlanT5-XXL, which belongs to the same FlanT5 family as the backbone (FlanT5-XL) used in InstructBLIP, and it is validated on only 200 samples with a Cohen's Kappa of 0.59 (moderate agreement). Because SemEq is used to reinterpret the accuracy results for USPC and Object, the paper should report whether the SemEq conclusions are robust to the choice of the LLM evaluator or to the threshold for semantic equivalence. As written, the SemEq improvements could partly reflect evaluator bias rather than genuine semantic agreement with the reference labels.","section":"Section 4.3, SemEq metric"}],"minor_comments":[{"comment":"The text states that DeepPatent2 covers '33 USPC classes' and later says the 'Miscellaneous' concept was excluded, resulting in '32 USPC' in Table 1; this transition should be stated explicitly when the final concept count is introduced.","section":"Section 3.2"},{"comment":"The clustering procedure for Objects is underspecified: the embedding model (PatentBERT is mentioned later, but the exact checkpoint and pooling) and the number of clusters or the similarity threshold are not given. This makes the normalization step hard to reproduce.","section":"Section 3.2"},{"comment":"There are several typos and wording issues, e.g., 'we postpend short instructions' (should be 'append'), 'the LLM bakbone FlanT5' (should be 'backbone'), and 'using both both traditional and deep learning methods' in Section 2. These should be corrected in a revision.","section":"Section 4.1"},{"comment":"The y-axis label says 'Accuracy (%)' but the values are plotted as fractions (0.1 to 0.9). Either change the axis to percent (10-90) or change the label to 'Accuracy'.","section":"Figure 3"},{"comment":"The notation for the number of queries in MC-TS appears to have a typo: the formula reads 'N = PR r=1 l |C| kr m' which is not rendered correctly; it should be a summation of ceil(|C|/k^r). Please clarify the formula.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the resources (PatFigVQA, PatFigCLS) are potentially useful to the community. The main risk is that the headline comparison against CNN baselines is not yet convincing because of the asymmetric access to class semantics and because the label noise in the automatically extracted concepts is not controlled. I recommend inviting a revision that adds a text-conditioned CNN baseline or an ablation that removes label names from MC-TS prompts, reports multiple seeds or confidence intervals, and provides a validation of the normalized labels. These are addressable within the manuscript's scope, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a useful applied paper: it introduces two new datasets (PatFigVQA, PatFigCLS), the first LVLM treatment of patent figure classification, and a tournament-style multiple-choice strategy (MC-TS) that scales LVLM classification to large label sets. Code and data are public, and the authors are honest about the label noise in DeepPatent2. Second, the headline claim that MC-TS beats CNN baselines for Type and USPC is not yet established as a claim about visual classification, because the comparison is asymmetric: the LVLM sees candidate class names in its prompt, while the ResNet/ResNext baselines see only numeric indices. For USPC in particular, the class names (e.g., 'edible products') carry strong language priors. The paper never ablates this, nor does it give the CNNs text-conditioned features like CLIP embeddings. So the two-out-of-four advantage may reflect language knowledge, not visual understanding.\n\nThe datasets themselves are the real contribution. The VQA fine-tuning results show a sensible few-shot curve and saturation around 80 examples per concept. MC-TS is a reasonable adaptation of setwise LLM ranking to the visual domain, and the paper correctly notes the parallel to passage reranking. The qualitative confusion matrices are helpful.\n\nSoft spots, in order of severity. The evaluation lacks error bars or multiple seeds; Table 2 reports single runs. The Object test set has exactly one figure per concept, so the accuracy estimate is very noisy. The SemEq metric uses FlanT5-XXL, from the same LLM family as the InstructBLIP backbone, and its human agreement is only moderate (kappa = 0.59). The label noise from automatically extracted DeepPatent2 concepts is acknowledged, but it interacts with the label-text asymmetry: since the LVLM has access to the class names at inference, it can align with the text-derived labels in a way the CNNs cannot. That does not destroy the paper, but it does mean the central comparative claim needs tempering or a control. The Type split taken from the authors' own prior work is fine, not a flaw by itself.\n\nWho gets value: people working on patent retrieval, faceted search, or practical few-shot LVLM deployment for technical drawings. The datasets and MC-TS strategy are likely to be used regardless of the comparison issue.\n\nRecommendation: send it to peer review, but ask for a proper control—e.g., a CLIP-based linear probe or text-initialized classifier—and for error bars or multiple seeds. With those, it would be a solid conditional accept.","headline":"Useful new datasets and a pragmatic LVLM tournament classifier, but the headline advantage over CNNs may come from seeing class names in prompts rather than from visual understanding.","tokens_in":15045,"tokens_out":4112,"would_cite":true,"duration_ms":40638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A series of multiple-choice questions lets a vision-language model classify patent figures by type, projection, object, and USPC class, with few-shot fine-tuning and a tournament-style strategy that scales to thousands of labels.","keywords":["patent figure classification","large vision-language models","visual question answering","few-shot learning","tournament-style classification","patent retrieval","PatFigVQA","PatFigCLS"],"falsifier":"Re-annotate a random sample of test figures from PatFigCLS with patent experts, then recompute Top-1 accuracy for MC-TS and the CNN baselines on the clean labels; if the reported LVLM advantages over CNNs largely vanish on clean labels, the published numbers would be measuring agreement with noisy references rather than true classification ability.","tokens_in":14027,"feed_emoji":"🏆","tokens_out":3122,"duration_ms":31875,"temperature":0.7,"pith_summary":"This paper tries to establish that large vision-language models (LVLMs), previously untested on patent figures, can be adapted to this domain with modest amounts of training data and can compete with, or beat, traditional CNN-based classifiers on several classification aspects. The authors introduce two new resources, PatFigVQA for visual question answering and PatFigCLS for classification, and propose a tournament-style multiple-choice strategy that keeps the number of model queries logarithmic in the number of candidate classes. If the approach is correct, patent retrieval systems could add faceted figure search over type, projection, patent class, and depicted objects using a single fine-tuned LVLM instead of separate supervised classifiers.","feed_headline":"Vision-language model tournament beats CNNs on patent figure types","feed_subtitle":"Few-shot multiple-choice method classifies four aspects of patent drawings, winning on type and USPC.","key_machinery":"The central object is the tournament-style multiple-choice classification strategy (MC-TS), which partitions a large concept set $\\mathcal{C}$ into smaller subsets of size $k$, asks the LVLM a multiple-choice question over each subset, then iteratively pits the winning concepts against each other in subsequent rounds until one concept remains. The number of rounds is $R = \\lceil \\log_k |\\mathcal{C}| \\rceil$, so a dataset with thousands of classes requires only a handful of queries per figure instead of one query per class. This mechanism is what makes LVLM-based classification computationally feasible for the 1,447-object and 32-class settings.","core_discovery":"The paper claims that a fine-tuned InstructBLIP model, queried through a tournament of multiple-choice questions, outperforms CNN-based supervised classifiers for two of four patent-figure classification aspects (Type and USPC) and consistently outperforms the binary-question classification (BC) baseline across all aspects. The same model, fine-tuned on the newly introduced PatFigVQA dataset with as few as 150 samples per concept, improves monotonically with training data and starts to saturate around 80 samples per concept. The authors also report that multiple-choice questions degrade as the number of options grows, and that open-ended classification remains competitive on Projection and Object, but that the tournament strategy offers the best balance of accuracy and query efficiency for large concept sets.","pith_inferences":["If the DeepPatent2-derived labels are as noisy as the authors suspect, a human-cleaned version of PatFigCLS might reveal larger LVLM advantages over CNNs on Projection and Object, because the SemEq metric already shows that LVLM predictions are often semantically equivalent to the reference.","The tournament mechanism is label-order agnostic only if the LVLM treats option order fairly; a useful stress test would be to shuffle option order across runs and check variance, since long-context sensitivity is known to affect multiple-choice performance.","The same querying scheme could be applied to hierarchical patent classification (e.g., IPC subclasses) by running tournaments within sibling groups first and then a final tournament across group winners, mirroring the hierarchy of the label space."],"forward_implications":["Patent examiners could use faceted figure search to filter prior art by figure type, projection, USPC class, and depicted objects without training a separate model for each aspect.","Few-shot fine-tuning with just dozens of samples per concept is enough to close much of the domain gap between natural-image-pretrained LVLMs and patent drawings.","The tournament-style querying scheme extends to any LVLM classification task with a large label space, since the number of model calls grows logarithmically rather than linearly with class count.","The reported saturation around 80 samples per concept suggests that further gains will come from better visual discrimination or label quality rather than from simply adding more training images."],"supporting_citations":[{"why":"Supplies the DeepPatent2 corpus whose automatically extracted Projection, Object, and USPC concepts form the raw material for PatFigCLS and PatFigVQA.","marker":"[1]"},{"why":"Provides the extended CLEF-IP 2011 Type labels, the USPTO-PIP projection dataset, and the CNN baseline setup that the paper compares against.","marker":"[14]"},{"why":"Defines InstructBLIP, the LVLM architecture that is fine-tuned and evaluated throughout the paper.","marker":"[9]"},{"why":"Provides the planar geometric projection schema that the authors extend and use to normalize noisy Projection concepts via rule-based keyword mapping.","marker":"[3]"},{"why":"FlanT5 is the LLM backbone inside InstructBLIP and the LLM evaluator used in the SemEq metric.","marker":"[7]"},{"why":"PatentBERT embeddings are used to cluster Object concepts and to compute semantic similarity in open-ended classification and semantic-equivalency evaluation.","marker":"[42]"},{"why":"The LAVE metric is adapted to define the SemEq evaluation metric used for USPC and Object results.","marker":"[29]"},{"why":"Provides evidence that long contexts degrade LLM performance, motivating the subset-based tournament strategy to keep multiple-choice questions short.","marker":"[27]"}],"fun_headline_variants":["LVLM tournament beats CNNs on patent figure type and USPC","Few-shot vision-language model wins patent type, USPC via tournament","New datasets and tournament MCQs push LVLMs past CNNs on patent figures","Tournament of MCQ choices gives LVLM edge over CNNs on patent type and USPC","Fine-tuned LVLM tournament outclasses CNNs on two patent aspects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels for Projection, Object, and USPC come from DeepPatent2, where concepts were extracted automatically from figure references and were not normalized; the authors normalize them with rule-based keyword mapping and embedding-based clustering, then treat those normalized labels as ground truth for both training and evaluation.","fun_headline_variants_meta":{"raw":{"variants":["LVLM tournament beats CNNs on patent figure type and USPC","Few-shot vision-language model wins patent type, USPC via tournament","New datasets and tournament MCQs push LVLMs past CNNs on patent figures","Tournament of MCQ choices gives LVLM edge over CNNs on patent type and USPC","Fine-tuned LVLM tournament outclasses CNNs on two patent aspects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3573,"prompt_tokens":874,"completion_tokens":2699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":2599}},"tokens_in":490,"tokens_out":2699,"duration_ms":18493,"temperature":1.0,"reasoning_tokens":2599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:49:52.353530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of test figures from PatFigCLS with patent experts, then recompute Top-1 accuracy for MC-TS and the CNN baselines on the clean labels; if the reported LVLM advantages over CNNs largely vanish on clean labels, the published numbers would be measuring agreement with noisy references rather than true classification ability.","supporting_citations":[{"cited_title":"Scientific Data 10(1), 772 (Nov 2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the DeepPatent2 corpus whose automatically extracted Projection, Object, and USPC concepts form the raw material for PatFigCLS and PatFigVQA."},{"cited_title":"In: International Conference on Theory and Practice of Digital Libraries, TPDL 2023, Zadar, Croatia, September 26-29, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the extended CLEF-IP 2011 Type labels, the USPTO-PIP projection dataset, and the CNN baseline setup that the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines InstructBLIP, the LVLM architecture that is fine-tuned and evaluated throughout the paper."},{"cited_title":"ACM Computing Surveys 10(4), 465–502 (Dec 1978)","cited_arxiv_id":null,"evidence_quote":"Provides the planar geometric projection schema that the authors extend and use to normalize noisy Projection concepts via rule-based keyword mapping."},{"cited_title":"Journal of Machine Learning Research25(70), 1–53 (2024), http://jmlr.org/papers/v25/23-0870.html","cited_arxiv_id":null,"evidence_quote":"FlanT5 is the LLM backbone inside InstructBLIP and the LLM evaluator used in the SemEq metric."},{"cited_title":"White paper (2020), https://services.google.com/fh/files/blogs/bert_for_patents_white_paper.pdf","cited_arxiv_id":null,"evidence_quote":"PatentBERT embeddings are used to cluster Object concepts and to compute semantic similarity in open-ended classification and semantic-equivalency evaluation."}],"review_version":1}