Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges

Hao Liu; Rong Lu; Song Hou

REVIEW 4 major objections 5 minor 10 references

Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges

T0 review · 4 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Generative vision-language models beat embedding models for zero-shot geoscience document classification, reaching 82% accuracy with Chain-of-Thought prompting.

desk verdict Solid practitioner bake-off: Qwen2.5-VL + CoT hits 82% zero-shot vs 63% for the best multimodal embedder on an 8-class geoscience first-page set; useful accuracy–cost map, but proprietary data and missing uncertainty leave the ranking unreplicable. read the letter →

arxiv 2604.04997 v1 submitted 2026-04-05 cs.IR cs.AIcs.CLcs.CVcs.LG

Rong Lu , Hao Liu , Song Hou This is my paper

classification cs.IRcs.AIcs.CLcs.CVcs.LG

keywords documentclassificationvision-languagemodelsmultimodalembeddingsChain-of-Thoughtpromptinggeosciencedocumentszero-shotsupervisedfine-tuningclassimbalance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Oil and gas archives hold vast piles of technical reports, logs, and surveys whose visual cues (charts, seismic sections, page layouts) matter as much as text, and whose OCR quality is often poor. This paper compares two ways of automating their classification: multimodal embedding models that turn pages into vectors for nearest-class voting, and generative vision-language models that output a class label directly. On a proprietary eight-class geoscience benchmark, generative models with Chain-of-Thought prompts and domain definitions reach 82% zero-shot accuracy, well above the best embedding model at 63%. Fine-tuning can raise performance further when training data are abundant, but it hurts minority classes. The practical message is that practitioners face a clear accuracy-versus-cost trade-off and that prompt engineering is a cheap, high-leverage way to inject domain knowledge into either paradigm.

What carries the argument

The comparison of two classification paradigms on the same first-page image benchmark: similarity-voting with multimodal embeddings versus direct label generation by VLMs under base versus CoT-plus-domain prompts, plus a controlled SFT experiment that exposes sensitivity to class imbalance.

What would settle it

Re-run the same models on a version of the benchmark that uses a randomly sampled interior page, or the full multi-page document, and check whether the 82% versus 63% ranking and absolute accuracies still hold.

Watch

Extended reading notes

Core claim

On a multi-disciplinary geoscience document benchmark, generative Vision-Language Models such as Qwen2.5-VL, guided by Chain-of-Thought prompting plus domain definitions, achieve substantially higher zero-shot classification accuracy (82%) than state-of-the-art multimodal embedding models such as QQMM (63%). Supervised fine-tuning can further improve the generative models, but only for classes that have enough training examples; under-represented classes suffer.

Load-bearing premise

Using only the first page of each multi-page document is assumed to be a fair and sufficient basis for assigning the full document’s class label.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing worth knowing is the clean empirical ranking: generative VLMs (especially Qwen2.5-VL-72B with a CoT-plus-domain prompt) reach 82% accuracy / 0.82 macro F1 zero-shot, while the strongest multimodal embedder they tried (QQMM) tops out at 63%/0.64 even with class-definition prompts. That gap, plus the observation that SFT helps majority classes a lot and hurts minority ones, is the actual new content.

What the paper does well is keep the comparison honest and practical. They run five embedders with clustering diagnostics (intra/inter, silhouette, DB, CH) and four open VLMs under base vs plus prompts, report both accuracy and macro F1, and are explicit about the cost–stability trade-off: VLMs reason better over layout and text but are slow and non-deterministic; embeddings are cheap and stable. The prompt ablations are clear (10-point F1 lift on the 7B model from CoT + domain definitions). For anyone building document triage in energy or similar technical archives, this is usable operational guidance.

Soft spots are real but proportionate. The corpus is proprietary, so the 82-vs-63 number cannot be recomputed; that is the load-bearing limitation. They only ever use the first page, which is a reasonable consistency choice but leaves open whether later-page seismic or log cues would change the ranking. No error bars or significance tests, and the SFT section is selective (0.93 only on classes with >150 samples). None of these break the central claim; they just keep it from being settled science.

Math and metrics are standard and correctly applied. Citations are appropriate (model cards, CoT, recent multimodal embedders). No circularity.

This is for practitioners and applied IR people who need a concrete accuracy–cost map on multimodal technical documents, not for theorists looking for a new method. I would send it to peer review at a workshop or applied venue; the numbers are clear enough to deserve referee time even if the data stay closed. Worth a look if you work on industrial document systems; not a must-read otherwise.

Referee Report

4 major / 5 minor

Summary. The manuscript compares embedding-based multimodal models against generative vision-language models (VLMs) for zero-shot and fine-tuned classification of geoscience technical documents. On a proprietary eight-class first-page benchmark spanning Geology & Geochemistry, Petrophysics, Geophysics, and Petroleum Engineering, the authors report that Qwen2.5-VL-72B with Chain-of-Thought plus domain-definition prompting reaches 82% accuracy / 0.82 macro F1, outperforming the strongest embedding baseline (QQMM-embed at 63% / 0.64 F1). They further show that supervised fine-tuning of Qwen2.5-VL-7B improves performance on well-represented classes but degrades it on minority classes, and they discuss accuracy–cost–stability trade-offs. Results are summarized in Tables 1–2 and Figure 1, with clustering metrics for embeddings and base-vs-plus prompt ablations for VLMs.

Significance. If the ranking holds, the work supplies a concrete, practitioner-oriented comparison of two dominant paradigms for multimodal document classification in a high-value industrial domain. The prompt-engineering ablations (class-definition enrichment for embeddings; CoT-style “plus” prompts for VLMs) and the explicit documentation of SFT sensitivity to imbalance are useful, low-cost findings. Strengths include consistent reporting of accuracy, macro F1 and several clustering indices across five embedders and four VLMs, and a clear discussion of inference cost versus determinism. The main limitation on significance is that the central 82% vs 63% claim rests on a single internal corpus whose size, class definitions, OCR quality and page-layout statistics are not released, so independent verification is currently impossible.

major comments (4)

§2.1 and Tables 1–2: The headline claim (Qwen2.5-VL + CoT at 82% zero-shot accuracy versus QQMM-embed at 63%) is measured exclusively on a proprietary eight-class first-page benchmark whose documents, labels, class definitions and train/test split are not released. Without at least a public subset, synthetic proxy, or detailed dataset card (document counts per class, OCR quality distribution, layout statistics), the ranking cannot be recomputed or stress-tested by others; this is load-bearing for the paper’s central empirical contribution.
§2.1 (“For all experiments, the first page of each document was used”): Restricting evaluation to page 1 is an untested free parameter. Geoscience reports frequently place decisive cues (seismic sections, log charts, later-page text) after the cover or abstract page. The manuscript reports no multi-page ablation or first-page-versus-full-document comparison; if later pages systematically alter class cues, both absolute accuracies and the VLM–embedding gap can shift.
Tables 1–2 and §3.2: The 19-point accuracy gap is presented without confidence intervals, bootstrap estimates, or statistical significance tests. Given a single fixed test split of unknown size, it is impossible to judge whether the gap is robust or an artifact of sampling variance; this weakens the claim that generative VLMs are “superior” in a general sense.
§3.3: SFT results are reported selectively (“0.93 … for those classes over 150 training samples”). Overall macro F1/accuracy on the full eight-class test set after fine-tuning is not given, nor is a confusion matrix or per-class sample counts. Because the abstract and conclusion emphasize SFT’s sensitivity to imbalance, the missing aggregate numbers leave the practical takeaway incomplete.

minor comments (5)

§2.3: The exact text of the “simple prompt” used for document embeddings and the full wording of the class-definition prompts are not provided; releasing them (or a short appendix) would aid reproducibility of the embedding results.
§2.4 / Table 2: The “base” versus “plus” prompt distinction is described only at a high level. A short example of each template would clarify how much of the 5–10 point lift is attributable to CoT versus domain definitions.
Figure 1 caption and body: The figure is referenced as summarizing “both embedding and VLMs under various configurations,” yet axis labels, exact model identifiers and whether SFT points are included are not fully specified in the text; a more self-contained caption would help.
§2.2 Eqs. (1)–(2): Cosine distance is defined as 1 − cosine similarity; stating the embedding dimensionality and whether L2-normalization is applied before distance computation would remove ambiguity for readers re-implementing the clustering metrics.
References: Several model cards and arXiv preprints are cited with access dates; ensuring stable DOIs or version pins (especially for QQMM, GME, mmE5, vdr) would improve long-term citability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pure empirical bake-off of measured accuracies on held-out labels; nothing is defined in terms of a fitted quantity then re-presented as a prediction.

full rationale

The paper is a comparative empirical evaluation of multimodal embedding models and generative VLMs on a proprietary geoscience document classification benchmark. All headline numbers (QQMM 63% accuracy / 0.64 F1; Qwen2.5-VL-72B + CoT 82% accuracy / 0.82 F1; SFT lifts on majority classes) are direct measurements against human ground-truth labels on a held-out test set, not quantities derived from parameters fitted to the same target. Prompt engineering (class-definition embeddings; CoT “plus” prompts) and SFT are standard interventions whose effects are reported as observed deltas, not as first-principles predictions. Clustering metrics (Lintra, Linter, silhouette, DB, CH) are descriptive diagnostics of the embedding space, not circular redefinitions of accuracy. Citations (Qwen2.5-VL technical report, Wei et al. CoT, CLIP, mmE5, GME, etc.) are independent prior work; there is no self-citation chain, uniqueness theorem, or ansatz smuggled from the authors’ own prior papers that forces the ranking. The first-page protocol and proprietary corpus raise replicability concerns, but those are external validity issues, not circularity of the derivation chain. Score 0 is therefore the correct finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central ranking rests on a proprietary labeled corpus, a first-page evaluation protocol, cosine-similarity voting for embeddings, and the assumption that free-form VLM generations can be reliably parsed into the eight class labels. No new physical entities or fitted theoretical constants are introduced; free choices are experimental design decisions rather than parameters of a predictive model.

free parameters (3)

first-page-only evaluation protocol
All documents reduced to page 1 before embedding or VLM inference; choice is stated but not ablated against multi-page or random-page baselines.
SFT training-set size and class counts (~7000 samples, highly imbalanced)
Exact per-class counts and sampling strategy are only described qualitatively; minority classes have “dozens” of examples, majority “thousands.”
image resize threshold (max dimension 8192 px)
Hand-chosen preprocessing cutoff for embedding pipeline; effect on accuracy not measured.

assumptions (4)

domain assumption The first page of a multi-page geoscience document is sufficiently representative of its discipline class for both embedding and generative classifiers.
Stated in §2.1 as the consistent evaluation basis; load-bearing for every reported accuracy.
domain assumption Cosine similarity between a document embedding and class-definition embeddings yields a valid hard classification via argmax (similarity voting).
Core of the embedding pipeline in §2.3; standard but unvalidated against other decision rules.
domain assumption Detailed domain-specific class definitions improve embedding-class alignment more than bare class names.
Empirically observed for QQMM (F1 0.55→0.64) and treated as a general design recommendation.
ad hoc to paper Free-form VLM text outputs can be deterministically parsed into one of the eight target labels without material label noise.
§2.4 states that generated output is parsed; no inter-annotator or parse-error rate is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges." pith.science (2026). https://pith.science/paper/2604.04997

@misc{pith2026260404997,
  author       = {Pith},
  title        = {Pith review of: Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.04997}},
  note         = {Machine review of arXiv:2604.04997}
}

read the original abstract

This work presents a comparative analysis of embedding-based and generative models for classifying geoscience technical documents. Using a multi-disciplinary benchmark dataset, we evaluated the trade-offs between model accuracy, stability, and computational cost. We find that generative Vision-Language Models (VLMs) like Qwen2.5-VL, enhanced with Chain-of-Thought (CoT) prompting, achieve superior zero-shot accuracy (82%) compared to state-of-the-art multimodal embedding models like QQMM (63%). We also demonstrate that while supervised fine-tuning (SFT) can improve VLM performance, it is sensitive to training data imbalance.

Figures

Figures reproduced from arXiv: 2604.04997 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 linked inside Pith

[1]

Qwen2.5-VL Technical Report.arXiv e-prints, art

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report.ar...

arXiv doi:10.48550/arxiv.2502.13923 2025
[2]

Semantic instance segmentation with a discriminative loss function.CoRR, abs/1708.02551, 2017

Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function.CoRR, abs/1708.02551, 2017. URL http://arxiv.org/abs/ 1708.02551

arXiv 2017
[3]

mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 8254–8275, Vienna, Au...

2025
[4]

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´ e, Morgane Rivi` ere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga¨ el Liu, Francesco Visin, Kathleen Kenealy, Luc...

arXiv doi:10.48550/arxiv.2503.19786 2025
[5]

Model card for vdr-2b-multi-v1

LlamaIndex. Model card for vdr-2b-multi-v1. https://huggingface.co/llamaindex/vdr-2b-multi- v1, 2025. Accessed: 2025-07-07

2025
[6]

Model card for mistral-small-3.2-24b-instruct-2506

Mistral AI. Model card for mistral-small-3.2-24b-instruct-2506. https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506, 2025. Accessed: 2025-07-07

2025
[7]

Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020, 2021

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020, 2021. URLhttps://arxiv.org/abs/2103.00020

arXiv 2021
[8]

Chi, Quoc V

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088

2022

Show all 10 references

[9]

Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying.arXiv e-prints, art

Youze Xue, Dian Li, and Gang Liu. Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying.arXiv e-prints, art. arXiv:2506.02020, May 2025. doi: 10.48550/arXiv.2506.02020

2025 doi arXiv
[10]

GME: Improving Universal Multimodal Retrieval by Multimodal LLMs.arXiv e-prints, art

Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs.arXiv e-prints, art. arXiv:2412.16855, December 2024. doi: 10.48550/ arXiv.2412.16855. 6

2024 arXiv

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.

[1] [1]

Qwen2.5-VL Technical Report.arXiv e-prints, art

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report.ar...

arXiv doi:10.48550/arxiv.2502.13923 2025

[2] [2]

Semantic instance segmentation with a discriminative loss function.CoRR, abs/1708.02551, 2017

Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function.CoRR, abs/1708.02551, 2017. URL http://arxiv.org/abs/ 1708.02551

arXiv 2017

[3] [3]

mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 8254–8275, Vienna, Au...

2025

[4] [4]

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´ e, Morgane Rivi` ere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga¨ el Liu, Francesco Visin, Kathleen Kenealy, Luc...

arXiv doi:10.48550/arxiv.2503.19786 2025

[5] [5]

Model card for vdr-2b-multi-v1

LlamaIndex. Model card for vdr-2b-multi-v1. https://huggingface.co/llamaindex/vdr-2b-multi- v1, 2025. Accessed: 2025-07-07

2025

[6] [6]

Model card for mistral-small-3.2-24b-instruct-2506

Mistral AI. Model card for mistral-small-3.2-24b-instruct-2506. https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506, 2025. Accessed: 2025-07-07

2025

[7] [7]

Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020, 2021

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision.CoRR, abs/2103.00020, 2021. URLhttps://arxiv.org/abs/2103.00020

arXiv 2021

[8] [8]

Chi, Quoc V

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088

2022