REVIEW 4 major objections 5 minor 14 references
Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that a two-LoRA pipeline with schema caching beats RAG baselines on on-device information extraction while cutting latency.
desk verdict A plausible, narrow engineering combination of dual LoRA adapters and schema caching for on-device IE, but the main comparison doesn't isolate the architecture and the evidence is too thin to trust the superiority claim yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanisms are the Dual-LoRA architecture and Incremental Schema Caching. Two low-rank adapters are merged into the same base LLM, producing an Identification model $\theta'_I$ that selects the relevant schemas $S$ from a query, and an Extraction model $\theta'_E$ that performs extraction from the prompt assembled from the matched schemas and the query. Incremental Schema Caching stores the computed key-value representations of the extraction meta-prompt and of already-seen matched schemas, so that when a schema reappears in a later query its cached computation is reused instead of being run again. This is what lets the method reduce redundant computation while keeping the extraction prompt schema-aware.
What would settle it
Re-run CrossNER_AI and DuEE-Fin after fine-tuning each retrieval baseline on the same schema-matching supervision used to train the Identification LoRA; if a retrieval-augmented pipeline then matches or exceeds DLISC's F1, the claimed advantage of the dual-LoRA identification stage is not confirmed. Separately, measure total latency on real on-device hardware including the cost of switching between the two LoRA adapters; if the switch cost erases the caching gap, the efficiency claim fails.
Extended reading notes
Core claim
The central claim is that a Dual-LoRA architecture over a single persistent on-device LLM can outperform retrieval-augmented two-stage IE while being faster, by replacing an external retriever with a trained Identification LoRA and caching incremental schema computations. Reported results on CrossNER_AI named entity recognition show DLISC reaching F1 0.4179 with Llama-3.2-1B versus 0.2341 for all three RAG baselines, and 0.4311 with Qwen2.5-3B versus at most 0.3968 for the baselines; on DuEE-Fin event extraction with TinyLlama-1.1B-Chat, DLISC achieves 0.2982 identification precision and 0.7746 extraction F1, ahead of every RAG baseline. Incremental Schema Caching lowers average extraction time from 1.38s to 0.66s per sample on Llama-3.2-1B and from 6.02s to 4.59s on Qwen2.5-3B.
Load-bearing premise
The effectiveness result assumes that comparing a fine-tuned Identification LoRA against frozen BM25, BGE, or LLM-Embedder retrievers, with the same extraction backbone, isolates the benefit of the dual-LoRA design; if fine-tuning alone is what produces the higher F1, the architecture comparison would not be supported.
Editorial extensions
If this is right
- A single on-device LLM can handle diverse extraction schemas without loading a separate retriever, since schema selection is done by a small LoRA merged into the same base model.
- Repeated schemas across queries become progressively cheaper because their computed representations are cached, so the efficiency benefit should scale with schema reuse.
- The same two-stage split can be applied to other schema-conditioned extraction tasks, not only the named entity recognition and event extraction datasets tested.
- Because both stages share one base model, updating the schema library or extraction rules only requires updating the corresponding LoRA weights, not redeploying the full model.
Reading between the lines
- Editorial inference: fine-tuning the RAG retrievers on the same schema-matching supervision used for the Identification LoRA would test whether the reported effectiveness gap comes from the dual-LoRA design itself or simply from the identification stage being trained.
- Editorial inference: the reported latency does not include the cost of switching between the two LoRA adapters on a single device; measuring total wall-clock time including that switch would clarify whether caching remains a win in deployment.
- Editorial inference: the caching benefit depends on how often schemas recur across queries, so a useful follow-up is to report cache hit rate and latency as a function of schema-library size.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DLISC, a two-stage schema-aware information extraction method for on-device LLMs built on the LLM-Adapters architecture. DLISC uses two LoRA modules on a single base LLM: an Identification LoRA that selects the most relevant schemas for a query and an Extraction LoRA that performs schema-conditioned extraction, together with an Incremental Schema Caching mechanism that reuses cached key-value states for previously seen schemas. The authors report experiments on CrossNER_AI (NER) and DuEE-Fin (event extraction) with Llama-3.2-1B, Qwen2.5-3B, and TinyLlama-1.1B, comparing against RAG baselines (BM25, BGE-Reranker-V2-M3, LLM-Embedder) that share the same extraction backbone. Reported results show consistent F1 gains for DLISC over the RAG baselines and latency reductions from the caching mechanism.
Significance. If the empirical claims are robust, the paper would make a useful contribution to on-device schema-aware IE by demonstrating that a dual-LoRA architecture with prompt caching can outperform retrieval-augmented two-stage extraction while reducing latency. The core architectural idea is clearly presented and builds on a plausible decomposition of schema identification and extraction. The paper is honest about its limitations, including the absence of real-device deployment. However, the current evidence is not sufficient to establish the central claim of consistent superiority, because the evaluation has unresolved confounds, a suspicious degenerate baseline result, and no variance or significance reporting.
major comments (4)
- [Section 3.2, Table 1] All three RAG baselines (BM25, BGE-Reranker-V2-M3, LLM-Embedder) produce exactly the same F1 score (0.2341) on CrossNER_AI with Llama-3.2-1B. This strongly suggests that the retrieval step has no effect in this condition, either because the schema library is too small or because the extraction backbone ignores the retrieved schemas. As reported, the comparison is not sensitive to retrieval quality, and the DLISC advantage in this row may reflect the fine-tuned Identification LoRA rather than a better schema-selection mechanism. The authors should explain this result and provide a condition where the baselines differ, or add additional analysis (e.g., oracle retrieval, varying K) to show the comparison is meaningful.
- [Section 2 / Section 3.1] The claim that DLISC outperforms RAG baselines while sharing the same extraction backbone does not isolate the benefit of the dual-LoRA architecture. In DLISC, the Identification LoRA is optimized (Section 2), while the three retrievers are used off-the-shelf and are not trained for schema matching. Consequently, the observed gains on CrossNER_AI and DuEE-Fin could be driven entirely by supervised fine-tuning of the schema-selection module. The paper provides no training details (e.g., construction of training data for schema matching, LoRA rank/alpha, number of epochs, learning rate, random seeds) and no control experiment with a fine-tuned retriever or a single-LoRA variant that receives the same schemas. These ablations are necessary to support the architectural claim.
- [Section 3.2, Table 2] The efficiency comparison is limited to Dual-LoRA without caching versus DLISC with caching, and it does not report latency for the RAG-based baselines. The observed latency reduction is largely a direct consequence of the caching mechanism (avoiding recomputation of the extraction prompt), so it is expected rather than a novel algorithmic result. Additionally, Eq. (1) states that the two LoRA parameter sets are merged with the base LLM, but the paper does not discuss whether both merged models can reside in memory simultaneously on an edge device or whether adapter switching occurs between the identification and extraction stages. The reported per-sample latency should account for this switching or memory-transfer overhead if it is not negligible.
- [Section 3.2 / Table 3] The paper reports a single F1 value per condition, with no confidence intervals, standard deviations across seeds, or significance tests. The results in Table 3 are based on TinyLlama only, and Table 1 on Llama-3.2-1B and Qwen2.5-3B, but there is no cross-model consistency analysis for DuEE-Fin. Given the small differences in some comparisons (e.g., 0.7746 vs. 0.7484 for LLM-Embedder), the authors should provide variance estimates or multiple runs to establish that the reported improvements are not due to chance.
minor comments (5)
- [Section 3.1] The evaluation metrics paragraph lists Precision as one of the three metrics, but Table 1 reports only F1. Please clarify where Precision is reported for the main effectiveness results or remove it from the metrics list if it is only used for the schema-matching phase in Table 3.
- [Section 1] The citation for LLM-Adapters (Hu et al., 2023) refers to the EMNLP paper, but the LLM-Adapters framework described here (multiple LoRA modules on one base model) is also discussed elsewhere; the reference should be checked for accuracy.
- [Section 3.1] The paper states the retrieval-based baselines use K=5 schemas, but it does not specify how K is chosen for DLISC or whether the same number of schemas is used in the identification stage. Please state this explicitly.
- [Section 3.1] The DuEE-Fin dataset is described as 'document-level' event extraction, but the evaluation appears to be sample-based (100 samples for latency). Please clarify the sample unit (sentence, document) used in the experiments.
- [Section 5] The limitation paragraph appropriately notes the lack of real-device deployment, but this limitation undercuts the 'on-device' framing of the title and abstract. It would strengthen the paper to state this limitation earlier and to frame the results as simulations of on-device inference.
Circularity Check
No significant circularity: DLISC's effectiveness and efficiency claims are empirically evaluated against external baselines, and the only self-citation is non-load-bearing.
full rationale
The paper does not contain a derivation chain in which a predicted quantity is equivalent to an input by construction. The effectiveness comparison in Section 3.2 is against external retrievers (BM25, BGE-Reranker-V2-M3, LLM-Embedder) that share DLISC's extraction backbone, which is a controlled comparison rather than a circular definition; any concern that the Identification LoRA is trained while the retrievers are off-the-shelf is an experimental-validity issue, not circularity. The efficiency gain in Table 2 is a measured consequence of the Incremental Schema Caching mechanism described in Equations 3-4; caching is defined as reusing previously computed schema prompts to avoid recomputation, and the paper reports the resulting latency rather than claiming this as a theoretical prediction. The self-citation to the authors' prior ASEE work (Liang et al., 2025) is used only to position the two-stage schema-matching and schema-augmented extraction paradigm, and it is not invoked to justify the reported F1 improvements or to rule out alternative designs. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known empirical pattern is merely relabeled. Concerns about training-detail omissions, identical baseline F1 values under Llama-3.2-1B, and unaccounted LoRA-switching overhead are legitimate correctness/robustness issues, but they do not constitute circularity.
Assumptions & free parameters
free parameters (2)
- K (top-K schemas) =
5
- LoRA rank and alpha
assumptions (4)
- domain assumption The schema library for each dataset is complete and correctly represents the labels to be extracted.
- domain assumption The on-device LLMs can perform schema-aware extraction after LoRA adaptation.
- domain assumption Merging and switching LoRA adapters on-device is fast enough not to dominate inference time.
- domain assumption KV cache and prompt cache can be reused across queries with matched schemas.
invented entities (1)
-
Incremental Schema Caching (ISC)
Cite this review
Pith. "Pith review of Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models." pith.science (2026). https://pith.science/paper/6SPFWO4G
@misc{pith2026250514992,
author = {Pith},
title = {Pith review of: Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/6SPFWO4G}},
note = {Machine review of arXiv:2505.14992}
}
read the original abstract
Information extraction (IE) plays a crucial role in natural language processing (NLP) by converting unstructured text into structured knowledge. Deploying computationally intensive large language models (LLMs) on resource-constrained devices for information extraction is challenging, particularly due to issues like hallucinations, limited context length, and high latency-especially when handling diverse extraction schemas. To address these challenges, we propose a two-stage information extraction approach adapted for on-device LLMs, called Dual-LoRA with Incremental Schema Caching (DLISC), which enhances both schema identification and schema-aware extraction in terms of effectiveness and efficiency. In particular, DLISC adopts an Identification LoRA module for retrieving the most relevant schemas to a given query, and an Extraction LoRA module for performing information extraction based on the previously selected schemas. To accelerate extraction inference, Incremental Schema Caching is incorporated to reduce redundant computation, substantially improving efficiency. Extensive experiments across multiple information extraction datasets demonstrate notable improvements in both effectiveness and efficiency.
Figures
Reference graph
Works this paper leans on
-
[2]
In 2024 IEEE International Conference on Knowledge Graph (ICKG), pages 33–49
Information extraction in low-resource scenarios: Survey and perspective. In 2024 IEEE International Conference on Knowledge Graph (ICKG), pages 33–49. IEEE. Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey
work page 2024
-
[3]
The llama 3 herd of models. ArXiv, abs/2407.21783. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang
-
[5]
LLM-adapters: An adapter fam- ily for parameter-efficient fine-tuning of large lan- guage models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5254–5276, Singapore. Association for Computational Linguistics. Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou
work page 2023
-
[6]
From matching to generation: A survey on generative in- formation retrieval. ArXiv, abs/2404.14851. Sheng Liang, Hang Lv, Zhihao Wen, Yaxiong Wu, Yongyue Zhang, Hao Wang, and Yong Liu
-
[8]
Keep the cost down: A review on methods to optimize llm’ s kv-cache consumption. ArXiv, abs/2407.18003. Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al
-
[9]
arXiv preprint arXiv:2404.14619
Openelm: An efficient lan- guage model family with open training and inference framework. arXiv preprint arXiv:2404.14619. Stephen E. Robertson and Hugo Zaragoza
-
[10]
Decompose, enrich, and extract! schema-aware event extraction using llms. Preprint, arXiv:2406.01045. Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen
-
[11]
arXiv preprint arXiv:2312.17617
Large language models for generative information extraction: A survey. arXiv preprint arXiv:2312.17617. Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling
Show all 14 references
-
[12]
arXiv preprint arXiv:2409.00088
On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088. An Yang, Baosong Yang, and Binyuan Hui
-
[13]
ArXiv, abs/2407.10671
Qwen2 technical report. ArXiv, abs/2407.10671. Yang Yang, Zhilei Wu, Yuexiang Yang, Shuangshuang Lian, Fengjie Guo, and Zhiwei Wang
-
[2022]
Applied Sciences, 12(19):9691
A survey of information extraction based on deep learning. Applied Sciences, 12(19):9691. Peitian Zhang, Zheng Liu, Shitao Xiao, Zhicheng Dou, and Jian-Yun Nie. 2024a. A multi-task embedder for retrieval augmented LLMs. In Proceedings of the 62nd Annual Meeting of the Associat...
-
[2023]
arXiv preprint arXiv:2312.10997
Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong
-
[2024]
Preprint, arXiv:2402.03216
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216. Shumin Deng, Yubo Ma, Ningyu Zhang, Yixin Cao, and Bryan Hooi
-
[2025]
arXiv preprint arXiv:2505.08690
Adaptive schema-aware event extraction with retrieval-augmented generation. arXiv preprint arXiv:2505.08690. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pas- cale Fung
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.