REVIEW 4 major objections 5 minor 1 cited by
The paper claims that a privacy-protecting, lightweight financial LLM can be fine-tuned with differential privacy and still match fully fine-tuned models on financial sentiment at strict privacy budgets.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
DPFinLLM is a standard LoRA plus DP-SGD fine-tuning recipe applied to Llama2 and ChatGLM2 for financial sentiment; the experiments are mixed, generally below state-of-the-art, and key details are missing.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection Routine DP-SGD + LoRA application whose own tables refute the 'comparable to fully fine-tuned models' claim; the ChatGLM2 NWGI row matches the base model to the fifth decimal. the 4 major comments →
When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper claims that DPFinLLM, a lightweight transformer fine-tuned with LoRA under an (epsilon, delta)-differential privacy mechanism, performs on par with fully fine-tuned financial sentiment models. Using Llama2-7B and ChatGLM2-6B as base architectures, the pipeline clips per-example gradients and adds Gaussian noise to batch gradients, following the standard DPSGD recipe. Across FPB, FIQA, TFNS, and NWGI, the DP-fine-tuned models improve over the base models and, at epsilon values from 2 to 8, reach accuracy and F1 scores comparable to the best fully fine-tuned baselines, with zero-shot transfer to unseen datasets mostly preserved. The authors interpret this as evidence that privacy-pro
What carries the argument
The load-bearing mechanism is the differential-private fine-tuning loop: per-sample gradient clipping with norm bound C (Eq. 7), averaging over a lot, and Gaussian noise injection scaled by sigma (Eq. 8), whose privacy cost is bounded by Theorem 1 in the style of DPSGD. The rank-restricted LoRA update (Eq. 4) is the companion mechanism that makes this affordable for large base models. Together they let the paper claim both privacy and task performance, since the noise is added to low-rank adapters rather than the full model.
Load-bearing premise
The claim stands or falls on whether the reported epsilon values truly bound the privacy loss of the actual fine-tuning runs, yet the paper never discloses the noise scale, clipping bound, lot size, sampling scheme, or privacy accountant used.
What would settle it
Recompute the privacy budget with a standard accountant (e.g., Rényi DP) using the training configuration from the released code; if the realized epsilon exceeds the reported values, the privacy claim is false. Alternatively, retrain the fully fine-tuned baselines on exactly the same per-dataset training subsets; if their accuracy then matches or exceeds the DP models, the utility claim fails.
If this is right
- If the reported results hold, on-device financial sentiment models can be fine-tuned with a formal privacy guarantee at a small accuracy cost, enabling use on user-facing devices.
- The non-monotonic relationship between epsilon and F1 across datasets implies that privacy budgets should be tuned per dataset rather than simply set to the largest acceptable value.
- The zero-shot results suggest DP fine-tuning can preserve generalization to unseen financial text, so a single privately fine-tuned model could be deployed across related sentiment tasks.
- Because the method uses LoRA, the privacy mechanism adds only small memory and compute overhead, making it viable on resource-constrained hardware.
Where Pith is reading between the lines
- We infer that the reported privacy guarantees are not yet independently verifiable: the paper omits the noise scale, clipping bound, lot size, sampling scheme, training steps, and the privacy accountant, so a reader cannot confirm that the reported epsilon values bound the actual runs.
- We infer that the comparison to fully fine-tuned baselines is potentially confounded, since the DP models are fine-tuned per dataset while the strongest baselines are trained on a multi-task corpus; a clean test would match training sets and data order.
- We infer that the architecture choices (grouped-query attention, RMSNorm, SwiGLU, rotary embeddings) are not themselves evaluated; a reader cannot tell whether the privacy-utility result depends on these choices or would transfer to any transformer base.
- We infer that the same pipeline should be tested on non-sentiment financial tasks (e.g., named-entity recognition, fraud classification) before claiming general on-device financial applicability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DPFinLLM, an on-device financial LLM that combines Llama2/ChatGLM2-style transformer components with LoRA fine-tuning and a DP-SGD-style privacy mechanism. It evaluates accuracy and F1 scores on four financial sentiment datasets (FPB, FIQA, TFNS, NWGI), comparing against base Llama2-7B, ChatGLM2-6B, and FinGPT baselines, and also reports zero-shot transfer results. The central claim is that DPFinLLM can safeguard user data under differential privacy while achieving performance comparable to fully fine-tuned models under strict privacy constraints.
Significance. If the claims were supported, the work would be relevant to privacy-preserving on-device financial sentiment analysis. The paper uses standard components (DP-SGD, LoRA, grouped-query attention, RMSNorm, SwiGLU) and does not claim new privacy theory: Theorem 1 is a restatement of the DP-SGD bound from Abadi et al. The zero-shot transfer tables are a useful attempt to study generalization under DP. However, the reported experiments do not support the headline claims: the paper's own tables show large accuracy gaps on key datasets, and the privacy accounting is not described in enough detail to verify the reported epsilon values. No reproducible artifact is actually provided; the GitHub link is unparsable and no hyperparameters are given.
major comments (4)
- [Abstract; §IV-B; Tables I, III, IV] The central claim that DPFinLLM 'achieve[s] performance comparable to fully fine-tuned models' is contradicted by the paper's own tables. On TFNS (Table III), Llama2-based DPFinLLM at ε=4 reaches 0.732 accuracy and 0.710 weighted F1, while FinGPT v3.2 reaches 0.890 accuracy and 0.889 weighted F1 — a 15.8-point gap. On FPB (Table I), the gap is about 6.8 points in accuracy (0.798 vs 0.866). In Table IV, the ChatGLM2-based DPFinLLM at ε=8 gives accuracy 0.56042 versus the base model's 0.56041 and a lower weighted F1 (0.44888 vs 0.44952), so the §IV-B claim of 'substantial increases ... across all evaluation metrics' is false for that row. These are not local presentation issues; they undermine the abstract's and conclusion's main assertion.
- [§III-B, Eq. (11); §IV-A5] The privacy guarantee is not instantiated. Theorem 1 is quoted with unspecified constants c1 and c2, and the experimental section does not report the noise scale σ, gradient clipping bound C, lot size L, sampling probability q, number of training steps T, or the privacy accountant used to convert these into ε. The paper sets δ=1/|d| but gives no values for the other quantities. In DP-SGD, ε is the output of a privacy accountant given those hyperparameters; simply selecting ε and invoking Eq. (11) is not a reproducible privacy argument. This is load-bearing for the paper's 'safeguard user data' claim.
- [§IV-A2, §IV-B] There is no non-private LoRA control. DPFinLLM is fine-tuned with LoRA, but the comparisons are against base models and FinGPT baselines, which are described as fully fine-tuned. Without a same-architecture LoRA fine-tune without DP, the reported accuracy differences cannot be attributed to the DP mechanism rather than to LoRA or the training setup. This is necessary to support the conclusion that DPFinLLM is comparable to fully fine-tuned models 'even under strict privacy constraints'.
- [§IV-A5, Tables I–IV] All results are single-run point estimates with no standard deviations, confidence intervals, or number of seeds. Several differences are small enough to be noise (e.g., FIQA ChatGLM2-based DPFinLLM accuracy 0.83636 versus ChatGLM2-6B 0.83636; NWGI ChatGLM2 row 0.56042 versus 0.56041). The empirical claims of 'substantial increases' therefore cannot be robustly assessed from the reported data.
minor comments (5)
- [Eq. (7)] Stray comma in the clipping formula: 'max(1, ,||g_t(x_i)||_2/C)' should read 'max(1, ||g_t(x_i)||_2/C)'.
- [Eq. (10)] The differential privacy inequality is malformed. It should be Pr[M(d) ∈ S] ≤ e^ε Pr[M(d') ∈ S] + δ, with set membership and proper exponent formatting.
- [§IV-A2] The sentence comparing against 'the original Llama2-7B and ChatGLM2-6B models, both trained on a financial multi-task dataset' conflates base pretrained models with FinGPT, which is the model trained on the financial multitask dataset.
- [§IV, github link] The open-source code URL contains a line break and a space ('DP FinLLM'), so it is not usable as given.
- [Fig. 1] The figure caption and text do not state which base model or dataset is shown, nor how the other hyperparameters are fixed when ε is varied. The figure is not self-contained.
Circularity Check
No significant circularity: the privacy result is an imported external theorem and the utility claims are empirical comparisons, not derivations from the paper's own outputs.
full rationale
DPFinLLM's privacy claim rests on Theorem 1 (Eq. 11), which is stated as the known DPSGD result from Abadi et al. (2016) with unspecified constants c1 and c2; it is not derived from this paper's experiments or fitted to its data, so there is no self-definitional reduction. The utility claim is an empirical comparison against FinGPT, FinBERT, Llama2, and ChatGLM2 baselines on standard financial sentiment datasets; no parameter is fitted to the headline outcome and then renamed a prediction. The paper contains no load-bearing self-citations: the cited works (FinGPT, Llama2, LoRA, DPSGD, etc.) are not by the present authors, and the architecture choices are attributed to external sources. The omission of concrete noise scale, clipping bound, lot size, sampling scheme, and privacy accountant details makes the reported epsilon values unverifiable, and Table IV shows one case (ChatGLM2-based DPFinLLM on NWGI at epsilon=8) where the DP fine-tuned model matches the base model to five decimals, contradicting Section IV-B's claim of 'substantial increases' across all metrics. These are correctness, reproducibility, and utility concerns, not circularity: the paper's derivation chain does not reduce to its own inputs. Therefore the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- Per-dataset privacy budget epsilon =
2, 4, 6, or 8 depending on dataset and base model
- Gradient clipping bound C =
not reported
- Noise scale sigma =
not reported
- LoRA rank r and scaling alpha =
not reported
- Training schedule (learning rate, epochs, batch and lot sizes) =
not reported
axioms (3)
- standard math Theorem 1: the DP-SGD privacy guarantee from Abadi et al. holds for the implemented training loop.
- domain assumption FinGPT baseline scores are directly comparable because they use the same splits and evaluation protocol.
- domain assumption The zero-shot tables in Section IV-C are interpreted with the base model column as reference.
Cite this review
Pith. "Pith review of When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning." pith.science (2026). https://pith.science/paper/ZCTRY2YX
@misc{pith2026250908995,
author = {Pith},
title = {Pith review of: When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZCTRY2YX}},
note = {Machine review of arXiv:2509.08995}
}
read the original abstract
The integration of Large Language Models (LLMs) into financial technology (FinTech) has revolutionized the analysis and processing of complex financial data, driving advancements in real-time decision-making and analytics. With the growing trend of deploying AI models on edge devices for financial applications, ensuring the privacy of sensitive financial data has become a significant challenge. To address this, we propose DPFinLLM, a privacy-enhanced, lightweight LLM specifically designed for on-device financial applications. DPFinLLM combines a robust differential privacy mechanism with a streamlined architecture inspired by state-of-the-art models, enabling secure and efficient processing of financial data. This proposed DPFinLLM can not only safeguard user data from privacy breaches but also ensure high performance across diverse financial tasks. Extensive experiments on multiple financial sentiment datasets validate the effectiveness of DPFinLLM, demonstrating its ability to achieve performance comparable to fully fine-tuned models, even under strict privacy constraints.
Figures
Forward citations
Cited by 1 Pith paper
-
Protecting On-Device AI Inference: A Systematic Review of Attacks and Defence Mechanisms
A systematic review of on-device AI inference security finds defenses are imbalanced, with roughly half focused on IP theft while one-third of attacks (adversarial examples) lack any associated defenses.
Reference graph
Works this paper leans on
-
[1]
Enhancing financial sentiment analysis via retrieval augmented large language models,
B. Zhang, H. Yang, T. Zhou, M. Ali Babar, and X.-Y . Liu, “Enhancing financial sentiment analysis via retrieval augmented large language models,” inProceedings of the fourth ACM international conference on AI in finance, 2023, pp. 349–356
2023
-
[2]
Financial knowledge large language model,
C. Yang, C. Xu, and Y . Qi, “Financial knowledge large language model,” arXiv preprint arXiv:2407.00365, 2024
Pith/arXiv arXiv 2024
-
[3]
Fraud detection with natural language processing,
P. Boulieris, J. Pavlopoulos, A. Xenos, and V . Vassalos, “Fraud detection with natural language processing,”Machine Learning, vol. 113, no. 8, pp. 5087–5108, 2024
2024
-
[4]
M. Sanz-Guerrero and J. Arroyo, “Credit risk meets large language models: Building a risk indicator from loan descriptions in p2p lending,” arXiv preprint arXiv:2401.16458, 2024
Pith/arXiv arXiv 2024
-
[5]
Domain knowledge is all you need: A field deployment of llm-powered test case generation in fintech domain,
Z. Xue, L. Li, S. Tian, X. Chen, P. Li, L. Chen, T. Jiang, and M. Zhang, “Domain knowledge is all you need: A field deployment of llm-powered test case generation in fintech domain,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 314–315
2024
-
[6]
Applications of federated learning in ai, iot, healthcare, finance, banking, and cross-domain learning,
W. Hassan and H. Mohamed, “Applications of federated learning in ai, iot, healthcare, finance, banking, and cross-domain learning,” inArtificial Intelligence Using Federated Learning. CRC Press, 2024, pp. 175–195
2024
-
[7]
On-device language models: A comprehensive review,
J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, and Z. Ling, “On-device language models: A comprehensive review,”arXiv preprint arXiv:2409.00088, 2024
Pith/arXiv arXiv 2024
-
[8]
Octo-planner: On-device language model for planner-action agents,
W. Chen, Z. Li, Z. Guo, and Y . Shen, “Octo-planner: On-device language model for planner-action agents,”arXiv preprint arXiv:2406.18082, 2024
Pith/arXiv arXiv 2024
-
[9]
Octopus v2: On-device language model for super agent,
W. Chen and Z. Li, “Octopus v2: On-device language model for super agent,”arXiv preprint arXiv:2404.01744, 2024
Pith/arXiv arXiv 2024
-
[10]
Octopus v3: Technical report for on-device sub-billion multi- modal ai agent,
——, “Octopus v3: Technical report for on-device sub-billion multi- modal ai agent,”arXiv preprint arXiv:2404.11459, 2024
Pith/arXiv arXiv 2024
-
[11]
Octopus v4: Graph of language models,
——, “Octopus v4: Graph of language models,”arXiv preprint arXiv:2404.19296, 2024
Pith/arXiv arXiv 2024
-
[12]
No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml,
Z. Zhang, C. Gong, Y . Cai, Y . Yuan, B. Liu, D. Li, Y . Guo, and X. Chen, “No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 3327–3345
2024
-
[13]
{SoK}: All you need to know about{On-Device}{ML}model extraction-the gap between research and practice,
T. Nayan, Q. Guo, M. Al Duniawi, M. Botacin, S. Uluagac, and R. Sun, “{SoK}: All you need to know about{On-Device}{ML}model extraction-the gap between research and practice,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 5233–5250
2024
-
[14]
Artificial intelligence and its im- plications for data privacy,
K. D. Martin and J. Zimmermann, “Artificial intelligence and its im- plications for data privacy,”Current Opinion in Psychology, p. 101829, 2024
2024
-
[15]
Dpdr: Gradient decomposition and reconstruction for differentially private deep learning,
Y . Liu, L. Xiong, Y . Liu, Y . Gu, R. Liu, and H. Chen, “Dpdr: Gradient decomposition and reconstruction for differentially private deep learning,”arXiv preprint arXiv:2406.02744, 2024
Pith/arXiv arXiv 2024
-
[16]
Finsosent: Advancing financial market sentiment analysis through pretrained large language models,
J. Delgadillo, J. Kinyua, and C. Mutigwe, “Finsosent: Advancing financial market sentiment analysis through pretrained large language models,”Big Data and Cognitive Computing, vol. 8, no. 8, p. 87, 2024
2024
-
[17]
Fingpt: Open-source financial large language models,
H. Yang, X.-Y . Liu, and C. D. Wang, “Fingpt: Open-source financial large language models,”arXiv preprint arXiv:2306.06031, 2023
arXiv 2023
-
[18]
Finbert: A large language model for extracting information from financial text,
A. H. Huang, H. Wang, and Y . Yang, “Finbert: A large language model for extracting information from financial text,”Contemporary Accounting Research, vol. 40, no. 2, pp. 806–841, 2023
2023
-
[19]
Extracting financial data from unstructured sources: Leveraging large language models,
H. Li, H. Gao, C. Wu, and M. A. Vasarhelyi, “Extracting financial data from unstructured sources: Leveraging large language models,”Journal of Information Systems, pp. 1–22, 2023
2023
-
[20]
Revolutionizing finance with llms: An overview of applications and insights,
H. Zhao, Z. Liu, Z. Wu, Y . Li, T. Yang, P. Shu, S. Xu, H. Dai, L. Zhao, G. Maiet al., “Revolutionizing finance with llms: An overview of applications and insights,”arXiv preprint arXiv:2401.11641, 2024
arXiv 2024
-
[21]
When ai meets information privacy: The adversarial role of ai in data sharing scenario,
A. Majeed and S. O. Hwang, “When ai meets information privacy: The adversarial role of ai in data sharing scenario,”IEEE Access, 2023
2023
-
[22]
Ai-driven anonymization: Protecting personal data privacy while leveraging ma- chine learning,
L. Yang, M. Tian, D. Xin, Q. Cheng, and J. Zheng, “Ai-driven anonymization: Protecting personal data privacy while leveraging ma- chine learning,”arXiv preprint arXiv:2402.17191, 2024
Pith/arXiv arXiv 2024
-
[23]
Sa-dpsgd: Differentially private stochas- tic gradient descent based on simulated annealing,
J. Fu, Z. Chen, and X. Ling, “Sa-dpsgd: Differentially private stochas- tic gradient descent based on simulated annealing,”arXiv preprint arXiv:2211.07218, 2022
Pith/arXiv arXiv 2022
-
[24]
Attention is all you need,
A. Vaswani, “Attention is all you need,”Advances in Neural Information Processing Systems, 2017
2017
-
[25]
Llama 2: Open foundation and fine-tuned chat models,
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023
Pith/arXiv arXiv 2023
-
[26]
Gqa: Training generalized multi-query transformer models from multi-head checkpoints,
J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebr ´on, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,”arXiv preprint arXiv:2305.13245, 2023
Pith/arXiv arXiv 2023
-
[27]
Root mean square layer normalization,
B. Zhang and R. Sennrich, “Root mean square layer normalization,” Advances in Neural Information Processing Systems, vol. 32, 2019
2019
-
[28]
Glu variants improve transformer,
N. Shazeer, “Glu variants improve transformer,”arXiv preprint arXiv:2002.05202, 2020
Pith/arXiv arXiv 2002
-
[29]
Roformer: En- hanced transformer with rotary position embedding,
J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024
2024
-
[30]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021
Pith/arXiv arXiv 2021
-
[31]
Our data, ourselves: Privacy via distributed noise generation,
C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” inAdvances in Cryptology-EUROCRYPT 2006: 24th Annual International Confer- ence on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25. Springer, 2006, pp. 486–503
2006
-
[32]
Deep learning with differential privacy,
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318
2016
-
[33]
Large language models can be strong differentially private learners,
X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,”arXiv preprint arXiv:2110.05679, 2021
Pith/arXiv arXiv 2021
-
[34]
Good debt or bad debt: Detecting semantic orientations in economic texts,
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala, “Good debt or bad debt: Detecting semantic orientations in economic texts,”Journal of the Association for Information Science and Technology, vol. 65, no. 4, pp. 782–796, 2014
2014
-
[35]
Www’18 open challenge: financial opinion mining and question answering,
M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur, “Www’18 open challenge: financial opinion mining and question answering,” inCompanion proceedings of the the web conference 2018, 2018, pp. 1941–1942
2018
-
[36]
Twitter financial news sentiment
N. Magic, “Twitter financial news sentiment.” https://huggingface.co/ datasets/zeroshot/twitter-financialnews-sentiment, 2022
2022
-
[37]
Data-centric fingpt. open-source for open finance,
H. Yang, “Data-centric fingpt. open-source for open finance,” .https: //github.com/AI4Finance-Foundation/FinGPT, 2023
2023
-
[38]
Chatglm: A family of large language models from glm-130b to glm-4 all tools,
T. GLM, “Chatglm: A family of large language models from glm-130b to glm-4 all tools,” 2024
2024
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.