REVIEW 5 major objections 6 minor 15 references
TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Texts
T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper argues that translating an established English financial benchmark into Korean cannot faithfully evaluate embedding models, because financial meaning depends on language- and culture-specific expressions, and it introduces a…
desk verdict KorFinMTEB is a real resource, but the paper's central claim about translated benchmarks is not supported by its own numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is KorFinMTEB itself: a benchmark of 26 datasets covering classification, clustering, retrieval, summarization, pair classification, reranking, and semantic textual similarity, assembled from native Korean financial sources such as news, disclosures, dictionaries, reports, and curated QA and summarization data, following FinMTEB's task format. The experimental design pairs it with Translated-FinMTEB, a GPT-4o translation of FinMTEB into Korean, and compares seven embedding models plus a Korean-fine-tuned model under identical hyperparameters to isolate the effect of linguistic and cultural authenticity.
What would settle it
A direct test would be to have Korean financial experts, blind to which benchmark an item came from, rate items from KorFinMTEB and Translated-FinMTEB for naturalness and domain faithfulness; if the translated items are judged equally natural and faithful, or if an independent second native Korean financial benchmark shows no systematic 5-8% gap against translated FinMTEB, the paper's central claim would be falsified.
Extended reading notes
Core claim
The central discovery is a consistent, task-dependent discrepancy: embedding models that look strong on translated FinMTEB lose 5-8% accuracy or correlation on KorFinMTEB's semantic-textual-similarity, pair-classification, and summarization tasks, while simple classification tasks show only small differences. Retrieval shows the opposite pattern, with models often scoring higher on the native benchmark, which the paper attributes to translation artifacts producing unnatural queries that misalign with genuine Korean corpus text. The authors interpret these patterns as evidence that translated benchmarks 'fail to reflect the linguistic and contextual depth of low-resource domains, leading to inflated or inconsistent performance metrics,' and that KorFinMTEB is the more reliable evaluation framework.
Load-bearing premise
The claim's load-bearing premise is that KorFinMTEB itself measures the real linguistic and cultural demands of Korean financial text, so that score differences between it and the translated benchmark reflect authenticity rather than differences in task difficulty, labeling standards, or data sources.
Editorial extensions
If this is right
- If KorFinMTEB is a faithful benchmark, model leaderboards for Korean finance should be re-run on it rather than on translated English tasks.
- Developers of embedding models for low-resource domains should expect performance drops on native benchmarks and should train on in-language data to close them.
- Benchmark builders for other low-resource languages should construct native datasets instead of relying on machine translation of high-resource benchmarks.
- The 5-8% gap on semantic tasks provides a specific, quantified target that future Korean financial embedding models could aim to close.
- Translation-based benchmarks may also distort retrieval evaluation, since translation artifacts can either inflate or deflate scores depending on task.
Reading between the lines
- The paper does not validate KorFinMTEB's own quality against an independent gold standard, so a fair test of its central claim would require measuring inter-annotator agreement on its labels and comparing it with another independently built Korean financial benchmark.
- The direction of the gap—lower on native semantic tasks but higher on native retrieval—suggests the two benchmarks differ in difficulty and content, not only in cultural authenticity; isolating nuance would require difficulty-matched task pairs.
- By analogy, other low-resource domains such as legal or medical Korean, or other languages such as Vietnamese or Arabic in finance, could exhibit the same benchmark-translation failure, providing a cheap test of the paper's generality.
- If the conclusion holds, multilingual embedding models that report high MTEB-style scores may systematically overstate their quality for domain-specific low-resource use, and evaluation suites should include native-domain tasks before deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces KorFinMTEB, a Korean financial-domain embedding benchmark with 7 tasks and 26 datasets assembled from native Korean sources, and compares eight embedding models on KorFinMTEB against Trans-ko-FinMTEB, a GPT-4o Korean translation of FinMTEB. The central claim is that direct translation of high-resource benchmarks fails to capture low-resource domain nuances and that the observed performance differences between the two benchmarks demonstrate this. The authors also argue that Korean-fine-tuned models are more robust and advocate native benchmark construction for other low-resource languages.
Significance. If the comparison were properly controlled, the paper would make a useful contribution: KorFinMTEB is a public, broad-coverage benchmark for an under-resourced domain/language pair, and the authors evaluate eight embedding models under a common FinMTEB-style protocol. The release of 26 datasets and the focus on native Korean financial text are concrete strengths. However, the paper's headline empirical claim depends on an uncontrolled comparison between two benchmarks that differ in source texts, topics, and annotation protocols. As reported, the tables also contain internal arithmetic inconsistencies and omit the summarization results invoked in the analysis. The benchmark itself may be a valuable resource, but the comparative evidence for the 'translation fails' thesis is not yet established at the level required by the claims.
major comments (5)
- [Section 3.3, Tables 6-9] The claimed 5-8% performance drop is contradicted by the reported numbers. For PairClassification, differences range from 0.086 to 0.455 in average precision; for Reranking, from 0.266 to 0.598; for STS, from -0.073 to 0.183. Even if these are read as percentage-point differences, the magnitudes are much larger than the claimed 5-8%. Moreover, summarization is listed as a task showing the drop, but no summarization results appear in Tables 6-9 or anywhere else in the paper. The abstract and conclusion therefore overstate the quantitative support.
- [Sections 3.1 and 3.2] The native-vs-translated comparison is confounded. KorFinMTEB is built from AI Hub, DART, BoK/FSS dictionaries, and in-house curation, while Trans-ko-FinMTEB is a GPT-4o translation of FinMTEB's English tasks. The two benchmarks share 'identical task formats' but not source texts, topic distributions, or difficulty. Without an aligned control (e.g., translating KorFinMTEB examples into English and comparing with the original, or human-validated equivalent pairs), the observed gaps can be explained by benchmark difficulty rather than by translation artifacts. Section 3.3 itself concedes that translation changes task difficulty via 'reduced linguistic variability,' which undermines the causal reading of the gap.
- [Table 7] The diff column contains arithmetic errors. For bge-en-icl, 0.135 - 0.030 = 0.105, not 0.096; for all-MiniLM-L12-v2, 0.050 - (-0.003) = 0.053, not 0.054; for e5-mistral-7b-instruct, 0.142 - 0.025 = 0.117, not 0.116; for instructor-base, -0.023 - 0.049 = -0.072, not -0.073. Since the diff column is the paper's main quantitative evidence, these errors need to be corrected and all tables rechecked.
- [Table 8] The FiQA2018Reranking block appears to reuse PairClassification scores. For bge-large-en-v1.5, text-embedding-3-small, all-MiniLM-L12-v2, instructor-base, and kure-v1, the FinMTEB and KorFinMTEB scores are identical to those listed for HeadlineACPairClassification (e.g., 0.751/0.437 for bge-large-en-v1.5), while the diff values for bge-en-icl and gte-Qwen2-1.5B-instruct correspond to the FiQA2018 rows. This suggests a copy-paste error that invalidates the reranking and possibly clustering columns. The entire table needs to be regenerated and verified.
- [Appendix A.1] No inter-annotator agreement or other reliability measure is reported for the in-house curated datasets, and the 'quality verification' of FinSTS-ko is described only as being performed by people with financial expertise, without quantitative evidence. Since KorFinMTEB is proposed as a more reliable benchmark than a translated one, the absence of reliability evidence is a load-bearing gap. The authors should report annotation procedures, agreement statistics, and a data statement.
minor comments (6)
- [Title and Abstract] The acronym 'TWICE' in the title is never defined in the text.
- [Tables 6-9] The column labeled 'FinMTEB Score' should clearly state that it refers to Trans-ko-FinMTEB, to avoid confusion with the original English FinMTEB.
- [Table 8] The table lacks visual separation between HeadlineACPairClassification, FiQA2018Reranking, and dartcompany2industryclustering; the three blocks run together on the page, making the table hard to read.
- [Section 3.1] The text mentions 'Sujet-Finance-Instruct-177k-ko' as a pair-classification source, but the appendix does not list this dataset in the pair-classification table; please add a source description or remove the mention.
- [Throughout] There are several typos: 'low-resources languages' in the abstract should be 'low-resource languages'; 'Korea Financial Texts' in the title should likely be 'Korean Financial Texts'; and 'FInancialMCQA-CLS-ko' has inconsistent capitalization.
- [Availability] The paper states that the dataset is 'fully open-sourced' but does not provide a URL or repository identifier in the main text; please add an availability section with a link to the code and data.
Circularity Check
No significant circularity: the central claim rests on an empirical benchmark comparison, not on a derivation or fitted identity.
full rationale
This paper is an empirical benchmark-construction and evaluation study. It contains no equations, no fitted parameters, and no derivation chain whose output is equivalent to its input by construction. The central claim—that translated FinMTEB underrepresents Korean financial linguistic and cultural nuances—is supported by observed performance differences in Tables 6–9 between a GPT-4o translation of FinMTEB and the newly constructed KorFinMTEB. Those differences are measurements, not consequences of the benchmark definition. The self-citations (Son et al., 2023; Son et al., 2024a,b) are used as data sources and as prior evidence that translation loses context, but the present comparison is independently computed on both benchmark variants and does not reduce to those citations. There is no fitted-input-called-prediction step, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The paper's in-text claim of a consistent 5–8% performance drop is not well supported by its own tables, which show much larger and inconsistent differences, and the native-vs-translated gap is confounded with task content and difficulty. However, confounding and internal inconsistency are correctness concerns, not circularity. Under the hard rules requiring a quoted reduction of an output to an input, no circular step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption GPT-4o Korean translation of FinMTEB is a fair and faithful comparator that preserves task format and difficulty (Section 3.2).
- domain assumption KorFinMTEB datasets are representative, high-quality samples of Korean financial text (Section 3.1).
- domain assumption Performance gaps between the two benchmarks measure linguistic and cultural nuance rather than unrelated benchmark differences (Section 3.3).
- domain assumption Standard MTEB and FinMTEB evaluation metrics and formats apply unchanged to Korean (Section 3.2).
Cite this review
Pith. "Pith review of TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Texts." pith.science (2026). https://pith.science/paper/X64VIAD7
@misc{pith2026250207131,
author = {Pith},
title = {Pith review of: TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/X64VIAD7}},
note = {Machine review of arXiv:2502.07131}
}
read the original abstract
Domain specificity of embedding models is critical for effective performance. However, existing benchmarks, such as FinMTEB, are primarily designed for high-resource languages, leaving low-resource settings, such as Korean, under-explored. Directly translating established English benchmarks often fails to capture the linguistic and cultural nuances present in low-resource domains. In this paper, titled TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Models Bring? A Case Study on Korea Financial Texts, we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages. Our experimental results reveal that while the models perform robustly on a translated version of FinMTEB, their performance on KorFinMTEB uncovers subtle yet critical discrepancies, especially in tasks requiring deeper semantic understanding, that underscore the limitations of direct translation. This discrepancy highlights the necessity of benchmarks that incorporate language-specific idiosyncrasies and cultural nuances. The insights from our study advocate for the development of domain-specific evaluation frameworks that can more accurately assess and drive the progress of embedding models in low-resource settings.
Figures
Reference graph
Works this paper leans on
-
[3]
Tianyu Gao, Xingcheng Yao, and Danqi Chen
URL https://arxiv.org/ abs/1810.04805. Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings,
-
[7]
Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives
URL https://arxiv.org/abs/2403.14341. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space,
-
[10]
URL http://dx.doi.org/10.1093/bioinformatics/ btz682
1093/bioinformatics/btz682. URL http://dx.doi.org/10.1093/bioinformatics/ btz682. Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. Making text embedders few-shot learners,
-
[11]
Guijin Son, Hanwool Lee, Nahyeon Kang, and Moonjeong Hahm
URL https://arxiv.org/abs/1908.10084. Guijin Son, Hanwool Lee, Nahyeon Kang, and Moonjeong Hahm. Removing non-stationary knowl- edge from pre-trained language models for entity-level sentiment classification in finance,
arXiv 1908
-
[12]
URL https://arxiv.org/abs/2301.03136. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean, 2024a. URL https://arxiv.org/abs/2402.11548. Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Wo...
-
[13]
URL https://arxiv.org/abs/2212.09741. Yixuan Tang and Yi Yang. Do we need domain-specific embedding models? an empirical investi- gation,
-
[14]
URL https://arxiv.org/abs/2409.18511. 6 Published as a conference paper at ICLR 2025 Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint arXiv:2401.00368,
arXiv 2025
-
[15]
URL https://arxiv.org/abs/2303.17564. A A PPENDIX ACKNOWLEDGMENTS This research was supported by Brian Impact Foundation, a non-profit organization dedicated to the advancement of science and technology for all. A.1 D ATASET DETAILS Dataset Description Source Law-Summ-ko Dataset for evaluating the accuracy of long text summarization in Korean legal docume...
arXiv 2025
Show all 15 references
-
[2013]
Niklas Muennighoff, Nouamane Tazi, Lo¨ıc Magne, and Nils Reimers
URL https://arxiv.org/abs/1301.3781. Niklas Muennighoff, Nouamane Tazi, Lo¨ıc Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark,
-
[2019]
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian- Ling Mao, Heyan Huang, and Ming Zhou
URL https://arxiv.org/abs/1908.10063. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian- Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information-theoretic framework for cross-lingual language model pre-training,
1908 arXiv
-
[2020]
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann
URL https://arxiv.org/abs/2002.10957. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance,
2002 arXiv
-
[2021]
URL https://arxiv.org/abs/2007. 07834. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding,
2007
-
[2022]
Youngjoon Jang, Junyoung Son, and Taemin Lee,
URL https://arxiv.org/abs/2104.08821. Youngjoon Jang, Junyoung Son, and Taemin Lee,
-
[2023]
Nils Reimers and Iryna Gurevych
URL https://arxiv.org/abs/2210.07316. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks,
-
[2024]
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang
URL https://arxiv.org/ abs/2409.15700. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281,
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.