Pith. sign in

REVIEW 4 major objections 6 minor 76 references

VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VIVID, a 1,636-item benchmark of Vietnamese idioms and proverbs, shows that current language models, including GPT-4o, explain them at less than half the maximum quality score.

desk verdict First systematic Vietnamese figurative-language benchmark with real empirical value, but the judge-reliability section has an internal inconsistency that needs fixing before the numbers are fully trusted. read the letter →

arxiv 2608.03095 v1 pith:PMJOTJWW submitted 2026-08-04 cs.CL cs.LG

Tu Tran Do , Nhat Ngoc Nguyen , Khanh-Tung Tran , Hoang D. Nguyen , Tu Minh Phuong , Long Hoang Dang This is my paper · ORCID
classification cs.CLcs.LG
keywords VietnameseidiomsproverbsfigurativelanguageculturalcompetenceLLM-as-a-Judgebenchmarkfew-shotpromptingGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VIVID is the first systematic benchmark for culturally grounded figurative language in Vietnamese: 1,636 idioms and proverbs, each paired with a dictionary-sourced explanation and labeled for five linguistic complexity traits and seven semantic themes. The paper uses it to argue that current language models lack cultural competence: on open-ended explanation generation, the best model tested, GPT-4o, averages 2.46 out of 5, below half of the maximum, while Vietnamese-specialized VinaLLaMA-7B manages only 0.13. A GPT-4.1 judge with aspect-based prompting, validated against human raters on 200 items (Cohen's kappa = 0.792), supplies the scores. The results also show that few-shot prompting is not universally helpful, degrading GPT-4o from 2.46 to 2.04 through stylistic overfitting. If correct, VIVID provides a measurable way to track whether future models are becoming culturally aware rather than merely fluent.

What carries the argument

The load-bearing mechanism is the evaluation loop built around VIVID. The benchmark itself is a curated set of 1,636 idiom/proverb–explanation pairs from two Vietnamese dictionaries, annotated with five complexity traits (literal expressions, pragmatic nuances, Sino-Vietnamese terms, uncommon vocabulary, folk knowledge) and seven semantic themes. On the generative side, models explain a phrase in Vietnamese, and GPT-4.1 scores the explanation 0–5 against the gold explanation using aspect-based prompting — explicit criteria for core-meaning accuracy, nuance capture, clarity, and completeness, with the rule that a score of 0 on core meaning forces an overall 0; this judge was validated against human raters on 200 items (kappa = 0.792) and chosen after comparing four prompting strategies. On the discriminative side, 3-shot exact-match classification tasks for topic and for complexity traits measure whether models can categorize the same cultural knowledge. The combination turns cultural competence into a number that can be compared across models and prompting choices.

What would settle it

Select a random sample of several hundred VIVID explanations, have native Vietnamese speakers score them on the same 0–5 scale, and compare model rankings with GPT-4.1's rankings; if human rankings disagree on which models are better, or if human scores for GPT-4o's explanations rise above half the maximum, the central 50% ceiling claim would fail.

Watch

Extended reading notes

Core claim

At the paper's core is the claim that cultural grounding is a distinct capability that current LLMs have not acquired, and that it can be benchmarked. On each of 1,636 Vietnamese idioms and proverbs, models are asked to write a short Vietnamese explanation; the explanations are scored 0–5 by GPT-4.1 against the dictionary explanation, with the judge asked to consider core-meaning accuracy, nuance, clarity, and completeness. Every model falls short: GPT-4o reaches 2.46, Gemini Flash 2.5 reaches 2.29, the best open-source model Llama-4-Scout reaches 1.24 with few-shot prompting, and Vietnamese-specialized models score at or below 1.09. The paper also identifies four recurring failure modes — literal over-interpretation, lexical gaps on archaic words, cultural disconnection on folk-knowledge idioms, and pragmatic flattening of sarcasm or irony — and shows that GPT-4o's few-shot degradation is driven by imitating the moralizing tone of the examples instead of preserving meaning. VIVID is offered as the first systematic tool for exposing and measuring those failures in Vietnamese.

Load-bearing premise

The whole scorecard depends on GPT-4.1's ratings matching what a native Vietnamese speaker would say about an explanation, but that match was checked on only 200 of the 1,636 entries.

Editorial extensions

If this is right

  • If VIVID measures what it claims, then language-specific pretraining is not a shortcut: Vietnamese-specialized GreenMind-14B and multilingual Qwen-3-14B score identically (1.09), so scale and data breadth, not Vietnamese tuning, drive figurative understanding.
  • The less-than-50% ceiling means state-of-the-art models cannot yet be trusted to explain or translate Vietnamese idioms without human oversight.
  • Few-shot prompting should be reported alongside zero-shot, because in this setting it materially lowers GPT-4o's score (2.46 to 2.04) by inducing stylistic imitation.
  • The taxonomy of failure modes — literal over-interpretation, lexical gaps, cultural disconnection, and pragmatic flattening — gives developers concrete targets: a model improves on VIVID only if those four error types decrease.
  • Researchers can reuse VIVID's 1,636 pairs and the validated judge prompt as a repeatable testbed for culturally aware NLP in Vietnamese.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the gold explanations come from a single dictionary entry per idiom, VIVID scores similarity to one canonical reading; a model that gives a different but culturally valid interpretation would be marked down, so a human acceptability study across multiple paraphrases could show that the true gap is smaller than 50% for some idioms.
  • The judge validation on 200 entries leaves open whether GPT-4.1 ranks models the same way humans would on harder categories such as folk knowledge; a larger human sample or an open-source judge could change the ranking.
  • The same benchmark design could be transplanted to other low-resource languages with rich oral traditions, with the aspect-based judge prompt as a reusable template; the 50% ceiling may be a general property of figurative language rather than Vietnamese-specific.
  • The exact-match scoring on discriminative tasks likely understates models that understand the trait but format the answer differently (one model scored zero on both tasks); relaxed matching could separate format failures from comprehension failures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. VIVID is a benchmark of 1,636 Vietnamese idioms and proverbs annotated with five complexity traits and seven semantic themes. The authors evaluate eight language models on two tasks: a generative explanation task scored by GPT-4.1 as a judge (with four prompting strategies compared against human ratings on a subset), and two discriminative classification tasks (topic and linguistic-characteristic identification). They report that all models, including the best-performing GPT-4o (2.46/5), achieve less than half of the maximum score; that Vietnamese-specialized models do not outperform similarly sized multilingual models; and that few-shot prompting degrades GPT-4o due to stylistic overfitting. The dataset and code are released.

Significance. If the reliability concerns are addressed, VIVID would fill a real gap: no existing benchmark targets Vietnamese figurative language, and the paper's dual-layer annotation (complexity traits plus semantic themes) is a useful resource for the community. The discriminative results (Table 5) independently show a large performance gap and thus provide partial support for the paper's broad conclusion. The systematic comparison of four LLM-as-a-judge prompting strategies (Table 1) is also a useful methodological contribution. However, the generative scorecard, which drives the headline 'below 50%' claim, rests on a judge whose human-agreement study is under-specified and internally inconsistent; the reliability certificate for the central claim therefore needs significant repair.

major comments (4)
  1. [§4.1.1] The reported human-annotation design is internally inconsistent. The text states that 200 idioms are randomly sampled for the reliability experiment, then says 'Two native Vietnamese speakers independently annotate 150 idiomatic expressions each, with a 50-sample overlap.' With 150+150−50 = 250 distinct annotations, this cannot be a 200-item sample; if each annotator covered 150 of the 200 items, the overlap would be 100, not 50. The paper also never reports inter-human agreement, saying only that reliability was 'strong.' Because the κ = 0.792 and the human baseline mean in Table 1 are the sole evidence that the GPT-4.1 judge is aligned with human judgment, the actual sample sizes, the overlap design, and the inter-human kappa must be reported precisely.
  2. [§4.1 and §3.3] There is a same-generator bias risk that is not addressed. GPT-4.1 is used both to generate the initial complexity-trait labels in Stage 1 of data validation and to score all generative outputs, and the evaluated models include GPT-4o, a sibling model from the same family. The 200-item human-agreement study does not rule out a systematic preference for GPT-family prose style, which could inflate GPT-4o's scores relative to smaller open-source models. I am not treating this as a circularity claim, but as an empirical correctness risk. The authors should report judge-human agreement separately for each evaluated model (at least for GPT-4o versus one or two open models) on the 200-item subset, and re-score a random subset with an alternative judge (e.g., an open-weight LLM or a second human annotator) to show that the rankings in Tables 3 and 4 are stable.
  3. [Abstract and §5.1] The headline 'less than 50% correctness' is not supported by the measurement protocol. The judge prompt (Figure 7) asks GPT-4.1 to rate 'overall similarity' between the model explanation and the gold human explanation on a 0–5 scale. A mean of 2.46 is 49.2% of the maximum score, but that does not imply that 49.2% of explanations are correct; the judge is not producing a binary correctness label. The abstract, introduction, and Section 5.1 should say 'less than 50% of the maximum judge-assigned similarity score' or otherwise avoid the term 'correctness' for a score that measures similarity to a reference explanation.
  4. [§3.3 and §7] The paper needs to separate the full human validation of all 1,636 annotations (Section 3.3) from the 200-item judge-reliability subset (Section 4.1.1). Section 3.3 says two native speakers 'independently reviewed all annotations,' while Section 7 limits the human–LLM agreement analysis to 200 samples and concedes that this subset 'may not fully capture the complete diversity.' These statements are not necessarily contradictory, but the manuscript should state explicitly whether the 200-item reliability subset is a subsample of the fully validated set and whether the same human labels were used in both places. Currently the reader cannot tell whether the judge was validated against the same gold-standard labels that constitute the benchmark, which is essential for interpreting κ = 0.792.
minor comments (6)
  1. [Table 2 vs Tables 3–4] The parameter count for Llama-4-Scout is inconsistent: Table 2 lists 'Llama-4-Scout (109B)', while Tables 3 and 4 and Section 5.1 refer to 'Llama-4-Scout-17B-16E'. The model should be described consistently with its actual configuration (17B active parameters, 16 experts).
  2. [References] Several references are duplicated under different citation tags: Le et al. 2022a/2022b (VIMQA), Ho et al. 2019a/2019b, Liu et al. 2022a/2022b, Tang et al. 2024a/2024b, and Wang et al. 2025a/2025b are the same papers in each pair. Also, Section 2.2 cites both ViGLUE and VLUE as '(Tran et al., 2024)' although the reference list contains distinct entries for Tran et al. (ViGLUE) and Do et al. (VLUE).
  3. [§4.1.1] The 'Human Baseline 3.47' row in Table 1 would be easier to interpret if the paper stated whether this is the mean of the two annotators' scores on the sampled items, how disagreements were resolved in computing this mean, and whether the human scores are the same gold explanations used in the full benchmark.
  4. [§5.1.2] The claim that few-shot prompting degrades GPT-4o is supported by aggregate means (2.46 zero-shot vs. 2.04 few-shot in Table 3) but no variance estimate or significance test is provided. Given that the two illustrative examples are anecdotal, a paired statistical test across the 1,636 items would substantially strengthen this secondary claim.
  5. [§3.3] The annotator qualification is described only as 'two native Vietnamese speakers with expertise in Vietnamese linguistics.' Given that the benchmark's validity depends on their judgments, the paper should briefly report their linguistic training or experience, and should describe the adjudication procedure for disagreements beyond 'discussion' (e.g., whether a third expert was consulted).
  6. [Throughout] The manuscript contains numerous ligature and spelling artifacts (e.g., 'difficult', 'official', 'sufficient') and inconsistent capitalization of model names (e.g., 'Vinallama' vs. 'VinaLLaMA'). These should be fixed in the camera-ready version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: gold labels and human-anchored judge are externally grounded; the annotation-count inconsistency is a reliability issue, not a circular reduction.

full rationale

The paper's central derivation—model scores on VIVID—does not reduce to its inputs. The generative scores are produced by a GPT-4.1 judge that compares each model explanation against dictionary-sourced gold explanations (Nguyen Lan 2010; Vu Dung et al. 2000); the gold is external to the evaluated models and to the judge's own prior annotations. The judge is anchored to human judgment on a random 200-item subset (κ=0.792), and the aspect-based prompt was selected by highest human agreement, not by the rankings it later produces; this is measurement validation, not a fitted parameter renamed as a prediction. The complexity-trait and theme labels were initially generated by GPT-4.1 but then independently reviewed, refined, and consensus-adjudicated by two native-speaker annotators, so the discriminative tasks' targets are not self-defined. The same-family risk (GPT-4.1 judges GPT-4o) is a validity threat but is broken by the human-agreement study and by the dictionary ground truth; no equation in the paper equates the judge's score with any input label. Section 7 explicitly concedes reliance on a closed model and the limited diversity of the 200-item validation subset, and §4.1.1's annotation counts (150+150−50=250 vs. a 200-item sample) are internally inconsistent, calling the kappa's evidential basis into question—but these are reliability and transparency problems, not circular reductions. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numbers are fitted to data; the central results are measurements from an evaluation benchmark. The main assumptions are that the selected dictionaries provide accurate gold explanations, that the taxonomy is complete and correctly applied, and that the LLM judge's agreement with humans generalizes beyond the small validation sample. The invented-entities list is empty because VIVID is a dataset, not a postulated scientific entity.

assumptions (4)
  • domain assumption The two source dictionaries (Nguyen Lan 2010; Vu Dung et al. 2000) provide correct and complete explanations for each idiom or proverb.
    Section 3.1 treats these dictionary explanations as the ground truth against which all models and the human baseline are scored; if an explanation is wrong or incomplete, all scores are systematically biased.
  • ad hoc to paper The five-trait taxonomy and seven theme categories capture the principal challenges of Vietnamese figurative understanding.
    Section 3.2.1 says the taxonomy is theory-driven and expert-consulted, but it is not independently validated against an external benchmark, and the initial trait labels were generated by GPT-4.1.
  • domain assumption The LLM-as-a-Judge (GPT-4.1) alignment with human judgment, measured on a small subset, extends to the full dataset.
    Section 4.1.1 and Section 7: the roughly 200-sample reliability study is the only evidence that judge scores match human judgment, and the authors concede it may not capture full diversity.
  • domain assumption Human judgment of two native speakers is reliable ground truth.
    Section 4.1.1 states 'strong inter-human annotator reliability' without reporting a coefficient, so the upper bound of judge-human agreement is unknown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP." pith.science (2026). https://pith.science/paper/PMJOTJWW

@misc{pith2026260803095,
  author       = {Pith},
  title        = {Pith review of: VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMJOTJWW}},
  note         = {Machine review of arXiv:2608.03095}
}
read the original abstract

We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese. VIVID comprises 1,636 idioms and proverbs annotated with five complexity traits (literal expressions, pragmatic nuances, Sino-Vietnamese terms, uncommon vocabulary, folk knowledge) and seven semantic themes. We establish an evaluation framework combining generative and discriminative tasks, proposing an LLM-as-a-Judge approach with aspect-based prompting validated against human judgment (Cohen's kappa = 0.792). Evaluating eight state-of-the-art models reveals critical gaps: Vietnamese-specialized models drastically underperform multilingual systems (VinaLLaMA-7B: 0.13 vs. GPT-4o: 2.46), and even top models achieve less than 50% of maximum scores. Notably, few-shot prompting does not universally improve performance, with GPT-4o exhibiting degradation due to stylistic overfitting. Our analysis exposes systematic failures including literal over-interpretation, lexical gaps, and pragmatic flattening, demonstrating that current models lack cultural competence for nuanced figurative interpretation. VIVID provides an essential tool for advancing figurative language understanding in culturally rich contexts.

Figures

Figures reproduced from arXiv: 2608.03095 by the authors.

Figure 1
Figure 1. Overview of the data collection and annotation pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Statistics of VIVID per topics. Vietnamese daily life, folk wisdom, and moral education. This classification schema was de￾rived through thematic analysis of idiomatic con￾tent commonly encountered in colloquial usage, folk literature, and educational materials. The seven themes are defined as follows: • Criticism: Expressions conveying disap￾proval, mockery, or critical judgment of behav￾iors, traits, or situations… view at source ↗
Figure 4
Figure 4. Taxonomy of idiomatic expressions that pose challenges for LLMs due to literal meaning, archaic vocabulary, cultural specificity, and other linguistic features. agricultural practices, natural phenomena, or traditional livelihoods • Other: Expressions addressing miscella￾neous topics including abstract concepts, spe￾cific scenarios, or descriptions • Social Relationships: Idioms reflecting in￾terpersonal dynamics wi… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Zero-shot prompt for idiom explanation generation (Vietnamese original left, English trans￾lation right). Few-shot Prompt. The few-shot prompt (Fig￾ure 6) situates the model as a Vietnamese lan￾guage expert and provides three demonstrative examples covering diverse idi…
Figure 6
Figure 6. Figure 6: Few-shot prompt for idiom explanation generation, with three in-context demonstrations (Viet￾namese original left, English translation right). B.1. Topic Classification Prompt The topic classification prompt is shown in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Aspect-based LLM-as-a-Judge prompt (Vietnamese original left, English translation right). GPT-4.1 outputs only the integer overall similarity score (0–5) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: In-context scoring examples embedded in the judge prompt to calibrate the 0–5 scale (Viet￾namese original left, English translation right) [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Topic classification prompt for the topic_labeling discriminative task (Vietnamese original left, English translation right) [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Linguistic complexity classification prompt for the pattern_labeling discriminative task (Vietnamese original left, English translation right). Models must predict all applicable traits; exact match is required for a correct prediction [PITH_FULL_IMAGE:figures/full_f…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 71 canonical work pages

  1. [1]

    Introduction Figurative language understanding remains a criti- cal challenge in natural language processing, re- quiring models to integrate linguistic knowledge, cultural context, and pragmatic reasoning ( Liu et al., 2022a). In Vietnamese linguistics, an idiom is defined as a fixed multi-word expression whose meaning cannot be straightforwardly inferre...

    work page Pith review arXiv 2021

  2. [2]

    Idiom and Proverb Datasets Existing figurative language datasets focus pri- marily on high-resource languages

    Related Work 2.1. Idiom and Proverb Datasets Existing figurative language datasets focus pri- marily on high-resource languages. ( Chakrabarty et al., 2022) introduce FLUTE, a synthetic dataset of 1000 English idiomatic sentences paired with synonymous or contradictory statements. ( T ay- yar Madabushi et al. , 2022) present SemEval- 2022 T ask 2 for mult...

    work page 2022

  3. [3]

    Da đồng lông móc

    Dataset We present VIVID, a culturally grounded evalua- tion benchmark dataset contains 1636 idioms and proverbs across 7 topics that capture Vietnamese cultural contexts, ranging from everyday life to spe- cific subject areas, as well as Vietnamese gram- mar and linguistics We further annotate the dataset into five dis- tinct characteristics that LLMs ar...

    work page 2010

  4. [4]

    Evaluation Framework and Experiment Setup In this section, we introduce our figurative lan- guage evaluation framework, which integrates both generative and discriminative evaluations of Vietnamese idioms and proverbs under the LLM- as-a-Judge paradigm. This setup not only ensures reliable benchmarking but also extends prior eval- uation research through ...

    work page 2025

  5. [5]

    Entries with clearly inaccurate or culturally inappropriate labels were flagged for removal, while uncer- tain cases were marked for collaborative dis- cussion

    Initial Review : Each annotator examined the LLM-generated complexity trait labels for con- textual relevance and accuracy. Entries with clearly inaccurate or culturally inappropriate labels were flagged for removal, while uncer- tain cases were marked for collaborative dis- cussion

    work page

  6. [6]

    Refinement: For the semantic theme classifi- cation (Section 3.2.2), both annotators inde- pendently assigned each idiom-explanation pair to one of the seven predefined themes based on its core semantic content. Cases with disagreement or ambiguity were jointly reviewed, and both annotators collaboratively revised the labels to ensure alignment with the p...

    work page

  7. [7]

    Only idiom-explanation pairs achieving unanimous agreement on both com- plexity traits and semantic themes were in- cluded in the final dataset

    Final Curation : All disagreements were re- solved through discussion between the two annotators. Only idiom-explanation pairs achieving unanimous agreement on both com- plexity traits and semantic themes were in- cluded in the final dataset. This validation process resulted in a high- quality curated dataset comprising 1,636 idiom- explanation pairs with...

    work page

  8. [8]

    For the purpose of Open Ac- cess, the author has applied a CC BY public copy- right licence to any Author Accepted Manuscript version arising from this submission

    Acknowledgements This publication has emanated from research sup- ported in part by grants from Research Ireland under Grant [12-RC-2289-P2] and [18/CRT/6223] which is co-funded under the European Regional Development Fund. For the purpose of Open Ac- cess, the author has applied a CC BY public copy- right licence to any Author Accepted Manuscript version...

    work page

Show all 76 references
  1. [9]

    Chưa lại người

    Result & Discussion 5.1. Performance on Generative Task We evaluate eight LLMs on the idiom/proverb ex- planation generation task in VIVID, under Zero- shot and Few-shot prompting across seven seman- tic themes (T able 3). Smaller models (7-8B pa- rameters) struggle significan...

    2024

  2. [10]

    Our ex- tensive evaluation reveals that even state-of-the- art models achieve less than 50% correctness on average, underscoring fundamental gaps in cultural language understanding

    Conclusion We present VIVID, the first systematic benchmark for evaluating culturally grounded figurative lan- guage understanding in Vietnamese, comprising 1,636 idioms and proverbs annotated with linguis- tic complexity traits and semantic themes. Our ex- tensive evaluation ...

  3. [11]

    De- pendence on a closed model raises concerns re- garding transparency and long-term reproducibil- ity

    Limitations First, our automatic evaluation framework relies on a proprietary closed-source model (GPT-4.1). De- pendence on a closed model raises concerns re- garding transparency and long-term reproducibil- ity. Commercial models may be updated or modi- fied over time withou...

  4. [13]

    Bibliographical References Israel Abebe Azime, Atnafu Lambebo Tonja, T adesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Der- artu Dagne Geremew, Assefa Atsbiha Tesfu, Philipp Slusa...

    2025 arXiv

  5. [15]

    arXiv preprint arXiv:2411.16594

    From generation to judgment: Opportu- nities and challenges of llm-as-a-judge . arXiv preprint arXiv:2411.16594. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022a. T esting the ability of language models to interpret figurative lan- guage. In Proceedings of the 2...

    2022

  6. [16]

    In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evaluation

    Konidioms corpus: A dataset of idioms in konkani language . In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evaluation. Quan Nguyen, Huy Pham, and Dung Dao. 2023. Vinallama: Llama-based vietnamese founda- tion mo...

    2024 arXiv

  7. [19]

    Độ chính xác của ý nghĩa cốt lõi (0–5 điểm): Giải thích của LLM nắm bắt đúng và chính xác ý nghĩa chính của câu thành ngữ/tục ngữ so với giải thích gốc? (5: Rất chính xác, 0: Sai lệch hoàn toàn)

  8. [20]

    Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý (0–5 điểm): Giải thích của LLM có phản ánh được các sắc thái, ngữ cảnh sử dụng điển hình, hoặc ý nghĩa ẩn sâu của câu nói như giải thích gốc không? (5: Rất tương đồng, 0: Không phản ánh được)

  9. [21]

    Chất lượng diễn đạt, sự rõ ràng, lưu loát (0–5 điểm): Giải thích của LLM có rõ ràng, dễ hiểu, mạch lạc và tự nhiên không? (5: Rất tốt, 0: Rất kém/khó hiểu)

  10. [22]

    Độ đầy đủ (0–5 điểm): Giải thích của LLM có bao gồm các ý chính được đề cập trong giải thích gốc không? (5: Rất đầy đủ, 0: Thiếu nhiều ý chính)

  11. [23]

    Độ tương đồng tổng thể (0–5 điểm): Tổng hợp các đánh giá trên, mức độ tương đồng chung là bao nhiêu? (5: Rất tương đồng, 0: Không tương đồng) [In-context examples — see Figure 8] [Nhiệm vụ cần đánh giá] Thành ngữ/Tục ngữ: {phrase} Giải thích của Con người: {ground_truth} Giải ...

  12. [24]

    Độ chính xác của ý nghĩa cốt lõi:

  13. [25]

    Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý:

  14. [26]

    Chất lượng diễn đạt, sự rõ ràng, lưu loát:

  15. [27]

    Độ đầy đủ: Nếu tiêu chí 1 được 0 điểm thì Độ tương đồng tổng thể được 0. Độ tương đồng tổng thể: hãy suy nghĩ sau đó chỉ trả về duy nhất đánh giá của độ tương đồng tổng thể và không giải thích gì thêm English (translation) [Instructions] You are a language expert. Your task is...

  16. [28]

    Accuracy of core meaning (0–5 points): Does the LLM explanation cor- rectly and precisely capture the main mean- ing of the idiom/proverb compared to the original explanation? (5: Very accurate, 0: Completely incorrect)

  17. [29]

    Capture of nuance, context, and im- plicit meaning (0–5 points): Does the LLM explanation reflect the nuances, typ- ical usage contexts, or deeper implied meanings of the expression as in the orig- inal explanation? (5: Highly similar, 0: Does not reflect them at all)

  18. [30]

    Expression quality, clarity, and flu- ency (0–5 points): Is the LLM explanation clear, easy to understand, coherent, and natural? (5: Excellent, 0: Very poor/difficult to understand)

  19. [31]

    Completeness (0–5 points): Does the LLM explanation include the key points mentioned in the original explanation? (5: Very complete, 0: Missing many key points)

  20. [32]

    Overall similarity (0–5 points): Syn- thesising the above assessments, what is the overall degree of similarity? (5: Highly similar, 0: Not similar at all) [In-context examples — see Figure 8] [Task to evaluate] Idiom/Proverb: {phrase} Human explanation: {ground_truth} LLM exp...

  21. [33]

    Accuracy of core meaning:

  22. [34]

    Capture of nuance, context, and implicit meaning:

  23. [35]

    Expression quality, clarity, and fluency:

  24. [36]

    Chưa đủ lông đủ cánh

    Completeness: If criterion 1 receives 0 points, the overall similarity score must also be 0. Overall similarity: Think carefully, then re- turn only the overall similarity score and provide no further explanation. Figure 7: Aspect-based LLM-as-a-Judge prompt (Vietnamese origin...

  25. [37]

    Độ chính xác của ý nghĩa cốt lõi: 5/5

  26. [38]

    Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý: 5/5

  27. [39]

    Chất lượng diễn đạt, sự rõ ràng, lưu loát: 5/5

  28. [40]

    Kim chỉ có đầu

    Độ đầy đủ: 5/5 Độ tương đồng tổng thể: 5/5 Ví dụ Minh họa 2 Thành ngữ/Tục ngữ: Kim chỉ có đầu Giải thích nghĩa gốc: Phải có trên dưới, đầu đuôi, trật tự văn phép. Giải nghĩa do LLM: Câu thành ngữ “Kim chỉ có đầu” mang ý nghĩa rằng mọi việc đều có nguyên nhân và nguồn gốc, khôn...

  29. [41]

    Độ chính xác của ý nghĩa cốt lõi: 1/5

  30. [42]

    Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý: 1/5

  31. [43]

    Chất lượng diễn đạt, sự rõ ràng, lưu loát: 3/5

  32. [44]

    Not yet fully feathered

    Độ đầy đủ: 1/5 Độ tương đồng tổng thể: 1/5 English (translation) Illustrative Example 1 Idiom/Proverb: Chưa đủ lông đủ cánh (“Not yet fully feathered”) Human explanation: Still inexperienced; not yet capable of independence. LLM explanation: The idiom “Chưa đủ lông đủ cánh” is...

  33. [45]

    Accuracy of core meaning: 5/5

  34. [46]

    Capture of nuance, context, and implicit meaning: 5/5

  35. [47]

    Expression quality, clarity, and fluency: 5/5

  36. [48]

    The nee- dle and thread have a head

    Completeness: 5/5 Overall similarity: 5/5 Illustrative Example 2 (low-quality re- sponse) Idiom/Proverb: Kim chỉ có đầu (“The nee- dle and thread have a head”) Human explanation: There must be a proper order — top and bottom, beginning and end, social hierarchy and etiquette. ...

  37. [49]

    Accuracy of core meaning: 1/5

  38. [50]

    Capture of nuance, context, and implicit meaning: 1/5

  39. [51]

    Expression quality, clarity, and fluency: 3/5

  40. [52]

    {phrase}

    Completeness: 1/5 Overall similarity: 1/5 Figure 8: In-context scoring examples embedded in the judge prompt to calibrate the 0–5 scale (Viet- namese original left, English translation right). Vietnamese Bạn là chuyên gia ngôn ngữ. Dưới đây là một câu thành ngữ/tục ngữ: “ {phr...

  41. [53]

    Tình yêu đôi lứa và hôn nhân Ví dụ: chẳng nên tình trước nghĩa sau, có con ta gả cho nhau thiệt gì

  42. [54]

    Ca ngợi phẩm chất và đạo đức con người Ví dụ: cân quắc anh hùng

  43. [55]

    Phê phán và châm biếm các thói quen xấu, hành vi tiêu cực Ví dụ: ai biết uốn câu cho vừa miệng cá

  44. [56]

    Kinh nghiệm sống về sản xuất, lao động và thiên nhiên Ví dụ: ác tắm thì ráo, sáo tắm thì mưa

  45. [57]

    Các mối quan hệ xã hội, gia đình Ví dụ: ắm con chồng hơn bồng cháu ngoại

  46. [58]

    Triết lý và bài học về cách sống Ví dụ: con ruồi đậu nặng đồng cân

  47. [59]

    {phrase}

    Các chủ đề khác (nếu không phù hợp với các mục trên) Chỉ trả lời duy nhất một số (1–7). Không giải thích, không thêm ký tự nào khác. English (translation) You are a language expert. Below is a Viet- namese idiom or proverb: “ {phrase}” Your task is to identify which of the fol...

  48. [60]

    Romantic love and marriage Example: chẳng nên tình trước nghĩa sau, có con ta gả cho nhau thiệt gì

  49. [61]

    Praising human virtues and moral character Example: cân quắc anh hùng

  50. [62]

    Criticising and satirising bad habits or negative behaviour Example: ai biết uốn câu cho vừa miệng cá

  51. [63]

    Life experience about production, labour, and nature Example: ác tắm thì ráo, sáo tắm thì mưa

  52. [64]

    Social and family relationships Example: ắm con chồng hơn bồng cháu ngoại

  53. [65]

    Philosophy and life lessons Example: con ruồi đậu nặng đồng cân

  54. [66]

    {phrase}

    Other topics (if none of the above ap- ply) Reply with a single number only (1–7). No explanation, no extra characters. Figure 9: Topic classification prompt for the topic_labeling discriminative task (Vietnamese original left, English translation right). Vietnamese Bạn là chu...

  55. [67]

    Ví dụ: Da đồng lông móc (kinh nghiệm chọn trâu, không hàm ý sâu xa)

    Là câu tục ngữ không mang nghĩa bóng (chỉ mang nghĩa đen/kỹ thuật). Ví dụ: Da đồng lông móc (kinh nghiệm chọn trâu, không hàm ý sâu xa)

  56. [68]

    Ví dụ: Nhất tự vi sư, bán tự vi sư

    Có từ Hán Việt xuất hiện. Ví dụ: Nhất tự vi sư, bán tự vi sư

  57. [69]

    Ví dụ: Am thanh cảnh vắng (từ “am” nghĩa là chùa nhỏ)

    Có từ ít phổ biến, ít dùng trong ngôn ngữ hiện đại. Ví dụ: Am thanh cảnh vắng (từ “am” nghĩa là chùa nhỏ)

  58. [70]

    Ví dụ: Cấy thưa thừa thóc, cấy mau dốc bồ

    Thể hiện kinh nghiệm dân gian xưa của người Việt Nam. Ví dụ: Cấy thưa thừa thóc, cấy mau dốc bồ

  59. [71]

    {phrase}

    Mang sắc thái mỉa mai, châm biếm hoặc tiêu cực . Ví dụ: Tốt mẽ khoe màu . Chỉ trả lời bằng các số đặc điểm, sắp xếp theo thứ tự tăng dần, cách nhau bằng dấu phẩy. Không giải thích, không thêm ký tự nào khác ngoài danh sách số. English (translation) You are a language expert. B...

  60. [72]

    Example: Da đồng lông móc (buffalo selec- tion criteria, no deeper implication)

    The proverb carries no figurative meaning (literal/technical meaning only). Example: Da đồng lông móc (buffalo selec- tion criteria, no deeper implication)

  61. [73]

    Example: Nhất tự vi sư, bán tự vi sư

    Contains Sino-Vietnamese vocabu- lary. Example: Nhất tự vi sư, bán tự vi sư

  62. [74]

    Example: Am thanh cảnh vắng (“am” means a small rural temple)

    Contains words rarely used in modern Vietnamese. Example: Am thanh cảnh vắng (“am” means a small rural temple)

  63. [75]

    Example: Cấy thưa thừa thóc, cấy mau dốc bồ

    Reflects traditional Vietnamese folk knowledge. Example: Cấy thưa thừa thóc, cấy mau dốc bồ

  64. [76]

    Example: Tốt mẽ khoe màu

    Carries a sarcastic, ironic, or negative tone. Example: Tốt mẽ khoe màu . Reply only with the characteristic numbers in ascending order, comma-separated. No explanation, no extra characters beyond the number list. Figure 10: Linguistic complexity classification prompt for the ...

  65. [2018]

    dân chỉ công bộc

    Glue: A multi-task benchmark and analy- sis platform for natural language understanding . Technical Report arXiv:1804.07461, arXiv. Minghan Wang, Viet Thanh Pham, Farhad Moghimifar, and Thuy-Trang Vu. 2025a. Proverbs run in pairs: Evaluating proverb trans- lation capability of...

    2025 arXiv

  66. [2019]

    Technical Report arXiv:1905.00537, arXiv

    Superglue: A stickier benchmark for general-purpose language understanding sys- tems. Technical Report arXiv:1905.00537, arXiv. Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman

    1905 arXiv

  67. [2023]

    In Proceedings of the 17th Con- ference of the European Chapter of the Associ- ation for Computational Linguistics , pages 652– 669, Dubrovnik, Croatia

    ViHOS: Hate speech spans detection for Vietnamese. In Proceedings of the 17th Con- ference of the European Chapter of the Associ- ation for Computational Linguistics , pages 652– 669, Dubrovnik, Croatia. Association for Compu- tational Linguistics. Tin Van Huynh, Kiet Van Nguy...

    2024

  68. [2024]

    and VLUE ( Tran et al. , 2024). VIMQA ( Le et al. , 2022b) evaluates multi-hop reasoning ca- pabilities. However, no dataset exists for assess- ing figurative and cultural language understanding in Vietnamese idioms and proverbs. VIVID ad- dresses this gap as the first benchma...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.