Pith. sign in

REVIEW 3 major objections 4 minor 152 references

Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

T0 review · 3 major / 4 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source smaller language models reach moderate agreement with human OPTION12 shared-decision scores and can support privacy-preserving coding with humans in the loop, while medical-domain models fail.

desk verdict Useful first pilot of local OS-sLLMs on OPTION12 Dutch melanoma data, but the headline correlations rest on only seven cleaned consultations and the test-set/Judge-LLM results are still missing. read the letter →

arxiv 2607.06127 v2 pith:F4KTKF57 submitted 2026-07-07 cs.CL

Tamara Wit , Lifeng Han , Carly Heipon , David Lindevelt , Anne Stiggelbout , Suzan Verberne This is my paper · ORCID
classification cs.CL
keywords shareddecisionmakingOPTION12open-sourceLLMsprivacy-preservingAIclinicalconsultationcodinghuman-in-the-loopmelanomatranscriptsdiscourseanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether open-source smaller language models, run locally, can score how well clinicians involve patients in treatment decisions using the twelve-item OPTION12 observer scale on Dutch melanoma consultation transcripts. It finds that general-domain models reach moderate correlation with expert human coders, while the tested medical-domain models hallucinate and fail to follow instructions. Gemma3:12b is strongest, with Pearson correlation 0.51 and Spearman 0.59 against human consensus. Because models still miss many items and struggle with timing of decision framing, speaker roles, and grounding evidence in the transcript, they cannot replace human annotators. They can, however, draft scores and justifications that humans check, and a Judge-LLM can reconcile disagreements among models the way human coders discuss until they agree. A sympathetic reader cares because manual OPTION12 coding is slow and often inconsistent, and commercial cloud models raise privacy and cost barriers for clinical data.

What carries the argument

The Judge-LLM consensus framework: several OS-sLLMs independently produce OPTION12 scores plus evidence and justification for each item; a selected best-performing model then acts as judge to reconcile those candidates into one consensus label, mirroring independent human coding followed by discussion to agreement.

What would settle it

Apply the same prompts, models, and Judge-LLM procedure to the held-out fifteen-consultation test set and check whether correlations with human consensus collapse toward zero, or whether medical-domain models suddenly outperform the general-domain ones that led on the development set.

Watch

Extended reading notes

Core claim

On expert-annotated Dutch melanoma consultations, general-domain open-source smaller language models achieve moderate agreement with human OPTION12 shared-decision scores, with Gemma3:12b strongest at Pearson r=0.51 and Spearman ρ=0.59, while medical-domain models fail through hallucination and instruction-following failures. Exact item matches remain low, and errors cluster around temporal discourse reasoning, role attribution, and evidence grounding. Current OS-sLLMs therefore cannot replace human annotators but form a usable foundation for privacy-preserving human-in-the-loop assessment, aided by a Judge-LLM that resolves multi-model disagreements into consensus scores.

Load-bearing premise

The reported correlations and model ranking rest on only seven successfully processed development consultations after four of eleven were dropped for formatting or suitability problems, so those results may not represent the full set of consultations.

Editorial extensions

If this is right

  • Locally run OS-sLLMs can draft OPTION12 scores and evidence so human coders spend time verifying rather than coding from scratch, while patient transcripts never leave institutional systems.
  • Item difficulty is systematic: temporal decision framing and clinician understanding-checks are hardest, so multi-model ensembles can cover complementary strengths across the twelve items.
  • Few-shot examples and later fine-tuning or parameter-efficient adaptation are expected to raise agreement on the remaining hard items toward usable human-in-the-loop levels.
  • Medical-domain smaller models need further work on instruction following and hallucination control before they can be trusted for discourse-level clinical coding tasks.
  • A Judge-LLM step can stand in for a third human when independent model scores disagree, reducing the cost of consensus meetings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistent superiority of general over medical models implies that OPTION12 coding is mainly a discourse and role-reasoning problem, not a medical-knowledge problem.
  • If the Judge-LLM procedure holds on the test set, the same multi-model-plus-judge pattern could transfer to other observer instruments used in clinical communication research.
  • Caregiver label noise and timing-format failures in preprocessing may have understated true model capability; cleaner speaker-normalized transcripts could lift exact-match rates without changing the models.
  • Privacy-preserving local scoring may enable multi-site SDM quality benchmarking that commercial cloud models cannot legally support under strict health-data rules.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces LLM4SDM, the first evaluation of open-source smaller LLMs (OS-sLLMs) for automated Observer OPTION12 coding of shared decision-making on Dutch melanoma consultation transcripts. It compares three general-domain models (Gemma3:12b, Llama3.1:8b, Mistral7b) and two medical-domain models (MedLlama2:7b, Meditron7b) on a development set of expert double-coded consultations, reports that general models outperform medical ones (which hallucinate and fail instruction following), finds Gemma3:12b strongest (Pearson r=0.51, Spearman ρ=0.59), provides item-level agreement tables and a qualitative error taxonomy (temporality, role attribution, evidence grounding), and proposes a Judge-LLM consensus framework to reconcile multi-model outputs for a planned human-in-the-loop pipeline.

Significance. If the pilot findings hold under larger evaluation, the work supplies a privacy-preserving, locally deployable alternative to commercial LLMs previously used only for the coarser OPTION5 instrument. The documented medical-model failure modes, the item-difficulty analysis, and the explicit error taxonomy are immediately useful to the clinical-NLP and SDM-measurement communities. The Judge-LLM design is a concrete, falsifiable proposal for multi-model consensus that mirrors human double-coding practice. These contributions are novel for OPTION12 and for open smaller models; the sustainability/privacy framing is a genuine advance over prior commercial-model studies.

major comments (3)
  1. [Section 4.3, Tables 1–4] Section 4.3 and Table 1: after excluding 4 of 11 development files for preprocessing failures or SDM unsuitability, all quantitative claims (Tables 2–4, including Gemma’s r=0.51 / ρ=0.59) rest on N=7 consultations (84 item-file pairs). Exact-match rates are already low (Gemma 20/84 ≈ 24 %). No confidence intervals, bootstrap, leave-one-out stability, or significance tests are supplied. With such a small cleaned sample the model ranking used to designate the Judge-sLLM and the claim of “moderate” alignment are statistically fragile and may not generalise to the held-out 15 or to deployment; this is load-bearing for the paper’s central recommendation.
  2. [Section 3, Figure 1, Section 6, Abstract] The methodology (Figure 1, Section 3) and conclusions (Section 6) describe a full testing-phase pipeline that deploys the selected Judge-sLLM on the 15 held-out consultations and evaluates consensus against human gold labels. No such results appear. Presenting the development-only pilot numbers in the abstract and as “findings” while leaving the planned evaluation unexecuted overstates the current evidence for the Judge-LLM framework and for the “promising foundation” claim.
  3. [Accepted Abstract vs. Tables 2 & 4] The “Accepted Abstract” block reports substantially higher correlations (0.83/0.80/0.64) and a different model ranking (Mistral best by consensus count) than the body (Tables 2 and 4: Gemma best, r≈0.51). These contradictory numbers cannot both be correct; the discrepancy must be resolved or the outdated abstract block removed, otherwise readers cannot trust the reported performance figures.
minor comments (4)
  1. [Section 4.1 / future-work paragraph] Human inter-rater reliability (before consensus) is never quantified, yet the paper repeatedly compares LLM–human agreement to an implicit human standard. Even a simple percentage agreement or weighted kappa on the original double codes would strengthen interpretation of the LLM numbers.
  2. [Appendix Table 6] Table 6 (speaker statistics) is presented without any statistical test or clear link to the OPTION12 scores; either analyse the relationship formally or move the table to an appendix.
  3. [Figures 2–4] Several figures (2–4) illustrate medical-model hallucinations well, but the captions and surrounding text could more explicitly state that these are representative failure modes rather than isolated runs.
  4. [Throughout] Minor typographical inconsistencies appear (e.g., “OPION12”, “Medtron” vs “Meditron”, “sLLM” vs “OS-sLLM”). A careful proof-read would remove them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical model-vs-human agreement metrics on independent expert consensus labels; no fitted parameters renamed as predictions and no load-bearing self-citation chain.

full rationale

The paper's central quantitative claims (exact-match consensus counts, per-item ratios, Pearson r and Spearman ρ between OS-sLLM scores and human OPTION12 scores) are computed by direct comparison of model outputs against independently double-coded, consensus-resolved human annotations on the development consultations (Tables 2–4, Section 4.3). Human labels are external to the models; nothing in the scoring equations or prompt design forces the reported r/ρ by construction. Prompt refinement and model selection occur on the development set (standard practice, disclosed), while the Judge-LLM consensus framework is only proposed and not yet executed on the held-out test set. Medical-model hallucination examples are qualitative observations, not circular. Self-citations are limited to the authors' own prior abstract and standard related-work references; none supply a uniqueness theorem or ansatz that the present results reduce to. The evaluation is therefore self-contained against external human benchmarks, with no self-definitional loop, fitted-input-as-prediction, or renaming of a known result.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The central empirical claims rest on standard LLM evaluation assumptions plus the validity of the human OPTION12 consensus as gold standard and the representativeness of the small cleaned development set. No free parameters are fitted to produce the reported correlations; the Judge-LLM is an invented procedural entity without independent evidence yet.

free parameters (2)
  • prompt and decoding configuration (temperature, context length, CoT wording)
    Chosen during pilot refinement on the development set; exact values not reported, yet they determine the scores used for model ranking and correlation claims.
  • development/test split and exclusion criteria
    11/15 split with 4 of 11 development files excluded post-hoc for preprocessing or SDM-unsuitability; the resulting N=7 drives all reported metrics.
assumptions (4)
  • domain assumption Human double-coding followed by consensus discussion yields a valid gold-standard OPTION12 score for each item.
    Used throughout Sections 3-4 as the reference for all agreement and correlation metrics; no IRR statistics are supplied.
  • domain assumption Exact score match (0-4) and Pearson/Spearman correlation across items are appropriate measures of LLM coding quality.
    Defined in Section 4.2 and used for all quantitative claims.
  • domain assumption General-domain vs medical-domain model comparison is fair when models are of roughly similar parameter count (~7-12B).
    Stated in Section 4.1; underpins the claim that medical models underperform.
  • ad hoc to paper A Judge-LLM can usefully mimic human consensus discussion by reconciling multiple model outputs.
    Introduced in Section 3 and Figure 1; not yet evaluated on the test set.
invented entities (2)
  • Judge-LLM consensus framework
    purpose: Resolve disagreements among multiple OS-sLLM annotations to produce a single consensus score and evidence set, analogous to human coder meetings.
    Proposed as a methodological contribution; no empirical results or external validation are provided in the current manuscript.
  • LLM4SDM framework / pipeline
    purpose: Name the overall development-then-deployment pipeline with prompt refinement, model selection, and planned Judge-LLM stage.
    Organizational label for the experimental design; not an independent scientific entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability." pith.science (2026). https://pith.science/paper/F4KTKF57

@misc{pith2026260706127,
  author       = {Pith},
  title        = {Pith review of: Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4KTKF57}},
  note         = {Machine review of arXiv:2607.06127}
}
read the original abstract

We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.

Figures

Figures reproduced from arXiv: 2607.06127 by the authors.

Figure 1
Figure 1. An overview of LLM4SDM framework: upper layer for development phase and lower layer for model testing. split this data set into development/validation and test sets (11, 15). To investigate both general and medi￾cal domain OS-LLMs, we used the follow￾ing comparable-sized lightweight models: 1) general domain pre-trained Gemma3:12b, Llama3.1:8b, and Mistral7b, and 2) medical domain finetuned models MedLlama2:7b and M… view at source ↗
Figure 2
Figure 2. Meditron Model Hallucination Example - repeated evidence and justification across pre￾dicted OPTIOIN12 items using the same score 3. {'item_number': 10, 'item_description': 'Hoeveel pijn voel je?', 'score': 0, 'evidence': 'verbal', 'justification': 'Asking the patient how much pain he ' 'feels is a standard question to ask in ' 'order to assess his general condition. ' 'A score of 1 indicates that the patient ' 'doe… view at source ↗
Figure 3
Figure 3. Meditron Model Hallucination Example - giving some score 0 for all OPTION12 items but using similar text to justify score 1 instead. {'item_number': 12, 'item_description': 'The patient has a history of ' 'exposure to environmental toxins', 'score': 1, 'evidence': 'Environmental toxin exposure diagnosis', 'justification': 'A history of exposure to environmental ' "toxins may increase the patient's risk " 'for develo… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: MedLlama Model Hallucination Example - giving score 1 for all OPTION12 items but mean score 10. and 12) using the same repeated evidence and justification across these items. However, in the 2nd run, after we fix its missing items in the output by more strict instructi…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

152 extracted references · 152 canonical work pages

  1. [1]

    Patient education and counseling , pages=

    Automating the Observer OPTION-5 measure of shared decision making: Assessing validity by comparing large language models to human ratings , author=. Patient education and counseling , pages=. 2025 , publisher=

    work page 2025

  2. [2]

    2005 , publisher=

    Shared decision making, measurement using the OPTION instrument , author=. 2005 , publisher=

    work page 2005

  3. [3]

    Patient Education and Counseling , volume=

    OPTION5 versus OPTION12 instruments to appreciate the extent to which healthcare providers involve patients in decision-making , author=. Patient Education and Counseling , volume=. 2016 , publisher=

    work page 2016

  4. [4]

    Academic Medicine , volume=

    Toward shared decision making: using the OPTION scale to analyze resident--patient consultations in family medicine , author=. Academic Medicine , volume=. 2011 , publisher=

    work page 2011

  5. [5]

    Circulation: Cardiovascular Quality and Outcomes , volume=

    Measuring shared decision making: a review of constructs, measures, and opportunities for cardiovascular care , author=. Circulation: Cardiovascular Quality and Outcomes , volume=. 2014 , publisher=

    work page 2014

  6. [6]

    Patient education and counseling , volume=

    Using a ‘talk’model of shared decision making to propose an observation-based measure: Observer OPTION5 Item , author=. Patient education and counseling , volume=. 2013 , publisher=

    work page 2013

  7. [7]

    Zeitschrift f

    The OPTION scale for the assessment of shared decision making (SDM): methodological issues , author=. Zeitschrift f. 2012 , publisher=

    work page 2012

  8. [8]

    Patient Education and Counseling , volume=

    The psychometric properties of Observer OPTION5, an observer measure of shared decision making , author=. Patient Education and Counseling , volume=. 2015 , publisher=

    work page 2015

Show all 152 references
  1. [9]

    Patient education and counseling , volume=

    Shared decision making: the reliability of the OPTION scale in Italy , author=. Patient education and counseling , volume=. 2007 , publisher=

    2007

  2. [10]

    Health Expectations , volume=

    Assessments of the extent to which health-care providers involve patients in decision making: a systematic review of studies using the OPTION instrument , author=. Health Expectations , volume=. 2015 , publisher=

    2015

  3. [11]

    Zeitschrift f

    Measurement of shared decision making--a review of instruments , author=. Zeitschrift f. 2011 , publisher=

    2011

  4. [12]

    Patient education and counseling , volume=

    Measuring shared decision making in the consultation: a comparison of the OPTION and Informed Decision Making instruments , author=. Patient education and counseling , volume=. 2008 , publisher=

    2008

  5. [13]

    Patient education and counseling , volume=

    Developing a dyadic OPTION scale to measure perceptions of shared decision making , author=. Patient education and counseling , volume=. 2010 , publisher=

    2010

  6. [14]

    Quality Management in Healthcare , volume=

    Shared decision-making measures: a systematic review , author=. Quality Management in Healthcare , volume=. 2020 , publisher=

    2020

  7. [15]

    Health Expectations , volume=

    Comparing the nine-item Shared Decision-Making Questionnaire to the OPTION Scale--an attempt to establish convergent validity , author=. Health Expectations , volume=. 2015 , publisher=

    2015

  8. [16]

    PloS one , volume=

    The quality of instruments to assess the process of shared decision making: a systematic review , author=. PloS one , volume=. 2018 , publisher=

    2018

  9. [17]

    Quality and Safety in health care , volume=

    Shared decision making: developing the OPTION scale for measuring patient involvement , author=. Quality and Safety in health care , volume=. 2003 , publisher=

    2003

  10. [18]

    Journal of clinical epidemiology , volume=

    Instruments to measure shared decision-making in outpatient chronic care: a systematic review and appraisal , author=. Journal of clinical epidemiology , volume=. 2020 , publisher=

    2020

  11. [19]

    Zeitschrift f

    Measuring (shared) decision-making--a review of psychometric instruments , author=. Zeitschrift f. 2007 , publisher=

    2007

  12. [20]

    BMJ evidence-based medicine , volume=

    Meet generative AI… your new shared decision-making assistant , author=. BMJ evidence-based medicine , volume=. 2024 , publisher=

    2024

  13. [21]

    Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages=

    Reliable and safe use of machine translation in medical settings , author=. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages=

    2022

  14. [22]

    arXiv preprint arXiv:2511.06427 , year=

    Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loop , author=. arXiv preprint arXiv:2511.06427 , year=

  15. [23]

    Patient education and counseling , volume=

    Shared decision making: concepts, evidence, and practice , author=. Patient education and counseling , volume=. 2015 , publisher=

    2015

  16. [24]

    American journal of preventive medicine , volume=

    Shared decision making in clinical medicine: past research and future directions , author=. American journal of preventive medicine , volume=. 1999 , publisher=

    1999

  17. [25]

    Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains

    Ramesh, Krithika and Gandhi, Nupoor and Madaan, Pulkit and Bauer, Lisa and Peris, Charith and Field, Anjalie. Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10...

    2024 doi

  18. [26]

    Scientific Data , volume=

    A dataset for plain language adaptation of biomedical abstracts , author=. Scientific Data , volume=. 2023 , publisher=

    2023

  19. [27]

    Evolutionary Intelligence , volume=

    Artificial intelligence trend analysis on healthcare podcasts using topic modeling and sentiment analysis: a data-driven approach , author=. Evolutionary Intelligence , volume=. 2024 , publisher=

    2024

  20. [28]

    JMIR cancer , volume=

    Ethical design of data-driven decision support tools for improving cancer care: embedded ethics review of the 4D PICTURE project , author=. JMIR cancer , volume=. 2025 , publisher=

    2025

  21. [29]

    International Journal of Medical Informatics , pages=

    Natural language processing in medical text processing: A scoping literature review , author=. International Journal of Medical Informatics , pages=. 2025 , publisher=

    2025

  22. [30]

    Informatics , volume=

    Large language models in healthcare and medical domain: A review , author=. Informatics , volume=. 2024 , organization=

    2024

  23. [31]

    INQUIRY: The Journal of Health Care Organization, Provision, and Financing , volume=

    The growing impact of natural language processing in healthcare and public health , author=. INQUIRY: The Journal of Health Care Organization, Provision, and Financing , volume=. 2024 , publisher=

    2024

  24. [32]

    ArXiv , pages=

    Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track , author=. ArXiv , pages=

  25. [33]

    Proceedings of the 24th Workshop on Biomedical Language Processing , pages=

    Overview of the biolaysumm 2025 shared task on lay summarization of biomedical research articles and radiology reports , author=. Proceedings of the 24th Workshop on Biomedical Language Processing , pages=

    2025

  26. [34]

    Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health). 2025. doi:10.18653/v1/2025.cl4health-1.0

    2025 doi

  27. [35]

    2022 , school=

    An investigation into multi-word expressions in machine translation , author=. 2022 , school=

    2022

  28. [36]

    Cancer Medicine , volume=

    The bigger picture of shared decision making: a service design perspective using the care path of locally advanced pancreatic cancer as a case , author=. Cancer Medicine , volume=. 2021 , publisher=

    2021

  29. [37]

    The Eleventh International Conference on Learning Representations , year=

    Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought , author=. The Eleventh International Conference on Learning Representations , year=

  30. [38]

    A Survey on In-context Learning

    Dong, Qingxiu and Li, Lei and Dai, Damai and Zheng, Ce and Ma, Jingyuan and Li, Rui and Xia, Heming and Xu, Jingjing and Wu, Zhiyong and Chang, Baobao and Sun, Xu and Li, Lei and Sui, Zhifang. A Survey on In-context Learning. Proceedings of the 2024 Conference on Empirical Met...

    2024 doi

  31. [39]

    Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023). 2023

    2023

  32. [40]

    2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) , pages=

    Topic modelling of swedish newspaper articles about coronavirus: a case study using latent dirichlet allocation method , author=. 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) , pages=. 2023 , organization=

    2023

  33. [41]

    2023 IEEE International Conference on Big Data (BigData) , pages=

    Extraction of medication and temporal relation from clinical text using neural language models , author=. 2023 IEEE International Conference on Big Data (BigData) , pages=. 2023 , organization=

    2023

  34. [42]

    2024 IEEE 12th International Conference on Healthcare Informatics (ICHI) , pages=

    Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts , author=. 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI) , pages=. 2024 , organization=

    2024

  35. [43]

    2025 , eprint=

    MaLei at the PLABA Track of TREC 2024: RoBERTa for Term Replacement -- LLaMA3.1 and GPT-4o for Complete Abstract Adaptation , author=. 2025 , eprint=

    2024

  36. [44]

    Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    Medication Extraction and Entity Linking using Stacked and Voted Ensembles on LLMs , author=. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

  37. [45]

    Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    The Manchester Bees at PerAnsSumm 2025: Iterative Self-Prompting with Claude and o1 for Perspective-aware Healthcare Answer Summarisation , author=. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    2025

  38. [46]

    Frontiers in Digital Health , volume=

    Synthetic4Health: generating annotated synthetic clinical letters , author=. Frontiers in Digital Health , volume=. 2025 , publisher=

    2025

  39. [47]

    2025 IEEE 13th International Conference on Healthcare Informatics (ICHI) , pages=

    MLM4SynMed: Masked Language Modelling for Synthetic Free-text Medical Records Generation , author=. 2025 IEEE 13th International Conference on Healthcare Informatics (ICHI) , pages=. 2025 , organization=

    2025

  40. [48]

    2025 IEEE 13th International Conference on Healthcare Informatics (ICHI) , pages=

    De-identifying Clinical Texts using Biomed-Clinical BERTs and Comprehensive Risk Assessment , author=. 2025 IEEE 13th International Conference on Healthcare Informatics (ICHI) , pages=. 2025 , organization=

    2025

  41. [49]

    International Conference on Biomedical Ontology , year=

    Towards Principles of Ontology-Based Annotation of Clinical Narratives , author=. International Conference on Biomedical Ontology , year=

  42. [50]

    Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

    Han, Lifeng and Erofeev, Gleb and Sorokina, Irina and Gladkoff, Serge and Nenadic, Goran. Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning. Proceedings of the 5th Clinical Natural Language Processing Workshop. ...

    2023 doi

  43. [51]

    Frontiers in Digital Health , volume=

    Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning , author=. Frontiers in Digital Health , volume=. 2024 , publisher=

    2024

  44. [52]

    M ed T em2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries

    Cui, Yang and Han, Lifeng and Nenadic, Goran. M ed T em2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). 2023. d...

    2023 doi

  45. [53]

    2024 , eprint=

    Exploration of Masked and Causal Language Modelling for Text Generation , author=. 2024 , eprint=

    2024

  46. [54]

    Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation , journal =

    Hang Dong and Víctor Suárez-Paniagua and William Whiteley and Honghan Wu , keywords =. Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation , journal =. 2021 , issn =. doi:https://doi.org/10.1016/j.j...

    2021 doi

  47. [55]

    Advances in neural information processing systems , volume=

    Xlnet: Generalized autoregressive pretraining for language understanding , author=. Advances in neural information processing systems , volume=

  48. [56]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    MHLAT: Multi-Hop Label-Wise Attention Model for Automatic ICD Coding , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  49. [57]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    arXiv

  50. [58]

    Journal of biomedical informatics , volume=

    Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation , author=. Journal of biomedical informatics , volume=. 2021 , publisher=

    2021

  51. [59]

    Journal of the American Medical Informatics Association , volume=

    Diagnosis code assignment: models and evaluation metrics , author=. Journal of the American Medical Informatics Association , volume=. 2014 , publisher=

    2014

  52. [60]

    2016 , institution=

    Applying deep learning to ICD-9 multi-label classification from medical records , author=. 2016 , institution=

    2016

  53. [61]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hum...

    2019 doi

  54. [62]

    arXiv preprint arXiv:1906.05474 , year=

    Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets , author=. arXiv preprint arXiv:1906.05474 , year=

    1906 arXiv

  55. [63]

    Advances in neural information processing systems , volume=

    Distributed representations of words and phrases and their compositionality , author=. Advances in neural information processing systems , volume=

  56. [64]

    Annual review of biomedical data science , volume=

    Modern clinical text mining: a guide and review , author=. Annual review of biomedical data science , volume=. 2021 , publisher=

    2021

  57. [65]

    BMC bioinformatics , volume=

    Automatic construction of rule-based ICD-9-CM coding systems , author=. BMC bioinformatics , volume=. 2008 , organization=

    2008

  58. [66]

    Journal of the American Medical Informatics Association , volume=

    A systematic literature review of automated clinical coding and classification systems , author=. Journal of the American Medical Informatics Association , volume=. 2010 , publisher=

    2010

  59. [67]

    , author=

    Automatic disease coding: the'fruit-machine'method in general practice. , author=. British journal of preventive & social medicine , volume=. 1973 , publisher=

    1973

  60. [68]

    Deloitte, MedTech Europe , year=

    The socio-economic impact of AI in healthcare , author=. Deloitte, MedTech Europe , year=

  61. [69]

    Journal of public health , volume=

    Systematic review of discharge coding accuracy , author=. Journal of public health , volume=. 2012 , publisher=

    2012

  62. [70]

    NPJ digital medicine , volume=

    Automated clinical coding: what, why, and where we are? , author=. NPJ digital medicine , volume=. 2022 , publisher=

    2022

  63. [71]

    Deep Long-Tailed Learning: A Survey , year=

    Zhang, Yifan and Kang, Bingyi and Hooi, Bryan and Yan, Shuicheng and Feng, Jiashi , journal=. Deep Long-Tailed Learning: A Survey , year=

  64. [72]

    Scientific data , volume=

    MIMIC-III, a freely accessible critical care database , author=. Scientific data , volume=. 2016 , publisher=

    2016

  65. [73]

    Hierarchical label-wise attention transformer model for explainable ICD coding , journal =

    Leibo Liu and Oscar Perez-Concha and Anthony Nguyen and Vicki Bennett and Louisa Jorm , keywords =. Hierarchical label-wise attention transformer model for explainable ICD coding , journal =. 2022 , issn =. doi:https://doi.org/10.1016/j.jbi.2022.104161 , url =

    2022 doi

  66. [74]

    Explainable Prediction of Medical Codes from Clinical Text

    Mullenbach, James and Wiegreffe, Sarah and Duke, Jon and Sun, Jimeng and Eisenstein, Jacob. Explainable Prediction of Medical Codes from Clinical Text. Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2018 doi

  67. [75]

    Hierarchical Attention Networks for Document Classification

    Yang, Zichao and Yang, Diyi and Dyer, Chris and He, Xiaodong and Smola, Alex and Hovy, Eduard. Hierarchical Attention Networks for Document Classification. Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...

    2016 doi

  68. [76]

    Measuring Uncertainty in Translation Quality Evaluation ( TQE )

    Gladkoff, Serge and Sorokina, Irina and Han, Lifeng and Alekseeva, Alexandra. Measuring Uncertainty in Translation Quality Evaluation ( TQE ). Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022

    2022

  69. [77]

    Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models , year=

    Tu, Hangyu and Han, Lifeng and Nenadic, Goran , booktitle=. Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models , year=

  70. [78]

    Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition , year=

    Belkadi, Samuel and Han, Lifeng and Wu, Yuping and Nenadic, Goran , booktitle=. Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition , year=

  71. [79]

    2023 , booktitle=

    Generating Medical Instructions with Conditional Transformer , author=. 2023 , booktitle=

    2023

  72. [80]

    2023 , url=

    Building healthcare software - clinical coding, classifications and terminology , author=. 2023 , url=

    2023

  73. [81]

    2023 , eprint=

    Large Language Models and Control Mechanisms Improve Text Readability of Biomedical Abstracts , author=. 2023 , eprint=

    2023

  74. [82]

    arXiv preprint arXiv:1904.05342 , year=

    Clinicalbert: Modeling clinical notes and predicting hospital readmission , author=. arXiv preprint arXiv:1904.05342 , year=

    1904 arXiv

  75. [83]

    Applied Sciences , volume=

    Research on Named Entity Recognition Methods in Chinese Forest Disease Texts , author=. Applied Sciences , volume=. 2022 , publisher=

    2022

  76. [84]

    Constructing Bi-Order-Transformer-CRF With Neural Cosine Similarity Function for Power Metering Entity Recognition , year=

    Zheng, Kaihong and Yang, Jingfeng and Zeng, Lukun and Gong, Qihang and Li, Sheng and Zhou, Shangli , journal=. Constructing Bi-Order-Transformer-CRF With Neural Cosine Similarity Function for Power Metering Entity Recognition , year=

  77. [85]

    Chinese Named Entity Recognition based on BERT-Transformer-BiLSTM-CRF Model , year=

    Gan, Yong and Yang, Ruisen and Zhang, Chenfang and Jia, Dongwei , booktitle=. Chinese Named Entity Recognition based on BERT-Transformer-BiLSTM-CRF Model , year=

  78. [86]

    Using Bidirectional Transformer-CRF for Spoken Language Understanding

    Zhang, Linhao and Wang, Houfeng. Using Bidirectional Transformer-CRF for Spoken Language Understanding. Natural Language Processing and Chinese Computing. 2019

    2019

  79. [87]

    Proceedings of The Fourth International Conference on Natural Language and Speech Processing (ICNLSP 2021) , pages=

    Arabic Named Entity Recognition Using Transformer-based-CRF Model , author=. Proceedings of The Fourth International Conference on Natural Language and Speech Processing (ICNLSP 2021) , pages=

    2021

  80. [88]

    CoRR , volume =

    Hang Yan and Bocao Deng and Xiaonan Li and Xipeng Qiu , title =. CoRR , volume =. 2019 , url =. 1911.04474 , timestamp =

    2019 arXiv

  81. [89]

    A transformer-based deep learning model for recognizing communication-oriented entities from patents of ICT in construction , journal =

    Hengqin Wu and Geoffrey Qiping Shen and Xue Lin and Minglei Li and Clyde Zhengdao Li , keywords =. A transformer-based deep learning model for recognizing communication-oriented entities from patents of ICT in construction , journal =. 2021 , issn =. doi:https://doi.org/10.101...

    2021 doi

  82. [90]

    and Chao, Lidia S

    Han, Aaron Li-Feng and Zeng, Xiaodong and Wong, Derek F. and Chao, Lidia S. C hinese Named Entity Recognition with Graph-based Semi-supervised Learning Model. Proceedings of the Eighth SIGHAN Workshop on C hinese Language Processing. 2015. doi:10.18653/v1/W15-3103

    2015 doi

  83. [91]

    Han, Aaron L. -F. and Wong, Derek F. and Chao, Lidia S. Chinese Named Entity Recognition with Conditional Random Fields in the Light of Chinese Characteristics. Language Processing and Intelligent Information Systems. 2013

    2013

  84. [92]

    Detection of Verbal Multi-Word Expressions via Conditional Random Fields with Syntactic Dependency Features and Semantic Re-Ranking

    Maldonado, Alfredo and Han, Lifeng and Moreau, Erwan and Alsulaimani, Ashjan and Chowdhury, Koel Dutta and Vogel, Carl and Liu, Qun. Detection of Verbal Multi-Word Expressions via Conditional Random Fields with Syntactic Dependency Features and Semantic Re-Ranking. Proceedings...

    2017 doi

  85. [93]

    2018 , publisher =

    Moreau, Erwan and Alsulaimani, Ashjan and Maldonado, Alfredo and Han, Lifeng and Vogel, Carl and Dutta Chowdhury, Koel , URL =. 2018 , publisher =. doi:10.5281/zenodo.1469559 , PDF =

    2018 doi

  86. [94]

    EDU-level Extractive Summarization with Varying Summary Lengths , publisher =

    Wu, Yuping and Tseng, Ching-Hsun and Shang, Jiayu and Mao, Shengzhong and Nenadic, Goran and Zeng, Xiao-Jun , keywords =. EDU-level Extractive Summarization with Varying Summary Lengths , publisher =. 2022 , copyright =. doi:10.48550/ARXIV.2210.04029 , url =

    2022 doi arXiv

  87. [95]

    Journal of the American Medical Informatics Association , volume=

    2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records , author=. Journal of the American Medical Informatics Association , volume=. 2020 , publisher=

    2018

  88. [96]

    Diagnosis Certainty and Progression: A Natural Language Processing Approach to Enable Characterisation of the Evolution of Diagnoses in Clinical Notes

    Alfredo Madrid and Caitlin Bullin and Lifeng Han and Judith Andrew and Warren Del-Pinto and Ghada Alfattni and Oswaldo Solarte and Ernestina M Ruiz and Luis Rodríguez and Ana María Aldana Gelves and Robyn Hamilton and Karim Webb and Meghna Jani and William G Dixonmand and Gora...

    2022

  89. [97]

    A Transformer-based Machine Learning Framework using Conditional Random Fields as Decoder for Clinical Text Mining

    Lifeng Han and Valerio Antonini and Ghada Alfattni and Alfredo Madrid and Warren Del-Pinto and Judith Andrew and Ana María Aldana Gelves and Robyn Hamilton and Karim Webb and Meghna Jani and William G Dixonmand and Goran Nenadic. A Transformer-based Machine Learning Framework ...

    2022

  90. [98]

    Publicly Available Clinical BERT Embeddings

    Alsentzer, Emily and Murphy, John and Boag, William and Weng, Wei-Hung and Jindi, Di and Naumann, Tristan and McDermott, Matthew. Publicly Available Clinical BERT Embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop. 2019. doi:10.18653/v1/W19-1909

    2019 doi

  91. [99]

    and McCallum, Andrew and Pereira, Fernando C

    Lafferty, John D. and McCallum, Andrew and Pereira, Fernando C. N. , title =. Proceedings of the Eighteenth International Conference on Machine Learning , pages =. 2001 , isbn =

    2001

  92. [100]

    and De Meulder, Fien

    Tjong Kim Sang, Erik F. and De Meulder, Fien. Introduction to the C o NLL -2003 Shared Task: Language-Independent Named Entity Recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003. 2003

    2003

  93. [101]

    and Surdeanu, Mihai and Bauer, John and Finkel, Jenny and Bethard, Steven J

    Manning, Christopher D. and Surdeanu, Mihai and Bauer, John and Finkel, Jenny and Bethard, Steven J. and McClosky, David , title =. Association for Computational Linguistics (ACL) System Demonstrations , year =

  94. [102]

    Dernoncourt, Franck and Lee, Ji Young and Szolovits, Peter , journal=

  95. [103]

    SM ed BERT : A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining

    Zhang, Taolin and Cai, Zerui and Wang, Chengyu and Qiu, Minghui and Yang, Bite and He, Xiaofeng. SM ed BERT : A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining. Proceedings of the 59th Annual Meeting of the Association for Comput...

    2021 doi

  96. [104]

    Investigating Prior Knowledge for Challenging C hinese Machine Reading Comprehension

    Sun, Kai and Yu, Dian and Yu, Dong and Cardie, Claire. Investigating Prior Knowledge for Challenging C hinese Machine Reading Comprehension. Transactions of the Association for Computational Linguistics. 2020. doi:10.1162/tacl_a_00305

    2020 doi

  97. [105]

    Extractive Financial Narrative Summarisation using S entence BERT Based Clustering

    Gokhan, Tuba and Smith, Phillip and Lee, Mark. Extractive Financial Narrative Summarisation using S entence BERT Based Clustering. Proceedings of the 3rd Financial Narrative Processing Workshop. 2021

    2021

  98. [106]

    Neural Machine Translation of Rare Words with Subword Units

    Sennrich, Rico and Haddow, Barry and Birch, Alexandra. Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2016. doi:10.18653/v1/P16-1162

    2016 doi

  99. [107]

    HuggingFace's Transformers: State-of-the-art Natural Language Processing , journal =

    Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and R. HuggingFace's Transformers: State-of-the-art Natural Language Processing , journal =. 2019 , url =. 1910.03771 , timestamp =

    2019 arXiv

  100. [108]

    Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books , year=

    Zhu, Yukun and Kiros, Ryan and Zemel, Rich and Salakhutdinov, Ruslan and Urtasun, Raquel and Torralba, Antonio and Fidler, Sanja , booktitle=. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books , year=

  101. [109]

    Gomez and Lukasz Kaiser and Illia Polosukhin , title=

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , title=. 2017 , cdate=

    2017

  102. [110]

    ACM Trans

    Gu, Yu and Tinn, Robert and Cheng, Hao and Lucas, Michael and Usuyama, Naoto and Liu, Xiaodong and Naumann, Tristan and Gao, Jianfeng and Poon, Hoifung , title =. ACM Trans. Comput. Healthcare , month =. 2021 , issue_date =. doi:10.1145/3458754 , abstract =

    2021 doi

  103. [111]

    Bioinformatics , volume =

    Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo , title = ". Bioinformatics , volume =. 2019 , month =. doi:10.1093/bioinformatics/btz682 , url =

    2019 doi

  104. [112]

    B io M ed BERT : A Pre-trained Biomedical Language Model for QA and IR

    Chakraborty, Souradip and Bisong, Ekaba and Bhatt, Shweta and Wagner, Thomas and Elliott, Riley and Mosconi, Francesco. B io M ed BERT : A Pre-trained Biomedical Language Model for QA and IR. Proceedings of the 28th International Conference on Computational Linguistics. 2020. ...

    2020 doi

  105. [113]

    Marian: Fast Neural Machine Translation in

    Junczys-Dowmunt, Marcin and Grundkiewicz, Roman and Dwojak, Tomasz and Hoang, Hieu and Heafield, Kenneth and Neckermann, Tom and Seide, Frank and Germann, Ulrich and Fikri Aji, Alham and Bogoychev, Nikolay and Martins, Andr\'. Marian: Fast Neural Machine Translation in. Procee...

    2018

  106. [114]

    Facebook AI’s WMT21 News Translation Task Submission , author=. Proc. of WMT , year=

  107. [115]

    Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

    Marie, Benjamin and Fujita, Atsushi and Rubino, Raphael. Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference...

    2021 doi

  108. [116]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  109. [117]

    Publications Manual , year = "1983", publisher =

    1983

  110. [118]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981 doi

  111. [119]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  112. [120]

    Dan Gusfield , title =. 1997

    1997

  113. [121]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  114. [122]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  115. [123]

    Why We Need New Evaluation Metrics for NLG

    Novikova, Jekaterina and Du s ek, Ond r ej and Cercas Curry, Amanda and Rieser, Verena. Why We Need New Evaluation Metrics for NLG. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1/D17-1238

    2017 doi

  116. [124]

    Monte Carlo modelling of confidence intervals in translation quality evaluation (TQE) and post-editing dstance (PED) measurement

    Alekseeva, Alexandra and Gladkoff, Serge and Sorokina, Irina and Han, Lifeng. Monte Carlo modelling of confidence intervals in translation quality evaluation (TQE) and post-editing dstance (PED) measurement. Metrics 2021: Workshop on Informetric and Scientometric Research (SIG...

    2021

  117. [125]

    C hinese Character Decomposition for Neural MT with Multi-Word Expressions

    Han, Lifeng and Jones, Gareth and Smeaton, Alan and Bolzoni, Paolo. C hinese Character Decomposition for Neural MT with Multi-Word Expressions. Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). 2021

    2021

  118. [126]

    arXiv e-prints , keywords =

    Meta-evaluation of Conversational Search Evaluation Metrics. arXiv e-prints , keywords =

  119. [127]

    Re-evaluating Evaluation in Text Summarization

    Bhandari, Manik and Gour, Pranav Narayan and Ashfaq, Atabak and Liu, Pengfei and Neubig, Graham. Re-evaluating Evaluation in Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.751

    2020 doi

  120. [128]

    arXiv e-prints , keywords =

    XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. arXiv e-prints , keywords =

  121. [129]

    Results of the WMT 13 Metrics Shared Task

    Mach \'a c ek, Matou s and Bojar, Ond r ej. Results of the WMT 13 Metrics Shared Task. Proceedings of the Eighth Workshop on Statistical Machine Translation. 2013

    2013

  122. [130]

    arXiv e-prints , keywords =

    Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. arXiv e-prints , keywords =

  123. [131]

    Algorithms for Hyper-Parameter Optimization , year =

    Bergstra, James and Bardenet, R\'. Algorithms for Hyper-Parameter Optimization , year =. Proceedings of the 24th International Conference on Neural Information Processing Systems , pages =

  124. [132]

    Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics , volume =

    Lommel, Arle and Burchardt, Aljoscha and Uszkoreit, Hans , year =. Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics , volume =. Tradumàtica: tecnologies de la traducció , doi =

  125. [133]

    and Chao, Lidia S

    Han, Aaron Li-Feng and Wong, Derek F. and Chao, Lidia S. and Lu, Yi and He, Liangye and Wang, Yiming and Zhou, Jiaji. A Description of Tunable Machine Translation Evaluation Systems in WMT 13 Metrics Task. Proceedings of the Eighth Workshop on Statistical Machine Translation. 2013

    2013

  126. [134]

    cush LEPOR uses LABSE distilled knowledge to improve correlation with human translations

    Erofeev, Gleb and Sorokina, Irina and Han, Lifeng and Gladkoff, Serge. cush LEPOR uses LABSE distilled knowledge to improve correlation with human translations. Proceedings of Machine Translation Summit XVIII: Users and Providers Track. 2021

    2021

  127. [135]

    Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

    Han, Lifeng and Smeaton, Alan and Jones, Gareth. Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods. Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age. 2021

    2021

  128. [136]

    CoRR , volume =

    Takuya Akiba and Shotaro Sano and Toshihiko Yanase and Takeru Ohta and Masanori Koyama , title =. CoRR , volume =. 2019 , url =

    2019

  129. [137]

    2009 , publisher =

    Steven Bird and Ewan Klein and Edward Loper , title =. 2009 , publisher =

    2009

  130. [138]

    arXiv e-prints , keywords =

    The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. arXiv e-prints , keywords =

  131. [139]

    Machine Translation , title =

    Marzouk, Shaimaa , da =. Machine Translation , title =. 2021 , Bdsk-Url-1 =. doi:10.1007/s10590-021-09266-0 , id =

    2021 doi

  132. [140]

    Recurrent Continuous Translation Models

    Kalchbrenner, Nal and Blunsom, Phil. Recurrent Continuous Translation Models. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013

    2013

  133. [141]

    Proceedings of the Six Conference on Machine Translation

    Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain. Proceedings of the Six Conference on Machine Translation. 2021

    2021

  134. [142]

    Findings of the 2019 Conference on Machine Translation ( WMT 19)

    Barrault, Lo. Findings of the 2019 Conference on Machine Translation ( WMT 19). Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). 2019. doi:10.18653/v1/W19-5301

    2019 doi

  135. [143]

    Findings of the 2020 Conference on Machine Translation ( WMT 20)

    Barrault, Lo. Findings of the 2020 Conference on Machine Translation ( WMT 20). Proceedings of the Fifth Conference on Machine Translation. 2020

    2020

  136. [144]

    CoRR , volume =

    KyungHyun Cho and Bart van Merrienboer and Dzmitry Bahdanau and Yoshua Bengio , title =. CoRR , volume =. 2014 , url =

    2014

  137. [145]

    CoRR , volume =

    Dzmitry Bahdanau and Kyunghyun Cho and Yoshua Bengio , title =. CoRR , volume =. 2014 , url =

    2014

  138. [146]

    2015 , url =

    Yvette Graham and Timothy Baldwin and Nitika Mathur , title =. 2015 , url =

    2015

  139. [147]

    2014 , school=

    LEPOR: An Augmented Machine Translation Evaluation Metric , author=. 2014 , school=

    2014

  140. [148]

    2014 , url=

    LEPOR: An Augmented Machine Translation Evaluation Metric , author=. 2014 , url=

    2014

  141. [149]

    Han, Aaron L. F. and Wong, Derek F. and Chao, Lidia S. LEPOR : A Robust Evaluation Metric for Machine Translation with Augmented Factors. Proceedings of COLING 2012: Posters. 2012

    2012

  142. [150]

    Machine Translation Summit XIV , pages=

    Language-independent Model for Machine Translation Evaluation with Reinforced Factors , author=. Machine Translation Summit XIV , pages=. 2013 , url =

    2013

  143. [151]

    CoRR , volume =

    Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang , title =. CoRR , volume =. 2020 , url =

    2020

  144. [152]

    Empirical evaluation methodology for target dependent sentiment analysis , author =

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.