HAMLET: Healthcare-focused Adaptive Multilingual Learning Embedding-based Topic Modeling

Hajar Sakai; Sarah S. Lam

REVIEW 4 major objections 5 minor 2 references

HAMLET: Healthcare-focused Adaptive Multilingual Learning Embedding-based Topic Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HAMLET uses GPT-4o-generated topics, then refines their embeddings with a graph neural network to deliver interpretable, cross-lingual healthcare topics that can serve as classification labels.

desk verdict HAMLET is a plausible LLM+GNN pipeline, but its main ablation gain is an artifact of scoring topic embeddings against the same graph neighbors the GNN averages in, so the central effectiveness claim is unsupported. read the letter →

arxiv 2505.07157 v1 pith:7LMEZR5X submitted 2025-05-12 cs.CL

Hajar Sakai , Sarah S. Lam This is my paper · ORCID

classification cs.CL

keywords topicmodelinglargelanguagemodelsgraphneuralnetworksBERTmultilingualhealthcareembeddingrefinementcoherence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to solve a real bottleneck: when healthcare text lacks labels, you need topics before you can classify, and traditional models like LDA or NMF produce overlapping, hard-to-interpret word bags. HAMLET proposes a pipeline where GPT-4o generates an initial pool of short topic phrases, a hybrid BERT–SBERT embedding scheme represents documents and topics, a graph neural network (EdgeRefineGNN) refines the topic embeddings by passing messages across document-topic-word edges, and a clustering plus coherence-based selection step extracts the top k topics. The authors claim this yields coherent, diverse, interpretable topics across English and French healthcare datasets, with refinement consistently improving a Composite Score over using the raw LLM embeddings. If correct, the approach would let practitioners skip manual topic naming and go straight to automated text classification in label-scarce healthcare settings.

What carries the argument

The load-bearing mechanism is the EdgeRefineGNN, an edge-conditioned graph neural network that updates each topic node's embedding by mean-pooling messages from its neighboring word and topic nodes, where the message for each edge is produced by an MLP that takes the edge's type and similarity score as input. It is trained with an MSE loss against the original topic embeddings, so the refinement is a denoising step constrained to stay near the LLM's semantics while absorbing structural information from the graph. The graph itself is built from topic-assignment edges, similar-topic edges, similar-word edges, and word-in-topic edges, where similarities come from the proposed SGS method; this graph supplies the structural context the GNN uses to adjust embeddings.

What would settle it

Train EdgeRefineGNN with an ablation that removes the word-in-topic edges from the graph (keeping only topic-topic, document-topic, and word-word edges), then compare the Composite Score gain over the original embeddings. If the gain mostly disappears when those edges are removed, the apparent coherence improvement is a direct consequence of the GNN's input graph, not of semantic refinement.

Watch

Extended reading notes

Core claim

The paper's central claim is that topic embeddings produced by an LLM are raw and under-representative, and that refining them through a graph neural network that respects heterogeneous relationships—documents, topics, words, similar topics, and similar words—produces topics that are more coherent and better separated than the original embeddings. Evidence is offered through ablation: on the English datasets the refined embeddings with coherence-based extraction achieve an overall Composite Score of 0.740 ± 0.023 versus 0.663 ± 0.008 for the original embeddings, and the French overall scores are 0.743 ± 0.018 versus 0.633 ± 0.029. The paper also introduces a Semantic-Geometric Similarity (SGS) method that blends WMD-inspired word-level alignment with cosine similarity under IDF weighting, and a Composite Score metric that weights topic diversity, pairwise Jaccard similarity, coherence, silhouette, and Davies-Bouldin in order to evaluate phrase-form topics.

Load-bearing premise

The paper's evidence that refinement improves quality rests on a Composite Score whose coherence component measures the average cosine similarity between a topic embedding and its associated word embeddings—and those exact 'word in topic' edges are already fed into the GNN during refinement, so the metric is not independent of the refinement mechanism it is used to validate.

Editorial extensions

If this is right

If the refinement claim holds, topic modeling pipelines that use LLM-generated phrases no longer need a human to interpret or name topics, since the extracted top-k phrases can be used directly as classification labels.
The approach is claimed to be language-agnostic in the tested sense, since the same pipeline with language-specific prompts worked on English patient feedback and French COVID news, producing stable Composite Scores across both.
The EdgeRefineGNN component, if genuinely beneficial, suggests that coupling an LLM's semantic priors with graph-structured message passing is a viable general recipe for unsupervised topic refinement, beyond the healthcare domain.
Since the ablation shows refined embeddings yield higher Silhouette and Davies-Bouldin scores, the refined embeddings may also serve better as features for downstream clustering or classification, not just as topic labels.
The SGS similarity method could be reused as a drop-in similarity measure for other phrase-level comparison tasks, since it combines word-level alignment, cosine similarity, and IDF weighting into a normalized score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A cautious reading: the coherence component of the Composite Score, $\text{Coherence}_t = \text{avg}(\cos(emb_t, emb_{words}))$, is computed after the GNN has already used word-in-topic edges in its message passing, so part of the observed coherence gain may be an artifact of the optimization target rather than independent semantic improvement. This is my inference as a reviewer, not a claim the p
A testable extension the authors do not propose: replace the coherence-based extraction with a holdout set of manually annotated topic labels (e.g., what a clinician would name the themes) and measure how often HAMLET's top-k phrases match those labels. That would directly validate the 'interpretable and usable as labels' claim.
If the SGS threshold (90th percentile of similarities) is truly adaptive, then the method should work across corpora with very different lexical densities; one could stress-test it by sampling a corpus with many near-duplicate articles (e.g., news wires) and checking whether the refined topics still collapse to a few redundant phrases.
The MSE loss that keeps refined embeddings close to the LLM's originals means the GNN cannot fully re-orient a topic that starts semantically wrong; a more aggressive refinement with an adversarial or contrastive loss might yield larger gains but also higher risk of drift. The authors do not explore this trade-off.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a plausible pipeline but the main piece of evidence is circular. The stress-test note is right. The coherence component of the Composite Score is computed as the average cosine similarity between a topic embedding and the embeddings of its associated words. EdgeRefineGNN updates a topic embedding by mean-pooling messages from exactly those neighboring word nodes (plus other nodes). So after refinement, the topic embedding is literally closer to those words by construction. The 0.740 versus 0.663 ablation gain in Coherence is not independent evidence of semantic improvement; it's the model pulling the embeddings toward the thing it is then scored against.

What's actually new: the specific combination of LLM-generated short phrases, SBERT-BERT hybrid embeddings, the SGS similarity (WMD-inspired plus cosine with IDF weighting), and an edge-conditioned GNN refinement. That combination is not in the cited prior work. The paper also runs a genuine cross-lingual study (English patient comments, French COVID news) across three document-length subsets each, and it includes a five-run statistical validation with t-tests and ANOVA. For a topic model meant to produce ready-to-use labels, getting interpretable short phrases is a real practical advantage. The approach is coherent and the experiments are described in enough detail to roughly reproduce, though they ship no code or data.

Soft spots, in order of severity. First, the circularity above is load-bearing. The central claim that HAMLET is effective rests on the ablation, and the ablation's main component is confounded. Second, the paper explicitly declines a direct comparison with LDA/NMF/BERTopic, saying 'a direct comparison is irrelevant' (§5). That is a dodge. They could map HAMLET phrase topics to words and compute C_v or NPMI, or do a human evaluation. Without an external anchor, the absolute Composite Score is uninterpretable. Third, the SGS weights are tuned with silhouette score on the same topics that later contribute to a Composite Score that includes silhouette; the Composite weights themselves are chosen based on sensitivity analysis on the same six datasets. This is tuning-to-the-test-set, though its impact is unclear. Fourth, no code or data.

I don't think this paper is ready for a serious referee. The architecture is worth a workshop paper or a preprint, and the authors show genuine thinking. But the evidence supporting the central claim is not independent. A revision needs a non-circular evaluation: compare against baselines on a shared metric, or report human-judged topic quality. If they do that, I'd be happy to look again. For now, I'd desk-reject rather than spend referee time.

Referee Report

4 major / 5 minor

Summary. The paper proposes HAMLET, a healthcare topic-modeling pipeline in which GPT-4o generates per-document topic phrases, SBERT-BERT hybrid embeddings represent documents and topics, a Semantic-Geometric Similarity (SGS) method builds similarity matrices, and an edge-conditioned GNN (EdgeRefineGNN) refines topic embeddings on a heterogeneous graph of documents, topics, and words. The final top-k topics are selected via k-means clustering followed by coherence-, centroid-, or connectivity-based extraction. The method is evaluated on three English and three French healthcare datasets of varying document lengths, using a newly introduced Composite Score, an ablation of refined versus original embeddings, and a statistical validation with t-tests and ANOVA. The paper claims that HAMLET produces interpretable, cross-lingual, document-length-insensitive topics without human intervention.

Significance. If the central effectiveness claim were supported, HAMLET would be a useful contribution to label-scarce healthcare text analysis, particularly because its outputs are short phrases usable as classification labels and because it is demonstrated on two languages. The paper also includes several positive features: six datasets with varying document lengths, repeated trials with descriptive statistics, t-tests, ANOVA, and sensitivity analysis of the Composite Score weights. However, no code or data are released, the evaluation metric is introduced in the same paper, and the main ablation evidence is confounded by the construction of the coherence metric. As a result, the paper currently does not provide independent evidence that HAMLET outperforms existing topic models or that EdgeRefineGNN improves semantic topic quality.

major comments (4)

[§4.6, §4.7.3, §4.5.2, §5.3] The central ablation claim is confounded by the coherence metric. Coherence_t is defined in §4.6 as avg(cosine_sim(emb_t, emb_associated_words)), where the associated words are exactly the word nodes connected to the topic by Word-in-Topic edges (§4.4). In EdgeRefineGNN's message passing (§4.5.2), the refined topic embedding is a learned mean-pooled aggregation over its neighboring word, topic, and document nodes, so the refinement directly moves topic embeddings toward the word embeddings used to score coherence. The reported gains in §5.3.1 (e.g., overall English coherence 0.470 for original versus 0.613 for refined, and Composite Score 0.663±0.008 versus 0.740±0.023) therefore do not constitute independent evidence of semantic improvement. The paper needs an external evaluation—standard coherence metrics on top words, human annotation, or downstream task performance—to support the claim that EdgeRefineGNN improves topic quality.
[§5 (intro) and §5.1–§5.2] The central effectiveness claim is not tested against existing topic models on any common metric. The paper states that because HAMLET outputs phrases and existing models output bag-of-words topics, 'a direct comparison is irrelevant.' Yet Tables 5–10 report LDA, LSA, NMF, and BERTopic on Cv, Cnpmi, Topic Diversity, and Jaccard similarity, while HAMLET is evaluated only on the new Composite Score; no shared metric is used across both. The assertion that 'HAMLET already has an edge' is therefore unsupported. A head-to-head comparison on at least one standard metric, or a human evaluation of topic quality, is required before the claim of effectiveness can be accepted.
[§4.7.6, Tables 9–14] The Composite Score is introduced and weighted within the same paper, and the weights are selected from a sensitivity analysis on the same datasets used for the main evaluation. This makes the headline scores dependent on a criterion whose validity is not independently established. In particular, Topic Diversity receives weight 0.4, but diversity measures are not quality measures; high diversity can coexist with fragmented or meaningless topics. The paper should validate the Composite Score against established metrics or human judgments before using it to rank HAMLET's variants and to justify the ablation conclusion.
[§4.1–§4.7 and Appendices] The experimental setup is not reproducible as reported. No code, model weights, or processed data are released; the GPT-4o version, temperature, and other decoding parameters are not specified; the edge-feature encoding for the GNN is not fully defined; and the graph-construction thresholds are only partially described. Without these details, the statistical validation in §5.4 cannot be checked, and the reader's take that nothing external validates the absolute Composite Score remains correct. The authors should release code and data, and specify all experimental settings.

minor comments (5)

[Throughout] The manuscript contains multiple typographical and grammatical errors, e.g., 'one key NLP task that enables that converts raw data' (§1), 'new approaches for for topic modeling' (§2.1), and 'an unbalanced and duplicates' in the reference list.
[Table and figure numbering] The numbering is inconsistent: Figure 5 is captioned 'Figure 0', Table 3 is used twice (topics counts and graph network details), and the text references 'Table 171' for ANOVA results. The numbering should be corrected throughout.
[§4.3.1] The IDF formula is garbled in the text: IDF(w)=log+Ncount(w)1 does not display the intended expression. The formula should be typeset correctly.
[References] The reference list duplicates entries (e.g., Blei et al. appears as [1] and [15]; Lee and Seung as [2] and [16]; several others are repeated), and a number of citations in the text are missing from the list. The reference list should be deduplicated and checked for completeness.
[§4.7.6] The Composite Score formula mixes terms with incompatible directions: Jaccard similarity is listed with a positive coefficient but the text states that lower values are better. The normalization direction should be stated explicitly for each component.

Circularity Check

2 steps flagged · score 8.0 of 10

The ablation gain is built into the metric: EdgeRefineGNN mean-pools the very word/topic neighbors that the Coherence score then measures, and coherence-based extraction selects on the same score.

self definitional [§4.4 (Hybrid Graph Network Construction), §4.5.2 (Message Passing Mechanism), §4.6 (Coherence-based Extraction)]
"• Word in topic edges: connecting words to topics they are associated with. ... Message Aggregation: The model resorts to mean pooling to aggregate messages from all neighboring nodes at layer l before feeding it to layer l+1 ... h_i^{(l+1)}=Dropout(σ(LayerNorm(1/|N(i)|∑_{j∈N(i)} f_e(e_ij) h_j + h_i^{(l)}))). ... Coherence_t = avg(cosine_sim(emb_t, emb_associated_words))"

For a topic node i, the word-in-topic edges make the topic's 'associated words' neighbors in N(i). EdgeRefineGNN writes the refined topic embedding as a learned mean-pooled function of exactly those neighboring node embeddings (word, topic, document). The Coherence_t metric then scores the refined embedding against emb_associated_words, i.e., against vectors that were aggregated into the refined embedding itself. A mean of a set of vectors tends to have higher average cosine similarity to the members of that set than an arbitrary input vector, so the reported Coherence gains in §5.3 (English 0.470→0.613; French 0.404→0.496) are largely a reward for the model's own averaging and are not independent evidence of semantic improvement.
self definitional [§4.6 (Coherence-based Extraction), §4.7.3 (Topic Coherence), §5.3 (Ablation Study)]
"Coherence-based Extraction: the coherence-based approach evaluates how well each topic's embedding aligns with its associated words ... Coherence_t = avg(cosine_sim(emb_t, emb_associated_words)). ... Coherence = 1/N Σ Score_coherence_t ... Score_coherence is the average of the similarities between topic i and its neighboring words."

The same Coherence_t formula is used both to select the top-k topics under 'coherence-based extraction' and, averaged, as the Coherence component of the Composite Score reported in the ablation. Therefore the 'Refined - Coherence' row in §5.3 does not test EdgeRefineGNN against an independent notion of topic quality; it reports the mean of the selection score over topics chosen by that very score. The ablation's headline composite gains (English 0.663→0.740; French 0.633→0.743) and the significant t-tests in §5.4 compare configurations that all maximize or are evaluated by the same entangled quantity.

full rationale

The central claim of the paper is that EdgeRefineGNN improves topic quality, and the principal evidence is the §5.3 ablation comparing original and refined embeddings. I find a specific, quotable circularity in that evidence: the Coherence metric used in both extraction and evaluation is average cosine similarity between a topic embedding and its associated word embeddings, while EdgeRefineGNN's message passing updates a topic embedding by mean-pooling over its neighboring nodes, which include exactly those associated word nodes via Word-in-Topic edges. Additionally, 'coherence-based extraction' selects topics by the same Coherence_t score that later appears as the Coherence component of the Composite Score. Thus the improvement in Coherence, and a substantial part of the Composite Score gain, is a built-in consequence of the evaluation design rather than an external validation of semantic quality. The paper also forgoes a shared benchmark ('Because the conventional methods and the proposed approach operate differently, a direct comparison is irrelevant', §5) and releases no code or data, so there is no independent anchor for the absolute Composite Scores. Topic Diversity and low Jaccard overlap are diversity/redundancy measures, not quality measures, and do not rescue the coherence-based conclusion. I find no load-bearing self-citation or imported uniqueness theorem; the circularity is internal to the metric and architecture. Score 8 reflects that the ablation's main claim is substantially forced by definition, though not every component of the composite (e.g., Topic Diversity) is equally entangled.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central evidence chain rests on four choices: LLM topic generation, embedding-similarity coherence, graph smoothing, and a custom weighted composite score. Three of these are domain assumptions introduced specifically for this method, and the coherence metric overlaps with the graph update, which raises the circularity burden.

free parameters (4)

weight_wmd = 0.900
A tunable weight balancing WMD-inspired and cosine similarities in SGS; optimized by grid search using silhouette score across all six datasets (Section 4.3.5).
weight_idf = 0.099
A tunable soft IDF weighting for word embeddings in SGS; optimized jointly with weight_wmd (Section 4.3.5).
similarity threshold percentile = 90th percentile per dataset
Edges between similar topics and similar words are kept above a threshold chosen as the 90th percentile for each dataset, after experimenting with fixed thresholds 0.7 to 0.9 (Section 4.4).
number of topics k = 50 (25 for English Dataset 3)
The number of topics to extract is set by the user or author, not derived from data; k = 50 except English Dataset 3 where k = 25 due to small size (Section 4.6).

assumptions (5)

domain assumption GPT-4o generated topic phrases are a reasonable and sufficient initial representation of the corpus topics.
The entire pipeline depends on the quality of the LLM's raw topics; no comparison is made against other topic-generation strategies (Section 4.1).
domain assumption Embedding-space cosine similarity between a topic and its associated words is a valid measure of topic coherence and quality.
Used both for coherence-based topic extraction and for the Coherence component of the Composite Score (Sections 4.6 and 4.7.3).
domain assumption Graph message passing that aggregates neighboring word and topic embeddings improves topic representation.
This is the ablation claim, but the evaluation metric is partly constructed from the same graph neighbors, so the assumption is not independently tested (Sections 4.5.2 and 5.3).
ad hoc to paper The Composite Score weights are a meaningful aggregate measure of topic model quality.
Weights (0.4 diversity, 0.15 Jaccard, 0.2 coherence, 0.2 silhouette, 0.05 Davies-Bouldin) are justified by a sensitivity analysis but are not validated against human judgments or downstream task performance (Section 4.7.6).
standard math The Hungarian algorithm and standard linear assignment machinery solve the WMD-inspired alignment correctly.
Used in SGS to align words between topics; assumes SciPy's linear_sum_assignment is correct (Section 4.3.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of HAMLET: Healthcare-focused Adaptive Multilingual Learning Embedding-based Topic Modeling." pith.science (2026). https://pith.science/paper/7LMEZR5X

@misc{pith2026250507157,
  author       = {Pith},
  title        = {Pith review of: HAMLET: Healthcare-focused Adaptive Multilingual Learning Embedding-based Topic Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LMEZR5X}},
  note         = {Machine review of arXiv:2505.07157}
}

read the original abstract

Traditional topic models often struggle with contextual nuances and fail to adequately handle polysemy and rare words. This limitation typically results in topics that lack coherence and quality. Large Language Models (LLMs) can mitigate this issue by generating an initial set of topics. However, these raw topics frequently lack refinement and representativeness, which leads to redundancy without lexical similarity and reduced interpretability. This paper introduces HAMLET, a graph-driven architecture for cross-lingual healthcare topic modeling that uses LLMs. The proposed approach leverages neural-enhanced semantic fusion to refine the embeddings of topics generated by the LLM. Instead of relying solely on statistical co-occurrence or human interpretation to extract topics from a document corpus, this method introduces a topic embedding refinement that uses Bidirectional Encoder Representations from Transformers (BERT) and Graph Neural Networks (GNN). After topic generation, a hybrid technique that involves BERT and Sentence-BERT (SBERT) is employed for embedding. The topic representations are further refined using a GNN, which establishes connections between documents, topics, words, similar topics, and similar words. A novel method is introduced to compute similarities. Consequently, the topic embeddings are refined, and the top k topics are extracted. Experiments were conducted using two healthcare datasets, one in English and one in French, from which six sets were derived. The results demonstrate the effectiveness of HAMLET.

Figures

Figures reproduced from arXiv: 2505.07157 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[5]

Existing topic models are first used and evaluated

Results and Discussion In this section, HAMLET's performance is discussed for the case of three English datasets and three French datasets. Existing topic models are first used and evaluated. Afterward, HAMLET is assessed through the proposed Composite Score. Because the conventional methods and the proposed approach operate differently, a direct comparis...

arXiv 2003
[21]

Islam, T., & Goldwasser, D. (2024). Discovering latent themes in social media messaging: A machine-in-the-loop approach integrating llms. arXiv preprint arXiv:2403.10707. [22] Koloski, B., Lavrač, N., Cestnik, B., Pollak, S., Škrlj, B., & Kastrin, A. (2024, April). AHAM: Adapt, Help, Ask, Model Harvesting LLMs for Literature Mining. In International Sympo...

arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [5]

Existing topic models are first used and evaluated

Results and Discussion In this section, HAMLET's performance is discussed for the case of three English datasets and three French datasets. Existing topic models are first used and evaluated. Afterward, HAMLET is assessed through the proposed Composite Score. Because the conventional methods and the proposed approach operate differently, a direct comparis...

arXiv 2003

[2] [21]

Islam, T., & Goldwasser, D. (2024). Discovering latent themes in social media messaging: A machine-in-the-loop approach integrating llms. arXiv preprint arXiv:2403.10707. [22] Koloski, B., Lavrač, N., Cestnik, B., Pollak, S., Škrlj, B., & Kastrin, A. (2024, April). AHAM: Adapt, Help, Ask, Model Harvesting LLMs for Literature Mining. In International Sympo...

arXiv 2024