Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining a Mixture of Experts layer with BERT and Visual Transformer embeddings improves healthy-food recommendation accuracy over using either approach alone.

desk verdict Plausible architecture, but the evaluation is too weak and internally inconsistent to support the headline comparison. read the letter →

arxiv 2412.11557 v1 pith:PUXRK52D submitted 2024-12-16 cs.IR cs.DB

Jingyu Xu , Yang Wang This is my paper · ORCID
classification cs.IRcs.DB
keywords healthcarerecommendationMixtureofExpertsmultimodalBERTVisualTransformercoldstartNDCGhealthyfood
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a hybrid recommendation architecture for healthy-food suggestions that combines a Mixture of Experts (MOE) module with pretrained BERT and Visual Transformer (ViT) embeddings. It claims that on a small self-built dataset of 177 patient-user entries, the hybrid model improves Precision@5 from 0.66 (text-only) to 0.73, and NDCG from 0.61 to 0.81, relative to single-modality and single-model baselines. The best variant, MOE-Transformers-Stacking, also reaches a MAP@5 of 0.24. The authors additionally report that image data adds only limited gains beyond text, particularly for new users, and that image reclassification errors hurt results. A sympathetic reader would care because the architecture offers a concrete template for multimodal, multi-task recommendations in a domain where data is scarce and heterogeneous.

What carries the argument

The central mechanism is the Mixture of Experts module: a shared neural collaborative filtering layer compresses the concatenated text, image, and user vectors into 648 dimensions; three transformer-block experts each process the shared input; a gating layer, built from XGBoost feeding a feedforward network, decides how to weight the experts; and task-specific heads produce the recommendation and description outputs. BERT and ViT provide the text and image embeddings, and simple concatenation fuses the modalities before the MOE module. The gating mechanism is what lets the model activate only a subset of experts per input, reducing computation while keeping task-specific behavior.

What would settle it

Re-running the comparison over many random 80/20 splits and computing bootstrap confidence intervals for Precision@5 and NDCG; if the confidence intervals for the hybrid model and the MOE-only baseline overlap across seeds, the claimed improvement is not established. Independently, showing that the expert relevance labels are inconsistent across labelers would also break the comparison.

Watch

Extended reading notes

Core claim

The paper claims that adding a Mixture of Experts layer on top of multimodal feature extractors -- BERT for text, ViT for images, with the embeddings concatenated to structured user data -- yields more accurate and better-ranked healthy-food recommendations than using either the large models or MOE alone. The best configuration, MOE-Transformers-Stacking, obtains Precision@5 of 0.73, NDCG of 0.81, and MAP@5 of 0.24 on the test split. The paper also argues that image features contribute less than text, that the cold-start problem remains for visual data, and that image reclassification failures degrade performance when photos are low-quality or taken in varied contexts.

Load-bearing premise

The evaluation is valid: the expert-generated relevance labels are correct, and the single random 80/20 split of the 177-entry dataset is representative enough that the reported metric gaps are meaningful.

Editorial extensions

If this is right

  • If the hybrid architecture is correct, the same MOE-plus-BERT/ViT template can be applied to other health recommendation tasks, such as exercise plans or medication reminders, where patient text and images are available.
  • The reported metric gains mean that combining a sparse-gating MOE module with pretrained embeddings can work on very small datasets, not just large-scale industrial ones.
  • The text-dominant results imply that future systems should invest in high-quality textual descriptions and treat images as an auxiliary signal rather than a primary one.
  • The cold-start finding suggests that visual-only recommendation will underperform for new users, so practical deployments should log user behavior or elicit exploratory interactions before relying on images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to test the same architecture on a public benchmark dataset to see whether the MOE-plus-BERT/ViT advantage holds beyond the self-built 177-entry corpus.
  • The reported NDCG difference between Table 1 and Table 2 suggests the gating upgrade contributes substantially, but the paper does not run the same gating on a non-MOE baseline; a direct ablation would isolate where the gain comes from.
  • The limited image contribution might be improved by cross-attention between text and image embeddings instead of simple concatenation, a testable design change that follows from the paper's own analysis of image inconsistencies.
  • The single random split means the exact gaps could change under a different split; recomputing with many seeds would show how stable the reported ordering is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a hybrid healthcare recommendation model that combines a Mixture-of-Experts (MOE) module with pretrained BERT and ViT encoders to recommend healthy food items from patient text, patient-uploaded images, and structured user data. The authors construct a self-built dataset of 177 entries, preprocess and merge text/image data, and evaluate the model against text-only, image-only, and multimodal (non-MOE) variants using Precision@5, NDCG, MAP@5, and (in the text) Recall. Additional ablation experiments compare different MOE internal architectures and different visual backbones. The central claim is that the proposed MOE+multimodal model outperforms baselines that use MOE or large language models individually, while the discussion also emphasizes that image data provides only limited improvement, particularly for cold-start cases.

Significance. If the empirical claims were supported, the paper would offer a reasonable engineering demonstration of combining off-the-shelf BERT/ViT embeddings with an MOE gating mechanism for a small-domain food recommendation task. The qualitative observations about image-based cold-start difficulties and image reclassification are potentially useful for practitioners. However, the significance is currently limited by the absence of code or data release, the very small dataset, and the reliance on a single 80/20 split with no uncertainty quantification. The architecture is not formally analyzed, and the novelty relative to standard multimodal fusion with an MOE layer is not deeply established. The paper is best read as a preliminary application study rather than a definitive comparative evaluation.

major comments (4)
  1. [Section 3.2 / Section 4 / Tables 1 and 2] The central comparative claim rests on a single random 80/20 split of 177 entries, yielding roughly 35 test records. The reported improvements (e.g., Precision@5 from 0.66 to 0.73, NDCG from 0.61 to 0.71 or 0.81) correspond to only a handful of correct top-5 recommendations, and the paper provides no confidence intervals, repeated splits, cross-validation, or significance tests. Given the small test set, the reported gaps are within plausible sampling noise, so the claim that the hybrid model 'outperforms' the baselines is not currently established. The authors should report repeated random splits or cross-validation with standard deviations and, where possible, a statistical significance test.
  2. [Section 5 vs. Tables 1 and 2] The numbers quoted in Section 5 do not consistently match the tables. Section 5 states that the MOE+multimodal model achieves a Recall of 0.81, but Table 1 contains no Recall column. It also states that the image-only NDCG is 0.63, whereas Table 1 lists 0.61 for image data. Moreover, Section 5's NDCG value of 0.81 for the multimodal MOE model matches Table 2 row 1 (MOE-Transformers-Stacking) rather than Table 1 row 4 (MOE + Multi-modal Data, NDCG 0.71). These inconsistencies make it impossible to determine which experimental outcome supports the central claim, and they must be reconciled or corrected.
  3. [Abstract / Section 4 / Section 7] The abstract and conclusion claim that the hybrid model outperforms baselines that use 'MOE or large language models individually.' The experimental design does not directly test this claim. Table 1 compares text-only BERT, image-only ViT, multimodal without MOE, and MOE+multimodal; there is no MOE-only condition that excludes the pretrained LLM/text encoders, and no large-language-model-based multimodal condition that excludes MOE. The reported ablations in Table 2 vary the MOE internal components but do not supply the missing baseline conditions. The claim as phrased is therefore not supported by the experiments.
  4. [Section 3.4 / Section 5] Section 3.4 says the paper evaluates models using four indices, but it gives formulas only for Precision@K, NDCG@K, and MAP@K. Recall is never defined, and no Recall column appears in any table, despite Recall figures being quoted in Section 5. Since Recall is one of the headline metrics in the abstract and analysis, the authors need to define it, add it to the evaluation tables, and make the reported Recall values traceable to the experimental results.
minor comments (6)
  1. [Table 2] The ID numbering in Table 2 skips from 4 to 6; either renumber the rows or explain the missing row.
  2. [Figures and captions] The figure numbering is inconsistent: Section 3.3 refers to 'Figure 1' for the proposed model, but Figure 1 in Section 2 is the neural matrix factorization diagram, and the proposed model appears as Figure 2. The caption for Figure 2 also says 'Figure 2. Structure of The hybrid recommender system Model.' Please renumber and reference all figures consistently.
  3. [Abstract] The abstract contains an incomplete sentence: 'The paper find mage data provided relatively limited improvement...' appears to be missing words. This needs copyediting throughout, as several other sentences in Sections 5 and 6 also read awkwardly.
  4. [Section 3.4] In the description of DCG, 'cash-in' should presumably be 'gain'; the sentence 'CG calculates the total relevance... while DCG prioritizes the impact of position loss' is unclear and should be rewritten for precision.
  5. [Section 4] The abbreviation 'VIR' appears in the first sentence of Section 4 ('the large model techniques of VIR and BERT'); this should be 'ViT.'
  6. [Section 3.3.2] The hyperparameters of the MOE components are not fully specified: the number of experts, the gating architecture, the expert hidden dimensions, and the training setup for the XGBoost base learner are only partially described. For reproducibility, these details should be provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported gains are empirical measurements on a self-built split, not consequences of the model's definition or of self-citations.

full rationale

The paper's central claim is an empirical comparison of measured Precision@K, NDCG, MAP@5, and (in Section 5) recall values across model variants. No equation defines a predicted quantity in terms of the fitted metric, and no parameter fitted to a subset of the data is subsequently relabeled as a prediction. The evaluation uses one random 80/20 split of the 177-entry self-built dataset described in Section 3.1 and 3.2; this is a statistical-validity limitation (roughly 35 test records, no error bars, no significance testing), not a circularity. The only apparent self-citation, reference [3] (Xu et al. 2023), appears in the introduction as an example of prior health-related systems and is not used to justify the architecture, the evaluation, or the comparative results, so it is not load-bearing. The Section 5 discussion quotes recall values (0.81 vs. 0.73/0.68) that do not appear in Table 1 and states an image-only NDCG of 0.63 that contradicts Table 1's 0.61; these are internal-consistency errors that undermine the empirical report but do not constitute circular reasoning. The model outputs are evaluated against held-out expert-labeled relevance data, so the reported improvements are not true by construction. Absence of an external benchmark weakens the evidentiary value of the claims, but the claimed derivation chain is not circular.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central claim rests entirely on a small, unreleased dataset and standard pretrained encoders. All architecture numbers (expert count, dimensions, K, epochs, gating design) are hand-set, and no external benchmark or error analysis validates the evaluation. There are no new physical or conceptual entities.

free parameters (7)
  • number_of_experts = 3
    Chosen without ablation over expert counts; central architecture choice.
  • single_modality_output_dimension = 648
    Chosen by hand for the NCF layer output; no justification.
  • multimodal_concatenated_dimension = 1296
    Concat of two 648-dim vectors, then compressed by gating FFN; arbitrary.
  • train_test_split_ratio = 80/20 random
    Single split, no cross-validation; with 177 entries this is high-variance.
  • evaluation_top_k = 5
    K fixed at 5 for all metrics; no sensitivity check.
  • training_epochs = 100
    Fixed without early stopping or convergence analysis.
  • gating_architecture = FFN + XGBoost stacking
    One gating design chosen; Table 2 varies expert backbones but not the gating mechanism.
assumptions (3)
  • domain assumption Pretrained BERT and ViT embeddings transfer to the health-food domain without fine-tuning for this task.
    Section 3.3.1 uses off-the-shelf BERT and ViT; no domain adaptation or fine-tuning is described.
  • domain assumption Expert-generated relevance labels in the self-built dataset are correct and complete.
    Section 3.1 says all data were generated through expert processing; no inter-annotator agreement or validation is reported.
  • domain assumption A single random 80/20 split of 177 entries produces stable model rankings.
    Section 3.2 and Section 4 use one random split; with roughly 35 test entries, metric gaps of 0.02-0.10 may be noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture." pith.science (2026). https://pith.science/paper/PUXRK52D

@misc{pith2026241211557,
  author       = {Pith},
  title        = {Pith review of: Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUXRK52D}},
  note         = {Machine review of arXiv:2412.11557}
}
read the original abstract

With the increasing availability of multimodal data, many fields urgently require advanced architectures capable of effectively integrating these diverse data sources to address specific problems. This study proposes a hybrid recommendation model that combines the Mixture of Experts (MOE) framework with large language models to enhance the performance of recommendation systems in the healthcare domain. We built a small dataset for recommending healthy food based on patient descriptions and evaluated the model's performance on several key metrics, including Precision, Recall, NDCG, and MAP@5. The experimental results show that the hybrid model outperforms the baseline models, which use MOE or large language models individually, in terms of both accuracy and personalized recommendation effectiveness. The paper finds image data provided relatively limited improvement in the performance of the personalized recommendation system, particularly in addressing the cold start problem. Then, the issue of reclassification of images also affected the recommendation results, especially when dealing with low-quality images or changes in the appearance of items, leading to suboptimal performance. The findings provide valuable insights into the development of powerful, scalable, and high-performance recommendation systems, advancing the application of personalized recommendation technologies in real-world domains such as healthcare.

Figures

Figures reproduced from arXiv: 2412.11557 by the authors.

Figure 1
Figure 1. Structure of the neural matrix factorization. However, these methods still struggle to fully capture the dynamic health needs of patients, especially in complex medical environments. Deep learning has made significant progress as a powerful tool in the field of recommendation systems. Deep neural networks are capable of learning nonlinear patterns from high-dimensional data, making them particularly suitable for han… view at source ↗
Figure 2
Figure 2. Structure of The hybrid recommender system Model. 3.3.1 VIT and BERT We chose this framework to use BERT and ViT as the basic model, and its model is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Structure of BERT and ViT. 3.3.2 Mixture of Experts (MOE) MOE is a multi-task learning framework that allows multiple tasks to share part of the network parameters while maintaining task-specific outputs. This architecture is particularly suitable for scenarios requiring simultaneous learning of multiple related tasks, such as classification and recommendation. Its structure includes the following components: ⚫ Neur… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 30 canonical work pages

  1. [1]

    Journal of medical Internet research, 2016

    Pang, P.C.-I., et al., Designing health websites based on users’ web-based information-seeking behaviors: A mixed -method observational study. Journal of medical Internet research, 2016. 18(6): p. e145

    work page 2016

  2. [2]

    Health and Technology, 2015

    Pang, P.C.-I., et al., Conceptualising health information seeking behaviours and exploratory search: result of a qualitative study. Health and Technology, 2015. 5: p. 45-55

    work page 2015

  3. [3]

    Automated Scoring of Clinical Patient Notes using Advanced NLP and Pseudo Labeling

    Xu, J., et al. Automated Scoring of Clinical Patient Notes using Advanced NLP and Pseudo Labeling. in 2023 5th International Conference on Artificial Intelligence and Computer Applications (ICAICA). 2023. IEEE

    work page 2023

  4. [4]

    Applied Sciences, 2024

    Lopez-Barreiro, J., et al., Artificial Intelligence-Powered Recommender Systems for Promoting Healthy Habits and Active Aging: A Systematic Review. Applied Sciences, 2024. 14(22): p. 10220

    work page 2024

  5. [5]

    Rendle, and R

    Koren, Y., S. Rendle, and R. Bell, Advances in collaborative filtering. Recommender systems handbook, 2021: p. 91-142

    work page 2021

  6. [6]

    Van Meteren, R. and M. Van Someren. Using content-based filtering for recommendation. in Proceedings of the machine learning in the new information age: MLnet/ECML2000 workshop

    work page

  7. [7]

    Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments

    Popescul, A., et al., Probabilistic models for unified collaborative and content -based recommendation in sparse-data environments. arXiv preprint arXiv:1301.2303, 2013

    work page Pith review arXiv 2013

  8. [8]

    Lu, J., Optimizing e-commerce with multi-objective recommendations using ensemble learning. 2024

    work page 2024

Show all 32 references
  1. [9]

    Customer Segment Classification Prediction in the Australian Retail Based on Machine Learning Algorithms

    Luo, Y., et al. Customer Segment Classification Prediction in the Australian Retail Based on Machine Learning Algorithms . in Proceedings of the 2023 4th International Conference on Machine Learning and Computer Application. 2023

    2023

  2. [10]

    Sun, Y., et al., A Multi -Objective Recommender System for Enhanced Consumer Behavior Prediction in E-Commerce. 2024

    2024

  3. [11]

    Masoudnia, S. and R. Ebrahimpour, Mixture of experts: a literature survey. Artificial Intelligence Review, 2014. 42: p. 275-293

    2014

  4. [12]

    Nature medicine, 2023

    Thirunavukarasu, A.J., et al., Large language models in medicine. Nature medicine, 2023. 29(8): p. 1930-1940

    2023

  5. [13]

    Hybrid Model Integration of LightGBM, DeepFM, and DIN for Enhanced Purchase Prediction on the Elo Dataset

    Lu, J., et al. Hybrid Model Integration of LightGBM, DeepFM, and DIN for Enhanced Purchase Prediction on the Elo Dataset . in 2024 IEEE 7th International Conference on Information Systems and Computer Aided Education (ICISCAE). 2024. IEEE

    2024

  6. [14]

    Applied Sciences, 2024

    Zheng, D., et al., MicroBERT: Distilling MoE -Based Knowledge from BERT into a Lighter Model. Applied Sciences, 2024. 14(14): p. 6171

    2024

  7. [15]

    Konstan, and J

    Herlocker, J.L., J.A. Konstan, and J. Riedl. Explaining collaborative filtering recommendations. in Proceedings of the 2000 ACM conference on Computer supported cooperative work. 2000

    2000

  8. [16]

    Zhou, and X

    Tsang, G., S.-M. Zhou, and X. Xie, Modeling large sparse data for feature selection: hospital admission predictions of the dementia patients using primary care electronic health records. IEEE Journal of Translational Engineering in Health and Medicine, 2020. 9: p. 1-13

    2020

  9. [17]

    Scientific Reports, 2022

    Kim, Y., et al., A pre-trained BERT for Korean medical natural language processing. Scientific Reports, 2022. 12(1): p. 13847

    2022

  10. [18]

    Lu, J., Enhancing Chatbot User Satisfaction: A Machine Learning Approach Integrating Decision Tree, TF-IDF, and BERTopic. 2024

    2024

  11. [19]

    arXiv preprint arXiv:2012.12556, 2020

    Han, K., et al., A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020

    2012 arXiv

  12. [20]

    Ye, and R

    Luo, Y., Z. Ye, and R. Lyu. Detecting student depression on Weibo based on various multimodal fusion methods . in Fourth International Conference on Signal Processing and Machine Learning (CONF-SPML 2024). 2024. SPIE

    2024

  13. [21]

    Neural Computation, 2020

    Gao, J., et al., A survey on deep learning for multimodal data fusion. Neural Computation, 2020. 32(5): p. 829-864

    2020

  14. [22]

    Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture

    Jiang, K., et al. Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture . in 2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE). 2024. IEEE

    2024

  15. [23]

    Multimedia systems,

    Atrey, P.K., et al., Multimodal fusion for multimedia analysis: a survey. Multimedia systems,

  16. [24]

    Advances in Neural Information Processing Systems, 2017

    Vaswani, A., Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  17. [25]

    Identifying Factors Influencing China Junior High Students' Cognitive Ability through Educational Data Mining: Utilizing LASSO, Random Forest, and XGBoost

    Luo, Y. Identifying Factors Influencing China Junior High Students' Cognitive Ability through Educational Data Mining: Utilizing LASSO, Random Forest, and XGBoost . in Proceedings of the 4th International Conference on Modern Education and Information Management, ICMEIM 2023, ...

    2023

  18. [26]

    arXiv preprint arXiv:1412.6980, 2014

    Kingma, D.P., Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  19. [27]

    Tianfield, and V

    Lazzarini, R., H. Tianfield, and V. Charissis, A stacking ensemble of deep learning models for IoT intrusion detection. Knowledge-Based Systems, 2023. 279: p. 110941

    2023

  20. [28]

    Da’u, A. and N. Salim, Recommendation system based on deep learning methods: a systematic review and new directions. Artificial Intelligence Review, 2020. 53(4): p. 2709-2748

    2020

  21. [29]

    An, H.-w. and N. Moon, Design of recommendation system for tourist spot using sentiment analysis based on CNN -LSTM. Journal of Ambient Intelligence and Humanized Computing,

  22. [30]

    Liu, T., P.C. -I. Pang, and Q. Xiong. Visualized Analysis of Research Trends of Digital Technology and Public Health Based on CiteSpace . in International Symposium on World Ecological Design. 2024. IOS Press

    2024

  23. [31]

    Cheong-Iao, and S

    Luo, Y., P. Cheong-Iao, and S. Chang, Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models. arXiv preprint arXiv:2408.08894, 2024

    2024 arXiv

  24. [32]

    Li, S., Harnessing multimodal data and mult -recall strategies for enhanced product recommendation in e-commerce. 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.