{"id":"3c19a736-ca9c-42ad-8914-9f0050cf25c5","arxiv_id":"2509.06576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MASH aligns medical codes across institutions using optimal transport and builds hierarchical graphs, including the first automated hierarchies for VA local laboratory codes.","lead":"Researchers built an automated system called MASH that aligns medical codes from two major US health systems and organizes them into tree-like hierarchies, including lab tests that had no standard structure before. If it holds up, it could help hospitals combine records across institutions for research.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bottom-set criterion D_ij = 0 is unproven for nodes at different depths, and the Supp B.3 converse proof is invalid; the recursive grouping algorithm's correctness therefore rests on an assumption that can fail.","rationale":"The reader's weakest_assumption correctly identifies the same concern: the D_ij = 0 criterion is not proven equivalent to bottom-set membership, and the Supp B.3 converse is invalid for nodes at different depths. My analysis confirms and sharpens this: not only is the converse unproven, it is actually false in general. The forward direction holds under additivity, but the converse requires more than the constant-difference property for k outside the nearest-common-ancestor subtree; considering all k as Eq. (6) requires can break the criterion even for legitimate parent-child pairs when edge lengths vary. This directly undermines Algorithm 2's correctness, which is central to the paper's claim of automated hierarchy construction. The concern is load-bearing because the paper's second headline outcome (hierarchical graphs, including for VA local lab codes) depends on Algorithm 2; if the criterion can merge nodes at different depths or split true bottom sets, the hierarchy is corrupted. The paper's practical k-means approximation and silhouette-based cluster number selection might mitigate the issue in practice, and the real-data case studies show plausible results, but the theoretical foundation is missing. The evaluation also has circularity issues and no code is released, but the correctness of the grouping criterion is the more fundamental concern; if it fails, the main methodological contribution is unjustified. Verdict remains CONDITIONAL: the proof gap is fixable with a corrected criterion (e.g., using all k or a different bottom-set characterization), and the empirical plausibility suggests the approach may still work in practice, but as written the central algorithm's correctness is not established.","tokens_in":18803,"tokens_out":4778,"duration_ms":31244,"concrete_test":"Construct a small explicit tree with known edge lengths (e.g., root R with children A and B, A with child i, B with child C, C with children j and k) and compute D_ij using Eq. (6) over all nodes. Vary edge lengths: (1) i and j at different depths (i is uncle of j as above) and (2) i parent of j. Show whether D_ij = 0 fails for non-bottom-set pairs and whether it also fails for some parent-child pairs, contradicting the claimed equivalence. If a counterexample is found, the Supp B.3 converse is invalid and Algorithm 2 lacks correctness guarantees.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MASH automatically reconstructs correct hierarchies depends on Algorithm 2's ability to identify bottom sets. The paper asserts in Eq. (6) and Supp B.3 that D_ij = 0 iff i and j belong to the same bottom set. The forward direction holds, but the converse is not established. The proof in Supp B.3 only considers k outside the subtree rooted at p, the nearest common ancestor. For an uncle-nephew pair (i at depth 2, j at depth 3, with sibling parents), D_ij can equal 0 when i and j are not in a parent-child or sibling relation, because for k outside p's subtree the difference is constant, yet the nodes are not in the same bottom set. Node pairs at different depths (e.g., a code and its grandchild, or an uncle and nephew) can produce vanishing D_ij despite not being directly related as required by Definition 1. Algorithm 2 then introduces false latent parents or groups nodes incorrectly, corrupting the hierarchy. This is a load-bearing assumption because the hierarchy construction is the paper's second headline outcome; if the recursive grouping can merge nodes at different depths, the resulting graph no longer reflects the true tree. The proof gap is acknowledged in Supp B.3's 'converse direction' but is not actually proven for all k; it only holds for k outside the rooted subtree, while the definition of D_ij requires considering all other nodes k, including those inside the subtree of p. The paper provides no argument that such k cannot break the criterion, and in fact they can, e.g., when k is in the subtree of i's parent but not of j's parent. A concrete counterexample: tree with leaves i, k1 under parent A, leaves j, k2 under parent B, with A and B siblings under root R. For k = k1, d(i,k1) = 2a, d(j,k1) = a + b + a = 2a + b, difference = -b. For k = k2, d(i,k2) = a + b + a = 2a + b, d(j,k2) = 2b, difference = 2a - b. Setting these equal gives -b = 2a - b => a = 0, impossible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MASH, a three-stage framework for harmonizing multi-institution electronic health record codes and automatically constructing hierarchical graphs over them. Stage I aligns institution-specific SPPMI-SVD embeddings through neural optimal transport, Stage II maps the harmonized embeddings into hyperbolic space with additivity and contrastive losses, and Stage III recovers a latent tree by recursively identifying \"bottom sets\" via the criterion D_ij=0 and k-means. The authors apply MASH to VA and MGB data for PheCode diagnoses, RxNorm medications, and LOINC/local laboratory codes, reporting improvements over baselines in embedding-quality AUC, NMI/ARI against known ontologies, sibling precision/sensitivity, and GPT-4-based interpretability scores. The claimed headline outcomes are a common representational space enabling cross-system analyses and automated hierarchies for previously unstructured VA local laboratory codes.","tokens_in":19118,"tokens_out":5452,"duration_ms":49323,"significance":"If the technical claims were fully supported, MASH would address a real and timely barrier in multi-institutional EHR research: the heterogeneity of local coding systems and the absence of curated hierarchies for local codes. The paper has genuine strengths: it operates at substantial scale (12.6M VA and 2.5M MGB patients), integrates several complementary information sources, provides ablation comparisons, and includes a simulation study in the supplement. However, the central hierarchy-recovery criterion is not rigorously established, and the main empirical evaluations contain multiple sources of circularity. These issues directly affect the paper's two headline claims, so the significance is currently conditional rather than demonstrated.","major_comments":[{"comment":"The claimed equivalence D_ij=0 iff i and j belong to the same bottom set is not proved. The converse direction in Supp B.3 only considers nodes k outside the subtree rooted at the nearest common ancestor p. For an uncle-nephew pair (i a child of p, j a grandchild of p through a sibling c of i), all such k yield a constant value of d(z_i,z_k)-d(z_j,z_k), so D_ij=0 even though i and j are neither siblings nor in a parent-child relationship. If the current node set V contains no node inside the subtrees that would expose the depth difference, the recursive grouping in Algorithm 2 will incorrectly merge such nodes into one bottom set and introduce a false latent parent. The proof must either handle k inside p's subtree or state an additional assumption (such as equal depth for all nodes in V) that is not satisfied in the EHR setting.","section":"Supplementary Section B.3, Eq. (6), Algorithm 2"},{"comment":"The evaluation of known-code hierarchies is circular. PheCode, NDF-RT/RxNorm, and LOINC hierarchies are used as supervision in Stage II (Section 2.3) and then scored with NMI and ARI against those same ontologies in Table 3. For local laboratory codes, the reference tree in Section 4.2.1 is built from a manually curated LOINC-to-local mapping, and the same kind of LOINC-local links are used to create supervised training pairs in Section 2.3. This common-source overlap can inflate all reported alignment numbers. The authors should separate the supervision and evaluation mappings, for example by holding out a portion of the curated mapping that is never used in training.","section":"Sections 2.3 and 4.2.1, Table 3"},{"comment":"The semantic interpretability evaluation is self-referential. GPT-4o generates the latent-node annotations using the prompts in Table S3, and the same model is then asked to judge whether child annotations are sub-concepts of the parent and whether sibling annotations are distinct. This measures prompt consistency rather than independent clinical validity. An independent judge, a different model, or human review is needed before the near-perfect hierarchy and divergence scores in Table 5 can be interpreted as evidence of clinically meaningful structure.","section":"Table 5 and Supplementary Section A.1"},{"comment":"The description of MASH as a fully automated hierarchy construction framework is not supported by the implementation details. Before Algorithm 2 runs, the user fixes 21 PheCode categories, 32 RxNorm categories, and the first 5 levels of the LOINC hierarchy, and the construction of latent nodes also reuses existing parent-child relations when labels are available. These manual choices should be explicitly scoped as user-specified inputs, and the paper should assess how sensitive the results in Tables 3 and 4 are to these choices.","section":"Supplementary Section A.1, Section 3.3"}],"minor_comments":[{"comment":"The notation for the marginal total is inconsistent: C^k(i,·) = Σ_i C^k(i,k) should presumably read C^k(i,·) = Σ_j C^k(i,j), since the sum should be over the second index.","section":"Equation (1)"},{"comment":"There are small typos: the legend in Figure 3 uses \"MESHI\" instead of \"MASH\", and Table S4 says \"consitent\" instead of \"consistent.\"","section":"Figure 3 and Table S4"},{"comment":"No code repository is provided. Given that the VA and MGB data cannot be shared, the supplemental simulation is helpful, but providing the exact code used for the pipeline would improve reproducibility.","section":"Data Availability"},{"comment":"The hierarchy evaluation reports only point estimates without confidence intervals or multiple-run variation. Since k-means initialization and the silhouette heuristic introduce randomness, the authors should report the variability of NMI/ARI and precision/sensitivity across seeds.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important applied problem and the empirical setup is ambitious, but the internal validity of the main claims currently rests on an unproven criterion in the hierarchy-recovery step and on several overlapping sources of supervision and evaluation. I would encourage the editor to seek a revision that fixes the Supp B.3 proof or explicitly restricts the claim, and that clearly separates supervised information from evaluation references."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a serious attempt at an important problem—harmonizing institution-specific EHR codes and building hierarchies automatically—and the MASH pipeline sensibly integrates known components. The case studies look clinically coherent. But the paper's central theoretical justification for the recursive grouping is unsound, and the evaluation has a circularity problem. I'd send it to referees, but with the expectation of major revision.\n\nWhat's actually new: the combination of neural optimal transport alignment, hyperbolic embeddings, and recursive latent-tree grouping applied to multi-institutional EHR codes is, as far as I know, novel. The VA local lab code hierarchy, even if preliminary, is a useful artifact. The authors also include a simulation study to make some results replicable despite protected data, which is a fair attempt.\n\nThe biggest soft spot is the bottom-set criterion D_ij in (6). The paper claims D_ij=0 iff nodes belong to the same bottom set. The forward direction holds; the converse does not. Supplementary B.3 only considers k outside the nearest common ancestor's subtree, and the step \"this implies both i and j must attach directly to p\" does not follow. Once latent nodes exist, the current node set V often contains nodes whose descendant subtrees are no longer represented. For such a pair with only one other node in V, D_ij=0 even for an uncle-nephew pair. So Algorithm 2 can merge unrelated branches. This is load-bearing, not a minor gap.\n\nThe evaluation also concerns me. The same ontologies (PheCode, RxNorm, LOINC) supply supervision and then serve as ground truth for NMI/ARI. The local-lab mapping dictionary is used both as training labels and as evaluation reference. And GPT-4o annotates latent nodes and then grades its own annotations—that is not independent evidence.\n\nThe \"fully automated\" claim is also overstated: 21 PheCode and 32 RxNorm top-level categories are manually defined (taken from the VA system). That is disclosed, but it softens the headline.\n\nCredit where due: the embedding-quality gains from OT aggregation and hyperbolic refinement are real in their experiments, the simulation protocol is thoughtful, and the prose is clear.\n\nWho is this for? Researchers working on EHR code harmonization and ontology learning. They would learn something, but they should not trust the hierarchy outputs without checking the grouping behavior.\n\nRecommendation: accept for peer review, but the proof gap and the circular evaluation need fixing before the results can be taken at face value. A corrected criterion, held-out evaluation, and released code/mappings would turn this into a solid contribution.","headline":"A sensible pipeline for a real EHR harmonization problem, but the recursive grouping criterion is unproven and the evaluation is partly circular; worth refereeing, but major revision needed.","tokens_in":19826,"tokens_out":14071,"would_cite":false,"duration_ms":120967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30","68T07","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fully automated three-stage pipeline—neural optimal transport alignment, hyperbolic embedding, and recursive grouping—can both harmonize medical codes across institutions and construct clinically coherent…","keywords":["electronic health records","medical code harmonization","optimal transport","hyperbolic embeddings","hierarchical clustering","latent tree models","multi-institutional EHR data","VA local laboratory codes"],"falsifier":"Construct a small known tree—say a root with two children A and B, where A has a child C and a grandchild D—and compute the hyperbolic or information distances. For the uncle C and nephew D, the criterion $D_{CD}$ will be zero for every node outside A's subtree, although C and D are not siblings or parent–child. Running Algorithm 2 on this tree should therefore place C and D in the same bottom set; if it does, the central recovery claim is contradicted.","tokens_in":18546,"feed_emoji":"🏥","tokens_out":9162,"duration_ms":77762,"temperature":0.7,"pith_summary":"MASH (Multi-source Automated Structured Hierarchy) is a fully automated pipeline that tackles two obstacles to multi-institution electronic health record research at once: hospitals record the same clinical idea under different codes, and many of those codes have no usable hierarchy. It aligns co-occurrence and text-based embeddings across the U.S. Department of Veterans Affairs (VA) and Mass General Brigham (MGB) systems with neural optimal transport, then trains hyperbolic embeddings and recursively recovers a latent tree, inserting abstract concepts where needed. If the method works as claimed, researchers can pool records from systems with incompatible coding practices, and previously unstructured local codes—especially the VA's extensive laboratory codes—gain an interpretable taxonomy. The paper reports that MASH reconstructs known structures such as PheCode and RxNorm, integrates local lab codes with LOINC-mapped concepts, and outperforms baselines on embedding and hierarchy-quality metrics.","feed_headline":"MASH harmonizes EHR codes across hospitals and builds hierarchies","feed_subtitle":"Maps VA and MGB codes into one space and infers taxonomies, including unstructured VA lab codes.","key_machinery":"The carrying mechanism is the recursive grouping step built on additivity of distances along tree paths. Given a node set $V$, the paper computes $D_{ij} = \\max_{k\\in V\\setminus\\{i,j\\}} (d(z_i,z_k)-d(z_j,z_k)) - \\min_{k\\in V\\setminus\\{i,j\\}} (d(z_i,z_k)-d(z_j,z_k))$, and treats $D_{ij}=0$ as the test for whether $i$ and $j$ belong to the same \"bottom set\"—nodes that are pairwise siblings or in a parent–child relation, with non-parent nodes being leaves. $k$-means on the $D$ matrix forms bottom sets, a latent node is inserted above each, distances are recomputed using the additivity relation, and the process iterates until the whole tree is built. The distances are Lorentz-model hyperbolic embeddings trained with an additivity loss, an information-preserving loss, and an InfoNCE contrastive loss, initialized from the optimal-transport-aligned Euclidean embeddings.","core_discovery":"On its own terms, the paper's central claim is that code harmonization and hierarchy construction can be solved jointly by one three-stage procedure: an optimal transport map aligns institution-specific embedding spaces, hyperbolic-space training imposes tree-like geometry with an additivity loss plus contrastive supervision, and recursive grouping recovers the latent tree from the learned distances. This yields a common representational space for codes from different health systems and hierarchical graphs for diagnosis, medication, and laboratory codes. A specific claimed outcome is the first automated hierarchy for the VA's collection of local laboratory codes, which have no standardized ontology support.","pith_inferences":["The paper leaves implicit that the design is naturally suited to federated learning settings: only summary SPPMI matrices and code descriptions need to leave each institution, never patient-level records.","The reported AUC and NMI gains are consistent with the claim, but the ablation shown means a reader should not attribute all gains to any single stage without further experiments.","A direct depth-aware audit on a known ontology—counting how often Algorithm 2 merges an uncle with a nephew—would test whether the $D_{ij}$ criterion's depth-sensitivity degrades the hierarchies in practice."],"forward_implications":["Institutions can contribute to shared analyses without manually mapping every local code to a standard ontology; aggregated co-occurrence matrices and text descriptions are the required inputs.","VA local laboratory codes, previously lacking any curated hierarchy, become navigable and can anchor cohort definitions and clinical feature selection.","Established ontologies such as PheCode and RxNorm are reproduced, and the same machinery can extend them when new codes enter the system.","The pipeline tolerates partial overlap between institutions: only a set of shared codes is needed to fit the optimal transport map."],"supporting_citations":[{"why":"Supplies the latent tree model and recursive grouping paradigm that Stage III adapts to learned embeddings.","marker":"Choi et al., 2011"},{"why":"Proves the additive information-distance property that the paper relies on in Eq. (4) and in the $D_{ij}$ criterion.","marker":"Erdős et al., 1999"},{"why":"Provides the discrete optimal-transport mapping estimation and block-coordinate descent loop used in Stage I.","marker":"Perrot et al., 2016"},{"why":"Establishes hyperbolic embeddings for representing hierarchies, the geometry used in Stage II.","marker":"Nickel and Kiela, 2017"},{"why":"Supplies the InfoNCE contrastive loss used to pull similar and related code pairs together.","marker":"Oord et al., 2018"},{"why":"Provides the CODER biomedical language model whose embeddings carry textual semantics into the pipeline.","marker":"Yuan et al., 2022"},{"why":"Defines the SPPMI co-occurrence embedding construction and the rareness thresholds used in preprocessing.","marker":"Hong et al., 2021"},{"why":"Precedent for aligning hyperbolic representations with optimal transport, informing the design of the alignment stage.","marker":"Hoyos-Idrobo, 2020"}],"fun_headline_variants":["MASH auto-builds EHR code hierarchies across hospitals","Automated mapping creates unified EHR code space","New framework harmonizes EHR codes and infers taxonomies","First automated hierarchy for unstructured VA lab codes","Joint alignment and tree-building for multi-source EHRs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire hierarchy rests on the equivalence $D_{ij}=0$ if and only if codes $i$ and $j$ lie in the same bottom set; this equivalence fails for nodes at different depths, such as an uncle and nephew, because the distance difference to outside nodes is still constant even though the two nodes do not attach at the same level.","fun_headline_variants_meta":{"raw":{"variants":["MASH auto-builds EHR code hierarchies across hospitals","Automated mapping creates unified EHR code space","New framework harmonizes EHR codes and infers taxonomies","First automated hierarchy for unstructured VA lab codes","Joint alignment and tree-building for multi-source EHRs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1289,"prompt_tokens":888,"completion_tokens":401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":504,"tokens_out":401,"duration_ms":4257,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:15:51.600771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a small known tree—say a root with two children A and B, where A has a child C and a grandchild D—and compute the hyperbolic or information distances. For the uncle C and nephew D, the criterion $D_{CD}$ will be zero for every node outside A's subtree, although C and D are not siblings or parent–child. Running Algorithm 2 on this tree should therefore place C and D in the same bottom set; if it does, the central recovery claim is contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the latent tree model and recursive grouping paradigm that Stage III adapts to learned embeddings."},{"cited_title":"Courty, R","cited_arxiv_id":null,"evidence_quote":"Provides the discrete optimal-transport mapping estimation and block-coordinate descent loop used in Stage I."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes hyperbolic embeddings for representing hierarchies, the geometry used in Stage II."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CODER biomedical language model whose embeddings carry textual semantics into the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SPPMI co-occurrence embedding construction and the rareness thresholds used in preprocessing."}],"review_version":1}