{"id":"32b7e3fe-5f4f-4df2-b026-497e6438b1a3","arxiv_id":"2501.11441","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A retrieve-identify-prompt pipeline plus prioritized depth-first search achieves state-of-the-art F-Measure on most OAEI 2024 tasks while sending only uncertain matches to an LLM.","lead":"MILA is an ontology-matching system that combines vector search with a search-tree strategy and only asks a large language model to judge borderline candidate pairs. On standard biomedical, anatomy, and biodiversity benchmarks, it reports the best F-Measure on five of seven unsupervised tasks and cuts execution time sharply.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F-measure SOTA rests on uncontrolled baseline scores: Tables 1-5 mix OAEI 2023, OAEI 2024, and literature results, so the 'best in five of seven' ranking is not yet established on a common benchmark.","rationale":"I read the paper as claiming that MILA's retrieve-identify-prompt/PDFS pipeline achieves state-of-the-art F-measure on five of seven OAEI 2024 unsupervised tasks and reduces runtime versus a retrieve-then-prompt baseline. The algorithmic construction is clear, and the released code is a positive sign. The reader's weakest assumption is the HCB shortcut: that mutual top-rank correspondences can be emitted without LLM confirmation. That concern is real in principle, but Table 9 partially answers it by reporting MILA-HCB precision of 0.911-0.988 across all tasks, so HCB false positives are unlikely to be the decisive flaw. The more load-bearing concern is that the F-measure ranking is built from a mixture of OAEI 2023, OAEI 2024, and literature-derived baseline scores, while MILA is evaluated only on OAEI 2024. Without a per-cell provenance table or re-runs on a common benchmark, the margins over competitors, especially the small 3-point margin on NCIT-DOID versus HybridOM, are not yet trustworthy. The runtime claim is better controlled: Table 8 compares MILA and the baseline with the same LLM (LLaMa-3.3-8B) on the same machine, so I do not see the runtime comparison as the central risk. The abstract/full-text inconsistencies about 'four of five' versus 'five of seven' and 'LLM requests' versus 'runtime' are editorial issues that do not change the technical concern. Overall, the CONDITIONAL verdict remains appropriate pending the benchmark provenance check.","tokens_in":22916,"tokens_out":11485,"duration_ms":128078,"concrete_test":"Produce a provenance table for every P/R/F cell in Tables 1-5: source paper, OAEI edition, task version, and reference file for each competing system. For any cell not taken from the official OAEI 2024 results, re-run that system on the exact OAEI 2024 reference alignments used for MILA (the released GitHub repository plus published system binaries or code) or remove it from the ranking. At minimum, verify whether BioGITOM has official OAEI 2024 unsupervised results and re-evaluate HybridOM and LogMapBio on OMIM-ORDO and NCIT-DOID with the same ontology files and reference mappings; if a corrected baseline F-measure exceeds MILA's point estimate or overlaps its confidence interval, the claimed ranking changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim depends on the F-measure comparisons in Tables 1-5, but those tables are not a controlled comparison. Section 5.4 states that baseline data were 'compiled from results published in the OAEI BIO-ML track (2023 and 2024 editions), as well as relevant literature,' and the cited sources indicate that several baselines (e.g., OLaLa, BERTMap, SORBETMatcher, LogMapBio) were evaluated in earlier editions or in standalone papers, not necessarily on the OAEI 2024 benchmark used for MILA. If any competing score comes from a different task version, reference alignment, or ontology release, the claimed margins (up to 17%) and the 'best in five of seven' ranking could be artifacts. This is more load-bearing than the HCB shortcut: Table 9 already reports MILA-HCB precision of 0.911-0.988 across tasks, which partially supports the shortcut, but no table gives the provenance of every baseline cell, and BioGITOM appears only in the semi-supervised tables even though it may also have OAEI 2024 unsupervised results.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MILA, a zero-shot ontology matching system that combines SBERT embedding-based retrieval with a retrieve-identify-prompt pipeline embedded in a prioritized depth-first search. The identify step emits high-confidence bidirectional correspondences without LLM confirmation, and the PDFS strategy limits LLM queries to borderline cases. The system is evaluated on seven OAEI 2024 tasks in the biomedical, anatomy, and biodiversity domains. The authors claim that MILA achieves the highest F-Measure in five of seven unsupervised tasks, outperforms state-of-the-art systems by up to 17%, and reduces runtime compared with a retrieve-then-prompt baseline, all without fine-tuning or domain-specific heuristics.","tokens_in":23192,"tokens_out":4787,"duration_ms":49361,"significance":"If the reported results hold, MILA is a valuable contribution to LLM-based ontology matching: it demonstrates that a combination of programmed search, learned embeddings, and a single LLM verification step can yield competitive or superior F-Measure while drastically reducing the number of LLM calls. The paper's strengths include the availability of code, the use of parameters (k, tau) taken from prior work rather than fitted to the test labels, evaluation against standard OAEI reference alignments, and a detailed runtime analysis. The claim of task-agnostic performance is also supported by stable results across several benchmark families. The main caveats lie in the provenance of baseline scores and the lack of statistical quantification, both of which affect the strength of the headline comparison.","major_comments":[{"comment":"The baseline performance scores are compiled from OAEI 2023 and 2024 reports and separate literature sources rather than re-run in a controlled setup on the same benchmark version. Because OAEI editions differ in ontology releases, reference alignments, and evaluation protocols, the claimed 'best in five of seven' ranking and the specific margins (e.g., 17% over LogMapBio on OMIM-ORDO) are not established on a common benchmark. The authors should either re-run the comparison systems on the exact same task versions or clearly restrict the claims to a subset with per-cell provenance (edition, ontology version, reference alignment) for every baseline score.","section":"Section 5.4, Tables 1-5"},{"comment":"The paper reports that each experiment was executed a minimum of 20 times to estimate variance, yet all F-Measure tables report only point estimates. The 'Average Variance in LLM' column in Table 7 does not quantify the uncertainty of the headline F-Measure differences, so margins of up to 17% could be within run-to-run variability. The authors should report means with standard deviations or confidence intervals, and where appropriate perform significance tests for the claimed improvements over the strongest baselines.","section":"Section 5.3 and Tables 1-5"},{"comment":"The runtime comparison between MILA and the retrieve-then-prompt baseline is not controlled. The MILA results in Tables 7 and 9 use LLaMa-3.1-70B via an inference endpoint, while the runtime comparison in Table 8 uses LLaMa-3.3-8B-Instruct, and the baseline pipeline is not described with the same algorithmic detail. Hardware, model, and serving differences confound the reported runtime reduction. The comparison should be made with the same LLM and execution environment, or the differences should be explicitly stated as limitations.","section":"Section 5.6.3 and Table 8"}],"minor_comments":[{"comment":"The abstract states 'achieved the highest F-Measure in four of the five unsupervised tasks,' while the Highlights section states 'five of the seven tasks in the unsupervised setting.' The counts should be reconciled to avoid ambiguity.","section":"Abstract vs Highlights"},{"comment":"The NCIT-DOID row reports F-Measure 0.970 for P=0.967 and R=0.928, but the harmonic mean is approximately 0.947. Please verify the reported values.","section":"Table 3"},{"comment":"The text contains a typo: 'OEAI' should be 'OAEI' in several places, including the phrase 'biomedical evaluation benchmark proposed by the OEAI in the 2024 edition'.","section":"Section 5.4"},{"comment":"The column header 'Average Variance in LLM' is unclear; it should specify what quantity is averaged (e.g., variance of F-Measure or of LLM responses) and over how many runs.","section":"Table 7"},{"comment":"Reference [41] lists 'T. Benson, G. Grieve, T. Benson, G. Grieve' as authors; the duplicated names should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The principal risk is the uncontrolled baseline compilation; if the authors can provide a controlled re-run or a per-cell provenance table for every baseline score, the paper would be much stronger. The HCB shortcut concern raised by the stress-test note is partially addressed by the MILA-HCB precision values in Table 9, which show high precision across tasks; however, the paper does not analyze failure cases of mutual top-rank correspondences, and a brief error analysis would be useful. The runtime claim also needs a fairer comparison. The core method is sound and well-described, so I would not reject, but the central SOTA claims need substantiation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering contribution with one genuinely nice idea, but the headline “best in five of seven” is not yet proven because the baseline scores come from mixed OAEI editions and standalone papers rather than a controlled re-run. I’d send it to review, but I’d want the comparison cleaned up first.\n\nWhat’s actually new: the retrieve-identify-prompt pipeline, where a mutual-top-k (HCB) gate emits high-confidence correspondences without consulting an LLM, and a prioritized depth-first search handles only the borderline cases. That combination is new in this space, and the runtime table shows it cuts LLM calls dramatically—e.g., SNOMED-FMA goes from ~47 hours to ~45 minutes. That’s a real practical win. The HCB shortcut is the load-bearing piece, and Table 9 gives some direct support: MILA-HCB precision ranges 0.911–0.988 across tasks. So the concern that mutual top-rank might flood the alignment with false positives is partially answered, though the paper never actually analyzes where those HCB false positives come from.\n\nWhat’s soft: the evaluation tables mix results from OAEI 2023, OAEI 2024, and independent literature. That means the claimed 17% margins and the “best in five of seven” ranking aren’t on a common benchmark. The stress-test note is right that this is more load-bearing than the HCB issue. Also, they run 20+ LLM executions but report F-Measure without confidence intervals, which makes the comparisons shaky. The runtime comparison uses LLaMa-3.3-8B for the baseline and MILA, but the main F-Measure runs use 70B—fine for the efficiency claim, but the two sets of numbers aren’t directly comparable. And the abstract says four of five while the highlights and full text say five of seven unsupervised tasks. That inconsistency is sloppy and needs fixing.\n\nWho this is for: anyone working on practical LLM-based ontology matching or RAG pipelines for structured data. The method is described clearly enough to reproduce, and the code is linked. It’s not a breakthrough, but it’s a credible step forward in efficiency and a reasonable zero-shot baseline against supervised systems.\n\nBottom line: the central idea holds up, the efficiency gain is real, and the HCB precision numbers give the shortcut a solid empirical basis. The uncontrolled baseline comparison and the missing confidence intervals are fixable. I’d recommend major revision—make the comparison controlled or at least transparently provenance-labeled, add intervals, and reconcile the abstract. Then it’s a page-worthy contribution.","headline":"MILA is a practical LLM-based OM system with a clever HCB gate and PDFS that cuts LLM queries, but the SOTA F-Measure claim rests on uncontrolled baseline tables and needs tightening before it can be taken at face value.","tokens_in":23698,"tokens_out":1508,"would_cite":true,"duration_ms":18617,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents MILA, a zero-shot matcher that wins five of seven unsupervised ontology-matching tasks by reserving large-language-model queries for borderline pairs.","keywords":["ontology matching","large language models","retrieval-augmented generation","prioritized depth-first search","bidirectional correspondence","zero-shot learning","SBERT embeddings","biomedical ontologies"],"falsifier":"Run the paper's retrieval step on the reference alignments and measure the precision of the HCB pairs alone: compute, for every pair of entities that are each other's top-ranked candidates, how many appear in the gold-standard equivalence alignments. If that precision is far below the reported system precision (0.93–0.98), the HCB shortcut is the source of false positives; if it is close to 1, the shortcut is sound and the claimed gains are real.","tokens_in":22746,"feed_emoji":"🔗","tokens_out":8003,"duration_ms":77130,"temperature":0.7,"pith_summary":"Ontology matching—deciding that two concepts in different ontologies are the same—is expensive to do with large language models because checking every candidate pair costs time and money. This paper claims that most correct matches can be decided by a fast embedding-based retrieval step, with the language model consulted only on uncertain pairs. The strategy, called MILA, achieved the best F-measure in five of the seven unsupervised tasks in the 2024 international ontology-alignment evaluation campaign, outperforming state-of-the-art systems by up to 17 percentage points while using far fewer LLM requests than a plain retrieve-then-prompt pipeline. The system needs no fine-tuning and no domain-specific heuristics.","feed_headline":"Five of seven ontology tasks won with fewer LLM calls","feed_subtitle":"MILA pairs SBERT retrieval with targeted prompting, beating fine-tuned models by up to 17 percent.","key_machinery":"The load-bearing object is the high-confidence bidirectional (HCB) correspondence: a source entity and a target entity that are the top-ranked candidates in each other's embedding-similarity candidate lists, which MILA treats as a confirmed equivalence without any LLM call. Around it, the paper builds a prioritized depth-first search (PDFS) over each entity's ranked candidate list, querying the LLM only when a pair is bidirectional but not mutual-top-rank and stopping as soon as the LLM answers 'Yes'. A cosine-similarity retrieval step over entity labels using an SBERT embedding model supplies the ranked lists, and a threshold ($\\tau = 0.75$) with a fixed top-$k=5$ bounds the candidate sets.","core_discovery":"In its own terms, MILA claims that ontology matching can be solved by a retrieve-identify-prompt pipeline: a fast embedding retriever proposes ranked candidate entities, an identify stage recognises when two entities mutually rank each other first—the high-confidence bidirectional (HCB) correspondence—and emits that equivalence directly, and a prioritized depth-first search prompts a large language model with a binary equivalence question only for the remaining bidirectional candidates, stopping at the first confirmation. The claim is that this zero-shot combination reaches the best F-measure in five of the seven unsupervised tasks of the 2024 international ontology-matching evaluation campaign, beats the strongest published systems by up to 17 percentage points, and with a smaller model cuts a 47-hour retrieve-then-prompt run to about 45 minutes on the largest task, all without fine-tuning, domain-specific heuristics, or post-processing.","pith_inferences":["The HCB shortcut is a form of symmetric nearest-neighbor agreement; in domains with many near-duplicates or homonyms, mutual top-rank could be far less reliable, so transferring the approach beyond terminology-dense biomedical ontologies is an open test.","The reported speedup depends on retrieval quality: on tasks where the true match never enters the top candidate list, MILA inherits the retriever's recall ceiling and cannot recover matches the retriever never proposed.","A concrete extension would be to run the retrieval step with varying thresholds and top-k values and measure whether HCB precision changes; if HCB precision depends strongly on k, the shortcut is a tuning artifact rather than a stable property of the embedding space."],"forward_implications":["LLM-based ontology matching cost can be cut by an order of magnitude or more by outsourcing confident matches to retrieval and using the LLM only for borderline candidates.","A zero-shot matcher with no domain-specific heuristics can reach or exceed the F-measure of fine-tuned biomedical matchers on most tasks, including a 17-point lead on OMIM-ORDO and SNOMED-NCIT Neoplasm.","The quality of the retriever is decisive: when the embedding model ranks the correct candidate first, HCB matches short-circuit the pipeline; when it does not, LLM queries multiply and runtime grows, as observed on the pharmacology task.","The same architecture yields stable results across biomedical, anatomy, and biodiversity tasks, indicating that the approach is task-agnostic.","Removing the LLM entirely (the MILA-HCB configuration) still achieves precision near 0.99 on some tasks, so the LLM's marginal role is mainly to raise recall on hard cases."],"supporting_citations":[{"why":"LLMs4OM: the retrieve-then-prompt baseline that MILA extends and compares against; supplied the top-k=5 setting and the observed recall drop after prompting.","marker":"[23]"},{"why":"OLaLa: introduced the LLM-based retrieve-then-prompt refinement used here; its ablation guided the choice of the SBERT model and the prompt design.","marker":"[20]"},{"why":"Sentence-BERT: the embedding model that generates the label vectors and ranked candidate lists underlying the HCB shortcut.","marker":"[24]"},{"why":"The OAEI proceedings: source of the 2024 evaluation benchmark, tasks, and the competitor results that define the paper's comparisons.","marker":"[10]"},{"why":"Machine-learning-friendly biomedical datasets: provides the reference alignments and the task-difficulty analysis, such as OMIM-ORDO being the hardest task.","marker":"[15]"},{"why":"BERTMap: a strong comparison system based on BERT and fine-tuning, and the source of the locality principle discussed in mapping refinement.","marker":"[17]"},{"why":"LogMap: a comparison baseline and the origin of the locality principle and logic-based mapping repair ideas referenced throughout.","marker":"[12]"},{"why":"The Llama 3 herd of models: the large language model used for the prompting stage in the reported experiments.","marker":"[44]"}],"fun_headline_variants":["MILA: five of seven ontology tasks won with fewer LLM calls","Zero-shot MILA tops 5/7 ontology benchmarks, cuts LLM prompts","Prioritized DFS plus LLM wins five of seven ontology tasks","MILA: efficient LLM ontology matching, beats SOTA by 17%","Retrieve-identify-prompt: MILA slashes LLM use, wins big"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that two entities that rank each other first in the embedding-similarity lists are in fact equivalent; if mutually top-ranked pairs can be not equivalent, those false matches enter the final alignment without any LLM check and the reported precision would degrade.","fun_headline_variants_meta":{"raw":{"variants":["MILA: five of seven ontology tasks won with fewer LLM calls","Zero-shot MILA tops 5/7 ontology benchmarks, cuts LLM prompts","Prioritized DFS plus LLM wins five of seven ontology tasks","MILA: efficient LLM ontology matching, beats SOTA by 17%","Retrieve-identify-prompt: MILA slashes LLM use, wins big"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1915,"prompt_tokens":1005,"completion_tokens":910,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":805}},"tokens_in":621,"tokens_out":910,"duration_ms":8887,"temperature":1.0,"reasoning_tokens":805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:15:11.765997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's retrieval step on the reference alignments and measure the precision of the HCB pairs alone: compute, for every pair of entities that are each other's top-ranked candidates, how many appear in the gold-standard equivalence alignments. If that precision is far below the reported system precision (0.93–0.98), the HCB shortcut is the source of false positives; if it is close to 1, the shortcut is sound and the claimed gains are real.","supporting_citations":[{"cited_title":"Hertling, H","cited_arxiv_id":null,"evidence_quote":"OLaLa: introduced the LLM-based retrieve-then-prompt refinement used here; its ablation guided the choice of the SBERT model and the prompt design."},{"cited_title":"Jim ´enez-Ruiz, O","cited_arxiv_id":null,"evidence_quote":"The OAEI proceedings: source of the 2024 evaluation benchmark, tasks, and the competitor results that define the paper's comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Machine-learning-friendly biomedical datasets: provides the reference alignments and the task-difficulty analysis, such as OMIM-ORDO being the hardest task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BERTMap: a strong comparison system based on BERT and fine-tuning, and the source of the locality principle discussed in mapping refinement."},{"cited_title":"Jim ´enez-Ruiz, B","cited_arxiv_id":null,"evidence_quote":"LogMap: a comparison baseline and the origin of the locality principle and logic-based mapping repair ideas referenced throughout."}],"review_version":1}