{"id":"2bbc2e7e-57e7-47d9-aebe-bf6a12c5267f","arxiv_id":"2507.03761","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fusing BM25 and BERT rankings with CombMNZ and ZMUV normalization gave the best reported nDCG and precision on four XMTC datasets, though no single-retriever baselines or significance results are reported.","lead":"Extreme multi-label text classification assigns many possible labels to each text, and the paper tests 60 combinations of score-normalization and fusion methods for blending sparse (BM25) and dense (BERT) label retrievers on four benchmarks. The authors report that a particular combination, CombMNZ with ZMUV normalization, works best, but the paper does not compare against the individual retrievers it fuses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fusion claim rests on missing single-retriever baselines and contradicts its own tables; 'highest effectiveness' has no defined criterion.","rationale":"The paper's central claim is empirical and load-bearing, but the evidence provided does not establish it. The most direct problem is the absence of single-retriever baselines: the introduction and abstract promise that fusing sparse and dense rankings improves label ranking by exploiting complementarity, yet no table reports the effectiveness of BM25 alone or BERT alone. If the best fused configuration is not better than one of its inputs, the claimed contribution collapses. A second, independent problem is that the paper's own numbers do not support the sharp conclusion that CombMNZ+ZMUV is best everywhere: on Eurlex-4k and Wiki10-31k, ZMUV+CombSUM wins several nDCG@1 cells. Without a stated criterion for aggregating across four datasets, twelve metrics, and head/tail partitions, 'highest effectiveness' is undefined. The paper also promises paired t-tests but never presents significance results, and the anomalous Borda Norm rows (identical values for four different algorithms) further indicate that the reported pipeline has an unverified component. I agree with the reader's rejection in substance: the empirical question is legitimate, but the paper is not currently a reliable benchmark. The concrete test I propose would settle the central claim by providing the missing baselines, measuring the asserted complementarity, and forcing a predefined and reproducible winner-selection rule.","tokens_in":33280,"tokens_out":5582,"duration_ms":62639,"concrete_test":"Run the same five folds and report for each dataset: (1) BM25-only and BERT-only nDCG@k and P@k; (2) Kendall tau between the two retrievers' 128-candidate rankings; (3) a pre-registered selection rule (e.g., mean over the 24 metric cells, or paired t-test with correction) applied to all fusion/normalization combinations. If a single retriever matches or beats ZMUV+CombMNZ, or if ZMUV+CombSUM is not significantly worse, the Section 6 claim should be revised; the same run also resolves whether the claimed complementarity exists.","verdict_should_be":"REJECT","load_bearing_attack":"The Section 6 conclusion cannot be verified from the paper. First, no effectiveness numbers are reported for BM25-only or BERT-only rankings, so the paper never tests the premise (Sec. 1) that sparse and dense rankings are complementary and that fusion improves label ranking. All tables show only fused outputs; if either input retriever already outperforms ZMUV+CombMNZ, the claimed benefit disappears. Second, the paper's own tables contradict the 'highest effectiveness' claim under any simple reading: on Eurlex-4k (Table 2), ZMUV+CombSUM beats ZMUV+CombMNZ on tail and head nDCG@1 (51.8 vs 51.5; 82.3 vs 81.8); on Wiki10-31k (Table 3) the same pattern appears (49.5 vs 49.1; 81.8 vs 81.2). The paper never states the aggregation rule over the 24 metric cells, and the t-tests promised in Section 4 are neither reported nor summarized. The missing code or repository link makes these anomalies uncheckable. Consequently, the empirical basis for the central claim is absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies ranking-based fusion algorithms for extreme multi-label text classification (XMTC), combining a BM25 sparse retriever with a fine-tuned BERT dense retriever. Six normalization strategies and ten fusion algorithms are evaluated on four XMTC datasets under head/tail label splits, with nDCG@k and Precision@k as metrics. The central claim, stated in Section 6, is that CombMNZ with ZMUV normalization yields the highest effectiveness across all datasets. The manuscript also asserts that sparse and dense rankings are complementary and that fusion improves label ranking, and it announces statistical significance testing and public code/data release.","tokens_in":33482,"tokens_out":3550,"duration_ms":39726,"significance":"If the central claim were supported, the paper would offer a simple, architecture-agnostic recipe for improving XMTC ranking by fusing BM25 and BERT outputs, which could be of practical value. The evaluation grid is broad (six normalizations, ten fusion algorithms, four datasets, head/tail breakdowns) and the focus on tail labels is relevant to XMTC. However, the paper currently provides no verifiable evidence for its main conclusion: the own tables contradict the 'highest across all datasets' statement, no single-retriever baselines are reported, and the promised significance tests and reproducibility artifacts are absent. The contribution is therefore not established in its present form.","major_comments":[{"comment":"The conclusion that ZMUV+CombMNZ yielded the highest effectiveness across all datasets is contradicted by the paper's own tables. On Eurlex-4k (Table 2), ZMUV+CombSUM achieves tail nDCG@1 of 51.8 vs. 51.5 for ZMUV+CombMNZ and head nDCG@1 of 82.3 vs. 81.8. On Wiki10-31k (Table 3), ZMUV+CombSUM again leads on tail nDCG@1 (49.5 vs. 49.1) and head nDCG@1 (81.8 vs. 81.2). On Amazon-670k (Table 4), ZMUV+CombSUM beats CombMNZ on tail nDCG@1 (48.0 vs. 47.3). The paper never specifies the aggregation rule over the 24 metric cells (2 metrics x 3 cutoffs x head/tail x 4 datasets), so 'highest effectiveness' is undefined. The conclusion must be revised or justified with an explicit ranking criterion.","section":"Section 6, Tables 2-5"},{"comment":"The paper claims that fusion of sparse and dense retrievers improves effectiveness and asserts complementarity (Section 1), but no effectiveness numbers are reported for BM25-only or BERT-only rankings. Every table shows only fused outputs, so the reader cannot determine whether fusion helps at all, nor whether the two retrievers are complementary. Without these single-retriever baselines, the central premise of the paper is untested. The authors should report per-retriever performance and, ideally, a quantitative complementarity measure (e.g., rank correlation between the two input rankings).","section":"Sections 4 and 5"},{"comment":"The text states that 'we assess the statistical significance of our results by employing the two-sided paired Student's t-test with 95% confidence' and that results are 'averaged across the five test splits', but no t-test outcomes, p-values, or significance markers appear in Tables 2-5. The reported standard deviations alone do not substantiate the significance claim, especially when differences as small as 0.1-0.3 percentage points are discussed. The authors must either report the significance results or remove the claim.","section":"Section 4"},{"comment":"The reproducibility statement says 'we make our source code, developed models, and data publicly available', but no repository URL, DOI, or access instructions are provided anywhere in the manuscript. This makes the empirical results unverifiable, particularly the anomalous 1.3-1.7 nDCG values for Borda Norm with ISR/Log_ISR/BordaFuse/Condorcet in Tables 2-5, which suggest a potential implementation issue (e.g., ties or zero normalization) that cannot be checked without code.","section":"Section 4"}],"minor_comments":[{"comment":"The caption of Table 5 reads 'Amazon-13k' while Section 4 and Table 1 refer to 'AmazonCat-13K'; the naming should be made consistent.","section":"Table 5 caption"},{"comment":"The abstract and introduction use 'proven effective' for sparse-dense fusion in XMTC, but the paper itself does not compare against the individual retrievers; this overstates what is demonstrated.","section":"Abstract and Introduction"},{"comment":"The definition of ZMUV Norm states it transforms scores 'into a standard normal distribution'; since the transformation is applied per ranked list, it would be more precise to say it standardizes the scores to zero mean and unit variance for each retrieval run, not that the scores follow a normal distribution.","section":"Section 2, ZMUV Norm"},{"comment":"Several references are incomplete or use placeholder formatting (e.g., the ACM template placeholders for the conference acronym and DOI), and reference [17] (Wei et al., WSDM 2022) appears unrelated to the cyclical learning rate citation in Section 4; this should be checked.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript resembles an unfinished short paper rather than a complete archival submission: the central conclusion is directly contradicted by three of the four result tables, the promised significance analysis and reproducibility artifacts are missing, and the headline fusion benefit is not testable without single-retriever baselines. The authors can address these issues by adding the missing baselines, defining and applying a clear comparison protocol, reporting significance, and revising the conclusion to match the data. If they do not, the paper should not be accepted. The scope is appropriate for a workshop or a short empirical track, but the current level of rigor falls below the journal standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper is a benchmark of standard rank-fusion algorithms on top of BM25 and BERT rankings for XMTC, and as a survey-by-benchmark it has decent coverage: six normalizations, ten fusion algorithms, four datasets, 5-fold CV, standard deviations. That part is honest work. But the paper's own conclusion, that CombMNZ with ZMUV normalization is best, is not supported by the numbers in its tables, and the missing single-retriever baselines mean the core premise that fusion helps is never actually tested.\n\nWhat is new: applying these well-known metasearch techniques to XMTC with a dense retriever is a routine extension, not an innovation. The specific comparison matrix is not in the literature, but the components are all from Fox and Shaw, Aslam and Montague, and Montague and Aslam. So novelty is low, but the empirical question is sensible.\n\nThe soft spots are serious. First, no BM25-only or BERT-only results appear anywhere. The reader cannot tell whether fusion improves on the better retriever alone; the complementarity claim in Section 1 is asserted, never measured. Second, the Section 6 statement is contradicted by the tables: on Eurlex-4k, ZMUV CombSUM has tail nDCG@1 51.8 versus 51.5 for CombMNZ, and head 82.3 versus 81.8; on Wiki10-31k, CombSUM again leads tail nDCG@1 49.5 versus 49.1. So 'highest effectiveness across all datasets' is not true on its face. Third, the t-tests promised in Section 4 are missing. Fourth, the Borda Norm rows are identical across multiple algorithms, including ISR, Log_ISR, BordaFuse, and Condorcet, which strongly suggests a bug in the fusion code or normalization. The paper says source code is available but provides no link or repository.\n\nWho should read this? Practitioners working on XMTC who want a cheap way to boost ranking quality by fusing a sparse and a dense retriever. The question is worth answering, and a revised version with baselines, corrected claims, significance tests, and code could be a useful benchmark. As submitted, I would not send it to peer review; the evidence needed to support the central claim is absent, and the apparent table bug makes the results uncheckable. I'd tell the authors to fix these issues before resubmitting.","headline":"A wide but flawed fusion benchmark for XMTC: the central claim is contradicted by the paper's own tables, and missing single-retriever baselines make the core premise untestable.","tokens_in":34108,"tokens_out":3896,"would_cite":false,"duration_ms":40541,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ranking fusion with ZMUV normalization and CombMNZ is the best-performing combination for fusing sparse and dense retrievers in extreme multi-label text classification.","keywords":["Extreme Multi-label Text Classification","ranking fusion","ranking normalization","CombMNZ","ZMUV","BM25","BERT","long-tail labels"],"falsifier":"Take any one of the four datasets and compute the overlap (or Kendall tau) between the BM25 and BERT candidate rankings; if the top-10 lists are nearly identical, then ZMUV+CombMNZ cannot beat either retriever alone, and the claimed fusion advantage would disappear.","tokens_in":33050,"feed_emoji":"🏷️","tokens_out":11063,"duration_ms":109454,"temperature":0.7,"pith_summary":"Extreme multi-label text classification asks a system to rank each document against a huge label vocabulary, where most labels are rare. This paper asks whether fusing the rankings of two very different retrievers—BM25, which matches words exactly, and a fine-tuned BERT model, which matches by semantic similarity—can improve that ranking, and which fusion recipe does it best. After testing six ways of normalizing scores and ten fusion algorithms on four benchmark datasets, it claims the winner is ZMUV normalization (converting each retriever's scores to zero mean and unit variance) followed by CombMNZ, which sums normalized scores and boosts labels that both retrievers support. The reported result is that this combination gives the highest ranking-quality scores at positions 1, 5 and 10, for both head and tail labels on every dataset. If correct, it is a drop-in improvement: no retraining or architecture change, just a pre-processing step on the retrievers' outputs.","feed_headline":"Fusing BM25 and BERT scores one way wins on all four tests","feed_subtitle":"Standardizing BM25 and BERT scores before merging lifts rare- and common-label ranking quality.","key_machinery":"The load-bearing mechanism is the pair (ZMUV, CombMNZ). ZMUV normalization takes each retriever's raw scores, subtracts the mean and divides by the standard deviation, putting BM25's unbounded scores and BERT's similarity scores on one comparable scale. CombMNZ then computes, for each label, the sum of its normalized scores and multiplies by the number of retrievers with a non-zero score for that label, so a label supported by both retrievers is amplified while a label seen by only one is discounted. The paper's experiments vary six normalizations (Min-Max, Max, Sum, ZMUV, Rank, Borda) and ten fusion algorithms (CombMIN, CombMAX, CombMED, CombSUM, CombANZ, CombMNZ, ISR, Log-ISR, BordaFuse, Condorcet), and this pair is what carries the claimed result.","core_discovery":"On the paper's own terms, the discovery is that fusing dense and sparse rankings for XMTC works best when the two score lists are first standardized to zero mean and unit variance and then merged with CombMNZ: for each candidate label the normalized scores from BM25 and BERT are summed, and that sum is multiplied by the number of retrievers that gave the label a non-zero score. This consensus-boosting step is what yields the highest nDCG@k and Precision@k on Eurlex-4K, Wiki10-31K, Amazon-670K and AmazonCat-13K, for both the 80% least frequent tail labels and the 20% most frequent head labels, with $k \\in \\{1,5,10\\}$. The claim is explicitly a comparative one: of the 60 normalization-plus-fusion combinations tested, ZMUV+CombMNZ is the top performer on all four datasets.","pith_inferences":["Editorial inference: ZMUV probably wins because it puts BM25's unbounded positive scores and BERT's similarity scores on a common scale; if that is the mechanism, other distribution-equalizing normalizations (e.g., quantile or rank-based z-scores) should produce similar gains.","Editorial inference: the head-tail split is fixed by a Pareto 80/20 rule; the reported tail improvements might shift if the threshold moved, so the recommendation's robustness to that threshold is a testable open question.","Editorial inference: because the recipe only consumes normalized score lists, it should transfer to other retriever pairs and to ranking tasks beyond XMTC; a direct test on those settings would show whether the mechanism is general."],"forward_implications":["A system that already has BM25 and BERT candidate lists can adopt ZMUV+CombMNZ as a drop-in post-processing step and, according to the paper, obtain the best measured nDCG@k and Precision@k among the 60 compared combinations.","The winning combination improves both tail and head labels together, so there is no measured tradeoff between rare-label coverage and common-label accuracy.","The recommendation generalizes across dataset scales, from Eurlex-4K with about 4K labels to Amazon-670K with over 670K labels, and across document counts from tens of thousands to millions.","Normalization choice is a first-order decision: the same fusion algorithm can differ by several points in nDCG depending on which normalization it is paired with."],"supporting_citations":[{"why":"Supplies the BM25 scoring model with b=0.75, k=1.5, the sparse retriever whose rankings are fused.","marker":"[1]"},{"why":"Establishes the metasearch framework that motivates combining rankings from multiple retrieval systems.","marker":"[2]"},{"why":"Provides the normalization and fusion implementations that the experiments run.","marker":"[3]"},{"why":"Supplies the metasearch library implementation used for the normalization and fusion code in the evaluation.","marker":"[4]"},{"why":"Original source of the CombSUM/CombMNZ score-based fusion family that carries the winning algorithm.","marker":"[9]"},{"why":"Defines the Normalized Temperature-Scaled Cross Entropy loss used to fine-tune BERT into the shared text-label embedding space of the dense retriever.","marker":"[12]"},{"why":"Supplies the Condorcet voting baseline that the paper compares against in the fusion experiments.","marker":"[13]"}],"fun_headline_variants":["ZMUV + CombMNZ fusion tops XMTC benchmarks","Standardized score fusion wins XMTC head and tail","BM25+BERT with ZMUV and CombMNZ best on 4 sets","Best XMTC fusion: standardize then CombMNZ","Normalize BM25 and BERT, then combine: XMTC winner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that BM25 and BERT produce complementary label rankings; the paper asserts this but never measures it, and if the two rankers largely agree, fusion adds nothing and the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["ZMUV + CombMNZ fusion tops XMTC benchmarks","Standardized score fusion wins XMTC head and tail","BM25+BERT with ZMUV and CombMNZ best on 4 sets","Best XMTC fusion: standardize then CombMNZ","Normalize BM25 and BERT, then combine: XMTC winner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000422,"raw_usage":{"total_tokens":2163,"prompt_tokens":936,"completion_tokens":1227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1134}},"tokens_in":552,"tokens_out":1227,"duration_ms":9797,"temperature":1.0,"reasoning_tokens":1134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:03:03.298799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any one of the four datasets and compute the overlap (or Kendall tau) between the BM25 and BERT candidate rankings; if the top-10 lists are nearly identical, then ZMUV+CombMNZ cannot beat either retriever alone, and the claimed fusion advantage would disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BM25 scoring model with b=0.75, k=1.5, the sparse retriever whose rankings are fused."},{"cited_title":"Aslam and Mark H","cited_arxiv_id":null,"evidence_quote":"Establishes the metasearch framework that motivates combining rankings from multiple retrieval systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the normalization and fusion implementations that the experiments run."},{"cited_title":"Fox and Joseph A","cited_arxiv_id":null,"evidence_quote":"Original source of the CombSUM/CombMNZ score-based fusion family that carries the winning algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Normalized Temperature-Scaled Cross Entropy loss used to fine-tune BERT into the shared text-label embedding space of the dense retriever."},{"cited_title":"Montague and Javed A","cited_arxiv_id":null,"evidence_quote":"Supplies the Condorcet voting baseline that the paper compares against in the fusion experiments."}],"review_version":1}