{"id":"528d091a-2623-4b39-8491-d8d98a444b57","arxiv_id":"2411.08309","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"CMiNet is an R package that builds a consensus microbiome network from ten inference algorithms, with no quantitative evidence that the consensus is more stable or accurate.","lead":"This paper introduces CMiNet, an R package that merges ten microbiome network inference algorithms into a single weighted consensus network. A generalist might read it to see whether consensus aggregation can make microbial interaction networks more trustworthy, but the paper provides no test of that claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consensus reliability is asserted without any ground-truth comparison; if algorithms share biases, agreement amplifies rather than cancels error, so the claim of a 'more stable and comprehensive' network is unsupported.","rationale":"The reader's weakest assumption is that agreement among multiple methods is a valid proxy for accuracy, which is exactly the load-bearing weakness I identify. The paper's entire justification for the consensus claim is the statement that, because no gold standard exists, combining different algorithms reduces algorithm-dependent bias and produces a more robust network. But agreement is only a valid proxy if the methods' errors are independent. The paper provides no evidence of such independence; in fact, correlation-based methods (Pearson, Spearman, Bicor, SparCC) share similar mathematical foundations and compositional datasets often induce common artifacts (e.g., closure, shared environmental drivers), so their errors can be positively correlated. In that case, the consensus simply amplifies the shared spurious signals. The only way to settle this is to evaluate the consensus against a known ground truth, which is possible through simulation. The paper does not do this, nor does it compare consensus stability across resampled data against single-method stability. Thus the central claim is not merely under-supported; it relies on an assumption that is known to be fragile. I agree with the reader's rejection. However, I do not think the paper is fraudulent or worthless: the package may be useful software, and the simulation test I propose could rescue the claim if the consensus performs well. But as submitted, the core claim is unvalidated, so the verdict of REJECT with moderate confidence remains appropriate. I therefore recommend no change to the reader's verdict. The concrete test is a single, feasible computational experiment that directly measures accuracy against known interactions, and a biased-scenario variant that directly challenges the 'shared biases' objection. If the consensus passes both, the claim would be substantially strengthened; if it fails, the paper would need to substantially revise its claims, perhaps rebranding CMiNet as a stability-exploration tool rather than a more reliable inference method.","tokens_in":4302,"tokens_out":3104,"duration_ms":33991,"concrete_test":"Simulate synthetic microbial communities with known interaction networks (e.g., generate OTU counts from a multivariate normal or logistic-normal model with a known sparse precision matrix). Run CMiNet and each of the ten individual methods on 100 simulated datasets. Compute edge-recovery AUPRC (or F1) for the consensus and each method. Then introduce a shared confounder (e.g., a common environmental gradient affecting all taxa) and repeat. If the consensus does not match or beat the best individual method in the unbiased scenario, or if its AUPRC drops below that of methods that directly adjust for compositionality in the biased scenario, the claim that consensus is more reliable fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract: 'more stable and comprehensive representation'; Conclusion: 'more robust and reliable network inference') rests on the assumption that agreement among methods implies accuracy. The paper explicitly acknowledges 'the lack of a gold standard' (Example Application section) and uses this absence to justify consensus. However, no quantitative evaluation is provided: there is no synthetic benchmark with known interactions, no comparison of consensus edge recovery against individual methods, and no stability analysis across data subsamples. The only numerical outputs are node and edge counts at thresholds (Figure 1b). This is insufficient because the consensus is essentially a majority vote on edge support: if several algorithms share a common bias—for example, correlation-based methods all detecting spurious co-occurrence due to a shared environmental driver or compositional closure—those spurious edges will receive high consensus scores and be 'confirmed' by the majority, producing a network that is confidently wrong. The assumption that algorithm diversity cancels biases is plausible but not demonstrated; it fails in known scenarios where methods are not independent (e.g., multiple methods rely on the same compositional normalization or on similar correlation heuristics). Without a ground-truth test, the manuscript cannot support the claim that consensus improves reliability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents CMiNet, an R package that constructs a consensus microbiome network by integrating outputs from ten network inference methods (Pearson, Spearman, Bicor, SparCC, SpiecEasi-MB, SpiecEasi-glassso, SPRING, GCoDA, CCLasso, and a novel CMIMN). The package provides functions for network construction, threshold-based visualization, and Hamming distance comparison. The central claim is that the consensus network is more stable and comprehensive than any single method, and that it ensures more robust and reliable inference. The paper includes an example application with node/edge counts at thresholds, but no quantitative evaluation of accuracy.","tokens_in":4550,"tokens_out":5325,"duration_ms":48201,"significance":"If the consensus approach were validated, the package would be a convenient and potentially useful tool for microbiome researchers, bundling diverse methods in one interface with customisable thresholds. The strengths are open availability, broad coverage of methods, and a clear software structure with documented functions. However, the paper's advertised advantages (stability, reliability, comprehensiveness) are asserted rather than demonstrated. There is no simulation with known ground truth, no comparison against individual methods, no uncertainty quantification, and no description of the example dataset. As presented, the evidence is entirely internal to the method, making the main claims unsupported. A validated version of the package would be a solid contribution to applied microbiome research.","major_comments":[{"comment":"The central claim, that CMiNet provides 'a more stable and comprehensive representation' (Abstract) and 'ensures more robust and reliable network inference' (Example Application), is not supported by any quantitative evaluation. No simulation with known ground truth is performed, no comparison of the consensus network to individual algorithms is reported, and no stability analysis across data subsamples is provided. The only numerical results are node and edge counts at thresholds in Figure 1b, which are descriptive and do not measure accuracy or robustness.","section":"Abstract; Example Application"},{"comment":"The manuscript uses the absence of a gold standard ('the lack of a gold standard in microbiome research', Example Application) to justify the consensus approach, but it never demonstrates that agreement among algorithms is a valid proxy for accuracy. Many of the included methods share underlying assumptions (e.g., compositional log-ratio transformations or correlation heuristics), so biases may be common rather than independent; a majority vote could amplify shared false edges. A synthetic benchmark with known interactions is required to test whether consensus improves edge recovery relative to individual methods.","section":"Example Application; Conclusion"},{"comment":"CMIMN, described as a novel algorithm, is never formally specified. There is no equation or algorithmic description for the conditional mutual information estimator, and the thresholds q1 and q2 are listed in Table 1 without any explanation of how they are chosen or what they control. The only reference for CMIMN is an in-preparation manuscript by the same authors [10], so the method cannot be independently evaluated or reproduced from this paper.","section":"Introduction; Table 1"},{"comment":"The dataset used to generate Figure 1b is not described: the text does not state the data source, sample size, type of sequencing or count data, preprocessing, or number of taxa. Without this context, the reported node/edge counts at thresholds 8 and 9 cannot be interpreted or reproduced, and the example does not illustrate the method's utility in a concrete setting.","section":"Example Application; Figure 1"}],"minor_comments":[{"comment":"The abstract says CMiNet incorporates 'nine widely used algorithms' but lists eight before introducing CMIMN; the Introduction lists nine widely recognized methods plus CMIMN, for a total of ten. Please make the counts consistent.","section":"Abstract"},{"comment":"The CMIMN bullet contains the typo 'cotructing'; it should be 'constructing'.","section":"Introduction"},{"comment":"The terms 'quantitative' and 'non-quantitative' are not defined. Clarify whether 'non-quantitative' means relative abundance/compositional data or something else.","section":"Table 1"},{"comment":"The algorithm labels 'SE_mb' and 'SE_glasso' in Table 1 do not match the names 'SpiecEas_MB' and 'SpiecEasi_Glasso' used in the Introduction; use consistent spelling throughout.","section":"Table 1"},{"comment":"The statement that CMiNet 'can serve as a benchmark for comparing newly developed algorithms' is an overclaim in the absence of validation; consider rewording to say the package provides a common reference point.","section":"Conclusion"},{"comment":"The choice of threshold (e.g., 9 vs. 8) is presented as a user preference without any guidance on how to select a threshold based on network properties; the package would benefit from a built-in diagnostic.","section":"Example Application"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an application note for an R package, but its central claim goes beyond describing software: it asserts improved reliability of consensus inference. The lack of external validation and the reliance on an in-preparation paper for CMIMN are, in my view, the main obstacles. If the journal regularly publishes software papers without validation, this may still be acceptable as a short communication, but the abstract's claims should be substantially softened until evidence is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: CMiNet is a useful R wrapper around existing microbiome network inference methods, with a simple majority-vote consensus and some convenient visualization. The package itself is probably fine. The paper, however, does not back its central claim that consensus gives \"more stable and comprehensive\" networks: there is no ground-truth benchmark, no comparison to individual methods, no stability analysis, and the one allegedly new algorithm CMIMN is described only by reference to an unpublished manuscript.\n\nWhat is genuinely new: the package integrates nine published algorithms plus one homegrown method into a single workflow with thresholding, weighted edges, and plotting utilities. That integration is a real convenience for practitioners. The code is on GitHub, and the authors note someone independently installed and ran it. The thresholding idea (keep edges confirmed by k of 10 methods) is simple but transparent.\n\nThe soft spots are in the claims. A majority vote over correlated methods can amplify shared biases — if several algorithms are sensitive to the same compositional closure artifact, those spurious edges get high consensus weights. The paper's only quantitative output is node/edge counts at two thresholds on an unspecified dataset. There is no simulation with known truth, no subsampling stability check, and no comparison to any single method's output. The authors acknowledge the absence of a gold standard and then lean on agreement as a proxy for reliability; that is exactly the step that needs external validation. In addition, CMIMN (the only new algorithm) is specified only by two parameters q1 and q2 and a citation to the authors' in-preparation work; a reader cannot evaluate whether it does anything beyond existing conditional-mutual-information methods.\n\nRecommendation: this is a solid software project that got oversold in the writing. It could be acceptable as a software/application note if the claims are scaled back to \"provides a convenient consensus tool,\" and if the example is replaced with or supplemented by a synthetic benchmark. As it stands, I would not accept it as a research paper, but I would send it to review with the expectation that the authors either provide validation or cut the reliability claims. If you use microbiome network tools, the package is worth knowing about, but don't cite it as evidence that consensus improves accuracy.","headline":"A convenient R wrapper with a simple majority-vote consensus, but the paper's central reliability claim is unsupported by any ground-truth validation or comparison to individual methods.","tokens_in":5003,"tokens_out":2292,"would_cite":false,"duration_ms":23766,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ten microbiome algorithms merged into one consensus network","keywords":["microbiome network","consensus network","network inference","R package","compositional data","conditional mutual information","microbial interactions"],"falsifier":"Construct a synthetic microbial community with known interaction structure, simulate compositional count data from it, run CMiNet, and compare the consensus network with each individual method's output; if a single method recovers the true edges more accurately than the consensus, or if a consensus built from deliberately biased but mutually agreeing methods still looks confident, the claim that consensus implies reliability would be refuted.","tokens_in":4130,"feed_emoji":"🦠","tokens_out":4374,"duration_ms":41803,"temperature":0.7,"pith_summary":"The paper presents CMiNet, an R package that builds a single weighted microbiome network by combining ten inference algorithms: Pearson, Spearman, Bicor, SparCC, SpiecEasi (MB and glasso), SPRING, GCoDA, CCLasso, and a new conditional mutual information method called CMIMN. The authors are trying to establish that a consensus approach reduces the algorithm-dependent variation that makes single-method networks unreliable, and that edge weights based on how many methods confirm an interaction give researchers a more stable and comprehensive representation of microbial relationships. This matters because microbiome research lacks a gold standard for interaction networks, so practitioners currently have no principled way to choose among divergent algorithm outputs. If the claim holds, CMiNet gives microbiome researchers a practical tool for turning method disagreement into a graded confidence score.","feed_headline":"Ten microbiome algorithms merged into one consensus network","feed_subtitle":"CMiNet R package scores every edge by how many methods confirm it, giving microbiome researchers a stabler network view.","key_machinery":"The central object is the weighted consensus network matrix: an $n \\times n$ matrix whose entries count how many of the included algorithms place an interaction between each pair of taxa, together with the thresholding step that converts those counts into a final network. The novel methodological ingredient is CMIMN, a conditional mutual information algorithm that captures non-linear conditional dependencies between taxa, complementing the nine existing methods. The weight-by-agreement construction does the argument's work: it converts a set of divergent binary network estimates into a single graded confidence score, lets users choose stringency via a threshold, and supplies Hamming distances as a measure of structural disagreement among methods.","core_discovery":"On its own terms, the paper's central claim is that no individual network inference method should be trusted as the sole picture of microbial interactions, because different algorithms—correlation-based, inverse-covariance based, and compositional-data specific—produce substantially different networks from the same data. CMiNet addresses this by running all ten methods and recording, for each pair of taxa, how many methods place an edge between them; the result is a weighted consensus network matrix in which edge weight equals the level of cross-method agreement. The authors show that thresholding this weight changes network size monotonically (for example, threshold 9 gives 47 edges, threshold 8 gives 94), and they argue that edges confirmed by all or most methods are more stable and reliable than those found by a single algorithm. The package also provides Hamming-distance comparisons, visualization, and per-algorithm parameter control. The paper's claim is that this consensus output is a more stable and comprehensive representation of microbial interactions than any single-method network, while acknowledging that no gold standard exists against which to check accuracy.","pith_inferences":["The paper's confidence argument is only as strong as the independence of the methods' errors; if several algorithms share the same bias, their agreement may overstate confidence rather than accuracy, a possibility the authors do not test empirically.","A natural testable extension is a synthetic benchmark with known interaction ground truth, comparing consensus recovery against each single method; if consensus does not beat the best method in such simulations, its value would be stability rather than accuracy.","Since CMIMN is new and only described in a manuscript in preparation, the package's 'nine plus one' consensus depends on an algorithm not yet independently validated; a standalone description of CMIMN's properties would strengthen the consensus claim.","The reported thresholding behavior suggests edge counts are highly concentrated near full agreement, so users may want guidance on choosing thresholds a priori rather than by inspecting network size."],"forward_implications":["Researchers can set a threshold to trade off sensitivity and specificity: high thresholds keep only edges every method agrees on, while lower thresholds admit edges supported by most methods.","Edge weights provide a per-interaction confidence score that can be used to prioritize candidate microbial interactions for experimental validation.","The package broadens applicability by handling quantitative and compositional data through algorithm-specific defaults and customizable parameters.","The consensus network can serve as a reference or benchmark when comparing newly developed network inference algorithms.","Because the package is open-source R, reproducible consensus networks can become a routine part of microbiome analysis pipelines."],"supporting_citations":[{"why":"Supplies the motivating observation that different algorithms yield substantially different network structures, the problem CMiNet is designed to solve.","marker":"[4]"},{"why":"Provides the SparCC method whose log-ratio correlation approach is integrated into the consensus.","marker":"[5]"},{"why":"Provides the two SpiecEasi variants (MB and glasso) integrated into the consensus.","marker":"[6]"},{"why":"Provides the SPRING semi-parametric rank-based method integrated into the consensus.","marker":"[7]"},{"why":"Provides the GCoDA method for conditional dependence in compositional data.","marker":"[8]"},{"why":"Provides the CCLasso lasso-based compositional correlation method.","marker":"[9]"},{"why":"Is the source of the novel CMIMN conditional mutual information algorithm included in the consensus.","marker":"[10]"}],"fun_headline_variants":["Ten algorithms, one consensus microbiome network","Microbiome consensus network from ten methods","Every edge voted on by ten microbiome methods","Ten-way agreement builds stabler microbiome networks","Consensus scoring across ten microbiome algorithms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that agreement among multiple inference methods is a trustworthy sign of a real microbial interaction, even though the paper does not test this against known true networks.","fun_headline_variants_meta":{"raw":{"variants":["Ten algorithms, one consensus microbiome network","Microbiome consensus network from ten methods","Every edge voted on by ten microbiome methods","Ten-way agreement builds stabler microbiome networks","Consensus scoring across ten microbiome algorithms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2727,"prompt_tokens":969,"completion_tokens":1758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1708}},"tokens_in":585,"tokens_out":1758,"duration_ms":12483,"temperature":1.0,"reasoning_tokens":1708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:43:45.885359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic microbial community with known interaction structure, simulate compositional count data from it, run CMiNet, and compare the consensus network with each individual method's output; if a single method recovers the true edges more accurately than the consensus, or if a consensus built from deliberately biased but mutually agreeing methods still looks confident, the claim that consensus implies reliability would be refuted.","supporting_citations":[{"cited_title":"Networks as tools for defining emergent properties of microbiomes and their stability","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating observation that different algorithms yield substantially different network structures, the problem CMiNet is designed to solve."},{"cited_title":"Inferring correlation networks from genomic survey data.PLoS computational biology, 8(9):e1002687, 2012","cited_arxiv_id":null,"evidence_quote":"Provides the SparCC method whose log-ratio correlation approach is integrated into the consensus."},{"cited_title":"Sparse and compositionally robust inference of microbial ecological networks","cited_arxiv_id":null,"evidence_quote":"Provides the two SpiecEasi variants (MB and glasso) integrated into the consensus."},{"cited_title":"Microbial networks in spring-semi-parametric rank-based correlation and partial correlation estimation for quantitative microbiome data","cited_arxiv_id":null,"evidence_quote":"Provides the SPRING semi-parametric rank-based method integrated into the consensus."},{"cited_title":"gcoda: conditional dependence network inference for compositional data","cited_arxiv_id":null,"evidence_quote":"Provides the GCoDA method for conditional dependence in compositional data."},{"cited_title":"Cclasso: correlation inference for compositional data through lasso","cited_arxiv_id":null,"evidence_quote":"Provides the CCLasso lasso-based compositional correlation method."},{"cited_title":"Leveraging machine learning and enhanced network-based methods in potato disease interactions","cited_arxiv_id":null,"evidence_quote":"Is the source of the novel CMIMN conditional mutual information algorithm included in the consensus."}],"review_version":1}