{"id":"fcb01f7a-e345-499f-9845-f7d6b9864f10","arxiv_id":"2505.00448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"NApy provides seven statistical tests with pairwise missing-value deletion, parallelized via OpenMP and Numba, and reports large runtime and memory gains over existing Python tools.","lead":"NApy is a new Python package that computes standard statistical tests on large biomedical datasets with missing values, using parallel C++ and Numba backends for speed. It aims to make on-the-fly analysis of all variable pairs in large cohort studies practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"For five of seven tests, the 'orders of magnitude' claim is measured against the authors' own naive Python-loop baseline, not an existing competitor; a vectorized SciPy baseline could close much of the gap, and the abstract's unqualified wording is already contradicted by the paper's single-thread…","rationale":"The paper is a solid software-engineering contribution with public code, unit tests against SciPy and R, and reproducible benchmark repositories. The strongest claim is the orders-of-magnitude speedup in runtime and memory. The most load-bearing assumption is that the comparison baselines represent realistic current practice. For correlations, real competitor tools are used, but for five of seven tests the only comparator is the authors' own per-pair Python loop around SciPy. That baseline pays O(F^2) Python call and serialization overhead, which NApy is specifically designed to avoid, so the reported speedup conflates algorithmic gains with the cost of a deliberately naive harness. This does not make the paper dishonest: the baseline is explicitly labeled naive, and the limitation is partly inherent because no existing library exposes the same all-pairs functionality. However, the abstract's unqualified wording is not supported by the paper's own single-thread Pearson result, where pandas is faster on real data. The right remedy is to keep the conditional verdict: require the abstract and conclusion to state the Pearson exception, and require benchmarks against a vectorized or otherwise optimized baseline before claiming general orders-of-magnitude improvement over optimized real-world usage. The proposed concrete test would settle the baseline concern directly and is feasible with the released benchmark scripts.","tokens_in":13213,"tokens_out":7807,"duration_ms":86142,"concrete_test":"Run the released benchmark script for the two-sample t-test on the 1000x1000 simulated data with 10% missing values using a vectorized baseline: compute per-feature group sums and sums of squares for all feature pairs via NumPy matrix operations (chunked over feature blocks to keep memory bounded), obtaining the same Welch/Student statistic and p-values as SciPy, and compare runtime and peak memory against NApy at 1 and 64 threads. If the vectorized baseline comes within a small factor (e.g., below 10x) of NApy, the 'orders of magnitude' claim is an artifact of the per-pair Python loop; if it remains more than 100x slower, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.3 defines the competitors. For Pearson and Spearman, real tools (pandas, SciPy, Pingouin) are benchmarked. For the remaining five tests (chi2, t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis), the paper states that no direct competitor exists and therefore compares NApy only against a self-constructed 'SciPy-Python Loop' baseline: itertools over feature pairs, per-pair SciPy calls, and joblib parallelization. The 'orders of magnitude' speedups in Section 4.2.2 and Figure 4 for these tests are thus measured against Python-level dispatch overhead that NApy's compiled inner loops eliminate by construction. A vectorized or chunked NumPy/SciPy implementation that computes the same test statistics over all feature pairs without a Python-level loop could plausibly remove most of the gap; if so, the headline claim overstates the advantage over realistic optimized usage. Additionally, the abstract's unqualified 'orders of magnitude' is internally qualified by the paper itself: Section 4.3 reports that pandas is slightly faster for single-thread Pearson on CHRIS data, and Table 3 shows a memory exception for that case. The central claim is credible for the parallel use case, but as stated it is broader than the evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents NApy, a Python package for computing seven statistical tests (Pearson, Spearman, chi-squared, t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis) with pairwise deletion of missing values and shared-memory parallelization using a C++/OpenMP backend and an alternative Numba implementation. The authors benchmark NApy against pandas, SciPy, Pingouin, and self-constructed Python-loop baselines on simulated data with varying feature/sample counts, thread counts, and missingness, and on the CHRIS population cohort. They report order-of-magnitude runtime speedups and lower memory usage, and validate correctness against SciPy and R libraries via unit tests. The paper claims that NApy enables on-the-fly statistical association analysis for large heterogeneous biomedical datasets.","tokens_in":13500,"tokens_out":6470,"duration_ms":59453,"significance":"If the performance claims are robust, NApy addresses a real gap: existing Python statistics libraries do not provide a single-call pairwise computation of these tests with pairwise missing-data removal and parallelization. The paper's strengths include open-source code, unit tests against SciPy and R, benchmark scripts, and a realistic real-world dataset (CHRIS). The shared-memory parallelization design avoids the memory multiplication typical of multiprocessing-based Python parallelism, which is a credible engineering contribution. However, the headline 'orders of magnitude' claim is only partially supported because for five of the seven tests the comparison is against a self-constructed naive Python loop rather than an optimized competitor, and the paper itself reports an exception for single-thread Pearson correlation.","major_comments":[{"comment":"The benchmark for the chi-squared, t-test, Mann-Whitney U, ANOVA, and Kruskal-Wallis tests compares NApy only against the authors' 'SciPy-Python Loop' baseline, which iterates over feature pairs with itertools and joblib and calls SciPy functions per pair. This baseline is not an existing competitor and is unrepresentative of optimized usage: a vectorized NumPy/SciPy implementation that processes all feature pairs in a chunked or matrix-wide manner (e.g., group sums and sums of squares for t-tests, bincount-based contingency tables for chi-squared) could eliminate much of the reported 10-1000x gap. Since the abstract's 'orders of magnitude' claim and the 'fastest competitor' language in Section 4.2.2 rest on this baseline, the authors should either add such vectorized baselines to the benchmark or restrict the claim to 'naive Python-loop baselines' throughout the abstract, introduction, and conclusion.","section":"§4.1.3, §4.2.2, Figure 4"},{"comment":"The abstract's unqualified claim that NApy 'outperforms competitor tools and baseline implementations with naïve Python-based parallelization by orders of magnitude' is internally contradicted by the paper's own results: Section 4.3 reports that pandas slightly outperforms NApy for single-thread Pearson on CHRIS data, and Table 3 shows pandas uses 0.009 GB versus NApy's 0.015 GB for Pearson on simulated data. The claim should be scoped to parallel execution and to tests where NApy provides pairwise missing-value handling not present in competitors; the conclusion should match this nuance rather than restating 'consistently outperforms Python competitors.'","section":"Abstract, §4.3, §6, Table 3"}],"minor_comments":[{"comment":"The statement that Cython 'inherently lacks support for intrinsic parallelization' is incorrect; Cython supports OpenMP parallelization via prange. Also, SciPy is not mainly implemented in Cython; it is a mix of Fortran, C, C++, and Python. Please correct these descriptions.","section":"§1"},{"comment":"The paper states that no direct competitors exist for the five tests, but vectorized approaches using SciPy's array operations are possible. Please clarify the rationale for not considering them as competitors in the benchmark, or add them as additional baselines.","section":"§4.1.3"},{"comment":"Memory measurements are reported as averages with no standard deviations, while runtime measurements include error bars; please add variability information or state that memory variability was negligible.","section":"Table 3"},{"comment":"The sentence 'For all tests and all numbers of features, samples, and threads, NApy is consistently faster than all tested competitors' is only valid for simulated data; Section 4.3 shows pandas is faster for single-thread Pearson on CHRIS. Please qualify this sentence.","section":"§4.2.2"},{"comment":"The correctness validation against SciPy and R is described but no numerical agreement results are shown; please include a summary of the unit test outcomes or reference a specific test report in the repository.","section":"§3.6"},{"comment":"The paper reports fold changes and memory comparisons but does not provide absolute runtime values for all competitors in the main text; consider adding a supplementary table with raw runtimes for reproducibility.","section":"Figure 4, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a credible software contribution, and the tool's functionality (pairwise missing-value deletion, shared-memory parallelism, and seven common tests in one package) is genuinely useful. The main issue is that the headline performance claim is not yet fully substantiated because the benchmark for five of the seven tests compares against a self-constructed naive Python loop rather than an optimized baseline. The authors could resolve this either by adding vectorized baselines or by carefully scoping the claims in the abstract and conclusion. I would not recommend rejection; this is a fixable issue within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nQuick take: NApy is a well-executed engineering contribution with a clear use case. The package combines seven standard statistical tests, pairwise missing-data deletion, and shared-memory parallelism (OpenMP and Numba) in a single Python library, which as far as I can tell no existing tool does. The benchmarks are reproducible in principle: code is public, unit tests compare against SciPy and R, and runtime results are averaged over three runs. The central claim—fast and memory-efficient all-pairs testing on mixed-type data with missing values—is credible for the parallel setting.\n\nWhat is genuinely new is the integration: the paper's own table shows pandas and SciPy only support pairwise deletion for correlations, and none parallelize. So for chi-square, t-test, Mann-Whitney, ANOVA, and Kruskal-Wallis, there is no off-the-shelf competitor, and the authors set up a naive Python loop as a baseline. That is the main soft spot. The \"orders of magnitude\" speedups for those five tests are measured against itertools + joblib + per-pair SciPy calls. A vectorized or chunked implementation that computes many pairs at once in NumPy would likely close much of that gap. The paper is somewhat careful—the abstract does say \"naive Python-based parallelization\"—but the intro's claim of \"outperforms state-of-the-art competitors\" overreaches for those five tests. Also, the paper itself reports that pandas is slightly faster for single-thread Pearson on CHRIS data; that exception is buried in Section 4.3 and not in the abstract. Minor issues: memory numbers are in one table without standard deviations, and no commit hash or exact benchmark script is referenced, only the repo.\n\nBut these are not fatal. The engineering effort is real, the correctness verification is solid (unit tests against SciPy and R), and the parallel scaling behavior looks genuinely useful for the interactive exploratory tools they motivate. I would ask the authors to soften the intro/abstract to say \"existing tools that support these tests\" or \"naive baselines,\" add the Pearson exception, and provide a pinned environment and benchmark entrypoint.\n\nVerdict: send to peer review. The paper deserves referee time; it is a useful contribution to the computational statistics tooling community. I would bring it to reading group as an example of solid software evaluation with a caution about baseline construction.","headline":"Solid engineering contribution with a credible parallel-performance claim; the soft spots are overbroad wording and weak baselines for five of seven tests, but they are fixable and not fatal.","tokens_in":13958,"tokens_out":2216,"would_cite":false,"duration_ms":22686,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62-04","65Y05"],"pacs":[],"model":"deepseek-v4-flash","headline":"NApy is a Python package that runs seven standard statistical tests on all feature pairs of large heterogeneous datasets with missing values, reporting orders-of-magnitude runtime and memory gains over established Python statistics tools.","keywords":["statistical software","efficient computing and parallelization","python","large-scale datasets","missing data","pairwise deletion","mixed-type data","OpenMP"],"falsifier":"Run an independent benchmark on a large missing-data matrix, for instance 10,000 features and 1,000 samples with 10% missingness, comparing NApy with vectorized SciPy and pandas code that uses optimized linear-algebra backends and precomputed complete-case filtering, measuring wall time and peak memory on a multi-core server; if NApy's runtime and memory advantage drops below the claimed order-of-magnitude range, the central claim is not as general as stated.","tokens_in":13052,"feed_emoji":"📊","tokens_out":6332,"duration_ms":60853,"temperature":0.7,"pith_summary":"The paper presents NApy, a Python package designed for computing statistical associations between every pair of features in large heterogeneous datasets when many values are missing. Its central claim is that by fusing pairwise missing-value deletion directly into the computation and parallelizing over feature pairs with shared-memory threads, NApy runs seven common statistical tests orders of magnitude faster and with far less memory than existing Python tools and naive Python baselines. The motivation is interactive biomedical data exploration: population cohorts with thousands of samples and thousands of variables need on-the-fly association networks, which current libraries cannot deliver because they lack built-in parallelization and only support pairwise deletion for a few tests. If the claim holds, NApy fills a practical gap between rich but slow Python statistics and the scale requirements of cohort studies.","feed_headline":"One Python package makes pairwise stats on missing data 1000x faster","feed_subtitle":"Seven statistical tests with pairwise missing-value deletion run in parallel on large mixed-type biomedical data.","key_machinery":"The central mechanism is pairwise feature-pair parallelization with fused pairwise missing-value deletion. For a pair of features $(g,h)$, only samples where both values are non-missing enter the test, i.e. the set $I(g,h)=\\{(g_i,h_i): g_i\\neq m, h_i\\neq m, i=1,\\dots,S\\}$; this subsetting happens inside the test loop rather than as a preprocessing step, so no intermediate copies are allocated. The outer loop over all feature pairs is distributed over threads via OpenMP or Numba on shared memory, and each test is implemented natively in C++ or JIT-compiled Numba. Pre-sorting and ranking continuous features once, rather than again for every pair, is what keeps nonparametric tests such as Spearman, Mann-Whitney U, and Kruskal-Wallis inexpensive.","core_discovery":"The paper claims that NApy is the first Python statistics package to combine seven standard statistical tests, pairwise missing-value removal, effect sizes, multiple-testing correction, and shared-memory parallelization in one library. The tests, covering every combination of continuous, dichotomous, and categorical features, are Pearson correlation, Spearman correlation, chi-square test, t-test, Mann-Whitney U, ANOVA, and Kruskal-Wallis. Efficiency comes from implementing the tests in C++ with OpenMP and in Numba, parallelizing at the outer level of pairwise feature analysis, and performing missing-value removal on the fly without creating copy-heavy preprocessing steps. Benchmarks on simulated matrices and on the CHRIS cohort report consistent runtime improvements over pandas, SciPy, Pingouin, and naive Python loops, with speedups over 1000-fold in the most extreme simulated case and over 400-fold with 64 threads on real data, while memory use stays constant as the thread count increases.","pith_inferences":["If the fused missing-value deletion were combined with sparse matrix storage, memory use could drop further for very high missingness; the paper does not explore this combination.","The benchmark design compares against author-constructed Python loops where no library function exists, so an independent benchmark that optimizes competitors' vectorization and linear-algebra settings would clarify how much of the speedup is algorithmic versus implementation-level.","The same shared-memory pair-parallel design could be extended to covariate-adjusted tests or mixed models, which the paper names as a limitation; that extension would broaden its applicability to confounded cohort analyses.","Because the exact Mann-Whitney mode is backed by a dynamic program, the package could potentially be extended to exact versions of other rank-based statistics."],"forward_implications":["Interactive data explorers can compute all pairwise association networks for large cohort datasets on the fly instead of relying only on precomputed, expert-curated networks.","Analysts can keep samples that have data for a given pair of features instead of discarding entire rows, reducing information loss when missingness is high.","The same library covers continuous, dichotomous, and categorical variables, so a mixed-type cohort can be analyzed without splitting it by data type.","Shared-memory parallelization makes runtime scale with core count while memory use stays flat, which is valuable on shared compute servers.","On the CHRIS cohort, the reported single-thread improvements range from 2-fold to 14-fold, and 64-thread improvements reach over 400-fold for several tests."],"supporting_citations":[{"why":"Supplies the SciPy statistical functions used both as baseline competitors and as the reference for correctness verification of p-values and test statistics.","marker":"[27]"},{"why":"Pingouin is a main competitor for pairwise correlation computations with p-values and effect sizes.","marker":"[26]"},{"why":"pandas provides the DataFrame.corr competitor used for pairwise Pearson and Spearman correlation benchmarks.","marker":"[29]"},{"why":"NumPy defines the input and output data structures and is also used in the naive Python baselines.","marker":"[10]"},{"why":"The CHRIS study supplies the real-world population cohort dataset with 10,464 features used for the practical benchmark.","marker":"[21]"},{"why":"Numba is the just-in-time compiler backend that provides the default parallel implementation for parametric tests.","marker":"[13]"},{"why":"pybind11 is the binding library that integrates the C++ OpenMP implementations into the Python wrapper.","marker":"[11]"},{"why":"Memray is the memory profiler used to measure allocated memory for NApy and all competitors.","marker":"[4]"},{"why":"The dynamic programming approach for exact Mann-Whitney U p-values is the basis of NApy's auto mode for small or tie-free samples.","marker":"[17]"}],"fun_headline_variants":["NApy: 1000x faster stats on messy data","Parallel stats with missing data: 1000x speedup","NApy crunches missing-data stats 1000x faster","Statistical tests on missing data, 1000x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark baselines fairly represent how scientists actually run SciPy, pandas, and Pingouin on large data; if those tools are used with the optimizations practitioners commonly apply, the reported orders-of-magnitude speedups could shrink.","fun_headline_variants_meta":{"raw":{"variants":["NApy: 1000x faster stats on messy data","Parallel stats with missing data: 1000x speedup","NApy crunches missing-data stats 1000x faster","Statistical tests on missing data, 1000x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1222,"prompt_tokens":882,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":498,"tokens_out":340,"duration_ms":3418,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:41:11.933470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent benchmark on a large missing-data matrix, for instance 10,000 features and 1,000 samples with 10% missingness, comparing NApy with vectorized SciPy and pandas code that uses optimized linear-algebra backends and precomputed complete-case filtering, measuring wall time and peak memory on a multi-core server; if NApy's runtime and memory advantage drops below the claimed order-of-magnitude range, the central claim is not as general as stated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CHRIS study supplies the real-world population cohort dataset with 10,464 features used for the practical benchmark."},{"cited_title":"2017.pybind11 – Seamless operability between C++11 and Python","cited_arxiv_id":null,"evidence_quote":"pybind11 is the binding library that integrates the C++ OpenMP implementations into the Python wrapper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Memray is the memory profiler used to measure allocated memory for NApy and all competitors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The dynamic programming approach for exact Mann-Whitney U p-values is the basis of NApy's auto mode for small or tie-free samples."}],"review_version":1}