Pith. sign in

REVIEW 2 major objections 6 minor 29 references

NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NApy is a Python package that runs seven standard statistical tests on all feature pairs of large heterogeneous datasets with missing values, reporting orders-of-magnitude runtime and memory gains over established Python statistics tools.

desk verdict Solid engineering contribution with a credible parallel-performance claim; the soft spots are overbroad wording and weak baselines for five of seven tests, but they are fixable and not fatal. read the letter →

arxiv 2505.00448 v1 pith:KNQGQJE2 submitted 2025-05-01 cs.MS cs.DCcs.PF

Fabian Woller , Lis Arend , Christian Fuchsberger , Markus List , David B. Blumenthal This is my paper · ORCID
classification cs.MScs.DCcs.PF MSC 62-0465Y05
keywords statisticalsoftwareefficientcomputingandparallelizationpythonlarge-scaledatasetsmissingdatapairwisedeletionmixed-typeOpenMP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents NApy, a Python package designed for computing statistical associations between every pair of features in large heterogeneous datasets when many values are missing. Its central claim is that by fusing pairwise missing-value deletion directly into the computation and parallelizing over feature pairs with shared-memory threads, NApy runs seven common statistical tests orders of magnitude faster and with far less memory than existing Python tools and naive Python baselines. The motivation is interactive biomedical data exploration: population cohorts with thousands of samples and thousands of variables need on-the-fly association networks, which current libraries cannot deliver because they lack built-in parallelization and only support pairwise deletion for a few tests. If the claim holds, NApy fills a practical gap between rich but slow Python statistics and the scale requirements of cohort studies.

What carries the argument

The central mechanism is pairwise feature-pair parallelization with fused pairwise missing-value deletion. For a pair of features $(g,h)$, only samples where both values are non-missing enter the test, i.e. the set $I(g,h)=\{(g_i,h_i): g_i\neq m, h_i\neq m, i=1,\dots,S\}$; this subsetting happens inside the test loop rather than as a preprocessing step, so no intermediate copies are allocated. The outer loop over all feature pairs is distributed over threads via OpenMP or Numba on shared memory, and each test is implemented natively in C++ or JIT-compiled Numba. Pre-sorting and ranking continuous features once, rather than again for every pair, is what keeps nonparametric tests such as Spearman, Mann-Whitney U, and Kruskal-Wallis inexpensive.

What would settle it

Run an independent benchmark on a large missing-data matrix, for instance 10,000 features and 1,000 samples with 10% missingness, comparing NApy with vectorized SciPy and pandas code that uses optimized linear-algebra backends and precomputed complete-case filtering, measuring wall time and peak memory on a multi-core server; if NApy's runtime and memory advantage drops below the claimed order-of-magnitude range, the central claim is not as general as stated.

Watch

Extended reading notes

Core claim

The paper claims that NApy is the first Python statistics package to combine seven standard statistical tests, pairwise missing-value removal, effect sizes, multiple-testing correction, and shared-memory parallelization in one library. The tests, covering every combination of continuous, dichotomous, and categorical features, are Pearson correlation, Spearman correlation, chi-square test, t-test, Mann-Whitney U, ANOVA, and Kruskal-Wallis. Efficiency comes from implementing the tests in C++ with OpenMP and in Numba, parallelizing at the outer level of pairwise feature analysis, and performing missing-value removal on the fly without creating copy-heavy preprocessing steps. Benchmarks on simulated matrices and on the CHRIS cohort report consistent runtime improvements over pandas, SciPy, Pingouin, and naive Python loops, with speedups over 1000-fold in the most extreme simulated case and over 400-fold with 64 threads on real data, while memory use stays constant as the thread count increases.

Load-bearing premise

The load-bearing premise is that the benchmark baselines fairly represent how scientists actually run SciPy, pandas, and Pingouin on large data; if those tools are used with the optimizations practitioners commonly apply, the reported orders-of-magnitude speedups could shrink.

Editorial extensions

If this is right

  • Interactive data explorers can compute all pairwise association networks for large cohort datasets on the fly instead of relying only on precomputed, expert-curated networks.
  • Analysts can keep samples that have data for a given pair of features instead of discarding entire rows, reducing information loss when missingness is high.
  • The same library covers continuous, dichotomous, and categorical variables, so a mixed-type cohort can be analyzed without splitting it by data type.
  • Shared-memory parallelization makes runtime scale with core count while memory use stays flat, which is valuable on shared compute servers.
  • On the CHRIS cohort, the reported single-thread improvements range from 2-fold to 14-fold, and 64-thread improvements reach over 400-fold for several tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fused missing-value deletion were combined with sparse matrix storage, memory use could drop further for very high missingness; the paper does not explore this combination.
  • The benchmark design compares against author-constructed Python loops where no library function exists, so an independent benchmark that optimizes competitors' vectorization and linear-algebra settings would clarify how much of the speedup is algorithmic versus implementation-level.
  • The same shared-memory pair-parallel design could be extended to covariate-adjusted tests or mixed models, which the paper names as a limitation; that extension would broaden its applicability to confounded cohort analyses.
  • Because the exact Mann-Whitney mode is backed by a dynamic program, the package could potentially be extended to exact versions of other rank-based statistics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents NApy, a Python package for computing seven statistical tests (Pearson, Spearman, chi-squared, t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis) with pairwise deletion of missing values and shared-memory parallelization using a C++/OpenMP backend and an alternative Numba implementation. The authors benchmark NApy against pandas, SciPy, Pingouin, and self-constructed Python-loop baselines on simulated data with varying feature/sample counts, thread counts, and missingness, and on the CHRIS population cohort. They report order-of-magnitude runtime speedups and lower memory usage, and validate correctness against SciPy and R libraries via unit tests. The paper claims that NApy enables on-the-fly statistical association analysis for large heterogeneous biomedical datasets.

Significance. If the performance claims are robust, NApy addresses a real gap: existing Python statistics libraries do not provide a single-call pairwise computation of these tests with pairwise missing-data removal and parallelization. The paper's strengths include open-source code, unit tests against SciPy and R, benchmark scripts, and a realistic real-world dataset (CHRIS). The shared-memory parallelization design avoids the memory multiplication typical of multiprocessing-based Python parallelism, which is a credible engineering contribution. However, the headline 'orders of magnitude' claim is only partially supported because for five of the seven tests the comparison is against a self-constructed naive Python loop rather than an optimized competitor, and the paper itself reports an exception for single-thread Pearson correlation.

major comments (2)
  1. [§4.1.3, §4.2.2, Figure 4] The benchmark for the chi-squared, t-test, Mann-Whitney U, ANOVA, and Kruskal-Wallis tests compares NApy only against the authors' 'SciPy-Python Loop' baseline, which iterates over feature pairs with itertools and joblib and calls SciPy functions per pair. This baseline is not an existing competitor and is unrepresentative of optimized usage: a vectorized NumPy/SciPy implementation that processes all feature pairs in a chunked or matrix-wide manner (e.g., group sums and sums of squares for t-tests, bincount-based contingency tables for chi-squared) could eliminate much of the reported 10-1000x gap. Since the abstract's 'orders of magnitude' claim and the 'fastest competitor' language in Section 4.2.2 rest on this baseline, the authors should either add such vectorized baselines to the benchmark or restrict the claim to 'naive Python-loop baselines' throughout the abstract, introduction, and conclusion.
  2. [Abstract, §4.3, §6, Table 3] The abstract's unqualified claim that NApy 'outperforms competitor tools and baseline implementations with naïve Python-based parallelization by orders of magnitude' is internally contradicted by the paper's own results: Section 4.3 reports that pandas slightly outperforms NApy for single-thread Pearson on CHRIS data, and Table 3 shows pandas uses 0.009 GB versus NApy's 0.015 GB for Pearson on simulated data. The claim should be scoped to parallel execution and to tests where NApy provides pairwise missing-value handling not present in competitors; the conclusion should match this nuance rather than restating 'consistently outperforms Python competitors.'
minor comments (6)
  1. [§1] The statement that Cython 'inherently lacks support for intrinsic parallelization' is incorrect; Cython supports OpenMP parallelization via prange. Also, SciPy is not mainly implemented in Cython; it is a mix of Fortran, C, C++, and Python. Please correct these descriptions.
  2. [§4.1.3] The paper states that no direct competitors exist for the five tests, but vectorized approaches using SciPy's array operations are possible. Please clarify the rationale for not considering them as competitors in the benchmark, or add them as additional baselines.
  3. [Table 3] Memory measurements are reported as averages with no standard deviations, while runtime measurements include error bars; please add variability information or state that memory variability was negligible.
  4. [§4.2.2] The sentence 'For all tests and all numbers of features, samples, and threads, NApy is consistently faster than all tested competitors' is only valid for simulated data; Section 4.3 shows pandas is faster for single-thread Pearson on CHRIS. Please qualify this sentence.
  5. [§3.6] The correctness validation against SciPy and R is described but no numerical agreement results are shown; please include a summary of the unit test outcomes or reference a specific test report in the repository.
  6. [Figure 4, Table 3] The paper reports fold changes and memory comparisons but does not provide absolute runtime values for all competitors in the main text; consider adding a supplementary table with raw runtimes for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NApy's performance claims are direct empirical measurements against external libraries and transparently disclosed self-constructed baselines.

full rationale

This is a software and benchmarking paper, not a derivation with fitted parameters, so the central claims cannot reduce to their inputs by construction. The paper's headline speedup claims are supported by runtime and memory measurements against external tools (pandas, SciPy, Pingouin) and against baselines that the paper explicitly describes as 'naive' Python loops constructed for the five tests lacking direct competitors. In Section 4.1.3, the authors state: 'For the chi2-test, the t-test, the Mann-Whitney U, ANOVA, and the Kruskal-Wallis tests, no direct competitors to NApy are currently available which allow to process all feature pairs via a single function call. Therefore, we only compared NApy against the Python-based baseline implementations of the corresponding functions in the SciPy stats module.' This is a transparent benchmark-design choice, not a circular reduction: the measured speedup is an observed empirical result against that specific baseline, not a prediction equivalent to a fitted quantity. The skeptical concern that a vectorized SciPy baseline might close much of the gap is a generalizability or benchmarking-validity issue, but it does not make the claim circular. The paper also candidly reports limitations on its strongest claim, noting in Section 4.3 that 'for computing Pearson correlation using a single thread, pandas slightly outperformed NApy in terms of runtime efficiency' and that pandas only computes correlations while NApy also computes p-values. The only self-referential element is that NApy chooses its default backend (Numba vs. C++) based on its own benchmarks in Section 4.2.1; this is standard engineering practice and does not bear on the correctness or independence of the statistical results. Verification of correctness uses external references: unit tests benchmarked against SciPy and R libraries such as Hmisc and R stats via rpy2. No load-bearing step reduces to self-citation, ansatz smuggling, or a renamed known result. Therefore no circularity is present, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is an engineering performance claim, not a scientific derivation. It depends on the statistical validity of pairwise deletion, on fair benchmark baselines, and on the specific input conventions. No free parameters are fitted to data, and no new entities are introduced.

assumptions (3)
  • domain assumption Pairwise missing data removal is an appropriate statistical handling strategy for the target use case.
    The paper motivates this with references (Eekhout et al., Shadbahr et al.) but does not derive it; if pairwise deletion is invalid for a given analysis, the tool's statistical results may be misleading. The assumption enters in Section 3.2 and the Limitations section.
  • domain assumption The benchmark baselines ('SciPy-Python Loop', etc.) represent reasonable upper bounds for competitor performance.
    The authors constructed the baselines using itertools and joblib (Section 4.1.3); the claim of orders-of-magnitude speedup depends on these being fair comparators to optimized real-world usage.
  • ad hoc to paper Missing values can be encoded as a special float sentinel, and label-encoded categories start at zero.
    This input convention is specific to NApy but reasonable for the target data (Section 3.2). It does not affect the efficiency comparison but is a design assumption users must accept.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data." pith.science (2026). https://pith.science/paper/KNQGQJE2

@misc{pith2026250500448,
  author       = {Pith},
  title        = {Pith review of: NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNQGQJE2}},
  note         = {Machine review of arXiv:2505.00448}
}
read the original abstract

Existing Python libraries and tools lack the ability to efficiently compute statistical test results for large datasets in the presence of missing values. This presents an issue as soon as constraints on runtime and memory availability become essential considerations for a particular usecase. Relevant research areas where such limitations arise include interactive tools and databases for exploratory analysis of biomedical data. To address this problem, we present the Python package NApy, which relies on a Numba and C++ backend with OpenMP parallelization to enable scalable statistical testing for mixed-type datasets in the presence of missing values. Both with respect to runtime and memory consumption, NApy outperforms competitor tools and baseline implementations with naive Python-based parallelization by orders of magnitude, thereby enabling on-the-fly analyses in interactive applications. NApy is publicly available at https://github.com/DyHealthNet/NApy.

Figures

Figures reproduced from arXiv: 2505.00448 by the authors.

Figure 1
Figure 1. Overview of NApy’s workflow (A) and benchmark analyses of runtime and memory consumption (B). NApy enables [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Distribution of missing values in the CHRIS study [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Runtime evaluation of NApy’s Numba and C++ im [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Runtime benchmarks of NApy and applicable competitors across features, sample, and thread counts. (A–C) The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Benchmark analysis of the impact of missing val [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 22 canonical work pages

  1. [1]

    Ben-Shachar, Dominique Makowski, Daniel Lüdecke, Indrajeet Patil, Brenton M

    Mattan S. Ben-Shachar, Dominique Makowski, Daniel Lüdecke, Indrajeet Patil, Brenton M. Wiernik, Rémi Thériault, and Philip Waggoner. 2019. effectsize: Indices of Effect Size. https://doi.org/10.32614/CRAN.package.effectsize

    work page doi:10.32614/cran.package.effectsize 2019

  2. [2]

    Y Benjamini and Y Hochberg. 1995. Controlling false discovery rate: practical powerful approach multiple hypothesis testing.J R Stat Soc B57 (1995), 289–300

    work page 1995

  3. [3]

    Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency.Ann. Stat.29, 4 (Aug. 2001), 1165–1188

    work page 2001

  4. [4]

    Bloomberg. [n. d.].Memray: A memory profiler for Python applications. https: //pypi.org/project/memray/

    work page

  5. [5]

    Boost. [n. d.].Boost C++ Libraries. http://www.boost.org/

    work page

  6. [6]

    Olive Jean Dunn. 1961. Multiple comparisons among means.J. Am. Stat. Assoc. 56, 293 (March 1961), 52–64

    work page 1961

  7. [7]

    Iris Eekhout, R Michiel de Boer, Jos W R Twisk, Henrica C W de Vet, and Mar- tijn W Heymans. 2012. Missing data: a systematic review of how they are reported and handled.Epidemiology23, 5 (Sept. 2012), 729–732

    work page 2012

  8. [8]

    Laurent Gautier. 2023. rpy2. https://github.com/rpy2/rpy2/releases/tag/ RELEASE_3_5_12

    work page 2023

Show all 29 references
  1. [9]

    Frank E Harrell Jr. 2003. Hmisc: Harrell Miscellaneous. https://doi.org/10.32614/ CRAN.package.Hmisc

    2003

  2. [10]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fe...

    2020

  3. [11]

    2017.pybind11 – Seamless operability between C++11 and Python

    Wenzel Jakob, Jason Rhinelander, and Dean Moldovan. 2017.pybind11 – Seamless operability between C++11 and Python. https://github.com/pybind/pybind11

    2017

  4. [12]

    Alboukadel Kassambara. 2019. rstatix: Pipe-Friendly Framework for Basic Statis- tical Tests. https://doi.org/10.32614/CRAN.package.rstatix

    2019 doi

  5. [13]

    Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: a LLVM-based Python JIT compiler. InProceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC(Austin, Texas)(LLVM ’15). Association for Computing Machinery, New York, NY, USA, Article 7, 6 page...

    2015

  6. [14]

    Jiahang Li, Shuxia Guo, Rulin Ma, Jia He, Xianghui Zhang, Dongsheng Rui, Yusong Ding, Yu Li, Leyao Jian, Jing Cheng, and Heng Guo. 2024. Comparison of the effects of imputation methods for missing data in predictive modelling of cohort study datasets.BMC Med. Res. Methodol.24,...

    2024

  7. [15]

    Blumenthal Weichung J Shih, Jay P Siegel, and Hal Stern

    Roderick J Little, Ralph D’Agostino, Michael L Cohen, Kay Dickersin, Scott S Emerson, John T Farrar, Constantine Frangakis, Joseph W Hogan, Geert Molen- berghs, Susan A Murphy, James D Neaton, Andrea Rotnitzky, Daniel Scharfstein, Fabian Woller, Lis Arend, Christian Fuchsberge...

    2012

  8. [16]

    ALGLIB LTD. [n. d.].ALGLIB®- numerical analysis library. https://www.alglib. net/

  9. [17]

    Andreas Löffler. [n. d.]. Über eine Partition der nat. Zahlen und ihr Anwendung beim U-Test”.Wiss. Z. Univ. Halle([n. d.])

  10. [18]

    Andreas Maier, Michael Hartung, Mark Abovsky, Klaudia Adamowicz, Gary D Bader, Sylvie Baier, David B Blumenthal, Jing Chen, Maria L Elkjaer, Carlos Garcia-Hernandez, Mohamed Helmy, Markus Hoffmann, Igor Jurisica, Max Kotl- yar, Olga Lazareva, Hagai Levi, Markus List, Sebastian...

    2024

  11. [19]

    Learning Statistics with R

    Danielle Navarro. 2011. lsr: Companion to "Learning Statistics with R". https: //doi.org/10.32614/CRAN.package.lsr

    2011 doi

  12. [20]

    2024.pandas-dev/pandas: Pandas

    The pandas development team. 2024.pandas-dev/pandas: Pandas. https://doi. org/10.5281/zenodo.10957263

    2024 doi

  13. [21]

    Cristian Pattaro, Martin Gögele, Deborah Mascalzoni, Roberto Melotti, Christine Schwienbacher, Alessandro De Grandi, Luisa Foco, Yuri D’Elia, Barbara Linder, Christian Fuchsberger, Cosetta Minelli, Clemens Egger, Lisa S Kofink, Stefano Zanigni, Torsten Schäfer, Maurizio F Fach...

    2015

  14. [22]

    2024.R: A Language and Environment for Statistical Computing

    R Core Team. 2024.R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project. org/

    2024

  15. [23]

    Sepideh Sadegh, Julian Matschinske, David B Blumenthal, Gihanna Galindez, Tim Kacprowski, Markus List, Reza Nasirigerdeh, Mhaned Oubounyt, Andreas Pichlmair, Tim Daniel Rose, Marisol Salgado-Albarrán, Julian Späth, Alexey Stukalov, Nina K Wenke, Kevin Yuan, Josch K Pauling, an...

    2020

  16. [24]

    Sepideh Sadegh, James Skelton, Elisa Anastasi, Judith Bernett, David B Blu- menthal, Gihanna Galindez, Marisol Salgado-Albarrán, Olga Lazareva, Keith Flanagan, Simon Cockell, Cristian Nogales, Ana I Casas, Harald H H W Schmidt, Jan Baumbach, Anil Wipat, and Tim Kacprowski. 202...

    2021

  17. [25]

    Tolou Shadbahr, Michael Roberts, Jan Stanczuk, Julian Gilbey, Philip Teare, Sören Dittmer, Matthew Thorpe, Ramon Viñas Torné, Evis Sala, Pietro Lió, Mishal Patel, Jacobus Preller, AIX-COVNET Collaboration, James H F Rudd, Tuomas Mirtti, Antti Sakari Rannikko, John A D Aston, J...

    2023

  18. [26]

    Raphael Vallat. 2018. Pingouin: statistics in Python.Journal of Open Source Software3, 31 (Nov. 2018), 1026. https://doi.org/10.21105/joss.01026

    2018 doi

  19. [27]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jar- rod Millman, Nikolay Mayorov, Andrew R. J. Nelson,...

    2020

  20. [28]

    Henry Völzke, Dietrich Alte, Carsten Oliver Schmidt, Dörte Radke, Roberto Lor- beer, Nele Friedrich, Nicole Aumann, Katharina Lau, Michael Piontek, Gabriele Born, Christoph Havemann, Till Ittermann, Sabine Schipf, Robin Haring, Se- bastian E Baumeister, Henri Wallaschofski, Ma...

    2011

  21. [29]

    Wes McKinney. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference, Stéfan van der Walt and Jarrod Millman (Eds.). 56 – 61. https://doi.org/10.25080/Majora-92bf1922-00a

    2010 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.