REVIEW 4 major objections 4 minor 12 references
Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assays
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper reports that replacing slope-based classifiers with non-parametric distribution tests on CRISPR-Cas12a fluorescence data raises sensitivity from as low as 38% to as high as 88%, and that an LSTM network reaches 100% specificity…
desk verdict Useful benchmark of EDF tests against slope methods for CRISPR-Cas12a data, but the LSTM's 100% specificity is not a valid generalization estimate because validation samples share patients with training data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the comparison of empirical cumulative distribution functions (ECDFs) of fluorescence intensity over time. For the statistical arm, each sample's most recent readings in a sliding window are compared with the ECDF of all 24 known negative trials using three quadratic EDF statistics: the Kolmogorov-Smirnov statistic (maximum vertical gap between ECDFs), the Anderson-Darling statistic (squared gap weighted toward distribution tails), and the Cramér-von Mises statistic (squared gap integrated across the whole distribution); the significance threshold, window length, and required run length are tuned by parameter sweeps. For the machine-learning arm, a two-layer long short-term memory (LSTM) recurrent network with gated memory cells reads the raw fluorescence sequence, outputs a confidence via a sigmoid layer, and classifies as positive only above a 0.95 threshold; ten random seed splits are averaged. These tools replace point-in-time slopes with whole-curve shape comparisons, which is the mechanism behind the reported gains.
What would settle it
Re-run both arms with strict separation: split the LSTM by patient so all three replicate assays of any swab stay in the same fold, and rebuild the negative reference distribution for each statistical test while excluding the sample being classified. If the LSTM's 100% specificity and 92.14% accuracy fall to much lower values, or the EDF methods' specificity drops below 92%, the reported gains are artifacts of leakage rather than measurable generalization; if the numbers survive, the paper's central claim stands.
Extended reading notes
Core claim
Using a clinical set of 16 HPV-16 vaginal swabs (8 positive and 8 negative, each assayed in triplicate) from an RPA-CRISPR-Cas12a one-pot reaction, the paper reports that conventional classifiers underperform: fluorescence magnitude and single-point slope give sensitivities of 42% and 38%, while the average-slope method gives 71% sensitivity and 88% specificity. The three quadratic empirical distribution function tests — Kolmogorov-Smirnov, Anderson-Darling, and Cramér-von Mises — compare a sliding window of each sample's fluorescence with the distribution of all known negatives. They report sensitivities of 83%, 75%, and 88%, specificity of 92% for all three, total accuracies of 87.5%, 83.3%, and 89.6%, and average time-to-result of 13–15 minutes. The LSTM network, averaged over ten random seeds with a 0.95 confidence threshold, reports 87% sensitivity, 100% specificity, 92.14% total accuracy, and a 25.4-minute average time-to-result; the authors note it was the only method to flag four low-signal positive samples that the distribution tests missed. The paper's conclusion is that the analysis method should be chosen according to whether the assay prioritizes rapid sensitive detection (EDF tests) or conservative specificity (LSTM).
Load-bearing premise
The reported performance rests on the assumption that the validation scheme is leak-free: the LSTM's random 70/30 split must not put replicate assays of the same patient in both training and validation, and the statistical tests must not compare a negative sample to a reference distribution that contains that same sample's own readings.
Editorial extensions
If this is right
- The fastest classifiers on this data set are the EDF tests, with average time-to-result of 13–15 minutes versus 35–43 minutes for the slope-based benchmarks, so point-of-care CRISPR assays could return positive calls minutes sooner if they adopt these methods.
- The LSTM's 87% sensitivity and 100% specificity make it the conservative classifier, suitable when a false positive is more harmful than a false negative.
- Because performance shifts with significance threshold, sliding-window length, and run length, the paper's parameter sweeps give users a template for tuning any CRISPR-Cas assay to its clinical priority.
- The comparison is carried out on raw time-series fluorescence, so the same statistical classifiers and LSTM can be retrained or retuned for other CRISPR-Cas and isothermal amplification assays rather than only for HPV-16.
Reading between the lines
- Editorial inference: the natural next step is a two-stage classifier in which a fast EDF test makes the initial call and the LSTM arbitrates low-confidence traces; that combination would join the EDF methods' speed with the LSTM's specificity, a direction the paper only gestures at with its mixture-of-experts suggestion.
- Editorial inference: the cleanest transfer test is to run the same seven classifiers on other real-time CRISPR-Cas or isothermal-amplification fluorescence data; if the ranking holds, the result is a general property of analysis choice rather than a quirk of this HPV-16 set.
- Editorial inference: because the LSTM split is random at the assay level, patient-level cross-validation, holding out all three replicate assays of a swab together, is the check that would tell whether the 92.14% accuracy is generalization or memorization; the paper does not report this check.
- Editorial inference: prospective use of the statistical tests requires a fixed negative reference distribution and frozen parameters; the paper's sweep gives a way to choose them but not the final deployed settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks fluorescence magnitude, average-slope, and single-point-slope methods against three quadratic empirical distribution function (EDF) statistical tests (Kolmogorov-Smirnov, Anderson-Darling, Cramer-von Mises) and an LSTM recurrent neural network for classifying CRISPR-Cas12a diagnostic fluorescence data. On a previously published clinical HPV-16 dataset (16 patients, 8 positive and 8 negative, each assayed in triplicate, giving 48 assays), the authors report that the EDF methods improve total accuracy and shorten time-to-result relative to slope-based benchmarks, and that the LSTM achieves 92.14% total accuracy with 100% specificity. They conclude that choice of analysis method substantially affects diagnostic performance and provide parameter-selection guidance.
Significance. If the reported performance held under honest evaluation, the paper would make a useful contribution by demonstrating that data-analysis choices can materially affect CRISPR-diagnostic accuracy and speed, and by introducing a machine-learning baseline to this application area. The work is clearly written, the benchmark comparisons are systematic, and the parameter-sweep figures in the Supporting Information give transparency into the optimization process. However, the central quantitative claims, especially the LSTM's 100% specificity and the EDF accuracy improvements, rest on evaluation protocols that mix training and validation data at the patient level and tune parameters on the same dataset used for reporting. As such, the headline numbers are not yet credible estimates of generalization to new patients or new runs.
major comments (4)
- [Long Short-Term Memory Analysis; Acquisition of the model data set] The LSTM evaluation uses a random 70/30 split of the 48 assays without grouping by patient. Since the dataset comprises triplicate assays of 16 clinical samples, replicate assays from the same patient can appear in both training and validation; for each patient the probability of such a split is approximately 0.63, and across 16 patients it is essentially certain that some patients contribute to both sets. The reported 92.14% accuracy and 100% specificity therefore likely reflect memorization of patient-specific fluorescence signatures rather than generalization to unseen patients. The paper acknowledges the small training set but never states this leakage as a limitation. A patient-level (or at least replicate-grouped) split is required before any generalization claim can be made.
- [Kolmogorov-Smirnov Test; Anderson-Darling Test; Cramér-von Mises Test; Supporting Figures S2-S10; Figure S11] For each EDF test, the paper states that 'cut-off values for run length, window length and alpha threshold were optimized to maximize total accuracy' on the same 48-assay dataset on which accuracy is then reported. This in-sample tuning inflates the reported accuracies (87.5%, 83.33%, 89.58%) relative to what would be seen on new data. Similarly, the LSTM prediction threshold was chosen by testing 0.90, 0.95, and 0.99 and retaining the value 'yielding the highest total accuracy' (Figure S11) on the validation set, which is threshold tuning on the test data. Nested cross-validation or a separate held-out threshold-selection set is needed to obtain unbiased performance estimates.
- [Impact of Classification Method on Time-To-Result] The text states 'Considering the high sensitivities and average TTRs of 4.67, 4.67 and 6 minutes for K-S, A-D, and C-vM, respectively,' which is inconsistent with the average TTR values of 13-14.7 minutes reported for these methods in the same section and in Figure 5A. Either the figure or the text contains an error, and this discrepancy must be resolved because the TTR comparison is a central part of the paper's message.
- [Impact of Classification Method on Time-To-Result; Figure 5A(ii)] The paper claims a 'statistically significant difference' between slope-based and EDF TTRs, but no statistical test, p-value, or effect size is reported. Given the small number of samples (24 positives) and large TTR variance, the reader cannot verify this claim. Please specify the test used (e.g., Wilcoxon signed-rank or paired t-test) and report the corresponding statistic.
minor comments (4)
- [Materials and Methods] The dataset is described as 16 clinical samples with triplicate assays in 'Acquisition of the model data set', but the Results section later says '24 positive and 24 negative assays' without explaining that these are triplicates; the text should state this explicitly to avoid confusion.
- [Long Short-Term Memory Analysis] The displayed binary cross-entropy loss equation contains garbled characters (e.g., '𝑦!', '𝑦2!', and the summation subscript). It should be typeset correctly.
- [Materials and Methods, Average Slope Test] The method by Fozouni et al. is misspelled as 'Fouzoni' in the text; please correct the spelling.
- [Impact of Classification Method on Time-To-Result] The paper uses 'TTR' both for the per-sample first positive call time and for the average over samples; clearly distinguishing these in the text and figure captions would improve readability.
Circularity Check
No significant circularity: the performance comparisons are empirical, externally labeled, and do not reduce to their inputs; score 2 reflects minor non-load-bearing reuse of the authors' prior dataset.
full rationale
This paper is an empirical benchmarking study rather than a derivation from first principles. Its central claims—that quadratic EDF tests and an LSTM-RNN improve sensitivity, specificity, and time-to-result relative to slope-based methods—are measurements made on a fixed clinical dataset (48 RPA–CRISPR-Cas12a assays from 16 HPV-16 vaginal swabs), with ground-truth labels taken from an external Anyplex qPCR assay. No equation is defined in terms of its own output, no fitted parameter is renamed as a prediction in a way that makes a headline number true by construction, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the chosen analysis methods. The reuse of the Lesinski et al. dataset (ref. 44, with overlapping authorship) is a self-citation, but it is a data source rather than a load-bearing argument; the paper's contribution is the comparative analysis, and the labels are externally sourced. Two statistical validity concerns exist but are not circularity: (i) the EDF parameters and LSTM threshold are optimized on the same data whose accuracy is then reported (e.g., 'Cutoffs of 0.90, 0.95, and 0.99 were tested, with the one yielding the highest total accuracy being selected'), and (ii) the LSTM's 70/30 split is at the assay level, not the patient level, so replicate assays from the same sample can appear in both training and validation, inflating the reported 100% specificity. The paper itself acknowledges the overfitting risk from the small training set. These issues undermine generalization claims but do not make the results equivalent to the inputs by definition. Accordingly, no circular step is identified.
Assumptions & free parameters
free parameters (11)
- K-S test threshold alpha =
0.003
- K-S window length =
3
- K-S run length =
1
- A-D test threshold alpha =
0.0015
- A-D window length =
3
- A-D run length =
1
- C-vM test threshold alpha =
0.0001
- C-vM window length =
6
- C-vM run length =
1
- LSTM prediction threshold =
0.95
- LSTM architecture hyperparameters (hidden size, layers, dropout) =
64, 2, 0.2
assumptions (3)
- domain assumption Anyplex qPCR provides correct ground-truth labels for the 16 clinical samples
- domain assumption The 48 assays can be treated as independent samples for evaluation purposes
- domain assumption Small-window empirical distribution comparisons yield valid p-values for classification
Cite this review
Pith. "Pith review of Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assays." pith.science (2026). https://pith.science/paper/YN2PFP7X
@misc{pith2026250104413,
author = {Pith},
title = {Pith review of: Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assays},
year = {2026},
howpublished = {\url{https://pith.science/paper/YN2PFP7X}},
note = {Machine review of arXiv:2501.04413}
}
read the original abstract
CRISPR-based diagnostics have gained increasing attention as biosensing tools able to address limitations in contemporary molecular diagnostic tests. To maximise the performance of CRISPR-based assays, much effort has focused on optimizing the chemistry and biology of the biosensing reaction. However, less attention has been paid to improving the techniques used to analyse CRISPR-based diagnostic data. To date, diagnostic decisions typically involve various forms of slope-based classification. Such methods are superior to traditional methods based on assessing absolute signals, but still have limitations. Herein, we establish performance benchmarks (total accuracy, sensitivity, and specificity) using common slope-based methods. We compare the performance of these benchmark methods with three different quadratic empirical distribution function statistical tests, finding significant improvements in diagnostic speed and accuracy when applied to a clinical data set. Two of the three statistical techniques, the Kolmogorov-Smirnov and Anderson-Darling tests, report the lowest time-to-result and highest total test accuracy. Furthermore, we developed a long short-term memory recurrent neural network to classify CRISPR-biosensing data, achieving 100% specificity on our model data set. Finally, we provide guidelines on choosing the classification method and classification method parameters that best suit a diagnostic assays needs.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[5]
Comparison of TTR and accuracy metrics across all analysis methods. A) TTR values for tests analysed using the seven different methods (i) as well as pairwise comparisons for statistically significant differences in the TTR of each test(ii). B) Ranking of each method, from fastest (1) to slowest (7). False negatives are grouped as FN. C) TTR, specificity,...
work page 2021
-
[12]
https://doi.org/10.48550/arXiv.2404.01365. S1 Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assays Nathan Khosla1#, Jake M. Lesinski1#, Marcus Haywood-Alexander2, Andrew J. deMello1* & Daniel A. Richards1* 1Institute for Chemical and Bioengineering, ETH Zurich, Vladimir-Prelog-Weg 1, 8093, Zürich, Switzerland. 2Institute of S...
-
[32]
25 (44) Lesinski, J. M.; Moragues, T.; Mathur, P.; Shen, Y.; Paganini, C.; Bezinge, L.; Verberckmoes, B.; Van Eenooghe, B.; Stavrakis, S.; deMello, A. J.; Richards, D. A. In Situ Complexation of SgRNA and Cas12a Improves the Performance of a One-Pot RPA–CRISPR-Cas12 Assay. Anal. Chem. 2024, 96 (25), 10443–10450. https://doi.org/10.1021/acs.analchem.4c0177...
arXiv 2024
-
[304]
(38) Lee, Y.; Kim, Y.-S.; Lee, D.; Jeong, S.; Kang, G.-H.; Jang, Y
https://doi.org/10.3390/v15020304. (38) Lee, Y.; Kim, Y.-S.; Lee, D.; Jeong, S.; Kang, G.-H.; Jang, Y. S.; Kim, W.; Choi, H. Y.; Kim, J. G.; Choi, S. The Application of a Deep Learning System Developed to Reduce the Time for RT-PCR in COVID-19 Detection. Sci. Rep. 2022, 12 (1),
-
[1234]
https://doi.org/10.1038/s41598-022-05069-2. (39) Darling, D. A. The Kolmogorov-Smirnov, Cramer-von Mises Tests. Ann. Math. Stat. 1957, 28 (4), 823–838. https://doi.org/10.1214/aoms/1177706788. (40) Borovkov, A. A. Kolmogorov and Boundary Problems of Probability Theory. Russ. Math. Surv. 2004, 59 (1), 91–102. https://doi.org/10.1070/RM2004v059n01ABEH000702...
arXiv 1957
-
[2019]
https://doi.org/10.48550/arXiv.1909.09586. (52) Oeschger, T. M.; McCloskey, D. S.; Buchmann, R. M.; Choubal, A. M.; Boza, J. M.; Mehta, S.; Erickson, D. Early Warning Diagnostics for Emerging Infectious Diseases in Developing into Late-Stage Pandemics. Acc. Chem. Res. 2021, 54 (19), 3656–3666. https://doi.org/10.1021/acs.accounts.1c00383. (53) Goldstein, ...
-
[2024]
(55) Dong, H.; Chen, B.; Chi, Y
https://doi.org/10.48550/arXiv.2405.05949. (55) Dong, H.; Chen, B.; Chi, Y. Prompt-Prompted Adaptive Structured Pruning for Efficient LLM Generation. arXiv August 11,
-
[2361]
https://doi.org/10.1038/s41467-023-38104-5. (36) Turbé, V.; Herbst, C.; Mngomezulu, T.; Meshkinfamfard, S.; Dlamini, N.; Mhlongo, T.; Smit, T.; Cherepanova, V.; Shimada, K.; Budd, J.; Arsenov, N.; Gray, S.; Pillay, D.; Herbst, K.; Shahmanesh, M.; McKendry, R. A. Deep Learning of HIV Field-Based Rapid Tests. Nat. Med. 2021, 27 (7), 1165–1170. https://doi.o...
Show all 12 references
-
[3923]
The Waiting Game
https://doi.org/10.1038/s41467-020-17419-7. (26) Abudayyeh, O. O.; Gootenberg, J. S.; Essletzbichler, P.; Han, S.; Joung, J.; Belanto, J. J.; Verdine, V.; Cox, D. B. T.; Kellner, M. J.; Regev, A.; Lander, E. S.; Voytas, D. F.; Ting, A. Y.; Zhang, F. RNA Targeting with CRISPR–C...
-
[4132]
(34) Sun, H.; Jiang, Q.; Huang, Y.; Mo, J.; Xie, W.; Dong, H.; Jia, Y
https://doi.org/10.1038/s41598-022-07954-2. (34) Sun, H.; Jiang, Q.; Huang, Y.; Mo, J.; Xie, W.; Dong, H.; Jia, Y. Integrated Smart Analytics of Nucleic Acid Amplification Tests via Paper Microfluidics and Deep Learning in Cloud Computing. Biomed. Signal Process. Control 2023,...
-
[5921]
(8) Gootenberg, J
https://doi.org/10.1038/s41467-020-19097-x. (8) Gootenberg, J. S.; Abudayyeh, O. O.; Lee, J. W.; Essletzbichler, P.; Dy, A. J.; Joung, J.; Verdine, V.; Donghia, N.; Daringer, N. M.; Freije, C. A.; Myhrvold, C.; Bhattacharyya, R. P.; Livny, J.; Regev, A.; Koonin, E. V.; Hung, D...
-
[6355]
(33) Waheed, W.; Saylan, S.; Hassan, T.; Kannout, H.; Alsafar, H.; Alazzam, A
https://doi.org/10.3390/s22176355. (33) Waheed, W.; Saylan, S.; Hassan, T.; Kannout, H.; Alsafar, H.; Alazzam, A. A Deep Learning-Driven Low-Power, Accurate, and Portable Platform for Rapid Detection of COVID-19 Using Reverse-Transcription Loop-Mediated Isothermal Amplificatio...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.