{"id":"7e16b351-bde4-48f0-a066-e809d35a41fa","arxiv_id":"2507.01924","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Pseudo-labeling with Isolation Forest lets an LSTM detect billing anomalies with high recall, but the hybrid LSTM-Transformer adds little and reduces precision.","lead":"This paper tests whether combining LSTM and Transformer models, trained on labels generated by unsupervised anomaly detectors, can spot unusual billing in mental healthcare data. In two proprietary datasets, the simpler LSTM trained on Isolation Forest labels performed best, while the hybrid model only improved recall at the cost of precision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation rests on a single confirmed fraud case (349 records, one client), which the paper itself says is too scarce for traditional metrics; the reported recall/precision therefore cannot support the central claim of effective anomaly detection.","rationale":"The reader's weakest assumption focuses on in-sample pseudo-labels. That is a valid concern about the training signal, but it does not directly invalidate the test-set metrics, since the test split is out-of-sample for the unsupervised models under a chronological split. The single-fraud-case ground truth is more load-bearing because it undermines the very measurements used for the headline. The paper itself acknowledges in Section 3.5.1 that a single fraud case is insufficient for traditional metrics, then uses those metrics in Section 4.2 to declare the approach effective. The high recall values (0.963, 0.744) are thus consistent with the model memorizing one client's pattern rather than detecting anomalies generally. This is not a question of implementation hygiene; it is a data limitation that no amount of algorithmic fixing can resolve. Therefore the central claim should be treated as unverified beyond this one case, and the conditional acceptance should hinge on obtaining multi-client labeled fraud data or a public benchmark demonstrating generalizability.","tokens_in":19630,"tokens_out":9954,"duration_ms":113818,"concrete_test":"Compute a per-client decomposition of the test-set true positives: report how many of the 349 fraud records fall in the test split and how many of the model's true positives belong to the single known fraudulent client. If all true positives are from that one client, the reported recall is a one-client retrieval rate. Additionally, re-run the pipeline on a temporally held-out period after removing all records of that client and report precision/recall against any remaining labels; if the positive class disappears or performance collapses, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that pseudo-labeled deep models detect anomalies in mental healthcare billing is evaluated against ground-truth labels from exactly one confirmed fraud case: 349 billing records from a single client out of 202 (Section 3.1). Section 3.5.1 states that with 'a single fraud case unlikely to represent the full range of anomaly patterns, traditional evaluation metrics were not applicable.' Despite this, Section 4.2 reports precision, recall, and F1 on the test set and concludes that iForest LSTM achieves recall 0.963 (declaration-level) and the hybrid iForest model 0.744 (operation-level). With a single positive client, these numbers describe retrieval of that one client's records, not generalizable fraud detection. The t-SNE and top-anomaly analyses (Section 4.1) show one client dominating the flagged anomalies, so the model may be exploiting client-specific billing signatures (e.g., early-payment features that dominate SHAP values) rather than learning a general anomaly concept. The in-sample pseudo-label generation (Section 3.5.1: unsupervised models fit on 80% of data that includes the downstream training split) is a related methodological flaw, but even fully out-of-sample pseudo-labeling would not rescue the evaluation from the single-case ground truth. The paper's own admission makes this the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid LSTM-Transformer anomaly detection framework for mental healthcare billing data, using pseudo-labels generated by Isolation Forest (iForest) and Autoencoder (AE) models to mitigate extreme label scarcity. The authors evaluate their approach on two proprietary datasets—declaration-level (71,196 records) and operation-level (1,589,951 records)—and report that the iForest-pseudo-labeled LSTM achieves the highest recall on declaration-level data (0.963) and that the hybrid iForest-based model achieves the highest recall on operation-level data (0.744), while all models trained on original labels fail to detect anomalies. The paper includes pseudo-code for the pseudo-labeling algorithms, detailed hyperparameter tables, SHAP interpretability analyses, and McNemar significance tests. The central claim is that pseudo-labeling with unsupervised detectors enables deep sequential models to detect billing anomalies under extreme label scarcity, with the iForest LSTM being the best overall performer.","tokens_in":19937,"tokens_out":4669,"duration_ms":50192,"significance":"The problem is practically relevant, and the use of two real-world proprietary billing datasets is a strength. The paper is transparent about its methodology: it provides algorithm pseudocode, hyperparameter grids, and extensive appendices, which support reproducibility within the constraints of private data. However, the evaluation is severely limited by three intertwined problems: ground-truth evaluation rests on a single confirmed fraud case, pseudo-labels for the training data are generated in-sample by the unsupervised detectors, and the original-label baselines score zero because the test splits lack positive labels. These issues mean the reported precision, recall, and F1 values do not support the general claim that the proposed approach provides effective anomaly detection in mental healthcare billing. The paper is best viewed as a case study or proof-of-concept; as it stands, it does not validate a general method, though it does illustrate a concrete pipeline and highlights challenges that future work should address.","major_comments":[{"comment":"The evaluation rests on a single confirmed fraud case: 349 billing records from one client out of 202. Section 3.5.1 explicitly states that with a single fraud case, traditional evaluation metrics were not applicable, yet Section 4.2 reports precision, recall, and F1 on the test set and uses these to conclude that iForest LSTM achieves the highest recall (0.963) on declaration-level data and the hybrid iForest model 0.744 on operation-level data. With only one positive client, these metrics measure retrieval of that one client's records, not generalizable fraud detection. This is the load-bearing weakness: even fully out-of-sample pseudo-labeling would not rescue the evaluation from the single-case ground truth. The authors should either reframe the paper as a case study with explicit caveats, or provide additional validation such as domain-expert review of flagged anomalies, precision-at-k analyses, or controlled experiments with synthetic anomalies.","section":"§3.1, §3.5.1, §4.2, Tables 1 and 2"},{"comment":"Pseudo-labels for the training data are generated in-sample by the unsupervised models. The unsupervised detectors are fit on an 80/20 split, while the LSTM and Transformer models use a separate 60/20/20 chronological split; the training portion of the downstream split is a subset of the unsupervised models' training data. Consequently, the pseudo-labels used to train the deep models are in-sample predictions from iForest and the AE, which can inflate the apparent quality of the pseudo-labels and cause the downstream model to partially replicate the unsupervised detector rather than learn generalizable patterns. The authors should generate pseudo-labels on a held-out split (or use a nested procedure) and report how performance changes when the pseudo-labels are out-of-sample.","section":"§3.5.1 and §3.5.2"},{"comment":"The original-label baselines score zero across all metrics, and the text states that in several test splits no labeled anomalies were present. This makes the comparison 'pseudo-labeling outperforms original-label training' unsupported by the reported numbers, because the baselines are evaluated against test sets that cannot contain any true positives. A meaningful comparison would require test splits that include the known fraud records, or evaluation metrics that do not depend on the presence of positives in the test set. As reported, the zero scores are an artifact of the evaluation split, not evidence about the value of pseudo-labeling.","section":"§4.2.5, Tables 1 and 2"},{"comment":"The 1.6% contamination threshold is set by hand ('based on domain expertise'), and the AE anomaly threshold is the same 1.6% percentile. This is a free parameter that directly controls the pseudo-label positive rate, yet no sensitivity analysis is reported. The claimed performance differences between iForest-based and AE-based models, and between pseudo-labeling and original labels, could be driven by this arbitrary threshold. The authors should report results across a range of contamination rates or justify the chosen value with external information.","section":"§3.5.1, Table 3"},{"comment":"The paper states that 'all models were run multiple times using the optimal threshold,' but no error bars, standard deviations, or confidence intervals are reported for the precision, recall, or F1 values. Without variance estimates, it is impossible to assess whether differences such as the iForest LSTM's recall of 0.963 versus the AE LSTM's 0.705 are statistically meaningful. The authors should report means and standard deviations across runs, and ideally bootstrap confidence intervals for the metrics.","section":"§4.2, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'to the best of the authors’ knowledge, the combination of RNNs and Transformer models for anomaly detection has not yet been explored' is too strong given that the related work itself cites Ileberi and Sun [18], who combined LSTM, CNN, and Transformers for credit card fraud detection. Please soften the novelty claim to specify that the specific LSTM-Transformer hybrid on pseudo-labeled healthcare billing data has not been studied.","section":"§2.3"},{"comment":"Several numerical entries in Tables 13 and 14 contain spacing errors such as '6 .35e-3' and '3 .93e8'; these should be corrected to standard decimal notation.","section":"Appendix B, Tables 13 and 14"},{"comment":"The SHAP analysis is applied only to the iForest LSTM model; the text would benefit from a sentence explaining how the SHAP results connect to the anomaly detection claim, particularly because the top features (payment term, early payment) may reflect a client-specific billing profile rather than a general anomaly concept.","section":"§4.3"},{"comment":"The sliding window is applied without grouping by entity, which is a reasonable choice for learning global patterns, but the paper does not discuss the risk that consecutive windows from the same billing file create temporal leakage between the training, validation, and test splits. Please clarify whether windows from the same entity or declaration file can cross split boundaries.","section":"§3.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, but the abstract and conclusion overstate the findings. The single-case ground truth makes the quantitative claims fragile, and the in-sample pseudo-label generation is a separate but fixable flaw. In my view, a major revision should either substantially temper the claims—framing the contribution as a case study or proof-of-concept—or add external validation such as expert review of flagged anomalies, additional confirmed fraud cases, or controlled experiments with known injected anomalies. The paper's strengths are its detailed methodology reporting and its use of real-world data, but those strengths do not compensate for the load-bearing evaluation gap. If the authors can obtain additional ground truth or re-frame the paper's scope, the work could be publishable as a case study; otherwise, the current evidence is insufficient for a general method claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the bottom line: the paper applies known semi-supervised anomaly detection machinery to a new domain, and the evaluation is too weak to support the central claim. The new bit is the application—pseudo-labeling via Isolation Forest and Autoencoder to train LSTM, Transformer, and stacked LSTM-Transformer models on two real mental-healthcare billing datasets. That is a legitimate research question, and the paper does several things well: careful preprocessing, chronological splits, two datasets, McNemar tests on architecture differences, SHAP analysis, and a thorough hyperparameter appendix. The iForest-LSTM baseline performing best is an honest, non-obvious finding.\n\nThe problem is that the evaluation rests on a single confirmed fraud case: 349 billing records from one client out of 202. Section 3.5.1 explicitly says 'traditional evaluation metrics were not applicable' because of this, yet Section 4.2 reports precision, recall, and F1 on the test set and uses them to conclude the models are effective. Those numbers describe whether the model retrieves that one client's records, not whether it detects fraud in any generalizable sense. The t-SNE and top-anomaly analysis reinforce this: one client dominates the flagged anomalies. A second serious issue is the in-sample pseudo-labeling: the unsupervised models are fit on 80% of the data that includes the downstream training split, so the pseudo-labels on training data are in-sample predictions, inflating their apparent quality. The original-label baselines score zero because the test splits happen to contain no positive labels, so the pseudo-label comparison is not informative. The hybrid model's advantage is also a recall-for-precision trade-off, not a clear win.\n\nNone of this kills the idea, but it does undermine this version of the paper. The authors are transparent about limitations and write clearly, but the contradiction between their own stated limitation and the reported metrics makes the paper incoherent on its own terms. The fix requires either more labeled fraud cases or a carefully argued synthetic evaluation, plus out-of-sample pseudo-labeling and variance reporting. I'd send it to a serious referee because the domain is important and the flaws are identifiable and fixable, but I wouldn't cite it yet.","headline":"A competent application paper whose evaluation rests on a single fraud case and in-sample pseudo-labels, so the central claim is not supported; the authors themselves admit the metrics are inapplicable before reporting them.","tokens_in":20420,"tokens_out":3634,"would_cite":false,"duration_ms":38826,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In mental-healthcare billing data, pseudo-labels generated by unsupervised anomaly detectors allow LSTM, Transformer, and hybrid models to catch anomalies, while models trained on the scarce real fraud labels detect nothing.","keywords":["anomaly detection","deep learning","hybrid models","mental healthcare","pseudo-labeling","Isolation Forest","LSTM","Transformer"],"falsifier":"Hold out the 349 billing records tied to the single confirmed fraudulent client, train the iForest-LSTM on pseudo-labels from the remaining data, and measure recall on those confirmed records; if recall is near zero while pseudo-labeled test recall stays high, the model is mostly reproducing the unsupervised detector rather than finding real fraud.","tokens_in":19441,"feed_emoji":"🩺","tokens_out":5855,"duration_ms":61473,"temperature":0.7,"pith_summary":"This paper tries to establish that pseudo-labeling can overcome the label scarcity that blocks deep learning for mental-healthcare billing fraud detection. The authors use Isolation Forest and Autoencoder anomaly detectors to assign synthetic labels to billing records, then train LSTM, Transformer, and stacked hybrid models on those labels. They report that the iForest-pseudo-labeled LSTM reaches 0.963 recall on declaration-level data and that the hybrid iForest model reaches 0.744 recall on operation-level data, while every model trained on the original labels scores zero. If the results hold, they would give payers and auditors a way to build high-recall screening systems without waiting for confirmed fraud labels.","feed_headline":"Computer-made labels teach LSTM to catch 96% of billing anomalies","feed_subtitle":"On scarce-label mental-health billing data, unsupervised labels beat training on real fraud labels.","key_machinery":"The carrying mechanism is an unsupervised pseudo-labeling pipeline. Isolation Forest scores each record by how quickly random isolation trees separate it, and an Autoencoder scores it by reconstruction error; both flag the most extreme 1.6% as anomalous, producing pseudo-labels without human fraud labels. Chronological sliding windows of length 100 (or 50 on operation-level data) then feed a two-layer LSTM and a Transformer encoder adapted from the Anomaly Transformer with a classification head replacing reconstruction, and a logistic-regression meta-learner stacks the two base models' label predictions. This machinery matters because it converts a label-scarce anomaly-detection problem into a supervised sequence-classification problem.","core_discovery":"The central claim is that unsupervised pseudo-labels are a usable substitute for scarce fraud labels in this domain. On the declaration-level dataset, the Isolation-Forest-labeled LSTM achieves the best balance across all metrics (0.999 accuracy, 0.995 precision, 0.963 recall, 0.959 F1). On the more complex operation-level dataset, the same model keeps the best overall balance (0.996 accuracy, 0.918 precision, 0.699 recall), while the hybrid iForest-based model maximizes recall at 0.744 with lower precision (0.279). The paper also reports that autoencoder pseudo-labels separate better in t-SNE but produce worse classifiers, that original-label models fail completely, and that LSTM and Transformer errors differ significantly even when trained on identical pseudo-labels.","pith_inferences":["Because the unsupervised detectors are fit on the same records used to train the downstream classifiers, the pseudo-labels on the training split are in-sample predictions; reported recall may partly measure how well the LSTM replicates the iForest detector rather than how well it generalizes to new fraud.","The known fraudulent client's 349 declaration records and 84 operations are a natural held-out gold standard; the paper does not report recall on them, so the method's ability to catch confirmed fraud remains untested.","The same pipeline could be tried in adjacent billing settings such as dentistry, outpatient surgery, or insurance claims, but the paper's own discussion implies it would need retraining because billing codes and policies differ.","A direct test of the pseudo-labeling assumption would compare it against a modest number of expert-reviewed labels at equal annotation cost, to see whether unsupervised pseudo-labels genuinely beat a small amount of human supervision."],"forward_implications":["On declaration-level billing data, the iForest-pseudo-labeled LSTM reaches 0.995 precision and 0.963 recall, so a single LSTM trained on unsupervised labels could serve as a high-sensitivity first-pass screener.","On operation-level data, the hybrid iForest model reaches 0.744 recall but only 0.279 precision, meaning recall-oriented deployment would flood reviewers with false positives unless a precision filter is added.","Models trained with the original fraud labels score zero on test metrics, so pseudo-labeling is not a small improvement but a precondition for these architectures to learn anything in this setting.","Since McNemar's test shows LSTM and Transformer errors differ significantly under identical pseudo-labels, architecture choice changes which anomalies are found even when the training labels are the same.","Autoencoder pseudo-labels look cleaner in t-SNE plots yet train worse classifiers, so visual label separation is not a reliable predictor of downstream anomaly-detection performance."],"supporting_citations":[{"why":"Cited for Isolation Forest's ability to isolate anomalies in high-dimensional data, the basis of the iForest pseudo-labeling branch.","marker":"[46]"},{"why":"Provides the adapted Anomaly Transformer encoder architecture whose attention mechanism is reused with a classification head.","marker":"[47]"},{"why":"Documents the hybrid LSTM-Transformer ensemble approach that the paper adapts to billing data, and whose claim of hybrid superiority the paper does not reproduce.","marker":"[18]"},{"why":"Shows LSTM networks succeeding at fraud detection in sequential financial data, the empirical precedent for the iForest-LSTM baseline.","marker":"[6]"},{"why":"Establishes the extreme class-imbalance challenge that motivates pseudo-labeling and explains why original-label models score zero.","marker":"[22]"},{"why":"Supplies the coarse-to-fine pseudo-labeling framework that the paper's unsupervised pseudo-labeling strategy extends.","marker":"[2]"},{"why":"Provides an unsupervised pseudo-label refinement method that supports the use of internally consistent pseudo-labels in imbalanced settings.","marker":"[19]"}],"fun_headline_variants":["Pseudo-labels teach LSTM to spot 96% of billing fraud","Unsupervised labels rival scarce fraud labels in billing anomaly detection","Isolation Forest pseudo-labels boost LSTM recall to 96%","AI finds billing anomalies with pseudo-labels when real labels are scarce"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-labels produced by the Isolation Forest and Autoencoder are a valid learning signal, even though the unsupervised detectors were fit on the same records that later train the LSTM and Transformer.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-labels teach LSTM to spot 96% of billing fraud","Unsupervised labels rival scarce fraud labels in billing anomaly detection","Isolation Forest pseudo-labels boost LSTM recall to 96%","AI finds billing anomalies with pseudo-labels when real labels are scarce"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2569,"prompt_tokens":879,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1613}},"tokens_in":495,"tokens_out":1690,"duration_ms":12630,"temperature":1.0,"reasoning_tokens":1613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:40:11.464910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out the 349 billing records tied to the single confirmed fraudulent client, train the iForest-LSTM on pseudo-labels from the remaining data, and measure recall on those confirmed records; if recall is near zero while pseudo-labeled test recall stays high, the model is mostly reproducing the unsupervised detector rather than finding real fraud.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited for Isolation Forest's ability to isolate anomalies in high-dimensional data, the basis of the iForest pseudo-labeling branch."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the hybrid LSTM-Transformer ensemble approach that the paper adapts to billing data, and whose claim of hybrid superiority the paper does not reproduce."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LSTM networks succeeding at fraud detection in sequential financial data, the empirical precedent for the iForest-LSTM baseline."},{"cited_title":"Challenges and Complexities in Machine Learning based Credit Card Fraud Detection","cited_arxiv_id":"2208.10943","evidence_quote":"Establishes the extreme class-imbalance challenge that motivates pseudo-labeling and explains why original-label models score zero."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the coarse-to-fine pseudo-labeling framework that the paper's unsupervised pseudo-labeling strategy extends."},{"cited_title":"On Pseudo-Labeling for Class-Mismatch Semi-Supervised Learning","cited_arxiv_id":"2301.06010","evidence_quote":"Provides an unsupervised pseudo-label refinement method that supports the use of internally consistent pseudo-labels in imbalanced settings."}],"review_version":1}