{"id":"242ed6b8-4aeb-4b04-b362-bf8c552730cc","arxiv_id":"2502.15724","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Instruction-tuned Mistral 7B achieves modestly higher F1 than CNN/LSTM on next merchant category prediction, but the evaluation lacks significance tests and the weighted F1 is dominated by an 'Other' class.","lead":"This paper fine-tunes the Mistral 7B large language model on bank transaction histories written as natural language prompts, then tests whether it predicts a customer's next purchase category better than CNN, LSTM, and a simple baseline. The fine-tuned model reports the highest weighted F1 (0.66 vs 0.62 for CNN and 0.60 for LSTM), but the gains are small, the data are private, and no error bars are given.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No error bars or significance tests; Table 2's Clothing F1 swings from 0.62 to 0.12 across lengths, so the claimed Mistral advantage may be sampling noise.","rationale":"The reader's weakest assumption and my concern align: without uncertainty quantification, the reported differences are not demonstrably real. I considered other concerns: the sequence-length comparisons use different ground-truth positions (5th, 8th, 10th, 15th) and different customer subsets, which undermines the 'across all sequence lengths' claim but not the headline last-9 comparison; the 'Other' class dominance affects weighted F1 but Table 2 directly addresses the three categories. The most load-bearing issue is statistical: the single-run evaluation on 1,000 customers cannot support a claim of significance, especially for minority classes. A concrete test with multiple seeds or bootstrap CIs would settle it. If the CI shows overlap, the paper's conclusion would need to be softened; if the advantage persists, the claim is strengthened. Thus conditional acceptance with that requirement is appropriate.","tokens_in":11361,"tokens_out":7041,"duration_ms":57859,"concrete_test":"Re-run the evaluation with at least 10 random seeds (or bootstrap the 1,000 test customers) and report per-class F1 with 95% confidence intervals for all four classes, including 'Other', at each sequence length. The central claim survives only if the lower bound of the CI for (Mistral − best baseline) at last-9 exceeds zero for the three key categories; otherwise, the 'significant outperformance' is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—that fine-tuned Mistral 'significantly outperforms' CNN/LSTM on the three key merchant categories—rests on single-run F1 values in Table 2 and weighted F1 in Table 1. No error bars, confidence intervals, or significance tests are reported anywhere. The instability is visible: for Clothing, Mistral's F1 is 0.62 at last-9 but 0.12 at last-7; for Gas Stations, 0.50 at last-7 vs 0.40 at last-9. With only 1,000 test customers and minority classes at ~11–12% of transactions, per-class support is roughly 110–120 examples, making F1 estimates highly variable. The weighted F1 is dominated by the unshown 'Other' class (45.5% in Bank A, 'similar' in Bank B), yet the paper never reports that class's F1. The claimed 0.04–0.06 overall F1 gap and the per-class gaps could easily arise from sampling noise. Section 2.2 also asserts 'remarkable consistency' that contradicts the observed variance across sequence lengths.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether fine-tuned open-source LLMs can predict the next merchant category of a customer's bank transactions. Transaction and demographic data are converted into natural-language instruction samples, and a Mistral-7B-Instruct-v0.2 model is fine-tuned with LoRA on 8,154 customers from Bank A and tested on 1,000 customers from Bank B. The authors compare this model against an averaging baseline, a CNN, and an LSTM at input sequence lengths of 4, 7, 9, and 14 transactions. The central claim is that the fine-tuned Mistral model achieves a weighted F1 of 0.66 at last-9 versus 0.62 for CNN and 0.60 for LSTM, and that it outperforms the sequential baselines specifically on the Grocery, Clothing, and Gas Stations categories, which the paper attributes to the LLM's semantic understanding and better handling of minority classes.","tokens_in":11590,"tokens_out":3856,"duration_ms":31420,"significance":"If the empirical claims are correct, the paper demonstrates a practical use of open-source instruction-tuned LLMs for a realistic financial prediction task with cross-bank generalization, and it makes a concrete contribution by showing how tabular transaction data can be formatted as natural-language instructions. The study design has notable strengths: a held-out test bank, a clearly described preprocessing pipeline, class-imbalance discussion, and a public code repository. However, the headline comparison currently rests on single-run F1 values without uncertainty quantification, and the class-level results show large fluctuations that the text does not address. The significance of the result is therefore conditional on the authors providing statistical support for the performance gaps.","major_comments":[{"comment":"The central claim that the fine-tuned Mistral model 'significantly outperforms' CNN and LSTM is not supported by any error bars, confidence intervals, or significance tests. The reported advantage at last-9 is 0.66 versus 0.62 and 0.60 on a single test set of 1,000 customers. Given that Clothing and Gas Stations each comprise roughly 11-12% of transactions, per-class support is around 110-120 examples, and the 0.04-0.06 F1 gaps could easily arise from sampling noise. Please report bootstrap confidence intervals, a paired significance test such as McNemar's test, or repeated runs with different seeds and random test samples.","section":"Section 2.1, Table 1"},{"comment":"The assertion of 'remarkable consistency' across categories and sequence lengths is directly contradicted by Table 2. Mistral's Clothing F1 is 0.620 at last-9, 0.480 at last-4, 0.120 at last-7, and 0.220 at last-14, and its Gas Stations F1 ranges from 0.400 to 0.500 across lengths. This variability is of the same order as the claimed advantage over the baselines, so the paper needs to explain this instability or temper the consistency claim.","section":"Section 2.2, Table 2"},{"comment":"The headline weighted F1 scores in Table 1 are dominated by the 'Other' class, which the paper reports as 45.5% of Bank A transactions and 'similar' in Bank B, yet Table 2 reports class-wise F1 only for Clothing, Gas Stations, and Grocery. Without the 'Other' class F1, the overall weighted F1 does not directly support the abstract's claim about the three named categories. Please report class-wise F1 for all four classes and, where possible, per-class statistical tests.","section":"Section 2.2, Table 2"},{"comment":"The evaluation strategy states that models were trained only on last-9 sequences but tested on last-4, last-7, and last-14 sequences. The paper should clarify whether the CNN and LSTM were retrained or adapted for the non-9 lengths, since the results are used to make general claims about 'consistency across different sequence lengths' in Section 2.2.","section":"Section 4.6"}],"minor_comments":[{"comment":"The phrase 'transaction datagrocery' is missing a space; it should read 'transaction data—grocery'.","section":"Abstract"},{"comment":"The text refers to the 'SFFtrainer library'; this appears to be a typo for 'SFTTrainer' from the TRL library.","section":"Section 4.4"},{"comment":"The table layout is difficult to parse: model names appear in the middle of the table, sequence-length labels repeat, and the raw-model rows are interleaved with baseline rows. Please use separate panels per model or otherwise restructure the table for clarity.","section":"Table 1"},{"comment":"The sentence 'The bold scores were depicted by comparing LSTM and CNN with Mistral model' is unclear, and the bold formatting is not visible in the manuscript text. Please state explicitly which comparisons are bolded.","section":"Section 2.1"},{"comment":"The paper uses the word 'significantly' repeatedly in the Discussion and Conclusion without any statistical test; please reserve this term for results with quantified uncertainty.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The core experimental design is sensible and the paper could be a useful empirical study if the statistical uncertainty is addressed. The main risk is that the performance gap is small relative to the observed variance in Table 2, and the authors' consistency claim is not supported by the data as presented. I recommend major revision rather than rejection because the missing analyses are within the paper's scope and do not require a fundamentally different study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one thing genuinely useful: it applies LoRA instruction tuning to tabular bank transaction data for next merchant category prediction, and tests cross-bank generalization (train on Bank A, test on Bank B) with a temporal split. That is a real, if incremental, extension of the LLM-for-recommendation line into a new domain, and the authors include code and honest preprocessing details. The baseline CNN/LSTM comparisons are standard and fairly described.\n\nThe soft spot is not the idea; it's the evidence for the headline claim. The Mistral model does achieve a weighted F1 of 0.66 at last-9 versus 0.62 for CNN and 0.60 for LSTM, but there are no error bars, confidence intervals, or significance tests anywhere in the paper. On a test set of only 1,000 customers, with minority classes around 11% of transactions, that 0.04–0.06 gap could easily be noise. The instability in Table 2 makes this concrete: Mistral's Clothing F1 swings from 0.62 at last-9 to 0.12 at last-7 and 0.22 at last-14. That is not the 'remarkable consistency' the text claims. It suggests the model's advantage is far from robust across sequence lengths.\n\nA second issue is the metric. The weighted F1 is dominated by the 'Other' class, which the authors say is 45.5% of transactions, but they never report class-wise F1 for Other in the main tables. So the headline gain could be coming mostly from the class that the paper itself treats as uninteresting, while the 'three key categories' claim rests on single-run F1 numbers that fluctuate wildly. The authors also say 'significantly outperforms' without any statistical test. These are fixable problems, but they are load-bearing for the central conclusion.\n\nMinor points: the preprocessing choices (top-3 categories plus Other, minimum transaction thresholds) are reasonable but not sensitivity-analyzed, and the private data limits independent replication, though code availability helps. The novelty is modest—this is an application paper, not a methodological advance—but the cross-bank evaluation is a genuine strength.\n\nWho is this for? Researchers working on LLMs for tabular or transactional data, and anyone thinking about how to evaluate instruction-tuned models in small-data, class-imbalanced settings. It deserves a serious referee, but the referee should ask for error bars or bootstrapping, class-wise results including Other, seed robustness, and a tempering of the 'significant' language. I would not desk-reject it, but I would send it back for major revision.","headline":"A legitimate but under-evidenced empirical claim that LoRA-tuned Mistral beats CNN/LSTM on next merchant category prediction; the cross-bank setup is nice, but the paper needs significance testing and class-wise transparency before I'd trust the headline.","tokens_in":12148,"tokens_out":1633,"would_cite":false,"duration_ms":16563,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Mistral-7B on bank transactions written as natural-language instructions predicts a customer's next purchase category, and in this study it outperforms CNN and LSTM baselines on a held-out bank's data.","keywords":["large language models","instruction tuning","LoRA","merchant category prediction","financial transactions","customer purchase behavior","class imbalance","sequential prediction"],"falsifier":"Re-run the Bank A training and Bank B evaluation multiple times, or bootstrap the 1,000 Bank B test customers, and compute confidence intervals for the weighted F1 at sequence length 9; if the CNN's 0.62 and the fine-tuned Mistral's 0.66 intervals overlap, the claimed superiority is not established.","tokens_in":11130,"feed_emoji":"💳","tokens_out":11724,"duration_ms":84827,"temperature":0.7,"pith_summary":"The paper tries to establish that a general-purpose open-source large language model, fine-tuned on bank transactions rewritten as natural-language instruction prompts, can predict a customer's next purchase category more accurately than standard CNN and LSTM sequential models. This matters because next-category prediction is what credit-card rewards and targeted marketing campaigns are built on, and the comparison is deliberately strict: models are trained on one bank's customers and tested on another bank's customers. In the paper's headline result, fine-tuned Mistral Instruct 7B v.2 reaches a weighted F1 of 0.66 on the second bank's data at sequence length 9, versus 0.62 for the CNN and 0.60 for the LSTM, with the largest gains in the clothing, grocery, and gas-station categories. If the result holds, banks could get better minority-class coverage and a way to fold demographics and transaction history into a single natural-language model without hand-built feature engineering.","feed_headline":"Fine-tuned Mistral beats CNN and LSTM at next-category prediction","feed_subtitle":"On a second bank's unseen data, fine-tuned Mistral scores 0.66 weighted F1 versus 0.62 for CNN.","key_machinery":"The load-bearing machinery is instruction fine-tuning of a frozen open-source LLM via LoRA. Each customer's demographic fields and recent transactions are rendered into a first-person natural-language prompt, such as 'I am 48 years old, married male, secondary school graduate, and I work as a private employee', followed by the chronological category and spending list, with the next merchant category as the output. LoRA adds two small trainable update matrices per adapted layer, so the pretrained Mistral weights stay frozen while a low-rank parameter set $\\Theta$ is optimized by maximizing the log-likelihood of the output tokens given the input sequence and preceding tokens. The same prompt format is used for Bank A training and Bank B testing, and the evaluation varies input sequence length at test time (last-4, last-7, last-9, last-14) even though the model was trained only at last-9.","core_discovery":"On its own terms, the paper claims that instruction-fine-tuning Mistral Instruct 7B v.2 with LoRA on Bank A transactions converted to natural-language prompts produces a next-merchant-category predictor that, tested on Bank B, reaches a weighted F1 of 0.66 at sequence length 9, above the CNN at 0.62 and the LSTM at 0.60. The gain is concentrated in the three named merchant categories: at the trained sequence length, clothing F1 is 0.62 versus near zero for the neural baselines, gas stations reaches 0.40 to 0.50 versus roughly 0.20 to 0.24, and grocery reaches 0.52 to 0.59 versus roughly 0.35 to 0.47. The paper attributes the advantage to the LLM's semantic understanding of categories and demographic context, and to its stronger handling of class imbalance in a dataset where grocery, clothing, and gas stations make up 11.2 to 31.3 percent of transactions and the remaining 45.5 percent are lumped as Other.","pith_inferences":["If the class-level differences are taken at face value, the real payoff is in minority-category coverage; the weighted-F1 advantage could be diluted by the unshown 'Other' class, which holds 45.5 percent of transactions.","The sharp swing in the Clothing score across sequence lengths (F1 0.62 at last-9 but 0.12 at last-7) suggests the LLM's advantage may depend on sequence position or prompt formatting, which repeated runs could clarify.","The template-based design makes it cheap to test whether the model is using semantics or surface cues: systematically reword the demographic and amount fields and measure how the category predictions move.","For deployment, the accuracy gain over the CNN and LSTM must be weighed against inference cost and latency; the paper's claims are about predictive accuracy, not operational efficiency."],"forward_implications":["A weighted F1 of 0.66 on Bank B at sequence length 9, versus 0.62 for the CNN and 0.60 for the LSTM, means the fine-tuned LLM is a competitive alternative to bespoke sequential models for next-category prediction.","Class-level results at the trained sequence length show the LLM handling the minority clothing and gas-station categories far better than the neural baselines, which supports the paper's argument about imbalanced financial datasets.","Testing at last-4, last-7, and last-14 despite training only at last-9 suggests the fine-tuned model transfers across input sequence lengths without additional retraining.","Because demographic and transaction information is embedded in the instruction prompt rather than engineered as numeric features, adding new customer attributes would require editing the template rather than redesigning the model."],"supporting_citations":[{"why":"supplies the standard classification baselines and the purchase-category prediction task context against which the LLM is compared.","marker":"[5]"},{"why":"provides the instruction-tuning framework from which the four-step dataset preparation is adapted.","marker":"[9]"},{"why":"defines the low-rank adaptation method that keeps the pretrained weights frozen while training small update matrices.","marker":"[24]"},{"why":"provides the parameter-efficient fine-tuning library used to apply LoRA.","marker":"[25]"},{"why":"provides the supervised fine-tuning trainer used to train the model on instruction-output pairs.","marker":"[26]"},{"why":"is the Mistral-7B-Instruct-v0.2 model that the study fine-tunes.","marker":"[27]"},{"why":"supplies the Bank A dataset used for training and validation.","marker":"[28]"},{"why":"supplies the Bank B dataset used only for testing cross-bank generalization.","marker":"[29]"},{"why":"grounds the instruction-following objective in the InstructGPT approach to training language models to follow instructions.","marker":"[30]"}],"fun_headline_variants":["Fine-tuned Mistral beats CNN and LSTM on bank data","LoRA-tuned Mistral excels at rare merchant categories","Mistral tops CNN and LSTM in cross-bank prediction","Instruction-tuned LLM wins on minority-class F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single-run F1 scores in Tables 1 and 2 are stable enough that the 0.04 to 0.06 weighted-F1 advantage for Mistral and the large class-level gaps reflect real model differences rather than chance.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Mistral beats CNN and LSTM on bank data","LoRA-tuned Mistral excels at rare merchant categories","Mistral tops CNN and LSTM in cross-bank prediction","Instruction-tuned LLM wins on minority-class F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1213,"prompt_tokens":960,"completion_tokens":253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":182}},"tokens_in":576,"tokens_out":253,"duration_ms":3047,"temperature":1.0,"reasoning_tokens":182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:04:12.993665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Bank A training and Bank B evaluation multiple times, or bootstrap the 1,000 Bank B test customers, and compute confidence intervals for the weighted F1 at sequence length 9; if the CNN's 0.62 and the fine-tuned Mistral's 0.66 intervals overlap, the claimed superiority is not established.","supporting_citations":[{"cited_title":"”Forecasting purchase categories by trans- actional data: A comparative study of classification methods.” Lecture Notes in Computer Science","cited_arxiv_id":null,"evidence_quote":"supplies the standard classification baselines and the purchase-category prediction task context against which the LLM is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the instruction-tuning framework from which the four-step dataset preparation is adapted."},{"cited_title":"”PEFT: State-of-the-art Parameter-Efficient Fine- Tuning methods.” GitHub repository","cited_arxiv_id":null,"evidence_quote":"provides the parameter-efficient fine-tuning library used to apply LoRA."},{"cited_title":"”TRL: Transformer Reinforcement Learning.” GitHub repository","cited_arxiv_id":null,"evidence_quote":"provides the supervised fine-tuning trainer used to train the model on instruction-output pairs."},{"cited_title":"”Behavioral attributes and financial churn prediction.” EPJ Data Science 7.1 (2018): 1-18","cited_arxiv_id":null,"evidence_quote":"supplies the Bank A dataset used for training and validation."},{"cited_title":"”Money Walks: Implicit Mobility Behavior and Financial Well-Being.” PLOS ONE 10.8 (2015): e0136628","cited_arxiv_id":null,"evidence_quote":"supplies the Bank B dataset used only for testing cross-bank generalization."}],"review_version":1}