REVIEW 3 major objections 6 minor 28 references
Transaction Categorization with Relational Deep Learning in QuickBooks
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Rel-Cat reframes transaction categorization as link prediction over the QuickBooks relational database and reports 68.67% Top-1 accuracy, beating the production Lynx model's 62.49%.
desk verdict Rel-Cat is a sensible industrial application of relational deep learning with strong ablations, but the paper never clarifies whether the inference graph leaks future test transactions into GNN message passing, which undermines the headline comparison until resolved. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a two-stage pipeline over a heterogeneous graph $G=(V,E,\phi,\psi)$ built from the QuickBooks relational database, where rows are nodes and primary-foreign key relations are edges. Transaction categorization is reformulated as link prediction: a new transaction node enters without an edge to any Category node, and the model scores candidate Category nodes by inner product with the transaction embedding. The pipeline first tries TopK NN, a similarity-based early exit that reuses categories from historical transactions whose cosine similarity with the new transaction exceeds 0.8; if fewer than five distinct categories emerge, a heterogeneous GraphSAGE-style GNN computes the remaining scores. Supporting components include Txn-Bert, a from-scratch transformer text encoder with a custom WordPiece tokenizer trained on transaction descriptions, two-hop transaction-transaction edges with GATv2 aggregation to reduce over-squashing, similarity-based neighbor sampling, and diversity-filtered loss computation with weighted negative sampling.
What would settle it
Take the same 15K test transactions and break out Top-1 accuracy by how often the true category appears in the company's own recent history, for example buckets of 0, 1–2, 3–9, and 10+ prior occurrences. If the Rel-Cat advantage over Lynx concentrates in the 10+ bucket and disappears in the 0–2 buckets, then the reported 68.67% is driven by the early exit's successful retrieval of routine transactions rather than by better generalization to new or rare categories.
Extended reading notes
Core claim
The central claim is that categorization accuracy improves when the model is trained on the full relational structure of the data rather than on flat per-transaction features: converting the four-table QuickBooks database into a heterogeneous graph and defining the task as link prediction lets the model transfer category knowledge across companies and users. The paper reports that in the Few Shot setting Rel-Cat achieves 68.67% Top-1 and 88.04% Top-5, outperforming Lynx's 62.49% Top-1. On categories already seen in a company's history, the TopK NN early exit alone reaches 79.72% Top-1, while for categories unseen in that company's history the GNN component contributes 22.05% Top-1, so the two mechanisms cover complementary regimes. The authors state the approach is being prepared for production deployment.
Load-bearing premise
The evaluation assumes the test set, which uses the two most recent labeled transactions per company after November 2023, represents the real production workload; if those recent transactions are disproportionately routine repeats, the reported accuracy overstates performance on rarer categories and the edge over Lynx may shrink.
Editorial extensions
If this is right
- QuickBooks can retire its per-company logistic regression models (Lynx) and run a single graph model for personalized categorization, cutting maintenance overhead while improving accuracy.
- New companies with little transaction history no longer face a cold start: category knowledge from similar businesses is reachable through the graph, giving 22.05% Top-1 accuracy on categories unseen in a company's own history.
- Users can be offered Top-5 category choices with 88.04% accuracy, potentially increasing trust in automated categorization and reducing manual review.
- Real-time constraints are met: the full Top-5 pipeline takes under one second per 1,000 transactions on GPU, and 68% of Top-1 predictions are resolved by the cheap early exit alone.
- The relational-database-to-graph recipe can be reapplied to other multi-table enterprise prediction tasks, such as invoice line-item classification or fraud rule assignment.
Reading between the lines
- The test set, consisting of the two most recent labeled transactions per company, likely overweights routine and repeated transactions; the reported 68.67% may overstate performance on long-tail categories, so the real production gain could be smaller than the headline.
- TopK NN is essentially a retrieval baseline that reuses categories from similar past transactions, so its high Top-1 score mainly reflects transaction regularity; the GNN's contribution is best measured on categories absent from a company's history, where it reaches only 22.05%.
- The method should transfer to other relational enterprise tasks wherever text attributes plus foreign-key relationships define the prediction target, but each domain would need its own trained-from-scratch tokenizer and a freshly tuned similarity cutoff.
- A live deployment with per-company feedback loops could test whether the Top-5 suggestions actually shift user adoption of automated categorization, a benefit claimed but not directly measured in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Rel-Cat, a graph-based model for transaction categorization in QuickBooks. The authors convert the relational database into a heterogeneous graph, treat categorization as link prediction between transaction and Category nodes, and combine a purpose-built text encoder (Txn-Bert) with a heterogeneous GNN and a TopK nearest-neighbor early exit. On a large private dataset (3M training transactions, 15K held-out test transactions), they report that Rel-Cat achieves 68.67% Top-1 and 88.04% Top-5 accuracy in the Few-Shot setting, outperforming the production Lynx model (62.49% Top-1). They also report ablations supporting each design choice and sub-second GPU inference for 1,000 transactions.
Significance. If the reported results hold, this is a practically significant contribution: a single unified model could replace millions of per-company logistic regression models, with better accuracy and simpler maintenance. The use of a large proprietary dataset, the extensive ablation study (Tables 1, 2, 4), and the attention to inference latency are commendable. However, the validity of the central comparison depends on a clear, leakage-free construction of the inference graph, which the paper does not specify; this needs to be resolved before the superiority claim can be accepted.
major comments (3)
- [Section 3.2, Eq. (1), Section 4.1, Appendix A.4] The paper never states whether the graph used for test-time inference contains the test transactions themselves or any future transactions. Section 3.2 adds E_aug edges between every pair of same-company transactions, and Appendix A.3 builds nodes from all rows; Section 4.1 fixes the test set as the two most recent labeled transactions per company. If the inference graph is built statically over all transactions, then when scoring one test transaction, the other (later) test transaction from the same company is a direct neighbor via E_aug. Even if its Category link is masked, its description, amount, and memo features propagate through Eq. (1) into the target's representation; and if labels are not masked during the TopK NN step (Section 3.4, Appendix A.4), the later transaction can directly supply its Category. This would constitute future-information leakage that Lynx, by construction, cannot access, and could materially inflate the reported 68.67% vs 62.49% Top-1 gap. The authors must specify how the inference graph is restricted to transactions preceding each target (or exclude all test transactions from the graph) and should rerun the evaluation with a clearly chronological graph if the current pipeline includes future transactions.
- [Section 4.1, Table 1] All reported accuracies are single point estimates without error bars, confidence intervals, or significance tests. The text uses 'significantly outperforming' (Section 4.2), but no evidence of statistical significance is provided. Given that the test set has 15K transactions, bootstrap confidence intervals on the accuracy differences would be easy to compute and should be reported; at minimum, results from multiple random seeds would establish that the differences are not due to optimization variance.
- [Section 4.1, Section 4.6] The test set consists only of the two most recent labeled transactions per company. Section 4.6 shows that the TopK NN early exit alone resolves 68% of Top-1 predictions, indicating a high fraction of routine, repeated transactions. If the recent two transactions per company are disproportionately routine, the overall accuracy overstates performance on less frequent categories and on new merchant relationships. To support the production-deployment claim, the authors should report the distribution of transaction frequency and category in the test set, and ideally evaluate on a continuous temporal held-out window with all transactions (or a stratified sample) as well as on the two-most-recent sample.
minor comments (6)
- [Section 4.1] The relationship between the 7.5K test companies and the 15K training companies is not stated; please clarify whether they are disjoint, overlapping, or a subset, since this affects interpretation of the few-shot results.
- [Appendix A.4] The TopK NN step is described as identifying 'similar historical transactions,' but 'historical' is never precisely defined with respect to timestamps; this is related to the inference-graph leakage issue and should be defined explicitly.
- [Table 3] Table 3 reports inference times for 1,000 transactions, but it is unclear whether graph construction and neighbor sampling are included in the reported times; please specify the exact components measured.
- [Section 4.4, Table 2] The ablation 'w/o two-hop connections' yields the best Historical Unseen accuracy (26.93%) while lowering overall accuracy; the text attributes this to a trade-off, but a fuller explanation of why removing the two-hop edges improves unseen-category generalization would strengthen the paper.
- [Section 3.3, Eq. (2)] The loss is described as AUCLoss with a squared hinge form, but the relationship to the cited AUCLoss [21] is not explained; please clarify or cite the specific variant.
- [Throughout] There are several formatting and spacing issues in the text (e.g., '3000K transactions', 'Rel-Cat67') that should be corrected in a revision.
Circularity Check
No significant circularity; the headline accuracy claims rest on held-out evaluation rather than on fitted inputs or self-citation chains.
full rationale
The paper's derivation is empirical, not analytic: the headline numbers in Table 1 are Top-1/Top-5 accuracies computed on a 15K-transaction test split (Section 4.1) after training on a separate 3M-transaction split. No equation in Sections 3.2-3.4 makes the predicted Category equal to an input by construction. TopK NN is a transparent nearest-neighbor early exit: it copies categories from historical transactions whose Txn-Bert cosine similarity exceeds 0.8; this is a legitimate baseline, and its failure on Historical Unseen samples (0.16% in Table 2) shows it is not a tautology. Txn-Bert is a trained encoder, not a fitted parameter renamed as a prediction. The only overlapping-author citation used in the method is FakeEdge [6] (Appendix A.5), adopted as a batch-mode training heuristic and validated by the 'w/o distribution shift mitigations' ablation; its role is a component, and its effectiveness is demonstrated rather than imported by citation. The unresolved '[?]' reference in Appendix B.1 is a completeness defect, not circularity. Potential concerns about future-transaction leakage or test-set representativeness are internal-validity questions about the evaluation, not circular reductions. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (4)
- TopK NN similarity cutoff =
0.8
- Diversity filtering schedule =
100% to 40%
- GNN hyperparameters =
not reported
- Txn-Bert layer count =
6
assumptions (3)
- domain assumption The relational database schema in Figure 1 fully determines the graph structure and the categorization target.
- domain assumption Cosine similarity in Txn-Bert embedding space identifies transactions that should share a category.
- domain assumption The two most recent labeled transactions per company are representative of the production workload.
Cite this review
Pith. "Pith review of Transaction Categorization with Relational Deep Learning in QuickBooks." pith.science (2026). https://pith.science/paper/4DLULUVQ
@misc{pith2026250609234,
author = {Pith},
title = {Pith review of: Transaction Categorization with Relational Deep Learning in QuickBooks},
year = {2026},
howpublished = {\url{https://pith.science/paper/4DLULUVQ}},
note = {Machine review of arXiv:2506.09234}
}
read the original abstract
Automatic transaction categorization is crucial for enhancing the customer experience in QuickBooks by providing accurate accounting and bookkeeping. The distinct challenges in this domain stem from the unique formatting of transaction descriptions, the wide variety of transaction categories, and the vast scale of the data involved. Furthermore, organizing transaction data in a relational database creates difficulties in developing a unified model that covers the entire database. In this work, we develop a novel graph-based model, named Rel-Cat, which is built directly over the relational database. We introduce a new formulation of transaction categorization as a link prediction task within this graph structure. By integrating techniques from natural language processing and graph machine learning, our model not only outperforms the existing production model in QuickBooks but also scales effectively to a growing customer base with a simpler, more effective architecture without compromising on accuracy. This design also helps tackle a key challenge of the cold start problem by adapting to minimal data.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
arXiv:2006.05205 [cs, stat] (Mar 2021), arXiv: 2006.05205
Alon, U., Yahav, E.: On the Bottleneck of Graph Neural Networks and its Practical Implications. arXiv:2006.05205 [cs, stat] (Mar 2021), arXiv: 2006.05205
arXiv 2006
-
[2]
Brody, S., Alon, U., Yahav, E.: How Attentive are Graph Attention Networks? (Jan 2022), arXiv:2105.14491 [cs]
arXiv 2022
-
[3]
Geometric deep learning: going beyond Euclidean data
Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine 34(4), 18–42 (Jul 2017), arXiv: 1611.08097
work page Pith review arXiv 2017
-
[4]
In: Burstein, J., Doran, C., Solorio, T
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)...
work page 2019
-
[5]
Dong, K., Guo, Z., Chawla, N.V.: Pure Message Passing Can Estimate Common Neighbor for Link Prediction (Oct 2023), arXiv:2309.00976 [cs]
work page Pith review arXiv 2023
-
[6]
Dong, K., Tian, Y., Guo, Z., Yang, Y., Chawla, N.: FakeEdge: Alleviate Dataset Shift in Link Prediction (Dec 2022)
work page 2022
-
[7]
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazare, P.E., Lomeli, M., Hosseini, L., Jegou, H.: The Faiss library (2024), _eprint: 2401.08281
arXiv 2024
-
[8]
Feng, J., Chen, Y., Li, F., Sarkar, A., Zhang, M.: How Powerful are K-hop Message Passing Graph Neural Networks (Jan 2023), arXiv:2205.13328 [cs]
work page Pith review arXiv 2023
Show all 28 references
-
[9]
Fey, M., Hu, W., Huang, K., Lenssen, J.E., Ranjan, R., Robinson, J., Ying, R., You, J., Leskovec, J.: Position: Relational Deep Learning - Graph Representation Learning on Relational Databases (Jun 2024)
2024
-
[10]
Frasca, F., Rossi, E., Eynard, D., Chamberlain, B., Bronstein, M., Monti, F.: SIGN: Scalable Inception Graph Neural Networks (Nov 2020), arXiv:2004.11198 [cs, stat]
2020 arXiv
-
[11]
CoRR (2017)
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural Message Passing for Quantum Chemistry. CoRR (2017)
2017
-
[12]
arXiv:1706.02216 [cs, stat] (Sep 2018), arXiv: 1706.02216
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive Representation Learning on Large Graphs. arXiv:1706.02216 [cs, stat] (Sep 2018), arXiv: 1706.02216
2018 arXiv
-
[13]
Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 9365–9372 (Jul 2019), number: 01 18 K
Lesner, C., Ran, A., Rukonic, M., Wang, W.: Large Scale Personalized Categoriza- tion of Financial Transactions. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 9365–9372 (Jul 2019), number: 01 18 K. Dong et al
2019
-
[14]
In: Proceedings of the 27th ACM SIGKDD Confer- enceonKnowledgeDiscovery&DataMining.pp.3299–3307.KDD’21,Association for Computing Machinery, New York, NY, USA (Aug 2021)
Liu, J., Pei, L., Sun, Y., Simpson, H., Lu, J., Ho, N.: Categorization of Financial Transactions in QuickBooks. In: Proceedings of the 27th ACM SIGKDD Confer- enceonKnowledgeDiscovery&DataMining.pp.3299–3307.KDD’21,Association for Computing Machinery, New York, NY, USA (Aug 2021)
2021
-
[15]
In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed Repre- sentations of Words and Phrases and their Compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems....
2013
-
[16]
In: Proceedings of the 2016 Conference on Design, Automation & Test in Europe
Panda, P., Sengupta, A., Roy, K.: Conditional deep learning for energy-efficient and enhanced pattern recognition. In: Proceedings of the 2016 Conference on Design, Automation & Test in Europe. pp. 475–480. DATE ’16, EDA Consortium, San Jose, CA, USA (Mar 2016)
2016
-
[17]
In: Proceedings of the 38th International Conference on Machine Learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. In: Proceedings of the 38th International Conference on Mac...
2021
-
[18]
In: Inui, K., Jiang, J., Ng, V., Wan, X
Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...
2019
-
[19]
In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Schuster, M., Nakajima, K.: Japanese and Korean voice search. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5149–5152 (Mar 2012), iSSN: 2379-190X
2012
-
[20]
In: Advances in Neural Information Processing Systems
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is All you Need. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
2017
-
[21]
arXiv:2112.02936 [cs] (Jan 2022), arXiv: 2112.02936
Wang, Z., Zhou, Y., Hong, L., Zou, Y., Su, H., Chen, S.: Pairwise Learning for Neural Link Prediction. arXiv:2112.02936 [cs] (Jan 2022), arXiv: 2112.02936
2022 arXiv
-
[22]
Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith,...
2016 arXiv
-
[23]
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How Powerful are Graph Neural Net- works? CoRR abs/1810.00826 (2018), arXiv: 1810.00826
2018 arXiv
-
[24]
transaction
Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W.L., Leskovec, J.: Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In: Proceed- ings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 974–983. KDD ’18, Asso...
2018
-
[25]
V: Node set, representing rows across all tables
-
[26]
E ⊆ V × V: Edge set, representing relationships based on primary-foreign key mappings
-
[27]
ϕ : V → T: Node type mapping function, assigning each node to its corre- sponding node type (table)
-
[28]
We then map the elements of the relational database(T , L) to the elements of the heterogeneous graphG = (V, E, ϕ, ψ)
ψ : E → R: Edge type mapping function, linking each edge to its relation type. We then map the elements of the relational database(T , L) to the elements of the heterogeneous graphG = (V, E, ϕ, ψ). We first define the node set in the converted graph as the union of all rows in...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.