{"id":"1584d1b1-7d30-4378-a4e0-8748db933a16","arxiv_id":"2412.10941","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"APAR pre-trains a tabular transformer on arithmetic combinations of target labels and fine-tunes it with adaptive feature masking, beating GBDT and neural baselines on 10 regression datasets.","lead":"APAR is a two-stage training method that helps neural networks predict numeric outcomes from tables, such as house prices, emissions, or flight fares. The first stage pre-trains on arithmetic relationships between example labels; the second stage fine-tunes with a self-learned feature masking regularizer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline benchmark gains may be inflated by an unspecified per-dataset choice of arithmetic operator; Section 4.2 and Algorithm 1 leave the selection rule undefined, and Table 4 shows the best operator varies by dataset.","rationale":"The reader's weakest assumption is exactly the arithmetic-operator selection protocol, and I agree that it is the most load-bearing issue. The paper itself contains the evidence: Section 4.2's footnote defers operator choice to empirical selection; Algorithm 1 leaves it as an unstated rule; and Section 5.5 shows performance varies across operators and datasets. Without a stated rule, the reported APAR numbers in Table 2 cannot be interpreted as a single method's performance. The concrete test I propose would settle this by forcing a validation-based selection and comparing the resulting test RMSEs. I do not think the concern justifies moving from CONDITIONAL to REJECT, because the framework is clearly described, the code is public, and the ablations broadly support the design; however, the headline benchmark claim remains contingent on the selection protocol being honest and pre-specified. The ER discrepancy between Table 2 (0.0338) and Table 3 APAR (0.0266) is an additional reporting inconsistency that should be checked when the operator-selection protocol is clarified, and standard deviations over the 5 seeds should be reported for all key numbers.","tokens_in":15083,"tokens_out":4228,"duration_ms":34960,"concrete_test":"Re-run the experiment with an explicit, pre-registered operator-selection rule: split each dataset's training data into an inner validation split, train APAR with each of the four operators (+, −, ×, ÷) on the training portion, select the operator with lowest validation RMSE, then evaluate on the official test set. Compare these test RMSEs with Table 2. If the validation-selected operators differ from those used in Table 2 on any dataset, or if the average improvement over the second-best baseline drops below the claimed 9.18–9.43%, the headline claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—APAR improves RMSE by 9.43–20.37% over GBDT-, supervised NN-, and pretrain-finetune baselines—rests on Table 2, which reports one APAR number per dataset. The paper never specifies how the arithmetic operator was selected for each dataset. Section 4.2's footnote says 'We empirically chose arithmetic operations in our experiments,' and Algorithm 1 (line 2) says 'Select an appropriate arithmetic operation' without defining the selection rule. Section 5.5 (Table 4) reports per-operator results showing that the best operator varies by dataset (e.g., Addition is best on AM/HS/PM/FP, Multiplication on BD/GS/BS/YE; Multiplication also gives 0.0266 on ER, better than the 0.0338 reported for APAR in Table 2). If the operator was chosen after inspecting test-set performance, or after many validation comparisons without multiplicity control, the reported gains are the selected maximum over four choices, not the performance of a single pre-specified method. The 'consistently superior' claim would then be overstated. This is the most load-bearing uncertainty because the entire benchmark comparison depends on it; missing standard deviations and the ER value inconsistency between Table 2 and Table 3 are secondary but reinforce the need for a precise protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APAR, a two-stage framework for tabular regression. In pre-training, pairs of samples are encoded by a Transformer with a feature tokenizer, and the model predicts the result of an arithmetic operation (addition, subtraction, multiplication, or division) applied to the two labels. In fine-tuning, the model is regularized by enforcing prediction consistency between an original sample and a version augmented with a gate vector drawn from a relaxed multivariate Bernoulli distribution, with a sparsity penalty on the gate. The method is evaluated on 10 tabular regression datasets against GBDT, supervised neural, and pretrain-finetune baselines, reporting average RMSE improvements of 9.43% to 20.37%, with ablations for the pre-training task, the adaptive regularization, and the choice of arithmetic operator.","tokens_in":15401,"tokens_out":4707,"duration_ms":40486,"significance":"If the reported gains are robust, APAR is a practically relevant recipe: it provides a simple arithmetic pretext task that uses continuous labels, an adaptive and structure-aware augmentation scheme, and a consistent empirical pipeline across 10 datasets with public code and data. The paper also includes ablations that isolate the pre-training task and the regularization component. However, the central benchmark claim is not yet fully supported because the per-dataset arithmetic operator is chosen without a stated protocol, no standard deviations are reported despite the stated use of 5 seeds, and several table entries for APAR itself are inconsistent across tables. These issues must be fixed before the headline 'consistently superior' claim can be accepted.","major_comments":[{"comment":"The per-dataset arithmetic operator is the most load-bearing unstated choice. The footnote in §4.2 says 'We empirically chose arithmetic operations in our experiments,' and Algorithm 1 line 2 says 'Select an appropriate arithmetic operation' without defining the selection rule. Table 4 shows that the best operator varies by dataset and, for ER, Multiplication achieves 0.0266 while the APAR entry in Table 2 is 0.0338 (the Addition value). If the operator was selected after inspecting test-set results, the reported gains are a post-selected maximum over four choices, not the performance of a single pre-specified method. The authors must specify a pre-registered or validation-based selection rule, report the operator chosen for each dataset, and present the resulting numbers; otherwise the headline 9.43–20.37% improvement is not established.","section":"§4.2, Algorithm 1, §5.5"},{"comment":"The paper states that 'All the results are the average of 5 different random seeds' but reports no standard deviations, per-seed values, or significance tests anywhere. Many adjacent entries in Table 2 differ by under 1% (e.g., BD: APAR 0.2397 vs. TabNet 0.2404), so without variance information the 'consistently superior' ranking claim cannot be distinguished from noise. Please report mean ± std or equivalent, and ideally a paired significance test or rank stability analysis across seeds.","section":"§5.1, Tables 2–5"},{"comment":"The RQ3 discussion claims that removing adaptive-regularized fine-tuning 'degrades the performance across all datasets.' This is contradicted by Table 3: on ER, both w/o AR and APAR report 0.0266, so the improvement is exactly zero there. The claim should be revised to reflect the actual per-dataset differences, and the ER row should be discussed explicitly.","section":"§5.4, Table 3"},{"comment":"The APAR results are internally inconsistent across tables. Table 2 lists APAR as ER=0.0338, BS=0.0139, PM=0.5239; Table 3 lists the same APAR row as ER=0.0266; and Table 5 lists APAR as ER=0.034, BS=0.014, PM=0.524. If these are different runs or configurations, this must be stated; otherwise the reported numbers should be unified. This discrepancy directly affects the reproducibility of the main result.","section":"Tables 2, 3, and 5"}],"minor_comments":[{"comment":"The phrase 'outperforms ... in RMSE (+9.43% ∼ 20.37%)' is ambiguous about direction; since lower RMSE is better, please write 'reduces RMSE by 9.43%–20.37%.'","section":"Abstract and §5.2"},{"comment":"There is a typo: 'supervised lavels' should be 'supervised labels.'","section":"Related Work"},{"comment":"The notation 'TabNet*' and 'TabNet' is confusing: the group is titled 'NN-based with a Pretrain-Finetune,' but the footnote says '* denotes without pre-training,' and TabNet appears both starred and unstarred. Please clarify which TabNet variant is in which group.","section":"Table 2"},{"comment":"The phrase 'arithmetic-related pertaining tasks' should be 'arithmetic-related pre-training tasks.'","section":"§5.2"},{"comment":"The division operator yi/yj has no guard for yj = 0; please clarify whether the log-scaled labels are always positive or how zero labels are handled.","section":"Eq. (8)"},{"comment":"The sentence 'achieving an average RMSE improvement of 9.18% compared to the second-best ranking method' is not directly derivable from Table 2; please specify which baseline is used for each dataset and how the average is computed.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the core idea is reasonable, but the experimental protocol needs significant tightening before publication. The arithmetic-operator selection issue is the main correctness risk; it must be resolved with a clear validation-based rule. I would also ask the authors to provide the code for the operator-selection step as part of the public repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the arithmetic-aware pre-training idea is genuinely new and worth engaging with, but the headline benchmark numbers are not trustworthy until the authors specify how they chose each dataset's arithmetic operator. Table 2's APAR numbers look like the selected maximum over four operator choices rather than a single pre-specified method.\n\nThe paper proposes two things: a pretext task where the model sees two samples and predicts the result of an arithmetic operation (+, -, ×, /) on their labels, and a fine-tuning regularizer that learns a correlated Bernoulli mask via a Gaussian copula. The first is the real contribution. Using continuous labels to build sample-pair supervision is a neat way to inject label structure without discrete classes, and the ablations show it beats feature reconstruction and mask reconstruction. The copula gate is a plausible improvement over independent Bernoulli masks. The writing is clear and the public code is a plus.\n\nThe soft spots are real and one is load-bearing. Section 4.2 says they \"empirically chose arithmetic operations\" and Algorithm 1 just says \"Select an appropriate arithmetic operation.\" Table 4 shows the best operator varies by dataset, so without a pre-specified selection rule (validation-based, with multiplicity control) the reported 9.43–20.37% improvement over baselines is the selected maximum of four configurations. That is not the same as the performance of a single method. There is also an internal inconsistency: Table 2 reports APAR ER=0.0338, Table 3 reports 0.0266; Table 4 shows Addition gives 0.0338 and Multiplication gives 0.0266, so the main table may have used different operators without saying so. Missing standard deviations and the RQ3 \"across all datasets\" overstatement (ER is equal, not improved) are minor in comparison.\n\nWho is this for: anyone working on deep learning for tabular regression. The idea is strong enough that a referee should ask for the selection protocol and variance before taking the numbers at face value. I'd send it out rather than desk reject, but the revision must be substantive.\n\nRecommendation: engage, but ask for a precise operator-selection protocol applied to validation data only, std deviations, a reconciliation of the ER values, and a corrected RQ3 sentence.","headline":"The arithmetic-aware pre-training idea is genuinely novel and worth pursuing, but the headline RMSE gains are not believable until the authors disclose how they picked each dataset's arithmetic operator.","tokens_in":15873,"tokens_out":3683,"would_cite":true,"duration_ms":28428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes APAR, a two-stage recipe that pre-trains a tabular encoder to predict arithmetic results on continuous labels and then fine-tunes it with adaptive gate-based augmentation, claiming RMSE gains of 9.43–20.37% over tree…","keywords":["tabular regression","arithmetic-aware pre-training","adaptive regularization","continuous labels","self-attention","gradient-boosted decision trees","data augmentation","irregular target functions"],"falsifier":"Run the full APAR protocol again on the same ten datasets with the arithmetic operator selected for each dataset using only validation data, and compare average RMSE to the same baselines; if the advantage disappears or falls to within run-to-run variation, the headline gains came from test-set-dependent operator selection.","tokens_in":14896,"feed_emoji":"📊","tokens_out":8729,"duration_ms":73079,"temperature":0.7,"pith_summary":"APAR is a bet that the reason neural networks lose to gradient-boosted trees on tabular regression is not feature encoding but irregular target functions: small feature shifts that produce large label jumps. The paper proposes a two-stage cure: pre-train a self-attention encoder to predict the result of an arithmetic operation on the continuous labels of two randomly paired samples, then fine-tune it under a consistency regularizer that augments each sample with a feature gate sampled from a learnable correlated distribution. The claim is that this uses the continuous labels far more directly than reconstruction-based or contrastive pre-training, which either ignore labels or need discrete classes. On ten regression datasets spanning housing, emissions, elections, air quality, bike sharing, music, kernel performance, and flight prices, the paper reports RMSE improvements of 9.43% to 20.37% over GBDT-based, supervised neural, and pretrain-finetune neural baselines.","feed_headline":"Arithmetic pretraining lets neural nets surpass tree models on tables","feed_subtitle":"A two-stage recipe reports 9.43–20.37% lower RMSE across ten regression datasets.","key_machinery":"Two mechanisms carry the argument. The arithmetic-aware pretext objective turns the continuous label into a supervised signal about sample relationships: from representations of two samples, the model must output $y_i \\odot y_j$, so the encoder has to organize the representation space so that label arithmetic is readable from concatenated CLS tokens. The adaptive regularizer is a relaxed multivariate Bernoulli gate: $\\tilde{Z}_i = Z_i \\odot \\tilde{m}$, with $\\tilde{m}$ sampled under a correlation matrix $R$ and learnable inclusion probabilities $\\pi$, trained so the augmented view and original view give consistent labels while $\\sum \\pi_i$ is penalized. This gives a differentiable, correlation-aware form of learned data augmentation.","core_discovery":"On the paper's own terms, the discovery is that arithmetic can act as a bridge between continuous labels and self-supervised representation learning. Pairs of samples are encoded; the concatenated CLS vectors must predict $y_i \\odot y_j$ for a chosen operator $\\odot \\in \\{+, -, \\times, /\\}$; after this pre-training, the model is fine-tuned by requiring the prediction from an augmented view—embeddings multiplied by a relaxed multivariate Bernoulli gate with learnable inclusion probabilities and a data-derived correlation matrix—to match the prediction from the original view. The paper argues that the first stage builds sample-wise awareness of label geometry, and the second stage keeps the model from latching onto uninformative features by letting the network learn which features to corrupt. The central evidence is that both stages contribute and that arithmetic pre-training beats reconstruction and mask pre-training on all ten datasets.","pith_inferences":["The per-operator study implies that no single arithmetic operator is always best; a validation-based selector for $\\odot$ is the most direct extension, and its success would make the method less dependent on human choice.","Because the pre-training signal only needs pairs of continuous labels, the same objective could be attached to different tabular encoders or combined with reconstruction tasks; the ablations suggest arithmetic and reconstruction capture different information.","The paper motivates 'irregular target functions' without defining them quantitatively; a formal irregularity measure could predict which datasets benefit most from APAR.","The learned gate probabilities $\\pi$ can be read as feature-importance estimates; comparing them with tree-based importances would test whether the adaptive regularizer is discovering the same features that drive predictions."],"forward_implications":["Pretrain-finetune neural methods can move from classification-style tabular benchmarks to regression without discarding continuous label information.","Using the arithmetic pretext task in place of feature-reconstruction or mask-reconstruction pre-training improves RMSE on all ten datasets tested.","The adaptive correlated gate is what turns fine-tuning into a regularized problem: without it, performance degrades across all datasets.","Addition and multiplication are the safe pre-training operators; subtraction is weaker and division often fails to converge, so practitioners can default to one of the two."],"supporting_citations":[{"why":"Establishes the motivating premise that irregular target functions make neural networks overfit tabular regression data.","marker":"Beyazit et al. 2023"},{"why":"Supplies the FT-Transformer baseline and the default backbone and hyperparameter configuration that APAR adopts.","marker":"Gorishniy et al. 2021"},{"why":"Provides the self-attention encoder used to build contextualized feature representations.","marker":"Vaswani et al. 2017"},{"why":"Contributes the relaxed multivariate Bernoulli distribution and reparameterization used for the adaptive gate.","marker":"Wang and Yin 2020"},{"why":"Motivates the feature-tokenizer design and the tree-versus-deep-learning gap that the paper targets.","marker":"Grinsztajn, Oyallon, and Varoquaux 2022"},{"why":"Defines TabNet, the main pretrain-finetune neural baseline that APAR compares against.","marker":"Arik and Pfister 2021"},{"why":"Defines VIME, the feature-corruption pretraining baseline that arithmetic pre-training is ablated against.","marker":"Yoon et al. 2020"},{"why":"Supplies XGBoost, the representative GBDT baseline.","marker":"Chen and Guestrin 2016"}],"fun_headline_variants":["Arithmetic pre-training boosts tabular regression by up to 20% RMSE","Neural nets beat GBDT on tables with arithmetic pre-training","Self-supervised arithmetic cuts regression error on tabular data by 9–20%","APAR: arithmetic-aware pretraining improves tabular regression across 10 datasets","Two-stage arithmetic pretraining beats tree models on tabular regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the arithmetic operation for each dataset was picked by a validation-based rule rather than by looking at test-set RMSE.","fun_headline_variants_meta":{"raw":{"variants":["Arithmetic pre-training boosts tabular regression by up to 20% RMSE","Neural nets beat GBDT on tables with arithmetic pre-training","Self-supervised arithmetic cuts regression error on tabular data by 9–20%","APAR: arithmetic-aware pretraining improves tabular regression across 10 datasets","Two-stage arithmetic pretraining beats tree models on tabular regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2081,"prompt_tokens":958,"completion_tokens":1123,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1024}},"tokens_in":574,"tokens_out":1123,"duration_ms":8123,"temperature":1.0,"reasoning_tokens":1024,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:27:36.915593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full APAR protocol again on the same ten datasets with the arithmetic operator selected for each dataset using only validation data, and compare average RMSE to the same baselines; if the advantage disappears or falls to within run-to-run variation, the headline gains came from test-set-dependent operator selection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the motivating premise that irregular target functions make neural networks overfit tabular regression data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the relaxed multivariate Bernoulli distribution and reparameterization used for the adaptive gate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the feature-tokenizer design and the tree-versus-deep-learning gap that the paper targets."},{"cited_title":"\\\"O .; and Pfister, T","cited_arxiv_id":null,"evidence_quote":"Defines TabNet, the main pretrain-finetune neural baseline that APAR compares against."}],"review_version":1}