Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-augmented LLMs make tabular in-context learning scalable to any training-set size, with median error falling as a power law.

desk verdict Retrieval helps—the main comparison holds—but the scaling-law claim is overreaching without a non-LLM control swept over D. read the letter →

arxiv 2502.03147 v1 pith:4W6NLENT submitted 2025-02-05 cs.CL cs.AI

Xumeng Wen , Shun Zheng , Zhen Xu , Yiming Sun , Jiang Bian This is my paper · ORCID
classification cs.CLcs.AI
keywords tabularin-contextlearningretrieval-augmentedgenerationlargelanguagemodelsdatascalingbehaviorpower-lawany-shotretrieval-guidedinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to remove the few-shot ceiling on LLM-based in-context learning for tabular data. Because each table row burns many tokens when written as text, earlier LLM approaches could only put a handful of examples in context; this work claims that a retrieval module that selects the most relevant rows for each test instance, combined with instruction tuning that mimics that selection, lets the same text interface use training sets of any size. On 29 classification and 40 regression datasets the median error falls as a power law of training-set size, with exponent near 0.102 for classification and 0.053 for regression. If true, a single chat-style model could serve as a universal, accessible predictor for tabular data without per-dataset training, while still improving as more data arrives.

What carries the argument

The load-bearing object is the non-parametric retrieval policy TabRAG, which scores feature-wise distances between the test row and every training row after quantile normalization, weights each feature by linear (Pearson correlation) and non-linear (single-feature tree score) importance, and aggregates these into a sample distance with a weighted L2 norm. It then feeds a fixed quota of nearest neighbors, up to 128 rows, to an LLM whose post-training objective was changed to predict from contexts produced by that same policy rather than from randomly selected rows. This alignment of retrieval and instruction tuning is what lets the text-based model convert more training data into better predictions instead of being overwhelmed by token limits.

What would settle it

Find a tabular task whose label depends on global structure rather than local neighbors—for example, predicting a row's rank in the full training distribution—and show that increasing training set size $D$ no longer lowers the median error along the reported power law $L(D)=(D_c/D)^\alpha$. A second, simpler check is to audit the overlap between the post-training and held-out datasets: if any held-out dataset appears in the training list, the power-law gains could be partly memorization rather than scalable learning.

Watch

Extended reading notes

Core claim

The central claim is that scalable tabular in-context learning can be decoupled into a retrieval step that picks a fixed-size support set for each test row and an LLM step that predicts from that support set alone. The paper formalizes this as $C_{x_{\mathrm{test}}}=\mathrm{TabRAG}(x_{\mathrm{test}}, D_{\mathrm{train}})$ and shows that when a Phi-3-based LLM is post-trained with contexts generated by this retrieval policy, its median error on held-out datasets follows $L(D)=(D_c/D)^\alpha$, with $\alpha\approx0.102$ for AUROC-based classification error and $\alpha\approx0.053$ for regression NMAE. It further reports that the retrieval-augmented model beats its random-context and KNN ablated variants, outperforms the leading numeric TabICL baseline on roughly a fifth of the datasets, contributes measurable diversity to ensembles, and still trails well-tuned numeric models on average.

Load-bearing premise

The method assumes that a small set of retrieved neighbors can carry enough information to predict any given test row, and the evaluation assumes the manual filtering fully removed overlap between the post-training dataset and the held-out datasets.

Editorial extensions

If this is right

  • LLM-based TabICL can escape the few-shot regime: a fixed-size retrieved context gives a forward pass per test row even when the training set has 100,000 rows.
  • More training data will keep helping the text interface, with median error improving roughly as a power law, until dataset-specific limits are reached.
  • Retrieval policy quality becomes the main lever: simple changes such as swapping normalization or adding a domain-informed constraint cut regression error by 14 percent to 59 percent on the case-study datasets.
  • Because the model's decision boundaries differ from numeric models, adding it to an ensemble of TabPFN-v2 and gradient-boosted trees improves aggregate robustness.
  • Text representation also captures discrete structure: the model achieves zero NMAE on an integer regression task where numeric models emit float outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corollary the paper leaves implicit is that 'retrieval engineering'—choosing feature weights, normalization, and similarity constraints per dataset—becomes a practical skill analogous to prompt engineering, and the case studies give a direct recipe for testing this on new datasets.
  • The power-law exponents imply strongly diminishing returns: the classification exponent of about 0.102 means multiplying training data by 1000 cuts median error by roughly half, so the practical value sits in the small-to-mid data range rather than in massive datasets.
  • A testable extension is to post-train the LLM on synthetic data with diverse feature distributions and feature-label interactions, mirroring TabPFN's recipe, and measure whether the held-out error gap closes.
  • One could also test whether retrieval-plus-LLM serves as a drop-in component for conversational data analysis by checking whether the same retrieved support set improves answer faithfulness on natural-language questions about a table.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes retrieval-augmented LLMs for tabular in-context learning (TabICL), decoupling context selection (TabRAG) from LLM prediction, and aligning the LLM with retrieval patterns via retrieval-guided instruction tuning. The authors post-train Phi-3 with a generative tabular learning objective and evaluate on 29 classification and 40 regression datasets. They report that RAG+Phi3-GTL outperforms random-context Phi3-GTL and RAG+KNN on nearly all datasets, that median error follows a power law L(D) = (Dc/D)^alpha with alpha ~ 0.102 (classification) and ~0.053 (regression), and that LLM-based TabICL adds ensemble diversity and excels on certain datasets, while still lagging behind well-tuned numeric models overall.

Significance. If the scaling claim is established, the paper would show that LLM-based TabICL can leverage full-sized training sets through retrieval, a meaningful step beyond the few-shot limitation of prior LLM-based TabICL. The benchmark breadth (69 datasets), the inclusion of tuned GBDT and neural baselines, the RAG+X controls, and the detailed per-dataset case studies are strengths. The retrieval-guided post-training and the decision-boundary analysis are useful contributions. However, the headline scaling-law claim is not yet supported by the evidence as presented, because the D-sweep lacks a non-LLM control that consumes the identical retrieved contexts, and the power-law parameters are fitted to the same data they are used to describe. The data-leakage auditability issue also needs resolution before the held-out generalization claims can be fully credited.

major comments (4)
  1. [Section 5.2, Figure 1] The power-law L(D) is reported only for RAG+Phi3-GTL, with no RAG+KNN or RAG+LR control evaluated across the same D sweep. Since the LLM context is capped at 128 instances (Appendix D.5) and the frozen LLM receives the same context size at every D, the observed decline in median error as D grows from 400 to 50,000 could be entirely a property of the retrieval pool (nearest neighbors become closer as the candidate pool grows), not of LLM-based TabICL. This is load-bearing for the abstract and Section 4's attribution of scalability to the LLM-based interface. Please add RAG+KNN (and ideally RAG+LR) scaling curves to Figure 1, or otherwise explicitly separate the retrieval-pool effect from the LLM's in-context reasoning effect.
  2. [Section 5.2, Eq. L(D) = (Dc/D)^alpha] The power-law claims rest on four median data points (D=400, 2000, 10000, 50000) from which both alpha and Dc are estimated, with no confidence intervals, no per-dataset fits, and no out-of-sample predictive check. As presented, L(D) is a fitted curve rather than a scaling law, and the statement that 'the expected prediction error approaches zero' extrapolates beyond the fitted range without uncertainty quantification. Please report uncertainties, show per-dataset fits, and test the law by predicting a held-out D value; also temper the extrapolative claim or justify it with a theoretical argument.
  3. [Appendix B.1 / Section 5.1] The claim that the 319 post-training datasets and the 69 held-out datasets are disjoint relies on a manual filtering process, but no list of post-training dataset identifiers or the overlap-filtering procedure is provided. Because several held-out datasets come from the GTL benchmark (Wen et al., 2024), which itself used a 350-dataset pool, the leakage risk is non-trivial and not auditable. Please release the full list of post-training dataset identifiers and the exact filtering steps, or otherwise make the split independently verifiable.
  4. [Section 4, Eq. (3)] The method is built on the stated assumption that a limited per-test-instance support set can suffice for accurate prediction. As the paper's own case study R-25 (Appendix E) shows, tasks that require global structure or feature combinations (e.g., day-of-week and year interactions) are not recoverable by the default local retrieval policy. This scope limitation directly bounds the 'any data size' claim in the abstract and should be stated prominently in the abstract or introduction, not only in the limitations/case-study sections.
minor comments (6)
  1. [Abstract / Appendix B.2] The abstract says 'any data size,' but the experiments cap training sets at 100,000 samples (Appendix B.2) and the LLM context at 128 instances; please clarify the intended scope.
  2. [Figure 1] The dashed lines are said to represent a power-law relationship, but the fitted parameters and the metric for L are not given in the caption; please report the fitted equation in the caption or in the text of Section 5.2.
  3. [Appendix A.2 / Appendix C] The text uses 'over 300 public datasets,' '319 public datasets,' and '146 classification and 173 regression datasets' in close proximity; please align the terminology and state explicitly that 146+173 equals the 319 total.
  4. [Section 5.1] The sentence about adopting Phi-3 and extending the effective context length from 4K to 128K conflates the base model's architecture with the post-training procedure; please specify which component (base LLM, post-training sequence length, or retrieval setup) determines the effective context.
  5. [Related Work, ICL paragraph] The citation 'Dong et al.' lacks a year and full venue information; please add the complete reference.
  6. [Table 4] Several numerical entries in Table 4 show an unusual spacing between the integer and fractional parts (e.g., '0 .960'); please reformat the table for clarity.

Circularity Check

1 steps flagged · score 4.0 of 10

The RAG-vs-random comparison is self-contained, but the power-law scaling 'finding' is a fit of L(D)=(Dc/D)^alpha to the same points it is claimed to explain.

  1. fitted input called prediction [Section 5.2, Scaling with Available Training Instances, Figure 1]
    "With the RAG policy, the median prediction error demonstrates a power-law relationship with the number of training instances, expressed as L(D) = ( Dc/D)α. For classification tasks, L = 1 − AUROC, Dc ∼ 6.05e−5, and α ∼ 0.102, whereas for regression tasks, L = NMAE, Dc ∼ 8.05e−8, and α ∼ 0.053. This finding highlights a favorable statistical learning characteristic: given a distinguishable feature space and sufficient training instances, the expected prediction error approaches zero."

    The exponents α and constants Dc are fitted to the same four median-error points (D = 400, 2000, 10000, 50000) that Figure 1 displays and that the 'law' is then used to describe. There is no independent hold-out in D, no per-dataset fit, and no out-of-sample check reported. The statement that expected error approaches zero is a reading of the fitted curve, not a prediction generated before fitting. Thus the scaling-law evidence reduces, by construction, to a parametric curve fit; it cannot independently certify that the LLM interface is what benefits from larger datasets. The Random-vs-RAG comparison at fixed D remains a genuine controlled measurement, so the circularity is partial and confined to the scaling-law framing.

full rationale

The paper's core method comparison (RAG+Phi3-GTL vs Phi3-GTL with random contexts, and vs externally tuned tabular models) is evaluated on 69 held-out datasets with ablated variants and standard baselines; it is not circular. The reliance on Wen et al. (2024) for the GTL post-training objective and for part of the benchmark is normal scientific inheritance and is not load-bearing: the present gains are shown against that prior method as the Phi3-GTL baseline. The main circularity concern is the power-law claim in Section 5.2: alpha and Dc are fitted to the same median-error points they are said to reveal, so the 'scaling behavior' is a descriptive fit rather than an independent prediction. The absence of a RAG+KNN scaling sweep is a missing control for attribution (kNN error naturally falls as the pool grows), but that is an evidential gap, not a definitional reduction. Overall partial circularity score 4.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method depends on the kNN-style locality assumption, the reliability of single-feature importance scores, and the assumption that the manually filtered post-training corpus does not overlap the held-out set. The only explicit fitted constants are the power-law alpha and Dc; the retrieval policy itself is parameter-free by design. No new entities are introduced.

free parameters (2)
  • Power-law exponent alpha = 0.102 (classification), 0.053 (regression)
    Fitted to the median errors across training-set sizes in Figure 1; presented as a scaling law but not derived.
  • Power-law constant Dc = 6.05e-5 (classification), 8.05e-8 (regression)
    Fitted jointly with alpha to the same median error data; has no independent predictive content.
assumptions (4)
  • domain assumption A limited support set of retrieved instances suffices for accurate prediction on each test row (kNN-style locality assumption).
    Section 4 states this explicitly as the basis for the decoupled retrieval formulation. If some datasets require global structure, the retrieval approach cannot recover it.
  • domain assumption Manual filtering of the post-training corpus removes all overlap with held-out datasets.
    Appendix B.1 asserts datasets were carefully filtered and manually, but no list or code is provided, so the no-leakage assumption is not checkable.
  • domain assumption Pearson correlation and PPS computed on the training pool are reliable proxies for which features matter for each test query.
    Appendix A.1 uses these single-feature scores to weight distances; the case studies in Appendix E show this fails on several datasets (e.g., R-25 needs feature combinations).
  • domain assumption The text serialization of tabular rows preserves enough information for LLM in-context learning to be competitive.
    Section 3 commits to token-based serialization; the paper's own failure analysis attributes some gaps to limited pattern coverage in post-training, showing this transfer is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models." pith.science (2026). https://pith.science/paper/4W6NLENT

@misc{pith2026250203147,
  author       = {Pith},
  title        = {Pith review of: Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4W6NLENT}},
  note         = {Machine review of arXiv:2502.03147}
}
read the original abstract

Recent studies have shown that large language models (LLMs), when customized with post-training on tabular data, can acquire general tabular in-context learning (TabICL) capabilities. These models are able to transfer effectively across diverse data schemas and different task domains. However, existing LLM-based TabICL approaches are constrained to few-shot scenarios due to the sequence length limitations of LLMs, as tabular instances represented in plain text consume substantial tokens. To address this limitation and enable scalable TabICL for any data size, we propose retrieval-augmented LLMs tailored to tabular data. Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs. This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets and demonstrating promising scaling behavior. Extensive comparisons with state-of-the-art tabular models reveal that, while LLM-based TabICL still lags behind well-tuned numeric models in overall performance, it uncovers powerful algorithms under limited contexts, enhances ensemble diversity, and excels on specific datasets. These unique properties underscore the potential of language as a universal and accessible interface for scalable tabular data learning.

Figures

Figures reproduced from arXiv: 2502.03147 by the authors.

Figure 1
Figure 1. We investigate the effects of increasing the number of training instances (|D T ′ train|) and the number of in-context instances per test example (N C ) on the TabICL performance of Phi3-GTL models. In each subplot, we compare the scaling effects of two Phi3-GTL models with different retrieval policies: one that randomly selects in-context instances, denoted as ”Random,” and the other employing our default TabRAG mo… view at source ↗
Figure 2
Figure 2. An overall performance comparison of all models. In the left subplot, we use violin plots to show the AUROC scores of different models across 29 classification tasks, while the right subplot displays the NMAE scores for 40 regression tasks. Models are sorted by their median metric score across the held-out datasets, with dashed lines indicating these median scores in each subplot. Our approach, RAG+Phi3-GTL, is pref… view at source ↗
Figure 3
Figure 3. Ensemble performance comparisons of RAG+Phi3-GTL, TabPFN-v2, LightGBM, and CatBoost are presented, where normalized AUROC or NMAE scores (min-max normalized across methods for each dataset) are plotted to highlight their relative strengths across multiple datasets, while omitting absolute metric differences. limited in-context instances. Metrics For classification tasks, we use the Area Under the Receiver Operating … view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Per-dataset performance comparisons between RAG+Phi3-GTL and the two most competitive baselines, TabPFN-v2 and CatBoost, are presented, with dataset IDs sorted by performance gaps. Dashed lines and annotations are used to indicate the proportion of datasets where RAG+P…
Figure 5
Figure 5. Figure 5: Decision boundary comparisons of various models, where each row corresponds to a specific set of training instances generated from a given data distribution. The first column visualizes these training instances, while the subsequent columns illustrate the decision boun…
Figure 6
Figure 6. Figure 6: Ablation study of the retrieval mechanism across 29 classification and 40 regression datasets. Normalized metrics compare variants where individual components of the full method (Section A.1) are removed. This approach significantly improves zero-shot and in-context le…
Figure 7
Figure 7. Figure 7: Comparison of base LLM performance with RAG and GTL approaches. Results are evaluated across 29 classification datasets and 40 regression datasets. Normalization Effects: While NoNorm degrades performance in classification tasks, it yields improvements in regression ta…
Figure 8
Figure 8. Figure 8: Comparison with Tabula-8B on 29 classification and 40 regression datasets. Tabula-8B LLaMA3-8B-GTL LLaMA2-13B-GTL 0.6 0.8 1.0 AUROC Tabula-8B LLaMA3-8B-GTL LLaMA2-13B-GTL 0.0 0.5 1.0 Normalized AUROC [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Comparison with Tabula-8B on 40 regression-transformed classification datasets. regression tasks ( [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Compare performance gap between Tabula-8B and LLaMA-13B-GTL in regression datasets and the corresponding quantile￾transformed classification datasets. −122.5 −120.0 −117.5 −115.0 Longitude 34 36 38 40 Latitude 0 1 2 3 4 5 Lat. Lon. Feature Name 0.0 0.2 0.4 Feature Sco…
Figure 11
Figure 11. Figure 11: Case Study on the California Dataset. The left figure displays the relationship between features and the label, with label values represented by point color. The middle figure shows feature importance scores computed using linear (Pearson Correlation) and non-linear (…
Figure 12
Figure 12. Figure 12: Case Study on the Pol Dataset. The figure displays the distribution of an important feature under three transformations: (left) the original feature values, (middle) standard normalized values, and (right) quantile normalized values 18 [PITH_FULL_IMAGE:figures/full_f…
Figure 13
Figure 13. Figure 13: The figure displays the relationship between features (e.g., Temperature and Hour) and the label. It also visualizes bike rental patterns across different years and working-day conditions [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Decision boundaries for the toy dataset circle, evaluated across noise levels from 0.1 to 0.3 and training data sizes from 16 to 128. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Decision boundaries for the toy dataset moon, evaluated across noise levels from 0.1 to 0.3 and training data sizes from 16 to 128. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Decision boundaries for the toy dataset linear rotation, evaluated across noise levels from 0.1 to 0.3 and training data sizes from 16 to 128. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Algorithmic Recourse of In-Context Learning for Tabular Data

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    The paper delivers the first theoretical analysis and practical zeroth-order framework for algorithmic recourse under in-context learning for tabular prediction.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [2]

    AUROC scores of Phi3-Medium-GTL and baseline models in RAG scenarios and full data Scenarios on the classif-cat-medium- 0-rl dataset. Phi3-Medium-GTL XGBoost TabPFN-v1 TabPFN-v2 RAG 0.7811 0.7845 0.7430 0.8621 Full Data - 0.8700 0.7307 0.9212 selecting normalization methods based on feature distributions and leveraging domain-specific knowledge for specif...

    work page 2021

  3. [6]

    Normalized metrics compare variants where individual components of the full method (Section A.1) are removed

    Ablation study of the retrieval mechanism across 29 classification and 40 regression datasets. Normalized metrics compare variants where individual components of the full method (Section A.1) are removed. This approach significantly improves zero-shot and in-context learning performance for base LLMs. As demonstrated in (Wen et al., 2024), GTL-enhanced LL...

    work page 2024

  4. [8]

    P., Grabocka, J., and Hutter, F

    M¨uller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510,

    arXiv

  5. [9]

    B., and Goldstein, T

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. SAINT: Improved neural net- works for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342,

    arXiv

  6. [11]

    URL https://doi.org/10.5281/zenodo. 4091345. Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. In ICLR,

    doi:10.5281/zenodo

  7. [12]

    A., Oguz, B., et al

    Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of founda- tion models. arXiv preprint arXiv:2309.16039,

    arXiv

  8. [14]

    To ensure the integrity of our evaluation, we carefully filtered these datasets to eliminate any potential data contamination between the training and held-out evaluation sets

    on data with retrieved contexts, we curated over 300 public datasets from Kaggle, following the collection methodology outlined in their paper. To ensure the integrity of our evaluation, we carefully filtered these datasets to eliminate any potential data contamination between the training and held-out evaluation sets. After filtering, we retained 146 cla...

    work page 2024

Show all 23 references
  1. [15]

    FTT is a widely recognized neural model for tabular learning

    and TabR (Gorishniy et al., 2024). FTT is a widely recognized neural model for tabular learning. TabR represents a significant successor in retrieval-based tabular models. This research proposes an innovative approach that leverages embedding distance-based retrieval methods t...

    2024

  2. [16]

    Neural Baselines We evaluate three neural baselines to benchmark performance on tabular data: MLP, FT- Transformer (Gorishniy et al., 2021), and TabR (Gorishniy et al., 2024)

    to maintain consistency and comparability. Neural Baselines We evaluate three neural baselines to benchmark performance on tabular data: MLP, FT- Transformer (Gorishniy et al., 2021), and TabR (Gorishniy et al., 2024). For our experiments, we use the implementations provided b...

    2021

  3. [18]

    123456” → “123

    For TabPFN and RAG+Ph3-GTL, the training data serves as the context data samples. Notably, the LLM exhibits unique decision boundaries characterized by higher uncertainty, contrasting with the smoother boundaries of other models. This suggests potential opportunities for LLMs ...

    2024

  4. [19]

    RAG-Tuned denotes results using dataset-specific retrieval methods (see Section E)

    Normalized Mean Absolute Error (NMAE) scores for case study datasets. RAG-Tuned denotes results using dataset-specific retrieval methods (see Section E). The first group highlights datasets where default TabRAG underperformed; customized retrieval contexts significantly improv...

    1904

  5. [20]

    The left figure displays the relationship between features and the label, with label values represented by point color

    Case Study on the California Dataset. The left figure displays the relationship between features and the label, with label values represented by point color. The middle figure shows feature importance scores computed using linear (Pearson Correlation) and non-linear (PPS) meas...

    1904

  6. [21]

    Case Study on the Pol Dataset. The figure displays the distribution of an important feature under three transformations: (left) the original feature values, (middle) standard normalized values, and (right) quantile normalized values 18 Scalable In-Context Learning on Tabular D...

    2021

  7. [128]

    Index Dataset Abbr

    22 Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models Table 3: Data statistics of held-out datasets used in this paper. Index Dataset Abbr. Data Source Tag Task # Train Ex. # Test Ex. # Class. # Num. Feat. # Cat. Feat. Max Label Value Mi...

    2022

  8. [1951]

    Retrieval-augmented generation for large language models: A survey

    Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997,

    arXiv

  9. [2009]

    To capture non-linear relationships, we fit decision trees to each feature and evaluate its contribution to target prediction in the context pool

    to measure the linear relationship between each feature and the target label in the context pool. To capture non-linear relationships, we fit decision trees to each feature and evaluate its contribution to target prediction in the context pool. This process does not require pa...

    2020

  10. [2016]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  11. [2020]

    D., Yue, X., and Chen, W

    Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long- ICLBench: Long-context llms struggle with long in- context learning. arXiv preprint arXiv:2404.02060,

    arXiv

  12. [2021]

    Llama 2: Open foundation and fine- tuned chat models

    10 Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat m...

    arXiv

  13. [2022]

    A survey on in-context learning

    Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey on in-context learning. arXiv preprint arXiv:2301.00234,

    arXiv

  14. [2024]

    Deepseek- v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Liu, H., Zaharia, M., and Abbeel, P. RingAttention with blockwise transformers for near-infinite context...

    arXiv

  15. [2025]

    Tab- Transformer: Tabular data modeling using contextual embeddings

    Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tab- Transformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678,

    2012 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.