Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Federated In-Context Learning: Iterative Refinement for Improved Answer Quality

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under a linear self-attention idealization, Fed-ICL's round-based label exchange converges to the pooled-data answer and outperforms debate and ensemble baselines on QA benchmarks.

desk verdict A genuinely new federated-ICL loop with a convergence proof that proves less than claimed; worth engaging, but the 'optimal' language needs revision. read the letter →

arxiv 2506.07440 v1 pith:2GEEXPGG submitted 2025-06-09 cs.LG

Ruhan Wang , Zhiyong Wang , Chengkai Huang , Rui Wang , Tong Yu , Lina Yao , John C.S. Lui , Dongruo Zhou This is my paper
classification cs.LG
keywords federatedlearningin-contextquestionansweringiterativerefinementlinearself-attentionlargelanguagemodelsparameter-freecommunicationprivacy-preserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Federated In-Context Learning (Fed-ICL), a round-based scheme in which a server and clients refine answers to common questions without transmitting model weights or raw local data. The central claim is that iterating this refinement converges to a fixed point that coincides with the pooled-data answer when client and query covariances align with the pretraining distribution, at least in the linear self-attention idealization used for the theory. The engine is a simple linear recurrence for the server's implicit weight vector, and the paper proves linear convergence whenever a covariance-based contraction matrix satisfies a spectral bound. Experiments on MMLU and TruthfulQA show the iterative scheme outperforming one-shot ensembling, debate, and federated fine-tuning baselines while communicating only predicted answers. This matters because high-quality in-context examples are exactly what privacy-conscious federated settings cannot share directly.

What carries the argument

The machinery is the recurrence $w_{k+1} = \frac{1}{2}H_{\mathrm{cont}}w_k + \frac{1}{2}w_{\mathrm{limit}}$ that emerges from composing two ICL passes inside Algorithm 1. In the first pass each client uses the server's current answers to relabel its own local examples; in the second it uses both its original and relabeled examples to answer the server's queries, and the server averages the resulting labels. For the single-layer linear self-attention model, each pass is a linear readout, and composing the two passes collapses into the affine map above. The matrix $H_{\mathrm{cont}}$ is the contraction that governs convergence, $w_{\mathrm{limit}}$ is the pooled-data target, and the bound $\|H_{\mathrm{cont}}\|_2 \le 2$ is the condition under which repeated application of the map contracts toward the fixed point.

What would settle it

In a controlled regression experiment with a single-layer linear self-attention model, compute $H_{\mathrm{cont}}$ and $w_{\mathrm{limit}}$ from the client data and check whether the observed server weights obey $w_{k+1} = \frac{1}{2}H_{\mathrm{cont}}w_k + \frac{1}{2}w_{\mathrm{limit}}$ across rounds; any systematic deviation, or a case with $\|H_{\mathrm{cont}}\|_2 > 2$ where the iteration still converges, would falsify the paper's central convergence claim.

Watch

Extended reading notes

Core claim

At the center of the paper is a linear dynamical story about in-context learning. For a single-layer linear self-attention (LSA) model, the paper shows that Fed-ICL's server-side predicted label for every query $x_m$ can be written as $\hat{y}_{k,m} = w_k^\top x_m$, and the implicit weight vector evolves as $w_{k+1} = \frac{1}{2}H_{\mathrm{cont}}w_k + \frac{1}{2}w_{\mathrm{limit}}$. Here $H_{\mathrm{cont}}$ is built from the empirical covariances of client example covariates and server query covariates, and $w_{\mathrm{limit}}$ is exactly the weight vector the same LSA model would learn from the union of all client datasets. Theorem 4.1 derives this recurrence, and Corollary 4.2 shows that under $\|H_{\mathrm{cont}}\|_2 \le 2$ the server weights converge at a linear rate to $w^* = (2I - H_{\mathrm{cont}})^{-1}w_{\mathrm{limit}}$. The intended reading is a federated analogue of classic convergence-to-the-global-model guarantees: Fed-ICL reaches pooled-data answer quality without pooling data.

Load-bearing premise

The guarantee assumes every client runs the same simplified single-layer attention model, that client and server questions are Gaussian with covariances matching the pretraining distribution, and that the matrix controlling contraction is not too large in operator norm; outside those conditions the claimed convergence to pooled-data answer quality is not proven.

Editorial extensions

If this is right

  • Federated QA can run by exchanging only predicted labels: per-round communication cost is proportional to the number of queries, not to the model size, while answer quality still improves across rounds.
  • The convergence target $w_{\mathrm{limit}}$ is the pooled-data weight vector, so the framework's asymptotic answer quality is capped by what a single model could achieve with all client data together.
  • The contraction condition ties guaranteed convergence to alignment between client data, server queries, and the pretraining distribution; when that alignment fails, the guarantee does not apply.
  • Even without ground-truth labels at clients, the Fed-ICL-Free variant builds pseudo-labels from the server's refined answers and still beats debate-style baselines that ignore local datasets.
  • Empirically, more interaction rounds and semantically filtered local examples improve accuracy, while adding more clients with stronger data heterogeneity hurts it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the contraction intuition carries over to nonlinear LLMs, the framework should converge fastest when each client's local question distribution matches both the server's query distribution and the model's pretraining distribution; this yields a testable client-selection or data-filtering principle.
  • Inference: the theorem leaves a gap between the fixed point $w^*$ and the pooled-data target $w_{\mathrm{limit}}$, so in realistic settings converged answers will likely differ from the pooled-data oracle; measuring that gap empirically would quantify the cost of federated instead of centralized ICL.
  • Inference: the reported privacy evaluation tests prompt extraction from individual client responses, not whether multiple refinement rounds leak more about local datasets over time, so multi-round leakage remains an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces Fed-ICL, a federated in-context learning framework for question answering in which a server iteratively refines predicted answers by having clients relabel their local data using the current global context and then predict the server's query examples using both original and relabeled local data. The server aggregates client predictions, and the process repeats over several rounds. The paper provides a convergence analysis for a simplified single-layer linear self-attention model and reports experiments on MMLU and TruthfulQA showing accuracy gains over FL and parameter-free baselines at lower communication cost.

Significance. The proposed framework is novel and practically motivated, and the experimental evaluation is comparatively thorough, including ablations, heterogeneity analysis, a privacy analysis, and communication-cost comparisons. The algebraic derivation of the recurrence in Theorem 4.1 is transparent and builds on prior work. However, the headline theoretical claim—that Fed-ICL converges to the centralized ICL optimum—is not actually established by the proved result: the fixed point of the recurrence is (2I-H_cont)^{-1}w_limit, which generally differs from w_limit. With a corrected statement and appropriately scoped claims, the paper would still be useful for its algorithmic contribution and empirical results, but the current theoretical framing overstates what is proven.

major comments (3)
  1. [Theorem 4.1 and Corollary 4.2; Section 1 contribution bullet and Section 7 conclusion] The recurrence w_{k+1} = (1/2)H_cont w_k + (1/2)w_limit has fixed point w^* = (2I-H_cont)^{-1}w_limit, not w_limit. The statement in the contributions (Section 1) and conclusion (Section 7) that Fed-ICL 'converges to the optimal answers conditioned on all client datasets' is therefore not supported by Corollary 4.2; it would require H_cont = I, which fails with probability one for finite N and M under the paper's Gaussian assumptions because H_cont is a product of sample covariance matrices. The asymptotic argument in Section 4.1 (H_cont ≈ I when T, N, M → ∞ and the distributions align) does not justify the finite-sample claim. Please restate the guarantee as convergence to w^* and analyze the bias w^* - w_limit, or prove conditions under which w^* = w_limit.
  2. [Corollary 4.2] The condition should be ||H_cont||_2 < 2, not ≤ 2. If ||H_cont||_2 = 2, the matrix (2I-H_cont) can be singular and w^* may not be well-defined; moreover, the contraction factor 1/2 ||H_cont||_2 in Eq. (3) equals 1 in that case, so geometric convergence is not guaranteed. The paper also does not provide primitive conditions on N, M, L, d, Λ, Λ_client, and Λ_server under which the spectral-norm assumption holds with high probability; as stated, Corollary 4.2 is conditional on an event whose probability is not analyzed.
  3. [Section 4, 'LM Setup' and Remark 4.3] The convergence theorem assumes all clients share the same pretrained single-layer linear self-attention model, Gaussian covariates, and distributional alignment with the pretraining distribution. These restrictions are acknowledged in Remark 4.3, but the abstract, introduction, and conclusion state the convergence-to-optimality claim without these caveats. The claims should be scoped to the simplified model, and the theory should not be described as a guarantee for the practical Fed-ICL setting with heterogeneous LLMs and non-Gaussian text data.
minor comments (5)
  1. [Algorithm 1, line 8] The phrase 'sends back to C i k to server' should read 'sends C^i_{k+1} to the server'.
  2. [Appendix B, proof of Theorem 4.1] The proof uses 'episode k' where the algorithm description uses 'round k'; please align the terminology.
  3. [Abstract and Section 1] The term 'parameter-free' is used to describe methods that do not transmit model parameters, but Fed-ICL does transmit predicted labels; consider clarifying this terminology to avoid confusion.
  4. [Figures 5–8] These ablation plots show a single trajectory per setting without error bars or repeated-seed variance; reporting standard deviations over multiple runs would strengthen the empirical claims.
  5. [Section 6.1, Evaluation Metric] The use of GPT-4o-generated answers as ground truth for TruthfulQA is non-standard; since the metric then measures agreement with a particular model rather than with human truthfulness judgments, this choice should be justified more carefully.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Fed-ICL's recurrence follows from an external LSA convergence theorem; the H_cont=I gap is a correctness overclaim, not a definitional/fitted-input loop.

full rationale

The derivation chain in Theorem 4.1 is self-contained: the recurrence w_{k+1} = 1/2 H_cont w_k + 1/2 w_limit is derived directly from Algorithm 1 by applying the external single-layer linear self-attention characterization in Eq. (14), which the paper quotes from Zhang et al. (2023a). That external theorem has stated assumptions (gradient flow on the population loss with the specified initialization, under the scale condition sigma^2 ||Gamma||_op sqrt(d) < 2) that do not include the Fed-ICL convergence claim, so it is independent support rather than a citation whose content reduces to the target result. The quantity w_limit is defined as the weight of the linear predictor obtained by ICL with the combined client datasets, and it is not fitted to the recurrence; the fixed point of the iteration is a consequence of the algorithm, not an input. The paper also does not rely on any self-citation: the load-bearing reference to Zhang et al. (2023a) is external to the present author set. The concern that the fixed point is w* = (2I - H_cont)^{-1} w_limit, which equals the claimed 'optimal answers conditioned on all client datasets' only when H_cont = I, and that the stated condition ||H_cont||_2 <= 2 should be strict to avoid a singular (2I - H_cont), is a correctness and overclaim issue about the theorem's interpretation, not a circular reduction: the fixed point is still derived from the algorithm and is not imposed by construction. Accordingly, no circular step was found, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The theoretical claim rests on the external LSA convergence result, on Gaussian-linear data assumptions, and on a norm bound that is asserted rather than proven. The experimental results depend on hand-chosen hyperparameters (C=5, K=6, temperature 0.1) and on the choice of embedding model for kNN filtering.

free parameters (4)
  • Number of context examples C = 5
    Set to 5 in main experiments; ablation compares 1 vs 5 but no validation-based selection reported. Affects ICL accuracy.
  • Number of rounds K = 6
    Six interaction rounds used as convergence point for Fed-ICL and Fed-ICL-Free; no early-stopping criterion is defined.
  • Generation temperature = 0.1
    Temperature 0.1 for answer generation; baseline methods use temperature 1.0, potentially giving baselines a disadvantage in diversity.
  • kNN filtering size and embedding model = C=5 with paraphrase-MiniLM-L6-v2
    Filtering to closest examples; choice of embedding model and C affects which local data is used as context.
assumptions (5)
  • domain assumption All clients share the same pre-trained language model
    Assumed in Section 4 LM Setup; in practice Fed-ICL allows different LMs per client, but the theory and the claim of optimality require identical LMs.
  • standard math Pretrained LSA converges to global optimum with W* as in Theorem A.3 (Zhang et al. 2023a)
    Adopted as external result in Appendix A; the paper relies on this to derive Eq (14).
  • domain assumption Covariates x ~ N(0, Lambda) and labels y = <w,x> in the theoretical analysis
    Section 4 and Appendix A specify the linear regression setting; experiments use different data distributions.
  • domain assumption Server aggregation is average for regression
    The theory analyzes average aggregation; for classification and generation, fusion LM and majority voting are used without theoretical justification.
  • domain assumption Norm bound ||H_cont||_2 <= 2 for convergence
    Corollary 4.2 assumes this bound; the paper only argues heuristically that when Lambda is close to Lambda_client and Lambda_server, H_cont is close to I.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated In-Context Learning: Iterative Refinement for Improved Answer Quality." pith.science (2026). https://pith.science/paper/2GEEXPGG

@misc{pith2026250607440,
  author       = {Pith},
  title        = {Pith review of: Federated In-Context Learning: Iterative Refinement for Improved Answer Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GEEXPGG}},
  note         = {Machine review of arXiv:2506.07440}
}
read the original abstract

For question-answering (QA) tasks, in-context learning (ICL) enables language models to generate responses without modifying their parameters by leveraging examples provided in the input. However, the effectiveness of ICL heavily depends on the availability of high-quality examples, which are often scarce due to data privacy constraints, annotation costs, and distribution disparities. A natural solution is to utilize examples stored on client devices, but existing approaches either require transmitting model parameters - incurring significant communication overhead - or fail to fully exploit local datasets, limiting their effectiveness. To address these challenges, we propose Federated In-Context Learning (Fed-ICL), a general framework that enhances ICL through an iterative, collaborative process. Fed-ICL progressively refines responses by leveraging multi-round interactions between clients and a central server, improving answer quality without the need to transmit model parameters. We establish theoretical guarantees for the convergence of Fed-ICL and conduct extensive experiments on standard QA benchmarks, demonstrating that our proposed approach achieves strong performance while maintaining low communication costs.

Figures

Figures reproduced from arXiv: 2506.07440 by the authors.

Figure 1
Figure 1. Workflow of the Fed-ICL framework. Clients use the global context C from the server to relabel local datasets (D 1 , D 2 , D 3 ) and refine context locally. The server aggregates updated contexts (C 1 , C 2 , C 3 ) from clients to update the global context C, enabling collaborative learning across heterogeneous data. 2023; Wolf et al., 2020; Qiu et al., 2020), excelling in tasks such as text generation, classificati… view at source ↗
Figure 2
Figure 2. Comparison of performance and communication costs among Fed-ICL variants and baseline methods across different α settings on the MMLU and TruthfulQA benchmarks, using a small LLM as the client model. The reported results reflect performance at convergence. For iterative methods, communication costs are measured upon convergence. The horizontal lines represent the performance of non-iterative methods, including Fed-I… view at source ↗
Figure 3
Figure 3. Comparison of performance and communication costs among Fed-ICL variants and baseline methods across different α settings on the MMLU and TruthfulQA benchmarks, using a powerful LLM as the client model. The reported results reflect performance at convergence. For iterative methods, communication costs are measured upon convergence. The horizontal lines represent the performance of non-iterative methods, including Fe… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Client Response represents the answer generated by a client in response to a server query. Client-Provided Context Examples are partial question–answer pairs drawn from the client’s local data to support the response. Reconstructed Client Context Examples show the ques…
Figure 5
Figure 5. Figure 5: Performance variation of Fed-ICL and Fed-ICL-Free as the number of interaction rounds increases on the MMLU and TruthfulQA benchmarks [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: Comparison of Fed-ICL and Fed-ICL-Free performance across different numbers of context examples [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: Client dataset distribution under different α settings on the TruthfulQA benchmark [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Client dataset distribution under different α settings on the MMLU benchmark. aggregate all client data into a single client. Upon receiving a query from the server, the client applies the kNN algorithm to retrieve the top C question–ground truth answer pairs that are…
Figure 11
Figure 11. Figure 11: Performance of Fed-ICL variants on MMLU benchmark under different α settings using LLaMA as the client model, as the number of interaction rounds increases [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Performance of Fed-ICL variants on MMLU benchmark under different α settings using GPT-4o-mini as the client model, as the number of interaction rounds increases [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Performance of Fed-ICL variants on TruthfulQA benchmark under different α settings using LLaMA as the client model, as the number of interaction rounds increases. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Performance of Fed-ICL variants on the TruthfulQA benchmark under different α settings using GPT-4o-mini as the client model, as the number of interaction rounds increases [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Performance of LLM-Debate on the TruthfulQA benchmark using Llama-2-7B-chat-h or GPT-4o-mini as the client model, as the number of interaction rounds increases. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Privacy Analysis 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A behavior-centered handbook generated from agent-harness code helps LLM planners find the right edit sites and produce better edit plans than direct repository exploration.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [4]

    Brown, T. B. Language models are few-shot learners.arXiv preprint arXiv:2005.14165,

    arXiv 2005

  3. [8]

    B., and Mor- datch, I

    Du, Y ., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in lan- guage models through multiagent debate.arXiv preprint arXiv:2305.14325,

    arXiv

  4. [9]

    On the privacy risk of in-context learning

    Duan, H., Dziedzic, A., Yaghini, M., Papernot, N., and Boenisch, F. On the privacy risk of in-context learning. arXiv preprint arXiv:2411.10512,

    arXiv

  5. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  6. [11]

    Fate-llm: A industrial grade federated learning framework for large language models.arXiv preprint arXiv:2310.10049,

    Fan, T., Kang, Y ., Ma, G., Chen, W., Wei, W., Fan, L., and Yang, Q. Fate-llm: A industrial grade federated learning framework for large language models.arXiv preprint arXiv:2310.10049,

    arXiv

  7. [12]

    Transformers learn higher-order optimization methods for in-context learning: A study with linear models.arXiv preprint arXiv:2310.17086,

    Fu, D., Chen, T.-Q., Jia, R., and Sharan, V . Transformers learn higher-order optimization methods for in-context learning: A study with linear models.arXiv preprint arXiv:2310.17086,

    arXiv

  8. [14]

    H., Qi, H., and Brown, M

    Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual clas- sification.arXiv preprint arXiv:1909.06335,

    arXiv 1909

Show all 38 references
  1. [16]

    Huang, Y ., Cheng, Y ., and Liang, Y

    URL https://arxiv.org/abs/2106.09685. Huang, Y ., Cheng, Y ., and Liang, Y . In-context convergence of transformers.arXiv preprint arXiv:2310.05249,

    arXiv

  2. [17]

    J., Lee, J

    Jeon, H. J., Lee, J. D., Lei, Q., and Van Roy, B. An information-theoretic analysis of in-context learning. arXiv preprint arXiv:2401.15530,

    arXiv

  3. [18]

    Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion.arXiv preprint arXiv:2306.02561,

    arXiv

  4. [19]

    and Suzuki, T

    10 Federated In-Context Learning: Iterative Refinement for Improved Answer Quality Kim, J. and Suzuki, T. Transformers learn nonlinear fea- tures in context: Nonconvex mean-field dynamics on the attention landscape.arXiv preprint arXiv:2402.01258,

    arXiv

  5. [20]

    Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension.arXiv preprint arXiv:1910.13461,

    Lewis, M. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension.arXiv preprint arXiv:1910.13461,

    1910 arXiv

  6. [21]

    Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    arXiv

  7. [22]

    What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804,

    Liu, J., Shen, D., Zhang, Y ., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804,

    arXiv

  8. [23]

    4o mini: Advancing cost-efficient intelli- gence, 2024.URL: https://openai

    OpenAI, G. 4o mini: Advancing cost-efficient intelli- gence, 2024.URL: https://openai. com/index/gpt-4o-mini- advancing-cost-efficient-intelligence,

    2024

  9. [24]

    K., Ganesh, M

    Qiu, C., Li, X., Mummadi, C. K., Ganesh, M. R., Li, Z., Peng, L., and Lin, W.-Y . Text-driven prompt generation for vision-language models in federated learning.arXiv preprint arXiv:2310.06123,

    arXiv

  10. [25]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    1908 arXiv

  11. [26]

    F., Aleksandrov, P., Qiu, X., et al

    Sani, L., Iacob, A., Cao, Z., Marino, B., Gao, Y ., Paulik, T., Zhao, W., Shen, W. F., Aleksandrov, P., Qiu, X., et al. The future of large language model pre-training is federated. arXiv preprint arXiv:2405.10853,

    arXiv

  12. [27]

    Sellam, T., Das, D., and Parikh, A. P. Bleurt: Learn- ing robust metrics for text generation.arXiv preprint arXiv:2004.04696,

    2004 arXiv

  13. [28]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  14. [29]

    F., Ilhan, F., Huang, T., Hu, S., and Liu, L

    Tekin, S. F., Ilhan, F., Huang, T., Hu, S., and Liu, L. Llm- topla: Efficient llm ensemble by maximising diversity. arXiv preprint arXiv:2410.03953,

    arXiv

  15. [30]

    Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  16. [31]

    Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,

    Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,

    arXiv

  17. [32]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V ., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. InProceedings of the 2020 conference on em- pirical methods in natural language processing: s...

    2020

  18. [33]

    Fedbiot: Llm local fine-tuning in federated learning without full model

    Wu, F., Li, Z., Li, Y ., Ding, B., and Gao, J. Fedbiot: Llm local fine-tuning in federated learning without full model. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3345–3355, 2024a. Wu, P., Li, K., Nan, J., and Wang, F. Federated...

    arXiv

  19. [34]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    arXiv

  20. [35]

    Textgrad: Automatic” dif- ferentiation” via text.arXiv preprint arXiv:2406.07496,

    Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., and Zou, J. Textgrad: Automatic” dif- ferentiation” via text.arXiv preprint arXiv:2406.07496,

    arXiv

  21. [36]

    X., and Shmatikov, V

    Zhang, C., Morris, J. X., and Shmatikov, V . Extract- ing prompts by inverting llm outputs.arXiv preprint arXiv:2405.15012, 2024a. Zhang, R., Frei, S., and Bartlett, P. L. Trained trans- formers learn linear models in-context.arXiv preprint arXiv:2306.09927, 2023a. Zhang, R., ...

    1904 arXiv

  22. [37]

    An analysis of attention via the lens of exchangeability and latent variable models.arXiv preprint arXiv:2212.14852,

    Zhang, Y ., Liu, B., Cai, Q., Wang, L., and Wang, Z. An analysis of attention via the lens of exchangeability and latent variable models.arXiv preprint arXiv:2212.14852,

    arXiv

  23. [38]

    Effective prompt extraction from language models.arXiv preprint arXiv:2307.06865, 2023b

    Zhang, Y ., Carlini, N., and Ippolito, D. Effective prompt extraction from language models.arXiv preprint arXiv:2307.06865, 2023b. Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-be...

    arXiv

  24. [39]

    prompts

    12 Federated In-Context Learning: Iterative Refinement for Improved Answer Quality A. Preliminaries for the Theoretical Analysis This section provides a foundational description of the in-context learning framework for function classes, following the general ideas outlined in ...

    2022

  25. [2020]

    Investigating data contamination in modern benchmarks for large language models.arXiv preprint arXiv:2311.09783,

    Deng, C., Zhao, Y ., Tang, X., Gerstein, M., and Co- han, A. Investigating data contamination in modern benchmarks for large language models.arXiv preprint arXiv:2311.09783,

    arXiv

  26. [2021]

    Can textual gradient work in federated learning?arXiv preprint arXiv:2502.19980,

    Chen, M., Jin, R., Deng, W., Chen, Y ., Huang, Z., Yu, H., and Li, X. Can textual gradient work in federated learning?arXiv preprint arXiv:2502.19980,

    arXiv

  27. [2022]

    Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

  28. [2023]

    Ensemw2s: Can an ensemble of llms be leveraged to obtain a stronger llm?arXiv preprint arXiv:2410.04571,

    Agrawal, A., Ding, M., Che, Z., Deng, C., Satheesh, A., Langford, J., and Huang, F. Ensemw2s: Can an ensemble of llms be leveraged to obtain a stronger llm?arXiv preprint arXiv:2410.04571,

    arXiv

  29. [2024]

    What learning algorithm is in-context learn- ing? investigations with linear models.arXiv preprint arXiv:2211.15661,

    Aky¨urek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learn- ing? investigations with linear models.arXiv preprint arXiv:2211.15661,

    arXiv

  30. [2025]

    Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality.arXiv preprint arXiv:2402.19442,

    Chen, S., Sheen, H., Wang, T., and Yang, Z. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality.arXiv preprint arXiv:2402.19442,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.