Large Language Model-Enhanced Multi-Armed Bandits

Chenjun Xiao; Jiahang Sun; John C.S. Lui; Runhan Yang; Zhiyong Wang; Zhongxiang Dai

REVIEW 5 major objections 5 minor 1 cited by

Large Language Model-Enhanced Multi-Armed Bandits

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM-based bandit agents should be built by splitting labor: a classical bandit algorithm explores, and the LLM only predicts rewards from history; this hybrid consistently beats direct LLM arm selection in synthetic…

desk verdict A useful empirical recipe for combining LLM reward prediction with classical exploration in bandits, but the synthetic evidence is weakened by tuning leakage and missing classical baselines. read the letter →

arxiv 2502.01118 v1 pith:5TGGHEPW submitted 2025-02-03 cs.LG cs.AI

Jiahang Sun , Zhiyong Wang , Runhan Yang , Chenjun Xiao , John C.S. Lui , Zhongxiang Dai This is my paper · ORCID

classification cs.LGcs.AI

keywords multi-armedbanditslargelanguagemodelsThompsonsamplingin-contextlearningrewardpredictionexploration-exploitationtradeoffduelingregressionoracle

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual way of using large language models in multi-armed bandits—asking the LLM to choose the next arm—puts exploration in the wrong hands. Its proposed alternative keeps a classical bandit algorithm in charge of exploration and uses the LLM only to predict rewards from the observed history through in-context learning. Three algorithms are built on this split: Thompson sampling with an LLM reward predictor, a regression-oracle bandit with an LLM predictor, and a dueling-bandit variant. In synthetic and real-text experiments the hybrids consistently beat direct-LLM arm selection, with the largest gains on tasks where arm labels carry no semantic meaning the LLM can exploit. The practical claim is that a simple division of labor—classical algorithm for exploration, LLM for prediction—makes LLM agents reliable in sequential decisions.

What carries the argument

The central mechanism is the separation of reward prediction from exploration. The LLM, given a history prompt, produces one reward (or loss) prediction per arm; a classical bandit algorithm—Thompson sampling, SquareCB, or a Borda-maximizing duel selection—turns those predictions into decisions. In TS-LLM the machine is the temperature schedule: stochastic LLM draws approximate posterior sampling at high temperature and converge to greedy selection as temperature decays. In RO-LLM the machine is SquareCB's sampling distribution, which converts predicted losses into arm-selection probabilities without needing stochastic LLM output. In TS-LLM-DB the machine is the equivalence between the maximizer of the Borda function and the maximizer of the latent reward function, plus the optimistic second-arm rule borrowed from an existing dueling-bandit algorithm.

What would settle it

Run TS-LLM on a small synthetic bandit whose posterior is computable in closed form and compare its arm-selection frequencies with true Thompson sampling; if the frequencies differ systematically, or if disabling the LLM's randomness by forcing temperature 0 while adding independent Gaussian noise of the same scale reproduces the same regret, then the posterior-sampling explanation is not what carries the algorithm.

Watch

Extended reading notes

Core claim

The central claim is that LLM-based bandit agents fail at direct arm selection not because the LLM cannot learn from history but because arm selection is an exploration problem, and stochastic text generation is a poor way to do it. The paper replaces that job: the LLM is asked, for each arm, to output a reward or loss given the current history prompt. In TS-LLM, the LLM's random outputs at a decaying temperature stand in for posterior samples in Thompson sampling, so the algorithm starts exploratory and becomes greedy; in RO-LLM, a temperature-zero LLM supplies loss estimates to SquareCB's explicit exploration distribution; in TS-LLM-DB, an LLM predicts pairwise preference probabilities and the first arm is chosen by approximately maximizing the Borda function, which is equivalent to maximizing the latent reward. Across synthetic bandits, dueling bandits, and contextual bandits built from text datasets, the authors report these hybrids consistently outperform previous direct-selection baselines, and in the AmazonCat-13K task, where arm tags are opaque integers, TS-LLM's advantage over direct selection is large and grows with the number of arms.

Load-bearing premise

The load-bearing premise is that the LLM's stochastic reward predictions, at a given temperature and with a given history prompt, behave enough like samples from the posterior distribution over rewards that maximizing them is a valid Thompson sampling step; the paper asserts this without derivation or calibration evidence.

Editorial extensions

If this is right

An LLM agent for a sequential decision task does not need to be a good explorer; it only needs to be a good reward predictor, because the surrounding classical algorithm can supply exploration.
On tasks where arm identities are semantically opaque to the LLM, the hybrid approach can be dramatically better than direct arm selection, and the gap widens as the number of arms grows.
Preference-only feedback (dueling bandits) can be handled in the same hybrid style, as long as the LLM predicts pairwise preference probabilities and the first arm is chosen by maximizing the Borda score.
A decaying temperature schedule is a workable exploration-to-exploitation schedule: fixed high temperatures keep exploring too long and fixed low temperatures stop exploring too early.
The regression-oracle variant trades some cumulative regret for much lower run-to-run variance, giving a practical choice between average performance and consistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper's results suggest the LLM's stochastic outputs do not need to be a calibrated posterior for TS-LLM to work; any predictor whose high-temperature outputs are diverse and whose low-temperature outputs are accurate might give similar regret, so the temperature schedule is doing the exploration work.
Editorial inference: the same decoupling should transfer to other sequential-decision settings, such as LLM value models inside tree search or reinforcement learning, wherever a classical high-level mechanism can absorb exploration.
Editorial inference: a cheap test of the posterior-sampling interpretation would be to replace the LLM's stochastic predictions with deterministic predictions plus explicit Gaussian noise; if regret is unchanged, the 'posterior' label is not needed.
Editorial inference: because the LLM predictor never commits to a function class, the hybrid framework should be robust to reward-function misspecification, a comparison the paper motivates but does not run.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth reading as an empirical recipe: put a black-box LLM in the reward-prediction role and let a classical bandit algorithm handle exploration. That separation is sensible, and the authors implement it cleanly for Thompson sampling, SquareCB-style regression oracles, and dueling bandits. The real-text experiments, especially AmazonCat-13K, show a plausible and interesting effect: when arm features carry no semantic signal for the LLM, direct arm selection collapses while the TS-LLM wrapper keeps exploring structurally. The appendix gives full prompts and enough detail to reproduce the runs. That is real work, and I would not desk-reject it.

The soft spots are real, though. The biggest is tuning leakage. The temperature decay schedule for TS-LLM is selected in Sec. 5.1 on the linear reward task, then used in Sec. 4.1 on that same task (and the others). Likewise, RO-LLM's gamma is picked in Sec. 5.3 on the same linear and square functions that are then reported in Fig. 1. That makes the headline "consistently outperforms" partly circular for the synthetic results. It does not sink the paper, because the text experiments are less affected, but it does mean the synthetic figures overstate the generality of the recipe. The fix is straightforward: choose those hyperparameters on a validation split or on different functions.

The second soft spot is the baseline set. The comparison is only against LLM-based direct arm selection. We never see UCB, LinUCB, or linear Thompson sampling on the same synthetic tasks. Maybe the proposed methods beat those classical baselines, maybe not. The paper's own framing in the abstract says "consistently outperform previous baseline methods based on direct arm selection," which is accurate, but it invites the question of whether the classical exploration alone would do just as well. The dueling bandit section is weaker still: random search is the only comparator. That is too thin to support the claim of non-trivial algorithmic value.

Third, the paper calls TS-LLM a Thompson sampling variant, but the justification is thin. There is no argument that the LLM's stochastic outputs, conditioned on a history prompt, approximate a posterior over reward means. The decayed temperature acts as an exploration schedule, which is a heuristic. Naming it TS oversells the connection. The algorithm may still work, but the framing should be softened.

Overall, the central idea is promising and the paper ships enough detail to be useful. It deserves a serious referee, but with major revision: add classical baselines, move hyperparameter selection off the evaluation tasks, strengthen the dueling bandit comparison, and tone down the Thompson-sampling language.

For a reading group, it is a reasonable "maybe"—good for discussing experimental methodology in LLM-agent research, less so for theory. I would cite it if I were working on LLM-based sequential decision-making.

Referee Report

5 major / 5 minor

Summary. The paper proposes to combine classical multi-armed bandit algorithms with pre-trained LLMs by keeping the classical algorithm as the high-level decision framework and using the LLM as a reward/loss/preference predictor. Three methods are presented: TS-LLM (Thompson sampling with temperature-decayed stochastic LLM predictions), RO-LLM (SquareCB with deterministic LLM regression), and TS-LLM-DB (Borda-score-based dueling bandits). The experiments compare these methods against LLM-based direct arm selection on synthetic MAB, synthetic dueling bandits, and contextual bandits built from OneShotWikiLinks and AmazonCat-13K, and the paper claims consistent improvement over direct arm selection, particularly in tasks where arm labels have no semantic meaning.

Significance. If the empirical claims were fully supported, the paper would provide a useful design pattern: let the classical algorithm handle exploration while the LLM handles supervised reward prediction, which would also avoid specifying the reward function's parametric form. The paper is transparent about prompts and ablations, tests two different LLMs, and includes real-world text datasets. However, the current evidence is weakened by hyperparameter tuning on the evaluation tasks, by the absence of classical MAB and dueling-bandit baselines, and by an unvalidated assumption that stochastic LLM outputs mimic posterior sampling. These issues affect the load-bearing claims, so the manuscript requires major revision.

major comments (5)

[App. A.2 / Sec. 5.1 / Sec. 4.1] The central synthetic comparison is circular. App. A.2 states that the experiments use 'the optimal schedule for the temperature discovered in Sec. 5.1 (Fig. 5)', and Sec. 5.1 evaluates temperature schedules on the same linear-reward task that is reported in Sec. 4.1. Similarly, Sec. 5.3 selects the exploration parameter gamma for RO-LLM using the same linear and square reward functions that appear in Fig. 1. The headline 'consistently outperforms' therefore partly reflects fitting hyperparameters to the test tasks. The authors should tune on separate development tasks or use a validation split and then report results on held-out tasks with fixed hyperparameters.
[Sec. 4.1] No classical MAB baseline appears in the synthetic comparisons. The only baselines are LLM-based direct arm selection (NoFeature, FramingFeature, HistoryFeature). Since the proposal is to enhance classical MAB with LLMs, the reader needs to see UCB, LinUCB, and/or linear Thompson sampling to judge whether the LLM reward predictor adds value beyond classical exploration. This absence also leaves open the possibility that the gap in Fig. 1 is due to weak direct-arm-selection baselines rather than to the proposed combination.
[Sec. 3.1] The justification for calling TS-LLM a Thompson-sampling method is the assertion that 'we approximate the posterior distribution of rewards in TS using the stochastic predictions generated by the LLM.' The paper provides no derivation, calibration check, or empirical evidence that the LLM's stochastic predictions at a given temperature are distributed like the posterior over reward means. Without such support, the decaying-temperature mechanism is an arbitrary exploration heuristic, and the phrase 'posterior sampling' is not substantiated. A concrete test would be to compare LLM predictive quantiles against empirical reward residuals, or to compare TS-LLM with a proper Bayesian linear TS baseline.
[Sec. 4.2] TS-LLM-DB is evaluated only against random search. Given the paper's own statement that adapting the algorithm requires non-trivial modifications, and given that LLM-based dueling bandits from Xia et al. (2024) exist, the absence of any classical or LLM dueling-bandit baseline makes the 'small regrets' claim unsupported. At minimum, the authors should include a Borda-maximizing algorithm and a Thompson-sampling dueling bandit baseline.
[Sec. 4.3] The real-world experiments compare TS-LLM only with a self-constructed direct-arm-selection baseline. While this matches the abstract's comparison, it does not support the broader claim that the LLM-enhanced MAB approach is competitive with classical contextual bandit algorithms. Adding a standard contextual bandit baseline such as LinUCB would show whether the LLM predictor is actually beneficial in this setting.

minor comments (5)

[Sec. 5.1] Sec. 5.1 does not state the exact temperature decay schedule for TS-LLM; the only explicit schedules are for dueling bandits in App. A.2. Please report the schedule used in Fig. 5 so the tuning procedure is reproducible.
[Introduction] The introduction cites 'Koh et al. (2024)' but this reference is missing from the bibliography.
[Sec. 6] The heading of Sec. 6 contains a typo: 'Sequantial' should be 'Sequential'.
[App. A.4] In the OneShotWikiLinks prompt, the option 'help' is lowercase while the other options are capitalized; please clarify if this is intentional.
[Fig. 5] Fig. 5 would be easier to interpret with error bars or multiple random seeds, since the differences among fixed temperatures appear visually small.

Circularity Check

1 steps flagged · score 5.0 of 10

Synthetic evaluation is partly circular: the TS-LLM temperature schedule is tuned on the same linear task that is then reported as a main result, with no validation split.

fitted input called prediction [Sec. 5.1, Sec. A.2, Sec. 4.1 (Fig. 1)]
"Sec. 5.1: 'We follow the same experimental setting as Sec. 4.1 and adopt the linear reward function. The results in Fig. 5 show that the best performance is achieved by adopting decaying LLM temperatures...' Sec. A.2: 'In all our experiments here, we have adopted the optimal schedule for the temperature discovered in Sec. 5.1 (Fig. 5).'"

The decaying temperature schedule is a free hyperparameter of TS-LLM. Sec. 5.1 selects it by comparing regrets on the linear reward function using the same experimental setting as Sec. 4.1, and Sec. A.2 says this 'optimal schedule' is then used in all Sec. 4.1 synthetic experiments, which include that same linear reward function (plus square, sinusoidal, and GP rewards). Thus the linear-reward panel of Fig. 1 is not an out-of-sample test of the algorithm: it evaluates the method on the exact task used to fit its exploration schedule. Because the same schedule is applied to all four Sec. 4.1 tasks, the 'consistently outperforms' claim is at least partially a fitted-input result rather than an independent comparison.

full rationale

The strongest independent evidence is in the real-text experiments (Sec. 4.3), where the TS-LLM reward-prediction prompt is compared against a direct-arm-selection baseline using the same prompt family on OneShotWikiLinks and AmazonCat-13K; no synthetic hyperparameter tuning feeds those curves. The dueling-bandit experiments (Sec. 4.2) use temperature schedules described directly in Sec. A.2 rather than a separately tuned ablation, and they are compared only against random search. The main circularity I can pin to specific quotes is the temperature-schedule leakage: Sec. 5.1 tunes the schedule on the linear reward function, and Sec. A.2 applies that same 'optimal schedule' to all Sec. 4.1 tasks, including the linear task itself. The RO-LLM gamma parameter is also explored in Sec. 5.3 on the same linear and square reward functions used in Sec. 4.1, but the paper never states explicitly that those Sec. 5.3-selected gamma values are the ones used in Fig. 1, so I do not count that as a separately exhibited circular step. The paper's assumption that stochastic LLM outputs approximate a TS posterior (Sec. 3.1) is unsupported but is an assumption about the algorithm, not a case where the conclusion is defined in terms of its inputs. There is no load-bearing self-citation chain or imported uniqueness theorem: the citation to Verma et al. (2024) for the dueling-bandit arm-selection strategy is a design inspiration with some overlap in authorship, but the dueling results are also empirically shown, so it does not by itself force the paper's central claim. Overall, the central empirical claim is not fully circular because the text experiments are independent, but the synthetic comparison that anchors the abstract's 'consistently outperform' statement is partially contaminated by tuning on the evaluation task. Score 5 reflects a real, load-bearing evaluation leak without the whole derivation collapsing into self-definition.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four tuned numeric hyperparameters (temperature schedule, gamma, N, BTL scaling) and on the unvalidated assumption that LLM output randomness behaves like posterior sampling. There are no invented physical or conceptual entities. The main burden is empirical: hyperparameters are tuned on the same benchmarks used for evaluation, and the posterior approximation is asserted rather than derived.

free parameters (4)

Temperature decay schedule for TS-LLM / TS-LLM-DB = temp(t) = 1.5 - min(0.1*sqrt(t), 1.4) for first arm in linear dueling case; similar schedules in Sec A.2
Tuned in Sec 5.1 on the linear reward function and then applied to all synthetic tasks (Sec A.2).
Exploration parameter gamma in RO-LLM = gamma=10 for linear, gamma=5 for square (best among {1,5,10})
Selected in Sec 5.3 on the same linear and square tasks whose results are reported as main evaluation in Sec 4.1.
Number of sampled arms N for Borda estimate in TS-LLM-DB = N=15
Chosen as a compromise between performance and API cost; ablation in Sec 5.2 shows larger N improves regret.
BTL scaling factor in preference probability = 10
Introduced in Eq (1) of App A.2 to reduce noise in preference observations and simplify the dueling bandit problem; chosen by hand.

assumptions (5)

domain assumption Preference feedback follows the BTL model with a logistic link.
Assumed in Sec 2.2; all dueling bandit experiments generate data from this model, so the algorithm is only tested under this assumption.
standard math The maximizer of the Borda function coincides with the maximizer of the latent reward function.
Invoked in Sec 3.3 to justify selecting the first arm by maximizing estimated Borda scores; the paper cites Mehta et al. (2023) for this equivalence.
ad hoc to paper LLM stochastic outputs at positive temperature approximate samples from the posterior distribution of rewards.
Asserted in Sec 3.1 'Justifications' without derivation; this is the entire basis for TS-LLM's exploration mechanism.
domain assumption The LLM can predict rewards, losses, and preference probabilities from a history of feature-reward pairs accurately enough for the outer MAB algorithm to learn.
This is the core premise of the method, tested only empirically on a small set of tasks with two specific LLMs.
domain assumption Synthetic rewards are stationary and generated by the specified functions with Gaussian noise of variance 0.02.
Experimental setup in App A.2; the algorithms are not tested under non-stationarity or model misspecification beyond the tested nonlinear functions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Language Model-Enhanced Multi-Armed Bandits." pith.science (2026). https://pith.science/paper/5TGGHEPW

@misc{pith2026250201118,
  author       = {Pith},
  title        = {Pith review of: Large Language Model-Enhanced Multi-Armed Bandits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5TGGHEPW}},
  note         = {Machine review of arXiv:2502.01118}
}

read the original abstract

Large language models (LLMs) have been adopted to solve sequential decision-making tasks such as multi-armed bandits (MAB), in which an LLM is directly instructed to select the arms to pull in every iteration. However, this paradigm of direct arm selection using LLMs has been shown to be suboptimal in many MAB tasks. Therefore, we propose an alternative approach which combines the strengths of classical MAB and LLMs. Specifically, we adopt a classical MAB algorithm as the high-level framework and leverage the strong in-context learning capability of LLMs to perform the sub-task of reward prediction. Firstly, we incorporate the LLM-based reward predictor into the classical Thompson sampling (TS) algorithm and adopt a decaying schedule for the LLM temperature to ensure a transition from exploration to exploitation. Next, we incorporate the LLM-based reward predictor (with a temperature of 0) into a regression oracle-based MAB algorithm equipped with an explicit exploration mechanism. We also extend our TS-based algorithm to dueling bandits where only the preference feedback between pairs of arms is available, which requires non-trivial algorithmic modifications. We conduct empirical evaluations using both synthetic MAB tasks and experiments designed using real-world text datasets, in which the results show that our algorithms consistently outperform previous baseline methods based on direct arm selection. Interestingly, we also demonstrate that in challenging tasks where the arms lack semantic meanings that can be exploited by the LLM, our approach achieves considerably better performance than LLM-based direct arm selection.

Figures

Figures reproduced from arXiv: 2502.01118 by the authors.

**Figure 1.** The performance of our TS-LLM and RO-LLM algorithms in classical stochastic MAB tasks. approximately maximizing the Borda function fborda (Xu et al., 2020), which is defined as the expected probability that an arm is preferred over a randomly selected arm: fborda(x) = Ej∈U([K])[P(x ≻ xj )] where U([K]) denotes the uniform distribution among all K arms. Specifically, we estimate the expectation in fborda(x) by unifo… view at source ↗

**Figure 2.** The performance of our TS-LLM-DB algorithm in dueling bandits with linear and square latent reward functions. schedule of temperature when selecting both arms. Since our TS-LLM-DB selects the first arm greedily (i.e., pure exploitation) and chooses the second arm optimistically by balancing exploration and exploitation (Sec. 3.3), we adopt a schedule of smaller temperatures when selecting the first arm to encourage… view at source ↗

**Figure 3.** The cumulative rewards in the text experiments using the OneShotWikiLinks and AmazonCat-13K datasets (Sec. 4.3). ducted an experiment using the AmazonCat-13K dataset in a more challenging setting, i.e., with a larger number of arms (i.e., 30). The results ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: The performance of our TS-LLM algorithm in stochastic MAB tasks with different temperatures. 5.2. Impact of the Number N of Samples When Selecting the First Arm in TS-LLM-DB Recall that our TS-LLM-DB algorithm selects the first arm by approximately maximizing the Borda…

**Figure 4.** Figure 4: The cumulative rewards in the text experiments using the AmazonCat-13K dataset with K = 30 arms. 5. Ablation Study 5.1. Impact of Different Temperatures Here we investigate the impact of the temperature of the LLM on the performance of our TS-LLM (Algo. 1). As we have …

**Figure 6.** Figure 6: The impact of the number N of uniformly sampled arms when estimating the Borda function to select the first arm in our TS-LLM-DB algorithm (lines 3-5 of Algo. 3). the results in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: The impact of the exploration parameter γ in our RO-LLM algorithm. 6. Related Work LLM-Based Multi-Armed Bandits (MAB). The work of Krishnamurthy et al. (2024) has used an LLM to sequentially choose the arms in MAB. They have consider standard MAB problems with a fini…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex
cs.LG 2026-07 conditional novelty 7.0 of 10

Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

Reference graph

Works this paper leans on

21 extracted references · 2 canonical work pages · cited by 1 Pith paper

[3]

Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects

Cheng, Y ., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., et al. Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428,

arXiv
[4]

In-context exploration- exploitation for reinforcement learning

Dai, Z., Tomasi, F., and Ghiassian, S. In-context exploration- exploitation for reinforcement learning. arXiv preprint arXiv:2403.06826,

arXiv
[5]

M., Hao, B., and Van Roy, B

Dwaracherla, V ., Asghari, S. M., Hao, B., and Van Roy, B. Efficient exploration for LLMs. arXiv preprint arXiv:2402.00396,

arXiv
[8]

J., Zhang, C., and Slivkins, A

Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore in-context? arXiv preprint arXiv:2403.15371,

arXiv
[9]

In-context reinforcement learning with algorithm distillation

Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215,

arXiv
[10]

Feel-Good Thompson Sam- pling for Contextual Dueling Bandits

Li, X., Zhao, H., and Gu, Q. Feel-Good Thompson Sam- pling for Contextual Dueling Bandits. arXiv preprint arXiv:2404.06013,

arXiv
[11]

Lin, X., Dai, Z., Verma, A., Ng, S.-K., Jaillet, P., and Low, B. K. H. Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346,

arXiv
[12]

DeepSeek- v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models to enhance bayesian optimization. arXiv preprint arXiv:2402.03921, 2024b. Liu, X., Yu, H., Zhang, H., Xu,...

arXiv

Show all 21 references

[14]

OpenAI. ChatGPT. https://chat.openai.com, 2023a. 9 Large Language Model-Enhanced Multi-Armed Bandits OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023b. Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances i...

arXiv
[15]

Wikilinks: A large-scale cross-document coreference cor- pus labeled via links to wikipedia

Singh, S., Subramanya, A., Pereira, F., and McCallum, A. Wikilinks: A large-scale cross-document coreference cor- pus labeled via links to wikipedia. University of Mas- sachusetts, Amherst, Tech. Rep. UM-CS-2012, 15,

2012
[17]

A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024a

Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y ., et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024a. Wang, Z., Xie, J., Liu, X., Li, S., and Lui, J. Online cluste...

arXiv
[18]

The rise and potential of large language model based agents: A survey

Xi, Z., Chen, W., Guo, X., He, W., Ding, Y ., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864,

arXiv
[19]

Agentgym: Evolving large language model-based agents across di- verse environments

Xi, Z., Ding, Y ., Chen, W., Hong, B., Guo, H., Wang, J., Yang, D., Liao, C., Guo, X., He, W., et al. Agentgym: Evolving large language model-based agents across di- verse environments. arXiv preprint arXiv:2406.04151,

arXiv
[20]

Beyond numeric awards: In-context dueling bandits with llm agents.arXiv preprint arXiv:2407.01887,

Xia, F., Liu, H., Yue, Y ., and Li, T. Beyond numeric awards: In-context dueling bandits with llm agents.arXiv preprint arXiv:2407.01887,

arXiv
[2004]

Y ., McAleer, S., Fried, D., and Salakhutdinov, R

Koh, J. Y ., McAleer, S., Fried, D., and Salakhutdinov, R. Tree search for language model agents. arXiv preprint arXiv:2407.01476,
[2012]

Llama-berry: Pairwise optimization for o1-like olympiad-level math- ematical reasoning

Zhang, D., Wu, J., Lei, J., Che, T., Li, J., Xie, T., Huang, X., Zhang, S., Pavone, M., Li, Y ., et al. Llama-berry: Pairwise optimization for o1-like olympiad-level math- ematical reasoning. arXiv preprint arXiv:2410.02884 ,

arXiv
[2016]

Bi, Z., Han, K., Liu, C., Tang, Y ., and Wang, Y

URL http://manikvarma.org/ downloads/XC/XMLRepository.html. Bi, Z., Han, K., Liu, C., Tang, Y ., and Wang, Y . Forest- of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078,

arXiv
[2017]

J., Wang, Z., Wang, D

Hao, S., Gu, Y ., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 ,

arXiv
[2018]

Verma, A., Dai, Z., Lin, X., Jaillet, P., and Low, B. K. H. Neural dueling bandits. arXiv preprint arXiv:2407.17112,

arXiv
[2023]

P., Xie, Q., and Nowak, R

Mukherjee, S., Hanna, J. P., Xie, Q., and Nowak, R. Pre- training decision transformers with reward prediction for in-context multi-task structured bandit learning. arXiv preprint arXiv:2406.05064,
[2024]

Efficient sequential deci- sion making with large language models

Chen, D., Zhang, Q., and Zhu, Y . Efficient sequential deci- sion making with large language models. arXiv preprint arXiv:2406.12125,

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [3]

Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects

Cheng, Y ., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., et al. Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428,

arXiv

[2] [4]

In-context exploration- exploitation for reinforcement learning

Dai, Z., Tomasi, F., and Ghiassian, S. In-context exploration- exploitation for reinforcement learning. arXiv preprint arXiv:2403.06826,

arXiv

[3] [5]

M., Hao, B., and Van Roy, B

Dwaracherla, V ., Asghari, S. M., Hao, B., and Van Roy, B. Efficient exploration for LLMs. arXiv preprint arXiv:2402.00396,

arXiv

[4] [8]

J., Zhang, C., and Slivkins, A

Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore in-context? arXiv preprint arXiv:2403.15371,

arXiv

[5] [9]

In-context reinforcement learning with algorithm distillation

Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215,

arXiv

[6] [10]

Feel-Good Thompson Sam- pling for Contextual Dueling Bandits

Li, X., Zhao, H., and Gu, Q. Feel-Good Thompson Sam- pling for Contextual Dueling Bandits. arXiv preprint arXiv:2404.06013,

arXiv

[7] [11]

Lin, X., Dai, Z., Verma, A., Ng, S.-K., Jaillet, P., and Low, B. K. H. Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346,

arXiv

[8] [12]

DeepSeek- v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models to enhance bayesian optimization. arXiv preprint arXiv:2402.03921, 2024b. Liu, X., Yu, H., Zhang, H., Xu,...

arXiv