Pith. sign in

REVIEW 2 major objections 3 minor 1 cited by

Combinatorial Reinforcement Learning with Preference Feedback

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that MNL-VQL, an algorithm for offering item sets and learning from user picks, achieves almost minimax-optimal regret in combinatorial RL, with matching upper and lower bounds for linear MDPs.

desk verdict Genuinely new problem setting and a plausible lower bound, but the main upper bound rests on a total-variance lemma that is false as stated, so the minimax claim does not follow. read the letter →

arxiv 2502.10158 v3 pith:LCHTXZLA submitted 2025-02-14 stat.ML cs.LG

Joongkyu Lee , Min-hwan Oh This is my paper · ORCID
classification stat.MLcs.LG MSC 68Q3290C40
keywords combinatorialreinforcementlearningpreferencefeedbackmultinomiallogitmodelassortmentoptimizationgeneralfunctionapproximationgeneralizedEluderdimensionlinearMDPminimaxregret
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that combinatorial reinforcement learning—where an agent offers a set of items and observes which one a user picks under a multinomial logit model—can be solved with a computationally tractable algorithm whose regret is near the information-theoretic minimum. It proposes MNL-VQL, which estimates both the user's MNL preferences and the long-run value of each item, selects assortments by solving a linear program, and uses variance-weighted regression to keep exploration tight. For general function approximation the algorithm achieves $\tilde{O}(d\sqrt{HK} + \frac{1}{\kappa}d^2H^2 + \sqrt{d_\nu H K \log N} + d_\nu H^5 \log N \log^2(N N_b))$ regret. For linear MDPs the bound becomes $\tilde{O}(d\sqrt{HK}+d_{\mathrm{lin}}\sqrt{HK})$, matched by a lower bound $\Omega(d\sqrt{HK}+d_{\mathrm{lin}}\sqrt{HK})$, making MNL-VQL nearly minimax-optimal in that class. If correct, this is the first statistical guarantee for combinatorial RL with preference feedback, extending MNL bandits from single-step decisions to long-horizon recommendation problems.

What carries the argument

The central object is the item-level Q-value decomposition $Q_h(s,A)=\sum_{a\in A} P_h(a|s,A)\,Q_h(s,a)$, with $Q_h(s,a)$ estimated by a general function class $\mathcal{F}_h$. The load-bearing mechanism is the alternating optimistic/pessimistic choice probability of Equation (7): when an estimated item value exceeds the outside option the algorithm inflates the utility, otherwise it deflates it, and Lemma D.5 shows a subset of any assortment dominates both the true and the optimistic expectations, guaranteeing optimism without enumerating all $O(|I|^M)$ assortments. Efficiency comes from rewriting assortment selection as a linear program via the Charnes-Cooper transformation. Tightness comes from variance-weighted least squares with a consistent bonus oracle and from the law of total variance, which saves a factor $\sqrt{H}$ relative to summing $H$ MNL-bandit regrets.

What would settle it

Run MNL-VQL on the Section 6.1 linear-MDP synthetic environment with $N=40$, $M=6$, $H=5$, $K=30000$, across at least ten seeds, and compare mean regret to $C(d\sqrt{HK}+d_{\mathrm{lin}}\sqrt{HK})$ for the Theorem 5.2 constant; if any seed's cumulative regret grows linearly in $K$ or systematically exceeds that envelope, the claimed regret order is wrong.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that optimism can be maintained while keeping assortment optimization tractable even though the item-level Q-values are unknown. The key decomposition is that the Q-value of an assortment is a weighted sum of item-level Q-values, with weights given by MNL choice probabilities. Because a naive pointwise bonus over all assortments would be exponentially expensive, MNL-VQL constructs optimistic and pessimistic MNL utilities and alternates between them depending on whether the estimated outside-option value is largest (Equation (7)); this provably yields an optimistic value estimate (Lemma D.15). The resulting assortment-selection subproblem is a fractional linear program, solvable in polynomial time in the item count. Regret is then bounded by splitting it into MNL-model error, controlled through an online mirror-descent confidence set and the law of total variance, and item-value estimation error, controlled through the generalized Eluder dimension and a consistent bonus oracle. Theorems 5.2 and 5.3 close the loop for linear MDPs with matching upper and lower bounds.

Load-bearing premise

The result stands on Assumption 3.3: for every horizon and every value function reachable by the algorithm, the same function class must contain the exact one-step Bellman image and the exact second-moment Bellman image; without that closure the optimism and variance estimates that carry the bound do not follow.

Editorial extensions

If this is right

  • For $H=1$ the bound reduces to $\tilde{O}(d\sqrt{K} + \kappa^{-1}d^2)$, matching the known minimax MNL-bandit rate, so the framework strictly generalizes MNL bandits.
  • For linear MDPs and large enough $K$, regret is $\tilde{O}(d\sqrt{HK} + d_{\mathrm{lin}}\sqrt{HK})$, which matches the new lower bound, so MNL-VQL is nearly minimax-optimal in that class.
  • The assortment-selection problem is solved in polynomial time via a linear-programming reformulation, so the algorithm avoids enumerating the exponentially many assortments.
  • The regret analysis uses the law of total variance rather than summing $H$ MNL-bandit regrets, saving a factor of $\sqrt{H}$ in the horizon dependence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the $\sqrt{H}$ saving suggests preference learning over a long session should not pay a separate exploration cost at every step; testing MNL-VQL against a per-step MNL-bandit baseline on a fixed linear MDP would isolate that effect.
  • Editorial extension: because the general-function bound is stated in terms of the generalized Eluder dimension, the same algorithm should have a workable regret guarantee for nonlinear item-value classes if that dimension is finite; the paper only works out the linear-MDP instantiation, so this application is a projection, not a claim.
  • Editorial extension: the second-moment completeness in Assumption 3.3 is what lets the variance estimator avoid pessimism; a natural follow-up would be a variant needing only first-moment closure, since many practical function classes satisfy the first but not the second.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces a combinatorial reinforcement learning setting in which, at each step, the agent offers an assortment of items and receives MNL preference feedback, with item-level Q-values approximated by a general function class. The proposed algorithm, MNL-VQL, combines online mirror descent for the MNL parameters with variance-weighted optimistic value learning, and uses an optimistic/pessimistic utility construction together with an LP reformulation to select assortments efficiently. The main claims are a regret upper bound O~(d√(HK) + (1/κ)d^2H^2 + √(dνHK log N) + dνH^5 log(NNb log)^2) for general function approximation, a specialization to linear MDPs, and a matching lower bound Ω(d√(HK) + dlin√(HK)). The proof structure is elaborate, with separate lemmas for MNL regret, optimism, variance estimation, and bonus bounds, plus a hard-instance construction for the lower bound.

Significance. If the main theorems were correct, this would be the first statistical guarantee for combinatorial RL with MNL preference feedback, and the linear-MDP lower bound together with the upper bound would establish near-minimax optimality. The problem formulation is timely and the algorithmic idea of alternating optimistic and pessimistic utilities to maintain optimism with unknown item values is a genuine technical novelty. The lower-bound construction, with its layered linear MDP and MNL preference model, is also a useful contribution. However, the central upper-bound proof relies on a false total-variance lemma, so the claimed guarantees are not currently established.

major comments (2)
  1. [Appendix D.5, Lemma D.12 and its use in Lemma D.13] Lemma D.12 states that for any f^k_{h,j} in [0,1], with probability at least 1-δ, ∑_{k,h} Var_h[f^k_{h,j}](s^k_h) = O(K + H log(1/δ)). This is false as stated. Consider S={0,1}, items {a0,a1,a2}, transition kernel P(s'=1|s,a)=1/2 for all (s,a), zero rewards, and F = span{1, 1_{a=a1}, 1_{a=a2}}. For every V:S→[0,1], T_hV and T_h^2V are constant functions, so Assumption 3.3 is satisfied. Choose f(s,a1)=1, f(s,a2)=0, f(s,a0)=1/2, and set the MNL utilities to zero so that p(a1)=p(a2) at every state; then Var_h[f] is a positive constant at every visited state, making the sum over K episodes and H steps Ω(KH). This contradicts the claimed O(K+H log(1/δ)). Since Lemma D.13 uses exactly this bound at display (D.35) — replacing √(∑ Var) by √(K+H log(1/δ)) — the subsequent Cauchy-Schwarz step yields dH√K rather than d√(HK) when the variance is Ω(KH). Therefore Theorem D.25 (Theorem 5.1), the linear-MDP upper bound in Theorem 5.2, and the claimed match with the lower bound in Theorem 5.3 are not supported by the supplied proof.
  2. [Appendix C, Eq. (C.4)] The Charnes-Cooper transformation used to solve (8) omits the constraints y_a ≥ 0 and y_a ≤ t. These constraints are required to recover x_a = y_a/t in [0,1] from a solution of the LP; without them, an optimal solution to (C.4) can correspond to an infeasible assortment that includes more than one copy of an item or exceeds the cardinality bound. The text also asserts, following Chen and Hausman, that the binary indicator can be relaxed, but it does not provide the integrality argument for the resulting fractional LP. Since the claim of polynomial-time assortment optimization in Remark 4.2 rests on this reformulation, the constraint set and the integrality claim need to be corrected and proved explicitly.
minor comments (3)
  1. [Appendix D.1, Lemma D.6] The statement defines J(k,h): K×[H] → {1,2} as a one-to-one function, which is impossible when K×[H] has more than two elements; it should instead say that J(k,h) is the index satisfying A^k_h = A^k_{h,J(k,h)}.
  2. [Appendix C, last paragraph] There is a typo: 'This, in ture' should read 'This, in turn'.
  3. [Assumption 3.3 and Remark 3.4] The second-moment completeness assumption, while explicitly acknowledged as essential, is very strong and excludes many practical nonlinear function classes; the main text would benefit from a more prominent discussion of this limitation and of which function classes beyond tabular and linear MDPs satisfy it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central regret bound is an a priori high-probability inequality; its imports from the authors' prior work are independent published theorems with stated assumptions, and the new assortment-optimism and lower-bound constructions are not definitionally equivalent to their inputs.

full rationale

The derivation chain was checked against all seven circularity patterns. The MNL confidence interval (Corollary D.2 / Proposition D.1, citing Lemma 1 of Lee & Oh 2024) and the MNL lower-bound lemma (Proposition F.4, citing Lee & Oh 2024) are imported from the same authors' earlier work, but they are published theorems with explicit, parameter-free assumptions that do not include the target regret bound of this paper. Under the review rules, such citations count as independent support rather than load-bearing circularity. The general-function-approximation machinery—consistent bonus oracle, generalized Eluder dimension, variance-weighted regression, and the associated confidence lemmas (Definitions B.1, D.3, Lemmas D.21–D.23)—is taken from Agarwal et al. (2023), an external source, and none of it is fitted to the present data or relabeled as a prediction. The genuinely new steps, especially Equation (7)'s optimistic/pessimistic utility construction and the LP reformulation in Appendix C, are self-contained reductions from the stated structural assumptions (Assumptions 3.1 and 3.3). The regret bound is an a priori high-probability statement over the random learning process, not a fitted quantity renamed as a prediction. The lower-bound construction in Appendix F combines independent published lower-bound tools (Lee & Oh 2024; Zhou et al. 2021a) with a new multi-layered linear MDP; it does not presuppose the upper bound it is meant to match. Accordingly, no circular step is exhibited. A possible mathematical defect in Lemma D.12's total-variance bound would be a correctness concern, not a circularity concern, and is therefore not scored here.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No data-fitted parameters appear in the central theory; all constants and radii are chosen explicitly in Table D.2. The central claim depends on four structural premises (MNL model, κ lower bound, strong completeness, bonus oracle) and on the imported generalized Eluder dimension and sub-sampling framework. No new entities are introduced.

assumptions (4)
  • domain assumption User choices follow the multinomial logistic model with linear utilities (Assumption 3.1).
    The entire algorithm and analysis are built on this choice model; if true choice probabilities are not MNL, the confidence sets and assortment-optimism lemmas do not apply.
  • domain assumption There exists a problem-dependent constant κ bounding P_h(a|s,A)P_h(a0|s,A) from below (Definition 3.2).
    κ controls the elliptical-potential bounds in Lemmas D.7, D.10, and D.13; the regret bound carries a 1/κ term that can be exponentially large, so this is a real restriction on the guarantee.
  • domain assumption Completeness and realizability, including second-moment completeness (Assumption 3.3).
    Requires F_h to contain T_h V and T^2_h V for every bounded V; the variance estimation and all version-space arguments rely on it, and it fails for many nonlinear function classes.
  • domain assumption A consistent bonus oracle B exists and its output class W has size Nb (Definition B.1).
    The optimism proof and bonus bounds in Lemmas D.14, D.21-D.23 assume this oracle; for general F it is implemented via online sub-sampling, which additionally requires finite generalized Eluder dimension.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combinatorial Reinforcement Learning with Preference Feedback." pith.science (2026). https://pith.science/paper/LCHTXZLA

@misc{pith2026250210158,
  author       = {Pith},
  title        = {Pith review of: Combinatorial Reinforcement Learning with Preference Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCHTXZLA}},
  note         = {Machine review of arXiv:2502.10158}
}
read the original abstract

In this paper, we consider combinatorial reinforcement learning with preference feedback, where a learning agent sequentially offers an action--an assortment of multiple items to--a user, whose preference feedback follows a multinomial logistic (MNL) model. This framework allows us to model real-world scenarios, particularly those involving long-term user engagement, such as in recommender systems and online advertising. However, this framework faces two main challenges: (1) the unknown value of each item, unlike traditional MNL bandits that only address single-step preference feedback, and (2) the difficulty of ensuring optimism while maintaining tractable assortment selection in the combinatorial action space with unknown values. In this paper, we assume a contextual MNL preference model, where the mean utilities are linear, and the value of each item is approximated by a general function. We propose an algorithm, MNL-VQL, that addresses these challenges, making it both computationally and statistically efficient. As a special case, for linear MDPs (with the MNL preference feedback), we establish the first regret lower bound in this framework and show that MNL-VQL achieves nearly minimax-optimal regret. To the best of our knowledge, this is the first work to provide statistical guarantees in combinatorial RL with preference feedback.

Figures

Figures reproduced from arXiv: 2502.10158 by the authors.

Figure 1
Figure 1. Synthetic experiment: Episodic returns averaged over 10 runs. Dotted lines indicate estimated returns for incomplete runs due to excessive runtime. Shading denotes ˘1 standard deviation. 2500 5000 7500 10000 Episodes 1.5 2.0 2.5 3.0 N=50, |A|=20875 2500 5000 7500 10000 Episodes 1.5 2.0 2.5 3.0 N=100, |A|=166750 2500 5000 7500 10000 Episodes 1.5 2.0 2.5 3.0 N=200, |A|=1333500 MNL-VQL (ours) LSVI-UCB (holistic) Myopic… view at source ↗
Figure 2
Figure 2. MovieLens experiment: The dotted lines represent estimated (virtual) episodic returns for cases that could not be run due to excessively long runtimes. Shaded regions represent ˘1 standard deviation. of movies a user u has watched after entering the system, denoted by s “ pu, nq, where n P t0, . . . , H ´ 1u is the number of movies watched during the session. We interpret the ratings as representing MNL utilities. I… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improved Online Confidence Bounds for Multinomial Logistic Bandits

    stat.ML 2025-02 conditional novelty 7.0 of 10

    New ℓ∞-self-concordant analysis and Ville's-inequality martingale control yield an online confidence bound of O(√(d log t) + B√d), leading to variance-dependent MNL bandit regret with no K dependence and only asymptot...

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Instance-wise minimax-optimal algorithms for logistic bandits

    Abeille, M., Faury, L., and Calauz \`e nes, C. Instance-wise minimax-optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics, pp.\ 3691--3699. PMLR, 2021

    work page 2021

  2. [2]

    Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation

    Agarwal, A., Jin, Y., and Zhang, T. Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. In The Thirty Sixth Annual Conference on Learning Theory, pp.\ 987--1063. PMLR, 2023

    work page 2023

  3. [3]

    A tractable online learning algorithm for the multinomial logit contextual bandit

    Agrawal, P., Tulabandhula, T., and Avadhanula, V. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research, 310 0 (2): 0 737--750, 2023

    work page 2023

  4. [4]

    and Goyal, N

    Agrawal, S. and Goyal, N. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pp.\ 127--135. PMLR, 2013

    work page 2013

  5. [5]

    Thompson sampling for the mnl-bandit

    Agrawal, S., Avadhanula, V., Goyal, V., and Zeevi, A. Thompson sampling for the mnl-bandit. In Conference on learning theory, pp.\ 76--78. PMLR, 2017

    work page 2017

  6. [6]

    Mnl-bandit: A dynamic learning approach to assortment selection

    Agrawal, S., Avadhanula, V., Goyal, V., and Zeevi, A. Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research, 67 0 (5): 0 1453--1485, 2019

    work page 2019

  7. [7]

    April: Active preference learning-based reinforcement learning

    Akrour, R., Schoenauer, M., and Sebag, M. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp.\ 116--131. Springer, 2012

    2012

  8. [8]

    and Thrampoulidis, C

    Amani, S. and Thrampoulidis, C. Ucb-based algorithms for multinomial logistic regression bandits. Advances in Neural Information Processing Systems, 34: 0 2913--2924, 2021

    work page 2021

Show all 67 references
  1. [9]

    Distributional off-policy evaluation for slate recommendations

    Chaudhari, S., Arbour, D., Theocharous, G., and Vlassis, N. Distributional off-policy evaluation for slate recommendations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 8265--8273, 2024

    2024

  2. [10]

    Chen, K. D. and Hausman, W. H. Mathematical properties of the optimal product line selection problem using choice-based conjoint analysis. Management Science, 46 0 (2): 0 327--332, 2000

    2000

  3. [11]

    Combinatorial multi-armed bandit: General framework and applications

    Chen, W., Wang, Y., and Yuan, Y. Combinatorial multi-armed bandit: General framework and applications. In International conference on machine learning, pp.\ 151--159. PMLR, 2013

    2013

  4. [12]

    Cheung, W. C. and Simchi-Levi, D. Thompson sampling for online personalized assortment optimization problems with multinomial logit choice models. Available at SSRN 3075658, 2017

    2017

  5. [13]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    S., Proutiere, A., et al

    Combes, R., Talebi Mazraeh Shahi, M. S., Proutiere, A., et al. Combinatorial bandits revisited. Advances in neural information processing systems, 28, 2015

    2015

  7. [15]

    Cooper, A. C. W. et al. Programming with linear fractional functionals. Naval Research logistics quarterly, 9 0 (3): 0 181--186, 1962

    1962

  8. [16]

    Assortment planning under the multinomial logit model with totally unimodular constraint structures

    Davis, J., Gallego, G., and Topaloglu, H. Assortment planning under the multinomial logit model with totally unimodular constraint structures. department of ieor, columbia university, 2013

    2013

  9. [17]

    Reinforcement learning with combinatorial actions: An application to vehicle routing

    Delarue, A., Anderson, R., and Tjandraatmadja, C. Reinforcement learning with combinatorial actions: An application to vehicle routing. Advances in Neural Information Processing Systems, 33: 0 609--620, 2020

    2020

  10. [18]

    Bilinear classes: A structural framework for provable generalization in rl

    Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., and Wang, R. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning, pp.\ 2826--2836. PMLR, 2021

    2021

  11. [19]

    Cascading reinforcement learning

    Du, Y., Srikant, R., and Chen, W. Cascading reinforcement learning. arXiv preprint arXiv:2401.08961, 2024

    2024 arXiv

  12. [20]

    Improved optimistic algorithms for logistic bandits

    Faury, L., Abeille, M., Calauz \`e nes, C., and Fercoq, O. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning, pp.\ 3052--3060. PMLR, 2020

    2020

  13. [21]

    Jointly efficient and optimal algorithms for logistic bandits

    Faury, L., Abeille, M., Jun, K.-S., and Calauz \`e nes, C. Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics, pp.\ 546--580. PMLR, 2022

    2022

  14. [22]

    Parametric bandits: The generalized linear case

    Filippi, S., Capp\' e , O., Garivier, A., and Szepesv\' a ri, C. Parametric bandits: The generalized linear case. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1, NIPS'10, pp.\ 586–594, Red Hook, NY, USA, 2010. Curran Ass...

    2010

  15. [23]

    J., Kakade, S

    Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487, 2021

    2021 arXiv

  16. [24]

    Gheshlaghi Azar, M., Munos, R., and Kappen, H. J. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning, 91: 0 325--349, 2013

    2013

  17. [25]

    Deep reinforcement learning with a combinatorial action space for predicting popular reddit threads

    He, J., Ostendorf, M., He, X., Chen, J., Gao, J., Li, L., and Deng, L. Deep reinforcement learning with a combinatorial action space for predicting popular reddit threads. arXiv preprint arXiv:1606.03667, 2016

    2016 arXiv

  18. [26]

    Slateq: A tractable decomposition for reinforcement learning with recommendation sets

    Ie, E., Jain, V., Wang, J., Narvekar, S., Agarwal, R., Wu, R., Cheng, H.-T., Chandra, T., and Boutilier, C. Slateq: A tractable decomposition for reinforcement learning with recommendation sets. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial I...

    2019

  19. [27]

    Randomized exploration in reinforcement learning with general value function approximation

    Ishfaq, H., Cui, Q., Nguyen, V., Ayoub, A., Yang, Z., Wang, Z., Precup, D., and Yang, L. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning, volume 139, pp.\ 4607--4616. PMLR, 2021

    2021

  20. [28]

    Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pp.\ 1704--1713. PMLR, 2017

    2017

  21. [29]

    Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. Is q-learning provably efficient? In Advances in Neural Information Processing Systems, volume 31, pp.\ 4868--4878, 2018

    2018

  22. [30]

    Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pp.\ 2137--2143. PMLR, 2020

    2020

  23. [31]

    Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms

    Jin, C., Liu, Q., and Miryoosefi, S. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 0 13406--13418, 2021

    2021

  24. [32]

    Kong, D., Salakhutdinov, R., Wang, R., and Yang, L. F. Online sub-sampling for reinforcement learning with general function approximation. arXiv preprint arXiv:2106.07203, 2021

    2021 arXiv

  25. [33]

    Cascading bandits: Learning to rank in the cascade model

    Kveton, B., Szepesvari, C., Wen, Z., and Ashkan, A. Cascading bandits: Learning to rank in the cascade model. In International conference on machine learning, pp.\ 767--776. PMLR, 2015 a

    2015

  26. [34]

    Combinatorial cascading bandits

    Kveton, B., Wen, Z., Ashkan, A., and Szepesvari, C. Combinatorial cascading bandits. Advances in Neural Information Processing Systems, 28, 2015 b

    2015

  27. [35]

    and Hutter, M

    Lattimore, T. and Hutter, M. Pac bounds for discounted mdps. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23, pp.\ 320--334. Springer, 2012

    2012

  28. [36]

    and Oh, M.-h

    Lee, J. and Oh, M.-h. Demystifying linear mdps and novel dynamics aggregation framework. In The Twelfth International Conference on Learning Representations, 2023

    2023

  29. [37]

    and Oh, M.-h

    Lee, J. and Oh, M.-h. Nearly minimax optimal regret for multinomial logistic bandit. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  30. [38]

    Online learning to rank with features

    Li, S., Lattimore, T., and Szepesv \'a ri, C. Online learning to rank with features. In International Conference on Machine Learning, pp.\ 3856--3865. PMLR, 2019

    2019

  31. [39]

    Modelling the choice of residential location

    McFadden, D. Modelling the choice of residential location. 1977

    1977

  32. [40]

    Counterfactual evaluation of slate recommendations with sequential reward interactions

    McInerney, J., Brost, B., Chandar, P., Mehrotra, R., and Carterette, B. Counterfactual evaluation of slate recommendations with sequential reward interactions. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.\ 1779--1788, 2020

    2020

  33. [41]

    Discrete sequential prediction of continuous actions for deep rl

    Metz, L., Ibarz, J., Jaitly, N., and Davidson, J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035, 2017

    2017 arXiv

  34. [42]

    M., and Van Erven, T

    Mhammedi, Z., Koolen, W. M., and Van Erven, T. Lipschitz adaptivity with multiple learning rates in online learning. In Conference on Learning Theory, pp.\ 2490--2511. PMLR, 2019

    2019

  35. [43]

    and Iyengar, G

    Oh, M.-h. and Iyengar, G. Thompson sampling for multinomial logit contextual bandits. Advances in Neural Information Processing Systems, 32, 2019

    2019

  36. [44]

    and Iyengar, G

    Oh, M.-h. and Iyengar, G. Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 9205--9213, 2021

    2021

  37. [45]

    A modern introduction to online learning

    Orabona, F. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019

    1912 arXiv

  38. [46]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  39. [47]

    and Goyal, V

    Perivier, N. and Goyal, V. Dynamic pricing and assortment under a contextual mnl demand. Advances in Neural Information Processing Systems, 35: 0 3461--3474, 2022

    2022

  40. [48]

    M., and Shmoys, D

    Rusmevichientong, P., Shen, Z.-J. M., and Shmoys, D. B. Dynamic assortment optimization with a multinomial logit choice model and capacity constraint. Operations research, 58 0 (6): 0 1666--1680, 2010

    2010

  41. [49]

    and Van Roy, B

    Russo, D. and Van Roy, B. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems, pp.\ 2256--2264, 2013

    2013

  42. [50]

    Caql: Continuous action q-learning

    Ryu, M., Chow, Y., Anderson, R., Tjandraatmadja, C., and Boutilier, C. Caql: Continuous action q-learning. arXiv preprint arXiv:1909.12397, 2019

    1909 arXiv

  43. [51]

    Dueling rl: Reinforcement learning with trajectory preferences

    Saha, A., Pacchiano, A., and Lee, J. Dueling rl: Reinforcement learning with trajectory preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 6263--6289. PMLR, 2023

    2023

  44. [52]

    and Zeevi, A

    Saur \'e , D. and Zeevi, A. Optimal dynamic assortment planning with demand learning. Manufacturing & Service Operations Management, 15 0 (3): 0 387--404, 2013

    2013

  45. [53]

    Deep reinforcement learning with attention for slate markov decision processes with high-dimensional states and actions

    Sunehag, P., Evans, R., Dulac-Arnold, G., Zwols, Y., Visentin, D., and Coppin, B. Deep reinforcement learning with attention for slate markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124, 2015

    2015 arXiv

  46. [54]

    Off-policy evaluation for slate recommendation

    Swaminathan, A., Krishnamurthy, A., Agarwal, A., Dudik, M., Langford, J., Jose, D., and Zitouni, I. Off-policy evaluation for slate recommendation. Advances in Neural Information Processing Systems, 30, 2017

    2017

  47. [55]

    Composite convex minimization involving self-concordant-like cost functions

    Tran-Dinh, Q., Li, Y.-H., and Cevher, V. Composite convex minimization involving self-concordant-like cost functions. In Modelling, Computation and Optimization in Information Systems and Management Sciences: Proceedings of the 3rd International Conference on Modelling, Comput...

    2015

  48. [56]

    Control variates for slate off-policy evaluation

    Vlassis, N., Chandrashekar, A., Amat, F., and Kallus, N. Control variates for slate off-policy evaluation. Advances in Neural Information Processing Systems, 34: 0 3667--3679, 2021

    2021

  49. [57]

    R., and Yang, L

    Wang, R., Salakhutdinov, R. R., and Yang, L. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems, 33, 2020

    2020

  50. [58]

    S., and Krishnamurthy, A

    Wang, Y., Wang, R., Du, S. S., and Krishnamurthy, A. Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=CBmJwzneppz

    2021

  51. [59]

    A survey of preference-based reinforcement learning methods

    Wirth, C., Akrour, R., Neumann, G., and F \"u rnkranz, J. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 0 (136): 0 1--46, 2017

    2017

  52. [60]

    and Wang, M

    Yang, L. and Wang, M. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pp.\ 6995--7004. PMLR, 2019

    2019

  53. [61]

    D., and Sun, W

    Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816, 2023

    2023 arXiv

  54. [62]

    and Sugiyama, M

    Zhang, Y.-J. and Sugiyama, M. Online (multinomial) logistic bandit: Improved regret and constant computation cost. Advances in Neural Information Processing Systems, 36, 2024

    2024

  55. [63]

    A nearly optimal and low-switching algorithm for reinforcement learning with general function approximation

    Zhao, H., He, J., and Gu, Q. A nearly optimal and low-switching algorithm for reinforcement learning with general function approximation. arXiv preprint arXiv:2311.15238, 2023

    2023

  56. [64]

    Nearly minimax optimal reinforcement learning for linear mixture markov decision processes

    Zhou, D., Gu, Q., and Szepesvari, C. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pp.\ 4532--4576. PMLR, 2021 a

    2021

  57. [65]

    Provably efficient reinforcement learning for discounted mdps with feature mapping

    Zhou, D., He, J., and Gu, Q. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning, pp.\ 12793--12802. PMLR, 2021 b

    2021

  58. [66]

    Principled reinforcement learning with human feedback from pairwise or k-wise comparisons

    Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp.\ 43037--43067. PMLR, 2023

    2023

  59. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.