Lorenzo Croissant

selected publications

1. 

   Near-continuous time Reinforcement Learning for continuous state-action spaces

   Lorenzo Croissant, Marc Abeille, and Bruno Bouchard

   In Proceedings of the 35th International Conference on Algorithmic Learning Theory 25–28 feb 2024

   Abs Bib HTML PDF Blog Project

   We consider the reinforcement learning problem of controlling an unknown dynamical system to maximise the long-term average reward along a single trajectory. Most of the literature considers system interactions that occur in discrete time and discrete state-action spaces. Although this standpoint is suitable for games, it is often inadequate for systems in which interactions occur at a high frequency, if not in continuous time, or those whose state spaces are large if not inherently continuous. Perhaps the only exception is the linear quadratic framework for which results exist both in discrete and continuous time. However, its ability to handle continuous states comes with the drawback of a rigid dynamic and reward structure. This work aims to overcome these shortcomings by modelling interaction times with a Poisson clock of frequency \varepsilon^-1 which captures arbitrary time scales from discrete (\varepsilon=1) to continuous time (\varepsilon\downarrow0). In addition, we consider a generic reward function and model the state dynamics according to a jump process with an arbitrary transition kernel on \mathbbR^d. We show that the celebrated optimism protocol applies when the sub-tasks (learning and planning) can be performed effectively. We tackle learning by extending the eluder dimension framework and propose an approximate planning method based on a diffusive limit (\varepsilon\downarrow0) approximation of the jump process. Overall, our algorithm enjoys a regret of order \tilde\mathcalO(\sqrtT) or \tilde\mathcalO(\varepsilon^1/2 T+\sqrtT) with the approximate planning. As the frequency of interactions blows up, the approximation error \varepsilon^1/2 T vanishes, showing that \tilde\mathcalO(\sqrtT) is attainable in near-continuous time.

   @inproceedings{CAB23,
     author = {Croissant, Lorenzo and Abeille, Marc and Bouchard, Bruno},
     title = {Near-continuous time {Reinforcement} {Learning} for continuous state-action spaces},
     pages = {444-498},
     booktitle = {Proceedings of the {35th} International Conference on Algorithmic Learning Theory},
     year = {2024},
     series = {Proceedings of Machine Learning Research},
     month = {25--28 Feb},
     publisher = {PMLR},
   }

2. 

   Diffusive limit approximation of pure jump optimal ergodic control problems

   Marc Abeille, Bruno Bouchard, and Lorenzo Croissant

   25–28 feb 2022

   Abs arXiv Bib HTML PDF Blog Project

   Motivated by the design of fast reinforcement learning algorithms, we study the diffusive limit of a class of pure jump ergodic stochastic control problems. We show that, whenever the intensity of jumps is large enough, the approximation error is governed by the Hölder continuity of the Hessian matrix of the solution to the limit ergodic partial differential equation. This extends to this context the results of [1] obtained for finite horizon problems. We also explain how to construct a first order error correction term under appropriate smoothness assumptions. Finally, we quantify the error induced by the use of the Markov control policy constructed from the numerical finite difference scheme associated to the limit diffusive problem, this seems to be new in the literature and of its own interest. This approach permits to reduce very significantly the numerical resolution cost.

   @misc{ABC22,
     doi = {10.48550/ARXIV.2209.15284},
     author = {Abeille, Marc and Bouchard, Bruno and Croissant, Lorenzo},
     keywords = {Optimization and Control (math.OC), FOS: Mathematics},
     title = {Diffusive limit approximation of pure jump optimal ergodic control problems},
     publisher = {arXiv},
     year = {2022},
     copyright = {arXiv.org perpetual, non-exclusive license},
   }