Offline reinforcement learning (RL) focuses on learning policies using static datasets without further exploration. With the introduction of distributional reinforcement learning into offline RL, current methods excel at quantifying the risk and ensuring the security of learned policies. However, these algorithms cannot effectively balance the distribution shift and robustness, and even a minor perturbation in observations can significantly impair policy performance. In this paper, we propose the algorithm of Offline Robustness of Distributional actor-critic Ensemble Reinforcement Learning (ORDER) to improve the robustness of policies. In ORDER, we introduce two approaches to enhance the robustness: 1) introduce the smoothing technique to policies and distribution functions for states near the dataset; 2) strengthen the quantile network. In addition to improving the robustness, we also theoretically prove that ORDER converges to a conservative lower bound, which can alleviate the distribution shift. In our experiments, we validate the effectiveness of ORDER in the D4RL benchmark through comparative experiments and ablation studies.
Lange, S., Gabel, T. and Riedmiller, M. (2012) Batch Reinforcement Learning. In: Wiering, M. and van Otterlo, M., Eds., Reinforcement Learning , Springer, 45-73. https://doi.org/10.1007/978-3-642-27645-3_2
Wang, R., Wu, Y., Salakhutdinov, R. and Kakade, S. (2021) Instabilities of Offline RL with Pretrained Neural Representation. Proceedings of the 38 th International Conference on Machine Learning , Virtual, 18-24 July 2021, 10948-10960.
Nguyen-Tang, T., Yin, M., Gupta, S., Venkatesh, S. and Arora, R. (2023) On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation. Proceedings of the AAAI Conference on Artificial Intelligence , 37, 9310-9318. https://doi.org/10.1609/aaai.v37i8.26116
Diehl, C., Sievernich, T.S., Kruger, M., Hoffmann, F. and Bertram, T. (2023) Uncertainty-Aware Model-Based Offline Reinforcement Learning for Automated Driving. IEEE Robotics and Automation Letters , 8, 1167-1174. https://doi.org/10.1109/lra.2023.3236579
Zhang, Z., Mei, H. and Xu, Y. (2023) Continuous-Time Decision Transformer for Healthcare Applications. Proceedings of the 26 th International Conference on Artificial Intelligence and Statistics , Valencia, 25-27 April 2023, 6245-6262.
Singh, B., Kumar, R. and Singh, V.P. (2022) Reinforcement Learning in Robotic Applications: A Comprehensive Survey. Artificial Intelligence Review , 55, 945-990. https://doi.org/10.1007/s10462-021-09997-9
Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning , Stockholm, 10-15 July 2018, 1861-1870.
Ashvin, N., Murtaza, D., Abhishek, G. and Sergey, L. (2020) AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv: 2006.09359. https://doi.org/10.48550/arXiv.2006.09359
Fujimoto, S., Meger, D. and Precup, D. (2019) Off-Policy Deep Reinforcement Learning without Exploration. Proceedings of the 36 th International Conference on Machine Learning , Long Beach, 9-15 June 2019, 2052-2062.
Fujimoto, S. and Gu, S.S. (2021) A Minimalist Approach to Offline Reinforcement Learning. Advances in Neural Information Processing Systems , 34, 20132-20145.
Lyu, J., Ma, X., Li, X. and Lu, Z. (2022) Mildly Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems , 35, 1711-1724.
Dabney, W., Ostrovski, G., Silver, D. and Munos, R. (2018) Implicit Quantile Networks for Distributional Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning , Stockholm, 10-15 July 2018, 1096-1105.
Dabney, W., Rowland, M., Bellemare, M. and Munos, R. (2018) Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence , 32, 2892-2901. https://doi.org/10.1609/aaai.v32i1.11791
Ma, X., Xia, L., Zhou, Z., Yang, J. and Zhao, Q. (2020) DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning. arXiv: 2004.14547. https://doi.org/10.48550/arXiv.2004.14547
Ma, Y., Jayaraman, D. and Bastani, O. (2021) Conservative Offline Distributional Reinforcement Learning. Advances in Neural Information Processing Systems , 34, 19235-19247.
Bai, C., Xiao, T., Zhu, Z., Wang, L., Zhou, F., Garg, A., et al . (2022) Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning. IEEE Transactions on Neural Networks and Learning Systems , 35, 8954-8968. https://doi.org/10.1109/TNNLS.2022.3217189
Kumar, A., Zhou, A., Tucker, G. and Levine, S. (2020) Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems , 33, 1179-1191.
Fu, J., Kumar, A., Nachum, O., Tucker, G. and Levine, S. (2020) D4RL: Datasets for Deep Data-Driven Reinforcement Learning. arXiv: 2004.07219. https://doi.org/10.48550/arXiv.2004.07219
Bellemare, M.G., Dabney, W. and Munos, R. (2017) A Distributional Perspective on Reinforcement Learning. Proceedings of the 34 th International Conference on Machine Learning , Sydney, 6-11 August 2017, 449-458.
Müller, A. (1997) Integral Probability Metrics and Their Generating Classes of Functions. Advances in Applied Probability , 29, 429-443. https://doi.org/10.2307/1428011
Shen, Q., Li, Y., Jiang, H., Wang, Z. and Zhao, T. (2020) Deep Reinforcement Learning with Robust and Smooth Policy. Proceedings of the 37 th International Conference on Machine Learning , Virtual, 13-18 July 2020, 8707-8718.
Koenker, R. and Hallock, K.F. (2001) Quantile Regression. Journal of Economic Perspectives , 15, 143-156. https://doi.org/10.1257/jep.15.4.143
Huber, P.J. (1992) Robust Estimation of a Location Parameter. In: Kotz, S. and Johnson, N.L., Eds., Breakthroughs in Statistics , Springer, 492-518. https://doi.org/10.1007/978-1-4612-4380-9_35
Wu, Y., Tucker, G. and Nachum, O. (2019) Behavior Regularized Offline Reinforcement Learning. arXiv: 1911.11361. https://doi.org/10.48550/arXiv.1911.11361
Kidambi, R., Rajeswaran, A., Netrapalli, P. and Joachims. T. (2020) MOReL: Model-Based Offline Reinforcement Learning. Advances in Neural Information Processing Systems , 33, 21810-21823.
Figueiredo Prudencio, R., Maximo, M.R.O.A. and Colombini, E.L. (2024) A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Transactions on Neural Networks and Learning Systems , 35, 10237-10257. https://doi.org/10.1109/tnnls.2023.3250269
Sutton, R.S. and Barto, A.G. (2018) Reinforcement Learning: An Introduction. MIT Press.
Singla, A., Rafferty, A.N., Radanovic, G. and Heffernan, N.T. (2021) Reinforcement Learning for Education: Opportunities and Challenges. arXiv:2107.08828. https://doi.org/10.48550/arXiv.2107.08828
Agarwal, R., Schuurmans, D. and Norouzi, M. (2020) An Optimistic Perspective on Offline Reinforcement Learning. Proceedings of the 37 th International Conference on Machine Learning , Vienna, 13-18 July 2020, 104-114.
Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P. and Wang, Z. (2022) Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. arXiv: 2202.11566. https://doi.org/10.48550/arXiv.2202.11566
Sohn, K., Lee, H. and Yan, X. (2015) Learning Structured Output Representation Using Deep Conditional Generative Models. Advances in Neural Information Processing Systems , 28, 3483-3491.
Kumar, A., Fu, J., Soh, M., Tucker, G. and Levine, S. (2019) Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Advances in Neural Information Processing Systems , 32, 11761-11771.
Panaganti, K., Xu, Z., Kalathil, D. and Ghavamzadeh, M. (2022) Robust Reinforcement Learning Using Offline Data. Advances in Neural Information Processing Systems , 35, 32211-32224.
Kostrikov, I., Nair, A. and Levine, S. (2021) Offline Reinforcement Learning with Implicit Q-Learning. arXiv: 2110.06169. https://doi.org/10.48550/arXiv.2110.06169
Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C. and Han, L. (2022) RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. Advances in Neural Information Processing Systems , 35, 23851-23866.