About Me

Hi! I am Zhishuai Liu, a forth-year PhD candidate in Biostatistics at Duke University, advised by Prof. Pan Xu. My research focuses on Reinforcement Learning theory and applications, with a particular focus on Robust Reinforcement Learning, Reinforcement Learning via Supervised Learning and Reinforcement Learning for Healthcare. My aim is to build theoretically grounded Reinforcement Learning algorithms that scale efficiently. I am currently seeking an RL-related research internship. It is exciting to see how deep RL algorithms—built upon our theoretical framework—are driving progress in real-world applications such as LLM RL fine-tuning, robotics, and beyond.

🔥 News

• 2025.12: I am presenting my poster at NeurIPS 2025 on Thu, Dec 4, at Exhibit Hall C,D,E #3317. Welcome to stop by if you are interested in robust RL!
• 2025.09: Our paper Linear Mixture Distributionally Robust Markov Decision Processes is accepted to NeurIPS 2025!
• 2025.08: Attended Princeton 2025 Machine Learning Summer School!
• 2025.08: Our paper An Interactive Framework for Generating Clinical Data with Human Feedback is accepted to IEEE BHI 2025!
• 2025.05: Our paper Censored C-Learning for Dynamic Treatment Regime in Colorectal Cancer Study is accepted to Annals of Applied Statistics!
• 2025.05: Our paper Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction is accepted to ICML 2025!
• 2025.05: Our paper Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization is accepted to ICML 2025!
• 2024.08: Our paper Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning is accepted to NeurIPS 2024!
• 2024.01: Our paper Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation is accepted to AISTATS 2024!

Publications

Conference Publications

• Linear Mixture Distributionally Robust Markov Decision Processes

  Zhishuai Liu, Pan Xu

  The Thirty-Ninth Annual Conference on Neural Information Processing Systems.

• An Interactive Framework for Generating Clinical Data with Human Feedback

  Yu Yang^*, Jiafeng Song^*, Zhishuai Liu, Henry P Foote, Rishikesan Kamaleswaran, Pan Xu

  IEEE-EMBS International Conference on Biomedical and Health Informatics 2025.

• Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction

  Yiting He^*, Zhishuai Liu^*, Weixin Wang, Pan Xu

  The 42nd International Conference on Machine Learning.

• Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization

  Cheng Tang^*, Zhishuai Liu^*, Pan Xu

  The 42nd International Conference on Machine Learning.

• Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning

  Zhishuai Liu, Pan Xu

  The 38th Annual Conference on Neural Information Processing Systems.

• Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation.

  Zhishuai Liu, Pan Xu

  The 28th International Conference on Artificial Intelligence and Statistics.

Journal Publications

• Censored C-Learning for Dynamic Treatment Regime in Colorectal Cancer Study

  Zishu Zhan, Zhishuai Liu, Cunjie Lin, Danhui Yi, Jian Liu, Yufei Yang

  The Annals of Applied Statistics, 19(2), pp.1426-1447.

• Penalized Regression Analysis with Individual-Specific Patterns of Missing Covariates

  Zhishuai Liu, Zishu Zhan and Cunjie Lin

  Communications in Statistics-Simulation and Computation 53 (7), 3126-3142.

• Deep Spatial Q-Learning for Infectious Disease Control

  Zhishuai Liu, Jesse Clifton, Eric B Laber, John Drake, Ethan X Fang

  Journal of Agricultural, Biological and Environmental Statistics.

• Estimation in Optimal Treatment Regimes Based on Mean Residual Lifetimes with Right-Censored Data

  Zhishuai Liu, Zishu Zhan, Cunjie Lin

  Biometrical Journal 65 (8).

• Wavelet Scattering Transform for ECG Beat Classification

  Zhishuai Liu, Guihua Yao, Qing Zhang, Junpu Zhang and Xueying Zeng

  Computational and Mathematical Methods in Medicine, 2020(7), 1-11.

Workshop Publications

• How to Provably Improve Return Conditioned Supervised Learning?

  Zhishuai Liu, Yu Yang, Ruhan Wang, Pan Xu, Dongruo Zhou

  NeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists.

• Near-Optimal Reinforcement Learning for Linear Distributionally Robust Markov Decision Processes

  Zhishuai Liu^*, Weixin Wang^*, Pan Xu

  NeurIPS 2025 Workshop: Reliable ML from Unreliable Data.

• Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning

  Ruhan Wang^*, Yu Yang^*, Zhishuai Liu, Dongruo Zhou, Pan Xu

  NeurIPS 2025 Workshop: Reliable ML from Unreliable Data.

Preprint

• Policy Regularized Distributionally Robust Markov Decision Processes with Linear Function Approximation

  Jingwen Gu^*, Yiting He^*, Zhishuai Liu^*, Pan Xu

  arXiv preprint arXiv:2510.14246.

• Convergence of Sign-Based Random Reshuffling Algorithms for Nonconvex Optimization

  Zhen Qin^*, Zhishuai Liu^* Pan Xu

  arXiv preprint arXiv:2310.15976

Talks

• 2025.12: Duke RL Reading Group: On Return Conditioned Supervised Learning for Reinforcement Learning
• 2025.12: ML×OR Workshop: Mathematical Foundations and Operational Integration of Machine Learning for Uncertainty-Aware Decision-Making, Poster
• 2025.11: The 39th Annual Conference on Neural Information Processing Systems, Poster.
• 2025.08: 2025 Princeton Machine Learning Summer School, Poster.
• 2025.07: The Forty-Second International Conference on Machine Learning, Poster.
• 2024.12: The 38th Annual Conference on Neural Information Processing Systems, Poster.
• 2024.11: Conference in Honor of Dr. Michael Kosorok, Poster.
• 2024.02: Duke OBGE PhD Recruitment Visit Poster Session, Poster.

Student Mentoring

• Cheng Tang: Previous undergraduate student at Tsinghua University, now phd candidate at University of Illinois Urbana-Champaign. Our work received the Dean’s Award—the highest academic honor awarded by Zhili College at Tsinghua University.
• Yiting He: Undergraduate student at University of Science and Technology of China (USTC). Our work received the Outstanding Graduation Thesis awarded by USTC.
• Jingwen Gu: Undergraduate student at Cornell University.

Teaching Experience

• BIOSTAT 718: Analysis of Correlated and Longitudinal Data, 2024 Spring, Duke University.
• BIOSTAT 825: Foundation of Reinforcement Learning, 2023 Fall, Duke University.
• BIOSTAT 905: Linear Models and Inference, 2023 Spring, Duke University.
• Linear Regression: 2021 Fall, Renmin University of China.

Honors and Awards

• 2024 NeurIPS Top Reviewer
• 2024 NeurIPS Travel Award
• 2021 National Scholarship
• 2019 Provincial Outstanding Graduation Thesis
• 2018 Mathematical Contest In Modeling (MCM), Meritorious Winner

Service

• Conference reviewer: NeurIPS (2023-2025), AISTATS (2024-2025), ICML (2024-2025), AAAI (2024-2025), ICLR 2024, UAI 2024
• Co-organizer of the Duke RL Reading Group (lead by Prof. Ron Parr)