A short intoduction to stochastic optimal control theory and path integral control theory

Course information

• A one day course as part of Control Theory and Reinforcement Learning: Connections and Challenges - Spring School
• Lecturer: Bert Kappen

Course material:

• chapter 1 from the book Dynamic programming and optimal control by Dimitri Bertsekas. Copies 1a Copies 1b Here are his slides for Bertsekas' course.
• my book chapter in Barber, D., Cemgil, A. T., & Chiappa, S. (Eds.). (2011). Bayesian time series models. Cambridge University Press.
• These are the slides that are used for the course.

  Schedule:

  	Date	Topic	Chapter	Exercises
  1	March 17 9:00-10:30	Discrete time control dynamic programming Bellman equation Continuous time control Hamilton-Jacobi-Bellman Equation Pontryagin Minimum Principle	Bertsekas 2-5, 13-14, 18, 21-32 (2nd ed.) Bertsekas 2-5, 10-12, 16-27, 30-32 (1nd ed.) book chapter 1.2,1.3	Ex: Verify that J0(1)=2.7 and J0(2)=2.818 in Bertsekas Example 3.2 on pg. 23 in Copies 1b extra exercise 1, 2a,b Construct the Bayesian solution of the two armed bandit problem using dynamic programming
  2	March 17 14:00-15:30	Stochastic differential equations Kolmogorov backward equation and Fokker Planck equation Stochastic optimal control LQ examples, Portfolio management Path integral control theory	book chapter 1.4 and 1.6	extra exercise 2c, 3 extra exercise 4
  4	March 20	Path integral control KL control theory and relation to path integral control Importance sampling Miscelaneous topics Comparison PI control and RL Dual control Multi agent systems Mean field approximation for control: n joint arm, multi agents	book chapter 1.5 and 1.7	Directory with matlab code for n joint problem See below.