Short course on control theory and dynamic programming - Madrid, February 2014

The course provides an introduction to stochastic optimal control theory.

Course information

3 hours at Universidad Autonoma Madrid
For: Ma students and PhD students
Lecturer: Bert Kappen

Course material:

chapter 1 from the book Dynamic programming and optimal control by Dimitri Bertsekas. Copies 1a Copies 1b Here are his slides for Bertsekas' course.
my book chapter in Barber, D., Cemgil, A. T., & Chiappa, S. (Eds.). (2011). Bayesian time series models. Cambridge University Press.
These are the slides that are used for the course. Not all slides are used:
- deterministic control. Bellman equation (9-34)
- PMP principle (35-43,50)
- stochastic processes (51-52)
- Kolmogorov and Fokker-Planckk equations (53-56)
- stochastic optimal control (61-72)
- example portfolio selection (55-59)
- path integral control, delayed choice (83-93)
- path integral expression (98-103)
- sampling methods for computing optimal control (104-105, 111-112)
- acrobot (121-127)
- variational method (132-138) for exercise

Schedule:

Date Topic Chapter Exercises

1 Feb 25
17:00-18:00 Discrete time control
dynamic programming
Bellman equation
Bertsekas 2-5, 13-14, 18, 21-32 (2nd ed.)
Bertsekas 2-5, 10-12, 16-27, 30-32 (1nd ed.)
Kappen ICML tutorial 1.2
slides up to 34 Ex: Verify that J0(1)=2.7 and J0(2)=2.818 in
Bertsekas Example 3.2 on pg. 23 in Copies 1b

2 Feb 27
16:00-17:00 Continuous time control
Hamilton-Jacobi-Bellman Equation
Pontryagin Minimum Principle
Stochastic differential equations
Stochastic optimal control
LQ examples, Portfolio management
Kappen ICML tutorial 1.3,1.4
slides up to 59 extra exercise 2a,b

3 Feb 27
17:00-18:00 Path integral control theory
Kappen ICML tutorial 1.5, 1.6, 1.7
slides up to 93 extra exercise 2c, 3
extra exercise 4,5

	Date	Topic	Chapter	Exercises
1	Feb 25 17:00-18:00	Discrete time control dynamic programming Bellman equation	Bertsekas 2-5, 13-14, 18, 21-32 (2nd ed.) Bertsekas 2-5, 10-12, 16-27, 30-32 (1nd ed.) Kappen ICML tutorial 1.2 slides up to 34	Ex: Verify that J0(1)=2.7 and J0(2)=2.818 in Bertsekas Example 3.2 on pg. 23 in Copies 1b
2	Feb 27 16:00-17:00	Continuous time control Hamilton-Jacobi-Bellman Equation Pontryagin Minimum Principle Stochastic differential equations Stochastic optimal control LQ examples, Portfolio management	Kappen ICML tutorial 1.3,1.4 slides up to 59	extra exercise 2a,b
3	Feb 27 17:00-18:00	Path integral control theory	Kappen ICML tutorial 1.5, 1.6, 1.7 slides up to 93	extra exercise 2c, 3 extra exercise 4,5