[美]德梅萃·P. 博塞克斯（Dimitri P. Bertseka），美国MIT终身教授，美国国家工程院院士，清华大学复杂与网络化系统研究中心客座教授。电气工程与计算机科学领域国际知名作者，著有《非线性规划》《网络优化》《动态规划》《凸优化》《强化学习与最优控制》等十几本畅销教材和专著。
目錄：
1. AlphaZero， Off-Line Training， and On-Line Play
1.1. Off-Line Training and Policy Iteration P. 3
1.2. On-Line Play and Approximation in Value Space -
Truncated Rollout p. 6
1.3. The Lessons of AlphaZero p. 8
1.4. A New Conceptual Framework for Reinforcement Learning p. 11
1.5. Notes and Sources p. 14
2. Deterministic and Stochastic Dynamic Programming
2.1. Optimal Control Over an Infinite Horizon p. 20
2.2. Approximation in Value Space p. 25
2.3. Notes and Sources p. 30
3. An Abstract View of Reinforcement Learning
3.1. Bellman Operators p. 32
3.2. Approximation in Value Space and Newton‘s Method p. 39
3.3. Region of Stability p. 46
3.4. Policy Iteration， Rollout， and Newton’s Method p. 50
3.5. How Sensitive is On-Line Play to the Off-Line
Training Process p. 58
3.6. Why Not Just Train a Policy Network and Use it Without
On-Line Play p. 60
3.7. Multiagent Problems and Multiagent Rollout p. 61
3.8. On-Line Simplified Policy Iteration p. 66
3.9. Exceptional Cases p. 72
3.10. Notes and Sources p. 79
4. The Linear Quadratic Case - Illustrations
4.1. Optimal Solution p. 82
4.2. Cost Functions of Stable Linear Policies p. 83
4.3. Value Iteration p. 86
vii
viii Contents
4.4. One-Step and Multistep Lookahead - Newton Step
Interpretations p. 86
4.5. Sensitivity Issues p. 91
4.6. Rollout and Policy Iteration p. 94
4.7. Truncated Rollout - Length of Lookahead Issues . . p. 97
4.8. Exceptional Behavior in Linear Quadratic Problems . p. 99
4.9. Notes and Sources p. 100
5. Adaptive and Model Predictive Control
5.1. Systems with Unknown Parameters - Robust and
PID Control p. 102
5.2. Approximation in Value Space， Rollout， and Adaptive
Control p. 105
5.3. Approximation in Value Space， Rollout， and Model
Predictive Control p. 109
5.4. Terminal Cost Approximation - Stability Issues . . . p. 112
5.5. Notes and Sources p. 118
6. Finite Horizon Deterministic Problems - Discrete
Optimization
6.1. Deterministic Discrete Spaces Finite Horizon Problems. p. 120
6.2. General Discrete Optimization Problems p. 125
6.3. Approximation in Value Space p. 128
6.4. Rollout Algorithms for Discrete Optimization . . . p. 132
6.5. Rollout and Approximation in Value Space with Multistep
Lookahead p. 149
6.5.1. Simplified Multistep Rollout - Double Rollout . . p. 150
6.5.2. Incremental Rollout for Multistep Approximation in
Value Space p. 153
6.6. Constrained Forms of Rollout Algorithms p. 159
6.7. Adaptive Control by Rollout with a POMDP Formulation p. 173
6.8. Rollout for Minimax Control p. 182
6.9. Small Stage Costs and Long Horizon - Continuous-Time
Rollout p. 190
6.10. Epilogue p. 197
Appendix A: Newton‘s Method and Error Bounds
A.1. Newton’s Method for Differentiable Fixed
Point Problems p. 202
A.2. Newtons Method Without Differentiability of the
Hellman Operator p. 207
Contents ix
A.3. Local and Global Error Bounds for Approximation in
Value Space p. 210
A.4. Local and Global Error Bounds for Approximate
Policy Iteration p. 212
References p. 217
內容試閱：
With four parameters I can fit an elephant， and with five I can make him wiggle his trunk4 John von Neumann The purpose of this monograph is to propose and develop a new concep-tual framework for approximate Dynamic Programming (DP) and Rein-forcement Learning (RL). This framework centers around two algorithms， which are designed largely independently of each other and operate in syn-ergy through the powerful mechanism of Newtons method. We call these the off-line training and the on-line play algorithms; the names are bor-rowed from some of the major successes of RL involving games. Primary examples are the recent (2017) AlphaZero program (which plays chess)， and the similarly structured and earlier (1990s) TD-Gammon program (which plays backgammon). In these game contexts， the off-line training algorithm is the method used to teach the program how to evaluate positions and to generate good moves at any given position， while the on-line play algo-rithm is the method used to play in real time against human or computer opponents. f From the meeting of Freeman Dyson and Enrico Fermi (p. 273 of the Segre and Hoerlin biography of Fermi， The Pope of Physics， Picador， 2017): ”When Dyson met with him in 1953， Fermi welcomed him politely， but he quickly put aside the graphs he was being shown indicating agreement between theory and experiment. His verdict， as Dyson remembered， was ”There are two ways of doing calculations in theoretical physics. One way， and this is the way I prefer， is to have a clear physical picture of the process you are calculating. The other way is to have a precise and self-consistent mathematical formalism. You have neither.” When a stunned Dyson tried to counter by emphasizing the agreement between experiment and the calculations， Fermi asked him how many free parameters he had used to obtain the fit. Smiling after being told ”Four，” Fermi remarked， ”I remember my old friend Johnny von Neumann used to say， with four parameters I can fit an elephant， and with five I can make him wiggle his trunk.\ See also the paper by Mayer， Khairy， and Howard [MKH10]， which provides a verification of the von Neumann quotation.
購物車/收銀台(0) | 在線留言板
| 付款方式
| 運費計算
| 聯絡我們
| 幫助中心 |
加入書簽
HOME
新書上架
