RL Course by David Silver - Lecture 7: Policy Gradient Methods

What people said

Recommends Neil Slater · 2016-07 · Data Science Stack Exchange · on “Difference between AlphaGo's policy network and value network”

I was just returning to this question having learnt about Actor-Critic, and how it offsets costs/benefits between pure Value based vs Policy based methods. I think this answer comes closest to explaining this detail, but it is covered very well in David Silver's lecture: [LINK] (lecture 7, it is also available on YouTube - [this video] ).

Recommends Imran · 2018-08 · Data Science Stack Exchange · on “How does action get selected in a Policy Gradient Method?”

You are on the right track. We no longer select an action that we think maximizes the score. Rather we predict what the best action to take is. This can be very effective in large or continuous state spaces where taking an argmax of the score over all possible actions would be prohibitively expensive. Our policy network computes the probability of taking an action, and during training we sample actions from the network. This continues until we receive a reward signal, which we multiply by the predicted log probabilities to propagate the error back through the network over past actions leading up to that reward. This will tend to adjust the past actions' predicted probabilities down for more

Recommends Kari · 2018-09 · Data Science Stack Exchange · on “RL Advantage function why A = Q-V instead of A=V-Q?”

In RL Course by David Silver - Lecture 7: Policy Gradient Methods [this video] , David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s) [LINK] Preliminary, from this post: [LINK] First recall that a policy $\pi$ is a mapping from each state, $s$ , action $a$ , to the probability $\pi(a \mid s)$ of taking action $a$ when in state $s$ . The state value function, $V^\pi(s)$ , is the expected return when starting in state $s$ and following $\pi$ thereafter. Similarly, the state-action value function, $Q^\pi(s, a)$ , is the expected return of when starting in state $s$ , taking action $a$ , and following policy $\pi$ thereafter. In my understanding, $V(s)

Recommends Andrei Poehlmann · 2018-09 · Data Science Stack Exchange · on “Policy Gradient Methods - ScoreFunction & Log(policy)”

You have to see this approach more general and not so much from a neural network point of view. The reason one uses the re-arranged version is really just because in the end the gradient of your objective function ends up being an expectation again. This is also pointed out by David Silver at min 44:06 [this video] . For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here [LINK] . The name of the video is "CS294-112 9/6/17" At some later point [LINK] , S. Levine also goes into more details regarding the math of the policy gradient and draws comparisons to the maximum likelihood approach with neural networks.

Recommends Kari · 2018-09 · Data Science Stack Exchange · on “Policy Gradient Methods - ScoreFunction & Log(policy)”

In Policy Gradient Methods, Lecture 7 (34:15), David [this video] describes a Score Function as being the Gradient of the Log of the policy [LINK] Question: If we have a Neural Network that holds parameters defining our policy, we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with: $\nabla_{\theta}\pi_{\theta}(s,a)$ and instead would wish to deal with its re-arranged version: $\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $ Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just