【强化学习】随机策略的策略梯度

Policy的目标函数

$J(\pi )={\mathbb{E}}_{\tau |\pi }[G_0]={\mathbb{E}}_{\tau |\pi }[\sum _{t=0}{\gamma }^t{r}_t]$

定理1

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau |\pi }[\sum _{t=0}^{\infty }{\gamma }^t{G}_t{\nabla }_{\theta }\ln \pi (a_t|s_t)]$

$G_t={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}$

证明:

$J(\theta )={\sum }_{\tau |\pi }{G}_0\pi (\tau ;\theta )$

${\nabla }_{\theta }J(\theta )={\sum }_{\tau |\pi }{G}_0{\nabla }_{\theta }\pi (\tau ;\theta )$

${\nabla }_{\theta }\pi (\tau ;\theta )=\pi (\tau ;\theta ){\nabla }_{\theta }\ln \pi (\tau ;\theta )$

$\pi (\tau ;\theta )=p_1(s_0){\Pi }_{i=0}^{\infty }\pi (a_i|s_i;\theta )T(s_i,a_i,s_{i+1})$

$\ln \pi (\tau ;\theta )={\sum }_{i=0}^{\infty }\pi (a_i|s_i;\theta )+{\sum }_{i=0}^{\infty }\ln T(s_i,a_i,s_{i+1})+\ln p_1(s_0)$

${\nabla }_{\theta }\ln \pi (\tau ;\theta )={\sum }_{i=0}^{\infty }{\nabla }_{\theta }\pi (a_i|s_i;\theta )$

${\nabla }_{\theta }J(\theta )={\sum }_{\tau }{G}_0\pi (\tau ;\theta ){\sum }_{i=0}^{\infty }{\nabla }_{\theta }\pi (a_i|s_i;\theta )={\mathbb{E}}_{\tau }[G_0{\sum }_{i=0}^{\infty }{\nabla }_{\theta }\pi (a_i|s_i;\theta )]$

定理2

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau |\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{G}_i{\nabla }_{\theta }\ln \pi (s_i)]$

证明:

$J(\theta )={\mathbb{E}}_{\tau |\pi }[G_0]={\mathbb{E}}_{s_0}{\mathbb{E}}_{\tau |s_0,\pi }[G_0]={\mathbb{E}}_{s_0}{V}^{\theta }(s_0)$

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_0}{\nabla }_{\theta }{V}^{\theta }(s_0)$

$V^{\theta }(s_0)={\sum }_a\pi (a|s_0;\theta )Q^{\theta }(s_0,a)$

${\nabla }_{\theta }{V}^{\theta }(s_0)={\sum }_a[{\nabla }_{\theta }\pi (a|s_0;\theta )Q^{\theta }(s_0,a)+{\nabla }_{\theta }{Q}^{\theta }(s_0,a)\pi (a|s_0;\theta )]$

∑ a 0 ∇ θ π ( a ∣ s 0 ; θ ) Q θ ( s 0 , a ) = ∑ a π ( a ∣ s 0 ; θ ) ∇ θ ln ⁡ π ( a ∣ s 0 ; θ ) E τ ∣ s 0 , a 0 = a , π [ G 0 ] = E a 0 ∣ s 0 , π { ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) E τ ∣ s 0 , a 0 , π [ G 0 ] } = E a 0 ∣ s 0 , π E τ ∣ s 0 , a 0 , π [ G 0 ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) ] = E τ ∣ s 0 , π { G 0 ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) } \begin{align} \sum_{a_0}\nabla_{\theta}\pi(a|s_0;\theta)Q^{\theta}(s_0,a) &=\sum_{a}\pi(a|s_0;\theta) \nabla_{\theta}\ln\pi(a|s_0;\theta) \mathbb{E}_{\tau|s_0,a_0=a,\pi}[G_0]\notag\\ &=\mathbb{E}_{a_0|s_0,\pi}\{\nabla_{\theta}\ln\pi(a_0|s_0;\theta) \mathbb{E}_{\tau|s_0,a_0,\pi}[G_0]\}\notag\\ &=\mathbb{E}_{a_0|s_0,\pi}\mathbb{E}_{\tau|s_0,a_0,\pi}[G_0\nabla_{\theta}\ln\pi(a_0|s_0;\theta)]\\ &=\mathbb{E}_{\tau|s_0,\pi}\{G_0\nabla_{\theta}\ln\pi(a_0|s_0;\theta)\} \end{align} a0​∑​∇θ​π(a∣s0​;θ)Qθ(s0​,a)​=a∑​π(a∣s0​;θ)∇θ​lnπ(a∣s0​;θ)Eτ∣s0​,a0​=a,π​[G0​]=Ea0​∣s0​,π​{∇θ​lnπ(a0​∣s0​;θ)Eτ∣s0​,a0​,π​[G0​]}=Ea0​∣s0​,π​Eτ∣s0​,a0​,π​[G0​∇θ​lnπ(a0​∣s0​;θ)]=Eτ∣s0​,π​{G0​∇θ​lnπ(a0​∣s0​;θ)}​​

${\sum }_a{\nabla }_{\theta }{Q}^{\theta }(s_0,a)\pi (a|s_0;\theta )=\gamma {\mathbb{E}}_{a_0|s_0,\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)$

其中 $Q^{\theta }(s_0,a)={\sum }_{s^{\prime }}T(s_0,a,s^{\prime })[r(s_0,a,s^{\prime })+\gamma V^{\theta }(s^{\prime })]$

${\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)={\sum }_{s^{\prime }}T(s_0,a_0,s^{\prime }){\nabla }_{\theta }{V}^{\theta }(s^{\prime })={\mathbb{E}}_{s_1|s_0,a_0}{\nabla }_{\theta }{V}^{\theta }(s_1)$

所以 ${\sum }_a{\nabla }_{\theta }{Q}^{\theta }(s_0,a)\pi (a|s_0;\theta )=\gamma {\mathbb{E}}_{a_0|s_0,\pi }{\mathbb{E}}_{s_1|s_0,a_0}{\nabla }_{\theta }{V}^{\theta }(s_1)=\gamma {\mathbb{E}}_{s_1|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_1)$

∇ θ V θ ( s 0 ) = E τ ∣ s 0 , π { G 0 ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) } + γ E s 1 ∣ s 0 , π ∇ θ V θ ( s 1 ) \nabla_{\theta}V^{\theta}(s_0)=\mathbb{E}_{\tau|s_0,\pi}\{G_0\nabla_{\theta}\ln\pi(a_0|s_0;\theta)\}+\gamma\mathbb{E}_{s_1|s_0,\pi}\nabla_{\theta}V^{\theta}(s_1) ∇θ​Vθ(s0​)=Eτ∣s0​,π​{G0​∇θ​lnπ(a0​∣s0​;θ)}+γEs1​∣s0​,π​∇θ​Vθ(s1​)

同理可得:

V θ ( s i ) = E τ i ∣ s i , π { G i ∇ θ ln ⁡ π ( a i ∣ s i ; θ ) } + γ E s i + 1 ∣ s i , π ∇ θ V θ ( s i + 1 ) ,   i = 1 , 2 , . . . V^{\theta}(s_i)=\mathbb{E}_{\tau_i|s_i,\pi}\{G_i\nabla_{\theta}\ln\pi(a_i|s_i;\theta)\}+\gamma \mathbb{E}_{s_{i+1}|s_{i},\pi}\nabla_{\theta}V^{\theta}(s_{i+1}), \ i=1,2,... Vθ(si​)=Eτi​∣si​,π​{Gi​∇θ​lnπ(ai​∣si​;θ)}+γEsi+1​∣si​,π​∇θ​Vθ(si+1​), i=1,2,...$

将 V θ ( s 1 ) = E τ 1 ∣ s 1 , π { G 1 ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ E s 2 ∣ s 1 , π ∇ θ V θ ( s 2 ) V^{\theta}(s_1)=\mathbb{E}_{\tau_1|s_1,\pi}\{G_1\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma \mathbb{E}_{s_{2}|s_{1},\pi}\nabla_{\theta}V^{\theta}(s_{2}) Vθ(s1​)=Eτ1​∣s1​,π​{G1​∇θ​lnπ(a1​∣s1​;θ)}+γEs2​∣s1​,π​∇θ​Vθ(s2​) 代入 $\gamma {\mathbb{E}}_{s_1|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_1)$, 得

γ E s 1 ∣ s 0 , π ∇ θ V θ ( s 1 ) = γ E s 1 ∣ s 0 , π { E τ 1 ∣ s 1 , π { G 1 ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ E s 2 ∣ s 1 , π ∇ θ V θ ( s 2 ) } = γ E τ 1 ∣ s 0 , π [ G 1 ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) ] + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) = E τ ∣ s 0 , π [ γ G 1 ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) ] + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) \begin{align} \gamma\mathbb{E}_{s_1|s_0,\pi}\nabla_{\theta}V^{\theta}(s_1)&=\gamma\mathbb{E}_{s_1|s_0,\pi}\{\mathbb{E}_{\tau_1|s_1,\pi}\{G_1\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma \mathbb{E}_{s_{2}|s_{1},\pi}\nabla_{\theta}V^{\theta}(s_{2})\}\notag\\ &=\gamma \mathbb{E}_{\tau_1|s_0, \pi}[G_1\nabla_{\theta}\ln \pi(a_1|s_1;\theta)]+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) \notag\\ &=\mathbb{E}_{\tau|s_0, \pi}[\gamma G_1\nabla_{\theta}\ln \pi(a_1|s_1;\theta)]+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) \end{align} γEs1​∣s0​,π​∇θ​Vθ(s1​)​=γEs1​∣s0​,π​{Eτ1​∣s1​,π​{G1​∇θ​lnπ(a1​∣s1​;θ)}+γEs2​∣s1​,π​∇θ​Vθ(s2​)}=γEτ1​∣s0​,π​[G1​∇θ​lnπ(a1​∣s1​;θ)]+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)=Eτ∣s0​,π​[γG1​∇θ​lnπ(a1​∣s1​;θ)]+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)​​

进而

∇ θ V θ ( s 0 ) = E τ ∣ s 0 , π { G 0 ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) + γ G 1 ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) \nabla_{\theta}V^{\theta}(s_0)=\mathbb{E}_{\tau|s_0,\pi}\{G_0\nabla_{\theta}\ln\pi(a_0|s_0;\theta)+\gamma G_1\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) ∇θ​Vθ(s0​)=Eτ∣s0​,π​{G0​∇θ​lnπ(a0​∣s0​;θ)+γG1​∇θ​lnπ(a1​∣s1​;θ)}+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)

再将 V θ ( s 2 ) = E τ 2 ∣ s 2 , π { G 2 ∇ θ ln ⁡ π ( a 2 ∣ s 2 ; θ ) } + γ E s 3 ∣ s 2 , π ∇ θ V θ ( s 3 ) V^{\theta}(s_2)=\mathbb{E}_{\tau_2|s_2,\pi}\{G_2\nabla_{\theta}\ln\pi(a_2|s_2;\theta)\}+\gamma \mathbb{E}_{s_{3}|s_{2},\pi}\nabla_{\theta}V^{\theta}(s_{3}) Vθ(s2​)=Eτ2​∣s2​,π​{G2​∇θ​lnπ(a2​∣s2​;θ)}+γEs3​∣s2​,π​∇θ​Vθ(s3​) 代入 ${\gamma }^2{\mathbb{E}}_{s_2|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_2)$ …

不断重复上述过程得到

${\nabla }_{\theta }{V}^{\theta }(s_0)={\mathbb{E}}_{\tau |s_0,\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{G}_i{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]$

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_0}{\mathbb{E}}_{\tau |s_0,\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{G}_i{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]={\mathbb{E}}_{\tau |\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{G}_i{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]$

定理3

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau |\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{Q}_i(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i)]$

以上同上一个证明, 不赘述.

∑ a 0 ∇ θ π ( a ∣ s 0 ; θ ) Q θ ( s 0 , a ) = ∑ a π ( a ∣ s 0 ; θ ) ∇ θ ln ⁡ π ( a ∣ s 0 ; θ ) Q θ ( s 0 , a ) = E a 0 ∣ s 0 , π { ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) Q θ ( s 0 , a 0 ) } = E τ ∣ s 0 , π { Q θ ( s 0 , a 0 ) ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) } \begin{align} \sum_{a_0}\nabla_{\theta}\pi(a|s_0;\theta)Q^{\theta}(s_0,a) &=\sum_{a}\pi(a|s_0;\theta) \nabla_{\theta}\ln\pi(a|s_0;\theta)Q^{\theta}(s_0,a)\notag\\ &=\mathbb{E}_{a_0|s_0,\pi}\{\nabla_{\theta}\ln\pi(a_0|s_0;\theta) Q^{\theta}(s_0,a_0)\}\notag\\ &=\mathbb{E}_{\tau|s_0,\pi}\{Q^{\theta}(s_0,a_0)\nabla_{\theta}\ln\pi(a_0|s_0;\theta)\} \end{align} a0​∑​∇θ​π(a∣s0​;θ)Qθ(s0​,a)​=a∑​π(a∣s0​;θ)∇θ​lnπ(a∣s0​;θ)Qθ(s0​,a)=Ea0​∣s0​,π​{∇θ​lnπ(a0​∣s0​;θ)Qθ(s0​,a0​)}=Eτ∣s0​,π​{Qθ(s0​,a0​)∇θ​lnπ(a0​∣s0​;θ)}​​

${\sum }_a{\nabla }_{\theta }{Q}^{\theta }(s_0,a)\pi (a|s_0;\theta )=\gamma {\mathbb{E}}_{a_0|s_0,\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)$

其中 $Q^{\theta }(s_0,a)={\sum }_{s^{\prime }}T(s_0,a,s^{\prime })[r(s_0,a,s^{\prime })+\gamma V^{\theta }(s^{\prime })]$

${\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)={\sum }_{s^{\prime }}T(s_0,a_0,s^{\prime }){\nabla }_{\theta }{V}^{\theta }(s^{\prime })={\mathbb{E}}_{s_1|s_0,a_0}{\nabla }_{\theta }{V}^{\theta }(s_1)$

所以 ${\sum }_a{\nabla }_{\theta }{Q}^{\theta }(s_0,a)\pi (a|s_0;\theta )=\gamma {\mathbb{E}}_{a_0|s_0,\pi }{\mathbb{E}}_{s_1|s_0,a_0}{\nabla }_{\theta }{V}^{\theta }(s_1)=\gamma {\mathbb{E}}_{s_1|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_1)$

∇ θ V θ ( s 0 ) = E τ ∣ s 0 , π { Q θ ( s 0 , a 0 ) ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) } + γ E s 1 ∣ s 0 , π ∇ θ V θ ( s 1 ) \nabla_{\theta}V^{\theta}(s_0)=\mathbb{E}_{\tau|s_0,\pi}\{Q^{\theta}(s_0,a_0)\nabla_{\theta}\ln\pi(a_0|s_0;\theta)\}+\gamma\mathbb{E}_{s_1|s_0,\pi}\nabla_{\theta}V^{\theta}(s_1) ∇θ​Vθ(s0​)=Eτ∣s0​,π​{Qθ(s0​,a0​)∇θ​lnπ(a0​∣s0​;θ)}+γEs1​∣s0​,π​∇θ​Vθ(s1​)

同理可得:

V θ ( s i ) = E τ i ∣ s i , π { Q θ ( s i , a i ) ∇ θ ln ⁡ π ( a i ∣ s i ; θ ) } + γ E s i + 1 ∣ s i , π ∇ θ V θ ( s i + 1 ) ,   i = 1 , 2 , . . . V^{\theta}(s_i)=\mathbb{E}_{\tau_i|s_i,\pi}\{Q^{\theta}(s_i,a_i)\nabla_{\theta}\ln\pi(a_i|s_i;\theta)\}+\gamma \mathbb{E}_{s_{i+1}|s_{i},\pi}\nabla_{\theta}V^{\theta}(s_{i+1}), \ i=1,2,... Vθ(si​)=Eτi​∣si​,π​{Qθ(si​,ai​)∇θ​lnπ(ai​∣si​;θ)}+γEsi+1​∣si​,π​∇θ​Vθ(si+1​), i=1,2,...$

将 V θ ( s 1 ) = E τ 1 ∣ s 1 , π { Q θ ( s 1 , a 1 ) ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ E s 2 ∣ s 1 , π ∇ θ V θ ( s 2 ) V^{\theta}(s_1)=\mathbb{E}_{\tau_1|s_1,\pi}\{Q^{\theta}(s_1,a_1)\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma \mathbb{E}_{s_{2}|s_{1},\pi}\nabla_{\theta}V^{\theta}(s_{2}) Vθ(s1​)=Eτ1​∣s1​,π​{Qθ(s1​,a1​)∇θ​lnπ(a1​∣s1​;θ)}+γEs2​∣s1​,π​∇θ​Vθ(s2​) 代入 $\gamma {\mathbb{E}}_{s_1|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_1)$, 得

γ E s 1 ∣ s 0 , π ∇ θ V θ ( s 1 ) = γ E s 1 ∣ s 0 , π { E τ 1 ∣ s 1 , π { Q θ ( s 1 , a 1 ) ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ E s 2 ∣ s 1 , π ∇ θ V θ ( s 2 ) } = γ E τ 1 ∣ s 0 , π [ Q θ ( s 1 , a 1 ) ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) ] + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) = E τ ∣ s 0 , π [ γ Q θ ( s 1 , a 1 ) ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) ] + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) \begin{align} \gamma\mathbb{E}_{s_1|s_0,\pi}\nabla_{\theta}V^{\theta}(s_1)&=\gamma\mathbb{E}_{s_1|s_0,\pi}\{\mathbb{E}_{\tau_1|s_1,\pi}\{Q^{\theta}(s_1,a_1)\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma \mathbb{E}_{s_{2}|s_{1},\pi}\nabla_{\theta}V^{\theta}(s_{2})\}\notag\\ &=\gamma \mathbb{E}_{\tau_1|s_0, \pi}[Q^{\theta}(s_1,a_1)\nabla_{\theta}\ln \pi(a_1|s_1;\theta)]+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) \notag\\ &=\mathbb{E}_{\tau|s_0, \pi}[\gamma Q^{\theta}(s_1,a_1)\nabla_{\theta}\ln \pi(a_1|s_1;\theta)]+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) \end{align} γEs1​∣s0​,π​∇θ​Vθ(s1​)​=γEs1​∣s0​,π​{Eτ1​∣s1​,π​{Qθ(s1​,a1​)∇θ​lnπ(a1​∣s1​;θ)}+γEs2​∣s1​,π​∇θ​Vθ(s2​)}=γEτ1​∣s0​,π​[Qθ(s1​,a1​)∇θ​lnπ(a1​∣s1​;θ)]+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)=Eτ∣s0​,π​[γQθ(s1​,a1​)∇θ​lnπ(a1​∣s1​;θ)]+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)​​

进而

∇ θ V θ ( s 0 ) = E τ ∣ s 0 , π { Q θ ( s 0 , a 0 ) ∇ θ ln ⁡ π ( a 0 ∣ s 0 ; θ ) + γ Q θ ( s 1 , a 1 ) ∇ θ ln ⁡ π ( a 1 ∣ s 1 ; θ ) } + γ 2 E s 2 ∣ s 0 , π ∇ θ V θ ( s 2 ) \nabla_{\theta}V^{\theta}(s_0)=\mathbb{E}_{\tau|s_0,\pi}\{Q^{\theta}(s_0,a_0)\nabla_{\theta}\ln\pi(a_0|s_0;\theta)+\gamma Q^{\theta}(s_1,a_1)\nabla_{\theta}\ln\pi(a_1|s_1;\theta)\}+\gamma^2 \mathbb{E}_{s_2|s_0,\pi}\nabla_{\theta}V^{\theta}(s_{2}) ∇θ​Vθ(s0​)=Eτ∣s0​,π​{Qθ(s0​,a0​)∇θ​lnπ(a0​∣s0​;θ)+γQθ(s1​,a1​)∇θ​lnπ(a1​∣s1​;θ)}+γ2Es2​∣s0​,π​∇θ​Vθ(s2​)

再将 V θ ( s 2 ) = E τ 2 ∣ s 2 , π { Q θ ( s 2 , a 2 ) ∇ θ ln ⁡ π ( a 2 ∣ s 2 ; θ ) } + γ E s 3 ∣ s 2 , π ∇ θ V θ ( s 3 ) V^{\theta}(s_2)=\mathbb{E}_{\tau_2|s_2,\pi}\{Q^{\theta}(s_2,a_2)\nabla_{\theta}\ln\pi(a_2|s_2;\theta)\}+\gamma \mathbb{E}_{s_{3}|s_{2},\pi}\nabla_{\theta}V^{\theta}(s_{3}) Vθ(s2​)=Eτ2​∣s2​,π​{Qθ(s2​,a2​)∇θ​lnπ(a2​∣s2​;θ)}+γEs3​∣s2​,π​∇θ​Vθ(s3​) 代入 ${\gamma }^2{\mathbb{E}}_{s_2|s_0,\pi }{\nabla }_{\theta }{V}^{\theta }(s_2)$ …

不断重复上述过程得到

${\nabla }_{\theta }{V}^{\theta }(s_0)={\mathbb{E}}_{\tau |s_0,\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{Q}^{\theta }(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]$

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_0}{\mathbb{E}}_{\tau |s_0,\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{Q}^{\theta }(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]={\mathbb{E}}_{\tau |\pi }[{\sum }_{i=0}^{\infty }{\gamma }^i{Q}^{\theta }(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )]$

推论.

${\nabla }_{\theta }J(\theta )={\sum }_s{\sum }_a{\sum }_{i=0}^{\infty }{\gamma }^i\mathrm{Pr}[s_t=s,a_t=a|\pi ]Q^{\theta }(s,a){\nabla }_{\theta }\ln \pi (a|s)$

证明: ${\nabla }_{\theta }J(\theta )={\sum }_{i=0}^{\infty }{\gamma }^i{\mathbb{E}}_{\tau |\pi }[Q_i(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i)]$

${\mathbb{E}}_{\tau |\pi }[Q_i(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i)]={\sum }_s{\sum }_a\mathrm{Pr}[s_i=s,a_i=a|\pi ][Q_i(s_i,a_i){\nabla }_{\theta }\ln \pi (a_i|s_i)]{|}_{s_i=s,a_i=a}$

${\nabla }_{\theta }J(\theta )={\sum }_{i=0}^{\infty }{\sum }_s{\sum }_a{\gamma }^i\mathrm{Pr}[s_i=s,a_i=a|\pi ]Q(s,a)\ln \pi (a|s)={\sum }_s{\sum }_a{\sum }_{i=0}^{\infty }{\gamma }^i\mathrm{Pr}[s_t=s,a_t=a|\pi ]Q^{\theta }(s,a){\nabla }_{\theta }\ln \pi (a|s)$

定理4

${\mathbb{E}}_{(s_i,a_i)|\pi }{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )Q^{\theta }(s_i,a_i)={\sum }_s{\sum }_a\mathrm{Pr}[s_i=s,a_i=a|\pi ]{\nabla }_{\theta }\ln \pi (a|s;\theta )Q^{\theta }(s,a)$

证明:

引理. $Q^{\theta }(s,a)={\sum }_{s^{\prime }}T(s,a,s^{\prime })[r(s,a,s^{\prime })+\gamma V^{\theta }(s^{\prime })]$

${\nabla }_{\theta }{Q}^{\theta }(s,a)={\sum }_{s^{\prime }}T(s,a,s^{\prime }){\nabla }_{\theta }{V}^{\theta }(s^{\prime })$

${\nabla }_{\theta }V(s^{\prime })={\nabla }_{\theta }{\sum }_{a^{\prime }}\pi (a^{\prime }|s^{\prime };\theta )Q^{\theta }(s^{\prime },a^{\prime })={\sum }_{a^{\prime }}[{\nabla }_{\theta }\pi (a^{\prime }|s^{\prime };\theta )Q^{\theta }(s^{\prime },a^{\prime })+\pi (a^{\prime }|s^{\prime };\theta ){\nabla }_{\theta }{Q}^{\theta }(s^{\prime },a^{\prime })]$

所以

$$\begin{array}{rl}{\nabla }_{\theta }{Q}^{\theta }(s,a)& =\gamma \sum _{s^{\prime }}\sum _{a^{\prime }}T(s,a,s^{\prime })[{\nabla }_{\theta }\pi (a^{\prime }|s^{\prime };\theta )Q^{\theta }(s^{\prime },a^{\prime })+\pi (s^{\prime }|a^{\prime };\theta ){\nabla }_{\theta }{Q}^{\theta }(s^{\prime },a^{\prime })]\\ & =\gamma \sum _{s^{\prime }}\sum _{a^{\prime }}T(s,a,s^{\prime })\pi (s^{\prime }|a^{\prime };\theta )[{\nabla }_{\theta }\ln \pi (a^{\prime }|s^{\prime };\theta )Q^{\theta }(s^{\prime },a^{\prime })+{\nabla }_{\theta }{Q}^{\theta }(s^{\prime },a^{\prime })]\end{array}$$

$$\begin{array}{rl}{\nabla }_{\theta }{Q}^{\theta }(s_i,a_i)& =\gamma \sum _{s_{i+1}}\sum _{a_{i+1}}T(s_i,a_i,s_{i+1})\pi (s_{i+1}|a_{i+1};\theta )[{\nabla }_{\theta }\ln \pi (a_{i+1}|s_{i+1};\theta )Q^{\theta }(s_{i+1},a_{i+1})+{\nabla }_{\theta }{Q}^{\theta }(s_{i+1},a_{i+1})]\\ & =\gamma {\mathbb{E}}_{(s_{i+1},a_{i+1})|(s_i,a_i),\pi }[{\nabla }_{\theta }\ln \pi (a_{i+1},s_{i+1};\theta )Q^{\theta }(s_{i+1},a_{i+1})+{\nabla }_{\theta }{Q}^{\theta }(s_{i+1},a_{i+1})]\end{array}$$

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_0}{V}^{\theta }(s_0)={\mathbb{E}}_{s_0}{\sum }_a\pi (a|s_0;\theta )Q^{\theta }(s_0,a)$

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{s_0}{\sum }_a{\nabla }_{\theta }\pi (a|s_0;\theta )Q^{\theta }(s_0,a)+{\mathbb{E}}_{s_0}{\sum }_a\pi (a|s_0;\theta ){\nabla }_{\theta }{Q}^{\theta }(s_0,a)$

${\mathbb{E}}_{s_0}{\sum }_a{\nabla }_{\theta }\pi (a|s_0;\theta )Q^{\theta }(s_0,a)={\mathbb{E}}_{s_0}{\sum }_a\pi (a|s_0;\theta ){\nabla }_{\theta }\ln \pi (a|s_0;\theta )Q^{\theta }(s_0,a)={\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }\ln \pi (a_0|s_0;\theta )Q^{\theta }(s_0,a_0)$

$$\begin{array}{rl}{\mathbb{E}}_{s_0}\sum _a\pi (a|s_0;\theta ){\nabla }_{\theta }{Q}^{\theta }(s_0,a)& ={\mathbb{E}}_{s_0}{\mathbb{E}}_{a_0|s_0,\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)\\ & ={\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)\end{array}$$

至此,

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }\ln \pi (a_0|s_0;\theta )Q^{\theta }(s_0,a_0)+{\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)$

由引理得:

${\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)=\gamma {\mathbb{E}}_{(s_1,a_1)|(s_0,a_0),\pi }[{\nabla }_{\theta }\ln \pi (a_1|s_1;\theta )Q^{\theta }(s_1,a_1)+{\nabla }_{\theta }{Q}^{\theta }(s_1,a_1)]$

${\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }{Q}^{\theta }(s_0,a_0)=\gamma {\mathbb{E}}_{(s_0,a_0)}{\mathbb{E}}_{(s_1,a_1)|(s_0,a_0),\pi }[{\nabla }_{\theta }\ln \pi (a_1|s_1;\theta )Q^{\theta }(s_1,a_1)+{\nabla }_{\theta }{Q}^{\theta }(s_1,a_1)]=\gamma {\mathbb{E}}_{(s_1,a_1)|\pi }[{\nabla }_{\theta }\ln \pi (a_1|s_1;\theta )Q^{\theta }(s_1,a_1)+{\nabla }_{\theta }{Q}^{\theta }(s_1,a_1)]$

至此,

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{(s_0,a_0)|\pi }{\nabla }_{\theta }\ln \pi (a_0|s_0;\theta )Q^{\theta }(s_0,a_0)+\gamma {\mathbb{E}}_{(s_1,a_1)|\pi }{\nabla }_{\theta }\ln \pi (a_1|s_1;\theta )Q^{\theta }(s_1,a_1)+\gamma {\mathbb{E}}_{(s_1,a_1)|\pi }{\nabla }_{\theta }{Q}^{\theta }(s_1,a_1)$

重复上述过程, 可得

${\nabla }_{\theta }J(\theta )={\gamma }^i\sum _{i=0}^{\infty }{\mathbb{E}}_{(s_i,a_i)|\pi }{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )Q^{\theta }(s_i,a_i)$

其中

${\mathbb{E}}_{(s_i,a_i)|\pi }{\nabla }_{\theta }\ln \pi (a_i|s_i;\theta )Q^{\theta }(s_i,a_i)={\sum }_s{\sum }_a\mathrm{Pr}[s_i=s,a_i=a|\pi ]{\nabla }_{\theta }\ln \pi (a|s;\theta )Q^{\theta }(s,a)$

进而有

${\nabla }_{\theta }J(\theta )={\sum }_s{\sum }_a{\sum }_{i=0}^{\infty }{\gamma }^i\mathrm{Pr}[s_i=s,a_i=a]{\nabla }_{\theta }\ln \pi (a|s;\theta )Q^{\theta }(s,a)$.

定理5

${\mathbb{E}}_{(s_t,a_t)|\pi }[{\gamma }^t{G}_t{\nabla }_{\theta }\ln \pi (a_t|s_t;\theta )]={\mathbb{E}}_{(s_t,a_t)|{\pi }^{\prime }}[\frac{1}{{\pi }^{\prime }(a_t|s_t)}{\gamma }^t{G}_t{\nabla }_{\theta }\pi (a_t|s_t;\theta )]$

其中 ${\pi }^{\prime }$ 为任意一个策略.

证明:

${\nabla }_{\theta }\ln (\pi (a_t|s_t;\theta ))={\nabla }_{\theta }\pi (a_t|s_t;\theta )/\pi (a_t|s_t;\theta )$, 代入得

$$\begin{array}{rl}{\mathbb{E}}_{(s_t,a_t)|\pi }[{\gamma }^t{G}_t{\nabla }_{\theta }\ln \pi (a_t|s_t;\theta )]& =\sum _{(s_t,a_t)\in \mathcal{S}\times \mathcal{A}}\pi (a_t|s_t;\theta ){\gamma }^t{G}_t{\nabla }_{\theta }\pi (a_t|s_t;\theta )/\pi (a_t|s_t;\theta )\\ & =\sum _{(s_t,a_t)\in \mathcal{S}\times \mathcal{A}}{\gamma }^t{G}_t{\nabla }_{\theta }\pi (a_t|s_t;\theta )\\ & =\sum _{(s_t,a_t)\in \mathcal{S}\times \mathcal{A}}{\pi }^{\prime }(a_t|s_t)\frac{1}{{\pi }^{\prime }(a_t|s_t)}{\gamma }^i{G}_t{\nabla }_{\theta }\pi (a_t|s_t;\theta )\\ & ={\mathbb{E}}_{(s_t,a_t)|{\pi }^{\prime }}[\frac{1}{{\pi }^{\prime }(a_t|s_t)}{\gamma }^t{G}_t{\nabla }_{\theta }\pi (a_t|s_t;\theta )]\end{array}$$