当前位置：首页 > article >正文

【深度学习】强化学习（RL）-A3C（Asynchronous Advantage Actor-Critic）

article 2025/2/25 22:34:26

A3C（Asynchronous Advantage Actor-Critic）详解

A3C（Asynchronous Advantage Actor-Critic） 是 深度强化学习（Deep Reinforcement Learning, DRL） 领域的重要算法，由 DeepMind 在 2016 年提出。它是一种异步并行的 Actor-Critic 方法，用于解决强化学习中收敛慢、数据利用率低等问题，适用于复杂的高维度环境（如 Atari 游戏、机器人控制等）。

1. A3C 的核心思想

A3C 是 Actor-Critic 方法的增强版，主要改进点是：

通过异步（Asynchronous）并行地训练多个代理（Agent），提高训练效率，减少计算资源的浪费。
采用 优势函数（Advantage Function） 来减少估计值的方差，从而提高训练的稳定性和收敛速度。
结合 策略梯度方法（Policy Gradient） 和 值函数估计（Value Function Estimation），平衡探索（Exploration）和利用（Exploitation）。

A3C 相比 DQN（Deep Q-Network） 的最大不同点在于：

无经验回放（No Experience Replay）：A3C 不需要存储和采样经验数据，而是让多个 Agent 在不同环境中异步探索，提高了训练稳定性。
异步更新（Asynchronous Updates）：多个 Agent 以不同的时间间隔独立探索环境，减少了训练数据的相关性，提高了探索的多样性。

2. A3C 的网络架构

A3C 采用 Actor-Critic 结构，包含两个关键的网络：

Actor（策略网络，Policy Network）：
- 负责学习策略（Policy），即在每个状态下采取不同动作的概率分布 $\pi(a|s;\theta)$ 。
- 通过梯度上升优化策略，使得高奖励的动作更可能被选中。
Critic（值函数网络，Value Network）：
- 负责学习状态值函数 $V(s;\theta_v)$ ，即在当前状态 s 下，未来可能获得的长期奖励。
- 计算 优势函数（Advantage Function），用于指导 Actor 更新策略。

A3C 采用的目标函数

A3C 采用 Actor-Critic 方法，即：

策略更新（Actor）：
$\nabla_{\theta} J = \mathbb{E} \left[ \nabla_{\theta} \log \pi(a | s; \theta) A(s, a) \right]$
其中，优势函数（Advantage Function）：
$A(s, a) = R - V(s;\theta_v)$
这里，RRR 是从当前状态出发获得的实际奖励， $V(s;\theta_v)$ 是 Critic 估计的状态值。
Critic 更新（使用均方误差）：
$L_v(\theta_v) = (R - V(s;\theta_v))^2$
Critic 通过最小化均方误差来优化值函数，使其能够更准确地估计状态值。

3. A3C 的训练流程

A3C 采用 多个异步环境并行运行，每个环境都有自己的 Actor 和 Critic，它们共享全局参数 θ 和 $\theta_v$ ，训练过程如下：

初始化全局网络参数 θ 和 $\theta_v$ 。
创建多个 Worker（子代理），每个 Worker 独立运行：
- 与环境交互，收集数据（状态、动作、奖励）。
- 计算优势函数和目标值。
- 计算梯度并更新全局网络参数。
所有 Worker 异步执行，独立探索，最终收敛到较优的策略。

A3C 通过 多线程并行执行，使得不同 Worker 在不同环境中探索，提高了数据效率和稳定性。

4. A3C 代码实现（PyTorch）

以下是 A3C 的简化版本，包括 Actor-Critic 网络和训练流程。

Actor-Critic 网络

import torch
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
import gym
import numpy as np


# 定义全局 Actor-Critic 网络
class ActorCritic(nn.Module):
    def __init__(self, input_dim, action_dim):
        super(ActorCritic, self).__init__()
        self.fc1 = nn.Linear(input_dim, 128)
        self.actor = nn.Linear(128, action_dim)
        self.critic = nn.Linear(128, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        policy_logits = self.actor(x)
        value = self.critic(x)
        return policy_logits, value

Worker 进程

# Worker 进程
def worker(global_model, optimizer, env_name, global_episode_counter):
    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    local_model = ActorCritic(state_dim, action_dim)
    local_model.load_state_dict(global_model.state_dict())

    while global_episode_counter.value < 1000:  # 限制训练轮数
        state_info = env.reset()
        if isinstance(state_info, tuple):  # 适配新的 Gym 版本
            state = state_info[0]  # 只获取 state，不要 info
        else:
            state = state_info

        state = np.array(state, dtype=np.float32)
        done = False
        episode_reward = 0

        while not done:
            with torch.no_grad():  # 关闭梯度计算
                state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
                policy_logits, _ = global_model(state_tensor)

                action_prob = torch.softmax(policy_logits, dim=1)
                action = torch.multinomial(action_prob, num_samples=1).item()

            # Gym 新版 step() 返回 5 个值：state, reward, terminated, truncated, info
            step_result = env.step(action)

            if len(step_result) == 4:  # 旧版 Gym
                next_state_info, reward, done, _ = step_result
            else:  # 新版 Gym
                next_state_info, reward, terminated, truncated, _ = step_result
                done = terminated or truncated  # 兼容新版本

            next_state = np.array(next_state_info, dtype=np.float32)
            state = next_state
            episode_reward += reward

        # 更新全局 episode 计数
        with global_episode_counter.get_lock():
            global_episode_counter.value += 1
            print(f"Episode: {global_episode_counter.value}, Reward: {episode_reward}")

主训练循环

# 主训练逻辑
if __name__ == "__main__":
    env_name = "CartPole-v1"
    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    global_model = ActorCritic(state_dim, action_dim)
    global_model.share_memory()  # 共享内存

    optimizer = optim.Adam(global_model.parameters(), lr=0.001)
    global_episode_counter = mp.Value('i', 0)  # 多进程共享变量

    num_workers = 4
    processes = []
    for _ in range(num_workers):
        p = mp.Process(target=worker, args=(global_model, optimizer, env_name, global_episode_counter))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

运行结果:

Episode: 1, Reward: 24.0
Episode: 2, Reward: 11.0
Episode: 3, Reward: 16.0
Episode: 4, Reward: 13.0
Episode: 5, Reward: 33.0
Episode: 6, Reward: 18.0
Episode: 7, Reward: 12.0
Episode: 8, Reward: 17.0
Episode: 9, Reward: 11.0
Episode: 10, Reward: 20.0
Episode: 11, Reward: 14.0
Episode: 12, Reward: 20.0
Episode: 13, Reward: 12.0
Episode: 14, Reward: 22.0
Episode: 15, Reward: 17.0
Episode: 16, Reward: 14.0
Episode: 17, Reward: 18.0
Episode: 18, Reward: 12.0
Episode: 19, Reward: 25.0
Episode: 20, Reward: 12.0
Episode: 21, Reward: 14.0
Episode: 22, Reward: 22.0
Episode: 23, Reward: 21.0
Episode: 24, Reward: 12.0
Episode: 25, Reward: 11.0
Episode: 26, Reward: 13.0
Episode: 27, Reward: 10.0
Episode: 28, Reward: 13.0
Episode: 29, Reward: 10.0
Episode: 30, Reward: 17.0
Episode: 31, Reward: 10.0
Episode: 32, Reward: 18.0
Episode: 33, Reward: 26.0
Episode: 34, Reward: 15.0
Episode: 35, Reward: 19.0
Episode: 36, Reward: 17.0
Episode: 37, Reward: 14.0
Episode: 38, Reward: 9.0
Episode: 39, Reward: 10.0
Episode: 40, Reward: 12.0
Episode: 41, Reward: 23.0
Episode: 42, Reward: 15.0
Episode: 43, Reward: 29.0
Episode: 44, Reward: 11.0
Episode: 45, Reward: 28.0
Episode: 46, Reward: 10.0
Episode: 47, Reward: 13.0
Episode: 48, Reward: 13.0
Episode: 49, Reward: 21.0
Episode: 50, Reward: 14.0
Episode: 51, Reward: 15.0
Episode: 52, Reward: 16.0
Episode: 53, Reward: 47.0
Episode: 54, Reward: 14.0
Episode: 55, Reward: 18.0
Episode: 56, Reward: 9.0
Episode: 57, Reward: 22.0
Episode: 58, Reward: 16.0
Episode: 59, Reward: 11.0
Episode: 60, Reward: 14.0
Episode: 61, Reward: 17.0
Episode: 62, Reward: 32.0
Episode: 63, Reward: 13.0
Episode: 64, Reward: 8.0
Episode: 65, Reward: 18.0
Episode: 66, Reward: 19.0
Episode: 67, Reward: 26.0
Episode: 68, Reward: 19.0
Episode: 69, Reward: 39.0
Episode: 70, Reward: 29.0
Episode: 71, Reward: 26.0
Episode: 72, Reward: 26.0
Episode: 73, Reward: 13.0
Episode: 74, Reward: 23.0
Episode: 75, Reward: 15.0
Episode: 76, Reward: 13.0
Episode: 77, Reward: 50.0
Episode: 78, Reward: 11.0
Episode: 79, Reward: 21.0
Episode: 80, Reward: 81.0
Episode: 81, Reward: 9.0
Episode: 82, Reward: 20.0
Episode: 83, Reward: 14.0
Episode: 84, Reward: 17.0
Episode: 85, Reward: 25.0
Episode: 86, Reward: 10.0
Episode: 87, Reward: 21.0
Episode: 88, Reward: 24.0
Episode: 89, Reward: 18.0
Episode: 90, Reward: 15.0
Episode: 91, Reward: 15.0
Episode: 92, Reward: 11.0
Episode: 93, Reward: 16.0
Episode: 94, Reward: 21.0
Episode: 95, Reward: 14.0
Episode: 96, Reward: 13.0
Episode: 97, Reward: 17.0
Episode: 98, Reward: 11.0
Episode: 99, Reward: 20.0
Episode: 100, Reward: 11.0
Episode: 101, Reward: 11.0
Episode: 102, Reward: 20.0
Episode: 103, Reward: 26.0
Episode: 104, Reward: 29.0
Episode: 105, Reward: 11.0
Episode: 106, Reward: 11.0
Episode: 107, Reward: 12.0
Episode: 108, Reward: 15.0
Episode: 109, Reward: 11.0
Episode: 110, Reward: 15.0
Episode: 111, Reward: 12.0
Episode: 112, Reward: 16.0
Episode: 113, Reward: 19.0
Episode: 114, Reward: 13.0
Episode: 115, Reward: 12.0
Episode: 116, Reward: 16.0
Episode: 117, Reward: 11.0
Episode: 118, Reward: 16.0
Episode: 119, Reward: 13.0
Episode: 120, Reward: 16.0
Episode: 121, Reward: 22.0
Episode: 122, Reward: 11.0
Episode: 123, Reward: 11.0
Episode: 124, Reward: 14.0
Episode: 125, Reward: 12.0
Episode: 126, Reward: 12.0
Episode: 127, Reward: 10.0
Episode: 128, Reward: 18.0
Episode: 129, Reward: 62.0
Episode: 130, Reward: 22.0
Episode: 131, Reward: 9.0
Episode: 132, Reward: 15.0
Episode: 133, Reward: 23.0
Episode: 134, Reward: 13.0
Episode: 135, Reward: 12.0
Episode: 136, Reward: 14.0
Episode: 137, Reward: 10.0
Episode: 138, Reward: 20.0
Episode: 139, Reward: 18.0
Episode: 140, Reward: 23.0
Episode: 141, Reward: 28.0
Episode: 142, Reward: 38.0
Episode: 143, Reward: 18.0
Episode: 144, Reward: 25.0
Episode: 145, Reward: 17.0
Episode: 146, Reward: 41.0
Episode: 147, Reward: 10.0
Episode: 148, Reward: 18.0
Episode: 149, Reward: 11.0
Episode: 150, Reward: 33.0
Episode: 151, Reward: 25.0
Episode: 152, Reward: 25.0
Episode: 153, Reward: 15.0
Episode: 154, Reward: 16.0
Episode: 155, Reward: 20.0
Episode: 156, Reward: 13.0
Episode: 157, Reward: 33.0
Episode: 158, Reward: 13.0
Episode: 159, Reward: 60.0
Episode: 160, Reward: 27.0
Episode: 161, Reward: 14.0
Episode: 162, Reward: 13.0
Episode: 163, Reward: 37.0
Episode: 164, Reward: 11.0
Episode: 165, Reward: 12.0
Episode: 166, Reward: 18.0
Episode: 167, Reward: 11.0
Episode: 168, Reward: 28.0
Episode: 169, Reward: 13.0
Episode: 170, Reward: 13.0
Episode: 171, Reward: 11.0
Episode: 172, Reward: 20.0
Episode: 173, Reward: 9.0
Episode: 174, Reward: 10.0
Episode: 175, Reward: 15.0
Episode: 176, Reward: 29.0
Episode: 177, Reward: 33.0
Episode: 178, Reward: 12.0
Episode: 179, Reward: 12.0
Episode: 180, Reward: 13.0
Episode: 181, Reward: 26.0
Episode: 182, Reward: 43.0
Episode: 183, Reward: 33.0
Episode: 184, Reward: 21.0
Episode: 185, Reward: 23.0
Episode: 186, Reward: 22.0
Episode: 187, Reward: 25.0
Episode: 188, Reward: 9.0
Episode: 189, Reward: 37.0
Episode: 190, Reward: 10.0
Episode: 191, Reward: 12.0
Episode: 192, Reward: 16.0
Episode: 193, Reward: 28.0
Episode: 194, Reward: 17.0
Episode: 195, Reward: 16.0
Episode: 196, Reward: 29.0
Episode: 197, Reward: 35.0
Episode: 198, Reward: 12.0
Episode: 199, Reward: 10.0
Episode: 200, Reward: 14.0
Episode: 201, Reward: 14.0
Episode: 202, Reward: 13.0
Episode: 203, Reward: 13.0
Episode: 204, Reward: 10.0
Episode: 205, Reward: 15.0
Episode: 206, Reward: 14.0
Episode: 207, Reward: 15.0
Episode: 208, Reward: 9.0
Episode: 209, Reward: 18.0
Episode: 210, Reward: 31.0
Episode: 211, Reward: 21.0
Episode: 212, Reward: 47.0
Episode: 213, Reward: 19.0
Episode: 214, Reward: 10.0
Episode: 215, Reward: 12.0
Episode: 216, Reward: 20.0
Episode: 217, Reward: 18.0
Episode: 218, Reward: 10.0
Episode: 219, Reward: 9.0
Episode: 220, Reward: 16.0
Episode: 221, Reward: 19.0
Episode: 222, Reward: 11.0
Episode: 223, Reward: 12.0
Episode: 224, Reward: 28.0
Episode: 225, Reward: 11.0
Episode: 226, Reward: 34.0
Episode: 227, Reward: 11.0
Episode: 228, Reward: 12.0
Episode: 229, Reward: 12.0
Episode: 230, Reward: 10.0
Episode: 231, Reward: 15.0
Episode: 232, Reward: 15.0
Episode: 233, Reward: 24.0
Episode: 234, Reward: 18.0
Episode: 235, Reward: 11.0
Episode: 236, Reward: 42.0
Episode: 237, Reward: 12.0
Episode: 238, Reward: 12.0
Episode: 239, Reward: 15.0
Episode: 240, Reward: 10.0
Episode: 241, Reward: 21.0
Episode: 242, Reward: 15.0
Episode: 243, Reward: 13.0
Episode: 244, Reward: 11.0
Episode: 245, Reward: 33.0
Episode: 246, Reward: 44.0
Episode: 247, Reward: 20.0
Episode: 248, Reward: 31.0
Episode: 249, Reward: 39.0
Episode: 250, Reward: 27.0
Episode: 251, Reward: 16.0
Episode: 252, Reward: 9.0
Episode: 253, Reward: 21.0
Episode: 254, Reward: 13.0
Episode: 255, Reward: 11.0
Episode: 256, Reward: 26.0
Episode: 257, Reward: 23.0
Episode: 258, Reward: 14.0
Episode: 259, Reward: 12.0
Episode: 260, Reward: 13.0
Episode: 261, Reward: 28.0
Episode: 262, Reward: 26.0
Episode: 263, Reward: 19.0
Episode: 264, Reward: 34.0
Episode: 265, Reward: 14.0
Episode: 266, Reward: 26.0
Episode: 267, Reward: 13.0
Episode: 268, Reward: 80.0
Episode: 269, Reward: 20.0
Episode: 270, Reward: 12.0
Episode: 271, Reward: 11.0
Episode: 272, Reward: 14.0
Episode: 273, Reward: 19.0
Episode: 274, Reward: 27.0
Episode: 275, Reward: 14.0
Episode: 276, Reward: 16.0
Episode: 277, Reward: 18.0
Episode: 278, Reward: 23.0
Episode: 279, Reward: 18.0
Episode: 280, Reward: 13.0
Episode: 281, Reward: 11.0
Episode: 282, Reward: 20.0
Episode: 283, Reward: 16.0
Episode: 284, Reward: 10.0
Episode: 285, Reward: 12.0
Episode: 286, Reward: 47.0
Episode: 287, Reward: 18.0
Episode: 288, Reward: 17.0
Episode: 289, Reward: 24.0
Episode: 290, Reward: 10.0
Episode: 291, Reward: 19.0
Episode: 292, Reward: 12.0
Episode: 293, Reward: 13.0
Episode: 294, Reward: 11.0
Episode: 295, Reward: 32.0
Episode: 296, Reward: 12.0
Episode: 297, Reward: 27.0
Episode: 298, Reward: 10.0
Episode: 299, Reward: 18.0
Episode: 300, Reward: 14.0
Episode: 301, Reward: 21.0
Episode: 302, Reward: 12.0
Episode: 303, Reward: 13.0
Episode: 304, Reward: 14.0
Episode: 305, Reward: 13.0
Episode: 306, Reward: 15.0
Episode: 307, Reward: 15.0
Episode: 308, Reward: 12.0
Episode: 309, Reward: 27.0
Episode: 310, Reward: 20.0
Episode: 311, Reward: 16.0
Episode: 312, Reward: 12.0
Episode: 313, Reward: 9.0
Episode: 314, Reward: 24.0
Episode: 315, Reward: 41.0
Episode: 316, Reward: 16.0
Episode: 317, Reward: 20.0
Episode: 318, Reward: 11.0
Episode: 319, Reward: 15.0
Episode: 320, Reward: 41.0
Episode: 321, Reward: 9.0
Episode: 322, Reward: 42.0
Episode: 323, Reward: 35.0
Episode: 324, Reward: 10.0
Episode: 325, Reward: 11.0
Episode: 326, Reward: 11.0
Episode: 327, Reward: 10.0
Episode: 328, Reward: 23.0
Episode: 329, Reward: 26.0
Episode: 330, Reward: 14.0
Episode: 331, Reward: 19.0
Episode: 332, Reward: 27.0
Episode: 333, Reward: 9.0
Episode: 334, Reward: 11.0
Episode: 335, Reward: 22.0
Episode: 336, Reward: 17.0
Episode: 337, Reward: 17.0
Episode: 338, Reward: 16.0
Episode: 339, Reward: 14.0
Episode: 340, Reward: 15.0
Episode: 341, Reward: 22.0
Episode: 342, Reward: 12.0
Episode: 343, Reward: 28.0
Episode: 344, Reward: 21.0
Episode: 345, Reward: 12.0
Episode: 346, Reward: 27.0
Episode: 347, Reward: 16.0
Episode: 348, Reward: 13.0
Episode: 349, Reward: 15.0
Episode: 350, Reward: 8.0
Episode: 351, Reward: 13.0
Episode: 352, Reward: 17.0
Episode: 353, Reward: 19.0
Episode: 354, Reward: 9.0
Episode: 355, Reward: 14.0
Episode: 356, Reward: 9.0
Episode: 357, Reward: 19.0
Episode: 358, Reward: 10.0
Episode: 359, Reward: 12.0
Episode: 360, Reward: 23.0
Episode: 361, Reward: 26.0
Episode: 362, Reward: 15.0
Episode: 363, Reward: 34.0
Episode: 364, Reward: 37.0
Episode: 365, Reward: 28.0
Episode: 366, Reward: 11.0
Episode: 367, Reward: 19.0
Episode: 368, Reward: 10.0
Episode: 369, Reward: 15.0
Episode: 370, Reward: 32.0
Episode: 371, Reward: 30.0
Episode: 372, Reward: 32.0
Episode: 373, Reward: 17.0
Episode: 374, Reward: 8.0
Episode: 375, Reward: 12.0
Episode: 376, Reward: 11.0
Episode: 377, Reward: 36.0
Episode: 378, Reward: 16.0
Episode: 379, Reward: 14.0
Episode: 380, Reward: 13.0
Episode: 381, Reward: 13.0
Episode: 382, Reward: 13.0
Episode: 383, Reward: 16.0
Episode: 384, Reward: 13.0
Episode: 385, Reward: 13.0
Episode: 386, Reward: 14.0
Episode: 387, Reward: 16.0
Episode: 388, Reward: 17.0
Episode: 389, Reward: 28.0
Episode: 390, Reward: 17.0
Episode: 391, Reward: 16.0
Episode: 392, Reward: 10.0
Episode: 393, Reward: 18.0
Episode: 394, Reward: 12.0
Episode: 395, Reward: 18.0
Episode: 396, Reward: 28.0
Episode: 397, Reward: 31.0
Episode: 398, Reward: 16.0
Episode: 399, Reward: 15.0
Episode: 400, Reward: 22.0
Episode: 401, Reward: 28.0
Episode: 402, Reward: 22.0
Episode: 403, Reward: 10.0
Episode: 404, Reward: 36.0
Episode: 405, Reward: 13.0
Episode: 406, Reward: 17.0
Episode: 407, Reward: 14.0
Episode: 408, Reward: 19.0
Episode: 409, Reward: 16.0
Episode: 410, Reward: 18.0
Episode: 411, Reward: 16.0
Episode: 412, Reward: 22.0
Episode: 413, Reward: 17.0
Episode: 414, Reward: 13.0
Episode: 415, Reward: 15.0
Episode: 416, Reward: 17.0
Episode: 417, Reward: 15.0
Episode: 418, Reward: 14.0
Episode: 419, Reward: 25.0
Episode: 420, Reward: 12.0
Episode: 421, Reward: 46.0
Episode: 422, Reward: 21.0
Episode: 423, Reward: 15.0
Episode: 424, Reward: 11.0
Episode: 425, Reward: 9.0
Episode: 426, Reward: 18.0
Episode: 427, Reward: 23.0
Episode: 428, Reward: 24.0
Episode: 429, Reward: 16.0
Episode: 430, Reward: 29.0
Episode: 431, Reward: 11.0
Episode: 432, Reward: 16.0
Episode: 433, Reward: 16.0
Episode: 434, Reward: 11.0
Episode: 435, Reward: 10.0
Episode: 436, Reward: 10.0
Episode: 437, Reward: 11.0
Episode: 438, Reward: 11.0
Episode: 439, Reward: 22.0
Episode: 440, Reward: 17.0
Episode: 441, Reward: 9.0
Episode: 442, Reward: 11.0
Episode: 443, Reward: 14.0
Episode: 444, Reward: 11.0
Episode: 445, Reward: 16.0
Episode: 446, Reward: 16.0
Episode: 447, Reward: 16.0
Episode: 448, Reward: 21.0
Episode: 449, Reward: 20.0
Episode: 450, Reward: 27.0
Episode: 451, Reward: 13.0
Episode: 452, Reward: 33.0
Episode: 453, Reward: 22.0
Episode: 454, Reward: 16.0
Episode: 455, Reward: 18.0
Episode: 456, Reward: 30.0
Episode: 457, Reward: 13.0
Episode: 458, Reward: 10.0
Episode: 459, Reward: 14.0
Episode: 460, Reward: 16.0
Episode: 461, Reward: 18.0
Episode: 462, Reward: 11.0
Episode: 463, Reward: 18.0
Episode: 464, Reward: 10.0
Episode: 465, Reward: 19.0
Episode: 466, Reward: 27.0
Episode: 467, Reward: 16.0
Episode: 468, Reward: 15.0
Episode: 469, Reward: 12.0
Episode: 470, Reward: 14.0
Episode: 471, Reward: 18.0
Episode: 472, Reward: 16.0
Episode: 473, Reward: 9.0
Episode: 474, Reward: 25.0
Episode: 475, Reward: 29.0
Episode: 476, Reward: 13.0
Episode: 477, Reward: 23.0
Episode: 478, Reward: 17.0
Episode: 479, Reward: 15.0
Episode: 480, Reward: 20.0
Episode: 481, Reward: 11.0
Episode: 482, Reward: 11.0
Episode: 483, Reward: 18.0
Episode: 484, Reward: 21.0
Episode: 485, Reward: 15.0
Episode: 486, Reward: 12.0
Episode: 487, Reward: 11.0
Episode: 488, Reward: 18.0
Episode: 489, Reward: 20.0
Episode: 490, Reward: 14.0
Episode: 491, Reward: 11.0
Episode: 492, Reward: 18.0
Episode: 493, Reward: 23.0
Episode: 494, Reward: 29.0
Episode: 495, Reward: 12.0
Episode: 496, Reward: 11.0
Episode: 497, Reward: 12.0
Episode: 498, Reward: 16.0
Episode: 499, Reward: 10.0
Episode: 500, Reward: 14.0
Episode: 501, Reward: 9.0
Episode: 502, Reward: 32.0
Episode: 503, Reward: 12.0
Episode: 504, Reward: 16.0
Episode: 505, Reward: 21.0
Episode: 506, Reward: 15.0
Episode: 507, Reward: 14.0
Episode: 508, Reward: 28.0
Episode: 509, Reward: 14.0
Episode: 510, Reward: 12.0
Episode: 511, Reward: 15.0
Episode: 512, Reward: 19.0
Episode: 513, Reward: 14.0
Episode: 514, Reward: 12.0
Episode: 515, Reward: 13.0
Episode: 516, Reward: 13.0
Episode: 517, Reward: 10.0
Episode: 518, Reward: 18.0
Episode: 519, Reward: 18.0
Episode: 520, Reward: 18.0
Episode: 521, Reward: 12.0
Episode: 522, Reward: 27.0
Episode: 523, Reward: 11.0
Episode: 524, Reward: 9.0
Episode: 525, Reward: 24.0
Episode: 526, Reward: 47.0
Episode: 527, Reward: 17.0
Episode: 528, Reward: 10.0
Episode: 529, Reward: 22.0
Episode: 530, Reward: 24.0
Episode: 531, Reward: 13.0
Episode: 532, Reward: 14.0
Episode: 533, Reward: 29.0
Episode: 534, Reward: 11.0
Episode: 535, Reward: 17.0
Episode: 536, Reward: 17.0
Episode: 537, Reward: 28.0
Episode: 538, Reward: 9.0
Episode: 539, Reward: 27.0
Episode: 540, Reward: 13.0
Episode: 541, Reward: 16.0
Episode: 542, Reward: 16.0
Episode: 543, Reward: 12.0
Episode: 544, Reward: 20.0
Episode: 545, Reward: 26.0
Episode: 546, Reward: 11.0
Episode: 547, Reward: 20.0
Episode: 548, Reward: 21.0
Episode: 549, Reward: 38.0
Episode: 550, Reward: 26.0
Episode: 551, Reward: 11.0
Episode: 552, Reward: 18.0
Episode: 553, Reward: 13.0
Episode: 554, Reward: 14.0
Episode: 555, Reward: 15.0
Episode: 556, Reward: 28.0
Episode: 557, Reward: 11.0
Episode: 558, Reward: 9.0
Episode: 559, Reward: 16.0
Episode: 560, Reward: 19.0
Episode: 561, Reward: 13.0
Episode: 562, Reward: 23.0
Episode: 563, Reward: 12.0
Episode: 564, Reward: 9.0
Episode: 565, Reward: 18.0
Episode: 566, Reward: 14.0
Episode: 567, Reward: 11.0
Episode: 568, Reward: 12.0
Episode: 569, Reward: 60.0
Episode: 570, Reward: 26.0
Episode: 571, Reward: 50.0
Episode: 572, Reward: 16.0
Episode: 573, Reward: 27.0
Episode: 574, Reward: 46.0
Episode: 575, Reward: 10.0
Episode: 576, Reward: 31.0
Episode: 577, Reward: 14.0
Episode: 578, Reward: 14.0
Episode: 579, Reward: 42.0
Episode: 580, Reward: 36.0
Episode: 581, Reward: 46.0
Episode: 582, Reward: 21.0
Episode: 583, Reward: 11.0
Episode: 584, Reward: 34.0
Episode: 585, Reward: 10.0
Episode: 586, Reward: 13.0
Episode: 587, Reward: 10.0
Episode: 588, Reward: 16.0
Episode: 589, Reward: 16.0
Episode: 590, Reward: 16.0
Episode: 591, Reward: 12.0
Episode: 592, Reward: 17.0
Episode: 593, Reward: 11.0
Episode: 594, Reward: 23.0
Episode: 595, Reward: 11.0
Episode: 596, Reward: 11.0
Episode: 597, Reward: 23.0
Episode: 598, Reward: 21.0
Episode: 599, Reward: 18.0
Episode: 600, Reward: 15.0
Episode: 601, Reward: 9.0
Episode: 602, Reward: 22.0
Episode: 603, Reward: 20.0
Episode: 604, Reward: 11.0
Episode: 605, Reward: 12.0
Episode: 606, Reward: 16.0
Episode: 607, Reward: 15.0
Episode: 608, Reward: 14.0
Episode: 609, Reward: 27.0
Episode: 610, Reward: 19.0
Episode: 611, Reward: 34.0
Episode: 612, Reward: 18.0
Episode: 613, Reward: 12.0
Episode: 614, Reward: 11.0
Episode: 615, Reward: 12.0
Episode: 616, Reward: 20.0
Episode: 617, Reward: 31.0
Episode: 618, Reward: 14.0
Episode: 619, Reward: 20.0
Episode: 620, Reward: 15.0
Episode: 621, Reward: 17.0
Episode: 622, Reward: 24.0
Episode: 623, Reward: 49.0
Episode: 624, Reward: 13.0
Episode: 625, Reward: 12.0
Episode: 626, Reward: 12.0
Episode: 627, Reward: 11.0
Episode: 628, Reward: 24.0
Episode: 629, Reward: 13.0
Episode: 630, Reward: 21.0
Episode: 631, Reward: 36.0
Episode: 632, Reward: 28.0
Episode: 633, Reward: 60.0
Episode: 634, Reward: 9.0
Episode: 635, Reward: 10.0
Episode: 636, Reward: 17.0
Episode: 637, Reward: 15.0
Episode: 638, Reward: 14.0
Episode: 639, Reward: 22.0
Episode: 640, Reward: 24.0
Episode: 641, Reward: 19.0
Episode: 642, Reward: 26.0
Episode: 643, Reward: 15.0
Episode: 644, Reward: 17.0
Episode: 645, Reward: 15.0
Episode: 646, Reward: 17.0
Episode: 647, Reward: 14.0
Episode: 648, Reward: 17.0
Episode: 649, Reward: 16.0
Episode: 650, Reward: 10.0
Episode: 651, Reward: 13.0
Episode: 652, Reward: 15.0
Episode: 653, Reward: 34.0
Episode: 654, Reward: 14.0
Episode: 655, Reward: 11.0
Episode: 656, Reward: 9.0
Episode: 657, Reward: 15.0
Episode: 658, Reward: 11.0
Episode: 659, Reward: 12.0
Episode: 660, Reward: 16.0
Episode: 661, Reward: 24.0
Episode: 662, Reward: 11.0
Episode: 663, Reward: 14.0
Episode: 664, Reward: 46.0
Episode: 665, Reward: 14.0
Episode: 666, Reward: 24.0
Episode: 667, Reward: 28.0
Episode: 668, Reward: 16.0
Episode: 669, Reward: 15.0
Episode: 670, Reward: 12.0
Episode: 671, Reward: 13.0
Episode: 672, Reward: 25.0
Episode: 673, Reward: 23.0
Episode: 674, Reward: 13.0
Episode: 675, Reward: 27.0
Episode: 676, Reward: 24.0
Episode: 677, Reward: 18.0
Episode: 678, Reward: 15.0
Episode: 679, Reward: 12.0
Episode: 680, Reward: 9.0
Episode: 681, Reward: 11.0
Episode: 682, Reward: 17.0
Episode: 683, Reward: 9.0
Episode: 684, Reward: 20.0
Episode: 685, Reward: 10.0
Episode: 686, Reward: 12.0
Episode: 687, Reward: 40.0
Episode: 688, Reward: 24.0
Episode: 689, Reward: 25.0
Episode: 690, Reward: 9.0
Episode: 691, Reward: 16.0
Episode: 692, Reward: 9.0
Episode: 693, Reward: 13.0
Episode: 694, Reward: 17.0
Episode: 695, Reward: 15.0
Episode: 696, Reward: 18.0
Episode: 697, Reward: 13.0
Episode: 698, Reward: 10.0
Episode: 699, Reward: 56.0
Episode: 700, Reward: 13.0
Episode: 701, Reward: 17.0
Episode: 702, Reward: 11.0
Episode: 703, Reward: 23.0
Episode: 704, Reward: 33.0
Episode: 705, Reward: 24.0
Episode: 706, Reward: 25.0
Episode: 707, Reward: 26.0
Episode: 708, Reward: 18.0
Episode: 709, Reward: 10.0
Episode: 710, Reward: 9.0
Episode: 711, Reward: 11.0
Episode: 712, Reward: 12.0
Episode: 713, Reward: 21.0
Episode: 714, Reward: 16.0
Episode: 715, Reward: 12.0
Episode: 716, Reward: 24.0
Episode: 717, Reward: 23.0
Episode: 718, Reward: 16.0
Episode: 719, Reward: 32.0
Episode: 720, Reward: 25.0
Episode: 721, Reward: 14.0
Episode: 722, Reward: 9.0
Episode: 723, Reward: 11.0
Episode: 724, Reward: 30.0
Episode: 725, Reward: 21.0
Episode: 726, Reward: 13.0
Episode: 727, Reward: 14.0
Episode: 728, Reward: 12.0
Episode: 729, Reward: 32.0
Episode: 730, Reward: 12.0
Episode: 731, Reward: 9.0
Episode: 732, Reward: 32.0
Episode: 733, Reward: 15.0
Episode: 734, Reward: 11.0
Episode: 735, Reward: 14.0
Episode: 736, Reward: 13.0
Episode: 737, Reward: 16.0
Episode: 738, Reward: 23.0
Episode: 739, Reward: 14.0
Episode: 740, Reward: 17.0
Episode: 741, Reward: 12.0
Episode: 742, Reward: 43.0
Episode: 743, Reward: 36.0
Episode: 744, Reward: 16.0
Episode: 745, Reward: 13.0
Episode: 746, Reward: 28.0
Episode: 747, Reward: 11.0
Episode: 748, Reward: 10.0
Episode: 749, Reward: 24.0
Episode: 750, Reward: 14.0
Episode: 751, Reward: 27.0
Episode: 752, Reward: 13.0
Episode: 753, Reward: 23.0
Episode: 754, Reward: 20.0
Episode: 755, Reward: 9.0
Episode: 756, Reward: 19.0
Episode: 757, Reward: 32.0
Episode: 758, Reward: 12.0
Episode: 759, Reward: 18.0
Episode: 760, Reward: 12.0
Episode: 761, Reward: 25.0
Episode: 762, Reward: 15.0
Episode: 763, Reward: 29.0
Episode: 764, Reward: 9.0
Episode: 765, Reward: 14.0
Episode: 766, Reward: 17.0
Episode: 767, Reward: 11.0
Episode: 768, Reward: 15.0
Episode: 769, Reward: 10.0
Episode: 770, Reward: 23.0
Episode: 771, Reward: 10.0
Episode: 772, Reward: 14.0
Episode: 773, Reward: 18.0
Episode: 774, Reward: 13.0
Episode: 775, Reward: 29.0
Episode: 776, Reward: 74.0
Episode: 777, Reward: 13.0
Episode: 778, Reward: 12.0
Episode: 779, Reward: 21.0
Episode: 780, Reward: 14.0
Episode: 781, Reward: 12.0
Episode: 782, Reward: 15.0
Episode: 783, Reward: 38.0
Episode: 784, Reward: 9.0
Episode: 785, Reward: 15.0
Episode: 786, Reward: 24.0
Episode: 787, Reward: 50.0
Episode: 788, Reward: 14.0
Episode: 789, Reward: 28.0
Episode: 790, Reward: 14.0
Episode: 791, Reward: 12.0
Episode: 792, Reward: 18.0
Episode: 793, Reward: 10.0
Episode: 794, Reward: 11.0
Episode: 795, Reward: 10.0
Episode: 796, Reward: 11.0
Episode: 797, Reward: 24.0
Episode: 798, Reward: 13.0
Episode: 799, Reward: 11.0
Episode: 800, Reward: 22.0
Episode: 801, Reward: 14.0
Episode: 802, Reward: 51.0
Episode: 803, Reward: 11.0
Episode: 804, Reward: 10.0
Episode: 805, Reward: 20.0
Episode: 806, Reward: 13.0
Episode: 807, Reward: 23.0
Episode: 808, Reward: 12.0
Episode: 809, Reward: 11.0
Episode: 810, Reward: 13.0
Episode: 811, Reward: 14.0
Episode: 812, Reward: 10.0
Episode: 813, Reward: 23.0
Episode: 814, Reward: 15.0
Episode: 815, Reward: 34.0
Episode: 816, Reward: 27.0
Episode: 817, Reward: 17.0
Episode: 818, Reward: 19.0
Episode: 819, Reward: 14.0
Episode: 820, Reward: 15.0
Episode: 821, Reward: 22.0
Episode: 822, Reward: 19.0
Episode: 823, Reward: 16.0
Episode: 824, Reward: 17.0
Episode: 825, Reward: 12.0
Episode: 826, Reward: 13.0
Episode: 827, Reward: 15.0
Episode: 828, Reward: 22.0
Episode: 829, Reward: 18.0
Episode: 830, Reward: 16.0
Episode: 831, Reward: 21.0
Episode: 832, Reward: 20.0
Episode: 833, Reward: 20.0
Episode: 834, Reward: 37.0
Episode: 835, Reward: 11.0
Episode: 836, Reward: 15.0
Episode: 837, Reward: 11.0
Episode: 838, Reward: 22.0
Episode: 839, Reward: 18.0
Episode: 840, Reward: 33.0
Episode: 841, Reward: 19.0
Episode: 842, Reward: 14.0
Episode: 843, Reward: 11.0
Episode: 844, Reward: 10.0
Episode: 845, Reward: 15.0
Episode: 846, Reward: 22.0
Episode: 847, Reward: 13.0
Episode: 848, Reward: 10.0
Episode: 849, Reward: 11.0
Episode: 850, Reward: 16.0
Episode: 851, Reward: 23.0
Episode: 852, Reward: 28.0
Episode: 853, Reward: 26.0
Episode: 854, Reward: 9.0
Episode: 855, Reward: 11.0
Episode: 856, Reward: 18.0
Episode: 857, Reward: 18.0
Episode: 858, Reward: 19.0
Episode: 859, Reward: 16.0
Episode: 860, Reward: 11.0
Episode: 861, Reward: 9.0
Episode: 862, Reward: 16.0
Episode: 863, Reward: 15.0
Episode: 864, Reward: 20.0
Episode: 865, Reward: 25.0
Episode: 866, Reward: 14.0
Episode: 867, Reward: 12.0
Episode: 868, Reward: 13.0
Episode: 869, Reward: 16.0
Episode: 870, Reward: 16.0
Episode: 871, Reward: 27.0
Episode: 872, Reward: 15.0
Episode: 873, Reward: 14.0
Episode: 874, Reward: 17.0
Episode: 875, Reward: 11.0
Episode: 876, Reward: 30.0
Episode: 877, Reward: 10.0
Episode: 878, Reward: 26.0
Episode: 879, Reward: 11.0
Episode: 880, Reward: 22.0
Episode: 881, Reward: 13.0
Episode: 882, Reward: 29.0
Episode: 883, Reward: 13.0
Episode: 884, Reward: 18.0
Episode: 885, Reward: 15.0
Episode: 886, Reward: 12.0
Episode: 887, Reward: 11.0
Episode: 888, Reward: 25.0
Episode: 889, Reward: 12.0
Episode: 890, Reward: 14.0
Episode: 891, Reward: 23.0
Episode: 892, Reward: 48.0
Episode: 893, Reward: 14.0
Episode: 894, Reward: 19.0
Episode: 895, Reward: 10.0
Episode: 896, Reward: 10.0
Episode: 897, Reward: 13.0
Episode: 898, Reward: 13.0
Episode: 899, Reward: 14.0
Episode: 900, Reward: 10.0
Episode: 901, Reward: 16.0
Episode: 902, Reward: 20.0
Episode: 903, Reward: 30.0
Episode: 904, Reward: 11.0
Episode: 905, Reward: 18.0
Episode: 906, Reward: 17.0
Episode: 907, Reward: 17.0
Episode: 908, Reward: 17.0
Episode: 909, Reward: 33.0
Episode: 910, Reward: 18.0
Episode: 911, Reward: 17.0
Episode: 912, Reward: 12.0
Episode: 913, Reward: 18.0
Episode: 914, Reward: 15.0
Episode: 915, Reward: 26.0
Episode: 916, Reward: 19.0
Episode: 917, Reward: 13.0
Episode: 918, Reward: 21.0
Episode: 919, Reward: 17.0
Episode: 920, Reward: 13.0
Episode: 921, Reward: 14.0
Episode: 922, Reward: 8.0
Episode: 923, Reward: 10.0
Episode: 924, Reward: 30.0
Episode: 925, Reward: 15.0
Episode: 926, Reward: 15.0
Episode: 927, Reward: 11.0
Episode: 928, Reward: 11.0
Episode: 929, Reward: 13.0
Episode: 930, Reward: 13.0
Episode: 931, Reward: 11.0
Episode: 932, Reward: 37.0
Episode: 933, Reward: 30.0
Episode: 934, Reward: 13.0
Episode: 935, Reward: 12.0
Episode: 936, Reward: 14.0
Episode: 937, Reward: 24.0
Episode: 938, Reward: 10.0
Episode: 939, Reward: 32.0
Episode: 940, Reward: 18.0
Episode: 941, Reward: 18.0
Episode: 942, Reward: 15.0
Episode: 943, Reward: 14.0
Episode: 944, Reward: 18.0
Episode: 945, Reward: 13.0
Episode: 946, Reward: 10.0
Episode: 947, Reward: 27.0
Episode: 948, Reward: 17.0
Episode: 949, Reward: 13.0
Episode: 950, Reward: 14.0
Episode: 951, Reward: 13.0
Episode: 952, Reward: 21.0
Episode: 953, Reward: 24.0
Episode: 954, Reward: 21.0
Episode: 955, Reward: 21.0
Episode: 956, Reward: 33.0
Episode: 957, Reward: 15.0
Episode: 958, Reward: 13.0
Episode: 959, Reward: 45.0
Episode: 960, Reward: 13.0
Episode: 961, Reward: 11.0
Episode: 962, Reward: 12.0
Episode: 963, Reward: 18.0
Episode: 964, Reward: 15.0
Episode: 965, Reward: 10.0
Episode: 966, Reward: 11.0
Episode: 967, Reward: 25.0
Episode: 968, Reward: 17.0
Episode: 969, Reward: 13.0
Episode: 970, Reward: 16.0
Episode: 971, Reward: 11.0
Episode: 972, Reward: 21.0
Episode: 973, Reward: 13.0
Episode: 974, Reward: 12.0
Episode: 975, Reward: 12.0
Episode: 976, Reward: 28.0
Episode: 977, Reward: 18.0
Episode: 978, Reward: 10.0
Episode: 979, Reward: 13.0
Episode: 980, Reward: 15.0
Episode: 981, Reward: 16.0
Episode: 982, Reward: 13.0
Episode: 983, Reward: 12.0
Episode: 984, Reward: 18.0
Episode: 985, Reward: 22.0
Episode: 986, Reward: 29.0
Episode: 987, Reward: 24.0
Episode: 988, Reward: 12.0
Episode: 989, Reward: 13.0
Episode: 990, Reward: 11.0
Episode: 991, Reward: 11.0
Episode: 992, Reward: 13.0
Episode: 993, Reward: 41.0
Episode: 994, Reward: 18.0
Episode: 995, Reward: 11.0
Episode: 996, Reward: 15.0
Episode: 997, Reward: 14.0
Episode: 998, Reward: 9.0
Episode: 999, Reward: 19.0
Episode: 1000, Reward: 14.0
Episode: 1001, Reward: 22.0
Episode: 1002, Reward: 16.0
Episode: 1003, Reward: 18.0

代码说明：

采用 多进程 来创建多个 Worker。
每个 Worker 进行独立的环境交互，并异步更新全局 Actor-Critic 网络。
使用 共享内存 来同步多个进程的参数。

5. A3C 的优势

提高训练效率：多个 Worker 并行执行，避免了 DQN 的经验回放机制，提高了数据利用率。
稳定性强：异步更新减少了数据相关性，提高了训练稳定性。
收敛速度快：相比传统 DQN，A3C 需要更少的训练时间即可获得较优策略。

6. A3C 的改进版本

由于 A3C 仍然存在计算资源消耗较高的问题，后续研究提出了 A2C（Advantage Actor-Critic），它采用 同步更新（Synchronous Updates），可以进一步提高训练的稳定性和效率。

7. 总结

特点	A3C 关键点
核心思想	结合 Actor-Critic，采用异步更新
网络结构	包括 Actor（策略）和 Critic（值函数）
训练方法	多线程异步训练，提高效率
目标函数	采用 Advantage Function 改进策略梯度
优点	提高数据效率、收敛快、适用于复杂任务
改进版本	A2C（同步版本）