Double Q-learning

使用 Python 的 Gymnasium 進行強化學習

Fouad Trad

Machine Learning Engineer

Q-learning

  • 估計最適動作價值函式
  • 因以最大 Q 更新而高估 Q 值
  • 可能導致次佳策略學習

 

Image showing the mathematical formula of the Q-learning update rule.

使用 Python 的 Gymnasium 進行強化學習

Double Q-learning

  • 維護兩個 Q 表
  • 彼此交互更新
  • 降低 Q 值高估風險

Image showing two Q-tables, Q0 and Q1, and each one is updated based on the other.

使用 Python 的 Gymnasium 進行強化學習

Double Q-learning 更新

  • 隨機選一個表

Image showing two Q-tables, Q0 and Q1, and each one is updated based on the other.

使用 Python 的 Gymnasium 進行強化學習

Q0 更新

Image showing two Q-tables, Q0 and Q1, and each one is updated based on the other.

Image showing how to find the best next action when updating Q0.

Image showing the update rule of Q0.

使用 Python 的 Gymnasium 進行強化學習

Q1 更新

Image showing two Q-tables, Q0 and Q1, and each one is updated based on the other.

Image showing how to find the best next action when updating Q1.

Image showing the update rule of Q1.

使用 Python 的 Gymnasium 進行強化學習

Double Q-learning

Image showing two Q-tables, Q1 and Q2, and each one is updated based on the other.

  • 降低高估偏差
  • 在 Q0 與 Q1 更新間交替
  • 兩個表共同促進學習
使用 Python 的 Gymnasium 進行強化學習

Frozen Lake 實作

env = gym.make('FrozenLake-v1', 
               is_slippery=False)

num_states = env.observation_space.n
n_actions = env.action_space.n
Q = [np.zeros((num_states, n_actions))] * 2


num_episodes = 1000
alpha = 0.5
gamma = 0.99

Image showing an agent navigating the Frozen Lake environment.

使用 Python 的 Gymnasium 進行強化學習

實作 update_q_tables()

def update_q_tables(state, action, reward, next_state):
    # Select a random Q-table index (0 or 1)
    i = np.random.randint(2)

    # Update the corresponding Q-table
    best_next_action = np.argmax(Q[i][next_state])

    Q[i][state, action] = (1 - alpha) * Q[i][state, action] 
                          + alpha * (reward + gamma * Q[1-i][next_state, best_next_action])

Image showing the update rule of Q1.

Image showing the update rule of Q2.

使用 Python 的 Gymnasium 進行強化學習

訓練

for episode in range(num_episodes):
    state, info = env.reset()
    terminated = False

    while not terminated:
        action = np.random.choice(n_actions)  
        next_state, reward, terminated, truncated,  info = env.step(action)
        update_q_tables(state, action, reward, next_state)
        state = next_state


final_Q = (Q[0] + Q[1])/2
# OR
final_Q = Q[0] + Q[1]
使用 Python 的 Gymnasium 進行強化學習

智能體的策略

policy = {state: np.argmax(final_Q[state]) 
          for state in range(num_states)}
print(policy)

{ 0: 1,  1: 0,  2: 0,  3: 0, 
  4: 1,  5: 0,  6: 1,  7: 0, 
  8: 2,  9: 1, 10: 1, 11: 0, 
 12: 0, 13: 2, 14: 2, 15: 0}

Image showing the policy learned by the agent, showing which action to perform in every state.

使用 Python 的 Gymnasium 進行強化學習

一起來練習吧！

使用 Python 的 Gymnasium 進行強化學習

Preparing Video For Download...