Q-learning

使用 Python 的 Gymnasium 進行強化學習

Fouad Trad

Machine Learning Engineer

Q-learning 入門

代表 quality learning（品質學習）
無模型技巧
透過互動學得最佳 Q-table

圖示 Q-learning 流程：初始化 Q-table、選擇動作、從環境獲得回饋、更新表格。代理持續重複直到經過若干回合後收斂。

Q-learning 與 SARSA 比較

SARSA

顯示 SARSA 更新規則數學式的圖片。

依實際採取的動作更新
內循環（on-policy）學習者

Q-learning

顯示 Q-learning 更新規則數學式的圖片。

更新不依賴已採取的動作
外循環（off-policy）學習者

Q-learning 實作

env = gym.make("FrozenLake", is_slippery=True)


num_episodes = 1000
alpha = 0.1
gamma = 1


num_states, num_actions = env.observation_space.n, env.action_space.n
Q = np.zeros((num_states, num_actions))


reward_per_random_episode = []

Q-learning 實作

for episode in range(num_episodes):
    state, info = env.reset()
    terminated = False
    episode_reward = 0

    while not terminated:

        # Random action selection
        action = env.action_space.sample()

        # Take action and observe new state and reward
        new_state, reward, terminated, truncated, info = env.step(action)

        # Update Q-table
        update_q_table(state, action, new_state)

        episode_reward += reward
        state = new_state

    reward_per_random_episode.append(episode_reward)

Q-learning 更新

顯示 Q-learning 更新規則數學式的圖片。

def update_q_table(state, action, reward, new_state):

  old_value = Q[state, action]

  next_max = max(Q[new_state])

  Q[state, action] = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)

使用策略

reward_per_learned_episode = []

policy = get_policy()

for episode in range(num_episodes):
    state, info = env.reset()
    terminated = False
    episode_reward = 0
    while not terminated:
        # Select the best action based on learned Q-table
        action = policy[state]
        # Take action and observe new state
        new_state, reward, terminated, truncated, info = env.step(action)
        state = new_state

        episode_reward += reward

    reward_per_learned_episode.append(episode_reward)

Q-learning 評估

import numpy as np
import matplotlib.pyplot as plt


avg_random_reward = np.mean(reward_per_random_episode)
avg_learned_reward = np.mean(reward_per_learned_episode)


plt.bar(['Random Policy', 'Learned Policy'],
        [avg_random_reward, avg_learned_reward],
        color=['blue', 'green'])

plt.title('Average Reward per Episode')
plt.ylabel('Average Reward')
plt.show()

長條圖顯示，學得的策略回報遠高於隨機策略（約 0.26 對 0.01）。

一起來練習吧！

使用 Python 的 Gymnasium 進行強化學習