기본 DQN 알고리즘

Python으로 배우는 Deep Reinforcement Learning

Timothée Carayol

Principal Machine Learning Engineer, Komment

기본 DQN

 

 

  • 전체 DQN으로 가는 첫 단계
  • 구성 요소:
    • 범용 DRL 학습 루프
    • Q-네트워크
    • Q-러닝 원리
for episode in range(1000):
  state, info = env.reset()
  done = False
  while not done:
    # Action selection
    action = select_action(network, state)
    next_state, reward, terminated, truncated, _ = (
                                   env.step(action))
    done = terminated or truncated
    # Loss calculation
    loss = calculate_loss(network, state, action, 
                          next_state, reward, done)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    state = next_state
Python으로 배우는 Deep Reinforcement Learning

기본 DQN의 행동 선택

 

def select_action(q_network, state):

    # Feed state to network to obtain Q-values

    q_values = q_network(state)

    # Obtain index of action with highest Q-value
    action = torch.argmax(q_values).item()

    return action
  • 정책: 최고 Q-값의 행동 선택
    • $ a_t = {\arg\max}_a Q(S_t, a) $
    • 여기서는 행동 2, Q-값 0.12

 

 

Q-values: [-0.01, 0.08, 0.12, -0.07]

Action selected: 2, with q-value 0.12
Python으로 배우는 Deep Reinforcement Learning

기본 DQN 손실 함수

 

  • 행동-가치 함수는 벨만 방정식을 만족함
  • 아이디어: 양쪽의 차이(= TD-오차/벨만 오차) 최소화
  • 손실: 제곱 벨만 오차 사용

DQN 손실(제곱 벨만 오차): L(theta) = 벨만 오차의 제곱

 

벨만 방정식 Q(S,A) = reward + discount * max(Q(S_next,A))

벨만 오차 Q(S,A) - (reward + discount * max(Q(S_next,A)))

Python으로 배우는 Deep Reinforcement Learning

기본 DQN 손실 함수

def calculate_loss(
      q_network, state, action,
      next_state, reward, done):

    q_values = q_network(state)

    current_state_q_value = q_values[action]

    next_state_q_value = q_network(next_state).max()

    target_q_value = reward +
                gamma * next_state_q_value * (1-done)

    loss = nn.MSELoss()(
              current_state_q_value, target_q_value)

    return loss
  • 현재 상태 Q-값:

       $Q(s_t, a_t)$

  • 다음 상태 Q-값:

       $\max_a Q(s_{t+1}, a)$

  • 타깃 Q-값:

       $r_{t+1} + \gamma \max_a Q(s_{t+1}, a)$

  • DQN 손실:

$$\left(Q(s_t, a_t) - (r_{t+1} + \gamma \max_a Q(s_{t+1}, a)\right)^2$$

Python으로 배우는 Deep Reinforcement Learning

에피소드 요약

describe_episode(episode, reward, episode_reward, step)

| Episode    1 | Duration:   84 steps | Return: -871.38 | Crashed  |

| Episode    2 | Duration:   53 steps | Return: -452.68 | Crashed  |

| Episode    3 | Duration:   57 steps | Return: -414.22 | Crashed  |
| Episode    4 | Duration:   54 steps | Return: -475.09 | Crashed  |

| Episode    5 | Duration:   67 steps | Return: -532.31 | Crashed  |
| Episode    6 | Duration:   53 steps | Return: -407.00 | Crashed  |
| Episode    7 | Duration:   52 steps | Return: -380.45 | Crashed  |
| Episode    8 | Duration:   55 steps | Return: -380.75 | Crashed  |
| Episode    9 | Duration:   88 steps | Return: -688.68 | Crashed  |
| Episode   10 | Duration:   76 steps | Return: -338.06 | Crashed  |
Python으로 배우는 Deep Reinforcement Learning

Vamos praticar!

Python으로 배우는 Deep Reinforcement Learning

Preparing Video For Download...