Prioritized replay

[toslunar]

Prioritized replay (for DQN)

[muupan]

Travis CI failed due to style checks. Can you install hacking, apply flake8 and fix warnings?

[toslunar]

Fixed the style. Thanks.

[muupan]

Thanks for the fixes. Can you add tests for SumTree, PrioritizedBuffer and Prioritized(Episodic)ReplayBuffer?

[muupan]

Can you remove comment-outed code like this?

[muupan]

What is the purpose of self.count_used?

[toslunar]

Sorry. count_used was not used.

[muupan]

I tried adding this test case to tests/test_replay_buffer.py to check capacity handling and it failed, exceeding the specified capacity. It seems capacity parameter doesn't work. Can you fix it?

    def test_capacity(self):
        capacity = 10
        rbuf = replay_buffer.PrioritizedReplayBuffer(capacity)
        # Fill the buffer
        for _ in range(capacity):
            trans1 = dict(state=0, action=1, reward=2, next_state=3,
                          next_action=4, is_state_terminal=True)
            rbuf.append(**trans1)
        self.assertEqual(len(rbuf), capacity)

        # Add a new transition
        trans2 = dict(state=1, action=1, reward=2, next_state=3,
                      next_action=4, is_state_terminal=True)
        rbuf.append(**trans2)
        # The size should not change
        self.assertEqual(len(rbuf), capacity)

[muupan]

Can you rename it to TestPrioritizedReplayBuffer to avoid confusion?

[toslunar]

Fixed the issue on capacity.

The argument 'capacity' of PrioritizedBuffer was used to limit len(self.data). Now, it limits len(self) (= len(self.data) + len(self.data_inf)).

[muupan]

self.data[i] = self.data.pop()

This would raise an out-of-range error if i == n - 1 because self.data has only n-1 elements after pop.

[muupan]

The example script fails. Can you fix it?

$ python examples/gym/train_dqn_gym.py --prioritized-replay --episodic-replay
Output files are saved in dqn_out/20170316174313282734
INFO:gym.envs.registration:Making new env: Pendulum-v0
INFO:gym.envs.registration:Making new env: Pendulum-v0
DEBUG:chainerrl.agents.dqn:t:0 q:-0.3146612048149109 action_value:QuadraticActionValue greedy_actions:[[-0.89204156]] v:[[-0.3146612]]
...
DEBUG:chainerrl.agents.dqn:t:200 r:-0.00918397948453 a:[-0.41467309]
Saved the agent to dqn_out/20170316174313282734/200_except
Traceback (most recent call last):
  File "examples/gym/train_dqn_gym.py", line 179, in <module>
    main()
  File "examples/gym/train_dqn_gym.py", line 175, in main
    max_episode_len=timestep_limit)
  File "/home/fujita/drill/chainerrl/experiments/train_agent.py", line 124, in train_agent_with_evaluation
    successful_score=successful_score)
  File "/home/fujita/drill/chainerrl/experiments/train_agent.py", line 55, in train_agent
    agent.stop_episode_and_train(obs, r, done=done)
  File "/home/fujita/drill/chainerrl/agents/dqn.py", line 449, in stop_episode_and_train
    self.stop_episode()
  File "/home/fujita/drill/chainerrl/agents/dqn.py", line 456, in stop_episode
    self.replay_buffer.stop_current_episode()
  File "/home/fujita/drill/chainerrl/replay_buffer.py", line 232, in stop_current_episode
    self.episodic_memory.append(self.current_episode)
  File "/home/fujita/drill/chainerrl/misc/prioritized.py", line 18, in append
    if len(self) > self.capacity:
TypeError: unorderable types: int() > NoneType()

[coveralls]

Changes Unknown when pulling 4cec485 on toslunar:prioritized-replay into ** on pfnet:master**.

[coveralls]

Changes Unknown when pulling 620fb17 on toslunar:prioritized-replay into ** on pfnet:master**.

[coveralls]

Changes Unknown when pulling b6ca346 on toslunar:prioritized-replay into ** on pfnet:master**.

[coveralls]

Changes Unknown when pulling b6ca346 on toslunar:prioritized-replay into ** on pfnet:master**.

[muupan]

Thanks for the fixes! Now it looks good.