Batch PPO Implementation

[ljvmiranda921]

Refactor branch: https://github.com/ljvmiranda921/chainerrl/pull/13

Description

This PR is built on top of @iory 's A2C implementation in #149. It provides a batch/parallel implementation of Proximal Policy Optimization. I'm using #149's VecEnv environment to achieve this task. Here are the main changes:

• Add batch implementation to PPO via the batch_act() and batch_observe() methods
• Add chainerrl.experiments.train_agent_batch() Status: In progress
• Add new tests for batch PPO Status: In progress

Changes in data structure

Previously, the computation resides in the self.memory and self.last_episode attributes of PPO. Now we're also using self.batch_memory to handle this task. During a batch run, the type signature looks like:

# Type signature for self.batch_memory during batch run
batch_memory :: [[dict], [dict], [dict]]
where len(batch_memory) == batch_env.num_envs

The same goes for self.last_episode:

# Type signature for self.last_episode
last_episode :: [[dict], [dict], [dict]]
where len(last_episode) == batch_env.num_envs

New methods: batch_act() and batch_observe()

These methods are supposed to handle all batch computations during the run. The method batch_act(batch_obs) performs a set of actions given a set of observations, while batch_observe(obs, r, done, info) updates the model. A simple way to use them can be seen below:

t = 0; steps = 100
# o_0, r_0 : Init observation and reward
obs = batch_env.reset()
r = np.zeros(num_process, dtype='f')
# Initialize episode reward
episode_r = np.zeros(num_process, dtype='f')

while t < steps:
    # a_t : First action
    action = agent.batch_act(obs)
    # o_{t+1}, r_{t+1} : Get observation and reward
    obs, r, done, info = batch_env.step(action)
    # Train model
    agent.batch_observe(obs, r, done, info)
    # Update counters
    t += 1
    update_or_reset_reward(episode_r, done, info)

This assumes that the environment in VectorEnv sends a reset signal in the form of a dictionary entry in info.

[muupan]

Each env in VectorEnv should be assigned different random seeds. See train_a3c_gym.py for how to assign different random seeds.

[muupan]

t should be the total number of transitions experienced so far. So, it should increase by num_envs, not by 1. By doing so, we can keep the other hyperparameters and only change num_envs to trade cpus with computation time.