We are happy to announce the release of nnablaRL, a deep reinforcement learning (RL) library built on top of Neural Network Libraries.
Reinforcement learning is one of the cutting edge machine learning technology that achieves super human performance in the field of gaming, robotics, etc..
We hope that this new library, nnablaRL, helps RL experts and also non-RL experts using reinforcement learning algorithms easily among our nnabla ecosystem.
Features of nnablaRL is the following.
Friendly API
nnablaRL has friendly Python APIs which enables to start training with only 3 lines of python code.
import nnabla_rl
import nnabla_rl.algorithms as A
from nnabla_rl.utils.reproductions import build_atari_env
env = build_atari_env("BreakoutNoFrameskip-v4") # 1
dqn = A.DQN(env) # 2
dqn.train(env) # 3
You can also customize the algorithm’s hyper parameters easily. For example, you can change the batch size of training data as follows.
import nnabla_rl
import nnabla_rl.algorithms as A
from nnabla_rl.utils.reproductions import build_atari_env
env = build_atari_env("BreakoutNoFrameskip-v4")
config = A.DQNConfig(batch_size=100)
dqn = A.DQN(env, config=config)
dqn.train(env)
In addition to algorithm hyper parameters, you can also flexibly change the training component such as neural network models and model solvers. For details, see sample codes and API documents.
Many builtin algorithms
Most of famous/SOTA deep reinforcement learning algorithms, such as DQN, SAC, BCQ, GAIL, etc., is already implemented in nnablaRL. Implemented algorithms are carefully tested and evaluated. You can easily start training your agent using these verified implementations.
Please check the sample codes and document for detail usage of each algorithm.
You can find the list of implemented algorithms here.
Seamless switching of online and offline training
In reinforcement learning, there are two main training procedures, online and offline, to train the agent. Online training is a training procedure that executes both data collection and network update alternately. Conversely, offline training is a training procedure that updates the network using only existing data. With nnablaRL, you can switch these two training procedures seamlessly. For example, as shown below, you can easily train a robot’s controller online using simulated environment and finetune it offline with real robot dataset.
import nnabla_rl
import nnabla_rl.algorithms as A
simulator = get_simulator() # This is just an example. Assuming that simulator exists
dqn = A.DQN(simulator, config=config)
dqn.train_online(simulator)
real_data = get_real_data() # This is also an example. Assuming that you have real robot data
dqn.train_offline(real_data)
Getting started
You can find both notebook style interactive demos and raw python scripts as a sample code to get started. If you are unfamiliar with reinforcement learning, we recommend trying the notebook as a starting point. You can immediately launch and start training through google colaboratory! Check the list of notebooks here.
Development of nnablaRL has just started. We will continue adding new reinforcement learning algorithms and SOTA techniques to nnablaRL. Feedbacks, feature requests and contributions are welcome! Check the contribution guide for details.
Resources
- Documentation: https://nnabla-rl.readthedocs.io/
- Code: https://github.com/sony/nnabla-rl
- Python Package hosted on PYPI, install with:
pip install nnabla-rl
References
- [Mnih15] Mnih, Volodymir et al. “Human-level control through deep reinforcement learning”, Nature, 2015
- [Lillicrap16] Lillicrap, Timothy et al. “Continuous control with deep reinforcement learning”, 4th International Conference on Learning Representations (ICLR 2016), 2016
- [Haarnoja18] Haarnoja, Tuomas et al. “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, Thirty-fifth International Conference on Machine Learning (IMCL 2018), 2018
- [Vieillard20] Vieillard, Nino et al. “Munchausen Reinforcement Learning”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020
