We have released Neural Network Libraries v1.35.0!
Along with this update, nnabla-rl has also been updated to v0.13.0!
Please see “Spotlight” for important changes.

Spotlight

Lion optimizer

We’ve added the new optimizer Lion, published in February, as a Solver! This optimizer has excellent memory efficiency since it only holds momentum as a state, and it has been reported to achieve reduced training time and high performance in several experiments. Please give it a try. For detailed technical information, refer to the following paper.

Xiangning Chen, et al. Symbolic Discovery of Optimization Algorithms. arXiv 2023.

prune the constant branches

The nnabla-converter supports various model conversions, but the tflite converter has significant improvements in this release!

# How to install nnabla-converter
pip install nnabla nnabla-converter

# convert nnp to tflite
nnabla_cli convert -b 1 input.nnp output.tflite

# convert nnp to tensorflow, then remove redundant operators on tensorflow model
nnabla_cli convert input.nnp output.pb
nnabla_cli optimize output.pb improved.pb

1. Precompute the constant branches
   Andoid GPU delegates for TFLite don’t accept the operators with 2 constant inputs. We improve the tflite converter to eliminate those branches by precomputation.

2. Remove Split and Slice operators if the input shape and output shape are the same
   Until previous versions, the converted TFLite model contained a few redundant operators. By eliminating these operators, the model is more compact and has better performance.

3. Replace operators for quantized model conversion
   The different backends (or versions) have different support statuses, and the quantization model operators are a subset of TFLite operators. The quantized TFLite supports the core set operators, but some operators are not supported. So, these unsupported operators are replaced with supported operators.

4. Support TRANSPOSE_CONV operator for quantized model
   TRANSPOSE_CONV seems to be supported on the Android GPU delegate for TFLite, so we optionally support this operator in the tflite converter.

5. Improve TensorFlow to TensorFlow optimization to remove redundant Transpose
   The functionality to remove redundant operators with the “optimize” subcommand has been supported, and now it also supports TensorFlow v2.x.

nnabla

Build

• Use vs2019 default path (CPU / GPU)
• Use user’s pip cache in build-with-docker
• change aarch64 docker image base
• Enable FFT/IFFT pytests run on win32 when CUDA>=11.4
• change protobuf3.19.4 to protobuf3.19.6 (CPU / GPU)
• fix stft test error with fixing librosa version

Document

• update windows documents
• Update nnabla-doc build requirements.txt

Format Converter

• Add 8 operators to ONNX importer

BugFix

• Fix wheel too big problem
• Fix FFT/iFFT cufftXtMakePlanMany usage
• Fix to run tests for CUDA functions and solvers
• Fix the generated images become black in DPM examples

Examples

• [Fairness] Massage Data (Relabeling) Algorithm Part-2

nnabla-rl

BugFix

• Fix colab notebook
• Add pygame installation for colab demos
• Fix float max/min value range of gaussian explorer

Algorithm

• Implement SACD
• Add decision transformer

Utility

• Add progress bar hook
• Add accepted shape decorator

Document

• Add docformatter