We have released Neural Network Libraries v1.27.0!
Along with our update, NNabla NAS has also been updated to v0.12.0. Furthermore, nnabla-browser, a tool for visualizing Neural Network Libraries, has been released! We will shortly upload a blog post introducing each of these updates!
We have implemented LAMB, a solver designed for layerwise adaptive large batch optimization. LAMB has been used in a number of benchmark tasks that require large batch sizes, such as BERT or ResNet-50 training. We have corrected definition of SGDW, and updated the Solver base class design to support fusion of weight decay operation into update operation.
We have implemented batch Cholesky function. We have referred to this paper for implementation details. Note that, while cholesky decomposition requires only the upper/lower part of the matrix, the backward of this function returns symmetric matrix for convenience.
We have replaced the sort library in SortCUDA from Thrust to CUB, achieving about 25 times better optimization.
We have implemented colab interactive demo for Eigen-CAM, which computes and visualizes the principle components of the learned features/representations from the convolutional layers.