Spotlight

Support CUDA11.4 and remove CUDA10.2 support

From v1.30.0, we started supporting CUDA 11.4 extension.
Please find packages at https://pypi.org/project/nnabla-ext-cuda114.
OpenMPI 4.1.3 is included in multi-gpu docker images with CUDA 11.4 wheel packages, and can be found at https://hub.docker.com/r/nnabla/nnabla-ext-cuda-multi-gpu/tags.

Please note that CUDA 10.2 extension is no longer supported.

Introduce cuda/cudnn included wheel (CPU / GPU)

We have prepared wheel packages with all necessary CUDA and cuDNN libraries, for both Linux and Windows platform.
The default versions of the packages are CUDA 11.0.3 and cuDNN 8. The names of the packages have alllib affix.

Users can use these wheel packages directly on a GPU-enabled device without installing CUDA toolkit and cuDNN library. Note that a GPU driver that supports the default CUDA version of the packages is still necessary.

If you would like to enable multi-GPU execution with these wheel packages on Linux, you only need to install OpenMPI on a multi-GPU-enabled device. NCCL is not required as it is already packed in the packages.

pip install -U pip
pip install https://nnabla.org/whl/nnabla_ext_cuda_alllib/nnabla_ext_cuda_alllib-1.30.0-cp39-cp39-manylinux_2_17_x86_64.whl

Other wheels are listed in https://nnabla.org/install/#all-in-one_list .

Optimize TransposeCuda with cuTENSOR

We introduce cuTensor-based implementation to acceralate transpose in CUDA, especially in inner 2D transposition (ex. transpose(..., axis=(0, 1, 3, 2))).

As an example of actual use case, transpose used in channel last layout to channel first layout conversion (and vice versa) can now be processed faster than before.

Better error message in CUDNN.

Previous cudnn error messages did not point out what actually caused the error. This improvement shows the error code and navigates users to the CUDNN API guide to identify the possible causes of the error.
Redundant error message is suppressed so that the error message becomes more informative.

Before:

RuntimeError: target_specific error in cudnn_set_tensor_nd_descriptor_force_dim
/home/gitlab-runner/builds/jmdP2aBr/1/nnabla/builders/all/nnabla-ext-cuda/src/nbla/cuda/cudnn/cudnn.cpp:40
Failed `status == CUDNN_STATUS_SUCCESS`: NOT_SUPPORTED

After:

Caught exception "target_specific error in TestBody
nnabla-ext-cuda/src/nbla/cuda/test/test_nbla_cuda_utils.cpp:26
CUDNN_STATUS_ALLOC_FAILED occured in `FakeCudnnFunc()`. Please see CUDNN API documentation for the cause.

[Fairness] Prejudice Remover Regularizer (part1 / part2)

Prejudice Remover Regularizer has been added for fairness to mitigate data bias in machine learning.
In addition to tabular version, an image version is also available.
We also provide Colab tutorial that enables training with bias mitigation.

Name	Notebook	Task	Example
Prejudice Remover Regularizer		Mitigate the model bias with Prejudice Removal Technique
Prejudice Remover Regularizer for Images		Mitigate the model bias with Prejudice Removal Technique for Images

Available training scripts for CLIP

Training code for CLIP is now available on top of the inference script we previously published.
Users can distribute large mini-batches required for contrastive learning across multiple GPUs.

nnabla

Bugfix

• fix training attribute problem of module
• Fix QNN recorder zero check
• Fix QNN param exists check
• fix proto to info.proto in load.py
• disable dl_nccl_generate when nccl_disabled
• Update nvidia/cuda base image
• Set to use openib by default
• fix crash for coding error in concatenate
• CenterNet code cleanup
• [XAI] update eigencam notebook
• [XAI ]resolve conflict

Build

• Integrate flatc to converter
• Skip FFT/IFFT pytests for win32+CUDA114
• Update scipy I/F from 1.9.x
• allow selection of python to be used when running build_cpplib.bat
• Add benchmark for F.transpose()
• add empty targz and merge cuda wheel
• Fix CUDA_VERSION_MINOR to various condition
• remove flatc build from Dockerfile

Format Converter

• add shape() function and importer
• Implementation of onnx import of quantize resize scatter compress shape
• add onnx exporter groupnormalization
• pack flatc binary files for windows/linux/mac into python wheel
• update version dependency of file format converter

Examples

• [Fairness] Prejudice Remover Regularizer for images (part2)
• [Fairness] Rejection Option-based classification on image data(part2)
• Add conditional adm examples

Known Issues

• FFT/iFFT pytest cases fails on Win32+CUDA11.4 Cuda backend

NNabla-NAS

Bugfix

• Bug fixes of OFA(Once-for-All) and add tutorials for OFA
• Fixed missing regularizer arguments
• Bug fixes of OFA DynamicBN layer
• Add proper extra_repr to all OFA modules.
• Bug fixes and refactoring of OFA utilities
• Fix parameters not being inherited.
• Fixed OFA searcher to get correct net parameters with keys
• fixed jobs.sh
• Update imagenet.yaml
• f-string compatibility with Python version < 3.8
• Fix mismatching order of params of AvgPool

Build

• Fix documentation build
• Fix/doc build requirements

Core

• Add the functions to save checkpoint and restart from it
• Get regulizers out of the optimizers dictionary.
• Extract elastic-nn-related codes to common packages for better code reuse
• Support nnabla.no_grad etc.
• Implementation of mixed operations fast mode
• Add OFA-Xception search space
• Add OFA-ResNet50

Documentation

• [OFA] Update Model Zoo README with results on imagenet
• updated docs, readme and tutorials and gave timestamps to the config logs

Utility

• save arch.json when save-nnp option is specified.
• add image augmentation for csv dataloader
• add no_image_normalization param in executor when saving nnp
• Update DALI Pipeline to the latest version for the ImageNet dataset
• Enable save nnp function for OFA searcher
• Switch to hydra to handle configurations