Support CUDA11.0 on CD

cd/Jenkinsfile_cd_pipeline

@@ -36,7 +36,7 @@ pipeline {
 
   parameters {
     // Release parameters
-    string(defaultValue: "cpu,native,cu92,cu100,cu101,cu102", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
+    string(defaultValue: "cpu,native,cu101,cu102,cu110", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
     booleanParam(defaultValue: false, description: 'Whether this is a release build or not', name: "RELEASE_BUILD")
   }
 

cd/Jenkinsfile_release_job

@@ -43,7 +43,7 @@ pipeline {
     // any disruption caused by different COMMIT_ID values chaning the job parameter configuration on
     // Jenkins.
     string(defaultValue: "mxnet_lib/static", description: "Pipeline to build", name: "RELEASE_JOB_TYPE")
-    string(defaultValue: "cpu,native,cu92,cu100,cu101,cu102", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
+    string(defaultValue: "cpu,native,cu101,cu102,cu110", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
     booleanParam(defaultValue: false, description: 'Whether this is a release build or not', name: "RELEASE_BUILD")
   }
 

cd/README.md

@@ -25,16 +25,15 @@ MXNet aims to support a variety of frontends, e.g. Python, Java, Perl, R, etc. a
 
 The CD process is driven by the [CD pipeline job](Jenkinsfile_cd_pipeline), which orchestrates the order in which the artifacts are delivered. For instance, first publish the libmxnet library before publishing the pip package. It does this by triggering the [release job](Jenkinsfile_release_job) with a specific set of parameters for each delivery channel. The release job executes the specific release pipeline for a delivery channel across all MXNet *variants*.
 
-A variant is a specific environment or features for which MXNet is compiled. For instance CPU, GPU with CUDA v10.0, CUDA v9.0 with MKL-DNN support, etc. 
+A variant is a specific environment or features for which MXNet is compiled. For instance CPU, GPU with CUDA v10.1, CUDA v10.2 with MKL-DNN support, etc. 
 
 Currently, below variants are supported. All of these variants except native have MKL-DNN backend enabled.
 
 * *cpu*: CPU
 * *native*: CPU without MKL-DNN
-* *cu92*: CUDA 9.2
-* *cu100*: CUDA 10
 * *cu101*: CUDA 10.1
 * *cu102*: CUDA 10.2
+* *cu110*: CUDA 11.0
 
 *For more on variants, see [here](https://github.com/apache/incubator-mxnet/issues/8671)*
 

cd/python/pypi/pypi_package.sh

@@ -18,7 +18,7 @@
 
 set -ex
 
-# variant = cpu, native, cu92, cu100, etc.
+# variant = cpu, native, cu101, cu102, etc.
 export mxnet_variant=${1:?"Please specify the mxnet variant"}
 
 # Due to this PR: https://github.com/apache/incubator-mxnet/pull/14899

cd/utils/artifact_repository.md

@@ -17,7 +17,7 @@
 
 # Artifact Repository - Pushing and Pulling libmxnet
 
-The artifact repository is an S3 bucket accessible only to restricted Jenkins nodes. It is used to store compiled MXNet artifacts that can be used by downstream CD pipelines to package the compiled libraries for different delivery channels (e.g. DockerHub, PyPI, Maven, etc.). The S3 object keys for the files being posted will be prefixed with the following distinguishing characteristics of the binary: branch, commit id, operating system, variant and dependency linking strategy (static or dynamic). For instance, s3://bucket/73b29fa90d3eac0b1fae403b7583fdd1529942dc/ubuntu16.04/cu92mkl/static/libmxnet.so
+The artifact repository is an S3 bucket accessible only to restricted Jenkins nodes. It is used to store compiled MXNet artifacts that can be used by downstream CD pipelines to package the compiled libraries for different delivery channels (e.g. DockerHub, PyPI, Maven, etc.). The S3 object keys for the files being posted will be prefixed with the following distinguishing characteristics of the binary: branch, commit id, operating system, variant and dependency linking strategy (static or dynamic). For instance, s3://bucket/73b29fa90d3eac0b1fae403b7583fdd1529942dc/ubuntu16.04/cu102mkl/static/libmxnet.so
 
 An MXNet artifact is defined as the following set of files:
 
@@ -53,13 +53,13 @@ If not set, derived through the value of sys.platform (https://docs.python.org/3
 
 **Variant**
 
-Manually configured through the --variant argument. The current variants are: cpu, native, cu92, cu100, cu101, cu102.
+Manually configured through the --variant argument. The current variants are: cpu, native, cu101, cu102, cu110.
 
 As long as the tool is being run from the MXNet code base, the runtime feature detection tool (https://github.com/larroy/mxnet/blob/dd432b7f241c9da2c96bcb877c2dc84e6a1f74d4/docs/api/python/libinfo/libinfo.md) can be used to detect whether the library has been compiled with MKL (library has MKL-DNN feature enabled) and/or CUDA support (compiled with CUDA feature enabled).
 
 If it has been compiled with CUDA support, the output of /usr/local/cuda/bin/nvcc --version can be mined for the exact CUDA version (eg. 8.0, 9.0, etc.).
 
-By knowing which features are enabled on the binary, and if necessary, which CUDA version is installed on the machine, the value for the variant argument can be calculated. Eg. if CUDA features are enabled, and nvcc reports cuda version 10, then the variant would be cu100. If neither MKL-DNN nor CUDA features are enabled, the variant would be native. 
+By knowing which features are enabled on the binary, and if necessary, which CUDA version is installed on the machine, the value for the variant argument can be calculated. Eg. if CUDA features are enabled, and nvcc reports cuda version 10.2, then the variant would be cu102. If neither MKL-DNN nor CUDA features are enabled, the variant would be native. 
 
 **Dependency Linking**
 

cd/utils/mxnet_base_image.sh

@@ -21,18 +21,15 @@
 mxnet_variant=${1:?"Please specify the mxnet variant as the first parameter"}
 
 case ${mxnet_variant} in
-    cu92*)
-    echo "nvidia/cuda:9.2-cudnn7-runtime-ubuntu16.04"
-    ;;
-    cu100*)
-    echo "nvidia/cuda:10.0-cudnn7-runtime-ubuntu16.04"
-    ;;
     cu101*)
     echo "nvidia/cuda:10.1-cudnn7-runtime-ubuntu16.04"
     ;;
     cu102*)
     echo "nvidia/cuda:10.2-cudnn7-runtime-ubuntu16.04"
     ;;
+    cu110*)
+    echo "nvidia/cuda:11.0-cudnn8-runtime-ubuntu16.04"
+    ;;
     cpu)
     echo "ubuntu:16.04"
     ;;

cd/utils/test_artifact_repository.py

@@ -140,13 +140,13 @@ def test_get_cuda_version(self, mock):
         Tests correct cuda version with the right format is returned
         :return:
         """
-        mock.return_value = b'Cuda compilation tools, release 10.0, V10.0.130'
+        mock.return_value = b'Cuda compilation tools, release 10.2, V10.2.130'
         cuda_version = get_cuda_version()
-        self.assertEqual(cuda_version, '100')
+        self.assertEqual(cuda_version, '102')
 
-        mock.return_value = b'Cuda compilation tools, release 9.2, V9.2.148'
+        mock.return_value = b'Cuda compilation tools, release 11.0, V11.0.148'
         cuda_version = get_cuda_version()
-        self.assertEqual(cuda_version, '92')
+        self.assertEqual(cuda_version, '110')
 
     @patch('artifact_repository.check_output')
     def test_get_cuda_version_not_found(self, mock):
@@ -178,11 +178,11 @@ def test_probe_variant_cpu(self, mock_features):
     @patch('artifact_repository.get_cuda_version')
     def test_probe_variant_cuda(self, mock_cuda_version, mock_features):
         """
-        Tests 'cu100' is returned if MKLDNN is OFF and CUDA is ON and CUDA version is 10.0
+        Tests 'cu102' is returned if MKLDNN is OFF and CUDA is ON and CUDA version is 10.2
         """
         mock_features.return_value = {'MKLDNN': True, 'CUDA': True}
-        mock_cuda_version.return_value = '100'
-        self.assertEqual(probe_mxnet_variant('libmxnet.so'), 'cu100')
+        mock_cuda_version.return_value = '102'
+        self.assertEqual(probe_mxnet_variant('libmxnet.so'), 'cu102')
 
     @patch('artifact_repository.get_libmxnet_features')
     def test_probe_variant_cuda_returns_none_on_no_features(self, mock_features):

ci/docker/docker-compose.yml

@@ -41,26 +41,6 @@ services:
         BASE_IMAGE: centos:7
       cache_from:
         - ${DOCKER_CACHE_REGISTRY}/build.centos7_cpu:latest
-  centos7_gpu_cu92:
-    image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu92:latest
-    build:
-      context: .
-      dockerfile: Dockerfile.build.centos7
-      target: base
-      args:
-        BASE_IMAGE: nvidia/cuda:9.2-cudnn7-devel-centos7
-      cache_from:
-        - ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu92:latest
-  centos7_gpu_cu100:
-    image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu100:latest
-    build:
-      context: .
-      dockerfile: Dockerfile.build.centos7
-      target: base
-      args:
-        BASE_IMAGE: nvidia/cuda:10.0-cudnn7-devel-centos7
-      cache_from:
-        - ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu100:latest
   centos7_gpu_cu101:
     image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu101:latest
     build:
@@ -81,6 +61,16 @@ services:
         BASE_IMAGE: nvidia/cuda:10.2-cudnn7-devel-centos7
       cache_from:
         - ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu102:latest
+  centos7_gpu_cu110:
+    image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu110:latest
+    build:
+      context: .
+      dockerfile: Dockerfile.build.centos7
+      target: base
+      args:
+        BASE_IMAGE: nvidia/cuda:11.0-cudnn8-devel-centos7
+      cache_from:
+        - ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu110:latest
   ###################################################################################################
   # Dockerfile.build.ubuntu based images. On Ubuntu we test more recent
   # toolchain and dependency versions compared to CentOS7. We attempt to update
@@ -118,6 +108,16 @@ services:
         BASE_IMAGE: nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04
       cache_from:
         - ${DOCKER_CACHE_REGISTRY}/build.ubuntu_gpu_cu102:latest
+  ubuntu_gpu_cu110:
+    image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_gpu_cu110:latest
+    build:
+      context: .
+      dockerfile: Dockerfile.build.ubuntu
+      target: gpu
+      args:
+        BASE_IMAGE: nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04
+      cache_from:
+        - ${DOCKER_CACHE_REGISTRY}/build.ubuntu_gpu_cu110:latest
   ubuntu_build_cuda:
     image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_build_cuda:latest
     build:

ci/docker/runtime_functions.sh

@@ -101,7 +101,7 @@ gather_licenses() {
 
 # Compiles the dynamic mxnet library
 # Parameters:
-# $1 -> mxnet_variant: the mxnet variant to build, e.g. cpu, native, cu100, cu92, etc.
+# $1 -> mxnet_variant: the mxnet variant to build, e.g. cpu, native, cu101, cu102, etc.
 build_dynamic_libmxnet() {
     set -ex
 
@@ -1344,11 +1344,11 @@ build_static_python_cpu() {
     popd
 }
 
-build_static_python_cu92() {
+build_static_python_cu102() {
     set -ex
     pushd .
-    export mxnet_variant=cu92
-    source /opt/rh/devtoolset-7/enable
+    export mxnet_variant=cu102
+    source /opt/rh/devtoolset-8/enable
     source /opt/rh/rh-python36/enable
     # Opt in to newer GCC C++ ABI. devtoolset defaults to ABI Version 2.
     export CXXFLAGS="-fabi-version=11 -fabi-compat-version=7"

ci/jenkins/Jenkins_steps.groovy

@@ -305,7 +305,7 @@ def compile_centos7_gpu(lib_name) {
         ws('workspace/build-centos7-gpu') {
           timeout(time: max_time, unit: 'MINUTES') {
             utils.init_git()
-            utils.docker_run('centos7_gpu_cu92', 'build_centos7_gpu', false)
+            utils.docker_run('centos7_gpu_cu102', 'build_centos7_gpu', false)
             utils.pack_lib(lib_name, mx_lib)
           }
         }
@@ -634,7 +634,7 @@ def compile_static_python_gpu() {
         ws('workspace/ut-publish-python-gpu') {
           timeout(time: max_time, unit: 'MINUTES') {
             utils.init_git()
-            utils.docker_run('centos7_gpu_cu92', 'build_static_python_cu92')
+            utils.docker_run('centos7_gpu_cu102', 'build_static_python_cu102')
           }
         }
     }
@@ -911,7 +911,7 @@ def test_centos7_python3_gpu(lib_name) {
           timeout(time: max_time, unit: 'MINUTES') {
             try {
               utils.unpack_and_init(lib_name, mx_lib)
-              utils.docker_run('centos7_gpu_cu92', 'unittest_centos7_gpu', true)
+              utils.docker_run('centos7_gpu_cu102', 'unittest_centos7_gpu', true)
               utils.publish_test_coverage()
             } finally {
               utils.collect_test_results_unix('tests_gpu.xml', 'tests_python3_centos7_gpu.xml')

tools/pip/doc/CPU_ADDITIONAL.md

@@ -18,10 +18,9 @@
 Prerequisites
 -------------
 This package supports Linux, Mac OSX, and Windows platforms. You may also want to check:
+- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
 - [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
 - [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
-- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.
-- [mxnet-cu92](https://pypi.python.org/pypi/mxnet-cu92/) with CUDA-9.2 support.
 - [mxnet](https://pypi.python.org/pypi/mxnet/).
 - [mxnet-native](https://pypi.python.org/pypi/mxnet-native/) CPU variant without MKLDNN.
 
@@ -33,7 +32,7 @@ a GPL library and MXNet part of the Apache Software Foundation, MXNet must not
 redistribute `libquadmath.so.0` as part of the Pypi package and users must
 manually install it.
 
-To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master/install/index.html) for instructions on building from source.
+To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master) for instructions on building from source.
 
 Installation
 ------------

tools/pip/doc/CU101_ADDITIONAL.md

@@ -18,10 +18,8 @@
 Prerequisites
 -------------
 This package supports Linux and Windows platforms. You may also want to check:
+- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
 - [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
-- [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
-- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.
-- [mxnet-cu92](https://pypi.python.org/pypi/mxnet-cu92/) with CUDA-9.2 support.
 - [mxnet](https://pypi.python.org/pypi/mxnet/).
 - [mxnet-native](https://pypi.python.org/pypi/mxnet-native/) CPU variant without MKLDNN.
 
@@ -35,7 +33,7 @@ a GPL library and MXNet part of the Apache Software Foundation, MXNet must not
 redistribute `libquadmath.so.0` as part of the Pypi package and users must
 manually install it.
 
-To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.incubator.apache.org/versions/master/install/index.html) for instructions on building from source.
+To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master) for instructions on building from source.
 
 Installation
 ------------

tools/pip/doc/CU102_ADDITIONAL.md

@@ -18,10 +18,8 @@
 Prerequisites
 -------------
 This package supports Linux and Windows platforms. You may also want to check:
-- [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
+- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
 - [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
-- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.
-- [mxnet-cu92](https://pypi.python.org/pypi/mxnet-cu92/) with CUDA-9.2 support.
 - [mxnet](https://pypi.python.org/pypi/mxnet/).
 - [mxnet-native](https://pypi.python.org/pypi/mxnet-native/) CPU variant without MKLDNN.
 
@@ -35,7 +33,7 @@ a GPL library and MXNet part of the Apache Software Foundation, MXNet must not
 redistribute `libquadmath.so.0` as part of the Pypi package and users must
 manually install it.
 
-To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.incubator.apache.org/versions/master/install/index.html) for instructions on building from source.
+To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master) for instructions on building from source.
 
 Installation
 ------------

tools/pip/doc/CU92_ADDITIONAL.md → tools/pip/doc/CU110_ADDITIONAL.md

@@ -20,7 +20,6 @@ Prerequisites
 This package supports Linux and Windows platforms. You may also want to check:
 - [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
 - [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
-- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.
 - [mxnet](https://pypi.python.org/pypi/mxnet/).
 - [mxnet-native](https://pypi.python.org/pypi/mxnet-native/) CPU variant without MKLDNN.
 
@@ -34,11 +33,11 @@ a GPL library and MXNet part of the Apache Software Foundation, MXNet must not
 redistribute `libquadmath.so.0` as part of the Pypi package and users must
 manually install it.
 
-To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master/install/index.html) for instructions on building from source.
+To install for other platforms (e.g. Windows, Raspberry Pi/ARM) or other versions, check [Installing MXNet](https://mxnet.apache.org/versions/master) for instructions on building from source.
 
 Installation
 ------------
 To install:
 ```bash
-pip install mxnet-cu92
+pip install mxnet-cu110
 ```