Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Commit bbe179e

Browse files
szhaRohit Kumar Srivastava
szha
authored and
Rohit Kumar Srivastava
committed
[PIP] add build variant for cuda 11.2 (#19764)
1 parent df60158 commit bbe179e

14 files changed

+288
-8
lines changed

‎cd/Jenkinsfile_cd_pipeline‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ pipeline {
3636

3737
parameters {
3838
// Release parameters
39-
string(defaultValue: "cpu,native,cu100,cu101,cu102,cu110", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
39+
string(defaultValue: "cpu,native,cu101,cu102,cu110,cu112", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
4040
booleanParam(defaultValue: false, description: 'Whether this is a release build or not', name: "RELEASE_BUILD")
4141
}
4242

‎cd/Jenkinsfile_release_job‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ pipeline {
4343
// any disruption caused by different COMMIT_ID values chaning the job parameter configuration on
4444
// Jenkins.
4545
string(defaultValue: "mxnet_lib/static", description: "Pipeline to build", name: "RELEASE_JOB_TYPE")
46-
string(defaultValue: "cpu,native,cu100,cu101,cu102,cu110", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
46+
string(defaultValue: "cpu,native,cu101,cu102,cu110,cu112", description: "Comma separated list of variants", name: "MXNET_VARIANTS")
4747
booleanParam(defaultValue: false, description: 'Whether this is a release build or not', name: "RELEASE_BUILD")
4848
}
4949

‎cd/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ MXNet aims to support a variety of frontends, e.g. Python, Java, Perl, R, etc. a
2525

2626
The CD process is driven by the [CD pipeline job](Jenkinsfile_cd_pipeline), which orchestrates the order in which the artifacts are delivered. For instance, first publish the libmxnet library before publishing the pip package. It does this by triggering the [release job](Jenkinsfile_release_job) with a specific set of parameters for each delivery channel. The release job executes the specific release pipeline for a delivery channel across all MXNet *variants*.
2727

28-
A variant is a specific environment or features for which MXNet is compiled. For instance CPU, GPU with CUDA v10.0, CUDA v9.0 with MKL-DNN support, etc.
28+
A variant is a specific environment or features for which MXNet is compiled. For instance CPU, GPU with CUDA v10.1, CUDA v10.2 with MKL-DNN support, etc.
2929

3030
Currently, below variants are supported. All of these variants except native have MKL-DNN backend enabled.
3131

@@ -35,6 +35,7 @@ Currently, below variants are supported. All of these variants except native hav
3535
* *cu101*: CUDA 10.1
3636
* *cu102*: CUDA 10.2
3737
* *cu110*: CUDA 11.0
38+
* *cu112*: CUDA 11.2
3839

3940
*For more on variants, see [here](https://github.com/apache/incubator-mxnet/issues/8671)*
4041

@@ -120,7 +121,7 @@ The "first mile" of the CD process is posting the mxnet binaries to the [artifac
120121

121122
##### Timeout
122123

123-
We shouldn't set global timeouts for the pipelines. Rather, the `step` being executed should be rapped with a `timeout` function (as in the pipeline example above). The `max_time` is a global variable set at the [release job](Jenkinsfile_release_job) level.
124+
We shouldn't set global timeouts for the pipelines. Rather, the `step` being executed should be rapped with a `timeout` function (as in the pipeline example above). The `max_time` is a global variable set at the [release job](Jenkinsfile_release_job) level.
124125

125126
##### Node of execution
126127

‎cd/utils/artifact_repository.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ If not set, derived through the value of sys.platform (https://docs.python.org/3
5353

5454
**Variant**
5555

56-
Manually configured through the --variant argument. The current variants are: cpu, native, cu92, cu100, cu101, cu102 and cu110.
56+
Manually configured through the --variant argument. The current variants are: cpu, native, cu101, cu102, cu110, cu112.
5757

5858
As long as the tool is being run from the MXNet code base, the runtime feature detection tool (https://github.com/larroy/mxnet/blob/dd432b7f241c9da2c96bcb877c2dc84e6a1f74d4/docs/api/python/libinfo/libinfo.md) can be used to detect whether the library has been compiled with MKL (library has MKL-DNN feature enabled) and/or CUDA support (compiled with CUDA feature enabled).
5959

‎cd/utils/mxnet_base_image.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ case ${mxnet_variant} in
4242
cu110*)
4343
echo "nvidia/cuda:11.0-cudnn8-runtime-ubuntu16.04"
4444
;;
45+
cu112*)
46+
echo "nvidia/cuda:11.2-cudnn8-runtime-ubuntu16.04"
47+
;;
4548
cpu)
4649
echo "ubuntu:16.04"
4750
;;

‎ci/docker/docker-compose.yml‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# We use the cache_from feature introduced in file form version 3.4 (released 2017-11-01)
19+
version: "3.4"
20+
21+
# For simplicity, only the centos7_cpu is commented. But the comments apply to
22+
# all other services as well.
23+
services:
24+
###################################################################################################
25+
# Dockerfile.build.centos7 based images used for building on CentOS7. On
26+
# CentOS7, we respectively test the oldest supported toolchain and dependency
27+
# versions
28+
###################################################################################################
29+
centos7_cpu:
30+
# The resulting image will be named build.centos7_cpu:latest and will be
31+
# pushed to the dockerhub user specified in the environment variable
32+
# ${DOCKER_CACHE_REGISTRY} (typicall "mxnetci") under this name
33+
image: ${DOCKER_CACHE_REGISTRY}/build.centos7_cpu:latest
34+
build:
35+
context: .
36+
dockerfile: Dockerfile.build.centos7
37+
# Use "base" target declared in Dockerfile.build.centos7 as "build.centos7_cpu:latest"
38+
target: base
39+
args:
40+
# BASE_IMAGE is used to dynamically specify the FROM image in Dockerfile.build.centos7
41+
BASE_IMAGE: centos:7
42+
cache_from:
43+
- ${DOCKER_CACHE_REGISTRY}/build.centos7_cpu:latest
44+
centos7_gpu_cu101:
45+
image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu101:latest
46+
build:
47+
context: .
48+
dockerfile: Dockerfile.build.centos7
49+
target: base
50+
args:
51+
BASE_IMAGE: nvidia/cuda:10.1-cudnn7-devel-centos7
52+
cache_from:
53+
- ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu101:latest
54+
centos7_gpu_cu102:
55+
image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu102:latest
56+
build:
57+
context: .
58+
dockerfile: Dockerfile.build.centos7
59+
target: base
60+
args:
61+
BASE_IMAGE: nvidia/cuda:10.2-cudnn7-devel-centos7
62+
cache_from:
63+
- ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu102:latest
64+
centos7_gpu_cu110:
65+
image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu110:latest
66+
build:
67+
context: .
68+
dockerfile: Dockerfile.build.centos7
69+
target: base
70+
args:
71+
BASE_IMAGE: nvidia/cuda:11.0-cudnn8-devel-centos7
72+
cache_from:
73+
- ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu110:latest
74+
centos7_gpu_cu112:
75+
image: ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu112:latest
76+
build:
77+
context: .
78+
dockerfile: Dockerfile.build.centos7
79+
target: base
80+
args:
81+
BASE_IMAGE: nvidia/cuda:11.2.0-cudnn8-devel-centos7
82+
cache_from:
83+
- ${DOCKER_CACHE_REGISTRY}/build.centos7_gpu_cu112:latest
84+
###################################################################################################
85+
# Dockerfile.build.ubuntu based images. On Ubuntu we test more recent
86+
# toolchain and dependency versions compared to CentOS7. We attempt to update
87+
# the Ubuntu base image every 6 months, following the Ubuntu release cycle,
88+
# and testing the dependencies in their version provided by the respective
89+
# Ubuntu release.
90+
###################################################################################################
91+
ubuntu_cpu:
92+
image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_cpu:latest
93+
build:
94+
context: .
95+
dockerfile: Dockerfile.build.ubuntu
96+
target: base
97+
args:
98+
BASE_IMAGE: ubuntu:20.04
99+
cache_from:
100+
- ${DOCKER_CACHE_REGISTRY}/build.ubuntu_cpu:latest
101+
ubuntu_tensorrt_cu111:
102+
image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_tensorrt_cu111:latest
103+
build:
104+
context: .
105+
dockerfile: Dockerfile.build.ubuntu
106+
target: gpu
107+
args:
108+
BASE_IMAGE: nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
109+
cache_from:
110+
- ${DOCKER_CACHE_REGISTRY}/build.ubuntu_tensorrt_cu111:latest
111+
ubuntu_gpu_cu111:
112+
image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_gpu_cu111:latest
113+
build:
114+
context: .
115+
dockerfile: Dockerfile.build.ubuntu
116+
target: gpu
117+
args:
118+
BASE_IMAGE: nvidia/cuda:11.1-cudnn8-devel-ubuntu20.04
119+
cache_from:
120+
- ${DOCKER_CACHE_REGISTRY}/build.ubuntu_gpu_cu111:latest
121+
###################################################################################################
122+
# Dockerfile.build.android based images used for testing cross-compilation for plain ARM
123+
###################################################################################################
124+
armv6:
125+
image: ${DOCKER_CACHE_REGISTRY}/build.armv6:latest
126+
build:
127+
context: .
128+
dockerfile: Dockerfile.build.arm
129+
target: armv6
130+
cache_from:
131+
- ${DOCKER_CACHE_REGISTRY}/build.armv6:latest
132+
armv7:
133+
image: ${DOCKER_CACHE_REGISTRY}/build.armv7:latest
134+
build:
135+
context: .
136+
dockerfile: Dockerfile.build.arm
137+
target: armv7
138+
cache_from:
139+
- ${DOCKER_CACHE_REGISTRY}/build.armv7:latest
140+
armv8:
141+
image: ${DOCKER_CACHE_REGISTRY}/build.armv8:latest
142+
build:
143+
context: .
144+
dockerfile: Dockerfile.build.arm
145+
target: armv8
146+
cache_from:
147+
- ${DOCKER_CACHE_REGISTRY}/build.armv8:latest
148+
###################################################################################################
149+
# Dockerfile.test.arm based images for testing ARM artefacts via QEMU
150+
###################################################################################################
151+
test.armv7:
152+
image: ${DOCKER_CACHE_REGISTRY}/test.armv7:latest
153+
build:
154+
context: .
155+
dockerfile: Dockerfile.test.arm
156+
args:
157+
BASE_IMAGE: arm32v7/ubuntu:20.04
158+
cache_from:
159+
- ${DOCKER_CACHE_REGISTRY}/test.armv7:latest
160+
test.armv8:
161+
image: ${DOCKER_CACHE_REGISTRY}/test.armv8:latest
162+
build:
163+
context: .
164+
dockerfile: Dockerfile.test.arm
165+
args:
166+
BASE_IMAGE: arm64v8/ubuntu:20.04
167+
cache_from:
168+
- ${DOCKER_CACHE_REGISTRY}/test.armv8:latest
169+
###################################################################################################
170+
# Dockerfile.build.android based images used for testing cross-compilation for Android
171+
###################################################################################################
172+
android_armv7:
173+
image: ${DOCKER_CACHE_REGISTRY}/build.android_armv7:latest
174+
build:
175+
context: .
176+
dockerfile: Dockerfile.build.android
177+
target: armv7
178+
cache_from:
179+
- ${DOCKER_CACHE_REGISTRY}/build.android_armv7:latest
180+
android_armv8:
181+
image: ${DOCKER_CACHE_REGISTRY}/build.android_armv8:latest
182+
build:
183+
context: .
184+
dockerfile: Dockerfile.build.android
185+
target: armv8
186+
cache_from:
187+
- ${DOCKER_CACHE_REGISTRY}/build.android_armv8:latest
188+
###################################################################################################
189+
# Dockerfile.publish.test based images used for testing binary artifacts on minimal systems.
190+
###################################################################################################
191+
publish.test.centos7_cpu:
192+
image: ${DOCKER_CACHE_REGISTRY}/publish.test.centos7_cpu:latest
193+
build:
194+
context: .
195+
dockerfile: Dockerfile.publish.test.centos7
196+
args:
197+
BASE_IMAGE: centos:7
198+
cache_from:
199+
- ${DOCKER_CACHE_REGISTRY}/publish.test.centos7_cpu:latest
200+
publish.test.centos7_gpu:
201+
image: ${DOCKER_CACHE_REGISTRY}/publish.test.centos7_gpu:latest
202+
build:
203+
context: .
204+
dockerfile: Dockerfile.publish.test.centos7
205+
args:
206+
BASE_IMAGE: nvidia/cuda:9.2-cudnn7-devel-centos7
207+
cache_from:
208+
- ${DOCKER_CACHE_REGISTRY}/publish.test.centos7_gpu:latest
209+
###################################################################################################
210+
# Miscellaneous containers
211+
###################################################################################################
212+
jetson:
213+
image: ${DOCKER_CACHE_REGISTRY}/build.jetson:latest
214+
build:
215+
context: .
216+
dockerfile: Dockerfile.build.jetson
217+
cache_from:
218+
- ${DOCKER_CACHE_REGISTRY}/build.jetson:latest
219+
ubuntu_cpu_jekyll:
220+
image: ${DOCKER_CACHE_REGISTRY}/build.ubuntu_cpu_jekyll:latest
221+
build:
222+
context: .
223+
dockerfile: Dockerfile.build.ubuntu_cpu_jekyll
224+
cache_from:
225+
- ${DOCKER_CACHE_REGISTRY}/build.ubuntu_cpu_jekyll:latest

‎tools/pip/doc/CPU_ADDITIONAL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
Prerequisites
1919
-------------
2020
This package supports Linux, Mac OSX, and Windows platforms. You may also want to check:
21+
- [mxnet-cu112](https://pypi.python.org/pypi/mxnet-cu112/) with CUDA-11.2 support.
2122
- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
2223
- [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
2324
- [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.

‎tools/pip/doc/CU101_ADDITIONAL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
Prerequisites
1919
-------------
2020
This package supports Linux and Windows platforms. You may also want to check:
21+
- [mxnet-cu112](https://pypi.python.org/pypi/mxnet-cu112/) with CUDA-11.2 support.
2122
- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
2223
- [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
2324
- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.0 support.

‎tools/pip/doc/CU102_ADDITIONAL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
Prerequisites
1919
-------------
2020
This package supports Linux and Windows platforms. You may also want to check:
21+
- [mxnet-cu112](https://pypi.python.org/pypi/mxnet-cu112/) with CUDA-11.2 support.
2122
- [mxnet-cu110](https://pypi.python.org/pypi/mxnet-cu110/) with CUDA-11.0 support.
2223
- [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
2324
- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.

‎tools/pip/doc/CU110_ADDITIONAL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
Prerequisites
1919
-------------
2020
This package supports Linux and Windows platforms. You may also want to check:
21+
- [mxnet-cu112](https://pypi.python.org/pypi/mxnet-cu112/) with CUDA-11.2 support.
2122
- [mxnet-cu102](https://pypi.python.org/pypi/mxnet-cu102/) with CUDA-10.2 support.
2223
- [mxnet-cu101](https://pypi.python.org/pypi/mxnet-cu101/) with CUDA-10.1 support.
2324
- [mxnet-cu100](https://pypi.python.org/pypi/mxnet-cu100/) with CUDA-10.0 support.

0 commit comments

Comments
 (0)