Errors with DeepMD GPU version

[Grace0987]

Summary

I installed DeepMD v 1.3.3 with the offline script "deepmd-kit-1.3.3-cuda10.0_gpu-Linux-x86_64.sh" .
Inorder to train a module, i used this command "dp train " on a GPU machine ,which started well with loading the
corresponding CUDA and tensorflow libraries . But the Job did crash with the below error:

[Grace0987]

Error :
E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(29440, 25), b.shape=(25, 50), m=29440, n=50, k=25
[[node filter_type_0/MatMul_1 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
[[l2_force_test/_59]]
(1) Internal: Blas GEMM launch failed : a.shape=(29440, 25), b.shape=(25, 50), m=29440, n=50, k=25
[[node filter_type_0/MatMul_1 (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
0 successful operations.
0 derived errors ignored.

[Grace0987]

So, i thought to try the next releases, and i got the same error in a couple of next releases. My last try was to install the latest release "deepmd-kit-2.0.0.b0-cuda10.1_gpu-Linux-x86_64.sh". This time, when i trained the same model as previous, the cublas error did not appear. Now , it is complaining about few parameters to be changed in input file.
name: A100-SXM4-40GB, pci bus id: 0000:47:00.0, compute capability: 8.0)
2021-05-27 03:06:07.971279: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
DEEPMD INFO ---Summary of DataSystem: training -----------------------------------------------
DEEPMD INFO found 3 system(s):
DEEPMD INFO system natoms bch_sz n_bch prob pbc
DEEPMD INFO ../data/data_0/ 192 1 80 0.250 T
DEEPMD INFO ../data/data_1/ 192 1 160 0.500 T
DEEPMD INFO ../data/data_2/ 192 1 80 0.250 T
DEEPMD INFO --------------------------------------------------------------------------------------
DEEPMD INFO ---Summary of DataSystem: validation -----------------------------------------------
DEEPMD INFO found 1 system(s):
DEEPMD INFO system natoms bch_sz n_bch prob pbc
DEEPMD INFO ../data/data_3 192 1 80 1.000 T
DEEPMD INFO --------------------------------------------------------------------------------------
DEEPMD INFO training without frame parameter

2021-05-27 03:06:08.659353: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2021-05-27 03:06:08.672916: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2245895000 Hz
cuda assert: DeePMD-kit: illegal nbor list sorting /home/conda/feedstock_root/build_artifacts/libdeepmd_1621486666421/work/source/lib/src/cuda/prod_env_mat.cu 509

The input works fine with CPU's.

Could you please suggest on the error ? is there a mismatch with deepmd and tensorflow versions ?

Thanks in Advance

[Grace0987]

I have started a discussion in the discussion page. Sorry for the trouble here

[amcadmus]

Have you tried v2.0.0.b1? v2.0.0.b0 is known to have bugs in the GPU implementation.

[Grace0987]

Have you tried v2.0.0.b1? v2.0.0.b0 is known to have bugs in the GPU implementation.

Hi,
I tried v2.0.0.b1 and v2.0.0.b2 (released 2 days ago), still no luck. I am ending up in the same error while running dp train ...json
Error: cuda assert: DeePMD-kit: illegal nbor list sorting

While running lmp exectuable from the above versions, I get a kernel error like :
Error : GPU:0 failed. Status: device kernel image is invalid?

I am running my tests on NVIDIA A100 GPU's

Could you please suggest us a version (may be earlier versions) with which you have successful results ?

Many Thanks.

[denghuilu]

Have you tried v2.0.0.b1? v2.0.0.b0 is known to have bugs in the GPU implementation.

Hi,
I tried v2.0.0.b1 and v2.0.0.b2 (released 2 days ago), still no luck. I am ending up in the same error while running dp train ...json
Error: cuda assert: DeePMD-kit: illegal nbor list sorting

While running lmp exectuable from the above versions, I get a kernel error like :
Error : GPU:0 failed. Status: device kernel image is invalid?

I am running my tests on NVIDIA A100 GPU's

Could you please suggest us a version (may be earlier versions) with which you have successful results ?

Many Thanks.

The CUDA version supported by A100 needs to be at least equal to 11.0. Could you check the CUDA version of the downloaded deepmd-kit installation package?

[Grace0987]

The cuda toolkit version is 10.1.243 . Let me try with another deepmd package that has CUDA 11 packages.
Thanks for the hint here.

[Ericwang6]

@Grace0987 I reported the same bug in 2.0 version (#675), but not in 1.3.3 version. Could you please briefly describe your training system, for example, material in condense phase or small molecules? If convenient, you can also upload part of your data so that developers (@iProzd @denghuilu ) are able to reproduce the case.