[BUG] _Multi-GPU version is much slower than the single GPU

[Manyi-Yang]

Summary
I am confused that the Multi-GPU version is much slower than the single GPU.

Deepmd-kit version, installation way, input file, running commands, error log, etc.
How to install： conda install deepmd-kit=2.0.3=*cpu libdeepmd=2.0.3=*cpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org
GPU: NVIDIA Tesla V100 16Gb
In my system, one core has four GPU

Test results

In both cases, the same input file was used:
If with one GPU:
dp train --mpi-log=master input.json 1>> train.log 2>> train.log
the run speed is like this:

horovodrun -np 2 dp train --mpi-log=master input.json 1>> train.log 2>> train.log
the run speed is like this:

Another problem

when training the model with version 2.0.2/3, It took me more than 1 hour to load the data, but it only took me a few mins if with version 2.0.0.b2. why?

[wanghan-iapcm]

@shishaochen could you plz take a look? thanks

[njzjz]

I am confused that the Multi-GPU version is much slower than the single GPU.

How to install： conda install deepmd-kit=2.0.3=*cpu libdeepmd=2.0.3=*cpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org

Why did you install the CPU version?

[Manyi-Yang]

I am confused that the Multi-GPU version is much slower than the single GPU.
How to install： conda install deepmd-kit=2.0.3=*cpu libdeepmd=2.0.3=*cpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org

Why did you install the CPU version?
Sorry, Actually I installed like this:
conda install deepmd-kit=2.0.3=*gpu libdeepmd=2.0.3=*gpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org

[njzjz]

Ok, but could you confirm your CPU resources are enough?

…

On Nov 17, 2021, at 01:46, Manyi-Yang ***@***.***> wrote:  I am confused that the Multi-GPU version is much slower than the single GPU. How to install： conda install deepmd-kit=2.0.3=*cpu libdeepmd=2.0.3=*cpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org Why did you install the CPU version? Sorry, Actually I installed like this: conda install deepmd-kit=2.0.3=*gpu libdeepmd=2.0.3=*gpu lammps-dp=2.0.0 horovod -c https://conda.deepmodeling.org — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[njzjz]

As was discussed, the behavior was caused by a improper OMP_NUM_THREADS. We may add some documents to encourage some one to adjust it.

The second problem should be tracked in #1088.