[BUG]: colossalai check -i failed using pip from https://www.colossalai.org/download/

[better629]

🐛 Describe the bug

install colossallai from pip install colossalai==0.2.0+torch1.12cu11.3 -f https://release.colossalai.org successed, and import successed.

But colossalai check -i failed, and colossalai run xx also failed.

logs:

(colos) [root@xxx hybrid_parallel]# colossalai check -i
Using /root/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Traceback (most recent call last):
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/kernel/__init__.py", line 4, in <module>
    from colossalai._C import fused_optim
ImportError: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/_C/fused_optim.cpython-39-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/colos/bin/colossalai", line 5, in <module>
    from colossalai.cli import cli
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/__init__.py", line 1, in <module>
    from .initialize import (
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/initialize.py", line 23, in <module>
    from colossalai.engine.schedule import NonPipelineSchedule, PipelineSchedule, InterleavedPipelineSchedule, get_tensor_shape
    fused_optim = FusedOptimBuilder().load()
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/kernel/op_builder/builder.py", line 77, in load
    op_module = load(name=self.name,
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1202, in load
  File "/root/anaconda3/envs/colos/lib/python3.9/site-packages/torch/utils/_cpp_extension_versioner.py", line 15, in hash_source_files
    with open(filename) as file:
FileNotFoundError: [Errno 2] No such file or directory: '/root/anaconda3/envs/colos/lib/python3.9/site-packages/colossalai/kernel/cuda_native/csrc/colossal_C_frontend.cpp'

(colos) [root@xxx hybrid_parallel]# strings /lib64/libm.so.6 | grep GLIBC
GLIBC_2.2.5
GLIBC_2.4
GLIBC_2.15
GLIBC_PRIVATE
GLIBC_2.15

So, we need to install glibc2.19, ref to https://blog.csdn.net/m0_37201243/article/details/123641552
After installed glibc2.19, colossalai check -i successed.

#### Installation Report ####

Colossal-AI version: 0.2.0
----------------------------
PyTorch Version: 1.12.0
PyTorch Version required by Colossal-AI: 1.12
PyTorch version match: ✓
----------------------------
System CUDA Version: 11.8
CUDA Version required by PyTorch: 11.3
CUDA Version required by Colossal-AI: 11.3
CUDA Version Match: x
----------------------------
CUDA Extension: ✓

but run hybrid_parallel still failed,

(colos) [colos@xxx hybrid_parallel]# colossalai run --nproc_per_node 2 train.py --config config.py
Error: failed to run torchrun --nproc_per_node=2 --nnodes=1 --node_rank=0 --rdzv_backend=c10d --rdzv_endpoint=127.0.0.1:29500 --rdzv_id=colossalai-default-job train.py --config config.py on 127.0.0.1

Environment

NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8
conda env has installed cudatoolkit = 11.3

[FrankLeeeee]

That's kind of strange. The good news is that we have released a new version to pypi, where you can install colossalai directly via pip install colossalai. With this, the CUDA extension is only built when you need it. Give it a try.

[haofanwang]

I previously met the same problem as you via pip install colossalai==0.2.0+torch1.12cu11.3 -f https://release.colossalai.org. But I didn't install glibc2.19. Instead, I re-install Pytorch to match the cuda version. In my case, cuda is 11.6.

You can try pip install colossalai first. If it still doesn't work, degrade your cuda may be a choice.

[better629]

@FrankLeeeee @haofanwang

glibc problem may be caused by the redhat system version, it's not important.

The pytorch version is re-install to match cudatoolkit version in conda env with version=11.3.

I have tried pip install colossalai with no-cache, but still Error: failed to run torchrun. How can I get more detail log? it's hard to find it out.

The host machine is public, if conda env can't meet, it's not convenient to use.

[haofanwang]

Have you tried to degrade System CUDA Version?

[better629]

Have you tried to degrade System CUDA Version?

Can try, but the host machine is public. The operation may affects other people.

Why conda env installed cudatoolkit==11.3 can't work ?

[better629]

Have you tried to degrade System CUDA Version?

How can I get more detail log of colossalai run, just failed to run torchrun message is brief.

[better629]

System CUDA Version: 11.8
CUDA Version required by PyTorch: 11.3
CUDA Version required by Colossal-AI: 11.3
CUDA Version Match: x

@haofanwang
You installed 11.6, Do u above three version are all 11.6 and Matched？

[FrankLeeeee]

Can you try this?

torchrun --nproc_per_node 2 --standalone train.py --config config.py

[FrankLeeeee]

There are some possible reasons:

1. torchrun is not in your PATH so colossalai cannot find it
2. torchrun is okay, but somehow colossalai failed for reasons I am not aware yet.

[better629]

The CentOS Linux release 7.9 & Ubuntu 18.04 have ImportError: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found`，maybe you can update support system list before install.

update env to

------------ Environment ------------
Colossal-AI version: 0.2.0
PyTorch version: 1.12.0
CUDA version: 11.8
CUDA version required by PyTorch: 11.6

# cuda version may not the problem, guess `torchrun xx` will run successfully also in cuda = 11.3.

torchrun --nproc_per_node 2 --standalone train.py --config config.py # need to change config.py depend on your card num run successfully, but
colossalai run --nproc_per_node 4 train.py --config config.py run failed.

[better629]

The CentOS Linux release 7.9 & Ubuntu 18.04 have ImportError: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found`，maybe you can update support system list before install.

update env to

------------ Environment ------------
Colossal-AI version: 0.2.0
PyTorch version: 1.12.0
CUDA version: 11.8
CUDA version required by PyTorch: 11.6

# cuda version may not the problem, guess `torchrun xx` will run successfully also in cuda = 11.3.

torchrun --nproc_per_node 2 --standalone train.py --config config.py # need to change config.py depend on your card num run successfully, but colossalai run --nproc_per_node 4 train.py --config config.py run failed.

@FrankLeeeee any suggestions?

[FrankLeeeee]

This might be the DNS resolution of "127.0.0.1", this issue has been fixed in #2465 . This fix is not released to PyPI yet. You can try to install the latest codebase with the following commands:

git clone https://github.com/hpcaitech/ColossalAI.git
cd ColossalAI
pip install -v .

Do let me know if the issue persists.

[better629]

This might be the DNS resolution of "127.0.0.1", this issue has been fixed in #2465 . This fix is not released to PyPI yet. You can try to install the latest codebase with the following commands:

git clone https://github.com/hpcaitech/ColossalAI.git
cd ColossalAI
pip install -v .

Do let me know if the issue persists.

Yes, it works now if installed from source code. Thx.

[binmakeswell]

We have updated a lot. This issue was closed due to inactivity. Thanks.
https://github.com/hpcaitech/ColossalAI#Installation