[REQUEST] DS_BUILD_UTILS no longer does anything

[felker]

As far as I can tell, DS_BUILD_UTILS no longer appears in the DeepSpeed source code. There was one appearance as of v0.9.0, but it was removed sometime between that release and v0.10.0:

❯ git checkout v0.9.0
Previous HEAD position was 5fe9d610 Tensor parallelism for Mixture of Experts (#2074)
HEAD is now at 0b5252bb [CPU support] Optionally bind each rank to different cores on host (#2881)

❯ grep -nri "ds_build_utils" .
./op_builder/utils.py:10:    BUILD_VAR = "DS_BUILD_UTILS"
./docs/_tutorials/advanced-install.md:67:* `DS_BUILD_UTILS` builds various optimized utilities

❯ git checkout v0.10.0
Previous HEAD position was 0b5252bb [CPU support] Optionally bind each rank to different cores on host (#2881)
HEAD is now at f5c834a6 fix(cpu_accelerator): :bug: Convert LOCAL_SIZE to integer (#3971)

❯ grep -nri "ds_build_utils" .
./docs/_tutorials/advanced-install.md:67:* `DS_BUILD_UTILS` builds various optimized utilities

Can its mention be removed from https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops etc. docs?

[loadams]

Hi @felker - good observation, it was removed in this commit. So I believe we should also remove it from the docs. It could be useful to folks using older DeepSpeed versions, but we've had a few versions since. Thoughts, @jeffra?

[felker]

thanks for the confirmation @loadams. I was going to open another somewhat related issue, but I thought it might be good to pass it by you first:

Is it advisable for users to ever build DeepSpeed from the unstable / nightly master branch, or is it best to stick to tagged releases? In particular, I have been having trouble pre-installing the ops for commits since 0.10.3. I know that the devs recommend JITing most of the ops in general, but for our particular system's users, it is better for us to precompile.

Specifically, I need to disable Evoformer via DS_BUILD_EVOFORMER_ATTN=0 in order to build DeepSpeed from source on the latest master commits. Obviously the feature is very new, but I couldn't find any documentation nor relevant Issues or discussions on GitHub:

• https://github.com/microsoft/DeepSpeed/issues?q=evoformer
• https://github.com/microsoft/DeepSpeed/discussions?discussions_q=evoformer

Looking at your CI, it seems like the DS team also disables precompiling this feature for your builds:
https://github.com/microsoft/DeepSpeed/blob/master/.github/workflows/nv-pre-compile-ops.yml

Maybe this limitation can be documented somewhere?

Also, I believe that the recommended flags to speedup the build process, --global-option="build_ext" --global-option="-j8" can cause errors that do not occur when they are omitted (in our case, DS_BUILD_SPARSE_ATTN=0 DS_BUILD_OPS=1 DS_BUILD_AIO=1 pip install --verbose ., which works fine building 0.10.3 from source). This problem seems to be regardless of the value of -j parallel jobs on our system (32 core AMD Epyc):

 ERROR:  /soft/datascience/conda/2023-09-29/mconda3/compiler_compat/ld: build/temp.linux-x86_64-cpython-310/csrc/transformer/normalize_kernels.o: error adding symbols: no error

[loadams]

Hi @felker - thanks for the comments on that as well. As for the newer build featuers, like evoformer, you're right we are still working on documenting those. However, do you need to pre-compile ops in your case? I think most users just use the JIT compilation or will just compile the features they need, only needing to turn things off if they need to pre-compile potentially all ops via DS_BUILD_OPS=1 and then disable those that their system does not support. Is that your use case? Just trying to see what we should put in the docs.

And I believe I've seen another issue as well that had issues with the other speedup flags, I'll investigate that as well and post those results here, too.

Thanks for the feedback and bug spotting!

[felker]

Unfortunately we do need to pre-compile the ops.

But my real concern is this: should I avoid installing DeepSpeed from master and instead stick to official releases? Are unbuildable/un-JITable features regularly merged to master, such that one would need to manually disable them via appending flags to DS_BUILD_OPS=1 DS_BUILD_SPARSE_ATTN=0 ... ?

[loadams]

Not a problem, we know that sometimes users do need to do this. I would say we rarely add new un-JITable features, SPARSE_ATTN has known issues since torch 2 that we've still not had time to resolve, and EVOFORMER_ATTN was the most recent new feature I can think of. I did link to the docs for how to install Evoformer and use this in the PR to cleanup the DS_BUILD_UTILS linked above. The reason it isn't built in the pre-compile ops test is the time and space to install the requirements needed and that it is still somewhat under development.

But overall, if you don't need features/bugfixes that are introduced into master between releases, then you may be better off with just using the official releases. It just depends on what features you are using/how stable they are/etc. if that makes sense?