External parameter clarification

[janEbert]

Hey, from both doc sites I have several questions regarding external parameters for ZeRO stage 3 usage.

When external parameters are involved in a model, is it possible that the code works even when they aren't registered? Would I get a warning? Or will my model fail silently? Or may it crash if my chosen parameters randomly partition the parameters the wrong way?

[samyam]

@janEbert in most cases if external parameters is not setup properly it will crash. If the parameter is partitioned when its about to be used, then the potential mismatch in shape will often cause it to crash.

However, there are cases where there can be silent failures. partitioned parameters in ZeRO-3 are represented by a scalar value but the scaler values are broadcastable, so if a parameter is used in element wise operations, it can fail silentl. For example:

Full parameter:
param.data = tensor of shape [1024,1024]

Current partitioned parameter representation
param.data = 1.0

Silent faliure case :
result = torch.rand([1024,1024]) + param (this will fail silently because 1.0 is broadcastable to [1024,1024])

We are thinking about what might be a good way to address this. One thing we have been thinking about is to use an odd shaped tensor to represent partitioned parameters, so that it will reduce the chances of silent failures.

Possible fix using an alternate partitioned parameter representation
param.data=[5,2,3] (some wierd shaped tensor)

So torch.rand([1024,1024]) + param will fail.

We will this to our TODO list, but very happy to get contributions.

[janEbert]

Hey @samyam, thank you for the detailed explanation! That really helps.
It sounds like the possibility of the program not failing is rather low.

I still have to dive into the codebase to understand what is happening in the background better. For now, your answer definitely helps; if I get the time I'm happy to look into this further.

@cli99 are you going to add this to the documentation?

[cli99]

Hi @janEbert, I am working on a PR as @samyam suggested. #981. Will also document this detail as well.

[janEbert]

All right, thanks! Looking forward to stumbling upon some errors when you're done. ;)

[janEbert]

I just read this while informing myself about ZeRO-Infinity (in ./docs/_tutorials/zero.md):

Registering external parameters

Deprecated: DeepSpeed version 0.3.15 introduced automatic external parameter registration and this step is no longer needed.

I'll close this, then. Thanks for all your support!

@cli99's PR still seems worthwhile due to this (in the code docs):

DeepSpeed version 0.3.15 includes automatic external parameter discovery and registration to support the most common cases. Parameters can still be manually registered if they cannot be automatically detected.