Add "zero_allow_untested_optimizer" option in conf file

[CalogeroZarbo]

Dear all,
I've added the flag, in the same flavor as all the other flags have been addressed, by changing the DeepSpeedConfig class. Also, the assertion has been added to raise an error message if the flag is not set, but the user wants to use the custom optimizer. If both zero_allow_untested_optimizer and zero_optimization are set to true a warning message appears and the computation starts normally.

[msftclas]

All CLA requirements met.

[ShadenSmith]

Thanks so much for putting this together @CalogeroZarbo !

Would you mind adding a unit test that should trigger the assertion? Here's a similar test it could be based off of: https://github.com/microsoft/DeepSpeed/blob/master/tests/unit/test_fp16.py#L251

You could catch the assertion error with pytest:

with pytest.raises(AssertionError):
    model, optim, _,_ = deepspeed.initialize(args=args,
                                             model=model,
                                             model_parameters=model.parameters())

and end the test after that point.

[CalogeroZarbo]

@ShadenSmith Sure, on it! May I add the test to the same test_fp16.py file?

Thank you!
Cal

[ShadenSmith]

That would be perfect. Just right under the other ZeRO tests.

[CalogeroZarbo]

@ShadenSmith pytests are done. I added a helper class to define a custom optimizer on the fly. Thank you for the suggestion.
Let me know if I can help further on.

Cheers,
Cal

[tjruwase]

@CalogeroZarbo Thanks so much for doing this.

[ShadenSmith]

Thanks so much @CalogeroZarbo !