Check cardon directory is not None

[thomasw21]

We've noticed that 13B training crashed.

The logs suggest that it's linked to the new carbon feature:

File "/gpfsssd/worksf/projects/rech/six/commun/code/tr1-13B/Megatron-DeepSpeed-tr1-13B/megatron/global_vars.py", line 169, in _set_codecarbon_tracker
  _set_codecarbon_tracker(args)
 File "/gpfsssd/worksf/projects/rech/six/commun/code/tr1-13B/Megatron-DeepSpeed-tr1-13B/megatron/global_vars.py", line 169, in _set_codecarbon_tracker
  _set_codecarbon_tracker(args)
   File "/gpfsssd/worksf/projects/rech/six/commun/code/tr1-13B/Megatron-DeepSpeed-tr1-13B/megatron/global_vars.py", line 169, in _set_codecarbon_tracker
_set_codecarbon_tracker(args)
 File "/gpfsssd/worksf/projects/rech/six/commun/code/tr1-13B/Megatron-DeepSpeed-tr1-13B/megatron/global_vars.py", line 169, in _set_codecarbon_tracker
  Path(output_dir).mkdir(parents=True, exist_ok=True)
   File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 1042, in __new__
Path(output_dir).mkdir(parents=True, exist_ok=True)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 1042, in __new__
  Path(output_dir).mkdir(parents=True, exist_ok=True)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 1042, in __new__
  Path(output_dir).mkdir(parents=True, exist_ok=True)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 1042, in __new__
  self = cls._from_parts(args, init=False)
   File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 683, in _from_parts
self = cls._from_parts(args, init=False)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 683, in _from_parts
  self = cls._from_parts(args, init=False)
self = cls._from_parts(args, init=False)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 683, in _from_parts
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 683, in _from_parts
  drv, root, parts = self._parse_args(args)
   File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 667, in _parse_args
drv, root, parts = self._parse_args(args)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 667, in _parse_args
  drv, root, parts = self._parse_args(args)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 667, in _parse_args
  drv, root, parts = self._parse_args(args)
 File "/gpfswork/rech/six/commun/conda/tr1-13B/lib/python3.8/pathlib.py", line 667, in _parse_args
  a = os.fspath(a)
TypeError:   expected str, bytes or os.PathLike object, not NoneTypea = os.fspath(a)

I'm pretty sure it's because we don't check the value of args.codecarbon_dir, it defaults to None using the argument parser.

Instead we need to check that that value is not None.

I don't think the fix is urgent since we'll be running with the option activated from now on. (though it might impact smaller experiments)

[stas00]

Thanks a lot, @thomasw21

[stas00]

codecarbon is not ready mlco2/codecarbon#238 that's why I was trying to disable it in the slurm script.

[stas00]

Got a chance to test it, your fix is perfect, @thomasw21! Thank you!