need to be able to restart the tracker

[stas00]

Requesting to make it easier to restart the tracker. In the HuggingFace BigScience project we need to be able to restart the tracker in various places. After writing the code I discovered start/stop aren't a pair. One can't start() again after stop(). So the current stop() is similar to destroy() but one can't tell that from the name. start() after stop() gives:

[codecarbon WARNING @ 19:55:24] <class 'Exception'>
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/codecarbon/core/util.py", line 10, in suppress
    yield
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/codecarbon/emissions_tracker.py", line 314, in stop
    self._scheduler.shutdown()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/apscheduler/schedulers/background.py", line 41, in shutdown
    super(BackgroundScheduler, self).shutdown(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/apscheduler/schedulers/blocking.py", line 24, in shutdown
    super(BlockingScheduler, self).shutdown(wait)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/apscheduler/schedulers/base.py", line 189, in shutdown
    raise SchedulerNotRunningError
apscheduler.schedulers.SchedulerNotRunningError: Scheduler is not running

Any chance you could adjust the code so that the following code could work?

cc = codecarbon.OfflineEmissionsTracker(...)
cc.start()
cc.stop(); cc.start() # restart
...
cc.stop(); cc.start() # restart
cc.stop()

or the longer version in the context of how I'm trying to write the wrapper to fit into Megatron-LM's setup:

_GLOBAL_CODECARBON_TRACKER = None
def _set_codecarbon_tracker(args):
    global _GLOBAL_CODECARBON_TRACKER
    if hasattr(args, 'codecarbon_dir'):
        import codecarbon
        print('> setting codecarbon ...')
        output_dir = args.codecarbon_dir
        output_file = f"emissions-{args.rank:03d}.csv"

        log_level = "info"
        country_iso_code="FRA"

        Path(output_dir).mkdir(parents=True, exist_ok=True)
        _GLOBAL_CODECARBON_TRACKER = codecarbon.OfflineEmissionsTracker(
            output_dir=output_dir,
            output_file=output_file,
            log_level=log_level,
            country_iso_code=country_iso_code,
        )


def codecarbon_tracker_start():
    global _GLOBAL_CODECARBON_TRACKER
    if _GLOBAL_CODECARBON_TRACKER is None:
        return

    print('codecarbon START')
    _GLOBAL_CODECARBON_TRACKER.start()


def codecarbon_tracker_stop():
    global _GLOBAL_CODECARBON_TRACKER
    if _GLOBAL_CODECARBON_TRACKER is None:
        return

    print('codecarbon STOP')
    _GLOBAL_CODECARBON_TRACKER.stop()


def codecarbon_tracker_restart():
    global _GLOBAL_CODECARBON_TRACKER
    if _GLOBAL_CODECARBON_TRACKER is None:
        return

    # output_dir = _GLOBAL_CODECARBON_TRACKER._output_dir
    # output_file = _GLOBAL_CODECARBON_TRACKER._output_file
    # log_level = _GLOBAL_CODECARBON_TRACKER._log_level
    # country_iso_code = _GLOBAL_CODECARBON_TRACKER._country_iso_code

    codecarbon_tracker_stop()
    codecarbon_tracker_start()

Otherwise I have to re-create the tracker all the time, but to do that, I can't even access the args passed to the tracker without going to private variables.

even better if there were an API method restart() which simply flushes the data to the file and continues. That would probably be a simpler change to do.

Thank you!

@JetRunner, @whobbes

[SaboniAmine]

Hello!

Thanks for documenting your use case, would you need to restart your runs in a distributed environment ?

One way to attach data to the correct run would be to save the run_id in the csv file (this is persisted anyway if the rest API mode is activated), to let you aggregate emissions by run in the local visualisation.
This may imply a change of this amplitude to your flow :

cc = codecarbon.OfflineEmissionsTracker(...)
RUN_ID = cc.start()
cc.stop(); 
cc.start(from_run=RUN_ID) # restart
...
cc.stop(); 
cc.start(from_run=RUN_ID) # restart
cc.stop()

Would it solve your issue ?

[stas00]

Sure that would work, in which case why not have cc.restart and have the cc object save RUN_ID so that the user doesn't have to store the extra data somewhere?

So cc.restart() would be just cc.start(from_run=cc.get_last_run_id())

This is just an idea of course.

Additionally, @jaketae has been trying a different solution here: bigscience-workshop#1

---

would you need to restart your runs in a distributed environment ?

Indeed, it would run on 64 nodes 4 gpus each at the moment, and I'm not sure we are doing it right. As we have each gpu/process log things separately.

Please let us know if there is a better way to approach this in a multi-node environment.

thank you!

[stas00]

@benoit-cty, just to check in if you're still planning on a new release so that we could set the requirements right.

Thank you!

[benoit-cty]

Yes, we will do a 2.0.0 @vict0rsch could you do it ? Because I've never done it on code carbon. I saw the instructions in contributing.md

[stas00]

Running into a warning bombardment on multi-node HPC #238
Going to turn CC off for now.
Let's hold off on release please until we sort it out?