Write periodically

[vict0rsch]

@kngoyal @Lothiraldan right now if the code fails the tracker does not write to emissions.csv, I cannot see the emissions for my comet exp which failed un-gracefully.

What do you think is best?

1. write periodically so that no matter what we have data
2. use atexit.register

@Lothiraldan how does comet handles the end/crash of a python script? So that codecarbon can write to the file before the last comet upload?

[Lothiraldan]

The Comet SDK clean up (and calls EmissionTracker.stop()) when:

• Experiment.end()
• The atexit callback is called

It's not sufficient to write the emission.csv file periodically on disk, you would need also to log the file as an asset.

@vict0rsch If you can send SIGINT to your script, it should be enough for the Comet SDK to clean up properly.

[cifkao]

@Lothiraldan I would like a tool like this to be robust to when the experiment gets killed abruptly (so that there's no time to even call the atexit callback). I think this happens on some clusters with time limits or preemptive scheduling.

[benoit-cty]

The API is not an option for you, because you are in an environment without internet ?

[cifkao]

@benoit-cty There are definitely environments without internet connection on compute nodes. Sometimes there is connection but it is not reliable (I've encountered problems with other tools that tried to log something to a server) or requires some configuration (proxy). In such environments, it would be great to have a backup option that logs to a file (with a unique name for each run to avoid collisions) and allows syncing manually later.

[vict0rsch]

There's a flush method implemented (not released yet on pip and conda but available from source), see #236