Skip to content

Method to flush emission to CSV#236

Merged
benoit-cty merged 21 commits intomasterfrom
flush_tracker
Aug 25, 2021
Merged

Method to flush emission to CSV#236
benoit-cty merged 21 commits intomasterfrom
flush_tracker

Conversation

@benoit-cty
Copy link
Copy Markdown
Contributor

@benoit-cty benoit-cty commented Aug 21, 2021

Addressing #217 after talking with @stas00 (see #235) I add a new method flush() that compute emissions and add them to the CSV, and/or call the API if you use it.
I use it in a Keras callback and it do the job of adding a line in the CSV after each epoch.
You can see it in mnist_callback.py

I've added documentation. But no unit test.

@benoit-cty benoit-cty requested a review from vict0rsch August 21, 2021 20:26
@benoit-cty benoit-cty mentioned this pull request Aug 21, 2021
5 tasks
Copy link
Copy Markdown
Contributor

@stas00 stas00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding this new API, @benoit-cty

I added a few language improvement suggestions.

Comment thread README.md Outdated
Comment thread examples/mnist_callback.py Outdated
Comment thread examples/README.md Outdated
benoit-cty and others added 7 commits August 22, 2021 08:11
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
@benoit-cty benoit-cty linked an issue Aug 22, 2021 that may be closed by this pull request
@benoit-cty benoit-cty linked an issue Aug 22, 2021 that may be closed by this pull request
@benoit-cty benoit-cty marked this pull request as ready for review August 22, 2021 09:18
@benoit-cty
Copy link
Copy Markdown
Contributor Author

Thanks for the language improvement !
I've added unit tests so it's ready for review.

Copy link
Copy Markdown
Contributor

@stas00 stas00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this last incarnation to work correctly for our needs

Added a few tiny language corrections.

Comment thread codecarbon/core/api_client.py Outdated
Comment thread codecarbon/core/api_client.py Outdated
Comment thread codecarbon/core/api_client.py Outdated
Comment thread codecarbon/emissions_tracker.py Outdated
stas00 added a commit to stas00/Megatron-DeepSpeed that referenced this pull request Aug 22, 2021
@vict0rsch
Copy link
Copy Markdown
Contributor

Should we have run_id: Optional[str] = None in the EmissionsData data class? Right now the API test does not pass because of this

@vict0rsch
Copy link
Copy Markdown
Contributor

Added on_csv_write: Optional[str]:

:param on_csv_write: "append" or "update". Whether to always append a new line
                        to the csv when writing or to update the existing `run_id`
                        row (useful when calling`tracker.flush()` manually).
                        Accepts one of "append" or "update".

Comment thread codecarbon/output.py Outdated
@vict0rsch vict0rsch requested a review from SaboniAmine August 25, 2021 08:01
Copy link
Copy Markdown
Contributor Author

@benoit-cty benoit-cty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice additions Victor 👍

@benoit-cty benoit-cty merged commit 03479b6 into master Aug 25, 2021
@benoit-cty benoit-cty deleted the flush_tracker branch August 25, 2021 15:51
@stas00
Copy link
Copy Markdown
Contributor

stas00 commented Aug 25, 2021

Thank you everybody who helped to make this happen!

stas00 added a commit to bigscience-workshop/Megatron-DeepSpeed that referenced this pull request Aug 25, 2021
* add codecarbon

* switch to offline

* rework to also restart the tracker at each checkpoint save to ensure as little as possible data is lost

* adjust API to match bigscience-workshop/codecarbon#1

* fix logging

* new implementation based on mlco2/codecarbon#236

* add test

* update requirements
stas00 added a commit to bigscience-workshop/Megatron-DeepSpeed that referenced this pull request Aug 25, 2021
* add codecarbon

* switch to offline

* rework to also restart the tracker at each checkpoint save to ensure as little as possible data is lost

* adjust API to match bigscience-workshop/codecarbon#1

* fix logging

* new implementation based on mlco2/codecarbon#236

* add test

* update requirements
@vict0rsch vict0rsch mentioned this pull request Sep 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

need to be able to restart the tracker [Problem] track_emissions decorator returns a None

4 participants