Skip to content

analytics: refactor into a module#2826

Merged
efiop merged 46 commits into
masterfrom
unknown repository
Dec 9, 2019
Merged

analytics: refactor into a module#2826
efiop merged 46 commits into
masterfrom
unknown repository

Conversation

@ghost
Copy link
Copy Markdown

@ghost ghost commented Nov 20, 2019

Follow up from discussion on #2819

TODO:

  • Write tests

Comment thread dvc/analytics/__main__.py Outdated
Comment thread dvc/analytics/system_info.py Outdated
Comment thread tests/func/test_analytics.py
Comment thread dvc/analytics/system_info.py Outdated
@ghost ghost changed the title analytics: refactor into a module [WIP] analytics: refactor into a module Nov 21, 2019
Copy link
Copy Markdown
Contributor

@Suor Suor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing a class, but splitting it into several files kind of defeat the purpose of trying to make it more readable.

Copy link
Copy Markdown
Contributor

@Suor Suor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I was too harsh on files, but I think we shouldn't separate them too much. I've read the old code now and see how is it confusing, you can't even understand the structure of the report at a glance.

Comment thread dvc/analytics/system_info.py Outdated
Comment thread dvc/analytics/user_id.py Outdated
Comment thread dvc/analytics/user_id.py Outdated
@ghost
Copy link
Copy Markdown
Author

ghost commented Nov 26, 2019

Removing a class, but splitting it into several files kind of defeat the purpose of trying to make it more readable.

Agree, @Suor 🙂

Comment thread dvc/analytics.py Outdated
@ghost ghost changed the title [WIP] analytics: refactor into a module analytics: refactor into a module Nov 27, 2019
@ghost ghost requested review from Suor and efiop November 27, 2019 01:28
@ghost
Copy link
Copy Markdown
Author

ghost commented Nov 27, 2019

Tests were sort of exhaustive

Comment thread dvc/analytics.py
@efiop
Copy link
Copy Markdown
Contributor

efiop commented Dec 6, 2019

@MrOutis Tests are still failing, please take a look.

@ghost ghost added the c21-full-week label Dec 6, 2019
Comment thread dvc/analytics.py Outdated
Comment thread dvc/analytics.py Outdated
Comment thread dvc/analytics.py
Comment thread tests/unit/test_analytics.py
efiop
efiop previously requested changes Dec 6, 2019
Copy link
Copy Markdown
Contributor

@efiop efiop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good! I'm still worried about breaking the backend analytics logic by this, so please see some requests for tests up above. Really don't fancy working on analytics backend these days, so we need to do our best to make sure that the reports are exactly the same as they were before and they won't break anything.

@ghost ghost requested a review from efiop December 8, 2019 03:19
@ghost ghost dismissed efiop’s stale review December 8, 2019 03:20

excellent observations, @efiop, thanks again for the review 👍 mind checking if my changes address your comments?

@efiop efiop requested a review from pared December 9, 2019 13:40
Copy link
Copy Markdown
Contributor

@efiop efiop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ghost ghost requested a review from Suor December 9, 2019 14:25
Fixture to prevent modifying the actual global config
"""
with mock.patch(
"dvc.config.Config.get_global_config_dir", return_value=str(tmp_path)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not fspath()?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the patch for fspath and now it doesn't work with pathlib.Path in 3.5: #2903

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does for me, patch is only needed for Python 3.5 builtin pathlib, but pytests tmp_path uses pathlib2.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Suor we've been using str in all other places, so it is no biggie. We need to revisit that more carefully later and fix everywhere.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Suor , didn't new that pytest uses pathlib2!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my bad, then, @Suor , I just assumed it was using builtin pathlib

def test_send(mock_post, tmp_path):
url = "https://analytics.dvc.org"
report = {"name": "dummy report"}
fname = str(tmp_path / "report")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use fspath().

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as this one #2826 (comment)

😞

Comment on lines +72 to +73
with open(fname, "w") as fobj:
json.dump(report, fobj)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(tmp_path / "report").write_text(json.dumps(report))

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Suor , would need to use fspath on analytics.send:

@mock.patch("requests.post")
def test_send(mock_post, tmp_path):
    url = "https://analytics.dvc.org"
    report = {"name": "dummy report"}
    file = (tmp_path / "report")
    data = convert_to_unicode(json.dumps(report))

    file.write_text(data)
    analytics.send(fspath(file))

    assert mock_post.called
    assert mock_post.call_args.args[0] == url

I'd prefer to leave it like this for now, since we lack support for python 3.5 🙁

Copy link
Copy Markdown
Contributor

@Suor Suor Dec 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only we don't, what exactly doesn't work in Python 3.5 for you?

@ghost ghost requested a review from Suor December 9, 2019 15:32
@efiop efiop merged commit 8890daf into treeverse:master Dec 9, 2019
@ghost ghost deleted the refactor-analytics branch December 9, 2019 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhances DVC refactoring Factoring and re-factoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants