download data as msgpack

[bertinatto]

Any thoughts?

I'm assuming current tests, like [1], still cover this patch.

Addresses #58.

[1] https://github.com/scrapinghub/python-hubstorage/blob/msgpack/tests/test_project.py#L228

[laurentsenta]

Can you explain why we don't stream msgpack?
The difference should be at the "accept" header and the decode function right? The iteration & retrying should be reused from json-lines.

[bertinatto]

I assumed BytesIO would stream the data while reading it, but I noticed requests downloads everything [1] anyway. I re-factored out the common code. Thanks for pointing it out.

Is there any reason the retry logic isn't done using retrying?

[1] https://github.com/kennethreitz/requests/blob/master/requests/models.py#L737

[laurentsenta]

How test_bulkdata checks the mpack implementation?
I can't find any hint of fixture that uses different accept headers.

[bertinatto]

Here [1] resource.iter_values() is called and if the resource in question has msgpack support the job data will be dowloaded as msgpack [2]. If the unpacked data is correct this assert [3] should pass.

[1] https://github.com/scrapinghub/python-hubstorage/blob/master/tests/test_project.py#L241
[2] https://github.com/scrapinghub/python-hubstorage/blob/msgpack/hubstorage/resourcetype.py#L92-L99
[3] https://github.com/scrapinghub/python-hubstorage/blob/master/tests/test_project.py#L242

[chekunkov]

it's possible to get method='get' or method='gEt' here, it's converted to upper only in requests lib

[bertinatto]

Last push has some improvements (suggestions) and fixes for test failures that were shadowed before #64.

[vshlapakov]

@chekunkov Updated, review please.

[chekunkov]

LGTM

[jdemaeyer]

Hey guys, there are two issues with this PR:

• It breaks py3 compatibility (xrange is now range)
• Items returned by iter_values() no longer contain the _key. This breaks shub log -f. I can possibly change it in shub but I wouldn't be sure that there's no other users using _key

[chekunkov]

@jdemaeyer we will fix py3 compatibility asap

I'm afraid for now you have to pin python-hubstorage version in shub setup.py to <= 0.23.2, logs are using msgpack format by default and it seems we have a bug in Hubstorage - it returns _key field only for json and jsonlines formats.