upload client

[mikeAdamss]

What is this

In order to upload files to the dp-upload-service we need to present files in chunks wirh very specific fields to enable uploading large files in peices (chunking) and in a resumable way.

What to do

Jim Bryant (DE team) has already written some code to do this and it works.

• https://github.com/ONS-OpenData/cmd-run-transform/blob/9b164dd758c01f6eed4d1bb7cbd4836083943007/clients.py#L1112
• https://github.com/ONS-OpenData/cmd-run-transform/blob/master/clients.py#L1161

We just need to:

• wrap it in a class to become an upload client
• add unit tests, split out the logic a bit to make it easier to test.
• not use the temp file approach if at all possible, splitting files into disk will be very time and resource intensive. If you do need to split them try and split them in memory.

Whatever you do, do some reading around chunking and uploading files in python, its highly likely to be a solved thing.

Do use the BaseHttpClient as that'll give you exponential retries which you're gonna want.

So it'll be something along the lines of:

class UploadClient(BaseHttpClient):

    def upload_to_dp_upload_service(self ... whatever you'll need here):
        # encapsulate the logic from Jims scripts.

It's difficult to try this in anger until we've got staging access sorted, so if you pick this up befoe then just look to understand and rewrite the logic and make it testable ready for when that's sorted out.

Acceptqance Criteria\

• Upload script transcribed to upload client
• Unit tests
• Works in the staging environment