Summary
Functions _response_to_container() and _response_to_object() in class AzureBlobsStorageDriver of azure_blobs hard-code the URL scheme to http in their extra objects, i.e.:
extra = { "url": "http://%s%s" % (response.connection.host, response.connection.action),
This will cause a request to download from Azure blob storage to fail if the storage URL is secure (i.e. uses https).
Detailed Information
Other functions in this class do check secure to dynamically assign the scheme, e.g. in get_object_cdn_url():
scheme="https" if self.secure else "http",
We can fix the bug in the response functions in a similar way by dynamically assigning the scheme in the extra objects:
scheme='https' if self.secure else 'http',
extra = {
'url': '%s://%s%s' % (scheme, response.connection.host,
response.connection.action),
Environment
Libcloud version: apache-libcloud==2.8.2
Python version: Python 3.8.10
Operating System: Ubuntu 20.04 LTS
Steps which are needed to reproduce problem:
I am using a secure blob storage account (i.e. its URL starts with https) for a CKAN project, version 2.9.
I am using AzureBlobsStorageDriver with the CKAN extension ckanext-cloudstorage, an extension that requires libcloud, and the extension ckanext-xloader (v 0.9.0), which accesses the blob storage account. Because of the bug reported here, Xloader is being passed an incorrect URL (starting with http instead of https), and therefore cannot download any data from the storage account:
ckanext.xloader.job_exceptions.HTTPError: b'Xloader received a bad HTTP response when trying to download the data file
Client Error: The account being accessed does not support http.
But with the fix to dynamically assign the scheme based on secure, the correct URL is passed and the data can be downloaded.
Summary
Functions
_response_to_container()and_response_to_object()in classAzureBlobsStorageDriverof azure_blobs hard-code the URL scheme tohttpin theirextraobjects, i.e.:This will cause a request to download from Azure blob storage to fail if the storage URL is secure (i.e. uses
https).Detailed Information
Other functions in this class do check
secureto dynamically assign thescheme, e.g. inget_object_cdn_url():We can fix the bug in the response functions in a similar way by dynamically assigning the scheme in the
extraobjects:Environment
Libcloud version: apache-libcloud==2.8.2
Python version: Python 3.8.10
Operating System: Ubuntu 20.04 LTS
Steps which are needed to reproduce problem:
I am using a secure blob storage account (i.e. its URL starts with
https) for a CKAN project, version 2.9.I am using
AzureBlobsStorageDriverwith the CKAN extension ckanext-cloudstorage, an extension that requires libcloud, and the extension ckanext-xloader (v 0.9.0), which accesses the blob storage account. Because of the bug reported here, Xloader is being passed an incorrect URL (starting withhttpinstead ofhttps), and therefore cannot download any data from the storage account:But with the fix to dynamically assign the scheme based on
secure, the correct URL is passed and the data can be downloaded.