From 5b30b27521cd64918c1f17fd1b5e616247a3dd6b Mon Sep 17 00:00:00 2001 From: dhirst1 <96255147+dhirst1@users.noreply.github.com> Date: Thu, 16 Dec 2021 09:57:31 -0700 Subject: [PATCH 1/3] Update config.rst Submitting an integration request on behalf of the StorJ team. StorJ has a vetted working S3 integration - added integration steps to reflect that integration. --- doc/sphinx-guides/source/installation/config.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index e213b08c704..828ac535697 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -240,7 +240,7 @@ As for the "Remote only" authentication mode, it means that: - ``:DefaultAuthProvider`` has been set to use the desired authentication provider - The "builtin" authentication provider has been disabled (:ref:`api-toggle-auth-provider`). Note that disabling the "builtin" authentication provider means that the API endpoint for converting an account from a remote auth provider will not work. Converting directly from one remote authentication provider to another (i.e. from GitHub to Google) is not supported. Conversion from remote is always to "builtin". Then the user initiates a conversion from "builtin" to remote. Note that longer term, the plan is to permit multiple login options to the same Dataverse installation account per https://github.com/IQSS/dataverse/issues/3487 (so all this talk of conversion will be moot) but for now users can only use a single login option, as explained in the :doc:`/user/account` section of the User Guide. In short, "remote only" might work for you if you only plan to use a single remote authentication provider such that no conversion between remote authentication providers will be necessary. -File Storage: Using a Local Filesystem and/or Swift and/or S3 object stores +File Storage: Using a Local Filesystem and/or Swift and/or object stores --------------------------------------------------------------------------- By default, a Dataverse installation stores all data files (files uploaded by end users) on the filesystem at ``/usr/local/payara5/glassfish/domains/domain1/files``. This path can vary based on answers you gave to the installer (see the :ref:`dataverse-installer` section of the Installation Guide) or afterward by reconfiguring the ``dataverse.files.\.directory`` JVM option described below. @@ -557,6 +557,11 @@ Reported Working S3-Compatible Storage **Can be used for quick testing, too:** just use the example values above. Uses the public (read: unsecure and possibly slow) https://play.minio.io:9000 service. +'StorJ Integration' +Per the general Dataverse Installation to use S3 Storage noted above, you’ll first set up the StorJ S3 store, by defining the id, type, and label. After Following the general installation set the following configurations to use a StorJ object store - dataverse.files..payload-signing=true This optional setting will enable payload signing for the S3 bucket. Set dataverse.files..chunked-encoding=false. This optional setting will disable chunked encoding. + +Note that for direct uploads and downloads, Dataverse redirects to the proxy-url but presigns the urls based on the dataverse.files..custom-endpoint-url. Also, consider that if you choose to enable dataverse.files..download-redirect note that the S3 URLs expire after 60 minutes by default. You can change that minute value to reflect a timeout value that’s more appropriate by using .dataverse.files..url-expiration-minutes. + `Surf Object Store v2019-10-30 `_ Set ``dataverse.files..payload-signing=true`` and ``dataverse.files..chunked-encoding=false`` to use Surf Object Store. From 556d4798df9f0777e0767aebdb63ef02cf7f18f0 Mon Sep 17 00:00:00 2001 From: dhirst1 <96255147+dhirst1@users.noreply.github.com> Date: Wed, 5 Jan 2022 07:13:56 -0700 Subject: [PATCH 2/3] Update config.rst --- doc/sphinx-guides/source/installation/config.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 828ac535697..5426c824e66 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -557,8 +557,8 @@ Reported Working S3-Compatible Storage **Can be used for quick testing, too:** just use the example values above. Uses the public (read: unsecure and possibly slow) https://play.minio.io:9000 service. -'StorJ Integration' -Per the general Dataverse Installation to use S3 Storage noted above, you’ll first set up the StorJ S3 store, by defining the id, type, and label. After Following the general installation set the following configurations to use a StorJ object store - dataverse.files..payload-signing=true This optional setting will enable payload signing for the S3 bucket. Set dataverse.files..chunked-encoding=false. This optional setting will disable chunked encoding. +`StorJ Object Store `_ + StorJ is a distributed object store that can be configured with an S3 gateway. Per the general Dataverse Installation to use S3 Storage noted above, you’ll first set up the StorJ S3 store, by defining the id, type, and label. After Following the general installation set the following configurations to use a StorJ object store - dataverse.files..payload-signing=true This optional setting will enable payload signing for the S3 bucket. Set dataverse.files..chunked-encoding=false. This optional setting will disable chunked encoding. Note that for direct uploads and downloads, Dataverse redirects to the proxy-url but presigns the urls based on the dataverse.files..custom-endpoint-url. Also, consider that if you choose to enable dataverse.files..download-redirect note that the S3 URLs expire after 60 minutes by default. You can change that minute value to reflect a timeout value that’s more appropriate by using .dataverse.files..url-expiration-minutes. From e9b64da7a598bfd5795071705cd3fd61fff83041 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 6 Jan 2022 10:59:15 -0500 Subject: [PATCH 3/3] clean up StorJ S3 note #8314 --- doc/sphinx-guides/source/installation/config.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 5426c824e66..1fbc98d622b 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -558,9 +558,9 @@ Reported Working S3-Compatible Storage possibly slow) https://play.minio.io:9000 service. `StorJ Object Store `_ - StorJ is a distributed object store that can be configured with an S3 gateway. Per the general Dataverse Installation to use S3 Storage noted above, you’ll first set up the StorJ S3 store, by defining the id, type, and label. After Following the general installation set the following configurations to use a StorJ object store - dataverse.files..payload-signing=true This optional setting will enable payload signing for the S3 bucket. Set dataverse.files..chunked-encoding=false. This optional setting will disable chunked encoding. + StorJ is a distributed object store that can be configured with an S3 gateway. Per the S3 Storage instructions above, you'll first set up the StorJ S3 store by defining the id, type, and label. After following the general installation, set the following configurations to use a StorJ object store: ``dataverse.files..payload-signing=true`` and ``dataverse.files..chunked-encoding=false``. -Note that for direct uploads and downloads, Dataverse redirects to the proxy-url but presigns the urls based on the dataverse.files..custom-endpoint-url. Also, consider that if you choose to enable dataverse.files..download-redirect note that the S3 URLs expire after 60 minutes by default. You can change that minute value to reflect a timeout value that’s more appropriate by using .dataverse.files..url-expiration-minutes. + Note that for direct uploads and downloads, Dataverse redirects to the proxy-url but presigns the urls based on the ``dataverse.files..custom-endpoint-url``. Also, note that if you choose to enable ``dataverse.files..download-redirect`` the S3 URLs expire after 60 minutes by default. You can change that minute value to reflect a timeout value that’s more appropriate by using ``dataverse.files..url-expiration-minutes``. `Surf Object Store v2019-10-30 `_ Set ``dataverse.files..payload-signing=true`` and ``dataverse.files..chunked-encoding=false`` to use Surf Object