Fix 5478 move Swift settings to domain.xml for better governability, no disk access of properties file, leaner class by carlosmcgregor · Pull Request #5680 · IQSS/dataverse

carlosmcgregor · 2019-03-22T19:45:20Z

Related Issues

connects to Refactor Swift storage driver properties / properties getter #5478: Refactor Swift storage driver properties / properties getter

Pull Request Checklist

Unit tests completed
Integration tests: None
Deployment requirements, SQL updates, Solr updates, etc.: None
Documentation completed
Merged latest from "develop" branch and resolved conflicts

Changes to Documentation

The following commands are now needed to configure Swift:

asadmin create-jvm-options "\-Ddataverse.file.swift-default-endpoint=endpoint1"
asadmin create-jvm-options "\-Ddataverse.file.swift-auth-type.endpoint1=$AUTH_TYPE"
asadmin create-jvm-options "\-Ddataverse.file.swift-auth-url.endpoint1=$AUTH_URL"
asadmin create-jvm-options "\-Ddataverse.file.swift-tenant.endpoint1=$TENANT"
asadmin create-jvm-options "\-Ddataverse.file.swift-username.endpoint1=$USERNAME"
asadmin create-jvm-options "\-Ddataverse.file.swift-password.endpoint1='${ALIAS=$PASSWORD}'"
asadmin create-jvm-options "\-Ddataverse.file.swift-endpoint.endpoint1=$ENDPOINT"
asadmin create-jvm-options "\-Ddataverse.file.swift-hash-key.endpoint1=$HASH_KEY"

It makes use of the create-password-alias approach that @pdurbin pointed out (thank you!). Otherwise, all settings remain the same as with swift.properties. Some additional settings that were hardcoded (i.e. separator and temporary URL expiry time) were moved to the settings. These are:

dataverse.files.swift-folder-path-separator
dataverse.files.swift-temporary-url-expiry-time

Please let me know if you have further questions regarding the changes that were made!

Remove function that loads swift.properties

Add public class variable identifier prefix

pdurbin

The docs at a future version of http://guides.dataverse.org/en/4.11/installation/config.html#swift-storage need to change, right?

That page can be found at doc/sphinx-guides/source/installation/config.rst if you could please edit it.

carlosmcgregor · 2019-03-22T19:48:40Z

@pdurbin, I'm on it!

coveralls · 2019-03-22T19:51:08Z

Coverage increased (+0.005%) to 18.066% when pulling e0ab352 on carlosmcgregor:develop into fe4dcfc on IQSS:develop.

poikilotherm · 2019-03-23T13:07:35Z

I am not sure if it's a good idea to move this to domain XML. For future platforms like Payara or Glassfish 5, the rising usage of configuration frameworks and more, this might be a long-term decision which is not easy to revert. After re-reading specs: you can actually follow that road - ConfigAPI is fine with this. But please read on, below is important.

At least pretty please avoid - and _ in the config names. - is an invalid char for environment variables, _ is used for substitution. See gdcc/dataverse-kubernetes#11 Best option is to use a "." for namespacing as used elsewhere plus camel / pascal case.

…entation Add full create-password-alias instructions in documentation Add missing swiftEndPoint

poikilotherm · 2019-03-25T22:03:48Z

doc/sphinx-guides/source/installation/config.rst

-    ./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.file.swift-username.endpoint1=your-username"
-    ./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.file.swift-password.endpoint1='${ALIAS=your-password}'"
-    ./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.file.swift-endpoint.endpoint1=your-swift-endpoint"
+    ./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.file.swift.defaultEndpoint=endpoint1"


Some typos here. In the docs you use .file., but in the code .files. is used.

@poikilotherm good catch

Thank you for noticing that! Fixed.

carlosmcgregor · 2019-03-26T15:47:39Z

I read http://irclog.iq.harvard.edu/dataverse/2019-03-25#i_88874 and I would like to add one more setting that has -:

dataverse.files.storage-driver-id

carlosmcgregor · 2019-03-26T15:48:23Z

Sorry! Closed by accident :/

pdurbin · 2019-03-26T15:52:11Z

I would like to add one more setting

@carlosmcgregor no problem, I pulled this out of code review until you're done. Please let us know when we should take another look. Thanks! Also, you're welcome to join us in IRC if you'd like. 😄

carlosmcgregor · 2019-03-26T15:55:11Z

Sorry, @pdurbin ! I should have added that dataverse.files.storage-driver-id is out of the scope of this pull. Just mentioned it so that it can be added to a list of Dataverse JVM options that need to be changed :)

pdurbin · 2019-03-26T17:11:22Z

@carlosmcgregor oh! I understand now. It looks like @poikilotherm is handling some of those "dash" JVM options over at https://github.com/IQSS/dataverse-kubernetes/blob/655b92d60b1a09e109154c9c237525827dbf4dff/dataverse-k8s/bin/init_1_configure.sh#L94

I'll move this issue and pull request back to code review. Thanks for explaining the scope.

poikilotherm · 2019-03-26T18:37:25Z

@carlosmcgregor Feel free to leave a comment on gdcc/dataverse-kubernetes#11 :-)

carlosmcgregor · 2019-03-27T20:33:16Z

Thank you @pdurbin and @poikilotherm !

kcondon · 2019-04-10T19:47:24Z

@carlosmcgregor Hi, would you mind refreshing this branch from /develop? We've since released v4.12 and your branch shows 4.11. This is causing a build issue. Once this is adjusted I will test and merge. Thanks! Kevin

carlosmcgregor · 2019-04-10T22:52:21Z

Done, @kcondon ! Thank you :)

carlosmcgregor added 5 commits March 4, 2019 18:31

Change from swift.properties config to System

f31768b

Remove function that loads swift.properties

Fix typo in property

509b950

Move swift default endpoint to class variable

f54b410

Add public class variable identifier prefix

Merge branch 'develop' of github.com:IQSS/dataverse into develop

9ae533c

Fix typo in temp URL expiry time setting

e2d324d

pdurbin reviewed Mar 22, 2019

View reviewed changes

carlosmcgregor closed this Mar 22, 2019

pdurbin reopened this Mar 22, 2019

Update Swift installation guide

87f4cad

pdurbin mentioned this pull request Mar 25, 2019

Refactor Swift storage driver properties / properties getter #5478

Closed

Change Swift settings to dot and camel case in Swift driver and docum…

1417237

…entation Add full create-password-alias instructions in documentation Add missing swiftEndPoint

poikilotherm reviewed Mar 25, 2019

View reviewed changes

carlosmcgregor added 2 commits March 25, 2019 18:03

Fix tiny typo

166c62f

Add missing s's

eed33a3

carlosmcgregor closed this Mar 26, 2019

carlosmcgregor reopened this Mar 26, 2019

kcondon self-assigned this Apr 9, 2019

carlosmcgregor added 2 commits April 10, 2019 18:47

Merge branch 'develop' of github.com:IQSS/dataverse into develop

ff26e2b

Merge remote-tracking branch 'origin/develop' into develop

2871cf7

carlosmcgregor mentioned this pull request Apr 10, 2019

Swift storage driver no longer working with MOC (because of StorageIdentifier changes) #5754

Closed

Add release notes for 5478

e0ab352

kcondon merged commit a371605 into IQSS:develop Apr 11, 2019

pdurbin added this to the 4.13 milestone Apr 22, 2019

carlosmcgregor mentioned this pull request Jan 10, 2020

Iqss/6485 - Multiple File Stores #6488

Merged

Comments

Conversation

carlosmcgregor commented Mar 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues

Pull Request Checklist

Changes to Documentation

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

carlosmcgregor commented Mar 22, 2019

Uh oh!

coveralls commented Mar 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

poikilotherm commented Mar 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

poikilotherm Mar 25, 2019

Choose a reason for hiding this comment

Uh oh!

pdurbin Mar 25, 2019

Choose a reason for hiding this comment

Uh oh!

carlosmcgregor Mar 26, 2019

Choose a reason for hiding this comment

Uh oh!

carlosmcgregor commented Mar 26, 2019

Uh oh!

carlosmcgregor commented Mar 26, 2019

Uh oh!

pdurbin commented Mar 26, 2019

Uh oh!

carlosmcgregor commented Mar 26, 2019

Uh oh!

pdurbin commented Mar 26, 2019

Uh oh!

poikilotherm commented Mar 26, 2019

Uh oh!

carlosmcgregor commented Mar 27, 2019

Uh oh!

kcondon commented Apr 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlosmcgregor commented Apr 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

carlosmcgregor commented Mar 22, 2019 •

edited

Loading

coveralls commented Mar 22, 2019 •

edited

Loading

poikilotherm commented Mar 23, 2019 •

edited

Loading

kcondon commented Apr 10, 2019 •

edited

Loading