3937 rewrite installer in python again

[landreev]

What this PR does / why we need it:
New Installer Implementation, in Python
Which issue(s) this PR closes:

Closes #3937

Special notes for your reviewer:

Suggestions on how to test this:

Does this PR introduce a user interface change?:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

Coverage decreased (-0.009%) to 19.504% when pulling a35e3f6 on 3937-rewrite-installer-in-python-again into b70cd18 on develop.

[pdurbin]

Overall, I think this looks good so I'm sending it to QA. That said, her are some (minor) suggestions, questions, and comments. @landreev should feel free to pull this out of QA if he wants to work on any of this.

Suggestions:

• in the Perl install script, mention the new Python script
• in requirements.txt, put version of psycopg2
• consider recommending Python 3's venv (virtual environment)

Questions:

• 10.5072 is configured but when they contact DataCite do they get a new authority for testing?
• Any thoughts on when we'll stop supporting the Perl installer?

Comments:

• @donsizemore is planning on testing the new Python installer as part of IQSS/dataverse-ansible#139
• I'm not sure if anyone will miss the "podName" stuff but I guess it doesn't hurt to leave it in there. Recently @poikilotherm mentioned OpenShift (which podName was added for) at gdcc/dataverse-kubernetes#129
• It would be nice to support Windows dev environments some day. Out of scope for now.

[djbrooke]

@donsizemore, @landreev mentioned that you were discussing this quite a bit. Your review here would be very appreciated if you have the time to do so. Thanks in advance!

[donsizemore]

@landreev what's the relationship between default.config and interactive.config? I popped the latter into Ansible as a template but the install script seems to want default.config's structure instead.

I can swap out the templates, but default.config has 30 lines of content while interactive.config has 47. It's looking like I'll need both?

[landreev]

@landreev what's the relationship between default.config and interactive.config? I popped the latter into Ansible as a template but the install script seems to want default.config's structure instead.

I can swap out the templates, but default.config has 30 lines of content while interactive.config has 47. It's looking like I'll need both?

(@pdurbin mentioned he had already answered this, but just to clarify and document this here:)

Yes, the script needs both.
The default.config is for supplying the configuration values. The installer also has a new option --config_file= that can be used to use a custom file, instead of the default.config.

intreractive.config stores the prompts and info/help messages for the interactive setup dialog. It's the stuff that's hard-coded in %CONFIG_PROMPTS and %CONFIG_COMMENTS in the old installer. Storing it in a config file outside the script should make it cleaner/easier to maintain (or even localize the installer in a different language, if anyone ever feels a need for that...).

Thinking about it, there's no need to load and parse this file at all, when the installer is running in the non-interactive mode. I'll add an if statement for that.

[donsizemore]

@landreev one more question: previously the dataverse installer captured the output of glassfish-setup into glassfish-setup.out — do we want to keep that, or offer an output file flag? helpful for debugging...

[poikilotherm]

Nobody asked for it, but my 2 cents: as long as @donsizemore is not relying on the OpenShift workarounds, feel free to let anything related to it RIP in good ol' Perl version.

@pdurbin and I where going to cut out the Openshift stuff from the guide anyway and move all cloud/container regarding OpenShift to dataverse-k8s. See also gdcc/dataverse-kubernetes#129

[pdurbin]

Oh, I guess I'll throw in a few cents too. Back when the (Perl) installer required root I wrote a little shell script called "devinstall" at f181543 that seemed to work for me as an alternative. These days you can also find a slight variation of this script at https://github.com/IQSS/dataverse/blob/v4.18.1/scripts/deploy/phoenix.dataverse.org/install

These days I just use the standard installer for dev environments rather than my "devinstall" script. But on servers, I use dataverse-ansible, which doesn't use the old Perl script. I guess if dataverse-ansible uses the new Python script I'll start using it on servers.

I'm not exactly sure where I'm going with this. I guess I'm happy that the new Python script will be exercised regularly by dataverse-ansible. I think somewhere in docker-aio there's an option to use the old Perl script but I'm not in the habit of using docker-aio. In short, I'm glad the new code will be tested regularly, at least the non-interactive mode.

[landreev]

@poikilotherm

Nobody asked for it, but my 2 cents: as long as @donsizemore is not replying on the OpenShift workarounds, feel free to let anything related to it RIP in good ol' Perl version.

@pdurbin and me where going to cut out the Openshift stuff from the guide anyway and move all cloud/container regarding OpenShift to dataverse-k8s. See also IQSS/dataverse-kubernetes#129

I would like to preserve all that Openshift functionality; I just wanted to open a new issue for it. See my comment in the issue: #3937 (comment)

[landreev]

@donsizemore

@landreev one more question: previously the dataverse installer captured the output of glassfish-setup into glassfish-setup.out — do we want to keep that, or offer an output file flag? helpful for debugging...

Yes, I just noticed/remembered it myself - that the new version is just dumping that output on screen. I'll make it saves it in a file, just like the old version.

[landreev]

@pdurbin

* 10.5072 is configured but when they contact DataCite do they get a new authority for testing?

Hmm. I didn't even think about this - I simply replicated what the old installer was doing... (same exact initial configuration, same "you have to contact DataCite ... and request a test account" message)
They do issue individual test name spaces to people these days, don't they?
What should we be doing then? Leave that setting blank?
(or should we be setting it to 10.5072, and configure the "fake" provider?) - I would appreciate any input.

[landreev]

I have opened a new issue (#6494) for (re-)implementing the OpenShift pod-based functionality in the installer. @poikilotherm will you be willing to consult/help testing this?
It should be easy, we just need to check the correct pod names and make sure we know what needs to be set up on different pods.

[landreev]

I'm going to send this to QA here, so that we could start our normal internal testing process. @donsizemore if you discover any problems in the Ansible context and/or have any other suggestions, please let us know.
No pressure, obviously; we can always add things later.

[donsizemore]

@landreev sounds good - i'm still hammering away (mostly in modifying ansible)

[donsizemore]

dataverse-ansible now deploys the 4.18.1 release normally using the python3 installer instead of a bunch of calls I had broken out into Ansible-ese. I did keep the JDBC jar placement in Ansible's realm as that's a configurable switch for testing of newer jars. If anyone would like to give the work-in-progress a whirl, it's https://github.com/IQSS/dataverse-ansible/tree/139_python3_installer

[kcondon]

Issues found so far:

1. Add to readme: yum install postgresql96-devel needed so .h files available for pip install psycopg2
2. default.config from original non-interactive installer has same name as default.config used for python install and may conflict, resulting in failed install if downloaded from doc.
3. *.config files for python missing from dvinstall.zip
4. Replace default system email, nobody@mailinator.com, with blank
5. Maybe tighten up README to indicate supported python versions for psycopg: http://initd.org/psycopg/docs/install.html v2.7 or v3.4-3.8
6. Need python-devel to be installed for Python.h to build psycopg.

[landreev]

Yay! I'll give it a try.

@donsizemore

dataverse-ansible now deploys the 4.18.1 release normally using the python3 installer instead of a bunch of calls I had broken out into Ansible-ese.

[pdurbin]

@donsizemore wow! Looking at https://github.com/IQSS/dataverse-ansible/compare/139_python3_installer I see you've been busy!! Thanks!! 🎉

[kcondon]

All set testing, passing back to dev.

[landreev]

@kcondon, I addressed the feedback below. With the exception of the old-style default.config - I figured it wasn't necessary, just because of how unlikely it is to happen to anyone else. (and easy to fix if it does :)

Issues found so far:

1. Add to readme: yum install postgresql96-devel needed so .h files available for pip install psycopg2

2. default.config from original non-interactive installer has same name as default.config used for python install and may conflict, resulting in failed install if downloaded from doc.

3. *.config files for python missing from dvinstall.zip

4. Replace default system email, [nobody@mailinator.com](mailto:nobody@mailinator.com), with blank

5. Maybe tighten up README to indicate supported python versions for psycopg: http://initd.org/psycopg/docs/install.html v2.7 or v3.4-3.8

6. Need python-devel to be installed for Python.h to build psycopg.

[landreev]

@pdurbin

* Any thoughts on when we'll stop supporting the Perl installer?

My thought was to release it as beta/experimental in the next release; invite people to give it a try; incorporate any feedback we receive and then switch to it as the default installer for the release after that.
But idk - I'm feeling a little discouraged by the fact that a user needs to do more work to run the new installer than the old one. For ex., having to install psycopg2 - at least some people will definitely run into some issues with that.
By which I mean, none of it is going to be a problem for another developer. But not everybody trying to install Dataverse from a release bundle is a developer, or super technical necessarily.

This is something we may want to discuss as a team at some point during the next release cycle. I personally would be ok to say that you have to be somewhat of a sysadmin/be comfortable with python, etc. to install Dataverse.