improve ability of developers to run integration tests

[pameyer]

Integration tests at https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/ are useful; documentation of how to run these tests on a local system can be confusing for a new developer.

possibly related: #3938, #1936, #3959 , #2443

[pdurbin]

@pameyer thanks for pull request #4370 ! I'm copying and pasting a comment I just left there because we tend to favor discussion in issues rather than pull requests:

@pameyer overall, this is a very interesting contribution. I was going to try to execute your code but I'm blocked on not knowing what "surgery" steps are necessary and am much more interested in a fully automated solution than one with manual steps. Both Vagrant and our existing Docker files set up Glassfish and Solr in an automated way. Can we do the same in this "all in one" solution? Also, there seems to be some duplication of files, such as the Solr schema.xml and the post script I use to deploy to the phoenix server. Let's talk some more about definition of done for this issue. Perhaps all we need is more documentation about the surgery? Thanks!

[pameyer]

• the "surgery" steps are the ones from the guides (and Vagrant / existing Docker files) for glassfish ans solr. My thinking for not doing those here is that they 1) introduce additional points of failure into the process 2) send un-necessary and easily-avoidable network traffic on every build. This might be something that could be moved into the setup script, but I was shooting for an incremental improvement.
• duplication of the solr schema.xml is an oversight (hangover from that being something frequently customized); probably reasonable to fix.
• post script needed to be modified, because I was reverse engineering "deploy to phoenix" to get to "do stuff necessary to setup data for integration tests"; any of the phoenix scripts that weren't modified should be used in their original forms/locations.

[pdurbin]

@pameyer talked this out a bit at http://irclog.iq.harvard.edu/dataverse/2017-12-12 and plan to meet up tomorrow if time allows. It would do me good to see a demo.

[pdurbin]

I got a great brain dump from @pameyer as well as some "deps" (dependencies, two tarballs) I need to try to execute the code in his pull request. The Day of Meetings prevented us from talking more about running tests on developers' laptops vs. Jenkins but I have a much better understanding of where @pameyer is coming from and I like the way he thinks!

[pdurbin]

@pameyer thanks for providing me with glassfish4dv.tgz and solr-4.6.0dv.tgz. I've been leaving comments in your pull request (#4370). While I don't think there's any harm in merging it, I'm somewhat troubled that running the REST Assured tests is hanging here for me:

------htmlStr4: com.jayway.restassured.internal.RestAssuredResponseImpl@70d577e9
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.004 sec
Running edu.harvard.iq.dataverse.api.FileMetadataIT
Dec 14, 2017 9:56:11 AM edu.harvard.iq.dataverse.api.UtilIT getRestAssuredBaseUri
INFO: Base URL for tests: http://localhost:8083
TOKEN: c24070cf-9f97-4b42-955c-801a74dde388
DATAVERSE: http://localhost:8083/dataverse/271f6688
{
    "status": "ERROR",
    "message": "Validation failed: Directory Name cannot contain leading or trailing file separators. Invalid value: 'data/subdir1/'."
}

Oh, also, your pull request includes a pg_hba.conf for postgres but we already have it at https://github.com/IQSS/dataverse/blob/v4.8.4/conf/vagrant/var/lib/pgsql/data/pg_hba.conf so it might be nice to re-use it if we can.

@donsizemore since the pull request uses CentOS 7 rather than CentOS 6, you might be interesting in taking a look as you poke at #4384.

The last thing I'll say for now is that I still want to talk to @pameyer about how we can execute all this great code automatically from Jenkins or Travis or some other continuous integration service. This pull request does definitely add value in the sense that it forces the user to run the installer. This differs from the phoenix setup where I ran a shell script that was somewhat equivalent to the installer a couple years ago. Right now we already say "To run more than one test at a time, separate by commas" over at http://guides.dataverse.org/en/4.8.4/developers/testing.html#integration-tests but developers are not in the habit of running all the tests on their laptops. I think it's too much to ask. These tests should be run automatically on a server. I think this pull request is incremental progress toward that goal, something @bjonnh and I have talked about.

Sorry so rambly.

[pameyer]

@pdurbin Longer term, I'd been thinking it would make sense to have a Jenkins declarative pipeline that would build everything (war, installer zip, docker images), run the unit and integration tests (and browser-based tests if/when they become available), and generate coverage reports (I'm pretty sure coveralls is missing integration test coverage).

I'd missed that other copy of pg_hba.conf.

It may be possible to extend Travis to build images / run integration tests, but it's an approach I'm less familiar with.

[pdurbin]

I would be nice to know why conf/docker-aio/dv/postgresql.conf is present.

[pdurbin]

Note to self: to get a shell in the Docker container: docker exec -it adb505218f64 bash

[pdurbin]

server.log is showing this which I believe is normal because it's what we're testing above:

[2017-12-14T14:56:12.383+0000] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dvn.core.index.DOIEZIdServiceBean] [tid: _ThreadID=29 _ThreadName=http-listener-1(2)] [timeMillis: 1513263372383] [levelValue: 900] [[
  message bad request - unrecognized identifier scheme]]

[2017-12-14T14:56:12.414+0000] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.DatasetVersion] [tid: _ThreadID=29 _ThreadName=http-listener-1(2)] [timeMillis: 1513263372414] [levelValue: 800] [[
  Constraint violation found in FileMetadata. Directory Name cannot contain leading or trailing file separators. The invalid value is "data/subdir1/".]]

[2017-12-14T14:56:12.476+0000] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=30 _ThreadName=Thread-8] [timeMillis: 1513263372476] [levelValue: 800] [[
  vals = [edu.harvard.iq.dataverse.ControlledVocabularyValue[ id=12 ]], contains: false]]