From 681d1d0e41fcccff29e64efb7dbb14db516a9d30 Mon Sep 17 00:00:00 2001 From: Brian Silverstein Date: Tue, 6 Jun 2017 15:26:36 -0400 Subject: [PATCH 01/27] Updated testing documentation page with actual guide documentation; updated documentation.rst and geospatial.rst to include table of contents preface --- .../source/developers/documentation.rst | 2 + .../source/developers/geospatial.rst | 2 + .../source/developers/testing.rst | 63 ++++++++++++++++++- 3 files changed, 66 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/documentation.rst b/doc/sphinx-guides/source/developers/documentation.rst index 1908dc59ba5..4716be3de8e 100755 --- a/doc/sphinx-guides/source/developers/documentation.rst +++ b/doc/sphinx-guides/source/developers/documentation.rst @@ -2,6 +2,8 @@ Documentation ============= +.. contents:: :local: + Quick Fix ----------- diff --git a/doc/sphinx-guides/source/developers/geospatial.rst b/doc/sphinx-guides/source/developers/geospatial.rst index 55fbf9e6808..12584c84c1c 100644 --- a/doc/sphinx-guides/source/developers/geospatial.rst +++ b/doc/sphinx-guides/source/developers/geospatial.rst @@ -2,6 +2,8 @@ Geospatial Data =============== +.. contents:: :local: + How Dataverse Ingests Shapefiles -------------------------------- diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index a79d2d08570..8fc9d7b4ab1 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -2,12 +2,73 @@ Testing ======= +For testing in Dataverse, JUnit passes along test information to utilities like `Coveralls `_ which builds a quantifiable report of how much code coverage (% of code with verifiable tests) the project’s codebase has. We encourage test-driven development to help ensure new code has functional backend logic that can be repeated and help diagnose issues within the app. + +.. contents:: :local: + Unit Tests ---------- -Write them. JUnit is your friend. +Write them. `JUnit `_ is your friend. Creating unit tests for your code is a helpful way to test what you’ve built piece by piece. + +You may notice when code is pushed, an automated Coveralls bot will let you know if it has increased or decreased code coverage. + +- If code coverage is increasing: good! You made some test(s). +- If code coverage is decreasing: write some unit tests to show the world just how good your code is! + +A unit test should execute an operation of your code in a controlled fashion. You must make an assertion of what the expected response gives back. It's important to test optimistic output and assertions, but also test controlled failures such as 400 errors or other issues your API may encounter (to simulate human or machine error) with assertions stating such. + +Know how your program should handle anticipated errors/exceptions and confirm with your test(s) that it does so properly. Integration Tests ----------------- Write them. `REST-assured `_ and `Selenium `_ are recommended. A Jenkins environment is available for automated testing: https://build.hmdc.harvard.edu:8443 + +Integration tests help ensure that your code gets along well with the rest of Dataverse. It’s good to test the different functions of your code with the different parts of your Dataverse which the feature may interact with. + +These tests already live in the code base under ``/dataverse/src/test/java/edu/harvard/iq/dataverse/api/`` in your respective Dataverse directory. When you write an integration test of your own, it should be saved here. + +To execute existing integration tests on your local Dataverse, a helpful command line tool to use is `Maven `_. You should have Maven installed as per the `Development Environment `_ guide, but if not it’s easily done via Homebrew: ``brew install maven``. + +Once installed, you may run commands with ``mvn [options] [] []``. + ++ If you want to run just one particular API test, it’s as easy as you think: + + ``mvn test -Dtest=FileRecordJobIT`` + ++ To run more than one test at a time, separate by commas: + + ``mvn test -Dtest=FileRecordJobIT,ConfirmEmailIT`` + ++ To run any test(s) on a particular domain, replace localhost:8080 with desired domain name: + + ``mvn test -Dtest=FileMetadataIT -Ddataverse.test.baseurl='http://localhost:8080'`` + +**Note:** if you’re trying to run the test suite and receive the error "Dataverse config issue: No API key defined for built in user management" just run the curl command below. + +``curl -X PUT -d 'burrito' http://localhost:8080/api/admin/settings/BuiltinUsers.KEY`` + +Writing Tests +------------- + +When writing tests, you may find it helpful to first map out which functions of your code you want to test, and then write a functional unit test for each which can later comprise a larger integration test. + +Recall that integration tests should live in the ``/dataverse/src/test/java/edu/harvard/iq/dataverse/api`` directory, or if you’re on Netbeans then navigate to the project’s Test Packages → edu.harvard.iq.dataverse.api and place your testnameIT.java file there! + +If writing tests is new to you: in NetBeans, poke around existing tests. If you read through any you may notice each test within a file (bearing the @Test prefix) has an assertion which specifies the expected result (expected, from the Run menu, select Test File. + +The output in the file’s test window (beside the console) corresponds to what each line of each @Test block is doing. Many existing tests provide readable Json output, and how verbose the output is up to you as a developer. + +You do not have to reinvent the wheel. There are many useful methods you can call in your own tests -- especially within UtilIT.java -- when you need your test to create and/or interact with generated accounts, files, datasets, etc. Similar methods can subsequently delete them to get them out of your way as desired before the test has concluded. + +For example, if you’re testing your code’s operations with user accounts, the method ``UtilIT.createRandomUser();`` can generate an account for your test to work with. The same account can then be deleted by your program by calling the ``UtilIT.deleteUser();`` method on the imaginary friend your test generated. + +Remember, it’s only a test (and it's not graded)! Some guidelines to bear in mind: + +- Map out which logical functions you want to test +- Understand what’s being tested and ensure it’s repeatable +- Assert the conditions of success / return values for each operation + * A useful resource would be `HTTP status codes `_ +- Let the code do the labor; automate everything that happens when you run your test file. +- Just as with any development, if you’re stuck: ask for help! From 8aaed4ffbe80290c2ae4092e695581b361b0e289 Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Wed, 7 Jun 2017 15:07:51 -0400 Subject: [PATCH 02/27] Fixing merge conflict [ref: #3431] Phil and I are confused about whether our merge conflict worked! Testing out this fix. --- .../source/developers/documentation.rst | 13 ++++++++++++- doc/sphinx-guides/source/developers/geospatial.rst | 5 +++-- doc/sphinx-guides/source/developers/testing.rst | 3 ++- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/doc/sphinx-guides/source/developers/documentation.rst b/doc/sphinx-guides/source/developers/documentation.rst index 4716be3de8e..472e44a01b1 100755 --- a/doc/sphinx-guides/source/developers/documentation.rst +++ b/doc/sphinx-guides/source/developers/documentation.rst @@ -2,7 +2,8 @@ Documentation ============= -.. contents:: :local: +.. contents:: |toctitle| + :local: Quick Fix ----------- @@ -63,3 +64,13 @@ After sphinx is done processing the files you should notice that the html folder You can click on the files in the html folder to preview the changes. Now you can make a commit with the changes to your own fork in GitHub and submit a pull request to the dataverse repository. + +Table of Contents +----------------- + +Every non-index page should use the following code to display a table of contents of internal sub-headings: :: + + .. contents:: |toctitle| + :local: + +This code should be placed below any introductory text/images and directly above the first subheading, much like a Wikipedia page. diff --git a/doc/sphinx-guides/source/developers/geospatial.rst b/doc/sphinx-guides/source/developers/geospatial.rst index 12584c84c1c..d43e23cbefc 100644 --- a/doc/sphinx-guides/source/developers/geospatial.rst +++ b/doc/sphinx-guides/source/developers/geospatial.rst @@ -2,7 +2,8 @@ Geospatial Data =============== -.. contents:: :local: +.. contents:: |toctitle| + :local: How Dataverse Ingests Shapefiles -------------------------------- @@ -173,4 +174,4 @@ The ``get_join_targets()`` function in ``dataverse_layer_services.py`` uses the Saving Join Target Information to Geoconnect Database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``get_latest_jointarget_information()`` in ``utils.py`` retrieves recent JoinTarget Information from the database. (See the `utils code in GitHub `_.) \ No newline at end of file +The ``get_latest_jointarget_information()`` in ``utils.py`` retrieves recent JoinTarget Information from the database. (See the `utils code in GitHub `_.) diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 8fc9d7b4ab1..18263656a3f 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -4,7 +4,8 @@ Testing For testing in Dataverse, JUnit passes along test information to utilities like `Coveralls `_ which builds a quantifiable report of how much code coverage (% of code with verifiable tests) the project’s codebase has. We encourage test-driven development to help ensure new code has functional backend logic that can be repeated and help diagnose issues within the app. -.. contents:: :local: +.. contents:: |toctitle| + :local: Unit Tests ---------- From 4aa8141a9185881b3f2d92b80cccaa8c0c3fd35a Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Wed, 7 Jun 2017 17:12:03 -0400 Subject: [PATCH 03/27] Changed "Branching Strategy" page to "Version Control" [Ref: #3431] Changed filename and title of page and added a section called "Using Git" with info on how to make commits and resolve merge conflicts. --- ...ching-strategy.rst => version-control.rst} | 62 +++++++++++++++++-- 1 file changed, 56 insertions(+), 6 deletions(-) rename doc/sphinx-guides/source/developers/{branching-strategy.rst => version-control.rst} (50%) mode change 100755 => 100644 diff --git a/doc/sphinx-guides/source/developers/branching-strategy.rst b/doc/sphinx-guides/source/developers/version-control.rst old mode 100755 new mode 100644 similarity index 50% rename from doc/sphinx-guides/source/developers/branching-strategy.rst rename to doc/sphinx-guides/source/developers/version-control.rst index 536db58668a..e55e86a7239 --- a/doc/sphinx-guides/source/developers/branching-strategy.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -1,12 +1,15 @@ ================== -Branching Strategy +Version Control ================== .. contents:: |toctitle| :local: +Branching Strategy +------------------ + Goals ------ +~~~~~ The goals of the Dataverse branching strategy are: @@ -16,21 +19,68 @@ The goals of the Dataverse branching strategy are: We follow a simplified "git flow" model described at http://nvie.com/posts/a-successful-git-branching-model/ involving a "master" branch, a "develop" branch, and feature branches such as "1234-bug-fix". Branches --------- +~~~~~~~~ The "master" Branch -~~~~~~~~~~~~~~~~~~~ +******************* The "`master `_" branch represents released versions of Dataverse. As mentioned in the :doc:`making-releases` section, at release time we update the master branch to include all the code for that release. Commits are never made directly to master. Rather, master is updated only when we merge code into it from the "develop" branch. The "develop" Branch -~~~~~~~~~~~~~~~~~~~~ +******************** The "`develop `_" branch represents code that was stable enough to merge from a "feature" branch (described below) and that will make it into the next release. Like master, commits are never made to the develop branch. The develop branch is where integration occurs. Your goal is have your code merged into the develop branch after it has been reviewed. Feature Branches -~~~~~~~~~~~~~~~~ +**************** Feature branches are used for both developing features and fixing bugs. They are named after the GitHub issue they are meant to address, which means the branch should not be created until the GitHub issue exists. For example, if https://github.com/IQSS/dataverse/issues/1234 had a title of "Bug fix", you would name your branch "1234-bug-fix" or some other short "slug" with the issue number at the start. Always create your feature branch from the latest code in develop, pulling if necessary. Push your feature branch either to your fork of Dataverse or to the main repo at IQSS if you have write access. Create a pull request based on the feature branch you pushed. As mentioned in https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md if you do not have access to advance your pull request into the "Code Review" column at https://waffle.io/IQSS/dataverse you should reach out to ask for it to be moved on your behalf. + +Using Git +--------- +This section explains step-by-step some of the most important processes involved in using Git to contribute to Dataverse. + +Contribution Checklist +~~~~~~~~~~~~~~~~~~~~~~ +To make a contribution to Dataverse's code: + +1. Find or create a GitHub issue explaining what you'd like to contribute. + +**e.g.** `Issue #3728 `_ points out a typo in a page of Dataverse's documentation. + +2. Create a new branch that you will commit your change to. + +**e.g.** *3728-doc-apipolicy-fix* + +3. Make your change in the form of a commit to that branch. Make sure the title of your commit includes a reference to the number of the issue it relates to. + +**e.g.** *Fixed BlockedApiPolicy [ref: #3728]* + +4. Make a pull request to get approval to merge your changes into the Develop branch. + +**For more details** see our `Pull Request Template `_. + +Merge Conflict Resolution Checklist +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When merging a branch to Develop, you may encounter a merge conflict. Here is a checklist for how to resolve this situation. Note that this checklist presumes that you're using the free software tools GitHub Desktop and Netbeans. There are other tools that can accomplish this, but your checklist may look different. + +**In GitHub Desktop:** + +1. Sync from Develop +2. Open specific branch that's having the merge conflict. +3. Click "Update from develop" + +**In Netbeans:** + +4. Click Window -> Favorites and open your local Dataverse project folder in the Favorites panel +5. In this file browser, you can follow the red cylinder icon to find files with merge conflicts +6. Double click the red merge conflicted file +7. Right click on the red tab for that file and select Git -> Resolve Conflicts +8. Resolve on right or left (if you select "both" you can do finer edits after) +9. Save all changes + +**In GitHub Desktop:** + +10. Commit the merge (append issue number to end, e.g. #3728) and leave note about what was resolved. \ No newline at end of file From 271137df5ffbc3aa47ccf9d4aa0601e0a939a7cb Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 8 Jun 2017 15:53:36 -0400 Subject: [PATCH 04/27] finish converting branching-strategy.rst into version-control.rst #3431 --- doc/sphinx-guides/source/developers/dev-environment.rst | 2 +- doc/sphinx-guides/source/developers/index.rst | 2 +- doc/sphinx-guides/source/developers/intro.rst | 2 +- doc/sphinx-guides/source/developers/making-releases.rst | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 09a1fe2ee17..cc3424bc3c6 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -92,7 +92,7 @@ From the terminal, ``ssh-keygen`` will create new ssh keys for you: Clone Project from GitHub ~~~~~~~~~~~~~~~~~~~~~~~~~ -Before making commits, please read about our :doc:`/developers/branching-strategy` to make sure you commit to the right branch. +Before cloning the repo, you are invited to read about our branching strategy in the :doc:`version-control` section but we'll explain the basics here. Determine Which Repo To Push To ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/sphinx-guides/source/developers/index.rst b/doc/sphinx-guides/source/developers/index.rst index 6530fdc94bb..461af55864d 100755 --- a/doc/sphinx-guides/source/developers/index.rst +++ b/doc/sphinx-guides/source/developers/index.rst @@ -12,7 +12,7 @@ Developer Guide intro dev-environment - branching-strategy + version-control testing documentation debugging diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst index 47ad45aaf3c..4d5d870fff3 100755 --- a/doc/sphinx-guides/source/developers/intro.rst +++ b/doc/sphinx-guides/source/developers/intro.rst @@ -12,7 +12,7 @@ Intended Audience This guide is intended primarily for developers who want to work on the main Dataverse code base at https://github.com/IQSS/dataverse -To get started, you'll want to set up your :doc:`/developers/dev-environment` and make sure you understand the :doc:`/developers/branching-strategy`. Thorough :doc:`/developers/testing` is encouraged (and expected). Opinions about :doc:`/developers/coding-style` are welcome! +To get started, you'll want to set up your :doc:`dev-environment` and make sure you understand the branching strategy described in the :doc:`version-control` section. :doc:`testing` is expected. Opinions about :doc:`coding-style` are welcome! If you have any questions at all, please reach out to other developers per https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index bb151d906d0..00f5358ae57 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -19,7 +19,7 @@ Here's an example commit where all three files were updated at once: https://git Merge "develop" into "master" ----------------------------- -The "develop" branch should be merged into "master" before tagging. See also :doc:`branching-strategy`. +The "develop" branch should be merged into "master" before tagging. See also the branching strategy described in the :doc:`version-control` section. Write Release Notes ------------------- From 3c0c5387b96413a8a779286a870e3e32aadda39d Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 8 Jun 2017 16:16:11 -0400 Subject: [PATCH 05/27] must build war before running installer #3431 --- .../source/developers/dev-environment.rst | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index cc3424bc3c6..56362522167 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -117,6 +117,25 @@ If you do have push access to https://github.com/IQSS/dataverse clone it: ``git clone git@github.com:IQSS/dataverse.git`` +Building the WAR File +~~~~~~~~~~~~~~~~~~~~~ + +Soon, we'll be running the Dataverse installer, but before we do, we must build the Dataverse application, which is delivered as a "WAR" file. WAR stands for "Web application ARchive" and you can read more about this packaging format at https://en.wikipedia.org/wiki/WAR_(file_format) + +The first time you build the war file, it may take a few minutes while dependencies are downloaded from Maven Central. + +We'll describe below how to build the WAR file from both Netbean and the terminal, but in both cases, you'll want to see the output "BUILD SUCCESS". + +Building the War File from Netbeans +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +From Netbeans, click "Run" and then "Build Project (dataverse)". + +Building the War File from the Terminal +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +After cloning the git repo, you need to ``cd`` into ``dataverse`` and run ``mvn package``. If you don't have the ``mvn`` command available to you, you need to install Maven, which is mentioned in the :doc:`tools` section. + Installing and Running Solr ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -139,7 +158,7 @@ Once some dataverses, datasets, and files have been created and indexed, you can Run Installer ~~~~~~~~~~~~~ -Once you install Glassfish and PostgreSQL, you need to configure the environment for the Dataverse app - configure the database connection, set some options, etc. We have a new installer script that should do it all for you. Again, assuming that the clone on the Dataverse repository was retrieved using NetBeans and that it is saved in the path ~/NetBeansProjects: +Now that you have all the prerequisites in place, you need to configure the environment for the Dataverse app - configure the database connection, set some options, etc. We have an installer script that should do it all for you. Again, assuming that the clone on the Dataverse repository was retrieved using NetBeans and that it is saved in the path ~/NetBeansProjects: ``cd ~/NetBeansProjects/dataverse/scripts/installer`` From e32b9de01af127721931487ab8c4b5f42f52cf1e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 9 Jun 2017 10:40:00 -0400 Subject: [PATCH 06/27] explain context-root problem and solution #3431 --- .../source/developers/dev-environment.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 56362522167..34d76a658c7 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -170,6 +170,22 @@ The script is a variation of the old installer from DVN 3.x that calls another s All the future changes to the configuration that are Glassfish-specific and can be done through ``asadmin`` should now go into ``scripts/install/glassfish-setup.sh``. +Troubleshooting +--------------- + +We've described above the "happy path" of when everything goes right with setting up your Dataverse development environment. Here are some common problems and solutions for when things go wrong. + +context-root in glassfish-web.xml Munged by Netbeans +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For unknown reasons, Netbeans will sometimes change the following line under ``src/main/webapp/WEB-INF/glassfish-web.xml``: + +``/`` + +Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete the line entirely. Either way, you will see very strange behavior when attempting to click around Dataverse in a browser. The home page will load but icons will be missing. Any other page will fail to load entirely and you'll see a Glassfish error. + +The solution is to put the file back to how it was before Netbeans touched it. If anyone knows of an open Netbeans bug about this, please let us know. + Rebuilding Your Dev Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 269301a1c6b9d0fce528d0b34f61f6348f8633b9 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 9 Jun 2017 13:37:12 -0400 Subject: [PATCH 07/27] reword version control page #3431 --- .../source/developers/version-control.rst | 94 +++++++++++++------ 1 file changed, 66 insertions(+), 28 deletions(-) diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index e55e86a7239..284ac85c368 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -2,9 +2,17 @@ Version Control ================== +The Dataverse Project uses git for version control and GitHub for hosting. On this page we'll explain where to find the code, our branching strategey, some tips on how to make a pull request, and other git tips. + .. contents:: |toctitle| :local: + +Where to Find the Dataverse Code +-------------------------------- + +The main code in question is at https://github.com/IQSS/dataverse but as explained in the :doc:`intro` section under "Related Projects", there are many other code bases you can hack on if you wish! + Branching Strategy ------------------ @@ -34,53 +42,83 @@ The "`develop `_" branch represents code that Feature Branches **************** -Feature branches are used for both developing features and fixing bugs. They are named after the GitHub issue they are meant to address, which means the branch should not be created until the GitHub issue exists. For example, if https://github.com/IQSS/dataverse/issues/1234 had a title of "Bug fix", you would name your branch "1234-bug-fix" or some other short "slug" with the issue number at the start. +Feature branches are used for both developing features and fixing bugs. They are named after the GitHub issue they are meant to address, so create a GitHub issue if you need to. + +"3728-doc-apipolicy-fix" is an example of a fine name for your feature branch. It tells us that you are addressing https://github.com/IQSS/dataverse/issues/3728 and the "slug" is short, descriptive, and starts with the issue number. + +How to Make a Pull Request +-------------------------- + +Pull requests take all shapes and sizes, from a one-character typo fix to hundreds of files changing at once. Generally speaking, smaller pull requests are better so that they are easier to code review. That said, don't hold back on writing enough code or documentation to address the issue to the best of your ability. + +If you are writing code, please see :doc:`testing` for guidance on writing tests. + +The example of creating a pull request below has to do with fixing an important issue with the :doc:`documentation` but applies to fixing code as well. -Always create your feature branch from the latest code in develop, pulling if necessary. Push your feature branch either to your fork of Dataverse or to the main repo at IQSS if you have write access. Create a pull request based on the feature branch you pushed. As mentioned in https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md if you do not have access to advance your pull request into the "Code Review" column at https://waffle.io/IQSS/dataverse you should reach out to ask for it to be moved on your behalf. +Find or Create a GitHub Issue +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Using Git ---------- -This section explains step-by-step some of the most important processes involved in using Git to contribute to Dataverse. +For guidance on which issue to work on, please ask! Also, see https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md -Contribution Checklist -~~~~~~~~~~~~~~~~~~~~~~ -To make a contribution to Dataverse's code: +Let's say you want to tackle https://github.com/IQSS/dataverse/issues/3728 which points out a typo in a page of Dataverse's documentation. -1. Find or create a GitHub issue explaining what you'd like to contribute. +Create a New Branch Off the develop Branch +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**e.g.** `Issue #3728 `_ points out a typo in a page of Dataverse's documentation. +Always create your feature branch from the latest code in develop, pulling the latest code if necessary. As mentioned above, your branch should have a name like "3728-doc-apipolicy-fix" that starts with the issue number you are addressing, and ends with a short, descriptive name. -2. Create a new branch that you will commit your change to. +Commit Your Change to Your New Branch +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**e.g.** *3728-doc-apipolicy-fix* +Make your change in the form of a commit to that branch. Make sure the title of your commit includes a reference to the number of the issue it relates to, such as ``Fixed BlockedApiPolicy #3728``. Ideally the title is 50 characters or fewer, but don't worry about it. Use as much space in the body of the commit message as you need! -3. Make your change in the form of a commit to that branch. Make sure the title of your commit includes a reference to the number of the issue it relates to. +Push Your Branch to GitHub +~~~~~~~~~~~~~~~~~~~~~~~~~~ -**e.g.** *Fixed BlockedApiPolicy [ref: #3728]* +Push your feature branch to your fork of Dataverse (or to the main repo at IQSS, if you have write access). Create a pull request based on the feature branch you pushed. As mentioned in https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md if you do not have access to advance your pull request into the "Code Review" column at https://waffle.io/IQSS/dataverse you should reach out to ask for it to be moved on your behalf. -4. Make a pull request to get approval to merge your changes into the Develop branch. +Make a Pull Request +~~~~~~~~~~~~~~~~~~~ -**For more details** see our `Pull Request Template `_. +Make a pull request to get approval to merge your changes into the develop branch. Feedback on the pull request template we use is welcome! The "connects to #3728" syntax is important because it's used at https://waffle.io/IQSS/dataverse to associate pull requests with issues. -Merge Conflict Resolution Checklist -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When merging a branch to Develop, you may encounter a merge conflict. Here is a checklist for how to resolve this situation. Note that this checklist presumes that you're using the free software tools GitHub Desktop and Netbeans. There are other tools that can accomplish this, but your checklist may look different. +Here's an example of a pull request for issue #3728: https://github.com/IQSS/dataverse/pull/3827 + +Make Sure Your Pull Request Has Been Advanced to Code Review +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://waffle.io/IQSS/dataverse + +Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for various ways to reach out to developers who have enough access to the GitHub repo to move your issue and pull request to the "Code Review" column. + +How to Resolve Conflicts in Your Pull Request +--------------------------------------------- + +Unfortunately, pull requests can quickly become "stale" and unmergable as other pull requests are merged into the develop branch ahead of you. This is completely normal because other developers made their pull requests before you did. + +The Dataverse team may ping you to ask you to merge the latest from the develop branch into your branch and resolve merge conflicts. If this sounds daunting, please just say so and we will assist you. + +If you'd like to resolve the merge conflicts yourself, here are some steps that work well GitHub Desktop and Netbeans. **In GitHub Desktop:** -1. Sync from Develop -2. Open specific branch that's having the merge conflict. -3. Click "Update from develop" +1. Sync from develop. +2. Open the specific branch that's having the merge conflict. +3. Click "Update from develop". **In Netbeans:** -4. Click Window -> Favorites and open your local Dataverse project folder in the Favorites panel -5. In this file browser, you can follow the red cylinder icon to find files with merge conflicts -6. Double click the red merge conflicted file -7. Right click on the red tab for that file and select Git -> Resolve Conflicts -8. Resolve on right or left (if you select "both" you can do finer edits after) +4. Click Window -> Favorites and open your local Dataverse project folder in the Favorites panel. +5. In this file browser, you can follow the red cylinder icon to find files with merge conflicts. +6. Double click the red merge conflicted file. +7. Right click on the red tab for that file and select Git -> Resolve Conflicts. +8. Resolve on right or left (if you select "both" you can do finer edits after). 9. Save all changes **In GitHub Desktop:** -10. Commit the merge (append issue number to end, e.g. #3728) and leave note about what was resolved. \ No newline at end of file +10. Commit the merge (append issue number to end, e.g. #3728) and leave note about what was resolved. + +**In GitHub Issues:** + +11. Leave a comment for the Dataverse team that you have resolved the merge conflicts. From 22209127e4246437f105b0aedb4b82e6f0c03846 Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Fri, 9 Jun 2017 16:45:56 -0400 Subject: [PATCH 08/27] Version Control page review [Ref: #3431] Fixed a couple typos in the Version Control page. Lookin' good! --- doc/sphinx-guides/source/developers/version-control.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index 284ac85c368..7cf52da49e4 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -2,7 +2,7 @@ Version Control ================== -The Dataverse Project uses git for version control and GitHub for hosting. On this page we'll explain where to find the code, our branching strategey, some tips on how to make a pull request, and other git tips. +The Dataverse Project uses git for version control and GitHub for hosting. On this page we'll explain where to find the code, our branching strategey, advice on how to make a pull request, and other git tips. .. contents:: |toctitle| :local: @@ -62,7 +62,7 @@ For guidance on which issue to work on, please ask! Also, see https://github.com Let's say you want to tackle https://github.com/IQSS/dataverse/issues/3728 which points out a typo in a page of Dataverse's documentation. -Create a New Branch Off the develop Branch +Create a New Branch off the develop Branch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Always create your feature branch from the latest code in develop, pulling the latest code if necessary. As mentioned above, your branch should have a name like "3728-doc-apipolicy-fix" that starts with the issue number you are addressing, and ends with a short, descriptive name. @@ -94,11 +94,11 @@ Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for variou How to Resolve Conflicts in Your Pull Request --------------------------------------------- -Unfortunately, pull requests can quickly become "stale" and unmergable as other pull requests are merged into the develop branch ahead of you. This is completely normal because other developers made their pull requests before you did. +Unfortunately, pull requests can quickly become "stale" and unmergable as other pull requests are merged into the develop branch ahead of you. This is completely norma, and often occurs because other developers made their pull requests before you did. The Dataverse team may ping you to ask you to merge the latest from the develop branch into your branch and resolve merge conflicts. If this sounds daunting, please just say so and we will assist you. -If you'd like to resolve the merge conflicts yourself, here are some steps that work well GitHub Desktop and Netbeans. +If you'd like to resolve the merge conflicts yourself, here are some steps to do so that make use of GitHub Desktop and Netbeans. **In GitHub Desktop:** From 682452c30d216f33d99974a9b0f983120c63431a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 12 Jun 2017 09:30:39 -0400 Subject: [PATCH 09/27] fix a typo, a couple tweaks #3431 --- doc/sphinx-guides/source/developers/intro.rst | 2 ++ doc/sphinx-guides/source/developers/version-control.rst | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst index 4d5d870fff3..0cf695b6ea6 100755 --- a/doc/sphinx-guides/source/developers/intro.rst +++ b/doc/sphinx-guides/source/developers/intro.rst @@ -39,6 +39,7 @@ Related Projects As a developer, you also may be interested in these projects related to Dataverse: +- Miniverse - expose metrics from a Dataverse database: https://github.com/IQSS/miniverse - `Zelig `_ (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig - `TwoRavens `_ (Javascript) - a `d3.js `_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens - :doc:`/developers/unf/index` (Java) - a Universal Numerical Fingerprint: https://github.com/IQSS/UNF @@ -46,4 +47,5 @@ As a developer, you also may be interested in these projects related to Datavers - GeoConnect (Python) - create a map by uploading files to Dataverse: https://github.com/IQSS/geoconnect - Dataverse API client libraries - use Dataverse APIs from various languages: :doc:`/api/client-libraries` - Third party apps - make use of Dataverse APIs: :doc:`/api/apps` +- chat.dataverse.org - chat interface for Dataverse users and developers: https://github.com/IQSS/chat.dataverse.org - [Your project here] :) diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index 7cf52da49e4..03e8ea2a4e6 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -11,7 +11,7 @@ The Dataverse Project uses git for version control and GitHub for hosting. On th Where to Find the Dataverse Code -------------------------------- -The main code in question is at https://github.com/IQSS/dataverse but as explained in the :doc:`intro` section under "Related Projects", there are many other code bases you can hack on if you wish! +The main Dataverse code at https://github.com/IQSS/dataverse but as explained in the :doc:`intro` section under "Related Projects", there are many other code bases you can hack on if you wish! Branching Strategy ------------------ @@ -94,7 +94,7 @@ Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for variou How to Resolve Conflicts in Your Pull Request --------------------------------------------- -Unfortunately, pull requests can quickly become "stale" and unmergable as other pull requests are merged into the develop branch ahead of you. This is completely norma, and often occurs because other developers made their pull requests before you did. +Unfortunately, pull requests can quickly become "stale" and unmergable as other pull requests are merged into the develop branch ahead of you. This is completely normal, and often occurs because other developers made their pull requests before you did. The Dataverse team may ping you to ask you to merge the latest from the develop branch into your branch and resolve merge conflicts. If this sounds daunting, please just say so and we will assist you. From 5b57bec282b646ced6c61640371be4161feb32aa Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 12 Jun 2017 14:17:29 -0400 Subject: [PATCH 10/27] rewrite of testing page #3431 --- .../source/developers/dev-environment.rst | 4 +- .../source/developers/testing.rst | 192 +++++++++++++++--- 2 files changed, 165 insertions(+), 31 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 34d76a658c7..fe9f479eca0 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -8,7 +8,7 @@ Development Environment Assumptions ----------- -This guide assumes you are using a Mac. If you are using Windows or Linux, please reach out to other developers at https://groups.google.com/forum/#!forum/dataverse-dev +This guide assumes you are using a Mac. With some tweaks, it's not hard to get a dev environment set up on Linux. If you are using Windows, you might have the most success using Vagrant, which is listed under the :doc:`tools` section. Requirements ------------ @@ -170,6 +170,8 @@ The script is a variation of the old installer from DVN 3.x that calls another s All the future changes to the configuration that are Glassfish-specific and can be done through ``asadmin`` should now go into ``scripts/install/glassfish-setup.sh``. +FIXME: Add a "dev" mode to the installer to allow REST Assured tests to be run. For now, refer to the steps in the :doc:`testing` section. + Troubleshooting --------------- diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 18263656a3f..9003113e447 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -2,64 +2,129 @@ Testing ======= -For testing in Dataverse, JUnit passes along test information to utilities like `Coveralls `_ which builds a quantifiable report of how much code coverage (% of code with verifiable tests) the project’s codebase has. We encourage test-driven development to help ensure new code has functional backend logic that can be repeated and help diagnose issues within the app. +In order to keep our codebase healthy, the Dataverse project encourages developers to write automated tests in the form of unit tests and integration tests. We also welcome ideas for how to improve our automated testing. .. contents:: |toctitle| :local: +The Health of a Codebase +------------------------ + +Before we dive into the nut and bolts of testing, let's back up for a moment and think about why we write automated tests in the first place. Writing automated tests is an investment and leads to better quality software. Counterintuitively, writing tests and executing them regularly allows a project to move faster. Martin Fowler explains this well while talking about the health of a codebase: + + "This is an economic judgment. Several times, many times, I run into teams that say something like, 'Oh well. Management isn't allowing us to do a quality job here because it will slow us down. And we've appealed to management and said we need to put more quality in the code, but they've said no, we need to go faster instead.' And my comment to that is well, as soon as you’re framing it in terms of code quality versus speed, you've lost. Because the whole point of refactoring is to go faster. + + "And this is why I quite like playing a bit more with the metaphor as the health of a codebase. If you keep yourself healthy then you'll be able to run faster. But if you just say, 'Well, I want to run a lot so I'm therefore going to run a whole load all the time and not eat properly and not pay attention about this shooting pain going up my leg,' then you’re not going to be able to run quickly very long. **You have to pay attention to your health. And same with the codebase. You have to continuously say, 'How do we keep it in a healthy state? Then we can go fast,' because we’re running marathons here with codebases. And if we neglect that internal quality of the codebase, it hits you surprisingly fast.**" + + --Martin Fowler at https://devchat.tv/ruby-rogues/178-rr-book-club-refactoring-ruby-with-martin-fowler + +Testing in Depth +---------------- + +`Security in depth `_ might mean that your castle has a moat as well as high walls. Likewise, when testing, you should consider testing a various layers of the stack using both unit tests and integration tests. + +When writing tests, you may find it helpful to first map out which functions of your code you want to test, and then write a functional unit test for each which can later comprise a larger integration test. + Unit Tests ---------- -Write them. `JUnit `_ is your friend. Creating unit tests for your code is a helpful way to test what you’ve built piece by piece. - -You may notice when code is pushed, an automated Coveralls bot will let you know if it has increased or decreased code coverage. +Creating unit tests for your code is a helpful way to test what you've built piece by piece. + +Unit tests can be executed without runtime dependencies on PostgreSQL, Solr, or any other external system. They are the lowest level of testing and are executed constantly on developers' laptops as part of the build process and via continous integration services in the cloud. + +A unit test should execute an operation of your code in a controlled fashion. You must make an assertion of what the expected response gives back. It's important to test optimistic output and assertions, the "happy path", as well as unexpected intput that leads to failure conditions. Know how your program should handle anticipated errors/exceptions and confirm with your test(s) that it does so properly. + +Unit Test Automation Overview +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We use a variety of tools to write, execute, and measure the code coverage of unit tests, including Maven, JUnit, Jacoco, GitHub, Travis, Coveralls. We'll explain the role of each tool below, but here's an overview of what you can expect from the automation we've set up in terms of + +As you prepare to make a pull request, as described in the :doc:`version-control` section, you will be working on a new branch you create from the "develop" branch. Let's say your branch is called ``1012-private-url``. As you work, you are constantly invoking Maven to build the war file. When you do a "clean and build" in Netbeans, Maven runs all the unit tests (anything ending with ``Test.java``) and the runs the results through a tool called Jacoco that calculates code coverage. When you push your branch to GitHub and make a pull request, a web service called Travis CI runs Maven and Jacoco on your branch and pushes the results to Coveralls, which is a web service that tracks changes to code coverage over time. + +To make this more concrete, observe that https://github.com/IQSS/dataverse/pull/3111 has comments from a GitHub user called ``coveralls`` saying things like "Coverage increased (+0.5%) to 5.547% when pulling dd6ceb1 on 1012-private-url into be5b26e on develop." Clicking on the comment should lead you to a URL such as https://coveralls.io/builds/7013870 which shows how code coverage has gone up or down and that page links to a page such as https://travis-ci.org/IQSS/dataverse/builds/144840165 which shows the build on the Travis side that pushed the results ton Coveralls. + +The main take away should be that we care about unit testing enough to measure the changes to code coverage over time using automation. Now let's talk about how you can help keep our code coverage up by writing unit tests with JUnit. + +Writing Unit Tests with JUnit +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We are aware that there are newer testing tools such as TestNG, but we use `JUnit `_ because it's tried and true. + +If writing tests is new to you, poke around existing unit tests which all end in ``Test.java`` and live under ``src/test``. Each test is annotated with ``@Test`` and should have at least one assertion which specifies the expected result. In Netbeans, you can run all the tests in it by clicking "Run" -> "Test File". From the test file, you should be able to navigate to the code that's being tested by right-clicking on the file and clicking "Navigate" -> "Go to Test/Tested class". Likewise, from the code, you should be able to use the same "Navigate" menu to go to the tests. -- If code coverage is increasing: good! You made some test(s). -- If code coverage is decreasing: write some unit tests to show the world just how good your code is! - -A unit test should execute an operation of your code in a controlled fashion. You must make an assertion of what the expected response gives back. It's important to test optimistic output and assertions, but also test controlled failures such as 400 errors or other issues your API may encounter (to simulate human or machine error) with assertions stating such. +Refactoring Code to Make It Unit-Testable +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Know how your program should handle anticipated errors/exceptions and confirm with your test(s) that it does so properly. +Existing code is not necessarily written in a way that lends itself to easy testing. Generally speaking, it is difficult to write unit tests for both JSF "backing" beans (which end in ``Page.java``) and "service" beans (which end in ``Service.java``) because they require the database to be running in order to test them. If service beans can be exercised via API they can be tested with integration tests (described below) but a good technique for making the logic testable it to move code to "util beans" (which end in ``Util.java``) that operate on Plain Old Java Objects (POJOs). ``PrivateUrlUtil.java`` is a good example of moving logic from ``PrivateUrlServiceBean.java`` to a "util" bean to make the code testable. + +Observing Changes to Code Coverage +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once you've written some tests, you're probably wondering how much you've helped to increase the code coverage. In Netbeans, do a "clean and build." Then, under the "Projects" tab, right-click "dataverse" and click "Code Coverage" -> "Show Report". For each Java file you have open, you should be able to see the percentage of code that is covered by tests and every line in the file should be either green or red. Green indicates that the line is being exercised by a unit test and red indicates that it is not. + +In addition to seeing code coverage in Netbeans, you can also see code coverage reports by opening ``target/site/jacoco/index.html`` in your browser. Integration Tests ----------------- -Write them. `REST-assured `_ and `Selenium `_ are recommended. A Jenkins environment is available for automated testing: https://build.hmdc.harvard.edu:8443 - -Integration tests help ensure that your code gets along well with the rest of Dataverse. It’s good to test the different functions of your code with the different parts of your Dataverse which the feature may interact with. +Unit tests are fantastic for low level testing of logic but aren't especially real-world because they do not exercise Dataverse as it runs in production with a database and other runtime dependencies. We test in depth by also writing integration tests to exercise a running system. -These tests already live in the code base under ``/dataverse/src/test/java/edu/harvard/iq/dataverse/api/`` in your respective Dataverse directory. When you write an integration test of your own, it should be saved here. - -To execute existing integration tests on your local Dataverse, a helpful command line tool to use is `Maven `_. You should have Maven installed as per the `Development Environment `_ guide, but if not it’s easily done via Homebrew: ``brew install maven``. +Unfortunately, the term "integration tests" can mean different things to different people. For our purposes, an integration test has the following qualities: -Once installed, you may run commands with ``mvn [options] [] []``. +- Integration tests exercise Dataverse APIs. +- Integration tests are not automatically on developers' laptops. +- Integration tests operate on an installation of Dataverse that is running and able to talk to both PostgreSQL and Solr. +- Integration tests are written using REST Assured. -+ If you want to run just one particular API test, it’s as easy as you think: +Getting Set Up to Run REST Assured Tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``mvn test -Dtest=FileRecordJobIT`` +Unit tests are run automatically on every build but dev environments and servers requires special setup to run REST Assured tests. In short, Dataverse needs to be placed into an insecure mode that allows arbitrary users and datasets to be created and destroyed. This differs greatly from the out of the box behavior of Dataverse, which we strive to keep secure for sysadming installing the software for their institutions in a production environment. -+ To run more than one test at a time, separate by commas: +The :doc:`dev-environment` section currently refers developers here for advice on getting set up to run REST Assured tests but we'd like to add some sort of "dev" flag to the installer to put Dataverse in "insecure" mode, with lots of scary warnings that this dev mode should not be used in production. - ``mvn test -Dtest=FileRecordJobIT,ConfirmEmailIT`` +The Burrito Key +^^^^^^^^^^^^^^^ -+ To run any test(s) on a particular domain, replace localhost:8080 with desired domain name: +For reasons that have been lost to the mists of time, Dataverse really wants you to to have a burrito. Specifically, if you're trying to run REST Assured tests and see the error "Dataverse config issue: No API key defined for built in user management", you must run the following curl command (or make an equivalent change to your database): - ``mvn test -Dtest=FileMetadataIT -Ddataverse.test.baseurl='http://localhost:8080'`` +``curl -X PUT -d 'burrito' http://localhost:8080/api/admin/settings/BuiltinUsers.KEY`` -**Note:** if you’re trying to run the test suite and receive the error "Dataverse config issue: No API key defined for built in user management" just run the curl command below. +Without this "burrito" key in place, REST Assured will not be able to create users. We create users to create objects we want to test, such as dataverses, datasets, and files. -``curl -X PUT -d 'burrito' http://localhost:8080/api/admin/settings/BuiltinUsers.KEY`` +Root Dataverse Permissions +^^^^^^^^^^^^^^^^^^^^^^^^^^ -Writing Tests -------------- +In your browser, log in as dataverseAdmin (password: admin) and click the "Edit" button for your root dataverse. Navigate to Permissions, then the Edit Access button. Under "Who can add to this dataverse?" choose "Anyone with a dataverse account can add sub dataverses" if it isn't set to this already. -When writing tests, you may find it helpful to first map out which functions of your code you want to test, and then write a functional unit test for each which can later comprise a larger integration test. +Alternatively, this same step can be done with this script: ``scripts/search/tests/grant-authusers-add-on-root`` + +Publish Root Dataverse +^^^^^^^^^^^^^^^^^^^^^^ + +The root dataverse must be published for some of the REST Assured tests to run. + +dataverse.siteUrl +^^^^^^^^^^^^^^^^^ + +When run locally (as opposed to a remote server), some of the REST Assured tests require the ``dataverse.siteUrl`` JVM option to be set to ``http://localhost:8080``. See "JVM Options" under the :doc:`/installation/config` section of the Installation Guide for advice changing JVM options. First you should check to check your JVM options with: -Recall that integration tests should live in the ``/dataverse/src/test/java/edu/harvard/iq/dataverse/api`` directory, or if you’re on Netbeans then navigate to the project’s Test Packages → edu.harvard.iq.dataverse.api and place your testnameIT.java file there! +``asadmin list-jvm-options | egrep 'dataverse|doi'`` -If writing tests is new to you: in NetBeans, poke around existing tests. If you read through any you may notice each test within a file (bearing the @Test prefix) has an assertion which specifies the expected result (expected, from the Run menu, select Test File. +If ``dataverse.siteUrl`` is absent, you can add it with: -The output in the file’s test window (beside the console) corresponds to what each line of each @Test block is doing. Many existing tests provide readable Json output, and how verbose the output is up to you as a developer. +``asadmin create-jvm-options "-Ddataverse.siteUrl=http\://localhost\:8080"`` + +Identifier Generation +^^^^^^^^^^^^^^^^^^^^^ + +``DatasetsIT.java`` exercises the a feature where the "identifier" of a DOI can be a digit and requires a sequence to be added to your database. See ``:IdentifierGenerationStyle`` under the :doc:`/installation/config` section for adding this sequence to your installation of PostgreSQL. + + +Writing Integration Tests with REST Assured +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before writing any new REST Assured tests, you should get the tests to pass in an existing REST Assured test file. ``BuiltinUsersIT.java`` is relatively small and requires less setup than other test files. You do not have to reinvent the wheel. There are many useful methods you can call in your own tests -- especially within UtilIT.java -- when you need your test to create and/or interact with generated accounts, files, datasets, etc. Similar methods can subsequently delete them to get them out of your way as desired before the test has concluded. @@ -73,3 +138,70 @@ Remember, it’s only a test (and it's not graded)! Some guidelines to bear in m * A useful resource would be `HTTP status codes `_ - Let the code do the labor; automate everything that happens when you run your test file. - Just as with any development, if you’re stuck: ask for help! + +To execute existing integration tests on your local Dataverse, a helpful command line tool to use is `Maven `_. You should have Maven installed as per the `Development Environment `_ guide, but if not it’s easily done via Homebrew: ``brew install maven``. + +Once installed, you may run commands with ``mvn [options] [] []``. + ++ If you want to run just one particular API test, it’s as easy as you think: + + ``mvn test -Dtest=FileRecordJobIT`` + ++ To run more than one test at a time, separate by commas: + + ``mvn test -Dtest=FileRecordJobIT,ConfirmEmailIT`` + ++ To run any test(s) on a particular domain, replace localhost:8080 with desired domain name: + + ``mvn test -Dtest=FileMetadataIT -Ddataverse.test.baseurl='http://localhost:8080'`` + +The Phoenix Server +------------------ + +A server at http://phoenix.dataverse.org has been set up to test the latest code from the develop branch. Testing done using chained builds of Jenkins jobs: + +- A war file is built from the latest code in develop: https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-build-develop/ +- The resulting war file is depoyed to the Phoenix server: https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-deploy-develop/ +- REST Assured Tests are run across the wire from the Jenkins server to the Phoenix server https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/ + +Future Work +----------- + +We'd like to make improvements to our automated testing. See also a [thread on the mailing list](https://groups.google.com/forum/#!topic/dataverse-community/X8OrRWbPimA) asking for ideas from the community and discussion at https://github.com/IQSS/dataverse/issues/2746 + +Future Work on Unit Tests +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Review pull requests from @bencomp for ideas for approaches to testing: https://github.com/IQSS/dataverse/pulls?q=is%3Apr+author%3Abencomp +- Come up with a way to test commands: http://irclog.iq.harvard.edu/dataverse/2015-11-04#i_26750 +- Test EJBs using Arquillian, embedded Glassfish, or similar. @bmckinney kicked the tires on Arquillian at https://github.com/bmckinney/bio-dataverse/commit/2f243b1db1ca704a42cd0a5de329083763b7c37a + +Future Work on Integration Tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Automate testing of the Python client: https://github.com/IQSS/dataverse-client-python/issues/10 +- Work with @leeper on testing the R client: https://github.com/IQSS/dataverse-client-r +- Review and attempt to implement "API Test Checklist" from @kcondon at https://docs.google.com/document/d/199Oq1YwQ4pYCguaeW48bIN28QAitSk63NbPYxJHCCAE/edit?usp=sharing +- Attempt to use @openscholar approach for running integration tests using Travis https://github.com/openscholar/openscholar/blob/SCHOLAR-3.x/.travis.yml (probably requires using Ubuntu rather than CentOS) +- Generate code coverage reports for **integration** tests: https://github.com/pkainulainen/maven-examples/issues/3 and http://www.petrikainulainen.net/programming/maven/creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoco-maven-plugin/ +- Consistent logging of API Tests. Show test name at the beginning and end and status codes returned. + +Browser-Based Testing +~~~~~~~~~~~~~~~~~~~~~ + +- Revisit Selenium/Open Sauce: https://github.com/IQSS/dataverse/commit/8a26404 and https://saucelabs.com/u/esodvn and https://saucelabs.com/u/wdjs and http://sauceio.com/index.php/2013/05/a-browser-matrix-widget-for-the-open-source-community/ + +Installation Testing +~~~~~~~~~~~~~~~~~~~~ + +- Run `vagrant up` on a server to test the installer: http://guides.dataverse.org/en/latest/developers/tools.html#vagrant . We haven't been able to get this working in Travis: https://travis-ci.org/IQSS/dataverse/builds/96292683 . Perhaps it would be possible to use AWS as a provider from Vagrant judging from https://circleci.com/gh/critical-alert/circleci-vagrant/6 +- Work with @lwo to automate testing of https://github.com/IQSS/dataverse-puppet . Consider using Travis: https://github.com/IQSS/dataverse-puppet/issues/10 +- Work with @donsizemore to automate testing of https://github.com/IQSS/dataverse-ansible with Travis or similar. + +Load/Perfomance Testing +~~~~~~~~~~~~~~~~~~~~~~~ + +- Run stress tests on a period basis: https://github.com/IQSS/dataverse-helper-scripts/tree/master/src/stress_tests +- Marcel Duran created a command-line wrapper for the WebPagetest API that can be used to test performance in your continuous integration pipeline (TAP, Jenkins, Travis-CI, etc): https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs#jenkins-integration +- Documentation +- Create top-down checklist, building off the spreadsheet at https://github.com/IQSS/dataverse/issues/3358#issuecomment-256400776 From a47b0c2b5c6d3ee1a8be552dafd9e8ca469bdbff Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Mon, 12 Jun 2017 18:50:08 -0400 Subject: [PATCH 11/27] Proofread of Testing and Dev Environment pages [ref: #3431] Made minor typo/style fixes to testing.rst. Also looked over dev-environment.rst and it's all good. --- .../source/developers/testing.rst | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 9003113e447..085e629118b 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -32,18 +32,18 @@ Creating unit tests for your code is a helpful way to test what you've built pie Unit tests can be executed without runtime dependencies on PostgreSQL, Solr, or any other external system. They are the lowest level of testing and are executed constantly on developers' laptops as part of the build process and via continous integration services in the cloud. -A unit test should execute an operation of your code in a controlled fashion. You must make an assertion of what the expected response gives back. It's important to test optimistic output and assertions, the "happy path", as well as unexpected intput that leads to failure conditions. Know how your program should handle anticipated errors/exceptions and confirm with your test(s) that it does so properly. +A unit test should execute an operation of your code in a controlled fashion. You must make an assertion of what the expected response gives back. It's important to test optimistic output and assertions (the "happy path"), as well as unexpected input that leads to failure conditions. Know how your program should handle anticipated errors/exceptions and confirm with your test(s) that it does so properly. Unit Test Automation Overview ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -We use a variety of tools to write, execute, and measure the code coverage of unit tests, including Maven, JUnit, Jacoco, GitHub, Travis, Coveralls. We'll explain the role of each tool below, but here's an overview of what you can expect from the automation we've set up in terms of +We use a variety of tools to write, execute, and measure the code coverage of unit tests, including Maven, JUnit, Jacoco, GitHub, Travis, and Coveralls. We'll explain the role of each tool below, but here's an overview of what you can expect from the automation we've set up. As you prepare to make a pull request, as described in the :doc:`version-control` section, you will be working on a new branch you create from the "develop" branch. Let's say your branch is called ``1012-private-url``. As you work, you are constantly invoking Maven to build the war file. When you do a "clean and build" in Netbeans, Maven runs all the unit tests (anything ending with ``Test.java``) and the runs the results through a tool called Jacoco that calculates code coverage. When you push your branch to GitHub and make a pull request, a web service called Travis CI runs Maven and Jacoco on your branch and pushes the results to Coveralls, which is a web service that tracks changes to code coverage over time. -To make this more concrete, observe that https://github.com/IQSS/dataverse/pull/3111 has comments from a GitHub user called ``coveralls`` saying things like "Coverage increased (+0.5%) to 5.547% when pulling dd6ceb1 on 1012-private-url into be5b26e on develop." Clicking on the comment should lead you to a URL such as https://coveralls.io/builds/7013870 which shows how code coverage has gone up or down and that page links to a page such as https://travis-ci.org/IQSS/dataverse/builds/144840165 which shows the build on the Travis side that pushed the results ton Coveralls. +To make this more concrete, observe that https://github.com/IQSS/dataverse/pull/3111 has comments from a GitHub user called ``coveralls`` saying things like "Coverage increased (+0.5%) to 5.547% when pulling dd6ceb1 on 1012-private-url into be5b26e on develop." Clicking on the comment should lead you to a URL such as https://coveralls.io/builds/7013870 which shows how code coverage has gone up or down. That page links to a page such as https://travis-ci.org/IQSS/dataverse/builds/144840165 which shows the build on the Travis side that pushed the results ton Coveralls. -The main take away should be that we care about unit testing enough to measure the changes to code coverage over time using automation. Now let's talk about how you can help keep our code coverage up by writing unit tests with JUnit. +The main takeaway should be that we care about unit testing enough to measure the changes to code coverage over time using automation. Now let's talk about how you can help keep our code coverage up by writing unit tests with JUnit. Writing Unit Tests with JUnit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -67,7 +67,7 @@ In addition to seeing code coverage in Netbeans, you can also see code coverage Integration Tests ----------------- -Unit tests are fantastic for low level testing of logic but aren't especially real-world because they do not exercise Dataverse as it runs in production with a database and other runtime dependencies. We test in depth by also writing integration tests to exercise a running system. +Unit tests are fantastic for low level testing of logic but aren't especially real-world-applicable because they do not exercise Dataverse as it runs in production with a database and other runtime dependencies. We test in-depth by also writing integration tests to exercise a running system. Unfortunately, the term "integration tests" can mean different things to different people. For our purposes, an integration test has the following qualities: @@ -79,9 +79,9 @@ Unfortunately, the term "integration tests" can mean different things to differe Getting Set Up to Run REST Assured Tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Unit tests are run automatically on every build but dev environments and servers requires special setup to run REST Assured tests. In short, Dataverse needs to be placed into an insecure mode that allows arbitrary users and datasets to be created and destroyed. This differs greatly from the out of the box behavior of Dataverse, which we strive to keep secure for sysadming installing the software for their institutions in a production environment. +Unit tests are run automatically on every build, but dev environments and servers require special setup to run REST Assured tests. In short, Dataverse needs to be placed into an insecure mode that allows arbitrary users and datasets to be created and destroyed. This differs greatly from the out-of-the-box behavior of Dataverse, which we strive to keep secure for sysadmins installing the software for their institutions in a production environment. -The :doc:`dev-environment` section currently refers developers here for advice on getting set up to run REST Assured tests but we'd like to add some sort of "dev" flag to the installer to put Dataverse in "insecure" mode, with lots of scary warnings that this dev mode should not be used in production. +The :doc:`dev-environment` section currently refers developers here for advice on getting set up to run REST Assured tests, but we'd like to add some sort of "dev" flag to the installer to put Dataverse in "insecure" mode, with lots of scary warnings that this dev mode should not be used in production. The Burrito Key ^^^^^^^^^^^^^^^ @@ -118,7 +118,7 @@ If ``dataverse.siteUrl`` is absent, you can add it with: Identifier Generation ^^^^^^^^^^^^^^^^^^^^^ -``DatasetsIT.java`` exercises the a feature where the "identifier" of a DOI can be a digit and requires a sequence to be added to your database. See ``:IdentifierGenerationStyle`` under the :doc:`/installation/config` section for adding this sequence to your installation of PostgreSQL. +``DatasetsIT.java`` exercises the feature where the "identifier" of a DOI can be a digit and requires a sequence to be added to your database. See ``:IdentifierGenerationStyle`` under the :doc:`/installation/config` section for adding this sequence to your installation of PostgreSQL. Writing Integration Tests with REST Assured @@ -158,16 +158,16 @@ Once installed, you may run commands with ``mvn [options] [] ['_ asking for ideas from the community, and discussion at 'this GitHub issue. '_ Future Work on Unit Tests ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -194,7 +194,7 @@ Browser-Based Testing Installation Testing ~~~~~~~~~~~~~~~~~~~~ -- Run `vagrant up` on a server to test the installer: http://guides.dataverse.org/en/latest/developers/tools.html#vagrant . We haven't been able to get this working in Travis: https://travis-ci.org/IQSS/dataverse/builds/96292683 . Perhaps it would be possible to use AWS as a provider from Vagrant judging from https://circleci.com/gh/critical-alert/circleci-vagrant/6 +- Run `vagrant up` on a server to test the installer: http://guides.dataverse.org/en/latest/developers/tools.html#vagrant . We haven't been able to get this working in Travis: https://travis-ci.org/IQSS/dataverse/builds/96292683 . Perhaps it would be possible to use AWS as a provider from Vagrant judging from https://circleci.com/gh/critical-alert/circleci-vagrant/6 . - Work with @lwo to automate testing of https://github.com/IQSS/dataverse-puppet . Consider using Travis: https://github.com/IQSS/dataverse-puppet/issues/10 - Work with @donsizemore to automate testing of https://github.com/IQSS/dataverse-ansible with Travis or similar. From 51ecfd9e80ef28dceabe8dd9a9dbfa8fe65966ed Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Tue, 13 Jun 2017 14:50:29 -0400 Subject: [PATCH 12/27] Typo fix [ref: #3431] Fixed a header typo I'd missed before. --- doc/sphinx-guides/source/developers/testing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 085e629118b..d930565b810 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -198,7 +198,7 @@ Installation Testing - Work with @lwo to automate testing of https://github.com/IQSS/dataverse-puppet . Consider using Travis: https://github.com/IQSS/dataverse-puppet/issues/10 - Work with @donsizemore to automate testing of https://github.com/IQSS/dataverse-ansible with Travis or similar. -Load/Perfomance Testing +Load/Performance Testing ~~~~~~~~~~~~~~~~~~~~~~~ - Run stress tests on a period basis: https://github.com/IQSS/dataverse-helper-scripts/tree/master/src/stress_tests From 24340e9a84dc4b3f0669f3814676880fc5a5e464 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 14 Jun 2017 14:39:03 -0400 Subject: [PATCH 13/27] fix glassfish link, smtp trouble, non-interactive mode #3431 --- doc/sphinx-guides/source/developers/dev-environment.rst | 7 ++++++- doc/sphinx-guides/source/developers/testing.rst | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index fe9f479eca0..8db4b325dd9 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -27,7 +27,7 @@ Glassfish As a `Java Enterprise Edition `_ 7 (Java EE 7) application, Dataverse requires an applications server to run. -Glassfish 4.1 is required (not 4.1.1 until https://github.com/IQSS/dataverse/issues/2628 is resolved), which can be downloaded from http://glassfish.java.net . If you have downloaded Glassfish as part of a Netbeans bundle, you can manually add the proper version by clicking "Tools", "Servers", "Add Server". +Glassfish 4.1 is required (not 4.1.1 until https://github.com/IQSS/dataverse/issues/2628 is resolved), which can be downloaded from https://javaee.github.io/glassfish . If you have downloaded Glassfish as part of a Netbeans bundle, you can manually add the proper version by clicking "Tools", "Servers", "Add Server". PostgreSQL ~~~~~~~~~~ @@ -158,6 +158,11 @@ Once some dataverses, datasets, and files have been created and indexed, you can Run Installer ~~~~~~~~~~~~~ +Please note the following: + +- If you have trouble with the SMTP server, consider editing the installer script to disable the SMTP check. +- Rather than running the installer in "interactive" mode, it's possible to put the values in a file. See "non-interactive mode" in the :doc:`/installation/installation-main` section of the Installation Guide. + Now that you have all the prerequisites in place, you need to configure the environment for the Dataverse app - configure the database connection, set some options, etc. We have an installer script that should do it all for you. Again, assuming that the clone on the Dataverse repository was retrieved using NetBeans and that it is saved in the path ~/NetBeansProjects: ``cd ~/NetBeansProjects/dataverse/scripts/installer`` diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index d930565b810..ef35664269b 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -199,7 +199,7 @@ Installation Testing - Work with @donsizemore to automate testing of https://github.com/IQSS/dataverse-ansible with Travis or similar. Load/Performance Testing -~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~ - Run stress tests on a period basis: https://github.com/IQSS/dataverse-helper-scripts/tree/master/src/stress_tests - Marcel Duran created a command-line wrapper for the WebPagetest API that can be used to test performance in your continuous integration pipeline (TAP, Jenkins, Travis-CI, etc): https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs#jenkins-integration From f4f0feb6001cb03ff2f50b52b5a2a843a447daca Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 14 Jun 2017 16:49:18 -0400 Subject: [PATCH 14/27] mention dos2unix is needed on Windows for Vagrant #3431 --- doc/sphinx-guides/source/developers/tools.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/sphinx-guides/source/developers/tools.rst b/doc/sphinx-guides/source/developers/tools.rst index f094fd5fb9b..6064e1dcb3f 100755 --- a/doc/sphinx-guides/source/developers/tools.rst +++ b/doc/sphinx-guides/source/developers/tools.rst @@ -67,6 +67,8 @@ Vagrantfile). Please note that running ``vagrant up`` for the first time should run the ``downloads/download.sh`` script for you to download required software such as Glassfish and Solr and any patches. However, these dependencies change over time so it's a place to look if ``vagrant up`` was working but later fails. +On Windows if you see an error like ``/usr/bin/perl^M: bad interpreter`` you might need to run ``dos2unix`` on the installation scripts. + MSV +++ From 2f43b35652de35513dd5a403444f3824ab8a6572 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 16 Jun 2017 11:14:59 -0400 Subject: [PATCH 15/27] mention read-only team and spreadsheet of contributors #3431 --- doc/sphinx-guides/source/developers/version-control.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index 03e8ea2a4e6..1c5f924a7e5 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -62,6 +62,8 @@ For guidance on which issue to work on, please ask! Also, see https://github.com Let's say you want to tackle https://github.com/IQSS/dataverse/issues/3728 which points out a typo in a page of Dataverse's documentation. +If you tell us your GitHub username we are happy to add you to the "read only" team at https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that we can assign the issue to you while you're working on it. You can also tell us if you'd like to be added to the Dataverse Community Contributors spreadsheet at https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing . + Create a New Branch off the develop Branch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 7076bdcc0ee1743d73a356f5ee542f7d76ea17f3 Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Mon, 19 Jun 2017 11:45:44 -0400 Subject: [PATCH 16/27] Changed formatting of Issue Number and Title Changed formatting to fix issues with issue syntax and Waffle. --- PULL_REQUEST_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PULL_REQUEST_TEMPLATE.md b/PULL_REQUEST_TEMPLATE.md index 91a2fc0d5f5..37f2ef7bd68 100644 --- a/PULL_REQUEST_TEMPLATE.md +++ b/PULL_REQUEST_TEMPLATE.md @@ -4,7 +4,7 @@ Welcome! New contributors should at least glance at [CONTRIBUTING.md](/CONTRIBUT ## Related Issues -- connects to [#issue number]: [issue title] +- connects to #ISSUE_NUMBER: ISSUE_TITLE ## Pull Request Checklist From fa95e12227538986cf790bda6b822bfbd214884a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 09:18:54 -0400 Subject: [PATCH 17/27] mention asadmin deployment in dev guide #3907 --- .../source/developers/dev-environment.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index e5f31dacd89..7fcb5a978be 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -177,6 +177,21 @@ All the future changes to the configuration that are Glassfish-specific and can FIXME: Add a "dev" mode to the installer to allow REST Assured tests to be run. For now, refer to the steps in the :doc:`testing` section. +Iterating on Code and Redeploying +--------------------------------- + +Deploy on Save +~~~~~~~~~~~~~~ + +Out of the box, Netbeans is configured to "Deploy on Save" which means that if you save any changes to project files such as Java classes, XHTML files, or "bundle" files (i.e. Bundle.properties), the project is recompiled and redeployed to Glassfish automatically. This behavior works well for many of us but if you don't like it, you can turn it off by right-clicking "dataverse" under the Projects tab, clicking "Run" and unchecking "Deploy on Save". + +Note that it's possible to deploy the war file using the ``asadmin`` command. The :doc:`/installation/installation-main` section of the Installation Guide has more information on this topic. + +Netbeans Connector Chrome Extension +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For faster iteration while working on JSF pages, it is highly recommended that you install the Netbeans Connector Chrome Extension listed in the :doc:`tools` section. When you save XHTML or CSS files, you will see the changes immediately. + Troubleshooting --------------- From 06bd186436861e9a62148429d3132548e886471b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 09:39:57 -0400 Subject: [PATCH 18/27] add previous and next links to dev guide #2163 --- doc/sphinx-guides/source/developers/coding-style.rst | 4 ++++ doc/sphinx-guides/source/developers/debugging.rst | 4 ++++ doc/sphinx-guides/source/developers/dev-environment.rst | 4 ++++ doc/sphinx-guides/source/developers/documentation.rst | 4 ++++ doc/sphinx-guides/source/developers/geospatial.rst | 4 ++++ doc/sphinx-guides/source/developers/intro.rst | 4 ++++ doc/sphinx-guides/source/developers/making-releases.rst | 4 ++++ doc/sphinx-guides/source/developers/selinux.rst | 4 ++++ doc/sphinx-guides/source/developers/testing.rst | 4 ++++ doc/sphinx-guides/source/developers/tools.rst | 4 ++++ doc/sphinx-guides/source/developers/unf/index.rst | 6 +++++- doc/sphinx-guides/source/developers/version-control.rst | 4 ++++ 12 files changed, 49 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/coding-style.rst b/doc/sphinx-guides/source/developers/coding-style.rst index 6682a99ddd4..fc3d3840946 100755 --- a/doc/sphinx-guides/source/developers/coding-style.rst +++ b/doc/sphinx-guides/source/developers/coding-style.rst @@ -10,3 +10,7 @@ Like all development teams, the `Dataverse developers at IQSS `_ guide. You will need Python and a few other prerequisites. As mentioned under "Architecture and Components" in the :doc:`/installation/prep` section of the Installation Guide, Geoconnect is an optional component of Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature. + +---- + +Previous: :doc:`intro` | Next: :doc:`version-control` diff --git a/doc/sphinx-guides/source/developers/documentation.rst b/doc/sphinx-guides/source/developers/documentation.rst index 472e44a01b1..fcca7aa0558 100755 --- a/doc/sphinx-guides/source/developers/documentation.rst +++ b/doc/sphinx-guides/source/developers/documentation.rst @@ -74,3 +74,7 @@ Every non-index page should use the following code to display a table of content :local: This code should be placed below any introductory text/images and directly above the first subheading, much like a Wikipedia page. + +---- + +Previous: :doc:`testing` | Next: :doc:`debugging` diff --git a/doc/sphinx-guides/source/developers/geospatial.rst b/doc/sphinx-guides/source/developers/geospatial.rst index d43e23cbefc..45d2cd3a368 100644 --- a/doc/sphinx-guides/source/developers/geospatial.rst +++ b/doc/sphinx-guides/source/developers/geospatial.rst @@ -175,3 +175,7 @@ Saving Join Target Information to Geoconnect Database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``get_latest_jointarget_information()`` in ``utils.py`` retrieves recent JoinTarget Information from the database. (See the `utils code in GitHub `_.) + +---- + +Previous: :doc:`unf/index` | Next: :doc:`selinux` diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst index 0cf695b6ea6..b8bf46b544f 100755 --- a/doc/sphinx-guides/source/developers/intro.rst +++ b/doc/sphinx-guides/source/developers/intro.rst @@ -49,3 +49,7 @@ As a developer, you also may be interested in these projects related to Datavers - Third party apps - make use of Dataverse APIs: :doc:`/api/apps` - chat.dataverse.org - chat interface for Dataverse users and developers: https://github.com/IQSS/chat.dataverse.org - [Your project here] :) + +---- + +Next: :doc:`dev-environment` diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index 00f5358ae57..23cc5c93e45 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -43,3 +43,7 @@ Publish Release --------------- Click the "Publish release" button. + +---- + +Previous: :doc:`coding-style` | Next: :doc:`tools` diff --git a/doc/sphinx-guides/source/developers/selinux.rst b/doc/sphinx-guides/source/developers/selinux.rst index c860d123e6a..582510c5847 100644 --- a/doc/sphinx-guides/source/developers/selinux.rst +++ b/doc/sphinx-guides/source/developers/selinux.rst @@ -109,3 +109,7 @@ Once your updated SELinux rules are in place, try logging in with Shibboleth aga Keep iterating until it works and then create a pull request based on your updated file. Good luck! Many thanks to Bill Horka from IQSS for his assistance in explaining how to construct a SELinux Type Enforcement (TE) file! + +---- + +Previous: :doc:`geospatial` diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index ef35664269b..99e2267a9a7 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -205,3 +205,7 @@ Load/Performance Testing - Marcel Duran created a command-line wrapper for the WebPagetest API that can be used to test performance in your continuous integration pipeline (TAP, Jenkins, Travis-CI, etc): https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs#jenkins-integration - Documentation - Create top-down checklist, building off the spreadsheet at https://github.com/IQSS/dataverse/issues/3358#issuecomment-256400776 + +---- + +Previous: :doc:`version-control` | Next: :doc:`documentation` diff --git a/doc/sphinx-guides/source/developers/tools.rst b/doc/sphinx-guides/source/developers/tools.rst index 6064e1dcb3f..e1d2f5b9fa0 100755 --- a/doc/sphinx-guides/source/developers/tools.rst +++ b/doc/sphinx-guides/source/developers/tools.rst @@ -80,3 +80,7 @@ MSV start parsing a grammar. validating ddi.xml the document is valid. + +---- + +Previous: :doc:`making-releases` | Next: :doc:`unf/index` diff --git a/doc/sphinx-guides/source/developers/unf/index.rst b/doc/sphinx-guides/source/developers/unf/index.rst index 589c6870f9d..7759a64383f 100644 --- a/doc/sphinx-guides/source/developers/unf/index.rst +++ b/doc/sphinx-guides/source/developers/unf/index.rst @@ -39,4 +39,8 @@ New York: John Wiley. unf-v3 unf-v5 - unf-v6 \ No newline at end of file + unf-v6 + +---- + +Previous: :doc:`/developers/tools` | Next: :doc:`/developers/geospatial` diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index 1c5f924a7e5..f95054eee87 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -124,3 +124,7 @@ If you'd like to resolve the merge conflicts yourself, here are some steps to do **In GitHub Issues:** 11. Leave a comment for the Dataverse team that you have resolved the merge conflicts. + +---- + +Previous: :doc:`dev-environment` | Next: :doc:`testing` From a299f8acc28c94387e93cb6d37c610b78f26c326 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 09:55:10 -0400 Subject: [PATCH 19/27] add section on testing commands #3431 #1797 --- doc/sphinx-guides/source/developers/testing.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 99e2267a9a7..229ed63ac3e 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -64,6 +64,17 @@ Once you've written some tests, you're probably wondering how much you've helped In addition to seeing code coverage in Netbeans, you can also see code coverage reports by opening ``target/site/jacoco/index.html`` in your browser. +Testing Commands +^^^^^^^^^^^^^^^^ + +You might find studying the following test classes helpful in writing tests for commands: + +- CreatePrivateUrlCommandTest.java +- DeletePrivateUrlCommandTest.java +- GetPrivateUrlCommandTest.java + +In addition, there is a writeup on "The Testable Command" at https://github.com/IQSS/dataverse/blob/develop/doc/theTestableCommand/TheTestableCommand.md . + Integration Tests ----------------- From dccd9b0b84e22c9f7bda24859e4f124059c26daa Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 10:36:25 -0400 Subject: [PATCH 20/27] add deploying manually section from @pameyer #3907 --- .../source/developers/dev-environment.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index b133ef8f144..7109b37ebf0 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -185,7 +185,18 @@ Deploy on Save Out of the box, Netbeans is configured to "Deploy on Save" which means that if you save any changes to project files such as Java classes, XHTML files, or "bundle" files (i.e. Bundle.properties), the project is recompiled and redeployed to Glassfish automatically. This behavior works well for many of us but if you don't like it, you can turn it off by right-clicking "dataverse" under the Projects tab, clicking "Run" and unchecking "Deploy on Save". -Note that it's possible to deploy the war file using the ``asadmin`` command. The :doc:`/installation/installation-main` section of the Installation Guide has more information on this topic. +Deploying Manually +~~~~~~~~~~~~~~~~~~ + +For developers not using Netbeans, or deploying to a non-local system for development, code can be deployed manually. +There are four steps to this process: + +1. Build the war file: ``mvn package`` +2. Undeploy the Dataverse application (if necessary): ``asadmin undeploy dataverse-VERSION`` +3. Copy the war file to the development server (if necessary) +4. Deploy the new code: ``asadmin deploy /path/to/dataverse-VERSION.war`` + +The :doc:`/installation/installation-main` section of the Installation Guide has more information on this topic. Netbeans Connector Chrome Extension ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 6535d9d0b7cfdf3fa2cb843b39847bf4023bf6bd Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 11:50:46 -0400 Subject: [PATCH 21/27] improve coding style page in dev guide #3418 #2724 #2644 #2575 #2574 --- .../source/developers/coding-style.rst | 57 ++++++++++++++++++- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/doc/sphinx-guides/source/developers/coding-style.rst b/doc/sphinx-guides/source/developers/coding-style.rst index fc3d3840946..6d88f9008e3 100755 --- a/doc/sphinx-guides/source/developers/coding-style.rst +++ b/doc/sphinx-guides/source/developers/coding-style.rst @@ -2,14 +2,65 @@ Coding Style ============ +Like all development teams, the `Dataverse developers at IQSS `_ have their habits and styles when it comes to writing code. Let's attempt to get on the same page. :) + .. contents:: |toctitle| :local: -Like all development teams, the `Dataverse developers at IQSS `_ have their habits and styles when it comes to writing code. +Java +---- + +Formatting Code +~~~~~~~~~~~~~~~ + +Tabs vs. Spaces +^^^^^^^^^^^^^^^ + +Don't use tabs. Use spaces. + +Format Code You Changed with Netbeans +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As you probably gathered from the :doc:`dev-environment` section, IQSS has standardized on Netbeans. It is much appreciated when you format your code (but only the code you touched!) using the out-of-the-box Netbeans configuration. If you have created an entirely new Java class, you can just click Source -> Format. If you are adjusting code in an existing class, highlight the code you changed and then click Source -> Format. Keeping the "diff" in your pull requests small makes them easier to code review. + +We would like to someday automate the detection and possibly correction of code that hasn't been formatted using our house style (the default Netbeans style). We've heard that https://maven.apache.org/plugins/maven-checkstyle-plugin/ can do this but we would be happy to see a pull request in this area, especially if it also hooks up to our builds at https://travis-ci.org/IQSS/dataverse . + +Logging +~~~~~~~ + +We have adopted a pattern where the top of every class file has a line like this:: + + private static final Logger logger = Logger.getLogger(DatasetUtil.class.getCanonicalName()); + +Use this ``logger`` field with varying levels such as ``fine`` or ``info`` like this:: + + logger.fine("will get thumbnail from dataset logo"); + +Generally speaking you should use ``fine`` for everything that you don't want to show up in Glassfish's ``server.log`` file by default. If you use a higher level such as ``info`` for common operations, you will probably hear complaints that your code is too "chatty" in the logs. These logging levels can be controlled a runtime both on your development machine and in production as explained in the :doc:`debugging` section. + +When adding logging, do not simply add ``System.out.println()`` lines because the logging level cannot be controlled. + +Avoid Hard-Coding Strings +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Special strings should be defined as public constants. For example, ``DatasetFieldConstant.java`` contains has a field for "title" and it's used in many places in the code (try "Find Usages" in Netbeans). This is better than writing the string "title" in all those places. + +Type Safety +~~~~~~~~~~~ + +If you just downloaded Netbeans and are using the out-of-the-box settings, you should be in pretty good shape. Unfortunately, the default configuration of Netbeans doesn't warn you about type-safety problems you may be inadvertently introducing into the code. To see these warnings, click Netbeans -> Preferences -> Editor -> Hints and check the following: + +- "Raw Types" under "Standard Javac Warnings" + +If you know of a way to easily share Netbeans configuration across a team, please get in touch. + + +Bike Shedding +------------- -A lot of it isn't written down, but a draft has been started at https://docs.google.com/document/d/1KTd3FpM1BI3HlBofaZjMmBiQEJtFf11jiiGpQeJzy7A/edit?usp=sharing +What color should the `bike shed `_ be? :) -Debate and comments are welcome! +Come debate with us about coding style in this Google doc that has public comments enabled: https://docs.google.com/document/d/1KTd3FpM1BI3HlBofaZjMmBiQEJtFf11jiiGpQeJzy7A/edit?usp=sharing ---- From 071a6bd2ff47a05a8996ae5d577e49da45bacd18 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 12:25:23 -0400 Subject: [PATCH 22/27] fix broken links to datascience.iq (deceased) #3431 --- doc/sphinx-guides/source/developers/dev-environment.rst | 6 +++--- doc/sphinx-guides/source/developers/intro.rst | 2 +- doc/sphinx-guides/source/developers/unf/index.rst | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 7109b37ebf0..34c855840cd 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -59,12 +59,12 @@ Recommendations Mac OS X ~~~~~~~~ -The setup of a Dataverse development environment assumes the presence of a Unix shell (i.e. bash) so an operating system with Unix underpinnings such as Mac OS X or Linux is recommended. (The `development team at IQSS `_ has standardized Mac OS X.) Windows users are encouraged to install `Cygwin `_. +The setup of a Dataverse development environment assumes the presence of a Unix shell (i.e. bash) so an operating system with Unix underpinnings such as Mac OS X or Linux is recommended. (The `development team at IQSS `_ has standardized Mac OS X.) Windows users are encouraged to install `Cygwin `_. Netbeans ~~~~~~~~ -While developers are welcome to use any editor or IDE they wish, Netbeans 8+ is recommended because it is free of cost, works cross platform, has good support for Java EE projects, and happens to be the IDE that the `development team at IQSS `_ has standardized on. +While developers are welcome to use any editor or IDE they wish, Netbeans 8+ is recommended because it is free of cost, works cross platform, has good support for Java EE projects, and happens to be the IDE that the `development team at IQSS `_ has standardized on. NetBeans can be downloaded from http://netbeans.org. Please make sure that you use an option that contains the Jave EE features when choosing your download bundle. While using the installer you might be prompted about installing JUnit and Glassfish. There is no need to reinstall Glassfish, but it is recommended that you install JUnit. @@ -97,7 +97,7 @@ Before cloning the repo, you are invited to read about our branching strategy in Determine Which Repo To Push To ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Developers who are not part of the `development team at IQSS `_ should first fork https://github.com/IQSS/dataverse per https://help.github.com/articles/fork-a-repo/ +Developers who are not part of the `development team at IQSS `_ should first fork https://github.com/IQSS/dataverse per https://help.github.com/articles/fork-a-repo/ Cloning the Project from Netbeans ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst index b8bf46b544f..633297e74b7 100755 --- a/doc/sphinx-guides/source/developers/intro.rst +++ b/doc/sphinx-guides/source/developers/intro.rst @@ -41,7 +41,7 @@ As a developer, you also may be interested in these projects related to Datavers - Miniverse - expose metrics from a Dataverse database: https://github.com/IQSS/miniverse - `Zelig `_ (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig -- `TwoRavens `_ (Javascript) - a `d3.js `_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens +- `TwoRavens `_ (Javascript) - a `d3.js `_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens - :doc:`/developers/unf/index` (Java) - a Universal Numerical Fingerprint: https://github.com/IQSS/UNF - `DataTags `_ (Java and Scala) - tag datasets with privacy levels: https://github.com/IQSS/DataTags - GeoConnect (Python) - create a map by uploading files to Dataverse: https://github.com/IQSS/geoconnect diff --git a/doc/sphinx-guides/source/developers/unf/index.rst b/doc/sphinx-guides/source/developers/unf/index.rst index 7759a64383f..0651891242d 100644 --- a/doc/sphinx-guides/source/developers/unf/index.rst +++ b/doc/sphinx-guides/source/developers/unf/index.rst @@ -29,7 +29,7 @@ available, for cross-validation. Learn more: Micah Altman, Jeff Gill and Michael McDonald, 2003, `Numerical Issues in Statistical Computing for the Social Scientist -`_, +`_, New York: John Wiley. **Contents:** From f7a93377e2a06767627a285ef5df82a23399084c Mon Sep 17 00:00:00 2001 From: Derek Murphy Date: Tue, 20 Jun 2017 12:49:05 -0400 Subject: [PATCH 23/27] Proofread fixes for #3418 Small fixes to Coding Style page. --- doc/sphinx-guides/source/developers/coding-style.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/developers/coding-style.rst b/doc/sphinx-guides/source/developers/coding-style.rst index 6d88f9008e3..1a1cb6e33c1 100755 --- a/doc/sphinx-guides/source/developers/coding-style.rst +++ b/doc/sphinx-guides/source/developers/coding-style.rst @@ -36,14 +36,14 @@ Use this ``logger`` field with varying levels such as ``fine`` or ``info`` like logger.fine("will get thumbnail from dataset logo"); -Generally speaking you should use ``fine`` for everything that you don't want to show up in Glassfish's ``server.log`` file by default. If you use a higher level such as ``info`` for common operations, you will probably hear complaints that your code is too "chatty" in the logs. These logging levels can be controlled a runtime both on your development machine and in production as explained in the :doc:`debugging` section. +Generally speaking you should use ``fine`` for everything that you don't want to show up in Glassfish's ``server.log`` file by default. If you use a higher level such as ``info`` for common operations, you will probably hear complaints that your code is too "chatty" in the logs. These logging levels can be controlled at runtime both on your development machine and in production as explained in the :doc:`debugging` section. When adding logging, do not simply add ``System.out.println()`` lines because the logging level cannot be controlled. Avoid Hard-Coding Strings ~~~~~~~~~~~~~~~~~~~~~~~~~ -Special strings should be defined as public constants. For example, ``DatasetFieldConstant.java`` contains has a field for "title" and it's used in many places in the code (try "Find Usages" in Netbeans). This is better than writing the string "title" in all those places. +Special strings should be defined as public constants. For example, ``DatasetFieldConstant.java`` contains a field for "title" and it's used in many places in the code (try "Find Usages" in Netbeans). This is better than writing the string "title" in all those places. Type Safety ~~~~~~~~~~~ From b5a4ed1e14f5454b6381f41847d9bc645a33a7fe Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 14:37:59 -0400 Subject: [PATCH 24/27] improve readme #3729 --- README.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 0bcc0132b33..e0a771a6cf3 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,36 @@ Dataverse® =============== -Dataverse is an open source web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/)). +Dataverse is an [open source][] web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]). -Institutions and organizations can choose to install the Dataverse software for their own use. -In this case, the institution will be responsible for maintaining the application; installing upgrades, -setting up backups and data replication as needed. Dataverse 4.0 is now released and used by the Harvard Dataverse: [dataverse.harvard.edu](http://dataverse.harvard.edu/). If you'd like to first test it, you can use our demo site: [demo.dataverse.org](http://demo.dataverse.org). +[dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our development [roadmap][], and more. -For more general information about Dataverse please visit [dataverse.org](http://dataverse.org). +We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse. -The Dataverse code is *open-source* and *free*. +To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. -Dataverse releases are available for download from https://github.com/IQSS/dataverse/releases and installation instructions can be found in the [Installation Guide](http://guides.dataverse.org/en/latest/installation/). +To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][]. + +We love contributors! Please see our [Contributing Guide][] for ways you can help. Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States. [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org) -[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=master)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=master&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=master) +[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=master&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=master) + +[dataverse.org]: https://dataverse.org +[demo.dataverse.org]: https://demo.dataverse.org +[Dataverse community]: https://dataverse.org/developers +[Installation Guide]: http://guides.dataverse.org/en/latest/installation/index.html +[latest release]: https://github.com/IQSS/dataverse/releases +[features]: https://dataverse.org/software-features +[roadmap]: https://dataverse.org/goals-roadmap-and-releases +[integrations]: https://dataverse.org/integrations +[REST APIs]: http://guides.dataverse.org/en/latest/api/index.html +[Contributing Guide]: CONTRIBUTING.md +[mailing list]: https://groups.google.com/group/dataverse-community +[community call]: https://dataverse.org/community-calls +[chat.dataverse.org]: http://chat.dataverse.org +[Dataverse Community Meeting]: https://dataverse.org/events +[open source]: LICENSE.md From 7dbb863cb316e9c6a688b1ed6305f85356fd31bd Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 14:40:13 -0400 Subject: [PATCH 25/27] finsh changing "master" to "develop" in readme #3729 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e0a771a6cf3..93f2f8f754f 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org) -[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=master&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=master) +[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop) [dataverse.org]: https://dataverse.org [demo.dataverse.org]: https://demo.dataverse.org From 7e9a8d949fe8b4462c957de43ea615e5b5ccda7b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 20 Jun 2017 14:52:52 -0400 Subject: [PATCH 26/27] we don't assign new issues anymore, link version control page #3729 Also link to list of contributors. --- CONTRIBUTING.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f7b83f8c492..84f07bae985 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,7 +28,7 @@ An issue is a bug (a feature is no longer behaving the way it should) or a featu Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing issue that matches the issue you want to report, please add a comment to it. -If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org). Someone on the Dataverse development team will appropriately tag and assign it to a member of the Dataverse development team. +If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org). If you do not receive a reply to your new issue or comment in a timely manner, please email support@dataverse.org with a link to the issue. @@ -53,12 +53,11 @@ The source for the documentation at http://guides.dataverse.org is in the GitHub Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. -We will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. +Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. -We hope you find the Developer Guide at http://guides.dataverse.org/en/latest/developers helpful. - -After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks! +After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. We maintain a list of [community contributors][] so please let us know if you'd like to be added or removed from the list. Thanks! [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community [dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev [IRC]: http://chat.dataverse.org +[community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing From db03e1d03dc0f7a4d621c5719c9a371fbae27155 Mon Sep 17 00:00:00 2001 From: Pete Meyer Date: Tue, 20 Jun 2017 15:00:16 -0400 Subject: [PATCH 27/27] asadmin syntax for disabling glassfish analytics --- doc/sphinx-guides/source/developers/dev-environment.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 34c855840cd..4614f53b9d1 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -29,6 +29,8 @@ As a `Java Enterprise Edition