Document ideas for improving automated testing

[pdurbin]

We'd like to make improvements to our automated testing. I'm going to iterate on this issue description to jot down some ideas which may spawn child issues. I also started a thread on the mailing list asking for ideas from the community.

• Unit Tests
  • Increase code coverage. As of Dataverse 4.2.1 code coverage was 4.54%: https://coveralls.io/builds/4021183
    • Review pull requests from @bencomp for ideas for approaches to testing: https://github.com/IQSS/dataverse/pulls?q=is%3Apr+author%3Abencomp
    • Come up with a way to test commands: http://irclog.iq.harvard.edu/dataverse/2015-11-04#i_26750
• EJB Testing
  • Test EJBs using Arquillian, embedded Glassfish, or similar. @bmckinney kicked the tires on Arquillian at bmckinney@2f243b1
• Integration Tests
  • API testing
    • Reinstate periodic (at least nightly) runs of integration tests. Update: good progress per https://groups.google.com/d/msg/dataverse-community/X8OrRWbPimA/wcmQx5VZCQAJ
    • Automate testing of the Python client: Run tests automatically with each deployment to a Dataverse server dataverse-client-python#10
    • Work with @leeper on testing the R client: https://github.com/IQSS/dataverse-client-r
    • Review and attempt to implement "API Test Checklist" from @kcondon at https://docs.google.com/document/d/199Oq1YwQ4pYCguaeW48bIN28QAitSk63NbPYxJHCCAE/edit?usp=sharing
    • Attempt to use @openscholar approach for running integration tests using Travis https://github.com/openscholar/openscholar/blob/SCHOLAR-3.x/.travis.yml (probably requires using Ubuntu rather than CentOS)
    • Generate code coverage reports for integration tests: Have ITFizzBuzzTest test a Jersey endpoint and show lines executed pkainulainen/maven-examples#3 and http://www.petrikainulainen.net/programming/maven/creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoco-maven-plugin/
    • Consistent logging of API Tests. Show test name at the beginning and end and status codes returned.
  • Automated browser testing
    • Revisit Selenium/Open Sauce: 8a26404 and https://saucelabs.com/u/esodvn and https://saucelabs.com/u/wdjs and http://sauceio.com/index.php/2013/05/a-browser-matrix-widget-for-the-open-source-community/
  • Installation testing
    • Reinstate periodic (at least nightly) runs of dropping the database and running the setup scripts. Update: good progress per https://groups.google.com/d/msg/dataverse-community/X8OrRWbPimA/wcmQx5VZCQAJ
    • Run vagrant up on a server to test the installer: http://guides.dataverse.org/en/latest/developers/tools.html#vagrant . We haven't been able to get this working in Travis: https://travis-ci.org/IQSS/dataverse/builds/96292683 . Perhaps it would be possible to use AWS as a provider from Vagrant judging from https://circleci.com/gh/critical-alert/circleci-vagrant/6
    • Work with @lwo to automate testing of https://github.com/IQSS/dataverse-puppet . Consider using Travis: Travis builds dataverse-puppet#10
    • With with ? to automate testing of https://github.com/IQSS/dataverse-ansible with Travis or similar.
  • Load/perfomance testing
    • Run stress tests on a period basis: https://github.com/IQSS/dataverse-helper-scripts/tree/master/src/stress_tests
    • Marcel Duran created a command-line wrapper for the WebPagetest API that can be used to test performance in your continuous integration pipeline (TAP, Jenkins, Travis-CI, etc): https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs#jenkins-integration
• Documentation
  • Set testing expectations in Developer Guide and document tools (Jacoco, Coveralls, REST Assured) #3431 Update http://guides.dataverse.org/en/latest/developers/testing.html to be more clear about how much automated testing is expected from developers.
  • Create top-down checklist, building off the spreadsheet at API cleanup #3358 (comment)

[bencomp]

👍 ! Of course I won't need to add much to what I have said before about testing.

I recently heard about QuickCheck on Software Engineering Radio. It's a system (the original works for Haskell, but there are various Java implementations) that generates lots of test cases automatically, based on defined assumptions about the context of a test. That could be useful.

Interestingly, in that episode of SE Radio, Dave Thomas argues that aiming for excellent code coverage is a waste of time. Automated acceptance tests should be enough to cover the most important use cases, with unit tests only adding to where they are really needed. I don't remember if this was about the legacy systems they were talking about and that for new systems it does make sense to have unit tests covering lots of code.

I do still believe in automated testing as Dataverse is huge and complex. You want to use the limited QA resources wisely. Having correct tests (not sure if #2460 shows code is not working or if the test is incorrect) helps to detect mistakes before QA or the users find it.

(edit: QuickTest should have been QuickCheck - link was correct)

[michbarsinai]

Great list, @pdurbin! Commands are normally testable by providing a test context. There's a PoC unit test for this. It's probably a good idea to add some real tests as well - this will allow for building reusable mock service beans.