Classifying junit tests for testing versatility

[matthew-a-dunlap]

There is a need to organize our tests such that only some are run during development while others are run by Travis. Some tests can be pretty slow and running the full suite every time can slow down dev work.

Looking into this a bit, there are at least two JUnit options to choose from: @suite and @category annotations . It may be best to choose the option that does not require annotating every test class, but maybe only annotating the slow ones?

More investigation is needed but either way it shouldn't be more work than adding some annotations and doing some minor edits to the build process and travis process

[pameyer]

@matthew-a-dunlap From a very quick look ~ a year ago, completely disabling running unit tests when building (with maven) didn't result in a significant enough speed-up for me personally to bother building that way. Assuming I'm remembering correctly (which is always in question), ~5-10ss of ~1-2m build.

More efficiency is always good - just advanced warning that there's a chance this may not speed things up all that much (hopefully I'm wrong, or things have changed, though).

[matthew-a-dunlap]

@pameyer if anything I know the provenace parser/validator test were slowing down my builds by ~12 seconds, but they may be the only ones with a real impact

[pameyer]

Good point @matthew-a-dunlap - this pre-dated the prov work.

[djbrooke]

assigning to @matthew-a-dunlap to talk about this at next estimation session

[benjamin-martinez]

@matthew-a-dunlap @pameyer Aside from the prov tests, what other tests do you think would be included in this "Slow Test" or "Non-Essential Test" category?

[oscardssmith]

One good category is test that depend on external resources

[benjamin-martinez]

Okay, so what I am thinking is that we should have a test suite called "NonEssential" and within this suite we'll have two+ specific categories. As of now the two categories will be "Slow" and "External Dependency". The name "Slow" is probably just a place holder for now until I can come up with something more specific.

If we want to go even further with this, we could create several test suites that correlate to the specific part of the code base that your working in so only things that you can affect will be tested.

[matthew-a-dunlap]

@benjamin-martinez I think the best approach is to keep it pretty simple for now with room to add more groupings of tests later. Your approach in the first paragraph seems on that path. Can we also create a suite called "Essential" and have it be everything that isn't "Slow" or "External Dependency"? We'll need that to be what is run by the developer builds every time.

I don't have a good understanding of how suite / category are used together. I'll be in shortly and maybe we can discuss more then.