Run Large Tensor Tests as part of PR tests

[leezu]

While there is an effort to enable large tensor feature by default (#18625), the feature is only tested nightly and in fact the tests are failing: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/752/pipeline/96

As this will be a default feature, and especially as the test only runs 30min, can we enable it as part of the PR checks?

#18744
#18718
#18715

[access2rohit]

@leezu simply running all tests within test_large_array.py takes more than 90mins. I don't think its practical to enable them on CI. It will simply timeout on all PRs

[leezu]

Why does the run linked above only take 26m then?

[access2rohit]

Why does the run linked above only take 26m then?

I think last successful run should be considered. And if we do then : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/609/pipeline

it takes over roughly 90 mins. @leezu do you agree ?

[leezu]

@access2rohit there's only one long-running test:

] 

[2020-02-25T20:36:21.873Z] OK (SKIP=1)

[2020-02-25T20:36:21.873Z] + nosetests-3.4 tests/nightly/test_large_vector.py:test_nn

[2020-02-25T21:25:14.016Z] .

[2020-02-25T21:25:14.016Z] ----------------------------------------------------------------------

[2020-02-25T21:25:14.016Z] Ran 1 test in 2936.624s

All other tests run in around 30 minutes. Could the long-running test be optimized to take less than 1 hour? In any case, I think the other tests could be moved to run on every PR?

[access2rohit]

We can split up and parallelize the jobs to run on multiple workers as a part of multiple test suites in CI. That will speed things up but will consume more CI resources hence CI run cost per PR will increase(depending on which platforms you want it to be tested on in CI and for Both CPU and GPU build to ensure correctness and consistency in behaviour). What do you think ?

[leezu]

As mentioned, I don't think 30 minutes is an issue here as this is a default feature, will run in parallel to other pipelines which already take 60 minutes.