Collect logs from integration test runs

[TheRealFalcon]

Proposed Commit Message

Collect logs from integration test runs

During teardown of every cloud instance, run 'cloud-init collect-logs',
then transfer and unpack locally. Two new integration settings have
been added to specify when to perform this action (ALWAYS,
ON_ERROR, NEVER), and where to store these logs.

Additional Context

Note that it runs once per client, so if you're using the class client, you'll get one set of the logs for the entire class. Currently logs take about 5M of space locally. The entire tree of files created after a single test run can be seen here: https://paste.ubuntu.com/p/v4rF9m9tPF/ . It includes a couple test_example runs that don't exist in the repo because I was testing parameterized tests.

Test Steps

Without overriding COLLECT_LOGS or LOCAL_LOG_PATH in the integration test settings, run:
pytest -s tests/integration_tests
A timestamped directory should get created at /tmp/cloud_init_test_logs containing all of the logging information collected from each test instance run during tests.

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[paride]

@TheRealFalcon one flake8 failure:

tests/integration_tests/conftest.py:157:1: E303 too many blank lines (3)

[paride]

LGTM, but will require a +1 from a committer.

[blackboxsw]

Future me wants a docstring describing params, specifically I'd like an example of what node_id value is generally expected to be so I don't have to look it up from the PR subtext.

[blackboxsw]

Since this is a fairly expensive operation, do we want to really do this on every instance, or gate it behind either a test failure or a force value? This request kind of asks whether the COLLECT_LOGS value should be an enum instead of bool. values could by ALWAYS|ON_ERROR|NEVER?

what do folks think?

[TheRealFalcon]

I think the thinking was that for the SRU logs we'd need to collect the logs from most/all of the tests anyway. I could default it to False too. But having an ON_ERROR option seems like a good idea too. I'll take a look at adding that.

[blackboxsw]

Thanks a lot for this @TheRealFalcon +1.

[paride]

There a merge conflict to be resolved

[blackboxsw]

+1 Thanks for this, please update your suggested squash commit message to reference the something about how collect_logs is optionally collected ON_ERROR.

[TheRealFalcon]

@blackboxsw I updated the commit message and rebased the branch