allow kvstore upgrade

[jewnix]

Problem description:

• When deploying a docker container (or probably some other scenarios) with splunk version 9.x that has an existing volume containing a KVStore that was previously used in version 8.x, the KVStore will fail to start. This is due to the mongo version used for that kvstore was still on version mondo 3.6. This happens when splunk is not running the built-in automatic migration script since the conntainer looks like a fresh install to splunk. Splunk 9.1 does not support mongo version 3.6.

• The method of setting the current splunk_current_version uses the manifest files, this creates a list and does not allow to use the version conditional which requires the value to be a string.

This PR allows mongo to be upgraded from 3.6 to 4.0 initially, and then migrates it to the wiredTiger engine, and then starts splunk. It also will always migrate the KVStore to the latest version

Changes in the splunk_common role:

• New task get_facts_current_version.yml:

  • The version conditional used in the upgrade_kvstore.yml task, requires the value to be a string, the manifest method causes the value to be a list. When etc/splunk.version is already present, or is populated with the update_etc.yml task, we can use the splunk.version file to get the version and build. If etc is not populated, we use the manifest method to get the version.

• Modified task get_facts.yml:

  • Determine the current version with the get_facts_current_version.yml task.
  • Checks if the var/lib directory exists. This will allow us to determine if there is a data volume present or mounted that may contain a KVStore.
  • Check if the present KVStore is still on the mmapv1 storage engine by checking if the var/lib/splunk/kvstore/mongo/WiredTiger exists.

• Modified update_etc.yml:

  • Run the get_facts_current_version.yml again if the etc directory was updated if this task was triggered in main.yml, and the splunk.version file was not present when the initial get_facts.yml task ran.

• Modified main.yml:

  • Run the kvstore_upgrade.yml task if var/lib exists.
  • Currently only supported for standalone.

• New task upgrade_kvstore.yml:

  • Upgrade mongo from 3.6 to 4.0 when the new version is greater than version 9.0
  • Add storageEngineMigration in server.conf in case this is being upgraded to version less than 9.0 (edge case, but maybe someone is still doing that).
  • Migrate engine to WiredTiger if var/lib/splunk/kvstore/mongo/WiredTiger does not exist.
  • Always update KVStore to the latest version.