Small cleanup

[blag]

As well as a few small cleanups, this PR also fixes the CI scripts and the Makefile to work on PRs that have very large numbers of changed files, like StackStorm-Exchange/stackstorm-aws#89.

This PR also fixes CI for this repository.

Related: #76

[punkrokk]

@blag Can we see this running somewhere? Everything makes sense, but would be cool to see if work.

[cognifloyd]

From reading the code, looks good.

[nmaludy]

LGTM!

[blag]

@punkrokk Good point. FWIW, I did check this manually by running the CircleCI job for StackStorm-Exchange/stackstorm-aws#89 manually with SSH.

However, I also opened StackStorm-Exchange/stackstorm-aws#100 to kick off CI with this. That PR is tiny, and it only triggers one part of this code:

==================== configs-check ====================

. ~/virtualenv/bin/activate; \
if [ "${FORCE_CHECK_ALL_FILES}" = "true" ]; then \
	find /home/circleci/ci/* -name "*.yaml" -o -name "*.yml" | while read yaml_file; do \
		st2-check-validate-yaml-file "$yaml_file" || exit 1 ; \
	done; \
elif [ 1 -gt 0 ]; then \
	echo "Checking 1 YAML files"; \
	/home/circleci/ci/utils/git-changes yaml >~/.git-changes-yaml; \
	while read -r file; do \
		if [ -n "$file" ]; then \
			st2-check-validate-yaml-file $file || exit 1 ; \
		fi; \
	done < ~/.git-changes-yaml; \
	if [ -e ~/.git-changes-yaml ]; then rm ~/.git-changes-yaml; fi; \
else \
	echo "No files have changed, skipping run..."; \
fi
Checking 1 YAML files
Validating YAML syntax for file .circleci/config.yml...

That verifies that it is properly writing and then reading from the ~/.git-changes-yaml file and running st2-check-validate-yaml-file on it.

I haven't tested the similar changes for JSON so please 👀 at that code closely, but the YAML and Python checks were run during my manual CI run for StackStorm-Exchange/stackstorm-aws#89.

[Kami]

LGTM assuming existing jobs for various packs pass :)

[blag]

Once I merge this I'll need to rerun CI for StackStorm-Exchange/stackstorm-aws#89 anyway (although that's exactly what I did manually), and that definitely exercises more of the changes here, so I'll know very quickly if I messed anything up.

[blag]

This code looks good.

Failed test run: https://circleci.com/gh/StackStorm-Exchange/stackstorm-aws/626 (Python 2)
Successful test run: https://circleci.com/gh/StackStorm-Exchange/stackstorm-aws/625 (Python 3)

==================== configs-check ====================

. ~/virtualenv/bin/activate; \
if [ "${FORCE_CHECK_ALL_FILES}" = "true" ]; then \
	find /home/circleci/ci/* -name "*.yaml" -o -name "*.yml" | while read yaml_file; do \
		st2-check-validate-yaml-file "$yaml_file" || exit 1 ; \
	done; \
elif [ 3584 -gt 0 ]; then \
	echo "Checking 3584 YAML files"; \
	/home/circleci/ci/utils/git-changes yaml >~/.git-changes-yaml; \
	while read -r file; do \
		if [ -n "$file" ]; then \
			st2-check-validate-yaml-file $file || exit 1 ; \
		fi; \
	done < ~/.git-changes-yaml; \
	if [ -e ~/.git-changes-yaml ]; then rm ~/.git-changes-yaml; fi; \
else \
	echo "No files have changed, skipping run..."; \
fi
Checking 3584 YAML files
Validating YAML syntax for file actions/acm_add_tags_to_certificate.yaml...
Validating YAML syntax for file actions/acm_delete_certificate.yaml...

(repeat for 3000+ more files)

Validating YAML syntax for file actions/xray_put_trace_segments.yaml...
Validating YAML syntax for file config.schema.yaml...
Validating YAML syntax for file pack.yaml...
Validating YAML syntax for file tests/fixtures/multiaccount.yaml...