Based on discussions we have decided that runners should be created with a separate playbook, e.g. https://github.com/coderefinery/openstack-gitlab-runner to create the runners as runner tweaking and base installation tweaking are done with different time cycles.
The feature has been disabled by default but it should be removed entirely after a while.