Large code bases cause rate limiting when booting

[curzonj]

We have a code base with lots of activities and the register_activity_type gets rate limited which causes the worker to crash.

We need an option for workers to not register activities at boot and be able to register them manually. This commit is my current attempt to solve this for us.

[mjsteger]

The first chunk in your commit is definitely addressing an oversight in the code, which should be resolved. Second chunk also seems pretty reasonable way to manually set up the workers. Mind if I cherry-pick this into my fork/bundle the changes you've made so far, add some tests, and submit a PR here?

Perhaps there should be a list of notes for "problems you may hit as you scale up your use of SWF, and how to solve them" for things such as this.

[curzonj]

@mjsteger certainly. feel free to take/modify anything you like. This is how I use it:

AWS::Flow::Runner.register(
  {
    'domain' => { 'name' => ENV['AWS_SWF_DOMAIN'] },
    'activity_workers' => {
      'task_list' => "activity_tasklist",
    },
    'workflow_workers' => {
      'task_list' => "workflow_tasklist",
    }
  }
)

I also directly call the runner because it does lots of useful things, but running it from the CLI directly is too clunky:

File.open('swf_worker.json', 'w') do |f|
  f.write(worker_config.to_json)
end

# Because this runner does tons of work for
# us but makes it hard to customeize the actual
# workers. So we just monkey patch it above.
AWS::Flow::Runner.main

I'm sure the Runner class wasn't intended to be called directly like this. Everything is tagged @api private