Consider driving storage requests via ZK rather than by API call from ingest

[terrywbrady]

Mark proposed this in a team discussion.

• This would prevent storage workers from being overloaded
• This would eliminate the need for the hostname api call
• This would require a new zookeeper state

[dloy]

Suggestion

Instead of creating, one or more daemons for storage with feedback into the zk decision tree, use a pool of storage servers that are single threaded. When ingest is ready to have storage processing, attach to a single storage process to handle all storage based processing for that specific update. As soon as the process is complete the storage docker/ecs/.... is returned to the pool and ingest resumes normal processing.

This mechanism will not be overwhelmed by multi-threading. The size of the pool will never be more than the total of all concurrent ingests.

The advantage of handling this process:

• storage handling will not be overwhelmed by multi-threading
• no zk decision tree processes (daemons, etc) needs to be added
• adjustments for daemon prioritization is strictly limited to ingest requirements

If async division is needed between storage and ingest then single threaded storage is still a much simpler model than attempting to determine the impacts of multi-threading in a single service. Async will also require a new ingest process to perform post storage processing:

• signaling inventory creation
• cleanup of successfully processed content.

[mreyescdl]

Update
Decoupling code is now running in Stage and load testing is ongoing.

Three code bases have been altered to support feature

• ZK library: Add new Store state
  branch "storageState" is now merged into main
• Ingest: HandlerTransfer does not submit to Storage, but add queue entry
  branch "storageState" not yet merged
• Store: Consumer Daemon added to pull ZK entries and submit to Store service
  branch "storageState" not yet merged

[mreyescdl]

Update: Changes to the Cloud library were made to handle exceptions more accurately.
Also, a change to ensure that an exception puts ZK data in a "Failed" state.
Additionally, David is looking into consolidating the Upload methods and make further Cloud changes in the near future.

Also, we are running large datasets into Stage for testing