Consider having Ingest run on separate server / Modularize

[scolapasta]

[bencomp]

@scolapasta what? Why?

[landreev]

@bencomp: We always wanted to make the application more modular; so that individual pieces/components could run independently; and, if necessary, run on different servers - to deal with load issues and such.
This ticket is for making ingest such an independent service; so that it could run on another host, similarly to how Solr and Postgres are integrated into the db.

This doesn't mean it will have to run on a separate server.

Ingest jobs can potentially be expensive, with both memory and CPU cycles. So if this makes the rest of the app run slow, this could be a simple solution - just move it onto another server and let it chew all the CPU time and memory there.

[landreev]

@bencomp : (I meant to say "... are integrated into the application")

[pdurbin]

@landreev check out the comment by @wbuchanan about how he has built https://github.com/wbuchanan/data-transfer processing Stata files in Java: #2883 (comment)