Improve shipment mover

[huangjqq]

Currently, when the cron job encounters an error, it keep retrying and sending notifications every 2 minutes. It would be good to fix it in the following ways:

• flag a shipment as having failed once it encounters errors instead of repeatedly re-trying and sending emails each time
• better system for flagging/emailing errors (daily report maybe?)
• improve interaction with partially downloaded files (details from Moses in comment)

[huangjqq]

Moses:
Another thing I worry about (which could even be implicated in this issue) is the potential race condition between the mkdir ...; cp -r ... approach to moving stuff around in the filesystem, and the cron job waking up and seeing a partially-copied directory to interact with. The better approach (if it's not being done now) is to copy everything into a temp directory somewhere on the same device and then rename. At the kernel level that's just swapping inodes and should be an atomic operation or close to it