Use MANIFEST in unload to guard against eventually-consistent S3 bucket listing

[JoshRosen]

This patch modifies spark-redshift's read path in order to guard against a potential source of missing data / inconsistent reads.

According to the Amazon S3 Data Consistency Model documentation, S3 bucket listing is eventually-consistent. In spark-redshift 1.6.0 and earlier, the S3 read path performs bucket-listing and thus may be impacted by this eventual-consistency, meaning that in rare circumstances reads may see only a subset of the unloaded data.

This patch fixes this issue by using UNLOAD command's the MANIFEST parameter, which causes it to write a JSON manifest which lists the paths of all unload output files. I modified the UNLOAD query to produce this manifest and added some code to the reader to consume it. In order to simplify parsing of the JSON manifest, I used the minimal-json JSON library, chosen because it's small and has no external dependencies (and thus is unlikely to cause the sorts of dependency conflicts that something like Jackson would lead to).

[codecov-io]

Current coverage is 89.27%

Merging #151 into master will increase coverage by +0.21% as of 2b64a94

@@            master    #151   diff @@
======================================
  Files           13      13       
  Stmts          649     662    +13
  Branches       144     146     +2
  Methods          0       0       
======================================
+ Hit            578     591    +13
  Partial          0       0       
  Missed          71      71       

Review entire Coverage Diff as of 2b64a94

Powered by Codecov. Updated on successful CI builds.

[JoshRosen]

This messy mocking is only to get the unit tests to pass. It's not the most high-fidelity mocking in the world, but that's okay because this code path is well-exercised by the integration tests.

[JoshRosen]

Ping @liancheng, could you help me to review this?

[liancheng]

LGTM

[JoshRosen]

Thanks for reviewing; I've updated the README to reflect the new semantics and have fixed the merge conflicts, so I'll merge this as soon as tests pass.