S3 file harvesting: only first 1000 files of each directory are read

[Alex-Ikanow]

(we list top-level directories recursively, which makes this even worse)

Issue INF-2131 on JIRA

[Alex-Ikanow]

August 2013 OSS release includes partial fix "S3 directories are now parsed recursively vs 1 one big list", which reduces the impact of this bug, also includes a "renameAfterParse" option for the file harvester that allows files to be moved after processing - together these workaround the bug completely (maintaining large numbers of files in a single "active" directory when they'll never be read again is undesirable anyway if it can possibly be avoided)