Cleanup open issues #142
Cleanup open issues #142
Conversation
|
@bethac07 This is ready to review/merge, other than merge conflicts, which send me into a panic trying to resolve. I'm sorry I've created another behemoth of a PR. It started in my head as just a bunch of little cleanup so it made sense to clump it all together, but seems to have grown... |
|
|
||
| ### Creating File Lists | ||
|
|
||
| Use any text editing software to create a `.txt` file where each line of the file is a path to a single image that you want to process. |
There was a problem hiding this comment.
We should mention here that it needs to be abspaths, not relpaths.
run.py
Outdated
| print("Service created") | ||
|
|
||
|
|
||
| def get_queue_url(sqs): |
There was a problem hiding this comment.
Why change this function instead of just run it twice, once for the dead letter queue and then once for the regular queue?
There was a problem hiding this comment.
We used to pass SQS_DEAD_LETTER_QUEUE as an ARN not a name so it had to handle them each differently. Then I needed to change it from ARN to a name as the easiest way to create it if it doesn't exist and then I missed cleaning this part up after myself :)
|
I did the best I could with the merge commit, but it had indeed gotten messy. It should be tested (preferably after addressing upstream comments) before we pull |
worker/cp-worker.py
Outdated
| else: | ||
| printandlog('== ERR \n'+err,logger) | ||
| mvtries+=1 | ||
| printandlog('Move attempt #'+str(mvtries+1),logger) |
worker/cp-worker.py
Outdated
| printandlog('== ERR \n'+err,logger) | ||
| mvtries+=1 | ||
| printandlog('Move attempt #'+str(mvtries+1),logger) | ||
| cmd = 'aws s3 mv ' + localOut + ' s3://' + DESTINATION_BUCKET + '/' + remoteOut + ' --recursive --exclude=cp.is.done' |
worker/cp-worker.py
Outdated
| cmd = 'aws s3 mv ' + localOut + ' s3://' + DESTINATION_BUCKET + '/' + remoteOut + ' --recursive --exclude=cp.is.done' | ||
| if UPLOAD_FLAGS: | ||
| cmd += ' ' + UPLOAD_FLAGS | ||
| printandlog('Uploading with command ' + cmd, logger) |
Cleanup of many open issues.