Update available archival config options

[bodom0015]

Description

Problem

Existing archival configuration options are kind of confusing.

Approach

• Replace archiveDebug with archiveAutoInterval and archiveAutoDelay to be more explicit
• Replace archiveMinimumStorageSize with archiveAutoAboveMinimumStorageSize to be more explicit
• Replace archiveAutoAfterDaysInactive with archiveAutoAfterInactiveCount and archiveAutoAfterInactiveUnit to be more flexible

How to Test

Prerequisite: Clowder running and configured to talk to RabbitMQ, RabbitMQ running, Clowder configured to store bytes on S3

1. Checkout and run this branch locally
2. Note the default "archive" configuration options in conf/application.conf

Manual Archival

1. To test manual archival, set the following values to your conf/custom.conf:

archiveEnabled=true                      # Enable archival features
archiveAllowUnarchive=true               # Offer the user an "Unarchive" button via the UI
archiveExtractorId="ncsa.archival.s3"    # The ID of the extractor to use for archiving

2. Run the S3 Archival extractor: docker run --net=host -it --rm -e AWS_ACCESS_KEY="${AWS_ACCESS_KEY}" -e AWS_SECRET_KEY="${AWS_SECRET_KEY}" -e AWS_BUCKET_NAME="your-bucket-name" -e RABBITMQ_URI="${RABBITMQ_URI}" clowder/extractors-archival-s3
   • NOTE: AWS_ACCESS_KEY, AWS_SECRET_KEY, and AWS_BUCKET_NAME must match your configuration for the S3ByteStorageDriver
3. Upload a file, navigate to it in the UI, and click "Archive"
   • You should see the extractor logs indicating that the file is being archived
   • You should see the Clowder UI refresh
   • After refresh, you should see that the file is marked as ARCHIVED
4. View the file in AWS S3 Management Console
   • You should see the S3 Object is now marked as INTELLIGENT_TIERING (by default)
5. Now click "Unarchive"
   • You should see the extractor logs indicating that the file is being unarchived
   • You should see the Clowder UI refresh
   • After refresh, you should see that the file is once again marked as PROCESS
6. View the file in AWS S3 Management Console
   • You should see the S3 Object is now marked as STANDARD (by default)

Automatic Archival

1. To test automatic archival of un-downloaded files, set the following configuration:

archiveAutoDelay=10                          # After waiting 10 seconds,
archiveAutoInterval=60                       # check every 60 seconds and archive
archiveAutoAboveMinimumStorageSize=1000000   # any file that is larger than 1MB
archiveAutoAfterInactiveCount=10             # and that has not been downloaded
archiveAutoAfterInactiveUnits="seconds"      # in the past 10 seconds

• WARNING: This configuration will archive every file in your system

2. Refresh the Clowder UI (to reload the configuration) and wait for the system to start
   • After 10 seconds (== archiveAutoDelay), you should see in the Clowder logs that the auto archival loop run is running its check
   • You should see your file is submitted to the extractor for archival
3. Manually refresh the page
   • You should see that your file is now marked as ARCHIVED
4. Click "Unarchive"
   • You should see that the file is marked as PROCESSED once again
5. Wait 60 seconds (== archiveAutoInterval)
   • You should see in the Clowder logs that the auto archival loop run is running its check once again
6. Manually refresh the page
   • You should see that your file has been automatically ARCHIVED

Review Time Estimate

• Immediately
• Within one week
• When possible

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• I have updated the CHANGELOG.md.
• I have signed the CLA
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.