Skip to content

Rework preservation audit batch creation #6975

@hackartisan

Description

@hackartisan

Summary or User Story

The new audit failed in prod with Connection reset by peer (redis://figgy-web-prod1.princeton.edu:6379/1) (RedisClient::ConnectionError)

We should try getting a count of IDs, then splitting off sub-jobs that add preservation check jobs to the batch. if we make sure the ids query is sorted we can use an offset and limit to get the right chunk of ids from the database.

Acceptance Criteria

  • the batch creation is broken up so that all preservation check jobs aren't enqueued at once.

First step

check sidekiq docs https://github.com/sidekiq/sidekiq/wiki/Batches#huge-batches

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions