in case of muliple instances of clowder extractors show up multiple times

[robkooper]

Describe the bug
If you run multiple instances of clowder using the same database, each of them will listen for the heartbeat and write it to the database.

To Reproduce
Steps to reproduce the behavior:

1. run clowder with multiple intances
2. start a new extractor
3. look at extractors
4. See error

This could be related to some of the issues we seen with elasticsearch as well as login, where multiple instances would pick up the messages. This could be resolved by picking a single master machine.

[robkooper]

if picking a single leader machine that is harcoded (easiest solution) we will end up where a single machine becomes point of failure for login, indexing and registration of extractors. However the majority of the pieces of clowder will work, and once this one machine is back, all other pieces will work as normal.

If implemented the default should be for the single machine to be a leader, and the others should be non leaders.

[lmarini]

Shouldn't the clowder instance that get it first get it off the queue? Should we change it that way?

Also, extractor ids should be unique in the database. If we don't overwrite already (doesn't look like it) we should probably
change it to that.

[robkooper]

Each clowder instance subscribes to the heartbeat separately, so there is no queue. Since each extractor sends a heartbeat every 5 minutes we don't need persistent queues.

As for the ids, i think we can have a race condition where 2 instances at the same time try to insert a new extractor. We can probably have a process that cleans up these duplicates.

I think the extractor.id + extractor.version should be the unique identifier.

[robkooper]

@max-zilla how will this impact the work you are doing with private extractractors?

[robkooper]

From @lmarini when return a list of all extractors make sure that there are no duplicates.

[robkooper]

fixed in #348