Metadata management seems broken to this noob

[snichols]

Affected Version

24.0.2

Description

Don't get me wrong here, I love so much about Druid! I love love love it!

But, I'm a noob at operating Druid cluster in a production environment. I'm using druid-operator and it works really well. I can stand up clusters and they work great! Fantastic.

Where I'm running into issues is when I delete streaming datasources and attempt to reconstitute them. Here's the repro steps:

1. Stand up a fresh Druid cluster using s3 for deep storage.
2. Set up a Kafka ingest supervisor to pull records from a topic.
3. Let that supervisor work long enough to persist segments. An hour, days, it's dealer's choice!
4. Terminate the supervisor.
5. Wait for the Kafka ingest task to finish.
6. Mark all datasource segments as unused.
7. Run a kill task for said datasource.
8. Wait for kill task to complete.
9. Observe that there's no datasource in the datasource list.
10. Observe that there's no segments listed in the segments list.
11. Set up a Kafka ingest supervisor to pull records from a topic with the same settings as step 2.
12. Watch as hilarious bugs occur. It could be that old segment metadata interferes or perhaps a topic name change causes weird exceptions. In any case, this never works cleanly.
13. Get frustrated as you realize that there's a tight coupling between topic and datasource name. You really can't reuse either one. Hate your life as all downstream queries need to be refactored due to this bug.

My workaround is to reinstall Druid from scratch and set up the ingest again. This works fine in development. But I'll need to stand up a permanent storage for all records so I can reconstitute the topic from scratch in the case of catastrophic failure in production. Oof.

I'd like to suggest that when a datasource is deleted then all references to the datasource are actually removed from the metadata store. Am I missing some reason why this should't be the case already? If so, that'd solve so many unhandled edge cases.

I'm happy to contribute a fix here but any guidance from an experienced Druid dev would be appreciated.

[kfaraz]

Hi @snichols , really glad to hear that you are enjoying using Druid!

Just for my understanding, do you reuse the same topic name in steps 2 and 11?
Could you please elaborate on the exceptions and issues that you see if you try to reuse the datasource name?

As you say, the topic name should be completely independent of the datasource name and this is indeed the case, unless there is a bug.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.