Assistance migrating from firehoses/parsers to input sources/formats

[gianm]

Continuing a conversation from #8903 (comment), #8903 (comment).

[gianm]

@vogievetsky wrote,

@gianm I made a thing that if you paste in a firehose based input spec into the data loader it will be magically converted to an input source based one. I need that because the data loader will soon only work with input sources. Do you think this is a bad idea?

I know there are a lot of Druid users that have ingestion specs saved somewhere outside of Druid what were you imagining these people will do to get onto the new format? Convert it by hand? I figured that the data loader could be helpful there.

I was imagining people would convert them by hand, based on the new docs.

The conversion isn't always possible, since there is stuff that firehoses can do right now that input sources can't. I think the data loader doing it for people would be really helpful if feasible, but doing it 'blindly' won't always work for that reason.

Btw I think it makes sense for the release notes to say not to use input sources and formats until you're pretty sure you won't want to roll back. Since using them will mean that you can't do a rolling downgrade anymore.

[vogievetsky]

I figured that the release notes would have a firehose-by-firehose conversion guide (as bullet points). I think it would also be good for the data loader (via me) to be aware of those conversions.

What I am saying is that I feel like I should have a 100% awareness of the differences between firehoses and input sources as that probably needs to be accounted in the data loader UI in some way. Then that conversion should be coded into a converter function that can fail to covert with a helpful error message and unit tests.

Other than some firehoses not being supported by input sources yet (like sqlFirehose) what are the notable differences between firehoses and input sources functionality wise?

It is a very good idea to move this conversion into its own ticket. Please comment (with links) on any changes that should be accounted for in the release notes.

[gianm]

Given the fact that docs would exist (and the old and new format will be documented next to each other) IMO spelling out each one in the release notes is overkill and would make them too long.

Does it work for you watch the input source and format PRs as they bring in doc updates? Anything relevant should be showing up there & some already has.