Skip to content

Add "lines" input format.#18433

Merged
gianm merged 2 commits intoapache:masterfrom
gianm:lines-format
Aug 26, 2025
Merged

Add "lines" input format.#18433
gianm merged 2 commits intoapache:masterfrom
gianm:lines-format

Conversation

@gianm
Copy link
Copy Markdown
Contributor

@gianm gianm commented Aug 23, 2025

This is useful for reading entire lines for later processing. For example, it can be used to read newline-delimited JSON into a single nested column, by combining this input format with the TRY_PARSE_JSON function.

This is useful for reading entire lines for later processing. For example,
it can be used to read newline-delimited JSON into a single nested column,
by combining this input format with the TRY_PARSE_JSON function.
Copy link
Copy Markdown
Contributor

@abhishekrb19 abhishekrb19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

It will be good to add web-console support for this feature separately.

return size * CompressionUtils.COMPRESSED_TEXT_WEIGHT_FACTOR;
}
return size;
return size * CompressionUtils.estimatedCompressionFactor(CompressionUtils.Format.fromFileName(path));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can extend CompressionUtils.estimatedCompressionFactor() or add a thin wrapper that accepts the path directly and returns the compression factor - internally invoking CompressionUtils.Format.fromFileName()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO these are equivalently useful for what we need them for right now, so I'll keep it the way I had it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for reviewing!

@abhishekrb19 abhishekrb19 added the Needs web console change Backend API changes that would benefit from frontend support in the web console label Aug 26, 2025
@gianm gianm merged commit 6ef9755 into apache:master Aug 26, 2025
70 checks passed
@gianm gianm deleted the lines-format branch August 26, 2025 23:11
@cecemei cecemei added this to the 35.0.0 milestone Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Documentation Needs web console change Backend API changes that would benefit from frontend support in the web console

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants