feat: support Spark expression regexp_extract#4146
Open
andygrove wants to merge 3 commits intoapache:mainfrom
Open
feat: support Spark expression regexp_extract#4146andygrove wants to merge 3 commits intoapache:mainfrom
regexp_extract#4146andygrove wants to merge 3 commits intoapache:mainfrom
Conversation
Implement regexp_extract using the Rust regex crate. The expression is marked Incompatible because the Rust regex engine differs from the Java engine that Spark uses; users must opt in via spark.comet.expression.RegExpExtract.allowIncompatible=true.
Audit follow-ups: - Align Rust error messages with Spark's `INVALID_PARAMETER_VALUE` templates so `expect_error` substrings can match both engines. - Override `getUnsupportedReasons` in `CometRegExpExtract` so the non-literal pattern and non-literal idx reasons are picked up by the Compatibility Guide generator. - Add Comet SQL test cases for: NULL pattern and NULL idx, idx=0 with no capture groups, multibyte / Unicode subjects, idx out of range, pattern with no groups + idx>=1, negative idx, invalid regex syntax, and a Java-only lookahead that Rust regex rejects (marked `ignore`). - Add fallback test cases for non-literal pattern and non-literal idx. - Mark the expression supported in `spark_expressions_support.md` with per-version audit notes.
Address review feedback: - Make `extract_array` build a `GenericStringBuilder<O>` matching the input offset size so a `LargeUtf8` subject no longer silently outputs `Utf8` (avoids potential i32-offset overflow on >2GB inputs). - Inline group extraction so the per-row `String` allocation is gone; the only remaining `to_string` is on the rare scalar code path. - Replace the manual append-null loop in `null_result` with `StringArray::new_null(n)`. - Borrow the pattern as `&str` instead of cloning it before calling `Regex::new`. - Pass `failOnError = false` to the proto, matching `CometStringSplit`. The Rust UDF does not branch on this flag, so `true` was misleading.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
regexp_extractis a common Spark SQL string function used to pull substrings out of input strings via regex capture groups. Adding native support lets queries that use it stay in Comet instead of falling back to Spark.What changes are included in this PR?
spark_regexp_extractinnative/spark-expr/src/string_funcs/regexp_extract.rs, backed by theregexcrate. Handles Utf8 and LargeUtf8 inputs (array and scalar), idx defaults to 1, idx=0 returns the whole match, no match returns the empty string, an unmatched optional group returns the empty string, null input returns null, and an out-of-range idx returns an execution error.regexp_extractincomet_scalar_funcs.rs.CometRegExpExtractserde mappingRegExpExtractto the native UDF. Reported asIncompatiblebecause the Rust regex engine has different semantics from Java's regex engine (POSIX classes, look-around, possessive quantifiers, etc.). Users opt in viaspark.comet.expression.RegExpExtract.allowIncompatible=true. Falls back when the pattern oridxis non-literal.How are these changes tested?
regexp_extract.rscovering basic group extraction, idx=0/default idx, null subject, null pattern, unmatched optional group, out-of-range idx, negative idx, and invalid regex.regexp_extract.sqlverifies that the expression falls back to Spark by default.regexp_extract_enabled.sqlexercises the happy path withallowIncompatible=true, including default and explicit idx, idx=0, no-match, null input, optional unmatched groups, anchors, and an all-literal expression. Run under aConfigMatrixfor both dictionary-encoded and plain Parquet input.