[SPARK-55062][Protobuf] Support proto2 extensions in protobuf functions

[dichlorodiphen]

What changes were proposed in this pull request?

This PR adds support for proto2 extensions to from_protobuf and to_protobuf (when file descriptor set is provided, as Java classes do not contain enough information to support extensions).

This is done by building an ExtensionRegistry and a map from descriptor name to its extensions. The registry is used during construction of the DynamicMessage to provide the Protobuf library with visibility of the extensions. The index is plumbed through the various helper classes for use in schema conversion and serde.

Why are the changes needed?

Proto2 extensions are a valid, if somewhat uncommon, feature of Protobuf, and it therefore makes sense to incorporate them into the schema when provided so as to not confuse the user.

Does this PR introduce any user-facing change?

Yes. Previously, extension fields would be dropped by both from_protobuf and to_protobuf. Now, they are retained. This can be demonstrated with the minimal example below. See the unit tests for more examples.

message Person {
    int32 id = 1;
    extensions 100 to 200;
}
extend Person {
    int32 age = 100;
}

How was this patch tested?

Unit tests were added for the new behavior, including basic behavior, extending nested messages, and extensions defined in separate files.

Was this patch authored or co-authored using generative AI tooling?

Initial draft authored with Claude Code.

Generated-by: claude-4.5-opus

[github-actions]

JIRA Issue Information

=== Improvement SPARK-55062 ===
Summary: from_protobuf and to_protobuf do not support proto2 extensions
Assignee: None
Status: Open
Affected: ["4.1.1"]

---

This comment was automatically generated by GitHub Actions

[zikangh]

What do you think about adding test cases for these edge cases?

1. extension field name collision with regular fields
2. schema evolution: read old data without extensions using new schema with extensions.
3. map extensions

[dichlorodiphen]

Good call, the original unit tests were in hindsight pretty lacking. I've added the above plus a few more. For #3, however, I believe the Protobuf grammar doesn't allow map fields in extensions (protoc will error), so there shouldn't be a need for a test

[dichlorodiphen]

A pretty flagrant error--I must have dropped this when reapplying changes from my other repo. I also realized as a result that the original tests were not explicitly checking for the added fields (round trip was working because the fields were being dropped), so I've added in those assertions.

EDIT: Also retested the operators manually in Spark Shell as a sanity check, and everything now looks good