Skip to content

PARQUET-2290: Add CI for Hadoop 2#1076

Merged
Fokko merged 1 commit intoapache:masterfrom
Fokko:fd-add-ci-hadoop2
Apr 23, 2023
Merged

PARQUET-2290: Add CI for Hadoop 2#1076
Fokko merged 1 commit intoapache:masterfrom
Fokko:fd-add-ci-hadoop2

Conversation

@Fokko
Copy link
Contributor

@Fokko Fokko commented Apr 19, 2023

Make sure you have checked all steps below.

While upgrading Parquet in Iceberg, I noticed that we broke compatibility. Iceberg is still at Hadoop 2.7.3. The hasCapability method that is used in #951 is not available in Hadoop <2.9.x. There is no serious workaround. I think it is good to add this CI to at least know the lower bound Hadoop 2 version that we're still supporting.

Jira

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

@Fokko Fokko force-pushed the fd-add-ci-hadoop2 branch 4 times, most recently from 4dab2dc to ebd6393 Compare April 19, 2023 13:24
import org.apache.hadoop.fs.GlobPattern;

import com.google.re2j.Pattern;
import com.google.re2j.PatternSyntaxException;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caused issues:

[ERROR] /Users/fokkodriesprong/Desktop/parquet-mr/parquet-thrift/src/main/java/org/apache/parquet/thrift/projection/deprecated/PathGlobPattern.java:[55,49] incompatible types: java.util.regex.Pattern cannot be converted to com.google.re2j.Pattern

Different libraries are used in Hadoop 2 and 3. I think it makes sense to remove this class since it is already marked deprecated for a very long time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the static function since it isn't being used in the project.

@Fokko Fokko force-pushed the fd-add-ci-hadoop2 branch 4 times, most recently from 2cf9975 to 79148e5 Compare April 19, 2023 20:49
@Fokko Fokko force-pushed the fd-add-ci-hadoop2 branch from 79148e5 to ffdf693 Compare April 20, 2023 08:16
@Fokko Fokko merged commit f70c529 into apache:master Apr 23, 2023
Fokko added a commit to Fokko/parquet-mr that referenced this pull request Apr 25, 2023
Fokko added a commit that referenced this pull request Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants