Skip to content

Conversation

@EronWright
Copy link
Contributor

@EronWright EronWright commented Jan 1, 2019

What is the purpose of the change

Based on PR #7389; please see commits that are prefixed with FLINK-11240.

Implements the descriptor and factory infrastructure for external catalogs.

I found it counterproductive to involve ConnectorDescriptor for external catalogs. That type was designed to support a mix-and-match between connector and format. See FLINK-11241 for more.

On the topic of the transition to flink-table-common, I found it convenient to keep the new types in flink-table until ExternalCatalog moves (FLINK-10755). Java was used whenever possible.

Closes FLINK-11240.

Brief change log

  • Add ExternalCatalogDescriptor
  • Add ExternalCatalogDescriptorValidator
  • Add ExternalCatalogFactory
  • Add TableFactoryUtil::findAndCreateExternalCatalog

Verifying this change

This change added tests and can be verified as follows:

  • Added unit test ExternalCatalogDescriptorTest for descriptors
  • Added unit test ExternalCatalogFactoryServiceTest for factory discovery
  • Added a sample factory TestExternalCatalogFactory for test purposes which serves the catalog provided by the existing CommonTestData::getInMemoryTestCatalog.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDocs

…y table

- fix the logic in supportsBatch to properly declare a batch-only table
- adjust CommonTestData to provide batch-only or streaming-only tables

Signed-off-by: EronWright <eronwright@gmail.com>
Signed-off-by: EronWright <eronwright@gmail.com>
…w/ user classloader

- use the context classloader for all interactions with TableEnvironment

Signed-off-by: EronWright <eronwright@gmail.com>
Signed-off-by: EronWright <eronwright@gmail.com>
@EronWright EronWright closed this Jan 1, 2019
- add is-streaming property for increased flexibility
- fix style issue

Signed-off-by: EronWright <eronwright@gmail.com>
@EronWright EronWright changed the title [FLINK-11237] [table] External Catalog Factory and Descriptor [FLINK-11240] [table] External Catalog Factory and Descriptor Jan 1, 2019
@EronWright EronWright reopened this Jan 1, 2019
@EronWright
Copy link
Contributor Author

@twalthr any reservation about merging this?

@EronWright
Copy link
Contributor Author

@twalthr ping

Copy link
Contributor

@twalthr twalthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @EronWright for this contribution. The changes look good to me. I will rebase this PR and merge this :) Sorry, for the delay.

@asfgit asfgit closed this in bd2dae6 Apr 2, 2019
HuangZhenQiu pushed a commit to HuangZhenQiu/flink that referenced this pull request Apr 22, 2019
sunhaibotb pushed a commit to sunhaibotb/flink that referenced this pull request May 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants