Skip to content

Conversation

@EronWright
Copy link
Contributor

@EronWright EronWright commented Jan 1, 2019

What is the purpose of the change

Please see commits prefixed with FLINK-11241.

This PR builds on #7390 (catalog descriptor/factory), to provide a typed API on TableEnvironment for connecting to a catalog via descriptor. You may see this API in action in the below snippet (from this example):

// register the taxi data tables under the "nyc" schema
tEnv.connect(new TaxiData()
        .ridesFile(ridesFile)
        .faresFile(faresFile)
        .maxEventDelaySecs(maxEventDelay)
        .servingSpeedFactor(servingSpeedFactor))
        .registerExternalCatalog("nyc");

// query across registered tables
tEnv.sqlQuery("SELECT f.rideId, f.driverId, f.totalFare FROM nyc.TaxiRides r INNER JOIN nyc.TaxiFares f ON r.rideId = f.rideId");

Brief change log

  • Added ConnectExternalCatalogDescriptor
  • Extended TableEnvironment with new connect method for catalogs

Verifying this change

This change added tests and can be verified as follows:

  • Added unit test ConnectExternalCatalogDescriptorTest for testing the new API

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDocs

…y table

- fix the logic in supportsBatch to properly declare a batch-only table
- adjust CommonTestData to provide batch-only or streaming-only tables

Signed-off-by: EronWright <eronwright@gmail.com>
Signed-off-by: EronWright <eronwright@gmail.com>
…w/ user classloader

- use the context classloader for all interactions with TableEnvironment

Signed-off-by: EronWright <eronwright@gmail.com>
Signed-off-by: EronWright <eronwright@gmail.com>
- add is-streaming property for increased flexibility
- fix style issue

Signed-off-by: EronWright <eronwright@gmail.com>
…og via a descriptor

- add ConnectExternalCatalogDescriptor
- add ConnectExternalCatalogDescriptorTest
- extend TableEnvironment with new connect method for catalogs

Signed-off-by: EronWright <eronwright@gmail.com>
…og via a descriptor

- add `TableEnvironment::listExternalCatalogs`

Signed-off-by: EronWright <eronwright@gmail.com>
…og via a descriptor

- fix scalastyle check

Signed-off-by: EronWright <eronwright@gmail.com>
@EronWright
Copy link
Contributor Author

@twalthr ping

@EronWright
Copy link
Contributor Author

EronWright commented Jul 7, 2019

Am closing since I think catalog subsystem is going in a different direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants