chore: add readme for google-cloud-bigquery-jdbc
#4071
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,317 @@ | ||||||
| # Google BigQuery JDBC Client for Java | ||||||
|
|
||||||
| Java idiomatic client for [BigQuery JDBC][product-docs]. | ||||||
|
|
||||||
| [![Maven][maven-version-image]][maven-version-link] | ||||||
| ![Stability][stability-image] | ||||||
|
|
||||||
| - [Product Documentation][product-docs] | ||||||
| - [Client Library Documentation][javadocs] | ||||||
|
|
||||||
|
|
||||||
| ## Quickstart | ||||||
|
|
||||||
|
|
||||||
| If you are using Maven, add this to your pom.xml file: | ||||||
|
|
||||||
| <!-- {x-version-update-start:google-cloud-bigquery-jdbc:released} --> | ||||||
|
|
||||||
| ```xml | ||||||
| <dependency> | ||||||
| <groupId>com.google.cloud</groupId> | ||||||
| <artifactId>google-cloud-bigquery-jdbc</artifactId> | ||||||
| <version>0.1.0-SNAPSHOT</version> | ||||||
| </dependency> | ||||||
| ``` | ||||||
|
|
||||||
| If you are using Gradle without BOM, add this to your dependencies: | ||||||
|
|
||||||
| ```Groovy | ||||||
| implementation 'com.google.cloud:google-cloud-bigquery-jdbc:0.1.0-SNAPSHOT' | ||||||
| ``` | ||||||
|
|
||||||
| If you are using SBT, add this to your dependencies: | ||||||
|
|
||||||
| ```Scala | ||||||
| libraryDependencies += "com.google.cloud" % "google-cloud-bigquery-jdbc" % "0.1.0-SNAPSHOT" | ||||||
| ``` | ||||||
| <!-- {x-version-update-end} --> | ||||||
|
|
||||||
| ## Authentication | ||||||
|
|
||||||
| See the [Authentication][authentication] section in the base directory's README. | ||||||
|
|
||||||
| ## Authorization | ||||||
|
|
||||||
| The client application making API calls must be granted [authorization scopes][auth-scopes] required for the desired BigQuery JDBC APIs, and the authenticated principal must have the [IAM role(s)][predefined-iam-roles] required to access GCP resources using the BigQuery JDBC API calls. | ||||||
|
|
||||||
| ## Developer Guide | ||||||
|
|
||||||
| ### Prerequisites | ||||||
|
|
||||||
| You need to have either Java with Maven installed or Docker. You might want to install [`Make`](https://www.gnu.org/software/make/) to simplify running commands, otherwise please look into Makefile to check for specific configurations. | ||||||
|
|
||||||
| ### Setup | ||||||
|
|
||||||
| `make install` primarily relies on `mvn install` command. All following commands are primarily applicable for the `google-cloud-bigquery-jdbc` project. | ||||||
|
|
||||||
| `bigquery-external-jdbc-tests` at this moment is indepdnent and has its own configuration. | ||||||
|
|
||||||
| ### Running tests | ||||||
|
|
||||||
| #### Unittests | ||||||
|
|
||||||
| Run all unittests | ||||||
|
|
||||||
| `make unittest` | ||||||
|
|
||||||
| Run specific unittests | ||||||
|
|
||||||
| `make unittest test=<filter>` | ||||||
|
|
||||||
| Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` | ||||||
|
|
||||||
| Example: `make unittest test=BigQueryArrowStructTest` | ||||||
|
|
||||||
| #### Integration tests | ||||||
|
|
||||||
| IMPORTANT: Running integration tests will skip unit tests. Primary focus of this command is to run specific set of tests without a lot of overhead. | ||||||
|
|
||||||
| Majority of Integration tests rely on certain environment variables: | ||||||
|
|
||||||
| ``` | ||||||
| # Default gcloud auth setup | ||||||
| export GOOGLE_APPLICATION_CREDENTIALS=<path-to-service-account-json-config-file> | ||||||
|
|
||||||
| # Test specfic envs | ||||||
| export SA_EMAIL=email@email.com | ||||||
| export SA_SECRET=<path-to-service-account-json-config-file> | ||||||
| # Alternatively it can be json content: | ||||||
| export SA_SECRET=`cat <path-to-service-account-json-config-file>` | ||||||
| ``` | ||||||
|
|
||||||
| Run all integration tests (currently takes 15+ minutes, so this is discouraged). | ||||||
|
|
||||||
| `make integration-test` | ||||||
|
|
||||||
| Run specific integration test | ||||||
|
|
||||||
| `make integration-test test=<filter>` | ||||||
|
|
||||||
| Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` | ||||||
|
|
||||||
| Example: `make unittest test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey` | ||||||
|
Contributor
There was a problem hiding this comment. The example command
Suggested change
|
||||||
|
|
||||||
| ### Dockerized environment | ||||||
|
|
||||||
| If you don't have Java or Maven, or if you want to test changes with a different Java version, you can leverage dockerized environment. | ||||||
|
|
||||||
| One-time run commands are similar to local development make commands: | ||||||
|
|
||||||
| ``` | ||||||
| make docker-build | ||||||
| make docker-unittest | ||||||
| make docker-integration-test | ||||||
| ``` | ||||||
|
|
||||||
| Please note that running unit or integration tests within docker doesn't leverage maven cache because it is not persisted. | ||||||
| If you want to run multiple commands, you can start a session and treat this shell session as your local environment. | ||||||
|
|
||||||
| ``` | ||||||
| make docker-session | ||||||
| ``` | ||||||
|
|
||||||
| All Docker commands rely only on `GOOGLE_APPLICATION_CREDENTIALS` env being present, it will create rest of env vars as needed. | ||||||
|
|
||||||
| ### Packaging | ||||||
|
|
||||||
| There are two ways to package Google JDBC Driver: thin jar and jar with all dependencies. | ||||||
|
|
||||||
| #### Thin jar | ||||||
|
|
||||||
| Thin jar can be generated by running `make docker-package`. It generates jar and list of dependencies in format compatible with copying into `pom.xml` file. | ||||||
|
|
||||||
| #### All dependencies | ||||||
|
|
||||||
| Jar with all dependencies can be generated by running `make docker-package-all-dependencies`. Produced jar includes all depenencies and can be used as a standalon jar with tools like R-Studio. | ||||||
|
|
||||||
| ### Custom query tests | ||||||
|
|
||||||
| Thin client using JDBC driver can be found in 'bigquery-jdbc-test-client' folder. It allows you to run custom queries with custom connection string ot generate logs or measure performance. | ||||||
|
Contributor
There was a problem hiding this comment. There is a typo here; 'ot' should be 'to'.
Suggested change
|
||||||
|
|
||||||
| It can be built for Google JDBC drivers or any other JDBC drivers. For other JDBC drivers, you need to specify a version and it will download it. | ||||||
|
|
||||||
| #### Google test | ||||||
|
|
||||||
| For Google, you need to build an uber jar following previous steps. When it is ready, you can use these | ||||||
| sample commands to run it: | ||||||
|
|
||||||
| ``` | ||||||
| cd bigquery-jdbc-test-client | ||||||
| make docker-package-for-validation JDBC_JAR_PATH=<path-to-uber-jar> | ||||||
| make run QUERY=small | ||||||
| ``` | ||||||
|
|
||||||
| ### Night builds | ||||||
|
|
||||||
| Nightly build is running full set of Integration tests and extended Nightly tests that includes some long-running tests (takes 20+ minutes to complete). | ||||||
|
|
||||||
| Nightly Integration Tests include a step to build a full and thin jars and upload them to Google Storage. | ||||||
|
|
||||||
| They can be retrieved via following commands: | ||||||
|
|
||||||
| ``` | ||||||
| gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest-full.jar . | ||||||
| gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest.zip . | ||||||
| ``` | ||||||
|
|
||||||
| #### Performance tests | ||||||
|
|
||||||
| Cloud Build Pipeline is uploading latest full jar to the internal location for perf tests once a week, at the moment scheduled to do this Sunday night. | ||||||
|
|
||||||
| ### Code Coverage | ||||||
|
|
||||||
| We're using [JaCoCo](https://www.eclemma.org/jacoco/) to track Code Coverage. `Makefile` has 2 separate set of commands for unittests and integration tests reports. | ||||||
|
|
||||||
| You can run `make docker-coverage` and it will produce both results, you can find `jacoco-(unittest|full).zip` file in the root with results. | ||||||
|
|
||||||
| ## Getting Started | ||||||
|
|
||||||
| ### Prerequisites | ||||||
|
|
||||||
| You will need a [Google Cloud Platform Console][developer-console] project with the BigQuery JDBC [API enabled][enable-api]. | ||||||
|
|
||||||
| [Follow these instructions][create-project] to get your project set up. You will also need to set up the local development environment by | ||||||
| [installing the Google Cloud Command Line Interface][cloud-cli] and running the following commands in command line: | ||||||
| `gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`. | ||||||
|
|
||||||
| ### Installation and setup | ||||||
|
|
||||||
| You'll need to obtain the `google-cloud-bigquery-jdbc` library. See the [Quickstart](#quickstart) section | ||||||
| to add `google-cloud-bigquery-jdbc` as a dependency in your code. | ||||||
|
|
||||||
| ## About BigQuery JDBC | ||||||
|
|
||||||
|
|
||||||
| [BigQuery JDBC][product-docs] | ||||||
|
|
||||||
| See the [BigQuery JDBC client library docs][javadocs] to learn how to | ||||||
| use this BigQuery JDBC Client Library. | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ## Troubleshooting | ||||||
|
|
||||||
| To get help, follow the instructions in the [shared Troubleshooting document][troubleshooting]. | ||||||
|
|
||||||
| ## Supported Java Versions | ||||||
|
|
||||||
| Java 8 or above is required for using this client. | ||||||
|
|
||||||
| Google's Java client libraries, | ||||||
| [Google Cloud Client Libraries][cloudlibs] | ||||||
| and | ||||||
| [Google Cloud API Libraries][apilibs], | ||||||
| follow the | ||||||
| [Oracle Java SE support roadmap][oracle] | ||||||
| (see the Oracle Java SE Product Releases section). | ||||||
|
|
||||||
| ### For new development | ||||||
|
|
||||||
| In general, new feature development occurs with support for the lowest Java | ||||||
| LTS version covered by Oracle's Premier Support (which typically lasts 5 years | ||||||
| from initial General Availability). If the minimum required JVM for a given | ||||||
| library is changed, it is accompanied by a [semver][semver] major release. | ||||||
|
|
||||||
| Java 11 and (in September 2021) Java 17 are the best choices for new | ||||||
| development. | ||||||
|
|
||||||
| ### Keeping production systems current | ||||||
|
|
||||||
| Google tests its client libraries with all current LTS versions covered by | ||||||
| Oracle's Extended Support (which typically lasts 8 years from initial | ||||||
| General Availability). | ||||||
|
|
||||||
| #### Legacy support | ||||||
|
|
||||||
| Google's client libraries support legacy versions of Java runtimes with long | ||||||
| term stable libraries that don't receive feature updates on a best efforts basis | ||||||
| as it may not be possible to backport all patches. | ||||||
|
|
||||||
| Google provides updates on a best efforts basis to apps that continue to use | ||||||
| Java 7, though apps might need to upgrade to current versions of the library | ||||||
| that supports their JVM. | ||||||
|
|
||||||
| #### Where to find specific information | ||||||
|
|
||||||
| The latest versions and the supported Java versions are identified on | ||||||
| the individual GitHub repository `github.com/GoogleAPIs/java-SERVICENAME` | ||||||
| and on [google-cloud-java][g-c-j]. | ||||||
|
|
||||||
| ## Versioning | ||||||
|
|
||||||
|
|
||||||
| This library follows [Semantic Versioning](http://semver.org/). | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ## Contributing | ||||||
|
|
||||||
|
|
||||||
| Contributions to this library are always welcome and highly encouraged. | ||||||
|
|
||||||
| See [CONTRIBUTING][contributing] for more information how to get started. | ||||||
|
|
||||||
| Please note that this project is released with a Contributor Code of Conduct. By participating in | ||||||
| this project you agree to abide by its terms. See [Code of Conduct][code-of-conduct] for more | ||||||
| information. | ||||||
|
|
||||||
|
|
||||||
| ## License | ||||||
|
|
||||||
| Apache 2.0 - See [LICENSE][license] for more information. | ||||||
|
|
||||||
| ## CI Status | ||||||
|
|
||||||
| Java Version | Status | ||||||
| ------------ | ------ | ||||||
| Java 8 | [![Kokoro CI][kokoro-badge-image-2]][kokoro-badge-link-2] | ||||||
| Java 8 OSX | [![Kokoro CI][kokoro-badge-image-3]][kokoro-badge-link-3] | ||||||
| Java 8 Windows | [![Kokoro CI][kokoro-badge-image-4]][kokoro-badge-link-4] | ||||||
| Java 11 | [![Kokoro CI][kokoro-badge-image-5]][kokoro-badge-link-5] | ||||||
|
|
||||||
| Java is a registered trademark of Oracle and/or its affiliates. | ||||||
|
|
||||||
| [product-docs]: https://cloud.google.com/bigquery | ||||||
| [javadocs]: https://cloud.google.com/java/docs/reference/google-cloud-bigquery/latest/history | ||||||
| [kokoro-badge-image-2]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8.svg | ||||||
| [kokoro-badge-link-2]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8.html | ||||||
| [kokoro-badge-image-3]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-osx.svg | ||||||
| [kokoro-badge-link-3]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-osx.html | ||||||
| [kokoro-badge-image-4]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-win.svg | ||||||
| [kokoro-badge-link-4]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-win.html | ||||||
| [kokoro-badge-image-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java11.svg | ||||||
| [kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java11.html | ||||||
| [stability-image]: https://img.shields.io/badge/stability-unknown-red | ||||||
| [maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-bigquery-jdbc.svg | ||||||
| [maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-bigquery-jdbc/0.0.0 | ||||||
| [authentication]: https://github.com/googleapis/google-cloud-java#authentication | ||||||
| [auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes | ||||||
| [predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles | ||||||
| [iam-policy]: https://cloud.google.com/iam/docs/overview#cloud-iam-policy | ||||||
| [developer-console]: https://console.developers.google.com/ | ||||||
| [create-project]: https://cloud.google.com/resource-manager/docs/creating-managing-projects | ||||||
| [cloud-cli]: https://cloud.google.com/cli | ||||||
| [troubleshooting]: https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md | ||||||
| [contributing]: https://github.com/googleapis/java-bigquery/blob/main/CONTRIBUTING.md | ||||||
| [code-of-conduct]: https://github.com/googleapis/java-bigquery/blob/main/CODE_OF_CONDUCT.md | ||||||
| [license]: https://github.com/googleapis/java-bigquery/blob/main/LICENSE | ||||||
|
|
||||||
| [semver]: https://semver.org/ | ||||||
| [cloudlibs]: https://cloud.google.com/apis/docs/client-libraries-explained | ||||||
| [apilibs]: https://cloud.google.com/apis/docs/client-libraries-explained#google_api_client_libraries | ||||||
| [oracle]: https://www.oracle.com/java/technologies/java-se-support-roadmap.html | ||||||
| [g-c-j]: http://github.com/googleapis/google-cloud-java | ||||||
keshavdandeva marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a typo here; 'indepdnent' should be 'independent'.