-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Bigtable: Add ability to run tests against direct path #5756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
6b8a042
Bigtable: Add ability to run tests against direct path
igorbernstein2 0652716
update docs
igorbernstein2 284aed9
cherrypick https://github.com/googleapis/google-cloud-java/pull/5762
igorbernstein2 303f0fa
include port in const
igorbernstein2 466d81e
Merge branch 'master' into directpath
igorbernstein2 442ac84
fix emulator system prop
igorbernstein2 9415eb9
Merge branch 'master' into directpath
igorbernstein2 7900ee1
make sure to download emulator before running the bigtable integratio…
igorbernstein2 146ff79
fix kokoro cmd line
igorbernstein2 73d9c8e
fix continuous & nightly jobs as well
igorbernstein2 42ee5e5
wrokaround flakes due to jdk bug
igorbernstein2 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
182 changes: 182 additions & 0 deletions
182
...-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/it/env/DirectPathEnv.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| /* | ||
| * Copyright 2019 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package com.google.cloud.bigtable.data.v2.it.env; | ||
|
|
||
| import com.google.api.core.ApiFuture; | ||
| import com.google.api.core.ApiFutures; | ||
| import com.google.api.gax.rpc.ServerStream; | ||
| import com.google.cloud.bigtable.data.v2.BigtableDataClient; | ||
| import com.google.cloud.bigtable.data.v2.BigtableDataSettings; | ||
| import com.google.cloud.bigtable.data.v2.models.Query; | ||
| import com.google.cloud.bigtable.data.v2.models.Row; | ||
| import com.google.cloud.bigtable.data.v2.models.RowMutation; | ||
| import com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStubSettings; | ||
| import com.google.common.collect.Lists; | ||
| import java.io.IOException; | ||
| import java.util.List; | ||
| import java.util.UUID; | ||
| import java.util.concurrent.ExecutionException; | ||
| import java.util.concurrent.TimeUnit; | ||
| import java.util.concurrent.TimeoutException; | ||
|
|
||
| /** | ||
| * Test environment that uses an existing bigtable table in the directpath environment. | ||
| * | ||
| * <p>The test environment can be accessed via both, CFE and direct path. The transport is | ||
| * controlled via the environment variable {@code GOOGLE_CLOUD_ENABLE_DIRECT_PATH}. Which contains | ||
| * comma delimited list of service names that should be accessed via direct path. | ||
| * | ||
| * <p>The target table must have a pre-existing family {@code cf}. The target table is configured | ||
| * via the system properties: | ||
| * | ||
| * <ul> | ||
| * <li>{@code bigtable.project} | ||
| * <li>{@code bigtable.instance} | ||
| * <li>{@code bigtable.table} | ||
| * </ul> | ||
| */ | ||
| public class DirectPathEnv implements TestEnv { | ||
| // TODO(igorbernstein): move direct path conditional logic to gax | ||
| private static final String DIRECT_PATH_ENV_VAR = "GOOGLE_CLOUD_ENABLE_DIRECT_PATH"; | ||
| private static final String DIRECT_PATH_END_POINT = | ||
| "testdirectpath-bigtable.sandbox.googleapis.com:443"; | ||
|
|
||
| private static final String PROJECT_PROPERTY_NAME = "bigtable.project"; | ||
| private static final String INSTANCE_PROPERTY_NAME = "bigtable.instance"; | ||
| private static final String TABLE_PROPERTY_NAME = "bigtable.table"; | ||
|
|
||
| private final boolean directPathEnabled; | ||
| private final String projectId; | ||
| private final String instanceId; | ||
| private final String tableId; | ||
| private static final String FAMILY_ID = "cf"; | ||
| private String rowPrefix; | ||
|
|
||
| private BigtableDataClient dataClient; | ||
|
|
||
| static DirectPathEnv create() { | ||
| return new DirectPathEnv( | ||
| isDirectPathEnabled(), | ||
| getRequiredProperty(PROJECT_PROPERTY_NAME), | ||
| getRequiredProperty(INSTANCE_PROPERTY_NAME), | ||
| getRequiredProperty(TABLE_PROPERTY_NAME)); | ||
| } | ||
|
|
||
| public DirectPathEnv( | ||
| boolean directPathEnabled, String projectId, String instanceId, String tableId) { | ||
| this.directPathEnabled = directPathEnabled; | ||
| this.projectId = projectId; | ||
| this.instanceId = instanceId; | ||
| this.tableId = tableId; | ||
| this.rowPrefix = UUID.randomUUID() + "-"; | ||
| } | ||
|
|
||
| @Override | ||
| public void start() throws IOException { | ||
| BigtableDataSettings.Builder settingsBuilder = | ||
| BigtableDataSettings.newBuilder().setProjectId(projectId).setInstanceId(instanceId); | ||
|
|
||
| // Direct path test environment uses a different endpoint. | ||
| settingsBuilder.stubSettings().setEndpoint(DIRECT_PATH_END_POINT); | ||
|
|
||
| // TODO(igorbernstein): move direct path conditional logic to gax | ||
| // Direct path environment can be accessed via CFE or direct path. When using direct path, | ||
| // disable connection pooling. | ||
| if (directPathEnabled) { | ||
| settingsBuilder | ||
| .stubSettings() | ||
| .setTransportChannelProvider( | ||
| EnhancedBigtableStubSettings.defaultGrpcTransportProviderBuilder() | ||
| .setPoolSize(1) | ||
| .build()); | ||
| } | ||
|
|
||
| dataClient = BigtableDataClient.create(settingsBuilder.build()); | ||
| } | ||
|
|
||
| @Override | ||
| public void stop() throws Exception { | ||
| deleteRows(); | ||
| dataClient.close(); | ||
| } | ||
|
|
||
| @Override | ||
| public BigtableDataClient getDataClient() { | ||
| return dataClient; | ||
| } | ||
|
|
||
| @Override | ||
| public String getProjectId() { | ||
| return projectId; | ||
| } | ||
|
|
||
| @Override | ||
| public String getInstanceId() { | ||
| return instanceId; | ||
| } | ||
|
|
||
| @Override | ||
| public String getTableId() { | ||
| return tableId; | ||
| } | ||
|
|
||
| @Override | ||
| public String getFamilyId() { | ||
| return FAMILY_ID; | ||
| } | ||
|
|
||
| @Override | ||
| public String getRowPrefix() { | ||
| return rowPrefix; | ||
| } | ||
|
|
||
| private void deleteRows() throws InterruptedException, ExecutionException, TimeoutException { | ||
| Query query = Query.create(tableId).prefix(rowPrefix); | ||
|
|
||
| List<ApiFuture<Void>> futures = Lists.newArrayList(); | ||
| ServerStream<Row> rows = dataClient.readRows(query); | ||
| for (Row row : rows) { | ||
| ApiFuture<Void> future = | ||
| dataClient.mutateRowAsync(RowMutation.create(tableId, row.getKey()).deleteRow()); | ||
| futures.add(future); | ||
| } | ||
|
|
||
| ApiFutures.allAsList(futures).get(10, TimeUnit.MINUTES); | ||
| } | ||
|
|
||
| private static String getRequiredProperty(String prop) { | ||
| String value = System.getProperty(prop); | ||
| if (value == null || value.isEmpty()) { | ||
| throw new RuntimeException("Missing system property: " + prop); | ||
| } | ||
| return value; | ||
| } | ||
|
|
||
| // TODO(igorbernstein): move direct path conditional logic to gax | ||
| private static boolean isDirectPathEnabled() { | ||
| String whiteList = System.getenv(DIRECT_PATH_ENV_VAR); | ||
| if (whiteList == null) { | ||
| return false; | ||
| } | ||
|
|
||
| for (String service : whiteList.split(",")) { | ||
| if (!service.isEmpty() && DIRECT_PATH_END_POINT.contains(service)) { | ||
| return true; | ||
| } | ||
| } | ||
| return false; | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.