From 84d29a61c48a37ed31e671fef676b7cdff0dd65e Mon Sep 17 00:00:00 2001 From: Blake Li Date: Mon, 5 Jun 2023 13:59:07 -0400 Subject: [PATCH 1/3] doc: Add test guide to DEVELOPMENT.md --- DEVELOPMENT.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 85fdd75ab4..6c8b42da61 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -38,4 +38,20 @@ for gapic-generator-java's Bazel build. ```sh mvn fmt:format - ``` \ No newline at end of file + ``` + +## Testing guide +There are 4 layers of testing in this repo: +1. Traditional unit tests +2. Golden unit tests +3. Showcase integration tests +4. Golden integration tests(We should stop adding new ones, and rely on golden unit tests and showcase tests) + +Based on where the code changes are, we should add different tests, in general + +- If the changes are in `gax/api-common` only, you must add traditional unit tests, you may add showcase integration tests if you are modifying a public setting. +- If the changes are in `gapic-generator-java` only, showcase integration tests are most likely not needed + - If it is in the `composer` module, you must add golden unit tests, you may add traditional unit tests for better coverage and easier testing. + - If it is in `other modules(ast, parser, writer etc.)`, you must add traditional unit tests, you may add golden unit tests to easily see the changes. +- If the changes are in both `gax` and `gapic-generator-java`, you must add all test layers, including traditional unit tests, golden unit tests and showcase integration tests. + From 1581a7fb68658161d3269d980af0050334072b46 Mon Sep 17 00:00:00 2001 From: Blake Li Date: Thu, 8 Jun 2023 17:54:38 -0400 Subject: [PATCH 2/3] doc: Add more details to test guide. --- DEVELOPMENT.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 6c8b42da61..92bbcd4034 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -42,16 +42,25 @@ for gapic-generator-java's Bazel build. ## Testing guide There are 4 layers of testing in this repo: -1. Traditional unit tests -2. Golden unit tests -3. Showcase integration tests -4. Golden integration tests(We should stop adding new ones, and rely on golden unit tests and showcase tests) +1. Traditional unit tests. These tests usually focus on testing one very specific test case at a time. For example, [MethodTest](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/test/java/com/google/api/generator/gapic/model/MethodTest.java) +2. Golden unit tests. These tests start with a test proto, we create different test cases in the proto, load the proto in unit tests and save the generated file as golden files. For example, [routing header proto](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/test/proto/explicit_dynamic_routing_header_testing.proto) -> Load in the [unit test](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/test/java/com/google/api/generator/gapic/composer/grpc/GrpcServiceStubClassComposerTest.java#L56-L64) -> Saved as [golden](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/test/java/com/google/api/generator/gapic/composer/grpc/goldens/GrpcRoutingHeadersStub.golden) +3. Showcase integration tests. They test the generated library's behavior against a mock server, see [README.md](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md) of showcase module for details. +4. Golden integration tests. These tests generate a full client library with real protos from googleapis, for example, golden integration tests for [compute](https://github.com/googleapis/sdk-platform-java/tree/main/test/integration/goldens/compute). However, we _should_ stop adding new ones, and rely on golden unit tests and showcase tests. Based on where the code changes are, we should add different tests, in general -- If the changes are in `gax/api-common` only, you must add traditional unit tests, you may add showcase integration tests if you are modifying a public setting. +- If the changes are in `gax` or `api-common` only, you _must_ add traditional unit tests, you _may_ add showcase integration tests if you are modifying a public setting. - If the changes are in `gapic-generator-java` only, showcase integration tests are most likely not needed - - If it is in the `composer` module, you must add golden unit tests, you may add traditional unit tests for better coverage and easier testing. - - If it is in `other modules(ast, parser, writer etc.)`, you must add traditional unit tests, you may add golden unit tests to easily see the changes. -- If the changes are in both `gax` and `gapic-generator-java`, you must add all test layers, including traditional unit tests, golden unit tests and showcase integration tests. + - If it is in the `composer` module, you _must_ add golden unit tests, you _may_ add traditional unit tests for better coverage and easier testing. + - If it is in the `parser` module, you _may_ add traditional unit tests with a test proto. For example, [routing_rule_parser_testing](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/test/proto/routing_rule_parser_testing.proto) that is only used for testing [RoutingRuleParser](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/src/main/java/com/google/api/generator/gapic/protoparser/RoutingRuleParser.java). + - If it is in `other modules(ast, model, writer etc.)`, you _must_ add traditional unit tests, you _may_ add golden unit tests to easily see the changes. +- If the changes are in both `gax` and `gapic-generator-java`, you _must_ add all test layers, including traditional unit tests, golden unit tests and showcase integration tests. +### How to run tests? + +See the [Unit Tests](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/DEVELOPMENT.md#unit-tests) section in gapic-generator-java/DEVELOPMENT.md for how to run traditional and golden unit tests. + +See [Running the Integration Tests](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#running-the-integration-tests) and [Update the Golden Showcase Files](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#update-the-golden-showcase-files) sections in showcase/README.md for how to run adn update showcase integration tests. + +See the [Integration Tests](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/DEVELOPMENT.md#integration-tests) section in gapic-generator-java/DEVELOPMENT.md for how to run golden integration tests. + \ No newline at end of file From a42b4ac80691f83c9efe964d8c4f3eba63119f2f Mon Sep 17 00:00:00 2001 From: Blake Li Date: Thu, 8 Jun 2023 18:09:21 -0400 Subject: [PATCH 3/3] Update DEVELOPMENT.md Co-authored-by: Emily Wang --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 92bbcd4034..835b72133d 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -60,7 +60,7 @@ Based on where the code changes are, we should add different tests, in general See the [Unit Tests](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/DEVELOPMENT.md#unit-tests) section in gapic-generator-java/DEVELOPMENT.md for how to run traditional and golden unit tests. -See [Running the Integration Tests](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#running-the-integration-tests) and [Update the Golden Showcase Files](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#update-the-golden-showcase-files) sections in showcase/README.md for how to run adn update showcase integration tests. +See [Running the Integration Tests](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#running-the-integration-tests) and [Update the Golden Showcase Files](https://github.com/googleapis/sdk-platform-java/blob/main/showcase/README.md#update-the-golden-showcase-files) sections in showcase/README.md for how to run and update showcase integration tests. See the [Integration Tests](https://github.com/googleapis/sdk-platform-java/blob/main/gapic-generator-java/DEVELOPMENT.md#integration-tests) section in gapic-generator-java/DEVELOPMENT.md for how to run golden integration tests. \ No newline at end of file