Skip to content

Run the e2e tests in parallel #1670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srinivashegde86 wants to merge 3 commits into from
Closed

Run the e2e tests in parallel #1670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b647049
Run the e2e tests in parallel
srinivashegde86 Jul 24, 2018
585abae
Run e2e tests in parallel
srinivashegde86 Aug 4, 2018
10db5c0
Merge remote-tracking branch 'upstream/master' into integrationTests
srinivashegde86 Aug 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ To run [the e2e tests](./e2e) and [the conformance tests](./conformance), you ne

```bash
go test -v -tags=e2e -count=1 ./test/conformance
go test -v -tags=e2e -count=1 ./test/e2e
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e
Copy link
Copy Markdown
Contributor

@adrcunha adrcunha Jul 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running tests in parallel should be optional. I suggest you mention it in the docs, but avoid repeating it every time.

```

### One test case
Expand Down Expand Up @@ -77,13 +77,14 @@ These tests require:
enabled is recommended (`-v`).
* Using [`--logverbose`](#output-verbose-log) to see the verbose log output from test as well as from k8s libraries.
* Using `-count=1` is [the idiomatic way to disable test caching](https://golang.org/doc/go1.10#test)
* Using `-parallel=4` to run upto 4 [parallel test functions](https://golang.org/pkg/testing/#T.Parallel) from the test package in [parallel](https://golang.org/cmd/go/#hdr-Testing_flags)

You can [use test flags](#flags) to control the environment
your tests run against, i.e. override [your environment variables](/DEVELOPMENT.md#environment-setup):

```bash
go test -v -tags=e2e -count=1 ./test/conformance --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
go test -v -tags=e2e -count=1 ./test/e2e --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
```

If you are running against an environment with no loadbalancer for the ingress, at the moment
Expand All @@ -92,7 +93,7 @@ your only option is to use a domain which will resolve to the IP of the running

```bash
go test -v -tags=e2e -count=1 ./test/conformance --resolvabledomain
go test -v -tags=e2e -count=1 ./test/e2e --resolvabledomain
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --resolvabledomain
```

## Test images
Expand Down Expand Up @@ -149,7 +150,7 @@ To run the tests with a non-default kubeconfig file:

```bash
go test -v -tags=e2e -count=1 ./test/conformance --kubeconfig /my/path/kubeconfig
go test -v -tags=e2e -count=1 ./test/e2e --kubeconfig /my/path/kubeconfig
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --kubeconfig /my/path/kubeconfig
```

### Specifying cluster
Expand All @@ -161,7 +162,7 @@ if not specified.

```bash
go test -v -tags=e2e -count=1 ./test/conformance --cluster your-cluster-name
go test -v -tags=e2e -count=1 ./test/e2e --cluster your-cluster-name
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --cluster your-cluster-name
```

The current cluster names can be obtained by running:
Expand All @@ -177,7 +178,7 @@ tests. By default, tests will use `serving-tests`.

```bash
go test -v -tags=e2e -count=1 ./test/conformance --namespace your-namespace-name
go test -v -tags=e2e -count=1 ./test/e2e --namespace your-namespace-name
go test -v -tags=e2e -parallel=4 -count=1 ./test/e2e --namespace your-namespace-name
```

### Overridding docker repo
Expand All @@ -189,7 +190,7 @@ if not specified.

```bash
go test -v -tags=e2e -count=1 ./test/conformance --dockerrepo gcr.myhappyproject
go test -v -tags=e2e -count=1 ./test/e2e --dockerrepo gcr.myhappyproject
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --dockerrepo gcr.myhappyproject
```

### Using a resolvable domain
Expand All @@ -212,7 +213,7 @@ If you have configured your cluster to use a resolvable domain, you can use the
The `--logverbose` argument lets you see verbose test logs and k8s logs.

```bash
go test -v -tags=e2e -count=1 ./test/e2e --logverbose
go test -v -tags=e2e -count=1 -parallel=4 ./test/e2e --logverbose
```

### Emit metrics
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/autoscale_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ func setScaleToZeroThreshold(clients *test.Clients, threshold string) error {
}

func setup(t *testing.T, logger *zap.SugaredLogger) *test.Clients {
clients := Setup(t)
clients := Setup(t, false)

configMap, err := getAutoscalerConfigMap(clients)
if err != nil {
Expand Down
4 changes: 2 additions & 2 deletions test/e2e/build_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import (
)

func TestBuildAndServe(t *testing.T) {
clients := Setup(t)
clients := Setup(t, true)

// Add test case specific name to its own logger.
logger := test.GetContextLogger("TestBuildAndServe")
Expand Down Expand Up @@ -101,7 +101,7 @@ func TestBuildAndServe(t *testing.T) {
}

func TestBuildFailure(t *testing.T) {
clients := Setup(t)
clients := Setup(t, true)

// Add test case specific name to its own logger.
logger := test.GetContextLogger("TestBuildFailure")
Expand Down
9 changes: 7 additions & 2 deletions test/e2e/e2e.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,17 @@ const (
defaultNamespaceName = "serving-tests"
)

// Setup creates the client objects needed in the e2e tests.
func Setup(t *testing.T) *test.Clients {
// Setup creates the client objects needed in the e2e tests. If parallel is true,
// the test will be marked so as to run in parallel with other parallel tests.
func Setup(t *testing.T, parallel bool) *test.Clients {
if test.Flags.Namespace == "" {
test.Flags.Namespace = defaultNamespaceName
}

if parallel {
t.Parallel()
Copy link
Copy Markdown
Contributor

@adrcunha adrcunha Jul 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, one shouldn't need to figure out what tests can and what tests cannot run in parallel.

}

clients, err := test.NewClients(
test.Flags.Kubeconfig,
test.Flags.Cluster,
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/errorcondition_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ const (

func TestContainerErrorMsg(t *testing.T) {
//t.Skip("Skipping until https://github.com/knative/serving/issues/1240 is closed")
clients := Setup(t)
clients := Setup(t, true)

//add test case specific name to its own logger
logger := test.GetContextLogger("TestContainerErrorMsg")
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/helloworld_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ const (
)

func TestHelloWorld(t *testing.T) {
clients := Setup(t)
clients := Setup(t, true)

//add test case specific name to its own logger
logger := test.GetContextLogger("TestHelloWorld")
Expand Down