Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9297acc
Add stubs for GetMetadata endpoint
TSultanov Feb 22, 2023
8d9de57
Add GetMetadata() endpoint to controller
TSultanov Feb 22, 2023
a9a5510
Add Beam SDK version to go build
TSultanov Feb 22, 2023
42b5dba
Remove commented out code
TSultanov Feb 23, 2023
4fd3753
Change BuildCommitTimestamp type to int64
TSultanov Feb 24, 2023
ecd840a
Reformat containers build.gradle
TSultanov Feb 24, 2023
5877451
Change syntax in git-functions.gradle
TSultanov Feb 24, 2023
92fe2b4
Make build_commit and beam_sdk fields optional
TSultanov Feb 24, 2023
90278a3
Update Tour of Beam protocol definitions to reflect Playground protoc…
TSultanov Mar 1, 2023
5cf5099
Show versions on the frontend (#22271)
alexeyinkin Mar 1, 2023
62fe1e7
Remove config.g.dart generation (#22271)
alexeyinkin Mar 1, 2023
0d41f15
Add config.g.dart (#22271)
alexeyinkin Mar 1, 2023
2ffc78e
Address review comments (#22271)
alexeyinkin Mar 2, 2023
ee51109
Clean up (#22271)
alexeyinkin Mar 2, 2023
097f468
Clean up (#22271)
alexeyinkin Mar 2, 2023
55c5c25
Add license to build_metadata.g.dart (#22271)
alexeyinkin Mar 2, 2023
b7b1e9e
Address review comments (#22271)
alexeyinkin Mar 6, 2023
3d581e5
Address review comments (#22271)
alexeyinkin Mar 6, 2023
0d31d6f
Merge remote-tracking branch 'upstream/master' into get-metadata-endp…
TSultanov Mar 7, 2023
26edffd
Add a TODO (#22271)
alexeyinkin Mar 8, 2023
e343078
Fix tabulation
TSultanov Mar 9, 2023
ce86f05
Remove optional fields from Playground protocol
TSultanov Mar 9, 2023
f738096
Document the backend lookup (#22271)
alexeyinkin Mar 9, 2023
222bcf3
Improve handling of empty values coming from GetMetadataResponse (#22…
alexeyinkin Mar 9, 2023
0e45fbc
Merge remote-tracking branch 'apache/master' into get-metadata-endpoi…
alexeyinkin Mar 10, 2023
96badc3
Fix integration tests (#22271)
alexeyinkin Mar 10, 2023
8a6c166
Merge remote-tracking branch 'upstream/master' into get-metadata-endp…
TSultanov Mar 16, 2023
1cff6fd
Remove protoc requirement from Readme
TSultanov Mar 16, 2023
1336e8e
Remove trailing whitespace
TSultanov Mar 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/playground_frontend_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,5 @@ jobs:

- name: 'Integration tests'
run: |
cp playground/frontend/lib/config.example.dart playground/frontend/lib/config.g.dart
chromedriver --port=4444 &
./gradlew :playground:frontend:integrationTest -PdeviceId=web-server
1 change: 1 addition & 0 deletions learning/tour-of-beam/backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ To update mocks for tests, run:
$ go generate -x ./...
```

> Note: [`moq`](https://github.com/matryer/moq) tool to be installed

### Datastore schema

Expand Down
521 changes: 340 additions & 181 deletions learning/tour-of-beam/backend/playground_api/api/v1/api.pb.go

Large diffs are not rendered by default.

38 changes: 38 additions & 0 deletions learning/tour-of-beam/backend/playground_api/api/v1/api_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

56 changes: 56 additions & 0 deletions learning/tour-of-beam/backend/playground_api/api/v1/mock.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions playground/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ cd beam

### Router, runners, and frontend

1. Edit `/playground/frontend/lib/config.g.dart` to set your local backend host and ports
found in `/playground/docker-compose.local.yaml`.
1. Edit `/playground/frontend/playground_components/lib/src/constants/backend_urls.dart`
to override backend URLs with yours found in `/playground/docker-compose.local.yaml`.
2. To start, run:

```bash
Expand All @@ -106,6 +106,8 @@ If you do not need particular runners, comment out:
1. Dependencies on them in `/playground/build.gradle.kts` in `dockerComposeLocalUp` task.
2. Their Docker image configurations in `/playground/docker-compose.local.yaml`.

See also [Backend Lookup](frontend/README.md#backend-lookup) in the Frontend.

## Removing old snippets

Run the method to remove unused code snippets from the Cloud Datastore. Unused snippets are snippets that are out of date. If the last visited date property less or equals than the current date minus dayDiff parameter then a snippet is out of date
Expand Down
14 changes: 14 additions & 0 deletions playground/api/v1/api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,17 @@ message GetSnippetResponse {
Complexity complexity = 4;
}

// GetMetadataRequest represents request for runner metadata
message GetMetadataRequest {}

// GetMetadataResponse contains metadata about the runner
message GetMetadataResponse {
string runner_sdk = 1;
string build_commit_hash = 2;
int64 build_commit_timestamp_seconds_since_epoch = 3;
string beam_sdk_version = 4;
}

service PlaygroundService {

// Submit the job for an execution and get the pipeline uuid.
Expand Down Expand Up @@ -372,4 +383,7 @@ service PlaygroundService {

// Get the snippet of playground.
rpc GetSnippet(GetSnippetRequest) returns (GetSnippetResponse);

// Get the runner metadata.
rpc GetMetadata(GetMetadataRequest) returns (GetMetadataResponse);
}
26 changes: 26 additions & 0 deletions playground/backend/cmd/server/build_info.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

// This file is intended to store build-time variables.
// To set a value to any of this variable use '-ldflags="-X main.<variable_name>=<variable_value>"'
// option when invoking 'go build'.

// BuildCommitHash will contain hash of git commit from which the application is built
var BuildCommitHash string

// BuildCommitTimestamp will contain timestamp of git commit from which the application is built
var BuildCommitTimestamp string
22 changes: 20 additions & 2 deletions playground/backend/cmd/server/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@
package main

import (
"cloud.google.com/go/datastore"
"context"
"errors"

"cloud.google.com/go/datastore"
"github.com/google/uuid"
"strconv"

pb "beam.apache.org/playground/backend/internal/api/v1"
"beam.apache.org/playground/backend/internal/cache"
Expand Down Expand Up @@ -529,3 +529,21 @@ func (controller *playgroundController) GetSnippet(ctx context.Context, info *pb
}
return &response, nil
}

// GetMetadata returns runner metadata
func (controller *playgroundController) GetMetadata(_ context.Context, _ *pb.GetMetadataRequest) (*pb.GetMetadataResponse, error) {
commitTimestampInteger, err := strconv.ParseInt(BuildCommitTimestamp, 10, 64)
if err != nil {
logger.Errorf("GetMetadata(): failed to parse BuildCommitTimestamp (\"%s\"): %s", BuildCommitTimestamp, err.Error())
commitTimestampInteger = 0
}

response := pb.GetMetadataResponse{
RunnerSdk: controller.env.BeamSdkEnvs.ApacheBeamSdk.String(),
BuildCommitHash: BuildCommitHash,
BuildCommitTimestampSecondsSinceEpoch: commitTimestampInteger,
BeamSdkVersion: controller.env.BeamSdkEnvs.BeamVersion,
}

return &response, nil
}
37 changes: 37 additions & 0 deletions playground/backend/containers/git-functions.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* License); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

ext.getGitCommitHash = () -> {
def stdout = new ByteArrayOutputStream()
exec {
executable('git')
args('rev-parse', 'HEAD')
standardOutput = stdout
}
return stdout.toString().trim()
} as Closure<String>

ext.getGitCommitTimestamp = () -> {
def stdout = new ByteArrayOutputStream()
exec {
executable('git')
args('show', '-s', '--format=%ct', 'HEAD')
standardOutput = stdout
}
return stdout.toString().trim()
} as Closure<String>
Comment on lines +19 to +37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a gradle wrapper needed? Why not just run the command in the context where the value is needed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this in 6 separate locations:

  • Building the frontend container.
  • Building the router container.
  • Building the 4 runners.

The 5 backends may be similar but frontend and backend are still two rather distinct users of this. This motivates to do this in a single location like this because duplicating git show -s --format=%ct HEAD and git rev-parse HEAD 2-6 times is more confusing.

Next, we cannot do this in Dockerfile because by the time it is built the directory is unaware of .git.

This leads to the requirement of having the values before the 6 containers' builds start. And their builds are triggered from Gradle. So we cannot shorten this to command $(git rev-parse HEAD), we would still have the whole exec { ... } thing but in 12 locations.

5 changes: 4 additions & 1 deletion playground/backend/containers/go/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ ARG BASE_IMAGE
FROM $BASE_IMAGE
ARG SDK_TAG
ARG SDK_TAG_LOCAL
ARG GIT_COMMIT="<unknown>"
ARG GIT_TIMESTAMP="0"

# Setup Go Environment
ENV GOPATH /go
Expand Down Expand Up @@ -48,8 +50,9 @@ WORKDIR /go/src/playground/backend
RUN go mod download &&\
go mod tidy &&\
cd cmd/server &&\
go build -o /opt/playground/backend/server_go_backend
go build -ldflags="-X main.BuildCommitHash=$GIT_COMMIT -X main.BuildCommitTimestamp=$GIT_TIMESTAMP" -o /opt/playground/backend/server_go_backend

ENV BEAM_VERSION=$SDK_TAG
ENV SERVER_IP=0.0.0.0
ENV SERVER_PORT=8080
ENV APP_WORK_DIR=/opt/playground/backend/
Expand Down
Loading