[Tavern] Post-MVP Refactor

docs/_docs/dev-guide/tavern.md

@@ -1,18 +1,22 @@
 ---
 title: Tavern
-tags: 
+tags:
  - Dev Guide
 description: Want to contribute to Tavern? Start here!
 permalink: dev-guide/tavern
 ---
-# Overview
+
+## Overview
+
 Tavern is a teamserver for Realm, providing a UI to control deployments and implants during an engagement. The majority of Tavern's functionality is exposed through a GraphQL API, which is used by both implants and the UI.
 
 If you would like to help contribute to Tavern, please take a look at our [open issues](https://github.com/KCarretto/realm/issues?q=is%3Aopen+is%3Aissue+label%3Atavern).
 
-# Configuration
-## MySQL
-By default, Tavern operates an in-memory SQLite database. To persist data, a MySQL backend is supported. In order to configure Tavern to use MySQL, the `MYSQL_ADDR` environment variable must be set to the `host[:port]` of the database (e.g. `127.0.0.1`, `mydb.com`, or `mydb.com:3306`). You can reference the [mysql.Config](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config) for additional information about Tavern's MySQL configuration. 
+## Configuration
+
+### MySQL
+
+By default, Tavern operates an in-memory SQLite database. To persist data, a MySQL backend is supported. In order to configure Tavern to use MySQL, the `MYSQL_ADDR` environment variable must be set to the `host[:port]` of the database (e.g. `127.0.0.1`, `mydb.com`, or `mydb.com:3306`). You can reference the [mysql.Config](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config) for additional information about Tavern's MySQL configuration.
 
 The following environment variables are currently supported for additional MySQL Configuration:
 
@@ -23,20 +27,25 @@ The following environment variables are currently supported for additional MySQL
 | MYSQL_USER| User to authenticate with | root | No |
 | MYSQL_PASSWD| Password to authenticate with | no password | No |
 | MYSQL_DB| Name of the database to use | tavern | No |
+| MYSQL_MAX_IDLE_CONNS | Integer value of max idle mysql connections to keep open | 10 | No |
+| MYSQL_MAX_OPEN_CONNS | Integer value of max mysql connections to open | 100 | No |
+| MYSQL_MAX_CONN_LIFETIME | Integer value of max mysql connection lifetime (in seconds) | 3600 | No |
 
-<br/>
 Here is an example of running Tavern locally with a MySQL backend:
-```
+
+```sh
 MYSQL_USER="admin" MYSQL_ADDR="127.0.0.1:3306" go run ./tavern
 ```
-<br/>
+
 When no value is set for `MYSQL_ADDR`, the default SQLite backend is used:
-```
+
+```sh
 MYSQL_USER="admin" go run ./tavern/
 2022/03/08 05:46:06 no value found for environment var 'MYSQL_ADDR', starting tavern with SQLite
 ```
 
-## OAuth
+### OAuth
+
 By default, user authentication is disabled for Tavern. This means that anyone can login and be granted a session. To restrict who is able to access your deployment, Tavern supports OAuth configuration (using [Google OAuth](https://developers.google.com/identity/protocols/oauth2)).
 
 To obtain a client_id and a client_secret for Google OAuth, please follow [their instructions](https://developers.google.com/identity/sign-in/web/sign-in#create_authorization_credentials) to create an application.
@@ -49,32 +58,32 @@ The following environment variables are required for OAuth Configuration:
 | OAUTH_CLIENT_SECRET | The [OAuth client_secret](https://www.oauth.com/oauth2-servers/client-registration/client-id-secret/) Tavern will use to authenticate to an identity provider (Google)
 | OAUTH_DOMAIN | The domain Tavern is being hosted at, that the identity provider (Google) should redirect users to after completing the consent flow |
 
-<br/>
 Here is an example of running Tavern locally with OAuth configured:
-```
+
+```sh
 OAUTH_CLIENT_ID=123 OAUTH_CLIENT_SECRET=456 OAUTH_DOMAIN=127.0.0.1 go run ./tavern
 2022/03/09 05:32:58 no value found for environment var 'MYSQL_ADDR', starting tavern with SQLite
 2022/03/09 05:32:58 listening on 0.0.0.0:80
 ```
 
-<br/>
 When no OAuth configuration is provided, authentication is disabled:
-```
+
+```sh
 go run ./tavern
 2022/03/09 05:24:43 no value found for environment var 'MYSQL_ADDR', starting tavern with SQLite
 2022/03/09 05:24:43 WARNING: OAuth is not configured, authentication disabled
 ```
 
-<br/>
 When partial OAuth configuration is provided, Tavern will error. This is to protect against inadvertently starting Tavern with authentication disabled.
-```
+
+```sh
 OAUTH_CLIENT_ID=123 go run ./tavern
 2022/03/09 05:31:46 no value found for environment var 'MYSQL_ADDR', starting tavern with SQLite
 2022/03/09 05:31:46 [FATAL] To configure OAuth, must provide value for environment var 'OAUTH_CLIENT_SECRET'
 exit status 1
 ```
 
-## Test Data
+### Test Data
 
 Running Tavern with the `ENABLE_TEST_DATA` environment variable set will populate the database with test data. This is useful for UI development, testing, or just interacting with Tavern and seeing how it works.
 
@@ -89,53 +98,60 @@ ENABLE_TEST_DATA=1 go run ./tavern
 #### How it Works
 
 Tavern hosts two endpoints to support OAuth:
+
 * A login handler (`/oauth/login`) which redirects users to Google's OAuth consent flow
-    * This endpoint sets a JWT cookie for the user, such that the [OAuth state parameter](https://auth0.com/docs/secure/attack-protection/state-parameters#csrf-attacks) can safely be verified later to prevent against CSRF attacks
-    * Currently the keys used to sign and verify JWTs are generated at server start, meaning if the server is restarted while a user is in the middle of an OAuth flow, it will fail and the user will need to restart the flow
+  * This endpoint sets a JWT cookie for the user, such that the [OAuth state parameter](https://auth0.com/docs/secure/attack-protection/state-parameters#csrf-attacks) can safely be verified later to prevent against CSRF attacks
+  * Currently the keys used to sign and verify JWTs are generated at server start, meaning if the server is restarted while a user is in the middle of an OAuth flow, it will fail and the user will need to restart the flow
 * An authorization handler (`/oauth/authorize`) which users are redirected to by Google after completing Google's OAuth consent flow
-    * This handler is responsible for obtaining a user's profile information from Google using an OAuth access token, and creates the user's account if it does not exist yet
+  * This handler is responsible for obtaining a user's profile information from Google using an OAuth access token, and creates the user's account if it does not exist yet
 
 ##### Trust on First Use
+
 Tavern supports a Trust on First Use (TOFU) authentication model, meaning the first user to successfully authenticate will be granted admin permissions. Subsequent users that login will have accounts created, but will require activation before they can interact with any Tavern APIs. Only admin users may activate other users.
 
-# User Interface
+## User Interface
 
-# CDN API
+## CDN API
+
+### Uploading Files
 
-## Uploading Files
 * File uploads require 2 form parameters: `fileName` and `fileContent`
 * A successful response returns JSON with the following content: `{"data":{"file":{"id":<FILE_ID>}}}`
 
-## Downloading Files
+### Downloading Files
+
 * TODO (CDN is not yet added to Tavern)
 
-# GraphQL API
+## GraphQL API
+
+### Playground
 
-## Playground
 If you'd like to explore the Graph API and try out some queries, head to the `/graphiql` endpoint of your Tavern deployment. This endpoint exposes an interactive playground for you to experiment with GraphQL queries.
 
 ![/assets/img/tavern/graphiql.png](/assets/img/tavern/graphiql.png)
 
+### Creating a New Model
 
-## Creating a New Model
 1. Initialize the schema `cd tavern && go run entgo.io/ent/cmd/ent init <NAME>`
 2. Update the generated file in `tavern/ent/schema/<NAME>.go`
 3. Ensure you include a `func (<NAME>) Annotations() []schema.Annotation` method which returns a `entgql.QueryField()` annotation to tell entgo to generate a GraphQL root query for this model (if you'd like it to be queryable from the root query)
 4. Update `tavern/graphql/gqlgen.yml` to include the ent types in the `autobind:` section (e.g.`- github.com/kcarretto/realm/tavern/ent/<NAME>`)
 5. **Optionally** update the `models:` section of `tavern/graphql/gqlgen.yml` to bind any GraphQL enum types to their respective `entgo` generated types (e.g. `github.com/kcarretto/realm/tavern/ent/<NAME>.<ENUM_FIELD>`)
 6. Run `go generate ./tavern/...` from the project root
 7. If you added an annotation for a root query field (see above), you will notice auto-generated the `query.resolvers.go` file has been updated with new methods to query your model (e.g. `func (r *queryResolver) <NAME>s ...`)
-    * This must be implemented (e.g. `return r.client.<NAME>.Query().All(ctx)` where <NAME> is the name of your model)
+    * This must be implemented (e.g. `return r.client.<NAME>.Query().All(ctx)` where NAME is the name of your model)
+
+### Adding Mutations
 
-## Adding Mutations
 1. Update the `mutation.graphql` schema file to include your new mutation and please include it in the section for the model it's mutating if applicable (e.g. createUser should be defined near all the related User mutations)
     * **Note:** Input types such as `Create<NAME>Input` or `Update<NAME>Input` will already be generated if you [added the approproate annotations to your ent schema](https://entgo.io/docs/tutorial-todo-gql#install-and-configure-entgql). If you require custom input mutations (e.g. `ClaimTasksInput`) then add them to the `inputs.graphql` file (Golang code will be generated in tavern/graphql/models e.g. `models.ClaimTasksInput`).
 2. Run `go generate ./...`
 3. Implement generated the generated mutation resolver method in `tavern/graphql/mutation.resolvers.go`
     * Depending on the mutation you're trying to implement, a one liner such as `return r.client.<NAME>.Create().SetInput(input).Save(ctx)` might be sufficient
-5. Please write a unit test for your new mutation in `<NAME>_test.go` <3
+4. Please write a unit test for your new mutation in `<NAME>_test.go` <3
+
+### Code Generation Reference
 
-## Code Generation Reference
 * After making a change, remember to run `go generate ./...` from the project root.
 * `tavern/ent/schema` is a directory which defines our graph using database models (ents) and the relations between them
 * `tavern/generate.go` is responsible for generating ents defined by the ent schema as well as updating the GraphQL schema and generating related code
@@ -145,16 +161,15 @@ If you'd like to explore the Graph API and try out some queries, head to the `/g
 * `tavern/graphql/schema/scalars.graphql` defines scalar GraphQL types that can be used to help with Go bindings (See [gqlgen docs](https://gqlgen.com/reference/scalars/) for more info)
 * `tavern/graphql/schema/inputs.graphql` defines custom GraphQL inputs that can be used with your mutations (e.g. outside of the default auto-generated CRUD inputs)
 
-## Resources
+### Resources
+
 * [Relay Documentation](https://relay.dev/graphql/connections.htm)
 * [entgo.io GraphQL Integration Docs](https://entgo.io/docs/graphql)
 * [Ent + GraphQL Tutorial](https://entgo.io/docs/tutorial-todo-gql)
 * [Example Ent + GraphQL project](https://github.com/ent/contrib/tree/master/entgql/internal/todo)
 * [GQLGen Repo](https://github.com/99designs/gqlgen)
 
-# Agent Development
-
-## Overview
+## Agent Development
 
 Tavern provides an HTTP(s) GraphQL API that agents may use directly to claim tasks and submit execution results. This is the standard request flow, and is supported as a core function of realm. To learn more about how to interface with GraphQL APIs, please read [this documentation](https://www.graphql.com/tutorials/#clients) or read on for a simple example.
 
@@ -164,7 +179,7 @@ This however restricts the available transport methods the agent may use to comm
 
 ![/assets/img/tavern/custom-usage-arch.png](/assets/img/tavern/custom-usage-arch.png)
 
-## GraphQL Example
+### GraphQL Example
 
 GraphQL mutations enable clients to _mutate_ or modify backend data. Tavern supports a variety of different mutations for interacting with the graph ([see schema](https://github.com/KCarretto/realm/blob/main/tavern/graphql/schema/mutation.graphql)). The two mutations agents rely on are `claimTasks` and `submitTaskResult` (covered in more detail below). GraphQL requests are submitted as HTTP POST requests to Tavern, with a JSON body including the GraphQL mutation. Below is an example JSON body that may be sent to the Tavern GraphQL API:
 
@@ -186,7 +201,7 @@ GraphQL mutations enable clients to _mutate_ or modify backend data. Tavern supp
 
 In the above example, `$input` is used to pass variables from code to the GraphQL mutation while avoiding sketchy string parsing. Fields that should be present in the output are included in the body of the query (e.g. 'id').
 
-## Claiming Tasks
+### Claiming Tasks
 
 The first GraphQL mutation an agent should utilize is `claimTasks`. This mutation is used to fetch new tasks from Tavern that should be executed by the agent. In order to fetch execution information, the agent should perform a graph traversal to obtain information about the associated job. For example:
 

go.mod

@@ -3,14 +3,14 @@ module github.com/kcarretto/realm
 go 1.20
 
 require (
-	entgo.io/contrib v0.3.4
-	entgo.io/ent v0.11.5-0.20221031135557-521f9b57bc3d
-	github.com/99designs/gqlgen v0.17.20
-	github.com/go-sql-driver/mysql v1.6.0
+	entgo.io/contrib v0.3.5
+	entgo.io/ent v0.11.9
+	github.com/99designs/gqlgen v0.17.26
+	github.com/go-sql-driver/mysql v1.7.0
 	github.com/golang-jwt/jwt v3.2.2+incompatible
 	github.com/hashicorp/go-multierror v1.1.1
 	github.com/mattn/go-sqlite3 v1.14.16
-	github.com/stretchr/testify v1.8.0
+	github.com/stretchr/testify v1.8.2
 	github.com/urfave/cli v1.22.5
 	github.com/vektah/gqlparser/v2 v2.5.1
 	github.com/vmihailenco/msgpack/v5 v5.3.5
@@ -21,35 +21,32 @@ require (
 )
 
 require (
-	ariga.io/atlas v0.8.2 // indirect
+	ariga.io/atlas v0.9.1 // indirect
 	cloud.google.com/go/compute/metadata v0.2.0 // indirect
 	github.com/agext/levenshtein v1.2.3 // indirect
 	github.com/agnivade/levenshtein v1.1.1 // indirect
 	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
-	github.com/cpuguy83/go-md2man/v2 v2.0.1 // indirect
+	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/go-openapi/inflect v0.19.0 // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/google/go-cmp v0.5.9 // indirect
 	github.com/google/uuid v1.3.0 // indirect
 	github.com/gorilla/websocket v1.5.0 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
-	github.com/hashicorp/golang-lru v0.5.4 // indirect
-	github.com/hashicorp/hcl/v2 v2.15.0 // indirect
-	github.com/logrusorgru/aurora/v3 v3.0.0 // indirect
-	github.com/mattn/go-colorable v0.1.13 // indirect
-	github.com/mattn/go-isatty v0.0.16 // indirect
+	github.com/hashicorp/golang-lru/v2 v2.0.1 // indirect
+	github.com/hashicorp/hcl/v2 v2.16.1 // indirect
 	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
-	github.com/zclconf/go-cty v1.12.1 // indirect
-	golang.org/x/mod v0.7.0 // indirect
-	golang.org/x/net v0.6.0 // indirect
-	golang.org/x/sys v0.5.0 // indirect
-	golang.org/x/text v0.7.0 // indirect
-	golang.org/x/tools v0.3.0 // indirect
+	github.com/zclconf/go-cty v1.13.0 // indirect
+	golang.org/x/mod v0.9.0 // indirect
+	golang.org/x/net v0.8.0 // indirect
+	golang.org/x/sys v0.6.0 // indirect
+	golang.org/x/text v0.8.0 // indirect
+	golang.org/x/tools v0.7.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/protobuf v1.28.1 // indirect
 )