Add Dgraph investigation

.github/workflows/pr-build.yaml

@@ -406,7 +406,7 @@ jobs:
     permissions:
       contents: read
     env:
-      CAPACT_BINARY: "${{ github.workspace}}/capact_linux_amd64/capact"
+      CAPACT_BINARY: "${{ github.workspace}}/capact"
 
     steps:
       - name: Checkout code

docs/investigation/local-hub-v2/README.md

@@ -0,0 +1,9 @@
+# Local Hub
+
+The following directory contains different investigations of using different backend for Local Hub implementation.
+We're looking for a good replacement to Neo4j, to resolve problems with high resource consumption and licensing.
+Ideally, we want to implement Local Hub in Go.
+
+There are the following proof of concept projects:
+- [PostgreSQL](./postresql/README.md) - The goal of this investigation is to find an efficient way to implement Local Hub backed with PostgreSQL.
+- [Dgraph](./dgraph/README.md) - The goal of this investigation is to check whether Dgraph can be used as a Local Hub replacement.

docs/investigation/local-hub-v2/dgraph/README.md

@@ -0,0 +1,114 @@
+# Dgraph as the Local Hub
+
+This is a second approach to Dgraph project after [the first investigation](../../graph-db/dgraph/README.md) done about a year ago.
+
+## Motivation
+
+We're looking for a good replacement to Neo4j, to resolve problems with high resource consumption and licensing. Ideally, we want to implement Local Hub in Go.
+
+### Goal
+
+The main goal is to check whether Dgraph can be used as a replacement for current Neo4j implementation to speed up Local Hub extensibility and reduce resource consumption.
+
+Phases:
+1. Check what was changed after a year.
+2. Try to port existing Local Hub in minimal scope:
+	- Query TypeInstances,
+	- and TypInstances mutation with relation.
+3. Use `@custom` or `@lambda` functionality to resolve delegated storage concern.
+	- Instead of official [JavaScript](https://github.com/dgraph-io/dgraph-lambda) implementation, use [dgraph-lambda-go](https://github.com/Schartey/dgraph-lambda-go) server created by community.
+4. If porting was possible, do the benchmarking between current and new solution.
+
+### Non-goal
+
+Even thought that, Public and Local Hubs share the same Neo4j instance, checking if Public Hub can be ported it's **not a goal** of this investigation.
+Additionally, we don't want to use the Dgraph only as of the database and create our own GraphQL server with dedicated resolvers' implementation for each entity.
+
+## Prerequisite
+
+- Install Go
+- Install Docker
+- Install [Insomnia](https://insomnia.rest/download)
+
+## What has changed?
+
+In general, they mostly do a [bug fix releases](https://github.com/dgraph-io/dgraph/blob/12c3ef564cde11ecc3de96ec1516b3148e52d795/CHANGELOG.md).
+
+## Overview
+
+Dgraph doesn't allow adding custom logic directly in Dgraph server. To change the Dgraph behavior, you can:
+- use `@custom` directive to redirect resolution of a given queries, mutations and fields. Resolution is done by calling the defined HTTP server.
+- use `@lambda` directive to redirect resolution to lambda server. Dgraph provides lambda server only for JavaScript. In general, this is the same concept of executing HTTP calls against a registered Lambda server.
+
+Dgraph is written in Go, but it cannot be directly extended in Go. Custom resolvers are always an HTTP call to your service.
+
+It's due to the fact that Dgraph's GraphQL calls are not translated but directly executed on Dgraph server. In the official docs, you read:
+> Dgraph is the world’s first, and at this time only, service to offer a specification-compliant GraphQL endpoint without the need for an additional translation layer in the tech stack. You will not find a hidden layer of GraphQL resolvers in Dgraph. GraphQL is natively executed within the core of Dgraph itself.
+
+In comparison to current Neo4j implementation. We can do exactly the same, but we need to write delegation logic in JavaScript, as a result we can use any protocol we want to. Dgraph just gives a syntax sugar for extensions, but at the same time restrict it to HTTP only.
+
+## Give it a try
+
+1. Ensure that Docker is running on your localhost.
+2. Run the Dgraph and lambda server and load GraphQL schema:
+   ```bash
+   docker compose up --build
+   ```
+   >**NOTE:** If the data was not populated successfully, run: `docker compose run populate`.
+3. Use [Insomnia](https://insomnia.rest/download) to run all queries and mutations from [scenario.graphql](./assets/scenario.graphql) one by one.
+   >**NOTE:** GraphQL address: http://localhost:8080
+
+### Findings
+
+After running the above example, you can notice that:
+
+- The `@custom` or `@lambda` directives are only allowed on fields where the type definition has a field with type `ID!` or a field with `@id` directive.
+- A type must have at least one field that is not of `ID!` type and doesn't have `@custom` or `@lambda` directive.
+- In `@custom` and `@lambda` you have access only to primitive fields available on the root. In our case, we are not able to get the **backend** property when resolving `version`.
+- If a field is marked with `@custom` or `@lambda` directive, it's not included in mutation.
+- To prepare the demo scenario I had to check the Dgraph code directly as snippets described in Dgraph documentation are sometimes invalid.
+
+At the beginning, I thought that lambdas on fields will be able to speed up our development. Due to, found issues, I don't think they will.
+
+### Migration problems
+
+I tried to migrate the current [Local Hub schema](../../../../hub-js/graphql/local/schema.graphql) to Dgraph and I spotted such issues:
+
+- No option to add webhooks to protect the mutation. The current webhooks are just asynchronously triggered with a given event.
+
+  **Why required?** This was needed to protect TypeInstance deletion when it's locked.
+
+- No option to hide or rename generated query/mutation.
+
+  **Why required?** Because of the above point, we will need to write a custom logic to protect the TypeInstance deletion when it's locked. To do so, I wanted to reuse automatically generated DQL instead of writing our own logic.
+
+- [Custom DQL is still not supported on GraphQL mutation](https://discuss.dgraph.io/t/why-does-custom-dql-not-support-mutation/10366/12)
+
+  **Why required?** We are not able to easily customize mutations as we can with Cypher in Neo4j.
+
+- [We are not able to customize generated names for queries/mutations/filters.](https://discuss.dgraph.io/t/custom-names-for-crud-operations/13780/10)
+
+  **Why required?** Without that, we are not able to keep backward compatibility. They suggest to use lambda for aliasing, but then GraphQL server will expose both variants. If we disable the default generation, then we need to write the DGL query by our own.
+
+- [No option to define custom scalars.](https://discuss.dgraph.io/t/graphql-custom-scalar-types/16742)
+
+  **Why required?** Without that, we won't have a readable API.
+
+- [JSONSchema needs to be exposed as String.](https://discuss.dgraph.io/t/json-blob-as-a-scalar/11034/10)
+
+  **Why required?** Without that, we are not able to keep backward compatibility.
+
+- [No option to skip a given field on create](https://discuss.dgraph.io/t/graphql-error-non-nullable-field-was-not-present-in-result-from-dgraph/9503/6).
+
+  **Why required?** We want to ignore the **lockedBy** property which is automatically generated for TypeInstance mutation. This field should be managed by other mutation but still visible in TypeInstance type for all queries. We could use `@lambda` resolver to overcome this issue. But this is more a hack than a proper solution.
+
+
+Conclusion:
+> Hmm… 😬 I started full-on with Dgraph’s GraphQL but have slowed down tremendously due to numerous feature requests/ bug fixes required for true production readiness. (...)
+
+_source: https://discuss.dgraph.io/t/graphql-error-non-nullable-field-was-not-present-in-result-from-dgraph/9503/11_
+
+## Summary
+
+After a year, Dgraph didn't change too much. I failed to easily port current logic into Dgraph because of its limitations. In Dgraph we will need to implement almost all by our self. As a result, this won't speed up our current development.
+Porting to Dgraph only makes sens when we want to reduce the resources and license issues. We need to be aware that there is no Cypher alternative in Dgraph.

docs/investigation/local-hub-v2/dgraph/assets/populate.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Retry a command
+n=0
+until [ "$n" -ge 5 ]
+do
+   curl -X POST http://dgraph:8080/admin/schema --data-binary '@schema.graphql' && break  # substitute your command here
+   n=$((n+1))
+   sleep 5
+done

docs/investigation/local-hub-v2/dgraph/assets/scenario.graphql

@@ -0,0 +1,116 @@
+# Create Backend
+mutation BackendRef {
+  addTypeInstanceBackendReference(input: { abstract: true }) {
+    typeInstanceBackendReference {
+      id
+      abstract
+    }
+  }
+}
+
+# Create first TypeInstance
+mutation TypeInstance1 {
+  addTypeInstance(
+    input: {
+      typeRef: { path: "cap.type.db.psql.config", revision: "0.1.0" }
+      spec: { backend: { id: "0x2" } backendID: "123"}
+    }
+  ) {
+    typeInstance {
+      id
+      uses {
+        id
+        typeRef {
+          path
+        }
+      }
+      usedBy {
+        id
+        typeRef {
+          path
+        }
+      }
+    }
+  }
+}
+
+# Create second TypeInstance
+# TODO: update in `uses` property to match the TypeInstance1
+mutation TypeInstance2 {
+  addTypeInstance(
+    input: {
+      typeRef: { path: "cap.type.db.psql.config", revision: "0.1.0" }
+      spec: { backend: { id: "0x2" }}
+      uses: [{ id: "0x5" }]
+    }
+  ) {
+    typeInstance {
+      id
+      uses {
+        id
+        typeRef {
+          path
+        }
+      }
+      usedBy {
+        id
+        typeRef {
+          path
+        }
+      }
+    }
+  }
+}
+
+query GetAllTypeInstances {
+  queryTypeInstance {
+    ...TypeInstanceFields
+    uses {
+      ...TypeInstanceFields
+    }
+    usedBy {
+      ...TypeInstanceFields
+    }
+  }
+}
+
+query FilterTypeInstance {
+  queryTypeInstance(filter: { id: "0x7" }) {
+    ...TypeInstanceFields
+    uses {
+      ...TypeInstanceFields
+    }
+    usedBy {
+      ...TypeInstanceFields
+    }
+  }
+}
+
+query QuerySingleTypeInstance {
+  getTypeInstance(id: "0x7" ) {
+    ...TypeInstanceFields
+    uses {
+      ...TypeInstanceFields
+    }
+    usedBy {
+      ...TypeInstanceFields
+    }
+  }
+}
+
+
+fragment TypeInstanceFields on TypeInstance {
+  id
+  lockedBy
+  typeRef {
+    path
+    revision
+  }
+  spec {
+    backend {
+      id
+      abstract
+    }
+    value
+  }
+}

docs/investigation/local-hub-v2/dgraph/assets/schema.graphql

@@ -0,0 +1,37 @@
+
+type TypeInstance {
+  id: ID!
+
+  # `lockedBy` field cannot be ignored only on mutation.
+  # Maybe they will support that in the future: https://discuss.dgraph.io/t/graphql-error-non-nullable-field-was-not-present-in-result-from-dgraph/9503/6
+  lockedBy: String
+
+  """
+  Common properties for all TypeInstances which cannot be changed
+  """
+  typeRef: TypeInstanceTypeReference!
+  uses: [TypeInstance!]
+  usedBy: [TypeInstance!] @hasInverse(field: uses)
+
+  spec: TypeInstanceResourceVersionSpec
+}
+
+
+type TypeInstanceResourceVersionSpec {
+  id: ID!
+  # Moved from TypeInstance to be visible in value's @lambda resolver.
+  backend: TypeInstanceBackendReference!
+  backendID: String!
+  value: String! @lambda
+}
+
+type TypeInstanceTypeReference {
+  path: String!
+  revision: String!
+}
+
+type TypeInstanceBackendReference {
+  id: ID!
+  abstract: Boolean!
+}
+

docs/investigation/local-hub-v2/dgraph/docker-compose.yml

@@ -0,0 +1,34 @@
+services:
+  dgraph:
+    image: dgraph/standalone:latest
+    environment:
+      DGRAPH_ALPHA_LAMBDA: "url=http://lambda:8686/graphql-worker"
+    ports:
+      - "8080:8080"
+      - "9080:9080"
+      - "8000:8000"
+    depends_on:
+      - lambda
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8080"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+  lambda:
+    build: ./lambda
+    ports:
+      - "8686:8686"
+    environment:
+      DQL_URL: dgraph:9080
+      DGRAPHQL_URL: http://dgraph:8080
+      DGRAPH_URL: http://dgraph:8080
+  populate:
+    image: alpine/curl:latest
+    entrypoint: sh
+    command: ./populate.sh
+    working_dir: /mnt
+    restart: on-failure
+    depends_on:
+      - dgraph
+    volumes:
+      - ./assets:/mnt

docs/investigation/local-hub-v2/dgraph/lambda/.dockerignore

@@ -0,0 +1,2 @@
+docker-compose.yml
+lambda.yaml

docs/investigation/local-hub-v2/dgraph/lambda/Dockerfile

@@ -0,0 +1,14 @@
+# syntax=docker/dockerfile:1
+
+FROM golang:1.17-alpine
+
+WORKDIR /app
+
+COPY . ./
+RUN go mod download
+
+RUN go build -o /lambda
+
+EXPOSE 8686
+
+CMD [ "/lambda" ]

docs/investigation/local-hub-v2/dgraph/lambda/README.md

@@ -0,0 +1,22 @@
+# Dgraph Go Lambda Server
+
+It is a Go implementation of Dgraph [lambda server](https://dgraph.io/docs/master/graphql/lambda/server/) with a custom logic for demo purposes.
+
+This project was generated using [dgraph-lambda-go](https://github.com/Schartey/dgraph-lambda-go).
+
+## Prerequisites
+
+- [Go](https://golang.org)
+- [Docker](https://docs.docker.com/get-docker/)
+
+## Usage
+
+To run the lambda server use the following command:
+```bash
+go run main.go
+```
+
+To start the whole Dgraph ecosystem, run:
+```bash
+docker compose up --build
+```

docs/investigation/local-hub-v2/dgraph/lambda/docker-compose.yml

@@ -0,0 +1,20 @@
+services:
+  dgraph:
+    image: dgraph/standalone:latest
+    environment:
+      DGRAPH_ALPHA_LAMBDA: "url=http://lambda:8686/graphql-worker"
+    ports:
+      - "8080:8080"
+      - "9080:9080"
+      - "8000:8000"
+    depends_on:
+      - lambda
+  lambda:
+    build: .
+
+    ports:
+      - "8686:8686"
+    environment:
+      DQL_URL: dgraph:9080
+      DGRAPHQL_URL: http://dgraph:8080
+      DGRAPH_URL: http://dgraph:8080