From 0f852554cbcd8f3749430e61313857575f213d5b Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 9 Dec 2019 13:44:19 -0600 Subject: [PATCH 01/25] engineering-plans: Start subgraph composition engineering plan --- .../0001-subgraph-composition.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 engineering-plans/0001-subgraph-composition.md diff --git a/engineering-plans/0001-subgraph-composition.md b/engineering-plans/0001-subgraph-composition.md new file mode 100644 index 0000000..d9654bf --- /dev/null +++ b/engineering-plans/0001-subgraph-composition.md @@ -0,0 +1,77 @@ +# PLAN-0001: Subgraph Composition + +
+
Author
+
Jorge Olivero
+ +
Implements
+
RFC-0001 Subgraph Composition
+ +
Engineering Plan pull request
+
Design PR
+ +
Obsoletes (if applicable)
+ +
Date of submission
+
2019-12-09
+ +
Date of approval
+
TBD
+ +
Approved by
+
TBD
+
+ +## Summary + +A description of what the Engineering Plan does in 1-2 paragraphs. + +## Implementation + +How is the change or the feature going to be implemented? Which parts need to be +changed and how? + +## Tests + +How is the change or the feature going to be tested? Outline what test cases +will be implemented and, roughly, how (e.g. as unit tests, integration tests +etc.). Describe what the corner cases are and how they are accounted for in the +tests. + +## Migration + +If the change is breaking, even if in parts, are (semi-)automated migrations +possible? If not, how are we going to move users forward (e.g. announce early to +prepare users in time for the change to be activated)? If migrations are +possible, how are they going to work? + +## Documentation + +How are the changes going to be documented? + +## Implementation Plan + +An iterative plan for implementing the change or new feature. + +Think hard to not skip over potential challenges. Break the implementation down +into a step-by-step plan where the scope of every step is clear and where there +is no, or little, uncertainty about each individual step and the sequence +overall. + +**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 +days. + +**Phases:** For big changes, it can make sense to break down the overall plan +into implementation phases. + +**Integration:** Explicit integration points must be included in the plan. What +are we going to submit for review and integration at what point? What can be +integrated early? + +In order for a plan to be approved, there must be _extremely high_ confidence +that the plan will work out. + +## Open Questions + +What are unresolved questions? Which areas may we have forgotten about or +neglected in the plan? From 1fdcb492a5ad7c317b7c343e107d3ed1f3a11615 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:23:25 -0600 Subject: [PATCH 02/25] engineering-plans: subgraph-schema-merging --- .../0001-subgraph-composition.md | 77 ------------- .../0002-subgraph-schema-merging.md | 107 ++++++++++++++++++ 2 files changed, 107 insertions(+), 77 deletions(-) delete mode 100644 engineering-plans/0001-subgraph-composition.md create mode 100644 engineering-plans/0002-subgraph-schema-merging.md diff --git a/engineering-plans/0001-subgraph-composition.md b/engineering-plans/0001-subgraph-composition.md deleted file mode 100644 index d9654bf..0000000 --- a/engineering-plans/0001-subgraph-composition.md +++ /dev/null @@ -1,77 +0,0 @@ -# PLAN-0001: Subgraph Composition - -
-
Author
-
Jorge Olivero
- -
Implements
-
RFC-0001 Subgraph Composition
- -
Engineering Plan pull request
-
Design PR
- -
Obsoletes (if applicable)
- -
Date of submission
-
2019-12-09
- -
Date of approval
-
TBD
- -
Approved by
-
TBD
-
- -## Summary - -A description of what the Engineering Plan does in 1-2 paragraphs. - -## Implementation - -How is the change or the feature going to be implemented? Which parts need to be -changed and how? - -## Tests - -How is the change or the feature going to be tested? Outline what test cases -will be implemented and, roughly, how (e.g. as unit tests, integration tests -etc.). Describe what the corner cases are and how they are accounted for in the -tests. - -## Migration - -If the change is breaking, even if in parts, are (semi-)automated migrations -possible? If not, how are we going to move users forward (e.g. announce early to -prepare users in time for the change to be activated)? If migrations are -possible, how are they going to work? - -## Documentation - -How are the changes going to be documented? - -## Implementation Plan - -An iterative plan for implementing the change or new feature. - -Think hard to not skip over potential challenges. Break the implementation down -into a step-by-step plan where the scope of every step is clear and where there -is no, or little, uncertainty about each individual step and the sequence -overall. - -**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 -days. - -**Phases:** For big changes, it can make sense to break down the overall plan -into implementation phases. - -**Integration:** Explicit integration points must be included in the plan. What -are we going to submit for review and integration at what point? What can be -integrated early? - -In order for a plan to be approved, there must be _extremely high_ confidence -that the plan will work out. - -## Open Questions - -What are unresolved questions? Which areas may we have forgotten about or -neglected in the plan? diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md new file mode 100644 index 0000000..43bb20c --- /dev/null +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -0,0 +1,107 @@ +# PLAN-0001: Subgraph Schema Merging for Subgraph Composition + +
+
Author
+
Jorge Olivero
+ +
Implements
+
RFC-0001 Subgraph Composition
+ +
Engineering Plan pull request
+
Design PR
+ +
Obsoletes (if applicable)
+ +
Date of submission
+
2019-12-09
+ +
Date of approval
+
TBD
+ +
Approved by
+
TBD
+
+ +## Summary + +Subgraph composition allows a subgraph to import types from a foreign subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. + +To generate an API schema for the subgraph, the local schema needs to be merged with each of the foreign schemas that it imports. The merging process needs to take the following into account: + +1. An imported schema not being available on the graph-node +2. An imported schema without a definition for the type which the local subgraph imports +3. A merged and cached api schema for a subgraph imported by name is updated +4. Ability to tell which subgraph/schema each type belongs to + +## Implementation + +There are two important parts to schema merging: + +1. Merging the schema + +2. Invalidating a cache entry when a subgraph's dependency, referenced by name, is updated and remerging that schema + +### Schema Merging + +Add a `merged_schema(Document, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported +types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. + +### Cache Invalidation + +In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. + +A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification it would scan the schemas in the cache for those which need to be remerged. + + +## Tests + +1. Schemas are merged correctly when all schemas and imported types are available + +2. Placeholder types are properly inserted into the merged schema when a schema is not available + +3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not + +4. The cache is invalidated when the store returns an updated version of a cache entry's dependency + +## Migration + +Subgraph composition is an additive feature which shouldn't require a special migration plan. + +## Documentation + +Documenation on https://thegraph.com/docs needs to outline: + +1. The reserved _Schema_ type and how to define imports on it +2. The semantics of importing by subgraph id vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema +3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query + +## Implementation Plan + +An iterative plan for implementing the change or new feature. + +Think hard to not skip over potential challenges. Break the implementation down +into a step-by-step plan where the scope of every step is clear and where there +is no, or little, uncertainty about each individual step and the sequence +overall. + +**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 +days. + +**Phases:** For big changes, it can make sense to break down the overall plan +into implementation phases. + +**Integration:** Explicit integration points must be included in the plan. What +are we going to submit for review and integration at what point? What can be +integrated early? + +In order for a plan to be approved, there must be _extremely high_ confidence +that the plan will work out. + +- Implment the `merged_schema` function (2d) +- Write tests for the `merged_schema` function (1d) +- Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) +- Add cache invalidation log to `Store::cached_schema` (2d) + +## Open Questions + +The details of how queries and subscriptions needs to be updated to leverage the types in a merged schema. From 9905197d020f67364b863f6d724b6c3a7a431f41 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:25:43 -0600 Subject: [PATCH 03/25] engineering-plans: Typo --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 43bb20c..cbc6e6e 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -100,7 +100,7 @@ that the plan will work out. - Implment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) -- Add cache invalidation log to `Store::cached_schema` (2d) +- Add cache invalidation logic to `Store::cached_schema` (2d) ## Open Questions From db843ae026d13a06c6db1af61d6b67962f82ccba Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:26:27 -0600 Subject: [PATCH 04/25] engineering-plans: Remove template content --- .../0002-subgraph-schema-merging.md | 20 ------------------- 1 file changed, 20 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index cbc6e6e..3217343 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -77,26 +77,6 @@ Documenation on https://thegraph.com/docs needs to outline: ## Implementation Plan -An iterative plan for implementing the change or new feature. - -Think hard to not skip over potential challenges. Break the implementation down -into a step-by-step plan where the scope of every step is clear and where there -is no, or little, uncertainty about each individual step and the sequence -overall. - -**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 -days. - -**Phases:** For big changes, it can make sense to break down the overall plan -into implementation phases. - -**Integration:** Explicit integration points must be included in the plan. What -are we going to submit for review and integration at what point? What can be -integrated early? - -In order for a plan to be approved, there must be _extremely high_ confidence -that the plan will work out. - - Implment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) From d841b1e17c53ff4cae79e4d4575a965c2ac97caa Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:27:33 -0600 Subject: [PATCH 05/25] engineering-plan: Update link for PR --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 3217343..5018ff0 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -8,7 +8,7 @@
RFC-0001 Subgraph Composition
Engineering Plan pull request
-
Design PR
+
Engineering Plan PR
Obsoletes (if applicable)
From 323c8ebba9305b29ee012510a64960f4f66e7fe7 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 17:23:53 -0600 Subject: [PATCH 06/25] engineering-plans: Add sammple GraphQL documents to describe merging --- .../0002-subgraph-schema-merging.md | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 5018ff0..64a2bb7 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,9 +43,40 @@ There are two important parts to schema merging: ### Schema Merging -Add a `merged_schema(Document, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "..." } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Schema after calling `merged_schema` + +```graphql +type A @entity @subgraphId("...") { + id: ID! + foo: B! +} + +type B @entity @subgraphId("...") { + id: ID! + bar: String +} +``` + +After the schema document is merged, the `api_schema` will be called. + ### Cache Invalidation In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. @@ -77,11 +108,11 @@ Documenation on https://thegraph.com/docs needs to outline: ## Implementation Plan -- Implment the `merged_schema` function (2d) +- Impelment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) - Add cache invalidation logic to `Store::cached_schema` (2d) ## Open Questions -The details of how queries and subscriptions needs to be updated to leverage the types in a merged schema. +The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. From 4856a89f2398555913a867d4ccae714dc0977243 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 17:42:51 -0600 Subject: [PATCH 07/25] engineering-plans: Fix sentence --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 64a2bb7..2500779 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -75,7 +75,7 @@ type B @entity @subgraphId("...") { } ``` -After the schema document is merged, the `api_schema` will be called. +After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation From 7e162111398a4b576282c8a902b1c3d44db3b7a2 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 09:46:23 -0600 Subject: [PATCH 08/25] engineering-plans: Shorten name and update plan # --- .../0002-subgraph-schema-merging.md | 50 +++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 2500779..147a15f 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -1,4 +1,4 @@ -# PLAN-0001: Subgraph Schema Merging for Subgraph Composition +# PLAN-0002: Subgraph Schema Merging
Author
@@ -46,13 +46,55 @@ There are two important parts to schema merging: Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +#### Example #1: Valid import and complete merge + +Local schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "X" } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema`: + +```graphql +type A @entity { + id: ID! + foo: B! +} + +type B @entity @subgraphId("X") { + id: ID! + bar: String +} +``` + +#### Example #2 + Schema before calling `merged_schema`: ```graphql type _Schema_ @import( types: ["B"], - from: { id: "..." } + from: { id: "X" } ) type A @entity { @@ -79,9 +121,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. +In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated and remerged before responding to the cache access. -A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification it would scan the schemas in the cache for those which need to be remerged. +A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. ## Tests From 8636b8e6e877c5bdaec1388c1a5871ab5d3e3094 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 09:48:12 -0600 Subject: [PATCH 09/25] engineering-plans: Update markup --- engineering-plans/0002-subgraph-schema-merging.md | 1 + 1 file changed, 1 insertion(+) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 147a15f..df6e75c 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -11,6 +11,7 @@
Engineering Plan PR
Obsoletes (if applicable)
+
-
Date of submission
2019-12-09
From a2d9b470f3dbdd3229d8c77aa7b2c86784357517 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 12:53:06 -0600 Subject: [PATCH 10/25] engineering-plans: Address feedback on pull #6 --- .../0002-subgraph-schema-merging.md | 121 ++++++++++++------ 1 file changed, 80 insertions(+), 41 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index df6e75c..fbe6911 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -25,22 +25,21 @@ ## Summary -Subgraph composition allows a subgraph to import types from a foreign subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. +Subgraph composition allows a subgraph to import types from another subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. -To generate an API schema for the subgraph, the local schema needs to be merged with each of the foreign schemas that it imports. The merging process needs to take the following into account: +To generate an API schema for the subgraph, the subgraph schema needs to be merged with the imported subgraph schemas. The merging process needs to take the following into account: -1. An imported schema not being available on the graph-node -2. An imported schema without a definition for the type which the local subgraph imports -3. A merged and cached api schema for a subgraph imported by name is updated +1. An imported schema not being available on the Graph Node +2. An imported schema mising a type imported by the subgraph schema +3. A schema imported by subgraph name changes 4. Ability to tell which subgraph/schema each type belongs to ## Implementation -There are two important parts to schema merging: +The schema merging implementation consists of two parts: -1. Merging the schema - -2. Invalidating a cache entry when a subgraph's dependency, referenced by name, is updated and remerging that schema +1. A cache of subgraph schemas +2. The schema merging logic ### Schema Merging @@ -53,14 +52,14 @@ Local schema before calling `merged_schema`: ```graphql type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) + @import( + types: ["B"], + from: { id: "X" } + ) type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } ``` @@ -68,8 +67,8 @@ Imported Schema X: ```graphql type B @entity { - id: ID! - bar: String + id: ID! + bar: String } ``` @@ -77,13 +76,13 @@ Schema after calling `merged_schema`: ```graphql type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } type B @entity @subgraphId("X") { - id: ID! - bar: String + id: ID! + bar: String } ``` @@ -93,14 +92,14 @@ Schema before calling `merged_schema`: ```graphql type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) + @import( + types: ["B"], + from: { id: "X" } + ) type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } ``` @@ -108,13 +107,53 @@ Schema after calling `merged_schema` ```graphql type A @entity @subgraphId("...") { - id: ID! - foo: B! + id: ID! + foo: B! } type B @entity @subgraphId("...") { - id: ID! - bar: String + id: ID! + bar: String +} +``` + +#### Example #3 + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId("X") @mergeInfo(originalName: "B") { + id: ID! + bar: String } ``` @@ -129,25 +168,25 @@ A more performant invalidation solution would be to have the cache maintain a li ## Tests -1. Schemas are merged correctly when all schemas and imported types are available +1. Schemas are merged correctly when all schemas and imported types are available. -2. Placeholder types are properly inserted into the merged schema when a schema is not available +2. Placeholder types are properly inserted into the merged schema when a schema is not available. -3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not +3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not. -4. The cache is invalidated when the store returns an updated version of a cache entry's dependency +4. The cache is invalidated when the store returns an updated version of a cache entry's dependency. ## Migration -Subgraph composition is an additive feature which shouldn't require a special migration plan. +Subgraph composition is an additive feature which doesn't require a special migration plan. ## Documentation -Documenation on https://thegraph.com/docs needs to outline: +Documentation on https://thegraph.com/docs needs to outline: -1. The reserved _Schema_ type and how to define imports on it -2. The semantics of importing by subgraph id vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema -3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query +1. The reserved _Schema_ type and how to define imports on it. +2. The semantics of importing by subgraph ID vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema. +3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query. ## Implementation Plan @@ -158,4 +197,4 @@ Documenation on https://thegraph.com/docs needs to outline: ## Open Questions -The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. +- The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. From 4139d9c89f4fc66a9e55fee75ba716b977a16d97 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 13:04:51 -0600 Subject: [PATCH 11/25] engineering-plans: @mergeInfo -> @originalName --- engineering-plans/0002-subgraph-schema-merging.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index fbe6911..7f0ac94 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,8 +43,9 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported -types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. +If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive. +The `api_schema` function will add all the necessary types and fields for the imported types without any changes. #### Example #1: Valid import and complete merge @@ -151,7 +152,7 @@ type B @entity { foo: BB! } -type BB @entity @subgraphId("X") @mergeInfo(originalName: "B") { +type BB @entity @subgraphId("X") @originalName("B") { id: ID! bar: String } From 20edb4b1aaf2134a8774843bee4f273aa27de8a0 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 13:41:39 -0600 Subject: [PATCH 12/25] engineering-plans: Add more detail to the merge algorithm --- .../0002-subgraph-schema-merging.md | 29 +++++++++++++------ 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 7f0ac94..053d967 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,11 +43,17 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. -If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive. -The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. -#### Example #1: Valid import and complete merge +The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. + +- If any schema node along that path is missing or if the type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId("...")` directive (if available), and a `@placeholder` directive denoting that type was not found. +- If the type is found, copy it and add a `@subgraphId("...")` directive. +- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive preserving the type name from the original schema. + +The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. + +#### Example #1: Complete merge Local schema before calling `merged_schema`: @@ -87,7 +93,7 @@ type B @entity @subgraphId("X") { } ``` -#### Example #2 +#### Example #2: Incomplete merge Schema before calling `merged_schema`: @@ -104,6 +110,12 @@ type A @entity { } ``` +Imported Schema X: + +```graphql +NOT AVAILABLE +``` + Schema after calling `merged_schema` ```graphql @@ -112,13 +124,12 @@ type A @entity @subgraphId("...") { foo: B! } -type B @entity @subgraphId("...") { +type B @entity @placeholder { id: ID! - bar: String } ``` -#### Example #3 +#### Example #3: Complete merge with `{ name: "...", as: "..." }` Schema before calling `merged_schema` @@ -191,7 +202,7 @@ Documentation on https://thegraph.com/docs needs to outline: ## Implementation Plan -- Impelment the `merged_schema` function (2d) +- Implement the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) - Add cache invalidation logic to `Store::cached_schema` (2d) From 84480da0e71dba94ba5d4eeae8c1ea307f51cad6 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 22:31:37 -0600 Subject: [PATCH 13/25] engineering-plans: Add detail to cache invalidation --- engineering-plans/0002-subgraph-schema-merging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 053d967..d2bd24a 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -173,7 +173,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated and remerged before responding to the cache access. +For each schema in the cache, keep a vector of volatile schemas containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recentschema merge. If there are any new versions, re merge the schema. + +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, and a `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`) A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 111c875e43fd6447f99203717ff80c5fd1f98ccf Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 22:35:25 -0600 Subject: [PATCH 14/25] engineering-plans: Add detail to schema merging cache invalidation --- engineering-plans/0002-subgraph-schema-merging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index d2bd24a..cdaea5a 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -173,9 +173,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -For each schema in the cache, keep a vector of volatile schemas containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recentschema merge. If there are any new versions, re merge the schema. +For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. -Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, and a `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`) +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 75ff610b649802aa9692d7b7343f727450f89cca Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 17 Dec 2019 12:21:32 -0600 Subject: [PATCH 15/25] engineering-plans: Add entry to approved.md --- engineering-plans/approved.md | 1 + 1 file changed, 1 insertion(+) diff --git a/engineering-plans/approved.md b/engineering-plans/approved.md index 8c1f549..5697314 100644 --- a/engineering-plans/approved.md +++ b/engineering-plans/approved.md @@ -3,3 +3,4 @@ - [PLAN-0001: GraphQL Query Prefetching](./0001-graphql-query-prefetching.md) - [PLAN-0002: Ethereum Tracing Cache](./0002-ethereum-tracing-cache.md) - [PLAN-0003: Remove JSONB Storage](./0003-remove-jsonb-storage.md) +- [PLAN-0004: Subgraph Schema Merging](./0002-subgraph-schema-merging.md) From b367c3f7c924d26f0e8d623ba609ed1822f37805 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 17 Dec 2019 12:52:29 -0600 Subject: [PATCH 16/25] SUMMARY: Add subgraph schema merging to list --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 3efaac7..9b8f840 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -12,5 +12,6 @@ - [PLAN-0001: GraphQL Query Prefetching](./engineering-plans/0001-graphql-query-prefetching.md) - [PLAN-0002: Ethereum Tracing Cache](./engineering-plans/0002-ethereum-tracing-cache.md) - [PLAN-0003: Remove JSONB Storage](./engineering-plans/0003-remove-jsonb-storage.md) + - [PLAN-0004: Subgraph Schema Merging](./engineering-plans/0004-subgraph-schema-merging.md) - [Obsolete Plans](./engineering-plans/obsolete.md) - [Rejected Plans](./engineering-plans/rejected.md) From 10acac4fd150b210f8ae067860669333bd3a2eca Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 20 Dec 2019 17:44:55 -0600 Subject: [PATCH 17/25] engineering-plans: Address importing types for non scalar fields --- .../0002-subgraph-schema-merging.md | 88 +++++++++++++++++-- 1 file changed, 79 insertions(+), 9 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index cdaea5a..480c29d 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,13 +43,13 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a `@subgraphId` directive denoting which subgraph the type came from. If any of the imported types have non scalar fields, import those types as well. The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. -- If any schema node along that path is missing or if the type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId("...")` directive (if available), and a `@placeholder` directive denoting that type was not found. -- If the type is found, copy it and add a `@subgraphId("...")` directive. -- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive preserving the type name from the original schema. +- If any schema node along that path is missing or if a type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId(id: "...")` directive (if available), and a `@placeholder` directive denoting that type was not found. +sc- If the type is found, copy it and add a `@subgraphId(id: "...")` directive. +- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName(name: "...")` directive preserving the type name from the original schema. The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. @@ -87,7 +87,7 @@ type A @entity { foo: B! } -type B @entity @subgraphId("X") { +type B @entity @subgraphId(id: "X") { id: ID! bar: String } @@ -119,7 +119,7 @@ NOT AVAILABLE Schema after calling `merged_schema` ```graphql -type A @entity @subgraphId("...") { +type A @entity @subgraphId(id: "...") { id: ID! foo: B! } @@ -163,19 +163,89 @@ type B @entity { foo: BB! } -type BB @entity @subgraphId("X") @originalName("B") { +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { id: ID! bar: String } ``` +#### Example #4: Complete merge with nested types + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type _Schema_ + @imports( + types: [{ name: "C", as: "CC" }] + from: { id: "Y" } + ) + +type B @entity { + id: ID! + bar: CC! + baz: DD! +} + +type DD @entity { + id: ID! +} +``` + +Imported Schema Y: + +```graphql +type C @entity { + id: ID! +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { + id: ID! + bar: CC! + baz: DD! +} + +type CC @entity @subgraphId(id: "Y") @originalName(name: "C") { + id: ID! +} + +type DD @entity @subgraphId(id: "X") { + id: ID! +} +``` + + + After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. +For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph ID which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. -Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported` (`Vec<(SchemaReference, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 1da8b0711ae4b30bcd773c82b114acd46530796b Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 14 Jan 2020 17:16:32 -0600 Subject: [PATCH 18/25] engineering-plans: Rename subgraph schema merging plan --- ...bgraph-schema-merging.md => 0004-subgraph-schema-merging.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename engineering-plans/{0002-subgraph-schema-merging.md => 0004-subgraph-schema-merging.md} (99%) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0004-subgraph-schema-merging.md similarity index 99% rename from engineering-plans/0002-subgraph-schema-merging.md rename to engineering-plans/0004-subgraph-schema-merging.md index 480c29d..72ac979 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0004-subgraph-schema-merging.md @@ -1,4 +1,4 @@ -# PLAN-0002: Subgraph Schema Merging +# PLAN-0004: Subgraph Schema Merging
Author
From 7bd8982af1f9df6a55e5fcd098cf0e02188c5694 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 17 Dec 2019 12:21:32 -0600 Subject: [PATCH 19/25] engineering-plans: Add entry to approved.md --- engineering-plans/approved.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/approved.md b/engineering-plans/approved.md index 5697314..0298de2 100644 --- a/engineering-plans/approved.md +++ b/engineering-plans/approved.md @@ -3,4 +3,4 @@ - [PLAN-0001: GraphQL Query Prefetching](./0001-graphql-query-prefetching.md) - [PLAN-0002: Ethereum Tracing Cache](./0002-ethereum-tracing-cache.md) - [PLAN-0003: Remove JSONB Storage](./0003-remove-jsonb-storage.md) -- [PLAN-0004: Subgraph Schema Merging](./0002-subgraph-schema-merging.md) +- [PLAN-0004: Subgraph Schema Merging](./0004-subgraph-schema-merging.md) From 77d2bc7f23062189c3fc2058ebd50a787dc5b126 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 14 Jan 2020 17:13:38 -0600 Subject: [PATCH 20/25] engineering-plans: Add plan for subgraph composition query execution --- ...05-subgraph-composition-query-execution.md | 276 ++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 engineering-plans/0005-subgraph-composition-query-execution.md diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md new file mode 100644 index 0000000..eee2b23 --- /dev/null +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -0,0 +1,276 @@ +# PLAN-0005: Subgraph Composition Query Execution + +
+
Author
+
Jorge Olivero
+ +
Implements
+
RFC-0001 Subgraph Composition
+ +
Engineering Plan pull request
+
Engineering Plan PR
+ +
Obsoletes (if applicable)
+
-
+ +
Date of submission
+
2020-1-14
+ +
Date of approval
+
TBD
+ +
Approved by
+
TBD
+
+ +## Summary + +Subgraph composition allows a subgraph to import types from another subgraph. This means that the GraphQL queries will need to traverse the subgraph's import graph to pull all of the necessary concrete types. + +## Implementation + +1. Ensure @subgraphId and @originalName directives on the imported types are used to generate the correct sql queries +2. Ensure @placeholder types are acknowledged during query execution since this means that a type can not be located in a subgraph +3. + +### Schema Merging + +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a `@subgraphId` directive denoting which subgraph the type came from. If any of the imported types have non scalar fields, import those types as well. + +The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. + +- If any schema node along that path is missing or if a type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId(id: "...")` directive (if available), and a `@placeholder` directive denoting that type was not found. +sc- If the type is found, copy it and add a `@subgraphId(id: "...")` directive. +- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName(name: "...")` directive preserving the type name from the original schema. + +The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. + +#### Example #1: Complete merge + +Local schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "X" } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema`: + +```graphql +type A @entity { + id: ID! + foo: B! +} + +type B @entity @subgraphId(id: "X") { + id: ID! + bar: String +} +``` + +#### Example #2: Incomplete merge + +Schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "X" } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Imported Schema X: + +```graphql +NOT AVAILABLE +``` + +Schema after calling `merged_schema` + +```graphql +type A @entity @subgraphId(id: "...") { + id: ID! + foo: B! +} + +type B @entity @placeholder { + id: ID! +} +``` + +#### Example #3: Complete merge with `{ name: "...", as: "..." }` + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { + id: ID! + bar: String +} +``` + +#### Example #4: Complete merge with nested types + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type _Schema_ + @imports( + types: [{ name: "C", as: "CC" }] + from: { id: "Y" } + ) + +type B @entity { + id: ID! + bar: CC! + baz: DD! +} + +type DD @entity { + id: ID! +} +``` + +Imported Schema Y: + +```graphql +type C @entity { + id: ID! +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { + id: ID! + bar: CC! + baz: DD! +} + +type CC @entity @subgraphId(id: "Y") @originalName(name: "C") { + id: ID! +} + +type DD @entity @subgraphId(id: "X") { + id: ID! +} +``` + + + +After the schema document is merged, the `api_schema` function will be called. + +### Cache Invalidation + +For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph ID which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. + +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported` (`Vec<(SchemaReference, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. + +A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. + + +## Tests + +1. Schemas are merged correctly when all schemas and imported types are available. + +2. Placeholder types are properly inserted into the merged schema when a schema is not available. + +3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not. + +4. The cache is invalidated when the store returns an updated version of a cache entry's dependency. + +## Migration + +Subgraph composition is an additive feature which doesn't require a special migration plan. + +## Documentation + +Documentation on https://thegraph.com/docs needs to outline: + +1. The reserved _Schema_ type and how to define imports on it. +2. The semantics of importing by subgraph ID vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema. +3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query. + +## Implementation Plan + +- Implement the `merged_schema` function (2d) +- Write tests for the `merged_schema` function (1d) +- Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) +- Add cache invalidation logic to `Store::cached_schema` (2d) + +## Open Questions + +- The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. From 2eb16c3ba186be9c580fecc203a1dfec4f6688a8 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Wed, 15 Jan 2020 10:28:52 -0600 Subject: [PATCH 21/25] engineering-plans: Subgraph composition query execution --- ...05-subgraph-composition-query-execution.md | 235 +++--------------- 1 file changed, 37 insertions(+), 198 deletions(-) diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md index eee2b23..ecd6c41 100644 --- a/engineering-plans/0005-subgraph-composition-query-execution.md +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -25,252 +25,91 @@ ## Summary -Subgraph composition allows a subgraph to import types from another subgraph. This means that the GraphQL queries will need to traverse the subgraph's import graph to pull all of the necessary concrete types. +Since subgraph composition allows a subgraph to import types from another subgraph, GraphQL query execution will need to traverse a subgraph's import graph to resolve concrete types. +Because imported schemas/types are loosely coupled, query execution also needs to address missing types in the api schema during resolution. +In addition, imported types which have been renamed need to be mapped back to their original name so that queries can resolve correctly. +Lastly, to simplify query execution graph-node needs to deny subgraphs with JSONB storage from using subgraph composition. ## Implementation -1. Ensure @subgraphId and @originalName directives on the imported types are used to generate the correct sql queries -2. Ensure @placeholder types are acknowledged during query execution since this means that a type can not be located in a subgraph -3. +1. Support Custom Type Names: Ensure @originalName directives on the imported types are used to query the correct sql tables. +2. Codify New Query Runtime Errors: Ensure @placeholder types are acknowledged and proper errors are returned during query execution when a type's subgraph is not deployed to the graph-node resolving the query. +3. Query Correct Subgraph: Ensure that @subgraphId directives on imported types are used correctly in all cases. +4. Add Subgraph Composition Feature Flag: Ensure subgraphs JSONB storage can not use subgraph composition. -### Schema Merging +### Support Custom Type Names -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a `@subgraphId` directive denoting which subgraph the type came from. If any of the imported types have non scalar fields, import those types as well. - -The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. - -- If any schema node along that path is missing or if a type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId(id: "...")` directive (if available), and a `@placeholder` directive denoting that type was not found. -sc- If the type is found, copy it and add a `@subgraphId(id: "...")` directive. -- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName(name: "...")` directive preserving the type name from the original schema. - -The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. - -#### Example #1: Complete merge - -Local schema before calling `merged_schema`: - -```graphql -type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) - -type A @entity { - id: ID! - foo: B! -} -``` - -Imported Schema X: - -```graphql -type B @entity { - id: ID! - bar: String -} -``` - -Schema after calling `merged_schema`: - -```graphql -type A @entity { - id: ID! - foo: B! -} - -type B @entity @subgraphId(id: "X") { - id: ID! - bar: String -} -``` - -#### Example #2: Incomplete merge - -Schema before calling `merged_schema`: - -```graphql -type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) - -type A @entity { - id: ID! - foo: B! -} -``` - -Imported Schema X: - -```graphql -NOT AVAILABLE -``` - -Schema after calling `merged_schema` - -```graphql -type A @entity @subgraphId(id: "...") { - id: ID! - foo: B! -} - -type B @entity @placeholder { - id: ID! -} -``` - -#### Example #3: Complete merge with `{ name: "...", as: "..." }` - -Schema before calling `merged_schema` +When a type is imported with the name/as syntax: ```graphql -type _Schema_ - @imports( - types: [{ name: "B", as: "BB" }] - from: { id: "X" } - ) - -type B @entity { - id: ID! - foo: BB! -} +type _Schema_ @import(types: [{ name: "A", as: "B" }], from: { id: "..." }) ``` -Imported Schema X: +an `@originalName` directive is added to it when it is merged into the target schema: ```graphql -type B @entity { - id: ID! - bar: String +type B @entity @subgraphId(id: "...") @originalName(name: "A") { + id: ID! + ... } ``` -Schema after calling `merged_schema` +GraphQL query resolution needs to make use of the `@originalName` directive to ensure SQL queries are made against the correct tables. -```graphql -type B @entity { - id: ID! - foo: BB! -} - -type BB @entity @subgraphId(id: "X") @originalName(name: "B") { - id: ID! - bar: String -} -``` +The follow code locations need to be updated: -#### Example #4: Complete merge with nested types +1. https://github.com/graphprotocol/graph-node/blob/master/graphql/src/store/resolver.rs#L362-L379f -Schema before calling `merged_schema` +2. https://github.com/graphprotocol/graph-node/blob/master/graphql/src/store/query.rs#L21-L27 -```graphql -type _Schema_ - @imports( - types: [{ name: "B", as: "BB" }] - from: { id: "X" } - ) - -type B @entity { - id: ID! - foo: BB! -} -``` +### Codify New Query Runtime Errors -Imported Schema X: +When a type is imported from a schema that is unavailable a placeholder is created to ensure the schema remains complete: ```graphql -type _Schema_ - @imports( - types: [{ name: "C", as: "CC" }] - from: { id: "Y" } - ) - -type B @entity { - id: ID! - bar: CC! - baz: DD! -} - -type DD @entity { - id: ID! -} +type _Schema_ @import(types: [{ name: "A", as: "B" }], from: { id: "..." }) ``` -Imported Schema Y: +when the subgraph with `id: "..."` is unavailable during the merge a placeholder will be created: ```graphql -type C @entity { +type B @entity @placeholder @originalName(name: "A") { id: ID! } ``` -Schema after calling `merged_schema` +Since the `@subgraphId` won't be available for this type during query execution, we have to handle this as query runtime error in the follow locations: -```graphql -type B @entity { - id: ID! - foo: BB! -} +1. https://github.com/graphprotocol/graph-node/blob/master/graphql/src/store/resolver.rs#L359 -type BB @entity @subgraphId(id: "X") @originalName(name: "B") { - id: ID! - bar: CC! - baz: DD! -} +### Query Correct Subgraph -type CC @entity @subgraphId(id: "Y") @originalName(name: "C") { - id: ID! -} - -type DD @entity @subgraphId(id: "X") { - id: ID! -} -``` +Imported types in a merged schema which are available on the Graph Node executing the query will have `@subgraphId` directives actings are pointers to the relevant database table. +In the prefetch portion of query resolution we run JOINs across tables for a subgraph's types to aggregate the children for a single parent or a set of parent objects. Instead of modifying sql for the JOIN +to ensure that a join across tables for two different subgraphs works, we can remove the JOIN and make the prefetch code only execute queries on a single table at a time. +Adapt the work described in this issue: https://github.com/graphprotocol/graph-node/issues/1450 -After the schema document is merged, the `api_schema` function will be called. - -### Cache Invalidation - -For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph ID which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. - -Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported` (`Vec<(SchemaReference, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. - -A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. +### Add Subgraph Composition Feature Flag +Do not allow subgraph composition for subgraphs which use the JSONB storage engine. ## Tests -1. Schemas are merged correctly when all schemas and imported types are available. - -2. Placeholder types are properly inserted into the merged schema when a schema is not available. - -3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not. - -4. The cache is invalidated when the store returns an updated version of a cache entry's dependency. +TODO ## Migration -Subgraph composition is an additive feature which doesn't require a special migration plan. +Subgraph composition is an additive feature and does require a migration plan. ## Documentation -Documentation on https://thegraph.com/docs needs to outline: - -1. The reserved _Schema_ type and how to define imports on it. -2. The semantics of importing by subgraph ID vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema. -3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query. ## Implementation Plan -- Implement the `merged_schema` function (2d) -- Write tests for the `merged_schema` function (1d) -- Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) -- Add cache invalidation logic to `Store::cached_schema` (2d) +TODO ## Open Questions -- The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. +TODO From 5a822c417993c9dedbcf5b1eee31be13ec4d0bbb Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Wed, 15 Jan 2020 12:09:05 -0600 Subject: [PATCH 22/25] engineering-plans: Address typo --- engineering-plans/0005-subgraph-composition-query-execution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md index ecd6c41..15829fe 100644 --- a/engineering-plans/0005-subgraph-composition-query-execution.md +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -101,7 +101,7 @@ TODO ## Migration -Subgraph composition is an additive feature and does require a migration plan. +Subgraph composition is an additive feature and does not require a migration plan. ## Documentation From 8b6aa110b38cbb8496fdf38e484431e129374388 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Thu, 16 Jan 2020 09:26:14 -0600 Subject: [PATCH 23/25] engineering-plans: Add more detail to query execution plan --- ...005-subgraph-composition-query-execution.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md index 15829fe..afc0351 100644 --- a/engineering-plans/0005-subgraph-composition-query-execution.md +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -97,7 +97,13 @@ Do not allow subgraph composition for subgraphs which use the JSONB storage engi ## Tests -TODO +### Unit Tests: + +- + +### Integration Tests: + +1. Querying an A and [A] with an imported field B and [B] with an imported field C ## Migration @@ -105,11 +111,15 @@ Subgraph composition is an additive feature and does not require a migration pla ## Documentation +Document the following aspects of query execution along side more general subgraph composition documentation: + +1. New runtime errors which could occur during query execution ## Implementation Plan -TODO +- Support custom type names (1d) +- Codify new query execution runtime errors (1d) +- Add subgraph composition feature flag (1d) +- Ensure correct subgraph is queried during single object resolution and prefetch paths (3d) ## Open Questions - -TODO From 8dbc3bb3b10c43950a4bd634d5c0296101d30672 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Thu, 16 Jan 2020 09:32:39 -0600 Subject: [PATCH 24/25] engineering-plans: touch up subgraph composition query execution --- .../0005-subgraph-composition-query-execution.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md index afc0351..7ad459c 100644 --- a/engineering-plans/0005-subgraph-composition-query-execution.md +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -84,7 +84,7 @@ Since the `@subgraphId` won't be available for this type during query execution, ### Query Correct Subgraph -Imported types in a merged schema which are available on the Graph Node executing the query will have `@subgraphId` directives actings are pointers to the relevant database table. +Imported types in a merged schema which are available on the Graph Node executing the query will have `@subgraphId` and `@originalName` directives actings are pointers to the relevant database table. In the prefetch portion of query resolution we run JOINs across tables for a subgraph's types to aggregate the children for a single parent or a set of parent objects. Instead of modifying sql for the JOIN to ensure that a join across tables for two different subgraphs works, we can remove the JOIN and make the prefetch code only execute queries on a single table at a time. @@ -99,11 +99,9 @@ Do not allow subgraph composition for subgraphs which use the JSONB storage engi ### Unit Tests: -- - ### Integration Tests: -1. Querying an A and [A] with an imported field B and [B] with an imported field C +1. Querying an A || [A] with an imported field B || [B] with an imported field C ## Migration From 5b0926ea245688bc5443037ee492273e874fff66 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 17 Feb 2020 15:19:12 -0600 Subject: [PATCH 25/25] subgraph-composition-query-execution: Small updates --- .../0005-subgraph-composition-query-execution.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/engineering-plans/0005-subgraph-composition-query-execution.md b/engineering-plans/0005-subgraph-composition-query-execution.md index 7ad459c..351949f 100644 --- a/engineering-plans/0005-subgraph-composition-query-execution.md +++ b/engineering-plans/0005-subgraph-composition-query-execution.md @@ -84,12 +84,8 @@ Since the `@subgraphId` won't be available for this type during query execution, ### Query Correct Subgraph -Imported types in a merged schema which are available on the Graph Node executing the query will have `@subgraphId` and `@originalName` directives actings are pointers to the relevant database table. -In the prefetch portion of query resolution we run JOINs across tables for a subgraph's types to aggregate the children for a single parent or a set of parent objects. Instead of modifying sql for the JOIN -to ensure that a join across tables for two different subgraphs works, we can remove the JOIN and make the prefetch code only execute queries on a single table at a time. - -Adapt the work described in this issue: https://github.com/graphprotocol/graph-node/issues/1450 - +Imported types in a merged schema which are available on the Graph Node executing the query will have `@subgraphId` and `@originalName` directives actings as pointers to the relevant database table. +Ensure that the changes introduced in https://github.com/graphprotocol/graph-node/pull/1483 work seamlessly with the merged schema. ### Add Subgraph Composition Feature Flag @@ -101,7 +97,7 @@ Do not allow subgraph composition for subgraphs which use the JSONB storage engi ### Integration Tests: -1. Querying an A || [A] with an imported field B || [B] with an imported field C +1. Querying an A || [A] with an imported field B || [B] with an imported field C || [C] ## Migration @@ -121,3 +117,5 @@ Document the following aspects of query execution along side more general subgra - Ensure correct subgraph is queried during single object resolution and prefetch paths (3d) ## Open Questions + +1. Is any work needed to ensure subgraph composition is not leveraged by a subgraph using the JSONB storage engine?