From 0f852554cbcd8f3749430e61313857575f213d5b Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 9 Dec 2019 13:44:19 -0600 Subject: [PATCH 01/18] engineering-plans: Start subgraph composition engineering plan --- .../0001-subgraph-composition.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 engineering-plans/0001-subgraph-composition.md diff --git a/engineering-plans/0001-subgraph-composition.md b/engineering-plans/0001-subgraph-composition.md new file mode 100644 index 0000000..d9654bf --- /dev/null +++ b/engineering-plans/0001-subgraph-composition.md @@ -0,0 +1,77 @@ +# PLAN-0001: Subgraph Composition + +
+
Author
+
Jorge Olivero
+ +
Implements
+
RFC-0001 Subgraph Composition
+ +
Engineering Plan pull request
+
Design PR
+ +
Obsoletes (if applicable)
+ +
Date of submission
+
2019-12-09
+ +
Date of approval
+
TBD
+ +
Approved by
+
TBD
+
+ +## Summary + +A description of what the Engineering Plan does in 1-2 paragraphs. + +## Implementation + +How is the change or the feature going to be implemented? Which parts need to be +changed and how? + +## Tests + +How is the change or the feature going to be tested? Outline what test cases +will be implemented and, roughly, how (e.g. as unit tests, integration tests +etc.). Describe what the corner cases are and how they are accounted for in the +tests. + +## Migration + +If the change is breaking, even if in parts, are (semi-)automated migrations +possible? If not, how are we going to move users forward (e.g. announce early to +prepare users in time for the change to be activated)? If migrations are +possible, how are they going to work? + +## Documentation + +How are the changes going to be documented? + +## Implementation Plan + +An iterative plan for implementing the change or new feature. + +Think hard to not skip over potential challenges. Break the implementation down +into a step-by-step plan where the scope of every step is clear and where there +is no, or little, uncertainty about each individual step and the sequence +overall. + +**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 +days. + +**Phases:** For big changes, it can make sense to break down the overall plan +into implementation phases. + +**Integration:** Explicit integration points must be included in the plan. What +are we going to submit for review and integration at what point? What can be +integrated early? + +In order for a plan to be approved, there must be _extremely high_ confidence +that the plan will work out. + +## Open Questions + +What are unresolved questions? Which areas may we have forgotten about or +neglected in the plan? From 1fdcb492a5ad7c317b7c343e107d3ed1f3a11615 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:23:25 -0600 Subject: [PATCH 02/18] engineering-plans: subgraph-schema-merging --- .../0001-subgraph-composition.md | 77 ------------- .../0002-subgraph-schema-merging.md | 107 ++++++++++++++++++ 2 files changed, 107 insertions(+), 77 deletions(-) delete mode 100644 engineering-plans/0001-subgraph-composition.md create mode 100644 engineering-plans/0002-subgraph-schema-merging.md diff --git a/engineering-plans/0001-subgraph-composition.md b/engineering-plans/0001-subgraph-composition.md deleted file mode 100644 index d9654bf..0000000 --- a/engineering-plans/0001-subgraph-composition.md +++ /dev/null @@ -1,77 +0,0 @@ -# PLAN-0001: Subgraph Composition - -
-
Author
-
Jorge Olivero
- -
Implements
-
RFC-0001 Subgraph Composition
- -
Engineering Plan pull request
-
Design PR
- -
Obsoletes (if applicable)
- -
Date of submission
-
2019-12-09
- -
Date of approval
-
TBD
- -
Approved by
-
TBD
-
- -## Summary - -A description of what the Engineering Plan does in 1-2 paragraphs. - -## Implementation - -How is the change or the feature going to be implemented? Which parts need to be -changed and how? - -## Tests - -How is the change or the feature going to be tested? Outline what test cases -will be implemented and, roughly, how (e.g. as unit tests, integration tests -etc.). Describe what the corner cases are and how they are accounted for in the -tests. - -## Migration - -If the change is breaking, even if in parts, are (semi-)automated migrations -possible? If not, how are we going to move users forward (e.g. announce early to -prepare users in time for the change to be activated)? If migrations are -possible, how are they going to work? - -## Documentation - -How are the changes going to be documented? - -## Implementation Plan - -An iterative plan for implementing the change or new feature. - -Think hard to not skip over potential challenges. Break the implementation down -into a step-by-step plan where the scope of every step is clear and where there -is no, or little, uncertainty about each individual step and the sequence -overall. - -**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 -days. - -**Phases:** For big changes, it can make sense to break down the overall plan -into implementation phases. - -**Integration:** Explicit integration points must be included in the plan. What -are we going to submit for review and integration at what point? What can be -integrated early? - -In order for a plan to be approved, there must be _extremely high_ confidence -that the plan will work out. - -## Open Questions - -What are unresolved questions? Which areas may we have forgotten about or -neglected in the plan? diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md new file mode 100644 index 0000000..43bb20c --- /dev/null +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -0,0 +1,107 @@ +# PLAN-0001: Subgraph Schema Merging for Subgraph Composition + +
+
Author
+
Jorge Olivero
+ +
Implements
+
RFC-0001 Subgraph Composition
+ +
Engineering Plan pull request
+
Design PR
+ +
Obsoletes (if applicable)
+ +
Date of submission
+
2019-12-09
+ +
Date of approval
+
TBD
+ +
Approved by
+
TBD
+
+ +## Summary + +Subgraph composition allows a subgraph to import types from a foreign subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. + +To generate an API schema for the subgraph, the local schema needs to be merged with each of the foreign schemas that it imports. The merging process needs to take the following into account: + +1. An imported schema not being available on the graph-node +2. An imported schema without a definition for the type which the local subgraph imports +3. A merged and cached api schema for a subgraph imported by name is updated +4. Ability to tell which subgraph/schema each type belongs to + +## Implementation + +There are two important parts to schema merging: + +1. Merging the schema + +2. Invalidating a cache entry when a subgraph's dependency, referenced by name, is updated and remerging that schema + +### Schema Merging + +Add a `merged_schema(Document, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported +types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. + +### Cache Invalidation + +In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. + +A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification it would scan the schemas in the cache for those which need to be remerged. + + +## Tests + +1. Schemas are merged correctly when all schemas and imported types are available + +2. Placeholder types are properly inserted into the merged schema when a schema is not available + +3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not + +4. The cache is invalidated when the store returns an updated version of a cache entry's dependency + +## Migration + +Subgraph composition is an additive feature which shouldn't require a special migration plan. + +## Documentation + +Documenation on https://thegraph.com/docs needs to outline: + +1. The reserved _Schema_ type and how to define imports on it +2. The semantics of importing by subgraph id vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema +3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query + +## Implementation Plan + +An iterative plan for implementing the change or new feature. + +Think hard to not skip over potential challenges. Break the implementation down +into a step-by-step plan where the scope of every step is clear and where there +is no, or little, uncertainty about each individual step and the sequence +overall. + +**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 +days. + +**Phases:** For big changes, it can make sense to break down the overall plan +into implementation phases. + +**Integration:** Explicit integration points must be included in the plan. What +are we going to submit for review and integration at what point? What can be +integrated early? + +In order for a plan to be approved, there must be _extremely high_ confidence +that the plan will work out. + +- Implment the `merged_schema` function (2d) +- Write tests for the `merged_schema` function (1d) +- Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) +- Add cache invalidation log to `Store::cached_schema` (2d) + +## Open Questions + +The details of how queries and subscriptions needs to be updated to leverage the types in a merged schema. From 9905197d020f67364b863f6d724b6c3a7a431f41 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:25:43 -0600 Subject: [PATCH 03/18] engineering-plans: Typo --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 43bb20c..cbc6e6e 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -100,7 +100,7 @@ that the plan will work out. - Implment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) -- Add cache invalidation log to `Store::cached_schema` (2d) +- Add cache invalidation logic to `Store::cached_schema` (2d) ## Open Questions From db843ae026d13a06c6db1af61d6b67962f82ccba Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:26:27 -0600 Subject: [PATCH 04/18] engineering-plans: Remove template content --- .../0002-subgraph-schema-merging.md | 20 ------------------- 1 file changed, 20 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index cbc6e6e..3217343 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -77,26 +77,6 @@ Documenation on https://thegraph.com/docs needs to outline: ## Implementation Plan -An iterative plan for implementing the change or new feature. - -Think hard to not skip over potential challenges. Break the implementation down -into a step-by-step plan where the scope of every step is clear and where there -is no, or little, uncertainty about each individual step and the sequence -overall. - -**Estimates:** Every step/task must come with an estimate of 1, 2, 3, 4 or 5 -days. - -**Phases:** For big changes, it can make sense to break down the overall plan -into implementation phases. - -**Integration:** Explicit integration points must be included in the plan. What -are we going to submit for review and integration at what point? What can be -integrated early? - -In order for a plan to be approved, there must be _extremely high_ confidence -that the plan will work out. - - Implment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) From d841b1e17c53ff4cae79e4d4575a965c2ac97caa Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 10:27:33 -0600 Subject: [PATCH 05/18] engineering-plan: Update link for PR --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 3217343..5018ff0 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -8,7 +8,7 @@
RFC-0001 Subgraph Composition
Engineering Plan pull request
-
Design PR
+
Engineering Plan PR
Obsoletes (if applicable)
From 323c8ebba9305b29ee012510a64960f4f66e7fe7 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 17:23:53 -0600 Subject: [PATCH 06/18] engineering-plans: Add sammple GraphQL documents to describe merging --- .../0002-subgraph-schema-merging.md | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 5018ff0..64a2bb7 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,9 +43,40 @@ There are two important parts to schema merging: ### Schema Merging -Add a `merged_schema(Document, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "..." } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Schema after calling `merged_schema` + +```graphql +type A @entity @subgraphId("...") { + id: ID! + foo: B! +} + +type B @entity @subgraphId("...") { + id: ID! + bar: String +} +``` + +After the schema document is merged, the `api_schema` will be called. + ### Cache Invalidation In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. @@ -77,11 +108,11 @@ Documenation on https://thegraph.com/docs needs to outline: ## Implementation Plan -- Implment the `merged_schema` function (2d) +- Impelment the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) - Add cache invalidation logic to `Store::cached_schema` (2d) ## Open Questions -The details of how queries and subscriptions needs to be updated to leverage the types in a merged schema. +The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. From 4856a89f2398555913a867d4ccae714dc0977243 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 13 Dec 2019 17:42:51 -0600 Subject: [PATCH 07/18] engineering-plans: Fix sentence --- engineering-plans/0002-subgraph-schema-merging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 64a2bb7..2500779 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -75,7 +75,7 @@ type B @entity @subgraphId("...") { } ``` -After the schema document is merged, the `api_schema` will be called. +After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation From 7e162111398a4b576282c8a902b1c3d44db3b7a2 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 09:46:23 -0600 Subject: [PATCH 08/18] engineering-plans: Shorten name and update plan # --- .../0002-subgraph-schema-merging.md | 50 +++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 2500779..147a15f 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -1,4 +1,4 @@ -# PLAN-0001: Subgraph Schema Merging for Subgraph Composition +# PLAN-0002: Subgraph Schema Merging
Author
@@ -46,13 +46,55 @@ There are two important parts to schema merging: Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +#### Example #1: Valid import and complete merge + +Local schema before calling `merged_schema`: + +```graphql +type _Schema_ + @import( + types: ["B"], + from: { id: "X" } + ) + +type A @entity { + id: ID! + foo: B! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema`: + +```graphql +type A @entity { + id: ID! + foo: B! +} + +type B @entity @subgraphId("X") { + id: ID! + bar: String +} +``` + +#### Example #2 + Schema before calling `merged_schema`: ```graphql type _Schema_ @import( types: ["B"], - from: { id: "..." } + from: { id: "X" } ) type A @entity { @@ -79,9 +121,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated an remerged before responding to the cache access. +In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated and remerged before responding to the cache access. -A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification it would scan the schemas in the cache for those which need to be remerged. +A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. ## Tests From 8636b8e6e877c5bdaec1388c1a5871ab5d3e3094 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 09:48:12 -0600 Subject: [PATCH 09/18] engineering-plans: Update markup --- engineering-plans/0002-subgraph-schema-merging.md | 1 + 1 file changed, 1 insertion(+) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 147a15f..df6e75c 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -11,6 +11,7 @@
Engineering Plan PR
Obsoletes (if applicable)
+
-
Date of submission
2019-12-09
From a2d9b470f3dbdd3229d8c77aa7b2c86784357517 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 12:53:06 -0600 Subject: [PATCH 10/18] engineering-plans: Address feedback on pull #6 --- .../0002-subgraph-schema-merging.md | 121 ++++++++++++------ 1 file changed, 80 insertions(+), 41 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index df6e75c..fbe6911 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -25,22 +25,21 @@ ## Summary -Subgraph composition allows a subgraph to import types from a foreign subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. +Subgraph composition allows a subgraph to import types from another subgraph. Imports are designed to be loosely coupled throughout the deployment and indexing phases, and tightly coupled during query execution for the subgraph. -To generate an API schema for the subgraph, the local schema needs to be merged with each of the foreign schemas that it imports. The merging process needs to take the following into account: +To generate an API schema for the subgraph, the subgraph schema needs to be merged with the imported subgraph schemas. The merging process needs to take the following into account: -1. An imported schema not being available on the graph-node -2. An imported schema without a definition for the type which the local subgraph imports -3. A merged and cached api schema for a subgraph imported by name is updated +1. An imported schema not being available on the Graph Node +2. An imported schema mising a type imported by the subgraph schema +3. A schema imported by subgraph name changes 4. Ability to tell which subgraph/schema each type belongs to ## Implementation -There are two important parts to schema merging: +The schema merging implementation consists of two parts: -1. Merging the schema - -2. Invalidating a cache entry when a subgraph's dependency, referenced by name, is updated and remerging that schema +1. A cache of subgraph schemas +2. The schema merging logic ### Schema Merging @@ -53,14 +52,14 @@ Local schema before calling `merged_schema`: ```graphql type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) + @import( + types: ["B"], + from: { id: "X" } + ) type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } ``` @@ -68,8 +67,8 @@ Imported Schema X: ```graphql type B @entity { - id: ID! - bar: String + id: ID! + bar: String } ``` @@ -77,13 +76,13 @@ Schema after calling `merged_schema`: ```graphql type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } type B @entity @subgraphId("X") { - id: ID! - bar: String + id: ID! + bar: String } ``` @@ -93,14 +92,14 @@ Schema before calling `merged_schema`: ```graphql type _Schema_ - @import( - types: ["B"], - from: { id: "X" } - ) + @import( + types: ["B"], + from: { id: "X" } + ) type A @entity { - id: ID! - foo: B! + id: ID! + foo: B! } ``` @@ -108,13 +107,53 @@ Schema after calling `merged_schema` ```graphql type A @entity @subgraphId("...") { - id: ID! - foo: B! + id: ID! + foo: B! } type B @entity @subgraphId("...") { - id: ID! - bar: String + id: ID! + bar: String +} +``` + +#### Example #3 + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type B @entity { + id: ID! + bar: String +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId("X") @mergeInfo(originalName: "B") { + id: ID! + bar: String } ``` @@ -129,25 +168,25 @@ A more performant invalidation solution would be to have the cache maintain a li ## Tests -1. Schemas are merged correctly when all schemas and imported types are available +1. Schemas are merged correctly when all schemas and imported types are available. -2. Placeholder types are properly inserted into the merged schema when a schema is not available +2. Placeholder types are properly inserted into the merged schema when a schema is not available. -3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not +3. Placeholder types are properly inserted into the merged schema when the relevant schemas are available but the types are not. -4. The cache is invalidated when the store returns an updated version of a cache entry's dependency +4. The cache is invalidated when the store returns an updated version of a cache entry's dependency. ## Migration -Subgraph composition is an additive feature which shouldn't require a special migration plan. +Subgraph composition is an additive feature which doesn't require a special migration plan. ## Documentation -Documenation on https://thegraph.com/docs needs to outline: +Documentation on https://thegraph.com/docs needs to outline: -1. The reserved _Schema_ type and how to define imports on it -2. The semantics of importing by subgraph id vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema -3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query +1. The reserved _Schema_ type and how to define imports on it. +2. The semantics of importing by subgraph ID vs. subgraph name, i.e. what happens when a subgraph imported by name removes expected types from the schema. +3. How queries are processed when imported subgraphs schemas or types are not available on the graph-node processing the query. ## Implementation Plan @@ -158,4 +197,4 @@ Documenation on https://thegraph.com/docs needs to outline: ## Open Questions -The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. +- The execution of queries and subscriptions needs to be updated to leverage the types in a merged schema. From 4139d9c89f4fc66a9e55fee75ba716b977a16d97 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 13:04:51 -0600 Subject: [PATCH 11/18] engineering-plans: @mergeInfo -> @originalName --- engineering-plans/0002-subgraph-schema-merging.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index fbe6911..7f0ac94 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,8 +43,9 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported -types to the proviced document with a @subgraphId diretive denoting which subgraph the type came from. The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. +If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive. +The `api_schema` function will add all the necessary types and fields for the imported types without any changes. #### Example #1: Valid import and complete merge @@ -151,7 +152,7 @@ type B @entity { foo: BB! } -type BB @entity @subgraphId("X") @mergeInfo(originalName: "B") { +type BB @entity @subgraphId("X") @originalName("B") { id: ID! bar: String } From 20edb4b1aaf2134a8774843bee4f273aa27de8a0 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 13:41:39 -0600 Subject: [PATCH 12/18] engineering-plans: Add more detail to the merge algorithm --- .../0002-subgraph-schema-merging.md | 29 +++++++++++++------ 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 7f0ac94..053d967 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,11 +43,17 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. -If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive. -The `api_schema` function will add all the necessary types and fields for the imported types without any changes. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. -#### Example #1: Valid import and complete merge +The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. + +- If any schema node along that path is missing or if the type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId("...")` directive (if available), and a `@placeholder` directive denoting that type was not found. +- If the type is found, copy it and add a `@subgraphId("...")` directive. +- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive preserving the type name from the original schema. + +The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. + +#### Example #1: Complete merge Local schema before calling `merged_schema`: @@ -87,7 +93,7 @@ type B @entity @subgraphId("X") { } ``` -#### Example #2 +#### Example #2: Incomplete merge Schema before calling `merged_schema`: @@ -104,6 +110,12 @@ type A @entity { } ``` +Imported Schema X: + +```graphql +NOT AVAILABLE +``` + Schema after calling `merged_schema` ```graphql @@ -112,13 +124,12 @@ type A @entity @subgraphId("...") { foo: B! } -type B @entity @subgraphId("...") { +type B @entity @placeholder { id: ID! - bar: String } ``` -#### Example #3 +#### Example #3: Complete merge with `{ name: "...", as: "..." }` Schema before calling `merged_schema` @@ -191,7 +202,7 @@ Documentation on https://thegraph.com/docs needs to outline: ## Implementation Plan -- Impelment the `merged_schema` function (2d) +- Implement the `merged_schema` function (2d) - Write tests for the `merged_schema` function (1d) - Integrate `merged_schema` into `Store::cached_schema` and update the cache to include the relevant information for imported schemas and types (1d) - Add cache invalidation logic to `Store::cached_schema` (2d) From 84480da0e71dba94ba5d4eeae8c1ea307f51cad6 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 22:31:37 -0600 Subject: [PATCH 13/18] engineering-plans: Add detail to cache invalidation --- engineering-plans/0002-subgraph-schema-merging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index 053d967..d2bd24a 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -173,7 +173,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -In the initial implementation, the cache will need to check if an entry needs to be updated when it is accessed. To determine whether an entry in the schema cache has been invalidated, each subgraph schema it imports by name needs to be queried in the store for its most current version. The schema cache will maintain a record of the versions used for merging in each entry, and if the version previously used to merge is no longer current that entry needs to be invalidated and remerged before responding to the cache access. +For each schema in the cache, keep a vector of volatile schemas containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recentschema merge. If there are any new versions, re merge the schema. + +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, and a `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`) A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 111c875e43fd6447f99203717ff80c5fd1f98ccf Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Mon, 16 Dec 2019 22:35:25 -0600 Subject: [PATCH 14/18] engineering-plans: Add detail to schema merging cache invalidation --- engineering-plans/0002-subgraph-schema-merging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index d2bd24a..cdaea5a 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -173,9 +173,9 @@ After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -For each schema in the cache, keep a vector of volatile schemas containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recentschema merge. If there are any new versions, re merge the schema. +For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. -Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, and a `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`) +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 75ff610b649802aa9692d7b7343f727450f89cca Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 17 Dec 2019 12:21:32 -0600 Subject: [PATCH 15/18] engineering-plans: Add entry to approved.md --- engineering-plans/approved.md | 1 + 1 file changed, 1 insertion(+) diff --git a/engineering-plans/approved.md b/engineering-plans/approved.md index 8c1f549..5697314 100644 --- a/engineering-plans/approved.md +++ b/engineering-plans/approved.md @@ -3,3 +3,4 @@ - [PLAN-0001: GraphQL Query Prefetching](./0001-graphql-query-prefetching.md) - [PLAN-0002: Ethereum Tracing Cache](./0002-ethereum-tracing-cache.md) - [PLAN-0003: Remove JSONB Storage](./0003-remove-jsonb-storage.md) +- [PLAN-0004: Subgraph Schema Merging](./0002-subgraph-schema-merging.md) From b367c3f7c924d26f0e8d623ba609ed1822f37805 Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 17 Dec 2019 12:52:29 -0600 Subject: [PATCH 16/18] SUMMARY: Add subgraph schema merging to list --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 3efaac7..9b8f840 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -12,5 +12,6 @@ - [PLAN-0001: GraphQL Query Prefetching](./engineering-plans/0001-graphql-query-prefetching.md) - [PLAN-0002: Ethereum Tracing Cache](./engineering-plans/0002-ethereum-tracing-cache.md) - [PLAN-0003: Remove JSONB Storage](./engineering-plans/0003-remove-jsonb-storage.md) + - [PLAN-0004: Subgraph Schema Merging](./engineering-plans/0004-subgraph-schema-merging.md) - [Obsolete Plans](./engineering-plans/obsolete.md) - [Rejected Plans](./engineering-plans/rejected.md) From 10acac4fd150b210f8ae067860669333bd3a2eca Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Fri, 20 Dec 2019 17:44:55 -0600 Subject: [PATCH 17/18] engineering-plans: Address importing types for non scalar fields --- .../0002-subgraph-schema-merging.md | 88 +++++++++++++++++-- 1 file changed, 79 insertions(+), 9 deletions(-) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0002-subgraph-schema-merging.md index cdaea5a..480c29d 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0002-subgraph-schema-merging.md @@ -43,13 +43,13 @@ The schema merging implementation consists of two parts: ### Schema Merging -Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a @subgraphId diretive denoting which subgraph the type came from. +Add a `merged_schema(&Schema, HashMap) -> Schema` function to the `graphql::schema` crate which will add each of the imported types to the provided document with a `@subgraphId` directive denoting which subgraph the type came from. If any of the imported types have non scalar fields, import those types as well. The `HashMap` includes all of the schemas in the subgraph's import graph which are available on the Graph Node. For each `@import` directive on the subgraph, find the imported types by tracing their path along the import graph. -- If any schema node along that path is missing or if the type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId("...")` directive (if available), and a `@placeholder` directive denoting that type was not found. -- If the type is found, copy it and add a `@subgraphId("...")` directive. -- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName("...")` directive preserving the type name from the original schema. +- If any schema node along that path is missing or if a type is missing in the schema, add a type definition to the subgraphs merged schema with the proper name, a `@subgraphId(id: "...")` directive (if available), and a `@placeholder` directive denoting that type was not found. +sc- If the type is found, copy it and add a `@subgraphId(id: "...")` directive. +- If the type is imported with the `{ name: "", as: "" }` format, the merged type will include an `@originalName(name: "...")` directive preserving the type name from the original schema. The `api_schema` function will add all the necessary types and fields for the imported types without requiring any changes. @@ -87,7 +87,7 @@ type A @entity { foo: B! } -type B @entity @subgraphId("X") { +type B @entity @subgraphId(id: "X") { id: ID! bar: String } @@ -119,7 +119,7 @@ NOT AVAILABLE Schema after calling `merged_schema` ```graphql -type A @entity @subgraphId("...") { +type A @entity @subgraphId(id: "...") { id: ID! foo: B! } @@ -163,19 +163,89 @@ type B @entity { foo: BB! } -type BB @entity @subgraphId("X") @originalName("B") { +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { id: ID! bar: String } ``` +#### Example #4: Complete merge with nested types + +Schema before calling `merged_schema` + +```graphql +type _Schema_ + @imports( + types: [{ name: "B", as: "BB" }] + from: { id: "X" } + ) + +type B @entity { + id: ID! + foo: BB! +} +``` + +Imported Schema X: + +```graphql +type _Schema_ + @imports( + types: [{ name: "C", as: "CC" }] + from: { id: "Y" } + ) + +type B @entity { + id: ID! + bar: CC! + baz: DD! +} + +type DD @entity { + id: ID! +} +``` + +Imported Schema Y: + +```graphql +type C @entity { + id: ID! +} +``` + +Schema after calling `merged_schema` + +```graphql +type B @entity { + id: ID! + foo: BB! +} + +type BB @entity @subgraphId(id: "X") @originalName(name: "B") { + id: ID! + bar: CC! + baz: DD! +} + +type CC @entity @subgraphId(id: "Y") @originalName(name: "C") { + id: ID! +} + +type DD @entity @subgraphId(id: "X") { + id: ID! +} +``` + + + After the schema document is merged, the `api_schema` function will be called. ### Cache Invalidation -For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph_id which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. +For each schema in the cache, keep a vector of subgraph pointers containing an element for each schema in the subgraph's import graph which was imported by name and the subgraph ID which was used during the schema merge. When a schema is accessed from the schema cache (and possibly only if this check hasn't happened in the last N seconds), check the current version for each of these schemas and run a diff against the versions used for the most recent schema merge. If there are any new versions, re merge the schema. -Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported_by_name` (`Vec<(SubgraphDeploymentName, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. +Currently the `schema_cache` in the `Store` is a `Mutex>`. A `SchemaPair` consists of two fields: `input_schema` and `api_schema`. To support the refresh flow, `SchemaPair` would be extended to be a `SchemaEntry`, with the fields `input_schema`, `api_schema`, `schemas_imported` (`Vec<(SchemaReference, SubgraphDeploymentId)>`), and a `last_refresh_check` timestamp. A more performant invalidation solution would be to have the cache maintain a listener notifying it every time a subgraph's current version changes. Upon receiving the notification the listener scans the schemas in the cache for those which should be remerged. From 1da8b0711ae4b30bcd773c82b114acd46530796b Mon Sep 17 00:00:00 2001 From: Jorge Olivero Date: Tue, 14 Jan 2020 17:16:32 -0600 Subject: [PATCH 18/18] engineering-plans: Rename subgraph schema merging plan --- ...bgraph-schema-merging.md => 0004-subgraph-schema-merging.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename engineering-plans/{0002-subgraph-schema-merging.md => 0004-subgraph-schema-merging.md} (99%) diff --git a/engineering-plans/0002-subgraph-schema-merging.md b/engineering-plans/0004-subgraph-schema-merging.md similarity index 99% rename from engineering-plans/0002-subgraph-schema-merging.md rename to engineering-plans/0004-subgraph-schema-merging.md index 480c29d..72ac979 100644 --- a/engineering-plans/0002-subgraph-schema-merging.md +++ b/engineering-plans/0004-subgraph-schema-merging.md @@ -1,4 +1,4 @@ -# PLAN-0002: Subgraph Schema Merging +# PLAN-0004: Subgraph Schema Merging
Author