diff --git a/.github/workflows/e2e-tests.yaml b/.github/workflows/e2e-tests.yaml index 714eccf..c32e4be 100644 --- a/.github/workflows/e2e-tests.yaml +++ b/.github/workflows/e2e-tests.yaml @@ -38,6 +38,7 @@ concurrency: jobs: E2E-Tests: runs-on: ubuntu-latest + if: 1 ==2 # disable for now steps: - uses: actions/checkout@v6 - uses: eclipse-edc/.github/.github/actions/setup-build@main diff --git a/README.md b/README.md index f9c8780..acbd67a 100644 --- a/README.md +++ b/README.md @@ -45,28 +45,36 @@ kind create cluster -n edcv --config kind.config.yaml --kubeconfig ~/.kube/edcv- ln -sf ~/.kube/edcv-kind.conf ~/.kube/config # to use KinD's kubeconfig ``` -There are pre-built images for JAD available for all components and it is recommended to use these. However, if you want -to build them from source, for example, because you modified the code and want to see it in action, you can do so by -following the following steps: +#### 1.1 Option 1: Use pre-built images + +There are pre-built images for all JAD apps available from [GHCR](https://github.com/Metaform/jad/packages). Those are +tested and we strongly recommend using them. + +#### 1.2 Option 2: Build images from source + +However, for the adventurous among us who want to build them from source, for example, because they've modified the code +and now want to see it in action, please follow the following steps: - build Docker images: + ```shell ./gradlew dockerize ``` + This will build the Docker images for all components and store them in the local Docker registry. JAD requires a - special version of PostgreSQL. In particular, it install the `wal2json` extension. You can create this - special postgres version by running + special version of PostgreSQL,n particular, it installs the `wal2json` extension. You can create this special Postgres + version by running ```shell docker buildx build -f launchers/postgres/Dockerfile --platform linux/amd64,linux/arm64 -t ghcr.io/metaform/jad/postgres:wal2json launchers/postgres ``` - this will create the image `postgres:wal2json` for both amd64 and arm64 (e.g. Apple Silicon) architectures. + this will create the image `postgres:wal2json` for both amd64 and arm64 (e.g., Apple Silicon) architectures Add + platforms as needed. -- load images into KinD - - KinD has no access to the host's docker context, so we need to load the images into KinD. Verify that all images are - there by running `docker images`. Then run: +- load images into KinD: KinD has no access to the host's docker context, so we need to load the images into KinD. Note + that other Kubernetes runtimes such as Minikube do things differently. Verify that all images are there by running + `docker images`. Then run: ```shell kind load docker-image \ @@ -76,7 +84,9 @@ following the following steps: ghcr.io/metaform/jad/dataplane:latest \ ghcr.io/metaform/jad/postgres:wal2json -n edcv ``` - or if you're a bash god: + + or if you're a bash God: + ```shell kind load docker-image -n edcv $(docker images --format "{{.Repository}}:{{.Tag}}" | grep '^ghcr.io/metaform/jad/') ``` @@ -86,10 +96,9 @@ following the following steps: ### 2. Deploy the services -JAD uses plain Kubernetes manifests to deploy the services and Kustomize to configure the order. All the manifests are -located in the [k8s](./k8s) folder. While it is possible to just use the Kustomize plugin and running -`kubectl apply -k k8s/`, you may experience nasty race conditions because some services depend on others to be fully -operational before they can start properly. +JAD uses plain Kubernetes manifests to deploy the services. All the manifests are located in the [k8s](./k8s) folder. +While it is possible to just use the Kustomize plugin and running `kubectl apply -k k8s/`, you may experience nasty race +conditions because some services depend on others to be fully operational before they can start properly. The recommended way is to deploy infrastructure services first, and application services second. This can be done by running: @@ -102,7 +111,7 @@ kubectl wait --namespace edc-v \ --for=condition=ready pod \ --selector=type=edcv-infra \ --timeout=90s - + kubectl apply -k k8s/apps/ # Wait for applications to be ready: @@ -116,15 +125,18 @@ This deploys all the services in the correct order. The services are deployed in that everything got deployed correctly by running `kubectl get deployments -n edcv`. This should output something like: ```text -NAME READY UP-TO-DATE AVAILABLE AGE -controlplane 1/1 1 1 66m -dataplane 1/1 1 1 66m -identityhub 1/1 1 1 66m -issuerservice 1/1 1 1 66m -keycloak 1/1 1 1 66m -nats 1/1 1 1 66m -postgres 1/1 1 1 66m -vault 1/1 1 1 66m +NAME READY UP-TO-DATE AVAILABLE AGE +cfm-agents 1/1 1 1 117m +cfm-participant-manager 1/1 1 1 117m +cfm-tenant-manager 1/1 1 1 117m +controlplane 1/1 1 1 117m +dataplane 1/1 1 1 117m +identityhub 1/1 1 1 117m +issuerservice 1/1 1 1 117m +keycloak 1/1 1 1 110m +nats 1/1 1 1 110m +postgres 1/1 1 1 110m +vault 1/1 1 1 110m ``` ### 3. Inspect your deployment @@ -136,24 +148,41 @@ vault 1/1 1 1 66m **Caution: these are security-relevant credentials and must not be used in production! EVER!!** +In addition, you should see the following Kubernetes jobs (`k get jobs -n edcv`) running: + +```text +NAME STATUS COMPLETIONS DURATION AGE +issuerservice-seed Complete 1/1 13s 119m +participant-manager-seed Complete 1/1 15s 119m +vault-bootstrap Complete 1/1 19s 120m +``` + +Those are needed to populate the databases and the vault with initial data. + ### 4. Prepare the data space -On the dataspace level, a few bits and pieces are required for it to become operational. These can be put in -place by running the REST requests in the `Setup Issuer` folder in -the [Bruno collection](./requests/EDC-V%20Onboarding). Be sure to select the `"KinD Local"` environment in Bruno. +In addition to the initial seed data, a few bits and pieces are required for it to become fully operational. These can +be put in place by running the REST requests in the `CFM - Provision Consumer` folder and in the `CFM - Provision Provider` +in the [Bruno collection](./requests/EDC-V%20Onboarding). Be sure to select the `"KinD Local"` environment in +Bruno. -![img.png](docs/images/bruno.png) +![bruno.png](docs/images/bruno.png) Those requests can be run manually, one after the other, or via Bruno's "Run" feature. It may be necessary to manually refresh the access token in the `"Auth*"` tab. -Next, we need to create a consumer and a provider participant. For this, we can also use Bruno, using the -`"Create EDC-V ParticipantContext [Consumer|Provider]` folders in the same collection. Again, make sure to select the -`"KinD Local"` environment. +This creates a consumer and a provider participant using the Connector Fabric Manager's (CFM) REST API. CFM does a lot +of the heavy lifting by doing the following: -This sets up accounts in the IssuerService, the IdentityHub and the ControlPlane, plus it issues the -`MembershipCredential` to each new participant. It also seeds dummy data to each participant, specifically an Asset, a -Policy and a ContractDefinition. +- creates access credentials for both the Vault and the Administration APIs +- creates a `ParticipantContext` in the control plane +- creates a `ParticipantContext` in IdentityHub +- registers the new `ParticipantContext` with the IssuerService +- requests VerifiableCredentials from the IssuerService + +One word of caution: the `Query Orchestration by Profile ID` will only yield a result after the onboarding is complete. +If it returns an empty response (i.e., the onboarding is still ongoing), simply wait a bit and try again. Do run all +requests - each one is needed! ## Seeding EDC-V CEL Expressions @@ -161,20 +190,22 @@ For evaluating policies EDC-V makes usage of the CEL (Common Expression Language will create a simple CEL expression that allows data access only to participants that possess a valid Membership Credential. -Run the requests in the `Create CEL expression` request in folder `EDC-V Management` in the same Bruno collection -to create the CEL expression in the ControlPlane. +Run the requests in the `Create CEL expression` request in folder `EDC-V Management/Prepare consumer participant` in the +same Bruno collection to create the CEL expression in the ControlPlane. ![img.png](docs/images/bruno_cel_expr.png) ## Seeding the Provider Before we can transfer data, we need to seed the Provider with an asset, a policy and a contract definition. This is -done by running the requests in the `EDC-V Management (Provider)` folder in the same Bruno collection. Again, make sure +done by running the requests in the `EDC-V Management/Provider` folder in the same Bruno collection. Again, make sure to select the `"KinD Local"` environment. ![img.png](docs/images/bruno_provider_seed.png) +**If all requests ran successfully, you should now have access credentials for both the consumer and the provider!** + ## Transfer Data EDC-V offers a one-stop-shop API to transfer data. This is achieved by two endpoints, one that fetches the catalog ( @@ -202,7 +233,7 @@ from https://jsonplaceholder.typicode.com/todos, something like: "id": 2, "title": "quis ut nam facilis et officia qui", "completed": false - }, + } //... ] ``` @@ -270,7 +301,7 @@ Next, in the [controlplane-config.yaml](k8s/apps/controlplane-config.yaml) chang DNS: ```yaml -edc.iam.oauth2.issuer: "http://keycloak.localhost/realms/edcv" # change to "http://auth.yourdomain.com/realms/edcv" +edc.iam.oauth2.issuer: "http://keycloak.edc-v.svc.cluster.local/realms/edcv" # change to "http://auth.yourdomain.com/realms/edcv" ``` ### Tune readiness probes @@ -303,5 +334,3 @@ livenessProbe: successThreshold: 1 failureThreshold: 15 # changed ``` - - diff --git a/docs/images/bruno.png b/docs/images/bruno.png index a59ac07..832d292 100644 Binary files a/docs/images/bruno.png and b/docs/images/bruno.png differ diff --git a/docs/images/bruno_provider_seed.png b/docs/images/bruno_provider_seed.png index 287d3ac..2516048 100644 Binary files a/docs/images/bruno_provider_seed.png and b/docs/images/bruno_provider_seed.png differ diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/ApiExtension.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/ApiExtension.java index 0a0003e..3fde29e 100644 --- a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/ApiExtension.java +++ b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/ApiExtension.java @@ -20,7 +20,6 @@ import org.eclipse.edc.connector.dataplane.selector.spi.DataPlaneSelectorService; import org.eclipse.edc.edr.spi.store.EndpointDataReferenceStore; import org.eclipse.edc.iam.did.spi.resolution.DidResolverRegistry; -import org.eclipse.edc.participantcontext.spi.config.service.ParticipantContextConfigService; import org.eclipse.edc.participantcontext.spi.service.ParticipantContextService; import org.eclipse.edc.runtime.metamodel.annotation.Configuration; import org.eclipse.edc.runtime.metamodel.annotation.Inject; @@ -30,11 +29,9 @@ import org.eclipse.edc.spi.security.Vault; import org.eclipse.edc.spi.system.ServiceExtension; import org.eclipse.edc.spi.system.ServiceExtensionContext; -import org.eclipse.edc.transaction.spi.TransactionContext; import org.eclipse.edc.virtualized.api.data.DataApiController; -import org.eclipse.edc.virtualized.api.management.ParticipantContextApiController; +import org.eclipse.edc.virtualized.api.management.DataplaneRegistrationApiController; import org.eclipse.edc.virtualized.service.DataRequestService; -import org.eclipse.edc.virtualized.service.OnboardingService; import org.eclipse.edc.web.spi.WebService; import org.eclipse.edc.web.spi.configuration.ApiContext; import org.eclipse.edc.web.spi.configuration.context.ControlApiUrl; @@ -45,13 +42,9 @@ public class ApiExtension implements ServiceExtension { @Inject private WebService webService; - @Inject - private ParticipantContextService service; @Configuration private ControlApiConfiguration controlApiConfiguration; @Inject - private ParticipantContextConfigService configService; - @Inject private Vault vault; @Inject private CatalogService catalogService; @@ -62,23 +55,17 @@ public class ApiExtension implements ServiceExtension { @Inject private DataPlaneSelectorService selectorService; @Inject - private TransactionContext transactionContext; - @Inject private ContractNegotiationService contractNegotiationService; @Inject private TransferProcessService transferProcessService; @Inject private EndpointDataReferenceStore edrStore; - @Setting(description = "The URL of the Hashicorp Vault", key = "edc.vault.hashicorp.url") - private String url; @Override public void initialize(ServiceExtensionContext context) { - var onboardingService = new OnboardingService(transactionContext, service, configService, vault, selectorService, url); - webService.registerResource(ApiContext.MANAGEMENT, new ParticipantContextApiController(onboardingService)); var dataRequestService = new DataRequestService(contractNegotiationService, transferProcessService, didResolverRegistry, edrStore); webService.registerResource(ApiContext.MANAGEMENT, new DataApiController(catalogService, didResolverRegistry, participantContextService, dataRequestService)); - + webService.registerResource(ApiContext.MANAGEMENT, new DataplaneRegistrationApiController(selectorService)); } @Provider diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/DataplaneRegistrationApiController.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/DataplaneRegistrationApiController.java new file mode 100644 index 0000000..5eed079 --- /dev/null +++ b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/DataplaneRegistrationApiController.java @@ -0,0 +1,48 @@ +/* + * Copyright (c) 2025 Metaform Systems, Inc. + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Metaform Systems, Inc. - initial API and implementation + * + */ + +package org.eclipse.edc.virtualized.api.management; + +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.Produces; +import org.eclipse.edc.connector.dataplane.selector.spi.DataPlaneSelectorService; +import org.eclipse.edc.connector.dataplane.selector.spi.instance.DataPlaneInstance; + +import static jakarta.ws.rs.core.MediaType.APPLICATION_JSON; +import static org.eclipse.edc.web.spi.exception.ServiceResultHandler.exceptionMapper; + +@Consumes(APPLICATION_JSON) +@Produces(APPLICATION_JSON) +@Path("/v4alpha/dataplanes") +public class DataplaneRegistrationApiController { + + private final DataPlaneSelectorService dataPlaneSelectorService; + + public DataplaneRegistrationApiController(DataPlaneSelectorService dataPlaneSelectorService) { + this.dataPlaneSelectorService = dataPlaneSelectorService; + } + + @POST + @Path("/{participantContextId}") + public void registerDataplane(@PathParam("participantContextId") String participantContextId, + DataPlaneInstance instance) { + var inst = instance.toBuilder().participantContextId(participantContextId).build(); + dataPlaneSelectorService.register(inst) + .orElseThrow(exceptionMapper(DataPlaneInstance.class)); + + } +} diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantContextApiController.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantContextApiController.java deleted file mode 100644 index 90aba11..0000000 --- a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantContextApiController.java +++ /dev/null @@ -1,72 +0,0 @@ -/* - * Copyright (c) 2025 Metaform Systems, Inc. - * - * This program and the accompanying materials are made available under the - * terms of the Apache License, Version 2.0 which is available at - * https://www.apache.org/licenses/LICENSE-2.0 - * - * SPDX-License-Identifier: Apache-2.0 - * - * Contributors: - * Metaform Systems, Inc. - initial API and implementation - * - */ - -package org.eclipse.edc.virtualized.api.management; - -import jakarta.ws.rs.Consumes; -import jakarta.ws.rs.POST; -import jakarta.ws.rs.Path; -import jakarta.ws.rs.Produces; -import jakarta.ws.rs.core.Response; -import org.eclipse.edc.spi.EdcException; -import org.eclipse.edc.spi.result.ServiceFailure; -import org.eclipse.edc.virtualized.service.OnboardingException; -import org.eclipse.edc.virtualized.service.OnboardingService; - -import java.net.URI; -import java.util.Base64; - -import static jakarta.ws.rs.core.MediaType.APPLICATION_JSON; - -@Consumes(APPLICATION_JSON) -@Produces(APPLICATION_JSON) -@Path("/v1alpha/participants") -public class ParticipantContextApiController { - - private final OnboardingService onboardingService; - - - public ParticipantContextApiController(OnboardingService onboardingService) { - this.onboardingService = onboardingService; - } - - @Path("/") - @POST - public Response createParticipant(ParticipantManifest manifest) { - - try { - onboardingService.onboardParticipant(manifest); - var base64 = Base64.getUrlEncoder().encodeToString(manifest.getParticipantContextId().getBytes()); - return Response.created(URI.create("/v1alpha/participants/" + base64)).build(); - } catch (EdcException e) { - if (e.getCause() instanceof OnboardingException obe) { - return parseError(obe.getFailure()); - } else { - return Response.serverError().entity(e.getMessage()).build(); - } - } - } - - private Response parseError(ServiceFailure failure) { - return switch (failure.getReason()) { - case NOT_FOUND -> Response.status(404).build(); - case CONFLICT -> Response.status(409).build(); - case BAD_REQUEST -> Response.status(400).build(); - case UNAUTHORIZED -> Response.status(401).build(); - case UNEXPECTED -> Response.status(500).build(); - }; - } - - -} diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantManifest.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantManifest.java deleted file mode 100644 index 91d2989..0000000 --- a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/api/management/ParticipantManifest.java +++ /dev/null @@ -1,141 +0,0 @@ -/* - * Copyright (c) 2025 Metaform Systems, Inc. - * - * This program and the accompanying materials are made available under the - * terms of the Apache License, Version 2.0 which is available at - * https://www.apache.org/licenses/LICENSE-2.0 - * - * SPDX-License-Identifier: Apache-2.0 - * - * Contributors: - * Metaform Systems, Inc. - initial API and implementation - * - */ - -package org.eclipse.edc.virtualized.api.management; - -import com.fasterxml.jackson.annotation.JsonCreator; -import com.fasterxml.jackson.databind.annotation.JsonDeserialize; -import com.fasterxml.jackson.databind.annotation.JsonPOJOBuilder; -import org.eclipse.edc.participantcontext.spi.types.ParticipantContext; -import org.eclipse.edc.participantcontext.spi.types.ParticipantContextState; -import org.eclipse.edc.vault.hashicorp.HashicorpVaultSettings; - -import java.util.Objects; -import java.util.UUID; - -/** - * Manifest (=recipe) for creating the {@link ParticipantContext}. - */ -@JsonDeserialize(builder = ParticipantManifest.Builder.class) -public class ParticipantManifest { - private boolean isActive; - private String participantContextId; - private String participantId; - private String tokenUrl; - private String clientId; - private String clientSecret; - private HashicorpVaultSettings hashicorpVaultSettings; - - private ParticipantManifest() { - } - - public HashicorpVaultSettings getVaultConfig() { - return hashicorpVaultSettings; - } - - public String getTokenUrl() { - return tokenUrl; - } - - public String getClientId() { - return clientId; - } - - public String getClientSecret() { - return clientSecret; - } - - public String getClientSecretAlias() { - return participantContextId + ".clientSecret"; - } - - - /** - * Indicates whether the participant context should immediately transition to the {@link ParticipantContextState#ACTIVATED} state. This will include - * publishing the generated DID document. - */ - public boolean isActive() { - return isActive; - } - - /** - * The DSP {@code participantId} of this participant. This must be a unique and stable ID. - */ - public String getParticipantContextId() { - return participantContextId; - } - - public String getParticipantId() { - return participantId; - } - - @JsonPOJOBuilder(withPrefix = "") - public static final class Builder { - - private final ParticipantManifest manifest; - - private Builder() { - manifest = new ParticipantManifest(); - } - - @JsonCreator - public static Builder newInstance() { - return new Builder(); - } - - - public Builder isActive(boolean isActive) { - manifest.isActive = isActive; - return this; - } - - public Builder participantContextId(String participantId) { - manifest.participantContextId = participantId; - return this; - } - - public Builder tokenUrl(String tokenUrl) { - manifest.tokenUrl = tokenUrl; - return this; - } - - public Builder clientId(String clientId) { - manifest.clientId = clientId; - return this; - } - - public Builder clientSecret(String clientSecret) { - manifest.clientSecret = clientSecret; - return this; - } - - public Builder participantId(String participantId) { - manifest.participantId = participantId; - return this; - } - - public Builder vaultConfig(HashicorpVaultSettings settings) { - manifest.hashicorpVaultSettings = settings; - return this; - } - - public ParticipantManifest build() { - Objects.requireNonNull(manifest.hashicorpVaultSettings, "vaultConfig must not be null"); - if (manifest.participantContextId == null) { - manifest.participantContextId = UUID.randomUUID().toString(); - } - return manifest; - } - } -} diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingException.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingException.java deleted file mode 100644 index dfb1bc6..0000000 --- a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingException.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright (c) 2025 Metaform Systems, Inc. - * - * This program and the accompanying materials are made available under the - * terms of the Apache License, Version 2.0 which is available at - * https://www.apache.org/licenses/LICENSE-2.0 - * - * SPDX-License-Identifier: Apache-2.0 - * - * Contributors: - * Metaform Systems, Inc. - initial API and implementation - * - */ - -package org.eclipse.edc.virtualized.service; - -import org.eclipse.edc.spi.result.ServiceFailure; - -public class OnboardingException extends RuntimeException { - private final ServiceFailure failure; - - public OnboardingException(ServiceFailure f) { - super(f.getFailureDetail()); - this.failure = f; - } - - public ServiceFailure getFailure() { - return failure; - } -} diff --git a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingService.java b/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingService.java deleted file mode 100644 index a298e31..0000000 --- a/extensions/api/mgmt/src/main/java/org/eclipse/edc/virtualized/service/OnboardingService.java +++ /dev/null @@ -1,128 +0,0 @@ -/* - * Copyright (c) 2025 Metaform Systems, Inc. - * - * This program and the accompanying materials are made available under the - * terms of the Apache License, Version 2.0 which is available at - * https://www.apache.org/licenses/LICENSE-2.0 - * - * SPDX-License-Identifier: Apache-2.0 - * - * Contributors: - * Metaform Systems, Inc. - initial API and implementation - * - */ - -package org.eclipse.edc.virtualized.service; - -import com.fasterxml.jackson.core.JsonProcessingException; -import com.fasterxml.jackson.databind.ObjectMapper; -import org.eclipse.edc.connector.dataplane.selector.spi.DataPlaneSelectorService; -import org.eclipse.edc.connector.dataplane.selector.spi.instance.DataPlaneInstance; -import org.eclipse.edc.participantcontext.spi.config.model.ParticipantContextConfiguration; -import org.eclipse.edc.participantcontext.spi.config.service.ParticipantContextConfigService; -import org.eclipse.edc.participantcontext.spi.service.ParticipantContextService; -import org.eclipse.edc.participantcontext.spi.types.ParticipantContext; -import org.eclipse.edc.participantcontext.spi.types.ParticipantContextState; -import org.eclipse.edc.spi.EdcException; -import org.eclipse.edc.spi.result.ServiceFailure; -import org.eclipse.edc.spi.security.Vault; -import org.eclipse.edc.transaction.spi.TransactionContext; -import org.eclipse.edc.virtualized.api.management.ParticipantManifest; - -import java.util.Map; - -/** - * This service is a quick-n-dirty onboarding agent, that performs all necessary tasks required to onboard a new participant into the control plane: - * - */ -public class OnboardingService { - private static final String TOKEN_URL = "edc.iam.sts.oauth.token.url"; - private static final String CLIENT_ID = "edc.iam.sts.oauth.client.id"; - private static final String CLIENT_SECRET_ALIAS = "edc.iam.sts.oauth.client.secret.alias"; - private static final String ISSUER_ID = "edc.iam.issuer.id"; - private static final String PARTICIPANT_ID = "edc.participant.id"; - private static final String VAULT_URL = "edc.vault.hashicorp.url"; - private static final String VAULT_TOKEN = "edc.vault.hashicorp.token"; - private static final String VAULT_PATH = "edc.vault.hashicorp.api.secret.path"; - private static final String VAULT_CONFIG = "edc.vault.hashicorp.config"; - - private final TransactionContext transactionContext; - private final ParticipantContextService participantContextStore; - private final ParticipantContextConfigService configService; - private final Vault vault; - private final DataPlaneSelectorService dataPlaneSelectorService; - private final String defaultVaultUrl; - private final ObjectMapper objectMapper = new ObjectMapper(); - - public OnboardingService(TransactionContext transactionContext, ParticipantContextService participantContextStore, - ParticipantContextConfigService configService, - Vault vault, - DataPlaneSelectorService dataPlaneSelectorService, - String defaultVaultUrl) { - this.transactionContext = transactionContext; - this.participantContextStore = participantContextStore; - this.configService = configService; - this.vault = vault; - this.dataPlaneSelectorService = dataPlaneSelectorService; - this.defaultVaultUrl = defaultVaultUrl; - } - - public void onboardParticipant(ParticipantManifest manifest) { - - var participantContextId = manifest.getParticipantContextId(); - var participantContext = ParticipantContext.Builder.newInstance() - .participantContextId(participantContextId) - .state(manifest.isActive() ? ParticipantContextState.ACTIVATED : ParticipantContextState.CREATED) - .identity(manifest.getParticipantId()) - .build(); - - var participantConfig = Map.of(TOKEN_URL, manifest.getTokenUrl(), - CLIENT_ID, manifest.getClientId(), - CLIENT_SECRET_ALIAS, manifest.getClientSecretAlias(), - ISSUER_ID, manifest.getParticipantId(), - PARTICIPANT_ID, manifest.getParticipantId()); - - var sensitiveConfig = Map.of(VAULT_CONFIG, toJson(manifest.getVaultConfig())); - - transactionContext.execute(() -> { - - participantContextStore.createParticipantContext(participantContext) - .orElseThrow(OnboardingException::new); - - var config = ParticipantContextConfiguration.Builder.newInstance() - .participantContextId(participantContextId) - .entries(participantConfig) - .privateEntries(sensitiveConfig) - .build(); - configService.save(config) - .orElseThrow(OnboardingException::new); - - vault.storeSecret(manifest.getParticipantContextId(), manifest.getClientSecretAlias(), manifest.getClientSecret()) - .orElseThrow(f -> new OnboardingException(new ServiceFailure(f.getMessages(), ServiceFailure.Reason.UNEXPECTED))); - - dataPlaneSelectorService.addInstance(DataPlaneInstance.Builder.newInstance() - .participantContextId(manifest.getParticipantContextId()) - .allowedSourceType("HttpData") - .allowedTransferType("HttpData-PULL") - .url("http://dataplane.edc-v.svc.cluster.local:8083/api/control/v1/dataflows") - .build()) - .orElseThrow(OnboardingException::new); - - }); - - } - - private String toJson(Object obj) { - try { - return objectMapper.writeValueAsString(obj); - } catch (JsonProcessingException e) { - throw new EdcException(e); - } - } -} diff --git a/k8s/apps/cfm-agents.yaml b/k8s/apps/cfm-agents.yaml new file mode 100644 index 0000000..80bf73d --- /dev/null +++ b/k8s/apps/cfm-agents.yaml @@ -0,0 +1,77 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cfm-agents + namespace: edc-v + labels: + app: cfm-agents + platform: edcv + type: edcv-app +spec: + replicas: 1 + selector: + matchLabels: + app: cfm-agents + template: + metadata: + labels: + app: cfm-agents + platform: edcv + type: edcv-app + spec: + containers: + - name: keycloak-agent + image: ghcr.io/metaform/connector-fabric-manager/kcagent:latest + imagePullPolicy: Always + volumeMounts: + - name: keycloak-agent-config + mountPath: /etc/appname + readOnly: true + - name: edcv-agent + image: ghcr.io/metaform/connector-fabric-manager/edcvagent:latest + imagePullPolicy: Always + volumeMounts: + - name: edcv-agent-config + mountPath: /etc/appname + readOnly: true + - name: registration-agent + image: ghcr.io/metaform/connector-fabric-manager/regagent:latest + imagePullPolicy: Always + volumeMounts: + - name: registration-agent-config + mountPath: /etc/appname + readOnly: true + - name: onboarding-agent + image: ghcr.io/metaform/connector-fabric-manager/obagent:latest + imagePullPolicy: Always + volumeMounts: + - name: onboarding-agent-config + mountPath: /etc/appname + readOnly: true + volumes: + - name: keycloak-agent-config + configMap: + name: keycloak-agent-config + - name: edcv-agent-config + configMap: + name: edcv-agent-config + - name: registration-agent-config + configMap: + name: reg-agent-config + - name: onboarding-agent-config + configMap: + name: ob-agent-config + restartPolicy: Always diff --git a/k8s/apps/controlplane-config.yaml b/k8s/apps/controlplane-config.yaml index 725b990..5a5c189 100644 --- a/k8s/apps/controlplane-config.yaml +++ b/k8s/apps/controlplane-config.yaml @@ -60,7 +60,7 @@ data: # to do this properly, we should probably configure the following properties on the ingress route: # proxy_set_header Host $host; # proxy_set_header X-Forwarded-Proto $scheme; - edc.iam.oauth2.issuer: "http://keycloak.localhost/realms/edcv" + edc.iam.oauth2.issuer: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv" edc.iam.oauth2.jwks.url: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/certs" # Default scopes config diff --git a/k8s/apps/controlplane.yaml b/k8s/apps/controlplane.yaml index 98b2115..7537a17 100644 --- a/k8s/apps/controlplane.yaml +++ b/k8s/apps/controlplane.yaml @@ -35,7 +35,7 @@ spec: containers: - name: controlplane image: ghcr.io/metaform/jad/controlplane:latest - imagePullPolicy: Always + imagePullPolicy: Never envFrom: - configMapRef: { name: controlplane-config } ports: diff --git a/k8s/apps/edcv-agent-config.yaml b/k8s/apps/edcv-agent-config.yaml new file mode 100644 index 0000000..4dff487 --- /dev/null +++ b/k8s/apps/edcv-agent-config.yaml @@ -0,0 +1,41 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: edcv-agent-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + edcvagent.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable + vault.url: http://vault.edc-v.svc.cluster.local:8200 + vault.path: secret + vault.clientId: provisioner + vault.clientSecret: provisioner-secret + vault.tokenUrl: http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token + vault.softDelete: true + keycloak.clientID: provisioner + keycloak.clientSecret: provisioner-secret + keycloak.tokenUrl: http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token + identityhub.url: http://identityhub.edc-v.svc.cluster.local:7081/api/identity + controlplane.url: http://controlplane.edc-v.svc.cluster.local:8081/api/mgmt \ No newline at end of file diff --git a/k8s/apps/identityhub-config.yaml b/k8s/apps/identityhub-config.yaml index e6b3059..41040a6 100644 --- a/k8s/apps/identityhub-config.yaml +++ b/k8s/apps/identityhub-config.yaml @@ -50,5 +50,5 @@ data: # to do this properly, we should probably configure the following properties on the ingress route: # proxy_set_header Host $host; # proxy_set_header X-Forwarded-Proto $scheme; - edc.iam.oauth2.issuer: "http://keycloak.localhost/realms/edcv" + edc.iam.oauth2.issuer: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv" edc.iam.oauth2.jwks.url: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/certs" \ No newline at end of file diff --git a/k8s/apps/issuerservice-config.yaml b/k8s/apps/issuerservice-config.yaml index 0ed16f1..ae54d3c 100644 --- a/k8s/apps/issuerservice-config.yaml +++ b/k8s/apps/issuerservice-config.yaml @@ -54,7 +54,7 @@ data: # to do this properly, we should probably configure the following properties on the ingress route: # proxy_set_header Host $host; # proxy_set_header X-Forwarded-Proto $scheme; - edc.iam.oauth2.issuer: "http://keycloak.localhost/realms/edcv" + edc.iam.oauth2.issuer: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv" edc.iam.oauth2.jwks.url: "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/certs" JAVA_TOOL_OPTIONS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1044" diff --git a/k8s/apps/issuerservice-seed-job.yaml b/k8s/apps/issuerservice-seed-job.yaml new file mode 100644 index 0000000..d08b8b0 --- /dev/null +++ b/k8s/apps/issuerservice-seed-job.yaml @@ -0,0 +1,265 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +## This seed job creates a tenant for the "issuer" in the IssuerService, as well as creates an Attestation Definition and +## Credential Definition. + +apiVersion: batch/v1 +kind: Job +metadata: + name: issuerservice-seed + namespace: edc-v + labels: + app: issuerservice-seed + platform: edcv + type: edcv-job +spec: + backoffLimit: 5 + template: + metadata: + labels: + app: issuerservice-seed + platform: edcv + type: edcv-job + spec: + restartPolicy: OnFailure + initContainers: + # Wait for issuerservice to be ready + - name: wait-for-issuerservice + image: curlimages/curl:latest + command: + - sh + - -c + - | + until curl -sf http://issuerservice.edc-v.svc.cluster.local:10010/api/check/readiness; do + echo "Waiting for issuerservice to be ready..." + sleep 5 + done + echo "" + echo "IssuerService is ready!" + containers: + - name: seed-issuerservice + image: curlimages/curl:latest + env: + - name: BASE_URL + value: "http://localhost" + - name: KC_HOST + value: "http://keycloak.edc-v.svc.cluster.local:8080" + - name: ISSUER_CLIENT_ID + value: "issuer" + - name: ISSUER_CLIENT_SECRET + value: "issuer-secret" + command: + - sh + - -c + - | + set -e + + echo "================================================" + echo "Step 1: Create Vault Access Token (Issuer)" + echo "================================================" + + # Get Keycloak admin token + KC_TOKEN=$(curl -sf -X POST "${KC_HOST}/realms/master/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=password" \ + -d "username=admin" \ + -d "password=admin" \ + -d "client_id=admin-cli" | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p') + + if [ -z "$KC_TOKEN" ]; then + echo "Failed to get Keycloak admin token" + exit 1 + fi + + # Create Keycloak client for Vault access + curl -sf -X POST "${KC_HOST}/admin/realms/edcv/clients" \ + -H "Authorization: Bearer ${KC_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "clientId": "'"${ISSUER_CLIENT_ID}"'", + "name": "Issuer Client", + "description": "Client for Vault Access (Issuer)", + "enabled": true, + "secret": "'"${ISSUER_CLIENT_SECRET}"'", + "protocol": "openid-connect", + "publicClient": false, + "serviceAccountsEnabled": true, + "standardFlowEnabled": false, + "directAccessGrantsEnabled": false, + "fullScopeAllowed": true, + "protocolMappers": [ + { + "name": "participantContextId", + "protocol": "openid-connect", + "protocolMapper": "oidc-hardcoded-claim-mapper", + "consentRequired": false, + "config": { + "claim.name": "participant_context_id", + "claim.value": "issuer", + "jsonType.label": "String", + "access.token.claim": "true", + "id.token.claim": "true", + "userinfo.token.claim": "true" + } + }, + { + "name": "role", + "protocol": "openid-connect", + "protocolMapper": "oidc-hardcoded-claim-mapper", + "consentRequired": false, + "config": { + "claim.name": "role", + "claim.value": "participant", + "jsonType.label": "String", + "access.token.claim": "true", + "id.token.claim": "true", + "userinfo.token.claim": "true" + } + } + ] + }' + + echo "✓ Vault Access Token created" + + echo "" + echo "================================================" + echo "Step 2: Create Issuer Tenant in IssuerService" + echo "================================================" + + # Get provisioner token + PROVISIONER_TOKEN=$(curl -sf -X POST "${KC_HOST}/realms/edcv/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=provisioner" \ + -d "client_secret=provisioner-secret" \ + -d "scope=issuer-admin-api:write" | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p') + + if [ -z "$PROVISIONER_TOKEN" ]; then + echo "Failed to get provisioner token" + exit 1 + fi + + # Create issuer tenant + TENANT_RESPONSE=$(curl -sf -X POST "http://issuerservice.edc-v.svc.cluster.local:10015/api/identity/v1alpha/participants" \ + -H "Authorization: Bearer ${PROVISIONER_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "roles": ["admin"], + "serviceEndpoints": [ + { + "type": "IssuerService", + "serviceEndpoint": "http://issuerservice.edc-v.svc.cluster.local:10012/api/issuance/v1alpha/participants/aXNzdWVy", + "id": "issuer-service-1" + } + ], + "active": true, + "participantContextId": "issuer", + "did": "did:web:issuerservice.edc-v.svc.cluster.local%3A10016:issuer", + "key": { + "keyId": "did:web:issuerservice.edc-v.svc.cluster.local%3A10016:issuer#key-1", + "privateKeyAlias": "did:web:issuerservice.edc-v.svc.cluster.local%3A10016:issuer#key-1", + "keyGeneratorParams": { + "algorithm": "EdDSA" + } + }, + "additionalProperties": { + "edc.vault.hashicorp.config": { + "credentials": { + "clientId": "'"${ISSUER_CLIENT_ID}"'", + "clientSecret": "'"${ISSUER_CLIENT_SECRET}"'", + "tokenUrl": "http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token" + }, + "config": { + "secretPath": "v1/participants", + "folderPath": "'"${ISSUER_CLIENT_ID}"'", + "vaultUrl": "http://vault.edc-v.svc.cluster.local:8200" + } + } + } + }') + + echo "✓ Issuer tenant created" + + echo "" + echo "================================================" + echo "Step 3: Create AttestationDefinition" + echo "================================================" + + # Get issuer token + ISSUER_TOKEN=$(curl -sf -X POST "${KC_HOST}/realms/edcv/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=issuer" \ + -d "client_secret=issuer-secret" \ + -d "scope=issuer-admin-api:write" | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p') + + if [ -z "$ISSUER_TOKEN" ]; then + echo "Failed to get issuer token" + exit 1 + fi + + # Create attestation definition + curl -sf -X POST "http://issuerservice.edc-v.svc.cluster.local:10013/api/admin/v1alpha/participants/aXNzdWVy/attestations" \ + -H "Authorization: Bearer ${ISSUER_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "attestationType": "membership", + "configuration": {}, + "id": "membership-attestation-def-1" + }' + + echo "✓ AttestationDefinition created" + + echo "" + echo "================================================" + echo "Step 4: Create CredentialDefinition" + echo "================================================" + + # Create credential definition + curl -sf -X POST "http://issuerservice.edc-v.svc.cluster.local:10013/api/admin/v1alpha/participants/aXNzdWVy/credentialdefinitions" \ + -H "Authorization: Bearer ${ISSUER_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "attestations": ["membership-attestation-def-1"], + "credentialType": "MembershipCredential", + "id": "membership-credential-def", + "jsonSchema": "{}", + "jsonSchemaUrl": "https://example.com/schema/membership-credential.json", + "mappings": [ + { + "input": "membership", + "output": "credentialSubject.membership", + "required": true + }, + { + "input": "membershipType", + "output": "credentialSubject.membershipType", + "required": "true" + }, + { + "input": "membershipStartDate", + "output": "credentialSubject.membershipStartDate", + "required": true + } + ], + "rules": [], + "format": "VC1_0_JWT", + "validity": "604800" + }' + + echo "✓ CredentialDefinition created" + echo "" + echo "================================================" + echo "IssuerService seeding completed successfully!" + echo "================================================" diff --git a/k8s/apps/keycloak-agent-config.yaml b/k8s/apps/keycloak-agent-config.yaml new file mode 100644 index 0000000..2f7243d --- /dev/null +++ b/k8s/apps/keycloak-agent-config.yaml @@ -0,0 +1,40 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: keycloak-agent-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + kcagent.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable + vault.url: http://vault.edc-v.svc.cluster.local:8200 + vault.path: secret + vault.clientId: provisioner + vault.clientSecret: provisioner-secret + vault.tokenUrl: http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token + keycloak.url: http://keycloak.edc-v.svc.cluster.local:8080 + keycloak.realm: edcv + keycloak.username: admin + keycloak.password: admin + keycloak.clientId: admin-cli \ No newline at end of file diff --git a/k8s/apps/onboarding-agent-config.yaml b/k8s/apps/onboarding-agent-config.yaml new file mode 100644 index 0000000..7195608 --- /dev/null +++ b/k8s/apps/onboarding-agent-config.yaml @@ -0,0 +1,34 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: ob-agent-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + obagent.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable + keycloak.tokenUrl: http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token + keycloak.clientId: admin + keycloak.clientSecret: edc-v-admin-secret + identityhub.url: http://identityhub.edc-v.svc.cluster.local:7081/api/identity \ No newline at end of file diff --git a/k8s/apps/participant-manager-config.yaml b/k8s/apps/participant-manager-config.yaml new file mode 100644 index 0000000..7f60a25 --- /dev/null +++ b/k8s/apps/participant-manager-config.yaml @@ -0,0 +1,30 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: pm-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + pm.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable \ No newline at end of file diff --git a/k8s/apps/participant-manager-seed-job.yaml b/k8s/apps/participant-manager-seed-job.yaml new file mode 100644 index 0000000..5490450 --- /dev/null +++ b/k8s/apps/participant-manager-seed-job.yaml @@ -0,0 +1,200 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +apiVersion: batch/v1 +kind: Job +metadata: + name: participant-manager-seed + namespace: edc-v + labels: + app: participant-manager-seed + platform: edcv + type: edcv-job +spec: + backoffLimit: 5 + template: + metadata: + labels: + app: participant-manager-seed + platform: edcv + type: edcv-job + spec: + restartPolicy: OnFailure + initContainers: + # Wait for participant-manager to be ready + - name: wait-for-participant-manager + image: curlimages/curl:latest + command: + - sh + - -c + - | + until curl -sf http://participant-manager.edc-v.svc.cluster.local:8080/api/v1alpha1/activity-definitions; do + echo "Waiting for participant-manager to be ready..." + sleep 5 + done + echo "Participant Manager is ready!" + containers: + - name: seed-participant-manager + image: curlimages/curl:latest + env: + - name: PM_BASE_URL + value: "http://participant-manager.edc-v.svc.cluster.local:8080" + - name: TM_BASE_URL + value: "http://tenant-manager.edc-v.svc.cluster.local:8080" + command: + - sh + - -c + - | + set -e + + echo "================================================" + echo "ParticipantManager & TenantManager Seeding" + echo "================================================" + + echo "" + echo "Step 1: Create network-activity ActivityDefinition" + echo "---------------------------------------------------" + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/activity-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "description": "Provisions DNS entries", + "inputSchema": {}, + "outputSchema": {}, + "type": "network-activity" + }' + + echo "✓ network-activity created" + + echo "" + echo "Step 2: Create edcv-activity ActivityDefinition" + echo "------------------------------------------------" + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/activity-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "description": "Provisions EDC-V Control plane entries", + "inputSchema": {}, + "outputSchema": {}, + "type": "edcv-activity" + }' + + echo "✓ edcv-activity created" + + echo "" + echo "Step 3: Create registration-activity ActivityDefinition" + echo "--------------------------------------------------------" + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/activity-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "description": "Creates Holder entries on the IssuerService", + "inputSchema": {}, + "outputSchema": {}, + "type": "registration-activity" + }' + + echo "✓ registration-activity created" + + echo "" + echo "Step 4: Create keycloak-activity ActivityDefinition" + echo "----------------------------------------------------" + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/activity-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "description": "Provisions Keycloak clients", + "inputSchema": {}, + "outputSchema": {}, + "type": "keycloak-activity" + }' + + echo "✓ keycloak-activity created" + + echo "" + echo "Step 5: Create onboarding-activity ActivityDefinition" + echo "------------------------------------------------------" + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/activity-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "description": "Onboards participants (= requests credentials)", + "inputSchema": {}, + "outputSchema": {}, + "type": "onboarding-activity" + }' + + echo "✓ onboarding-activity created" + + echo "" + echo "Step 6: Create Orchestration Definition (deploy)" + echo "------------------------------------------------" + + DEPLOY_ORCH_ID=$(cat /proc/sys/kernel/random/uuid) + + curl -sf -X POST "${PM_BASE_URL}/api/v1alpha1/orchestration-definitions" \ + -H "Content-Type: application/json" \ + -d '{ + "activities": [ + { + "dependsOn": [], + "discriminator": "deploy", + "inputs": [], + "type": "network-activity", + "id": "dns-provisioner" + }, + { + "id": "kc-client-provisioner", + "type": "keycloak-activity", + "discriminator": "deploy", + "dependsOn": [] + }, + { + "id": "holder-entry-creator", + "type": "registration-activity", + "discriminator": "deploy", + "dependsOn": [ + "kc-client-provisioner" + ] + }, + { + "id": "connector-provisioner", + "type": "edcv-activity", + "discriminator": "deploy", + "dependsOn": [ + "kc-client-provisioner" + ] + }, + { + "id": "onboarder", + "type": "onboarding-activity", + "discriminator": "deploy", + "dependsOn": [ + "connector-provisioner", + "holder-entry-creator" + ] + } + ], + "description": "Orchestrates the deployment of a new dataspace member", + "schema": {}, + "type": "cfm.orchestration.vpa.deploy", + "id": "'"${DEPLOY_ORCH_ID}"'" + }' + + echo "✓ Deploy orchestration definition created (ID: ${DEPLOY_ORCH_ID})" + + + echo "" + echo "================================================" + echo "ParticipantManager seeding completed!" + echo "================================================" diff --git a/k8s/apps/participant-manager.yaml b/k8s/apps/participant-manager.yaml new file mode 100644 index 0000000..b5f2eef --- /dev/null +++ b/k8s/apps/participant-manager.yaml @@ -0,0 +1,82 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cfm-participant-manager + namespace: edc-v + labels: + app: participant-manager + platform: edcv + type: edcv-app +spec: + replicas: 1 + selector: + matchLabels: + app: participant-manager + template: + metadata: + labels: + app: participant-manager + platform: edcv + type: edcv-app + spec: + containers: + - name: participant-manager + image: ghcr.io/metaform/connector-fabric-manager/pmanager:latest + imagePullPolicy: Always + ports: + - containerPort: 8080 + volumeMounts: + - name: config + mountPath: /etc/appname + readOnly: true + volumes: + - name: config + configMap: + name: pm-config + restartPolicy: Always + +--- +apiVersion: v1 +kind: Service +metadata: + name: participant-manager + namespace: edc-v +spec: + type: NodePort + selector: + app: participant-manager + ports: + - port: 8080 + targetPort: 8080 + +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: participant-manager + namespace: edc-v +spec: + rules: + - host: pm.localhost + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: participant-manager + port: + number: 8080 diff --git a/k8s/apps/registration-agent-config.yaml b/k8s/apps/registration-agent-config.yaml new file mode 100644 index 0000000..06e471d --- /dev/null +++ b/k8s/apps/registration-agent-config.yaml @@ -0,0 +1,35 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: reg-agent-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + regagent.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable + keycloak.tokenUrl: http://keycloak.edc-v.svc.cluster.local:8080/realms/edcv/protocol/openid-connect/token + keycloak.clientId: provisioner + keycloak.clientSecret: provisioner-secret + issuerservice.url: http://issuerservice.edc-v.svc.cluster.local:10013/api/admin + issuer.id: aXNzdWVy # base64 encoded "issuer" \ No newline at end of file diff --git a/k8s/apps/tenant-manager-config.yaml b/k8s/apps/tenant-manager-config.yaml new file mode 100644 index 0000000..843df79 --- /dev/null +++ b/k8s/apps/tenant-manager-config.yaml @@ -0,0 +1,30 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config + namespace: edc-v + +data: + # the file must be called "tm", and the extension must be one of + # "json", "toml", "yaml", "yml", "properties", "props", "prop", "hcl", "tfvars", "dotenv", "env", "ini" + tm.env: | + uri: nats://nats.edc-v.svc.cluster.local:4222 + bucket: cfm-bucket + stream: cfm-stream + httpport: 8080 + postgres: true + dsn: postgres://cfm:cfm@postgres.edc-v.svc.cluster.local:5432/cfm?sslmode=disable \ No newline at end of file diff --git a/k8s/apps/tenant-manager.yaml b/k8s/apps/tenant-manager.yaml new file mode 100644 index 0000000..216dc37 --- /dev/null +++ b/k8s/apps/tenant-manager.yaml @@ -0,0 +1,82 @@ +# +# Copyright (c) 2025 Metaform Systems, Inc. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License, Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# +# Contributors: +# Metaform Systems, Inc. - initial API and implementation +# + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cfm-tenant-manager + namespace: edc-v + labels: + app: tenant-manager + platform: edcv + type: edcv-app +spec: + replicas: 1 + selector: + matchLabels: + app: tenant-manager + template: + metadata: + labels: + app: tenant-manager + platform: edcv + type: edcv-app + spec: + containers: + - name: tenant-manager + image: ghcr.io/metaform/connector-fabric-manager/tmanager:latest + imagePullPolicy: Always + ports: + - containerPort: 8080 + volumeMounts: + - name: config + mountPath: /etc/appname + readOnly: true + volumes: + - name: config + configMap: + name: tm-config + restartPolicy: Always + +--- +apiVersion: v1 +kind: Service +metadata: + name: tenant-manager + namespace: edc-v +spec: + type: NodePort + selector: + app: tenant-manager + ports: + - port: 8080 + targetPort: 8080 + +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: tenant-manager + namespace: edc-v +spec: + rules: + - host: tm.localhost + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: tenant-manager + port: + number: 8080 diff --git a/k8s/base/keycloak.yaml b/k8s/base/keycloak.yaml index 1b0f14e..b63b947 100644 --- a/k8s/base/keycloak.yaml +++ b/k8s/base/keycloak.yaml @@ -51,8 +51,14 @@ spec: value: "admin" - name: KC_HTTP_ENABLED value: "true" + - name: KC_HOSTNAME + value: http://keycloak.edc-v.svc.cluster.local:8080 - name: KC_HOSTNAME_STRICT value: "false" + - name: KC_HOSTNAME_URL + value: "http://keycloak.edc-v.svc.cluster.local:8080" + - name: KC_PROXY + value: "edge" - name: KC_HEALTH_ENABLED value: "true" - name: KC_METRICS_ENABLED @@ -212,6 +218,14 @@ data: "userinfo.token.claim": "true" } } + ], + "defaultClientScopes": [ + "issuer-admin-api:write", + "issuer-admin-api:read", + "identity-api:write", + "identity-api:read", + "management-api:write", + "management-api:read" ] }, { diff --git a/k8s/base/postgres.yaml b/k8s/base/postgres.yaml index 6a8fae8..edd2e9e 100644 --- a/k8s/base/postgres.yaml +++ b/k8s/base/postgres.yaml @@ -32,24 +32,24 @@ spec: type: edcv-infra spec: containers: - - name: postgres - image: ghcr.io/metaform/jad/postgres:wal2json - ports: - - containerPort: 5432 - env: - - name: POSTGRES_DB - value: "controlplane" - - name: POSTGRES_USER - value: "cp" - - name: POSTGRES_PASSWORD - value: "cp" - volumeMounts: - - name: init-script - mountPath: /docker-entrypoint-initdb.d + - name: postgres + image: ghcr.io/metaform/jad/postgres:wal2json + ports: + - containerPort: 5432 + env: + - name: POSTGRES_DB + value: "controlplane" + - name: POSTGRES_USER + value: "cp" + - name: POSTGRES_PASSWORD + value: "cp" + volumeMounts: + - name: init-script + mountPath: /docker-entrypoint-initdb.d volumes: - - name: init-script - configMap: - name: postgres-init + - name: init-script + configMap: + name: postgres-init --- apiVersion: v1 kind: Service @@ -60,8 +60,8 @@ spec: selector: app: postgres ports: - - port: 5432 - targetPort: 5432 + - port: 5432 + targetPort: 5432 --- --- @@ -95,7 +95,7 @@ data: GRANT ALL PRIVILEGES ON DATABASE identityhub TO ih; \c identityhub GRANT ALL ON SCHEMA public TO ih; - + CREATE DATABASE issuerservice; CREATE USER issuer WITH PASSWORD 'issuer'; GRANT ALL PRIVILEGES ON DATABASE issuerservice TO issuer; @@ -112,4 +112,10 @@ data: CREATE USER kc WITH PASSWORD 'kc'; GRANT ALL PRIVILEGES ON DATABASE keycloak TO kc; \c keycloak - GRANT ALL ON SCHEMA public TO kc; \ No newline at end of file + GRANT ALL ON SCHEMA public TO kc; + + CREATE DATABASE cfm; + CREATE USER cfm WITH PASSWORD 'cfm'; + GRANT ALL PRIVILEGES ON DATABASE cfm TO cfm; + \c cfm + GRANT ALL ON SCHEMA public TO cfm; \ No newline at end of file diff --git a/k8s/base/vault.yaml b/k8s/base/vault.yaml index f8ed9d0..696a20c 100644 --- a/k8s/base/vault.yaml +++ b/k8s/base/vault.yaml @@ -96,7 +96,7 @@ spec: done echo "Keycloak is ready!" - + # Enable JWT auth method vault auth enable jwt || true @@ -141,10 +141,35 @@ spec: "clock_skew_leeway": 60 } EOF - + + + # Allow full CRUD + list for provisioner + cat <