diff --git a/README.md b/README.md index 624772557..867bb388e 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ The rest of this README is structured as follows: - [Using the REST API](#using-the-rest-api) - [Using the Java API](#using-the-java-api) - [Performance benchmark](#performance-benchmark) +- [Current development](#current-development) - [Developer information](#developer-information) - [Components](#components) - [Release steps](#release-steps) @@ -167,7 +168,7 @@ In the [`Java API Example` module](./examples/java-api), the Java API is used to Another knowledge base receives those bindings through a REACT knowledge interaction an prints them to the console. # Performance benchmark -A preliminary performence benchmark of the Knowledge Engine is available in [this repository](https://github.com/faclc4/YCSB-KE/tree/master). +A preliminary performance benchmark of the Knowledge Engine is available in [this repository](https://github.com/faclc4/YCSB-KE/tree/master). Based on this benchmark, the minimum requirements for the Knowledge Engine are: - i3-3040 Intel processor at 3.4 GHz @@ -177,6 +178,11 @@ Based on this benchmark, the minimum requirements for the Knowledge Engine are: Of course, it highly depends on how you want to use the Knowledge Engine, because the reasoner, for example, increases these requirements. +# Current development +The Knowledge Engine currently still undergoes further development. +An integration with [Eclipse Dataspace Components (EDC)](https://eclipse-edc.github.io/documentation/) is among the features in development, which is not stable in its current form and cannot be guaranteed to not crash frequently and contain bugs. It will be subject to many changes in the future. However, an example on the intended (future) use of this functionality is detailed in the [example](./examples/edc-example/README.md) with a description of the setup and instructions on parameters to set and how to execute it. + + # Developer information This section gives more detailed information about the project's structure, and is targeted towards developers who contribute code to the project. diff --git a/examples/edc-example/README.md b/examples/edc-example/README.md new file mode 100644 index 000000000..a3b0006a9 --- /dev/null +++ b/examples/edc-example/README.md @@ -0,0 +1,90 @@ +# TKE-EDC Example +This example uses EDC-IDS Connectors for communication between two Knowledge Engine Runtimes (KERs). +All messages that are sent contain an authentication code. +If a message is received, the authentication code is validated unless it is a meta Knowledge Interaction. + +## Introduction to the Knowledge Engine +The Knowledge Engine is a system for seamlessly connecting data sources. +Each data source, which can provide and/or consume information, is also called a Knowledge Base. +Multiple Knowledge Bases that communicate together form a network. +To communicate with such a network, each Knowledge Base uses a component called a Smart Connector. +This Smart Connector takes care of all communication between a Knowledge Base and others in the network. +The Smart Connectors are part of the Knowledge Engine solution and should not be confused with the EDC-IDS Connectors. + + +![A single Knowledge Base communicates with a network through a Smart Connector.](./single-kb.png) + +Within a network, each Knowledge Base announces what information it wants to _receive_, and what information it can _provide_ through its Smart Connector. +The Knowledge Engine will determine who to contact for each information request. + +There are 4 types of information requests, also called Knowledge Interactions, in the Knowledge Engine: Ask, Answer, Post, React. +Ask is to request information. An answer provides an Answer to a request for information, i.e. Ask. +Post is to announce information. +A React gives you the ability to subscribe to information and thus react to information that is announced through a Post. + +These Knowledge Interactions are first registered at the Smart Connector. +After they have been registered, they can be executed. + +For more information on the Knowledge Engine, check out the [documentation](https://docs.knowledge-engine.eu/). + +## About the Integration with EDC-IDS +The current integration between the Knowledge Engine and EDC-IDS focuses on the authentication of messages. +All messages that are sent contain an authentication code. +This authentication code is validated whenever the message is received. +This way we can be sure that the message was sent by the correct party, thus it establishes trust within the network. + +We currently use the standard EDC-IDS Connector without any modifications. +We use the Connector to establish and check the identity of all parties in the network. +The communication between KERs is still direct, meaning that messages that are sent do not go through the Connector. + +The authentication tokens are valid for a limited amount of time. +You can set the duration of validity of authentication tokens in the EDC Connector properties file (`edc.transfer.proxy.token.validity.seconds`). +While tokens can expire in the current implementation, there is not yet a mechanism to renew them. +That's why we currently advise you to set it to a high number. + + +## Running the TKE-EDC example +This example uses 3 knowledge bases as depicted below. + +![Picture with 3 knowledge bases. Each knowledge base uses a Smart Connector to communicate with the other knowledge bases.](./illustration-example-situation.png) + +One knowledge base asks for information and the other two provide an answer to the question. + +### Executing the example +Execute the following steps to run the example: +1. In this project, execute a `mvn clean install`. +2. In the `knowledge-directory` directory in this project, execute `docker build . -t testkd:1.3.3-SNAPSHOT`. +3. In the `smart-connector-rest-dist` directory in this project, execute `docker build . -t testsc:1.3.3-SNAPSHOT`. +4. In the `examples/edc-example` directory in this project, execute `docker compose build`. +5. In the `examples/edc-example` directory in this project, execute `docker compose up -d tke-edc-one tke-edc-two tke-edc-three`. This starts three EDC-IDS Connectors. +6. Wait around 10 seconds to give the EDC Connectors time to finish setting up. Then, execute `docker compose up -d` to start three KERs, three linked Knowledge Bases and a Knowledge Directory. + +You can inspect the logs with `docker compose logs -f`. +After a moment (+-30 seconds), the logs will stabilise when the connectors have finished initiating the various data flows. +You can then see that one KER (`runtime-1`) asks for information, a second KER (`runtime-2`) answers with `http://example.org/Math, http://example.org/Science` and the third (`runtime-3`) answers with `http://example.org/Magazines, http://example.org/Books`. + +To stop the example, execute `docker compose down`. + +## Adding another participant to the network +For each additional KER with an EDC-IDS Connector, we need the following files in the `examples/edc-example` directory: +- `connector/configuration/ker-configuration.properties` contains settings for the EDC-IDS Connector +- `connector/configuration/ker-vault.properties` contains a public key + +The `docker-compose.yml` in `examples/edc-example/` should also be modified to include: +- An additional KER (currently named `runtime-1`, `runtime-2`, ...) + - The `image` setting refers to the image build in the execution steps of this document. + - The `depends_on` setting refers to the Docker component for the EDC-IDS Connector + - The `KE_RUNTIME_EXPOSED_URL` is a unique URL for the new KER. + - The EDC related environment variables are: + - `KE_RUNTIME_USE_EDC` -> Turn EDC functionality on or off. + - `KE_EDC_PROTOCOL_URL` -> URL of the protocal API of the associated EDC-IDS connector. + - `KE_EDC_MANAGEMENT_URL` -> URL of the management API of the associated EDC-IDS connector. + - `KE_EDC_DATAPLANE_CONTROL_URL` -> URL of the dataplane control API of the associated EDC-IDS connector. + - `KE_EDC_DATAPLANE_PUBLIC_URL` -> URL of the dataplane public API of the associated EDC-IDS connector. + - `KE_EDC_TOKEN_VALIDATION_ENDPOINT` -> URL of the token validation endpoint of the associated EDC-IDS connector. +- An additional EDC-IDS Connector (currently named `tke-edc-one`, `tke-edc-two`, ...) + - Requires 4 ports to be forwarded + - The `command` used to start this connector refers to the previously mentioned configuration files and thus the names of those files should be modified if you copy the command from another EDC-IDS Connector. + - The `hostname` is used in the properties files to refer to this entity +- An additional knowledge base (`kb1`, `kb2`, ...) + - The `KE_URL` refers to the `KE_RUNTIME_EXPOSED_URL` of the KER Docker component (`runtime-1`, `runtime-2`, ...) \ No newline at end of file diff --git a/examples/edc-example/connector/certs/cert.pem b/examples/edc-example/connector/certs/cert.pem new file mode 100644 index 000000000..c7dc26fa7 --- /dev/null +++ b/examples/edc-example/connector/certs/cert.pem @@ -0,0 +1,21 @@ +-----BEGIN CERTIFICATE----- +MIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL +BQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM +GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy +MjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw +HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB +AQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g +E0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS +PbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H +I6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W +EGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0 +h5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud +DgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B +2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn +QHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/ +rySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe +Aqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy ++NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR +IvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/ +g3h+15GuzbsSzOCOEYOT +-----END CERTIFICATE----- diff --git a/examples/edc-example/connector/certs/cert.pfx b/examples/edc-example/connector/certs/cert.pfx new file mode 100644 index 000000000..7ac9c73e0 Binary files /dev/null and b/examples/edc-example/connector/certs/cert.pfx differ diff --git a/examples/edc-example/connector/certs/key.pem b/examples/edc-example/connector/certs/key.pem new file mode 100644 index 000000000..e72229e86 --- /dev/null +++ b/examples/edc-example/connector/certs/key.pem @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDBl6XaJnXTL+6D +Wip3aBhU+MzmY4d1V9hbTm1tiZ3gE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7 +EddidN0ITHB9cQNdAfdUJ5njmsGSPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7 +DHacZT/+OztBH1RwkG2ymM94Hf8HI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvj +X5qASakBtXISKIsOU84N0/2HDN3WEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga7 +75bPXN3M+JTSaIKE7dZbKzvx0Zi0h5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2 +YmnneYoVAgMBAAECggEBAJHXiN6bctAyn+DcoHlsNkhtVw+Jk5bXIutGXjHTJtiU +K//siAGC78IZMyXmi0KndPVCdBwShROVW8xWWIiXuZxy2Zvm872xqX4Ah3JsN7/Q +NrXdVBUDo38zwIGkxqIfIz9crZ4An+J/eq5zaTfRHzCLtswMqjRS2hFeBY5cKrBY +4bkSDGTP/c5cP7xS/UwaiTR2Ptd41f4zTyd4l5rl30TYHpazQNlbdxcOV4jh2Rnp +E0+cFEvEfeagVq7RmfBScKG5pk4qcRG0q2QHMyK5y00hdYvhdRjSgN7xIDkeO5B8 +s8/tSLU78nCl2gA9IKxTXYLitpISwZ81Q04mEAKRRtECgYEA+6lKnhn//aXerkLo +ZOLOjWQZhh005jHdNxX7DZqLpTrrfxc8v15KWUkAK1H0QHqYvfPrbbsBV1MY1xXt +sKmkeu/k8fJQzCIvFN4K2J5W5kMfq9PSw5d3XPeDaQuXUVaxBVp0gzPEPHmkKRbA +AkUqY0oJwA9gMKf8dK+flmLZfbsCgYEAxO4Roj2G46/Oox1GEZGxdLpiMpr9rEdR +JlSZ9kMGfddNLV7sFp6yPXDcyc/AOqeNj7tw1MyoT3Ar454+V0q83EZzCXvs4U6f +jUrfFcoVWIwf9AV/J4KWzMIzfqPIeNwqymZKd6BrZgcXXvAEPWt27mwO4a1GhC4G +oZv0t3lAsm8CgYAQ8C0IhSF4tgBN5Ez19VoHpDQflbmowLRt77nNCZjajyOokyzQ +iI0ig0pSoBp7eITtTAyNfyew8/PZDi3IVTKv35OeQTv08VwP4H4EZGve5aetDf3C +kmBDTpl2qYQOwnH5tUPgTMypcVp+NXzI6lTXB/WuCprjy3qvc96e5ZpT3wKBgQC8 +Xny/k9rTL/eYTwgXBiWYYjBL97VudUlKQOKEjNhIxwkrvQBXIrWbz7lh0Tcu49al +BcaHxru4QLO6pkM7fGHq0fh3ufJ8EZjMrjF1xjdk26Q05o0aXe+hLKHVIRVBhlfo +ArB4fRo+HcpdJXjox0KcDQCvHe+1v9DYBTWvymv4QQKBgBy3YH7hKz35DcXvA2r4 +Kis9a4ycuZqTXockO4rkcIwC6CJp9JbHDIRzig8HYOaRqmZ4a+coqLmddXr2uOF1 +7+iAxxG1KzdT6uFNd+e/j2cdUjnqcSmz49PRtdDswgyYhoDT+W4yVGNQ4VuKg6a3 +Z3pC+KTdoHSKeA2FyAGnSUpD +-----END PRIVATE KEY----- diff --git a/examples/edc-example/connector/configuration/consumer-configuration.properties b/examples/edc-example/connector/configuration/consumer-configuration.properties new file mode 100644 index 000000000..5102b3dc7 --- /dev/null +++ b/examples/edc-example/connector/configuration/consumer-configuration.properties @@ -0,0 +1,25 @@ +edc.participant.id=http://runtime-2:8081 +# communication between the docker containers using the hostname of the container +edc.dsp.callback.address=http://two:29194/protocol +web.http.port=29191 +web.http.path=/api +web.http.management.port=29193 +web.http.management.path=/management +web.http.protocol.port=29194 +web.http.protocol.path=/protocol + +# use: host.docker.internal if the docker container needs access localhost +edc.receiver.http.endpoint=http://runtime-2:8081/token +edc.public.key.alias=public-key +edc.transfer.dataplane.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=29291 +web.http.public.path=/public +web.http.control.port=29192 +web.http.control.path=/control +edc.dataplane.token.validation.endpoint=http://two:29192/control/token + +# authCode token expiration in seconds +edc.transfer.proxy.token.validity.seconds=999999 +#edc.transfer.proxy.token.validity.seconds=120 diff --git a/examples/edc-example/connector/configuration/consumer-vault.properties b/examples/edc-example/connector/configuration/consumer-vault.properties new file mode 100644 index 000000000..6ebdebd56 --- /dev/null +++ b/examples/edc-example/connector/configuration/consumer-vault.properties @@ -0,0 +1 @@ +public-key=-----BEGIN CERTIFICATE-----\r\nMIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL\r\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\r\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy\r\nMjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\r\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\r\nAQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g\r\nE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS\r\nPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H\r\nI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W\r\nEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0\r\nh5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud\r\nDgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B\r\n2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn\r\nQHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/\r\nrySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe\r\nAqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy\r\n+NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR\r\nIvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/\r\ng3h+15GuzbsSzOCOEYOT\r\n-----END CERTIFICATE----- diff --git a/examples/edc-example/connector/configuration/provider-configuration.properties b/examples/edc-example/connector/configuration/provider-configuration.properties new file mode 100644 index 000000000..dfa93ae10 --- /dev/null +++ b/examples/edc-example/connector/configuration/provider-configuration.properties @@ -0,0 +1,25 @@ +edc.participant.id=http://runtime-1:8081 +# communication between the docker containers using the hostname of the container +edc.dsp.callback.address=http://one:19194/protocol +web.http.port=19191 +web.http.path=/api +web.http.management.port=19193 +web.http.management.path=/management +web.http.protocol.port=19194 +web.http.protocol.path=/protocol + +# use: host.docker.internal if the docker container needs access localhost +edc.receiver.http.endpoint=http://runtime-1:8081/token +edc.public.key.alias=public-key +edc.transfer.dataplane.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=19291 +web.http.public.path=/public +web.http.control.port=19192 +web.http.control.path=/control +edc.dataplane.token.validation.endpoint=http://one:19192/control/token + +# authCode token expiration in seconds +edc.transfer.proxy.token.validity.seconds=999999 +#edc.transfer.proxy.token.validity.seconds=120 diff --git a/examples/edc-example/connector/configuration/provider-vault.properties b/examples/edc-example/connector/configuration/provider-vault.properties new file mode 100644 index 000000000..6ebdebd56 --- /dev/null +++ b/examples/edc-example/connector/configuration/provider-vault.properties @@ -0,0 +1 @@ +public-key=-----BEGIN CERTIFICATE-----\r\nMIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL\r\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\r\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy\r\nMjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\r\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\r\nAQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g\r\nE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS\r\nPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H\r\nI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W\r\nEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0\r\nh5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud\r\nDgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B\r\n2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn\r\nQHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/\r\nrySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe\r\nAqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy\r\n+NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR\r\nIvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/\r\ng3h+15GuzbsSzOCOEYOT\r\n-----END CERTIFICATE----- diff --git a/examples/edc-example/connector/configuration/three-configuration.properties b/examples/edc-example/connector/configuration/three-configuration.properties new file mode 100644 index 000000000..6b0e231d0 --- /dev/null +++ b/examples/edc-example/connector/configuration/three-configuration.properties @@ -0,0 +1,25 @@ +edc.participant.id=http://runtime-3:8081 +# communication between the docker containers using the hostname of the container +edc.dsp.callback.address=http://three:39194/protocol +web.http.port=39191 +web.http.path=/api +web.http.management.port=39193 +web.http.management.path=/management +web.http.protocol.port=39194 +web.http.protocol.path=/protocol + +# use: host.docker.internal if the docker container needs access localhost +edc.receiver.http.endpoint=http://runtime-3:8081/token +edc.public.key.alias=public-key +edc.transfer.dataplane.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=39291 +web.http.public.path=/public +web.http.control.port=39192 +web.http.control.path=/control +edc.dataplane.token.validation.endpoint=http://three:39192/control/token + +# authCode token expiration in seconds +edc.transfer.proxy.token.validity.seconds=999999 + diff --git a/examples/edc-example/connector/configuration/three-vault.properties b/examples/edc-example/connector/configuration/three-vault.properties new file mode 100644 index 000000000..6ebdebd56 --- /dev/null +++ b/examples/edc-example/connector/configuration/three-vault.properties @@ -0,0 +1 @@ +public-key=-----BEGIN CERTIFICATE-----\r\nMIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL\r\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\r\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy\r\nMjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\r\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\r\nAQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g\r\nE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS\r\nPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H\r\nI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W\r\nEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0\r\nh5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud\r\nDgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B\r\n2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn\r\nQHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/\r\nrySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe\r\nAqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy\r\n+NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR\r\nIvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/\r\ng3h+15GuzbsSzOCOEYOT\r\n-----END CERTIFICATE----- diff --git a/examples/edc-example/connector/libs/connector.jar b/examples/edc-example/connector/libs/connector.jar new file mode 100644 index 000000000..8fbed623b Binary files /dev/null and b/examples/edc-example/connector/libs/connector.jar differ diff --git a/examples/edc-example/docker-compose.yml b/examples/edc-example/docker-compose.yml new file mode 100644 index 000000000..a8bea651b --- /dev/null +++ b/examples/edc-example/docker-compose.yml @@ -0,0 +1,180 @@ +services: + + # This is the knowledge directory, facilitating discovery between different + # runtimes. It exposes its service over port 8282. + knowledge-directory: + image: testkd:1.3.3-SNAPSHOT + networks: + - tke-edc-manager_default + + # These services are seperate Knowledge Engine runtime, which can host + # multiple smart connectors. Note that the REST API port is a DIFFERENT port + # number than the ones configured below. It is still the default 8280. + runtime-1: + image: docker.io/library/testsc:1.3.3-SNAPSHOT + networks: + - tke-edc-manager_default + environment: + KE_RUNTIME_PORT: 8081 # The port that the KE uses to listen for inter-KE-runtime communication. + KE_RUNTIME_EXPOSED_URL: http://runtime-1:8081 # The URL where the runtime is available for inter-runtime communication from the outside. + KD_URL: http://knowledge-directory:8282 + JAVA_TOOL_OPTIONS: "-Dorg.slf4j.simpleLogger.defaultLogLevel=info" + KE_RUNTIME_USE_EDC: true + KE_EDC_PROTOCOL_URL: "http://one:19194/protocol" + KE_EDC_MANAGEMENT_URL: "http://one:19193/management" + KE_EDC_DATAPLANE_CONTROL_URL: "http://one:19192/control/transfer" + KE_EDC_DATAPLANE_PUBLIC_URL: "http://one:19191/public" + KE_EDC_TOKEN_VALIDATION_ENDPOINT: "http://one:19192/control/token" + depends_on: + - tke-edc-one + runtime-2: + image: docker.io/library/testsc:1.3.3-SNAPSHOT + networks: + - tke-edc-manager_default + environment: + KE_RUNTIME_PORT: 8081 + KE_RUNTIME_EXPOSED_URL: http://runtime-2:8081 + KD_URL: http://knowledge-directory:8282 + JAVA_TOOL_OPTIONS: "-Dorg.slf4j.simpleLogger.defaultLogLevel=info" + KE_RUNTIME_USE_EDC: true + KE_EDC_PROTOCOL_URL: "http://two:29194/protocol" + KE_EDC_MANAGEMENT_URL: "http://two:29193/management" + KE_EDC_DATAPLANE_CONTROL_URL: "http://two:29192/control/transfer" + KE_EDC_DATAPLANE_PUBLIC_URL: "http://two:29191/public" + KE_EDC_TOKEN_VALIDATION_ENDPOINT: "http://two:29192/control/token" + depends_on: + - tke-edc-two + runtime-3: + image: docker.io/library/testsc:1.3.3-SNAPSHOT + networks: + - tke-edc-manager_default + environment: + KE_RUNTIME_PORT: 8081 + KE_RUNTIME_EXPOSED_URL: http://runtime-3:8081 + KD_URL: http://knowledge-directory:8282 + JAVA_TOOL_OPTIONS: "-Dorg.slf4j.simpleLogger.defaultLogLevel=info" + KE_RUNTIME_USE_EDC: true + KE_EDC_PROTOCOL_URL: "http://three:39194/protocol" + KE_EDC_MANAGEMENT_URL: "http://three:39193/management" + KE_EDC_DATAPLANE_CONTROL_URL: "http://three:39192/control/transfer" + KE_EDC_DATAPLANE_PUBLIC_URL: "http://three:39191/public" + KE_EDC_TOKEN_VALIDATION_ENDPOINT: "http://three:39192/control/token" + depends_on: + - tke-edc-three + tke-edc-one: + image: openjdk:17 + entrypoint: ["/bin/sh","-c"] + hostname: one + networks: + - tke-edc-manager_default + ports: + - "19191:19191" + - "19192:19192" + - "19193:19193" + - "19194:19194" + command: + - | + java -Dedc.keystore=/edc/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.vault=/edc/configuration/provider-vault.properties -Dedc.fs.config=/edc/configuration/provider-configuration.properties -jar /edc/libs/connector.jar + volumes: + - ./connector/certs:/edc/certs + - ./connector/libs:/edc/libs + - ./connector/configuration:/edc/configuration + extra_hosts: + - "host.docker.internal:host-gateway" + tke-edc-two: + image: openjdk:17 + entrypoint: [ "/bin/sh","-c" ] + hostname: two + networks: + - tke-edc-manager_default + ports: + - "29191:29191" + - "29192:29192" + - "29193:29193" + - "29194:29194" + command: + - | + java -Dedc.keystore=/edc/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.vault=/edc/configuration/consumer-vault.properties -Dedc.fs.config=/edc/configuration/consumer-configuration.properties -jar /edc/libs/connector.jar + volumes: + - ./connector/certs:/edc/certs + - ./connector/libs:/edc/libs + - ./connector/configuration:/edc/configuration + extra_hosts: + - "host.docker.internal:host-gateway" + tke-edc-three: + image: openjdk:17 + entrypoint: [ "/bin/sh","-c" ] + hostname: three + networks: + - tke-edc-manager_default + ports: + - "39191:39191" + - "39192:39192" + - "39193:39193" + - "39194:39194" + command: + - | + java -Dedc.keystore=/edc/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.vault=/edc/configuration/three-vault.properties -Dedc.fs.config=/edc/configuration/three-configuration.properties -jar /edc/libs/connector.jar + volumes: + - ./connector/certs:/edc/certs + - ./connector/libs:/edc/libs + - ./connector/configuration:/edc/configuration + extra_hosts: + - "host.docker.internal:host-gateway" + # These Knowledge Bases use the different runtimes, and exchange data with eachother. + kb1: # linked to edc consumer + build: ../common/asking_kb + networks: + - tke-edc-manager_default + environment: + KE_URL: http://runtime-1:8280/rest + KB_ID: http://example.org/kb1 + PREFIXES: | + { + "ex": "http://example.org/" + } + GRAPH_PATTERN: | + ?a ex:relatedTo ?b . + kb2: # linked to edc provider + build: ../common/answering_kb + networks: + - tke-edc-manager_default + environment: + KE_URL: http://runtime-2:8280/rest + KB_ID: http://example.org/kb2 + PREFIXES: | + { + "ex": "http://example.org/" + } + GRAPH_PATTERN: | + ?a ex:relatedTo ?b . + KB_DATA: | + [ + { + "a": "", + "b": "" + } + ] + kb3: # linked to edc provider + build: ../common/answering_kb + networks: + - tke-edc-manager_default + environment: + KE_URL: http://runtime-3:8280/rest + KB_ID: http://example.org/kb3 + PREFIXES: | + { + "ex": "http://example.org/" + } + GRAPH_PATTERN: | + ?a ex:relatedTo ?b . + KB_DATA: | + [ + { + "a": "", + "b": "" + } + ] +networks: + tke-edc-manager_default: + external: false diff --git a/examples/edc-example/edc/certs/cert.pem b/examples/edc-example/edc/certs/cert.pem new file mode 100644 index 000000000..c7dc26fa7 --- /dev/null +++ b/examples/edc-example/edc/certs/cert.pem @@ -0,0 +1,21 @@ +-----BEGIN CERTIFICATE----- +MIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL +BQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM +GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy +MjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw +HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB +AQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g +E0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS +PbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H +I6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W +EGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0 +h5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud +DgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B +2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn +QHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/ +rySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe +Aqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy ++NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR +IvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/ +g3h+15GuzbsSzOCOEYOT +-----END CERTIFICATE----- diff --git a/examples/edc-example/edc/certs/cert.pfx b/examples/edc-example/edc/certs/cert.pfx new file mode 100644 index 000000000..7ac9c73e0 Binary files /dev/null and b/examples/edc-example/edc/certs/cert.pfx differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/checksums/checksums.lock b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/checksums/checksums.lock new file mode 100644 index 000000000..cb9b2a8f1 Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/checksums/checksums.lock differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/executionHistory/executionHistory.lock b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/executionHistory/executionHistory.lock new file mode 100644 index 000000000..959e6b66a Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/executionHistory/executionHistory.lock differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileChanges/last-build.bin b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileChanges/last-build.bin new file mode 100644 index 000000000..f76dd238a Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileChanges/last-build.bin differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileHashes/fileHashes.lock b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileHashes/fileHashes.lock new file mode 100644 index 000000000..dd67b2cb8 Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/fileHashes/fileHashes.lock differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/gc.properties b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/8.13/gc.properties new file mode 100644 index 000000000..e69de29bb diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/buildOutputCleanup.lock b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/buildOutputCleanup.lock new file mode 100644 index 000000000..eea4dc9bc Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/buildOutputCleanup.lock differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/cache.properties b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/cache.properties new file mode 100644 index 000000000..87ae10a05 --- /dev/null +++ b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/buildOutputCleanup/cache.properties @@ -0,0 +1,2 @@ +#Mon Jul 07 17:25:01 CEST 2025 +gradle.version=8.13 diff --git a/examples/edc-example/edc/consumer-pull-backend-service/.gradle/vcs-1/gc.properties b/examples/edc-example/edc/consumer-pull-backend-service/.gradle/vcs-1/gc.properties new file mode 100644 index 000000000..e69de29bb diff --git a/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/BackendService.class b/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/BackendService.class new file mode 100644 index 000000000..cf55124ee Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/BackendService.class differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/handler/ReceiverHandler.class b/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/handler/ReceiverHandler.class new file mode 100644 index 000000000..166848b75 Binary files /dev/null and b/examples/edc-example/edc/consumer-pull-backend-service/bin/main/org/eclipse/edc/handler/ReceiverHandler.class differ diff --git a/examples/edc-example/edc/consumer-pull-backend-service/build.gradle.kts b/examples/edc-example/edc/consumer-pull-backend-service/build.gradle.kts new file mode 100644 index 000000000..76eeccf66 --- /dev/null +++ b/examples/edc-example/edc/consumer-pull-backend-service/build.gradle.kts @@ -0,0 +1,23 @@ +/* + * Copyright (c) 2022 Bayerische Motoren Werke Aktiengesellschaft (BMW AG) + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Bayerische Motoren Werke Aktiengesellschaft (BMW AG) - initial API and implementation + * + */ + +plugins { + id("java") +} + +tasks.withType { + manifest { + attributes["Main-Class"] = "org.eclipse.edc.BackendService" + } +} \ No newline at end of file diff --git a/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/BackendService.java b/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/BackendService.java new file mode 100644 index 000000000..e9aabb6be --- /dev/null +++ b/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/BackendService.java @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2022 Bayerische Motoren Werke Aktiengesellschaft (BMW AG) + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Bayerische Motoren Werke Aktiengesellschaft (BMW AG) - initial API and implementation + * + */ + +package org.eclipse.edc; + +import com.sun.net.httpserver.HttpServer; +import org.eclipse.edc.handler.ReceiverHandler; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.util.Optional; + +public class BackendService { + + static final String HTTP_PORT = "server.port"; + + public static void main(String[] args) { + int port = Integer.parseInt(Optional.ofNullable(System.getenv(HTTP_PORT)).orElse("4000")); + var server = createHttpServer(port); + server.createContext("/receiver", new ReceiverHandler()); + server.setExecutor(null); + server.start(); + System.out.println("server started at " + port); + } + + private static HttpServer createHttpServer(int port) { + try { + return HttpServer.create(new InetSocketAddress(port), 0); + } catch (IOException e) { + throw new RuntimeException("Unable to start server at port " + port, e); + } + } +} diff --git a/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/handler/ReceiverHandler.java b/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/handler/ReceiverHandler.java new file mode 100644 index 000000000..4a5499fab --- /dev/null +++ b/examples/edc-example/edc/consumer-pull-backend-service/src/main/java/org/eclipse/edc/handler/ReceiverHandler.java @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2022 Bayerische Motoren Werke Aktiengesellschaft (BMW AG) + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Bayerische Motoren Werke Aktiengesellschaft (BMW AG) - initial API and implementation + * + */ + +package org.eclipse.edc.handler; + +import com.sun.net.httpserver.HttpExchange; +import com.sun.net.httpserver.HttpHandler; + +import java.io.IOException; + +public class ReceiverHandler implements HttpHandler { + + /** + * This method just prints the request body to the console and returns a 200 OK response. + */ + @Override + public void handle(HttpExchange exchange) throws IOException { + System.out.println("Request Body: " + new String(exchange.getRequestBody().readAllBytes())); + exchange.sendResponseHeaders(200, 0); + } +} diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/checksums.lock b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/checksums.lock new file mode 100644 index 000000000..a18f5ea31 Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/checksums.lock differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/md5-checksums.bin b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/md5-checksums.bin new file mode 100644 index 000000000..d415ae925 Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/md5-checksums.bin differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/sha1-checksums.bin b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/sha1-checksums.bin new file mode 100644 index 000000000..10c9e0e4f Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/checksums/sha1-checksums.bin differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileChanges/last-build.bin b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileChanges/last-build.bin new file mode 100644 index 000000000..f76dd238a Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileChanges/last-build.bin differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileHashes/fileHashes.lock b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileHashes/fileHashes.lock new file mode 100644 index 000000000..0dda85384 Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/fileHashes/fileHashes.lock differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/8.13/gc.properties b/examples/edc-example/edc/http-pull-connector/.gradle/8.13/gc.properties new file mode 100644 index 000000000..e69de29bb diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/buildOutputCleanup.lock b/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/buildOutputCleanup.lock new file mode 100644 index 000000000..677c45b1e Binary files /dev/null and b/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/buildOutputCleanup.lock differ diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/cache.properties b/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/cache.properties new file mode 100644 index 000000000..55a8ec893 --- /dev/null +++ b/examples/edc-example/edc/http-pull-connector/.gradle/buildOutputCleanup/cache.properties @@ -0,0 +1,2 @@ +#Mon Jul 07 17:24:30 CEST 2025 +gradle.version=8.13 diff --git a/examples/edc-example/edc/http-pull-connector/.gradle/vcs-1/gc.properties b/examples/edc-example/edc/http-pull-connector/.gradle/vcs-1/gc.properties new file mode 100644 index 000000000..e69de29bb diff --git a/examples/edc-example/edc/http-pull-connector/build.gradle.kts b/examples/edc-example/edc/http-pull-connector/build.gradle.kts new file mode 100644 index 000000000..d75a45f71 --- /dev/null +++ b/examples/edc-example/edc/http-pull-connector/build.gradle.kts @@ -0,0 +1,51 @@ +/* + * Copyright (c) 2022 Bayerische Motoren Werke Aktiengesellschaft (BMW AG) + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Bayerische Motoren Werke Aktiengesellschaft (BMW AG) - initial API and implementation + * + */ + +plugins { + `java-library` + id("application") + id("com.github.johnrengelman.shadow") version "8.1.1" +} + +dependencies { + implementation(libs.edc.control.plane.core) + implementation(libs.edc.dsp) + implementation(libs.edc.configuration.filesystem) + implementation(libs.edc.vault.filesystem) + implementation(libs.edc.iam.mock) + implementation(libs.edc.management.api) + implementation(libs.edc.transfer.data.plane) + implementation(libs.edc.transfer.pull.http.receiver) + + implementation(libs.edc.data.plane.selector.api) + implementation(libs.edc.data.plane.selector.core) + implementation(libs.edc.data.plane.selector.client) + + implementation(libs.edc.data.plane.api) + implementation(libs.edc.data.plane.core) + implementation(libs.edc.data.plane.http) +} + +application { + mainClass.set("$group.boot.system.runtime.BaseRuntime") +} + +var distTar = tasks.getByName("distTar") +var distZip = tasks.getByName("distZip") + +tasks.withType { + mergeServiceFiles() + archiveFileName.set("pull-connector.jar") + dependsOn(distTar, distZip) +} diff --git a/examples/edc-example/edc/http-pull-consumer/consumer-configuration.properties b/examples/edc-example/edc/http-pull-consumer/consumer-configuration.properties new file mode 100644 index 000000000..64a00dc6e --- /dev/null +++ b/examples/edc-example/edc/http-pull-consumer/consumer-configuration.properties @@ -0,0 +1,19 @@ +edc.participant.id=consumer +edc.ids.id=urn:connector:consumer +edc.dsp.callback.address=http://localhost:29194/protocol +web.http.port=29191 +web.http.path=/api +web.http.management.port=29193 +web.http.management.path=/management +web.http.protocol.port=29194 +web.http.protocol.path=/protocol +edc.receiver.http.endpoint=http://localhost:4000/receiver/urn:connector:provider/callback +edc.public.key.alias=public-key +edc.transfer.dataplane.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=29291 +web.http.public.path=/public +web.http.control.port=29192 +web.http.control.path=/control +edc.dataplane.token.validation.endpoint=http://localhost:29192/control/token diff --git a/examples/edc-example/edc/http-pull-consumer/consumer-vault.properties b/examples/edc-example/edc/http-pull-consumer/consumer-vault.properties new file mode 100644 index 000000000..6ebdebd56 --- /dev/null +++ b/examples/edc-example/edc/http-pull-consumer/consumer-vault.properties @@ -0,0 +1 @@ +public-key=-----BEGIN CERTIFICATE-----\r\nMIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL\r\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\r\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy\r\nMjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\r\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\r\nAQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g\r\nE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS\r\nPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H\r\nI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W\r\nEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0\r\nh5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud\r\nDgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B\r\n2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn\r\nQHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/\r\nrySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe\r\nAqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy\r\n+NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR\r\nIvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/\r\ng3h+15GuzbsSzOCOEYOT\r\n-----END CERTIFICATE----- diff --git a/examples/edc-example/edc/http-pull-provider/provider-configuration.properties b/examples/edc-example/edc/http-pull-provider/provider-configuration.properties new file mode 100644 index 000000000..7e1c50015 --- /dev/null +++ b/examples/edc-example/edc/http-pull-provider/provider-configuration.properties @@ -0,0 +1,20 @@ +edc.participant.id=provider +edc.ids.id=urn:connector:provider +edc.dsp.callback.address=http://localhost:19194/protocol +web.http.port=19191 +web.http.path=/api +web.http.management.port=19193 +web.http.management.path=/management +web.http.protocol.port=19194 +web.http.protocol.path=/protocol +# todo change to runtime-1 +edc.receiver.http.endpoint=http://localhost:4000/receiver/urn:connector:provider/callback +edc.public.key.alias=public-key +edc.transfer.dataplane.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.signer.privatekey.alias=1 +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=19291 +web.http.public.path=/public +web.http.control.port=19192 +web.http.control.path=/control +edc.dataplane.token.validation.endpoint=http://localhost:19192/control/token diff --git a/examples/edc-example/edc/http-pull-provider/provider-vault.properties b/examples/edc-example/edc/http-pull-provider/provider-vault.properties new file mode 100644 index 000000000..6ebdebd56 --- /dev/null +++ b/examples/edc-example/edc/http-pull-provider/provider-vault.properties @@ -0,0 +1 @@ +public-key=-----BEGIN CERTIFICATE-----\r\nMIIDazCCAlOgAwIBAgIUZ3/sZXYzW4PjmOXKrZn6WBmUJ+4wDQYJKoZIhvcNAQEL\r\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\r\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjAyMjMxNTA2MDNaFw0zMjAy\r\nMjExNTA2MDNaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\r\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\r\nAQUAA4IBDwAwggEKAoIBAQDBl6XaJnXTL+6DWip3aBhU+MzmY4d1V9hbTm1tiZ3g\r\nE0VbUrvGO3LoYaxpPv6zFmsg3uJv6JxVAde7EddidN0ITHB9cQNdAfdUJ5njmsGS\r\nPbdQuOQTHw0aG7/QvTI/nsvfEE6e0lbV/0e7DHacZT/+OztBH1RwkG2ymM94Hf8H\r\nI6x7q6yfRTAZOqeOMrPCYTcluAgE9NskoPvjX5qASakBtXISKIsOU84N0/2HDN3W\r\nEGMXvoHUQu6vrij6BwiwxKaw1AKwWENKoga775bPXN3M+JTSaIKE7dZbKzvx0Zi0\r\nh5X+bxc3BJi3Z/CsUBCzE+Y0SFetOiYmyl/2YmnneYoVAgMBAAGjUzBRMB0GA1Ud\r\nDgQWBBTvK1wVERwjni4B2vdH7KtEJeVWFzAfBgNVHSMEGDAWgBTvK1wVERwjni4B\r\n2vdH7KtEJeVWFzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBn\r\nQHiPA7OBYukHd9gS7c0HXE+fsWcS3GZeLqcHfQQnV3pte1vTmu9//IVW71wNCJ1/\r\nrySRyODPQoPehxEcyHwupNZSzXK//nPlTdSgjMfFxscvt1YndyQLQYCfyOJMixAe\r\nAqrb14GTFHUUrdor0PyElhkULjkOXUrSIsdBrfWrwLTkelE8NK3tb5ZG8KPzD9Jy\r\n+NwEPPr9d+iHkUkM7EFWw/cl56wka9ryBb97RI7DqbO6/j6OXHMk4GByxKv7DSIR\r\nIvF9/Dw20qytajtaHV0pluFcOBuFc0NfiDvCaQlbTsfjzbc6UmZWbOi9YOJl3VQ/\r\ng3h+15GuzbsSzOCOEYOT\r\n-----END CERTIFICATE----- diff --git a/examples/edc-example/illustration-example-situation.png b/examples/edc-example/illustration-example-situation.png new file mode 100644 index 000000000..023227ae4 Binary files /dev/null and b/examples/edc-example/illustration-example-situation.png differ diff --git a/examples/edc-example/single-kb.png b/examples/edc-example/single-kb.png new file mode 100644 index 000000000..3013a70cd Binary files /dev/null and b/examples/edc-example/single-kb.png differ diff --git a/knowledge-directory/pom.xml b/knowledge-directory/pom.xml index 99e0b8416..7dddd5dae 100644 --- a/knowledge-directory/pom.xml +++ b/knowledge-directory/pom.xml @@ -161,6 +161,9 @@ true java8 jersey3 + + @com.fasterxml.jackson.annotation.JsonInclude(com.fasterxml.jackson.annotation.JsonInclude.Include.NON_NULL) + diff --git a/knowledge-directory/src/main/resources/openapi-kd.yaml b/knowledge-directory/src/main/resources/openapi-kd.yaml index 9e338bdf5..64fa9d956 100644 --- a/knowledge-directory/src/main/resources/openapi-kd.yaml +++ b/knowledge-directory/src/main/resources/openapi-kd.yaml @@ -143,6 +143,9 @@ components: lastRenew: type: string format: date-time + edcConnectorUrl: + type: string + format: uri required: - exposedUrl - protocolVersion diff --git a/smart-connector-api/src/main/java/eu/knowledge/engine/smartconnector/api/SmartConnectorConfig.java b/smart-connector-api/src/main/java/eu/knowledge/engine/smartconnector/api/SmartConnectorConfig.java index 844354d76..17f7f7d9a 100644 --- a/smart-connector-api/src/main/java/eu/knowledge/engine/smartconnector/api/SmartConnectorConfig.java +++ b/smart-connector-api/src/main/java/eu/knowledge/engine/smartconnector/api/SmartConnectorConfig.java @@ -59,6 +59,36 @@ public class SmartConnectorConfig { */ public static final String CONF_KEY_KE_RUNTIME_PORT = "ke.runtime.port"; + /** + * Key to configure if a KER should use the EDC functionality or not. + */ + public static final String CONF_KEY_KE_RUNTIME_USE_EDC = "ke.runtime.use.edc"; + + /** + * Key to configure where a KER can reach the protocol API of its own control plane if using EDC. + */ + public static final String CONF_KEY_KE_EDC_PROTOCOL_URL = "ke.edc.protocol.url"; + + /** + * Key to configure where a KER can reach the management API of its own control plane if using EDC. + */ + public static final String CONF_KEY_KE_EDC_MANAGEMENT_URL = "ke.edc.management.url"; + + /** + * Key to configure where a KER can reach its data plane control API if using EDC. + */ + public static final String CONF_KEY_KE_EDC_DATAPLANE_CONTROL_URL = "ke.edc.dataplane.control.url"; + + /** + * Key to configure where a KER can reach its data plane public API if using EDC. + */ + public static final String CONF_KEY_KE_EDC_DATAPLANE_PUBLIC_URL = "ke.edc.dataplane.public.url"; + + /** + * Key to configure the URL where a KER can do token validation through the control plane if using EDC. + */ + public static final String CONF_KEY_KE_EDC_TOKEN_VALIDATION_ENDPOINT = "ke.edc.token.validation.endpoint"; + /** * Key to configure the default reasoner level (1-5) that is used in the current * KER when no reasoner level is provided by the user. The meaning of each diff --git a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerReactWithGapsEnabled.java b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerReactWithGapsEnabled.java index ef77a5b82..2fb372328 100644 --- a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerReactWithGapsEnabled.java +++ b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerReactWithGapsEnabled.java @@ -64,7 +64,7 @@ public void run() { try { // register the AnswerKB HttpTester registerAnswerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerAnswerKb.expectStatus(200); @@ -123,7 +123,7 @@ public void run() { try { // register the ReactKB HttpTester registerReactKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationReactor\", \"knowledgeBaseName\": \"RelationReactor\", \"knowledgeBaseDescription\": \"A KB that reacts to supply related people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationReactor\", \"knowledgeBaseName\": \"RelationReactor\", \"knowledgeBaseDescription\": \"A KB that reacts to supply related people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerReactKb.expectStatus(200); @@ -178,7 +178,7 @@ public void run() { // register the AskKB HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerKb.expectStatus(200); diff --git a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabled.java b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabled.java index 16541abc5..ad3947a58 100644 --- a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabled.java +++ b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabled.java @@ -55,7 +55,7 @@ public void run() { try { // register the AnswerKB HttpTester registerAnswerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerAnswerKb.expectStatus(200); @@ -103,7 +103,7 @@ public void run() { KBReady.await(); // register the AskKB HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerKb.expectStatus(200); diff --git a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabledNoGaps.java b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabledNoGaps.java index 8ef721fe8..069f3a606 100644 --- a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabledNoGaps.java +++ b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskAnswerWithGapsEnabledNoGaps.java @@ -55,7 +55,7 @@ public void run() { try { // register the AnswerKB HttpTester registerAnswerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationProvider\", \"knowledgeBaseName\": \"RelationProvider\", \"knowledgeBaseDescription\": \"A KB that provides relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerAnswerKb.expectStatus(200); @@ -103,7 +103,7 @@ public void run() { KBReady.await(); // register the AskKB HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerKb.expectStatus(200); diff --git a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsEnabled.java b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsEnabled.java index 14fee24f9..3e15f7658 100644 --- a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsEnabled.java +++ b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsEnabled.java @@ -34,7 +34,7 @@ public void testAskWithGaps() throws IOException { URL url = new URL("http://localhost:" + PORT + "/rest"); HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerKb.expectStatus(200); diff --git a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsNotEnabled.java b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsNotEnabled.java index a681da211..59247aae3 100644 --- a/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsNotEnabled.java +++ b/smart-connector-rest-dist/src/test/java/eu/knowledge/engine/rest/api/TestAskWithGapsNotEnabled.java @@ -35,7 +35,7 @@ public void testAskWithoutGaps() throws IOException { URL url = new URL("http://localhost:" + PORT + "/rest"); HttpTester registerKb = new HttpTester(new URL(url + "/sc"), "POST", - "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerEnabled\" : true}", + "{\"knowledgeBaseId\": \"https://www.tno.nl/example/relationAsker\", \"knowledgeBaseName\": \"RelationAsker\", \"knowledgeBaseDescription\": \"A KB that asks for relations between people\", \"reasonerLevel\" : 2}", Map.of("Content-Type", "application/json", "Accept", "*/*")); registerKb.expectStatus(200); diff --git a/smart-connector/pom.xml b/smart-connector/pom.xml index 5a482f15e..42fb2c714 100644 --- a/smart-connector/pom.xml +++ b/smart-connector/pom.xml @@ -207,6 +207,11 @@ smallrye-config 3.14.0 + + org.awaitility + awaitility + 4.3.0 + diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorClient.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorClient.java new file mode 100644 index 000000000..af8ae96c8 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorClient.java @@ -0,0 +1,313 @@ +package eu.knowledge.engine.smartconnector.edc; + +import org.awaitility.Awaitility; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.Executors; + +import static eu.knowledge.engine.smartconnector.edc.JsonUtil.findByJsonPointerExpression; + +public class EdcConnectorClient { + + private final Logger log = LoggerFactory.getLogger(EdcConnectorClient.class); + private final String edcConnectorManagementUrl; + private final HttpClient httpClient; + + public EdcConnectorClient(String edcConnectorManagementUrl) { + this.edcConnectorManagementUrl = edcConnectorManagementUrl; + var executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); + this.httpClient = HttpClient.newBuilder().executor(executorService).build(); + } + + /** + * Each TKE needs to register a dataplane whenever it wants to expose assets. + * This function registers a data plane for HttpProxy/HttpData. + */ + public String registerDataPlane(String dataPlaneId, String dataPlaneControlUrl, String dataPlanePublicUrl) { + var url = getManagementUrl("/v2/dataplanes"); + log.info("registerDataPlane at: {}", url); + var payload = """ + { + "@context": { + "edc": "https://w3id.org/edc/v0.0.1/ns/" + }, + "@id": "%s", + "url": "%s", + "allowedSourceTypes": [ + "HttpData" + ], + "allowedDestTypes": [ + "HttpProxy", + "HttpData" + ], + "properties": { + "https://w3id.org/edc/v0.0.1/ns/publicApiUrl": "%s" + } + } + """.formatted(dataPlaneId, dataPlaneControlUrl, dataPlanePublicUrl); + HttpResponse response = httpPost(url, payload); + log.info("Register data plane response: {}", response.body()); + return response.body(); + } + + /** + * Register an assert given the provided assetId and tkeUrl. + */ + public String registerAsset(String assetId, String tkeUrl, String tkeAssetName) { + var url = getManagementUrl("/v3/assets"); + log.info("registerAsset at: {}", url); + var payload = """ + { + "@context": { + "@vocab" : "https://w3id.org/edc/v0.0.1/ns/" + }, + "@id": "%s", + "properties": { + "name": "%s", + "contenttype": "application/json" + }, + "dataAddress": { + "type": "HttpData", + "name": "%s", + "baseUrl": "%s", + "proxyPath": "true" + } + } + """.formatted(assetId, tkeAssetName, tkeAssetName, tkeUrl); + HttpResponse response = httpPost(url, payload); + log.info("Register asset response: {}", response.body()); + return response.body(); + } + + /** + * The policy defines permissions which can be applied to an asset. + * + * @return + */ + public String registerPolicy(String policyId) { + var url = getManagementUrl("/v2/policydefinitions"); + log.info("getManagementUrl at: {}", url); + + var payload = """ + { + "@context": { + "edc": "https://w3id.org/edc/v0.0.1/ns/", + "odrl": "http://www.w3.org/ns/odrl/2/" + }, + "@id": "%s", + "policy": { + "@type": "set", + "odrl:permission": [], + "odrl:prohibition": [], + "odrl:obligation": [] + } + } + """.formatted(policyId); + HttpResponse response = httpPost(url, payload); + log.info("Register policy response: {}", response.body()); + return response.body(); + } + + public String registerContractDefinition(String contractDefinitionId, String accessPolicyId, String contractPolicyId) { + var url = getManagementUrl("/v2/contractdefinitions"); + log.info("registerContractDefinition at: {}", url); + var payload = """ + { + "@context": { + "edc": "https://w3id.org/edc/v0.0.1/ns/" + }, + "@id": "%s", + "accessPolicyId": "%s", + "contractPolicyId": "%s", + "assetsSelector": [] + } + """.formatted(contractDefinitionId, accessPolicyId, contractPolicyId); + HttpResponse response = httpPost(url, payload); + log.info("Register contract definition response: {}", response.body()); + return response.body(); + } + + /** + * Catalog requests are sent to ones own connector. + * + * @param counterPartyAddress + * @return + */ + public String catalogRequest(String counterPartyAddress) { + var url = getManagementUrl("/v2/catalog/request"); + log.info("catalogRequest at: {}", url); + var payload = """ + { + "@context": { + "@vocab": "https://w3id.org/edc/v0.0.1/ns/" + }, + "counterPartyAddress": "%s", + "protocol": "dataspace-protocol-http" + } + """.formatted(counterPartyAddress); + HttpResponse postResponse = httpPost(url, payload); + log.info("Catalog request response: {}", postResponse.body()); + return postResponse.body(); + } + + /** + * In order to request any data, a contract gets negotiated, and an agreement is resulting has to be negotiated between providers and consumers. + *

+ * The consumer now needs to initiate a contract negotiation sequence with the provider. That sequence looks as follows: + *

+ * Consumer sends a contract offer to the provider (currently, this has to be equal to the provider's offer!) + * Provider validates the received offer against its own offer + * Provider either sends an agreement or a rejection, depending on the validation result + * In case of successful validation, provider and consumer store the received agreement for later reference + * + * @return + */ + public String negotiateContract(String consumerParticipantId, String providerParticipantId, String counterPartyAddress, String assetId) { + var catalogRequest = catalogRequest(counterPartyAddress); + String catalogOfferIdForAsset = findByJsonPointerExpression(catalogRequest, "/dcat:dataset/odrl:hasPolicy/@id"); + + var url = getManagementUrl("/v2/contractnegotiations"); + log.info("negotiateContract at: {}", url); + var payload = """ + { + "@context": { + "@vocab": "https://w3id.org/edc/v0.0.1/ns/" + }, + "@type": "NegotiationInitiateRequestDto", + "consumerId": "%s", + "connectorId": "%s", + "providerId": "%s", + "counterPartyAddress": "%s", + "protocol": "dataspace-protocol-http", + "policy": { + "@context": "http://www.w3.org/ns/odrl.jsonld", + "@id": "%s", + "@type": "Set", + "permission": [], + "prohibition": [], + "obligation": [], + "target": "%s" + } + } + """.formatted(consumerParticipantId, providerParticipantId, providerParticipantId, counterPartyAddress, catalogOfferIdForAsset, assetId); + + HttpResponse postResponse = httpPost(url, payload); + log.info("Negotiate contract response: {}", postResponse.body()); + return postResponse.body(); + } + + public String contractAgreement(String json) { + String contractAgreementId = findByJsonPointerExpression(json, "/@id"); + var url = getManagementUrl("/v2/contractnegotiations/" + contractAgreementId); + log.info("contractAgreement at: {}", url); + final List responses = new ArrayList<>(); + + Awaitility.await().pollInterval(Duration.ofSeconds(1)).atMost(Duration.ofSeconds(10)).until(() -> { + HttpResponse response = httpGet(url); + responses.add(response.body()); + String state = findByJsonPointerExpression(response.body(), "/state"); + return Objects.equals(state, "FINALIZED"); + }); + + return responses.get(responses.size() - 1); + } + + public String transferProcess(String counterPartyAddress, String counterPartyParticipantId, String contractAgreementId, String assetId) { + var url = getManagementUrl("/v2/transferprocesses"); + log.info("transferProcess at: {}", url); + var payload = """ + { + "@context": { + "@vocab": "https://w3id.org/edc/v0.0.1/ns/" + }, + "@type": "TransferRequestDto", + "counterPartyAddress": "%s", + "connectorId": "%s", + "contractId": "%s", + "assetId": "%s", + "protocol": "dataspace-protocol-http", + "dataDestination": { + "type": "HttpProxy" + } + } + """.formatted(counterPartyAddress, counterPartyParticipantId, contractAgreementId, assetId); + HttpResponse postResponse = httpPost(url, payload); + log.info("Transfer process response: {}", postResponse.body()); + return postResponse.body(); + } + + /** + * Contract the EDC connector URL for the /management endpoint. + */ + private String getManagementUrl(String suffix) { + return edcConnectorManagementUrl + suffix; + } + + private HttpResponse httpGet(String url) { + return httpGet(url, "application/json"); + } + + private HttpResponse httpGet(String url, String accept) { + log.info("Calling: {}", url); + + HttpRequest request = HttpRequest.newBuilder() + .uri(toURI(url)) + .headers("Accept", accept) + .GET() + .build(); + + return sendRequest(request); + } + + private HttpResponse httpPost(String url, String payload) { + return httpPost(url, "application/json", payload); + } + + private HttpResponse httpPost(String url, String contentType, String payload) { + log.info("Calling: {}, Payload: {}", url, payload); + + HttpRequest request = HttpRequest.newBuilder() + .uri(toURI(url)) + .headers("Content-Type", contentType) + .POST(HttpRequest.BodyPublishers.ofString(payload)) + .build(); + + return sendRequest(request); + } + + private HttpResponse sendRequest(HttpRequest request) { + try { + HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); + int statusCode = response.statusCode(); + + if (statusCode >= 200 && statusCode <= 299) { + return response; + } else if (response.statusCode() >= 400 && statusCode <= 499) { + throw new RuntimeException("HttpClient exception, request failed with statusCode: " + statusCode + ", response: " + response.body()); + } else if (response.statusCode() >= 500) { + throw new RuntimeException("HttpServer exception, request failed with statusCode: " + statusCode + ", response: " + response.body()); + } + } catch (Exception e) { + throw new RuntimeException(e.getMessage(), e); + } + return null; + } + + private URI toURI(String url) { + try { + return new URI(url); + } catch (Exception e) { + throw new RuntimeException(e.getMessage(), e); + } + } + +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorProperties.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorProperties.java new file mode 100644 index 000000000..40be19b63 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorProperties.java @@ -0,0 +1,18 @@ +package eu.knowledge.engine.smartconnector.edc; + +public record EdcConnectorProperties(String participantId, String protocolUrl, String managementUrl, String dataPlaneId, + String dataPlaneControlUrl, String dataPlanePublicUrl, String tkeAssetUrl, String tkeAssetName) { + + public EdcConnectorProperties(String participantId, String protocolUrl) { + this( + participantId, + protocolUrl, + "", + "tke-dataplane", + "", + "", + "https://www.knowledge-engine.eu/", + "TNO Knowledge Engine Runtime" + ); + } +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorService.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorService.java new file mode 100644 index 000000000..d4ccc7ac5 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/EdcConnectorService.java @@ -0,0 +1,156 @@ +package eu.knowledge.engine.smartconnector.edc; + +import jakarta.inject.Inject; +import jakarta.inject.Named; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * The EdcConnectorService can manage all the configuration of and + * interactions between the specified EDC connectors in the + * TkeEdcConnectorConfiguration. + */ +@Named +public class EdcConnectorService { + + private final Logger log = LoggerFactory.getLogger(EdcConnectorService.class); + private final Map connectors = new HashMap<>(); + private final Map configuration = new HashMap<>(); + + // these are all static for every connector + public final static String DATA_PLANE_ID = "tke-dataplane"; + public final static String ASSET_NAME = "TNO Knowledge Engine Runtime"; + public final static String ASSET_URL = "https://www.knowledge-engine.eu/"; + + @Inject + public EdcConnectorService(List configuration) { + for (EdcConnectorProperties connector : configuration) { + addConnector(connector); + } + } + + public void addConnector(EdcConnectorProperties connector) { + log.info("Adding connector for [participant id: {}] with [management url: {}]", + connector.participantId(), connector.managementUrl()); + connectors.put(connector.participantId(), new EdcConnectorClient(connector.managementUrl())); + configuration.put(connector.participantId(), connector); + } + + /** + * Configure the connector with the provided connector's participantId + * + * @param participantId connector to configure + * @return map with all the responses + */ + public HashMap configureConnector(String participantId) { + log.info("configureConnector for participantId: {}", participantId); + EdcConnectorProperties properties = configuration.get(participantId); + EdcConnectorClient connector = connectors.get(participantId); + + String existingAssetId = getAssetIdFromCatalogForAssetName(properties.participantId(), + properties.participantId(), properties.tkeAssetName()); + if (existingAssetId != null) { + log.info("Connector already configured and TKE asset present for participantId: {}", participantId); + throw new RuntimeException("Connector already configured and TKE asset present."); + } + + // properties needed when creating a data plane. + var dataPlaneId = EdcConnectorService.DATA_PLANE_ID; + var dataPlaneControlUrl = properties.dataPlaneControlUrl(); + var dataPlanePublicUrl = properties.dataPlanePublicUrl(); + // properties needed when creating an asset. + var assetId = UUID.randomUUID().toString(); // generate an unique assetId + var tkeAssetUrl = EdcConnectorService.ASSET_URL; + var tkeAssetName = EdcConnectorService.ASSET_NAME; + // properties needed when creating a policy and contract definition. + var policyId = UUID.randomUUID().toString(); + var contractId = UUID.randomUUID().toString(); + + // Create the mandatory edc resources. + var map = new HashMap(); + map.put("registerDataPlane", connector.registerDataPlane(dataPlaneId, dataPlaneControlUrl, dataPlanePublicUrl)); + map.put("registerPolicy", connector.registerPolicy(policyId)); + map.put("registerAsset", connector.registerAsset(assetId, tkeAssetUrl, tkeAssetName)); + map.put("registerContractDefinition", connector.registerContractDefinition(contractId, policyId, policyId)); + return map; + } + + public String getAssetIdFromCatalogForAssetName(String participantId, String counterPartyParticipantId, + String assetName) { + String response = catalogRequest(participantId, counterPartyParticipantId); + return JsonUtil.findByJsonPointerExpression(response, "/dcat:dataset/@id"); + } + + /** + * Catalog request is always done from one connector (your own) to another + * party's connector (counterparty). Using a catalog request one can figure out + * what assets are provided by a connector. Asset identifiers can later be used + * to negotiate contracts between parties. + * + * @param participantId from which connector the request should be + * made + * @param counterPartyParticipantId to whom the request should be make + * @return response + */ + public String catalogRequest(String participantId, String counterPartyParticipantId) { + log.info("catalogRequest for participantId: {}", participantId); + EdcConnectorClient connector = connectors.get(participantId); + EdcConnectorProperties counterPartyProperties = configuration.get( + counterPartyParticipantId); + + var counterPartyProtocolUrl = counterPartyProperties.protocolUrl(); + return connector.catalogRequest(counterPartyProtocolUrl); + } + + /** + * Negotiate a contract between two connectors for the provided asset + * identifier. + * + * @param participantId from which connector the request should be + * made + * @param counterPartyParticipantId to whom the request should be make + * @param assetId determines what asset the participant wants + * to use + * @return response + */ + public String negotiateContract(String participantId, String counterPartyParticipantId, String assetId) { + log.info("negotiateContract for participantId: {}, counterPartyParticipantId: {}, assetId: {}", participantId, + counterPartyParticipantId, assetId); + EdcConnectorProperties participantProperties = configuration.get(participantId); + EdcConnectorProperties counterPartyProperties = configuration.get(counterPartyParticipantId); + EdcConnectorClient connector = connectors.get(participantId); + + // note that the counterparty protocol url could also be extract from the + // catalog request + var counterPartyAddress = counterPartyProperties.protocolUrl(); // dsp protocol address of the + // counterparty/provider + var consumerId = participantProperties.participantId(); + var providerId = counterPartyProperties.participantId(); + + // The consumer will negotiate a contract using its own connector, the + // counterPartyAddress + // is the party which we need to negotiate the contract with. + String negotiateContract = connector.negotiateContract(consumerId, providerId, counterPartyAddress, assetId); + return connector.contractAgreement(negotiateContract); + } + + public String transferProcess(String participantId, String counterPartyParticipantId, String contractAgreementId, + String assetId) { + log.info( + "transferProcess for participantId: {}, counterPartyParticipantId: {}, contractAgreementId: {}, assetId: {}", + participantId, counterPartyParticipantId, contractAgreementId, assetId); + EdcConnectorProperties counterPartyProperties = configuration.get(counterPartyParticipantId); + EdcConnectorClient connector = connectors.get(participantId); + + var counterPartyAddress = counterPartyProperties.protocolUrl(); // dsp protocol address of the + // counterparty/provider + var providerId = counterPartyProperties.participantId(); + + return connector.transferProcess(counterPartyAddress, providerId, contractAgreementId, assetId); + } +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/InMemoryTokenManager.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/InMemoryTokenManager.java new file mode 100644 index 000000000..6e8827ba7 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/InMemoryTokenManager.java @@ -0,0 +1,51 @@ +package eu.knowledge.engine.smartconnector.edc; + +import jakarta.annotation.Nullable; +import jakarta.inject.Named; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.stream.Stream; + +/** + * We store token data in memory here. + * For a production grade system it makes sense to store the data in a (persistent) datastore. + */ +@Named +public class InMemoryTokenManager { + + private final List tokens = new ArrayList<>(); + private final List transferProcessInitiations = new ArrayList<>(); + + public void tokenReceived(Token token) { + tokens.add(token); + } + + public List getTokensFor(@Nullable String participantId, @Nullable String counterPartyParticipantId) { + if (participantId == null && counterPartyParticipantId == null) { + return tokens; + } + + Stream transferProcessesStream = transferProcessInitiations.stream() + .filter((it) -> Objects.equals(it.participantId(), participantId)); + + if (counterPartyParticipantId != null) { + transferProcessesStream = transferProcessesStream.filter((it) -> Objects.equals(it.counterPartyParticipantId(), counterPartyParticipantId)); + } + + List transferProcessIds = transferProcessesStream.map(TransferProcess::transferProcessResponseId).toList(); + + // Get all the tokens belonging to the transferProcesses between participantId and counterPartyParticipantId + List list = transferProcessIds.stream() + .flatMap((transferProcessId) -> + tokens.stream().filter((token) -> Objects.equals(transferProcessId, token.id())) + ).toList(); + + return list; + } + + public void transferProcessInitiated(TransferProcess transferProcess) { + transferProcessInitiations.add(transferProcess); + } +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/JsonUtil.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/JsonUtil.java new file mode 100644 index 000000000..0983d55ec --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/JsonUtil.java @@ -0,0 +1,22 @@ +package eu.knowledge.engine.smartconnector.edc; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class JsonUtil { + + private static final ObjectMapper objectMapper = new ObjectMapper().findAndRegisterModules(); + + public static String findByJsonPointerExpression(String json, String jsonPointerExpression) { + try { + if (json == null) return null; + JsonNode jsonNode = objectMapper.readTree(json); + JsonNode id = jsonNode.at(jsonPointerExpression); + + if (id == null) return null; + return id.textValue(); + } catch (Exception e) { + throw new RuntimeException(e.getMessage(), e); + } + } +} diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/Token.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/Token.java new file mode 100644 index 000000000..c801e29f7 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/Token.java @@ -0,0 +1,17 @@ +package eu.knowledge.engine.smartconnector.edc; + +import static eu.knowledge.engine.smartconnector.edc.JsonUtil.findByJsonPointerExpression; + +public record Token(String tokenJson, String id, String contractId, String authKey, String authCode, String endpoint) { + + public Token(String tokenJson) { + this( + tokenJson, + findByJsonPointerExpression(tokenJson, "/id"), + findByJsonPointerExpression(tokenJson, "/contractId"), + findByJsonPointerExpression(tokenJson, "/authKey"), + findByJsonPointerExpression(tokenJson, "/authCode"), + findByJsonPointerExpression(tokenJson, "/endpoint") + ); + } +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/TransferProcess.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/TransferProcess.java new file mode 100644 index 000000000..d56787549 --- /dev/null +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/edc/TransferProcess.java @@ -0,0 +1,16 @@ +package eu.knowledge.engine.smartconnector.edc; + +import static eu.knowledge.engine.smartconnector.edc.JsonUtil.findByJsonPointerExpression; + +public record TransferProcess(String participantId, String counterPartyParticipantId, String contractAgreementId, String responseJson, String transferProcessResponseId) { + + public TransferProcess(String participantId, String counterPartyParticipantId, String contractAgreementId, String responseJson) { + this( + participantId, + counterPartyParticipantId, + contractAgreementId, + contractAgreementId, + findByJsonPointerExpression(responseJson, "/@id") + ); + } +} \ No newline at end of file diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/KeRuntime.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/KeRuntime.java index c57db66fe..d3211aa5f 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/KeRuntime.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/KeRuntime.java @@ -130,7 +130,11 @@ public static MessageDispatcher getMessageDispatcher() { .getConfigValue(SmartConnectorConfig.CONF_KEY_KE_RUNTIME_EXPOSED_URL); URI myExposedUrl = new URI(exposedUrl.getValue()); - messageDispatcher = new MessageDispatcher(myPort, myExposedUrl, new URI(kdUrl.getValue())); + ConfigValue useEdc = config + .getConfigValue(SmartConnectorConfig.CONF_KEY_KE_RUNTIME_USE_EDC); + var myUseEdc = Boolean.parseBoolean(useEdc.getValue()); + + messageDispatcher = new MessageDispatcher(myPort, myExposedUrl, new URI(kdUrl.getValue()), myUseEdc); } } catch (NumberFormatException | URISyntaxException e) { LOG.error("Could not parse configuration properties, cannot start Knowledge Engine", e); diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnection.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnection.java index 8bf8cd724..cc6d8f295 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnection.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnection.java @@ -20,6 +20,7 @@ import org.slf4j.Logger; +import com.fasterxml.jackson.annotation.JsonInclude; import com.fasterxml.jackson.databind.DeserializationFeature; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.SerializationFeature; @@ -51,12 +52,14 @@ public static enum State { private State currentState; private final URI kdUrl; private final URI myExposedUrl; + private final URI myEdcConnectorUrl; private final Object lock = new Object(); private ScheduledFuture scheduledFuture; - public KnowledgeDirectoryConnection(URI kdUrl, URI myExposedUrl) { + public KnowledgeDirectoryConnection(URI kdUrl, URI myExposedUrl, URI myEdcConnectorUrl) { this.myExposedUrl = myExposedUrl; + this.myEdcConnectorUrl = myEdcConnectorUrl; this.currentState = State.UNREGISTERED; var builder = HttpClient.newBuilder(); @@ -84,8 +87,9 @@ protected PasswordAuthentication getPasswordAuthentication() { this.httpClient = builder.build(); this.objectMapper = new ObjectMapper().disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES) - .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS).findAndRegisterModules() - .setDateFormat(new RFC3339DateFormat()); + .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS) + .setSerializationInclusion(JsonInclude.Include.NON_NULL).findAndRegisterModules() + .setDateFormat(new RFC3339DateFormat()); } public void start() { @@ -188,6 +192,7 @@ private void tryRegister() { KnowledgeEngineRuntimeConnectionDetails ker = new KnowledgeEngineRuntimeConnectionDetails(); ker.setExposedUrl(myExposedUrl); ker.setProtocolVersion(PROTOCOL_VERSION); + ker.setEdcConnectorUrl(myEdcConnectorUrl); try { HttpRequest registerRequest = HttpRequest diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/MessageDispatcher.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/MessageDispatcher.java index a0238a332..172cf173b 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/MessageDispatcher.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/MessageDispatcher.java @@ -58,6 +58,7 @@ private static enum State { private final URI myExposedUrl; private final URI kdUrl; + private boolean useEdc = false; private State state; private Server httpServer; @@ -84,11 +85,16 @@ private static enum State { * @param myExposedUrl * @param kdUrl */ - public MessageDispatcher(int myPort, URI myExposedUrl, URI kdUrl) { + public MessageDispatcher(int myPort, URI myExposedUrl, URI kdUrl, boolean useEdc) { this.myPort = myPort; this.myExposedUrl = myExposedUrl; this.kdUrl = kdUrl; this.state = State.NEW; + this.useEdc = useEdc; + } + + public MessageDispatcher(int myPort, URI myExposedUrl, URI kdUrl) { + this(myPort, myExposedUrl, kdUrl, false); } /** @@ -96,7 +102,7 @@ public MessageDispatcher(int myPort, URI myExposedUrl, URI kdUrl) { * external Knowledge Directory. */ public MessageDispatcher() { - this(0, null, null); + this(0, null, null, false); } boolean runsInDistributedMode() { @@ -115,14 +121,16 @@ public void start() throws Exception { localSmartConnectorConnectionsManager.start(); if (runsInDistributedMode()) { - // Start Knowledge Directory Connection Manager - this.knowledgeDirectoryConnectionManager = new KnowledgeDirectoryConnection(kdUrl, myExposedUrl); - this.getKnowledgeDirectoryConnectionManager().start(); - // Start the RemoteSmartConnnectorConnectionsManager - remoteSmartConnectorConnectionsManager = new RemoteKerConnectionManager(this); + remoteSmartConnectorConnectionsManager = new RemoteKerConnectionManager(this, this.myExposedUrl, this.useEdc); getRemoteSmartConnectorConnectionsManager().start(); + URI myEdcConnectorUrl = remoteSmartConnectorConnectionsManager.getEdcConnectorUrl(); + // Start Knowledge Directory Connection Manager + this.knowledgeDirectoryConnectionManager = new KnowledgeDirectoryConnection(kdUrl, myExposedUrl, myEdcConnectorUrl); + + this.getKnowledgeDirectoryConnectionManager().start(); + // Start HTTP Server this.startHttpServer(); } @@ -342,4 +350,8 @@ void notifySmartConnectorsChanged() { } } + protected boolean usesEdc() { + return this.useEdc; + } + } diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnection.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnection.java index 1b1b92033..f7c3b0cdc 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnection.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnection.java @@ -1,7 +1,9 @@ package eu.knowledge.engine.smartconnector.runtime.messaging; import static eu.knowledge.engine.smartconnector.runtime.messaging.Utils.stripUserInfoFromURI; +import static eu.knowledge.engine.smartconnector.edc.JsonUtil.findByJsonPointerExpression; +import java.io.FileInputStream; import java.io.IOException; import java.net.Authenticator; import java.net.PasswordAuthentication; @@ -18,6 +20,7 @@ import java.time.LocalDateTime; import java.util.ArrayList; import java.util.List; +import java.util.Properties; import org.eclipse.microprofile.config.ConfigProvider; import org.slf4j.Logger; @@ -37,6 +40,8 @@ import eu.knowledge.engine.smartconnector.runtime.messaging.inter_ker.api.RFC3339DateFormat; import eu.knowledge.engine.smartconnector.runtime.messaging.inter_ker.model.KnowledgeEngineRuntimeDetails; import eu.knowledge.engine.smartconnector.runtime.messaging.kd.model.KnowledgeEngineRuntimeConnectionDetails; +import eu.knowledge.engine.smartconnector.edc.EdcConnectorService; +import eu.knowledge.engine.smartconnector.edc.InMemoryTokenManager; /** * This class is responsible for sending messages to a single remote Knowledge @@ -47,6 +52,10 @@ public class RemoteKerConnection { public static final Logger LOG = LoggerFactory.getLogger(RemoteKerConnection.class); + private URI myExposedUri = null; + private EdcConnectorService edcService; + private InMemoryTokenManager tokenManager; + private final KnowledgeEngineRuntimeConnectionDetails remoteKerConnectionDetails; private final URI remoteKerUri; private KnowledgeEngineRuntimeDetails remoteKerDetails; @@ -59,10 +68,32 @@ public class RemoteKerConnection { private int errorCounter = 0; private LocalDateTime logStillIgnoringAfter = null; - public RemoteKerConnection(MessageDispatcher dispatcher, - KnowledgeEngineRuntimeConnectionDetails kerConnectionDetails) { + // edc info + /** + * The transfer id of this KERs data transfer with this Remote KER. + */ + private String transferId; + + /** + * The contract agreement id this KERs EDC Connector agreed upon with this + * Remote KERs EDC Connector. + */ + private String contractAgreementId; + + /** + * The authentication token which proves this KER has a valid EDC contract with + * this remote KER. + */ + private String authToken; + private String validationEndpoint; + + public RemoteKerConnection(MessageDispatcher dispatcher, URI myExposedUri, EdcConnectorService edcService, + InMemoryTokenManager tokenManager, KnowledgeEngineRuntimeConnectionDetails kerConnectionDetails) { + this.myExposedUri = myExposedUri; this.dispatcher = dispatcher; this.remoteKerConnectionDetails = kerConnectionDetails; + this.edcService = edcService; + this.tokenManager = tokenManager; var builder = HttpClient.newBuilder(); @@ -94,6 +125,43 @@ protected PasswordAuthentication getPasswordAuthentication() { objectMapper = new ObjectMapper().disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES) .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS).findAndRegisterModules() .setDateFormat(new RFC3339DateFormat()); + + if (!(this.edcService == null || this.tokenManager == null)) { + String file = "./edc.properties"; + Properties properties = new Properties(); + FileInputStream configReader; + try { + configReader = new FileInputStream(file); + properties.load(configReader); + configReader.close(); + } catch (IOException e) { + e.printStackTrace(); + } + + this.validationEndpoint = properties.getProperty("tokenValidationEndpoint"); + setupTransferProcess(); + } + } + + /** + * Make sure we have a valid authToken to communicate with the remote KER. This + * involves fetching their catalog, negotiating a contract, starting a transfer + * process and receiving the token. + */ + private void setupTransferProcess() { + String assetId = this.edcService.getAssetIdFromCatalogForAssetName(this.myExposedUri.toString(), + this.remoteKerUri.toString(), EdcConnectorService.ASSET_NAME); + String contractAgreementJson = this.edcService.negotiateContract(this.myExposedUri.toString(), + this.remoteKerUri.toString(), assetId); + + this.contractAgreementId = findByJsonPointerExpression(contractAgreementJson, "/contractAgreementId"); + + String transferJson = this.edcService.transferProcess(this.myExposedUri.toString(), + this.remoteKerUri.toString(), this.contractAgreementId, assetId); + this.transferId = findByJsonPointerExpression(transferJson, "/@id"); + + LOG.info("EDC Data Transfer with Remote KER {} started with Contract Agreement Id: {} and Transfer Id: {}", + this.remoteKerUri.toString(), this.contractAgreementId, this.transferId); } private int getHttpTimeout() { @@ -122,10 +190,17 @@ private int errorOccurred() { * {@link KnowledgeEngineRuntimeDetails} */ private void updateRemoteKerDataFromPeer() { + if (this.edcService != null && !tokenAvailable()) { + LOG.warn("No token available yet!"); + return; + } try { - HttpRequest request = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails")) - .header("Content-Type", "application/json").GET().build(); - + HttpRequest.Builder requestBuilder = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails")) + .headers("Content-Type", "application/json"); + if (this.edcService != null) + requestBuilder = requestBuilder.setHeader("Authorization", authToken); + + HttpRequest request = requestBuilder.GET().build(); HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); if (response.statusCode() == 200) { KnowledgeEngineRuntimeDetails runtimeDetails = objectMapper.readValue(response.body(), @@ -153,6 +228,10 @@ private void updateRemoteKerDataFromPeer() { dispatcher.notifySmartConnectorsChanged(); } + private boolean tokenAvailable() { + return this.authToken != null; + } + public boolean isAvailable() { if (tryAgainAfter != null) { boolean after = LocalDateTime.now().isAfter(tryAgainAfter); @@ -228,12 +307,19 @@ public void start() { public void stop() { if (this.isAvailable()) { + if (this.edcService != null && !tokenAvailable()) { + LOG.warn("No token available yet!"); + return; + } try { String ker_id = URLEncoder.encode(dispatcher.getMyKnowledgeEngineRuntimeDetails().getRuntimeId(), StandardCharsets.UTF_8); - HttpRequest request = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails/" + ker_id)) - .header("Content-Type", "application/json").DELETE().build(); - + HttpRequest.Builder requestBuilder = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails/" + ker_id)) + .headers("Content-Type", "application/json"); + if (this.edcService != null) + requestBuilder = requestBuilder.headers("Authorization", authToken); + + HttpRequest request = requestBuilder.DELETE().build(); HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); if (response.statusCode() == 204) { LOG.trace("Successfully said goodbye to {}", this.remoteKerUri); @@ -272,73 +358,90 @@ public void sendToRemoteSmartConnector(KnowledgeMessage message) throws IOExcept assert (getRemoteKerDetails() == null ? true : getRemoteKerDetails().getSmartConnectorIds().contains(message.getToKnowledgeBase().toString())); - if (this.isAvailable()) { - - try { - String jsonMessage = objectMapper.writeValueAsString(MessageConverter.toJson(message)); - HttpRequest request = HttpRequest - .newBuilder(new URI(this.remoteKerUri + getPathForMessageType(message))) - .header("Content-Type", "application/json").POST(BodyPublishers.ofString(jsonMessage)).build(); + if (!this.isAvailable()) { + logStillIgnoring(); + throw new IOException("KER " + this.remoteKerUri + " is currently unavailable. Trying again later."); + } - HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); + if (this.edcService != null && !tokenAvailable()) { + LOG.warn("No token available yet!"); + return; + } - if (response.statusCode() == 202) { - this.noError(); - LOG.trace("Successfully sent message {} to {}", message.getMessageId(), this.remoteKerUri); - } else { - this.remoteKerDetails = null; - int time = this.errorOccurred(); - LOG.warn("Ignoring KER {} for {} minutes. Failed to send message {} to {}, got response {}: {}", - this.remoteKerUri, time, message.getMessageId(), this.remoteKerUri, response.statusCode(), - response.body()); - this.dispatcher.notifySmartConnectorsChanged(); - throw new IOException("Message not accepted by remote host, status code " + response.statusCode() - + ", body " + response.body()); - } - } catch (URISyntaxException | InterruptedException | IOException | IllegalArgumentException e) { + try { + String jsonMessage = objectMapper.writeValueAsString(MessageConverter.toJson(message)); + HttpRequest.Builder requestBuilder = HttpRequest + .newBuilder(new URI(this.remoteKerUri + getPathForMessageType(message))) + .headers("Content-Type", "application/json"); + if (this.edcService != null) + requestBuilder = requestBuilder.setHeader("Authorization", authToken); + + HttpRequest request = requestBuilder.POST(BodyPublishers.ofString(jsonMessage)).build(); + HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); + if (response.statusCode() == 202) { + this.noError(); + LOG.trace("Successfully sent message {} to {}", message.getMessageId(), this.remoteKerUri); + } else { this.remoteKerDetails = null; int time = this.errorOccurred(); - LOG.warn("Ignoring KER {} for {} minutes. Error '{}' occurred.", this.remoteKerUri, time, - e.getMessage()); + LOG.warn("Ignoring KER {} for {} minutes. Failed to send message {} to {}, got response {}: {}", + this.remoteKerUri, time, message.getMessageId(), this.remoteKerUri, response.statusCode(), + response.body()); this.dispatcher.notifySmartConnectorsChanged(); - throw new IOException(e); + throw new IOException("Message not accepted by remote host, status code " + response.statusCode() + + ", body " + response.body()); } - } else { - logStillIgnoring(); - throw new IOException("KER " + this.remoteKerUri + " is currently unavailable. Trying again later."); + } catch (URISyntaxException | InterruptedException | IOException | IllegalArgumentException e) { + this.remoteKerDetails = null; + int time = this.errorOccurred(); + LOG.warn("Ignoring KER {} for {} minutes. Error '{}' occurred.", this.remoteKerUri, time, + e.getMessage()); + this.dispatcher.notifySmartConnectorsChanged(); + throw new IOException(e); } } public void sendMyKerDetailsToPeer(KnowledgeEngineRuntimeDetails details) { - if (this.isAvailable()) { - try { - String jsonMessage = objectMapper.writeValueAsString(details); - HttpRequest request = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails")) - .header("Content-Type", "application/json").POST(BodyPublishers.ofString(jsonMessage)).build(); + if (!this.isAvailable()) { + logStillIgnoring(); + return; + } - HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); - if (response.statusCode() == 200) { - this.noError(); - LOG.trace("Successfully sent updated KnowledgeEngineRuntimeDetails to {}", this.remoteKerUri); - } else { - this.remoteKerDetails = null; - int time = this.errorOccurred(); - this.dispatcher.notifySmartConnectorsChanged(); - LOG.warn( - "Ignoring KER {} for {} minutes. Failed to send updated KnowledgeEngineRuntimeDetails, got response {}: {}", - this.remoteKerUri, time, response.statusCode(), response.body()); - } - } catch (IOException | URISyntaxException | InterruptedException | IllegalArgumentException e) { + if (this.edcService != null && !tokenAvailable()) { + LOG.warn("No token available yet!"); + return; + } + + try { + String jsonMessage = objectMapper.writeValueAsString(details); + HttpRequest.Builder requestBuilder = HttpRequest.newBuilder(new URI(this.remoteKerUri + "/runtimedetails")) + .headers("Content-Type", "application/json"); + + if (this.edcService != null) + requestBuilder = requestBuilder.headers("Authorization", authToken); + + HttpRequest request = requestBuilder.POST(BodyPublishers.ofString(jsonMessage)).build(); + HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); + if (response.statusCode() == 200) { + this.noError(); + LOG.trace("Successfully sent updated KnowledgeEngineRuntimeDetails to {}", this.remoteKerUri); + } else { this.remoteKerDetails = null; int time = this.errorOccurred(); this.dispatcher.notifySmartConnectorsChanged(); LOG.warn( - "Ignoring KER {} for {} minutes. Failed to send updated KnowledgeEngineRuntimeDetails due to '{}'", - this.remoteKerUri, time, e.getMessage()); - LOG.debug("", e); + "Ignoring KER {} for {} minutes. Failed to send updated KnowledgeEngineRuntimeDetails, got response {}: {}", + this.remoteKerUri, time, response.statusCode(), response.body()); } - } else - logStillIgnoring(); + } catch (IOException | URISyntaxException | InterruptedException | IllegalArgumentException e) { + this.remoteKerDetails = null; + int time = this.errorOccurred(); + this.dispatcher.notifySmartConnectorsChanged(); + LOG.warn( + "Ignoring KER {} for {} minutes. Failed to send updated KnowledgeEngineRuntimeDetails due to '{}'", + this.remoteKerUri, time, e.getMessage()); + LOG.debug("", e); + } } private String getPathForMessageType(KnowledgeMessage message) { @@ -356,4 +459,59 @@ private String getPathForMessageType(KnowledgeMessage message) { return null; } } + + public String getConfigProperty(String key, String defaultValue) { + // We might replace this with something a bit more fancy in the future... + String value = System.getenv(key); + if (value == null) { + value = defaultValue; + LOG.trace("No value for the configuration parameter '{}' was provided, using the default value '{}'", key, + defaultValue); + } + return value; + } + + public boolean hasConfigProperty(String key) { + return System.getenv(key) != null; + } + + public boolean checkAuthorizationToken(String authorizationToken) { + if (validationEndpoint != null) { + LOG.info("Contacting validation endpoint {}", validationEndpoint); + HttpRequest request = null; + try { + request = HttpRequest.newBuilder(new URI(validationEndpoint)) + .headers("Content-Type", "application/json", "Authorization", authorizationToken).GET().build(); + } catch (URISyntaxException e) { + LOG.warn("Invalid URI for the validationEndpoint: " + validationEndpoint); + } + + try { + HttpResponse response = this.httpClient.send(request, BodyHandlers.ofString()); + if (response.statusCode() == 200) { + return true; + } else { + LOG.warn("Validating failed with status code {} and message '{}'", response.statusCode(), + response.body()); + } + } catch (IOException | InterruptedException e) { + LOG.error("Encountered a problem during authenticating the EDC token.", e); + } + + } + return false; + } + + public String getTransferId() { + return this.transferId; + } + + public String getContractAgreementId() { + return this.contractAgreementId; + } + + public void setToken(String aToken) { + this.authToken = aToken; + this.updateRemoteKerDataFromPeer(); + } } diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnectionManager.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnectionManager.java index a8cb82aac..53fb0c63b 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnectionManager.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteKerConnectionManager.java @@ -1,29 +1,40 @@ package eu.knowledge.engine.smartconnector.runtime.messaging; +import java.io.FileInputStream; +import java.io.IOException; import java.net.URI; +import java.net.URISyntaxException; import java.util.ArrayList; import java.util.Date; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.Properties; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ScheduledFuture; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; -import jakarta.ws.rs.core.Response; -import jakarta.ws.rs.core.SecurityContext; - +import org.eclipse.microprofile.config.Config; +import org.eclipse.microprofile.config.ConfigProvider; +import org.eclipse.microprofile.config.ConfigValue; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import eu.knowledge.engine.smartconnector.api.SmartConnector; +import eu.knowledge.engine.smartconnector.api.SmartConnectorConfig; import eu.knowledge.engine.smartconnector.runtime.KeRuntime; import eu.knowledge.engine.smartconnector.runtime.messaging.inter_ker.api.NotFoundException; import eu.knowledge.engine.smartconnector.runtime.messaging.inter_ker.api.SmartConnectorManagementApiService; import eu.knowledge.engine.smartconnector.runtime.messaging.inter_ker.model.KnowledgeEngineRuntimeDetails; import eu.knowledge.engine.smartconnector.runtime.messaging.kd.model.KnowledgeEngineRuntimeConnectionDetails; +import eu.knowledge.engine.smartconnector.edc.EdcConnectorProperties; +import eu.knowledge.engine.smartconnector.edc.EdcConnectorService; +import eu.knowledge.engine.smartconnector.edc.InMemoryTokenManager; +import eu.knowledge.engine.smartconnector.edc.Token; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; /** * The class is responsible for detecting new or removed remote @@ -45,10 +56,62 @@ public class RemoteKerConnectionManager extends SmartConnectorManagementApiServi private ScheduledFuture scheduledKnowledgeDirectoryQueryFuture; private final MessageDispatcher messageDispatcher; private Date knowledgeDirectoryUpdateCooldownEnds = null; + private EdcConnectorService edcService = null; + private InMemoryTokenManager tokenManager = null; + private URI myExposedUrl; + private URI myEdcConnectorUrl = null; + private boolean useEdc; - public RemoteKerConnectionManager(MessageDispatcher messageDispatcher) { + public RemoteKerConnectionManager(MessageDispatcher messageDispatcher, URI myExposedUrl, boolean useEdc) { this.messageDispatcher = messageDispatcher; + this.myExposedUrl = myExposedUrl; messageReceiver = new RemoteMessageReceiver(messageDispatcher); + this.useEdc = useEdc; + + if (this.useEdc) { + List config = loadConfig(); + + this.edcService = new EdcConnectorService(config); + this.tokenManager = new InMemoryTokenManager(); + } + } + + /** + * TODO We do not want to load these manually, is there a better way? + * + * @return A configuration object with properties for the two connectors. + */ + private List loadConfig() { + + Config config = ConfigProvider.getConfig(); + + ConfigValue protocolUrl = config.getConfigValue(SmartConnectorConfig.CONF_KEY_KE_EDC_PROTOCOL_URL); + ConfigValue managementUrl = config.getConfigValue(SmartConnectorConfig.CONF_KEY_KE_EDC_MANAGEMENT_URL); + ConfigValue dataPlaneControlUrl = config.getConfigValue(SmartConnectorConfig.CONF_KEY_KE_EDC_DATAPLANE_CONTROL_URL); + ConfigValue dataPlanePublicUrl = config.getConfigValue(SmartConnectorConfig.CONF_KEY_KE_EDC_DATAPLANE_PUBLIC_URL); + + EdcConnectorProperties props = new EdcConnectorProperties( + this.myExposedUrl.toString(), + protocolUrl.getValue(), + managementUrl.getValue(), + "tke-dataplane", + dataPlaneControlUrl.getValue(), + dataPlanePublicUrl.getValue(), + "TNO Knowledge Engine Runtime", + "https://www.knowledge-engine.eu/" + ); + + LOG.info("Setting management url to: {}", managementUrl); + + try { + this.myEdcConnectorUrl = new URI(props.protocolUrl()); + } catch (URISyntaxException e) { + LOG.error("Invalid syntax for EDC Connector URL"); + } + + List connectors = List.of(props); + + return connectors; } public void start() { @@ -60,6 +123,11 @@ public void start() { LOG.error("", t); } }, 5, KNOWLEDGE_DIRECTORY_UPDATE_INTERVAL, TimeUnit.SECONDS); + + // configure our EDC Connector with the TKE asset + if (useEdc) { + edcService.configureConnector(this.myExposedUrl.toString()); + } } public void scheduleQueryKnowledgeDirectory() { @@ -113,7 +181,21 @@ private synchronized void queryKnowledgeDirectory() { if (!remoteKerConnections.containsKey(knowledgeEngineRuntime.getId())) { // This must be a new remote KER LOG.info("Discovered new peer " + knowledgeEngineRuntime.getId()); - RemoteKerConnection messageSender = new RemoteKerConnection(messageDispatcher, knowledgeEngineRuntime); + + RemoteKerConnection messageSender; + if (useEdc) { + EdcConnectorProperties prop = new EdcConnectorProperties( + knowledgeEngineRuntime.getExposedUrl().toString(), + knowledgeEngineRuntime.getEdcConnectorUrl().toString() + ); + this.edcService.addConnector(prop); + + messageSender = new RemoteKerConnection(messageDispatcher, this.myExposedUrl, + this.edcService, this.tokenManager, knowledgeEngineRuntime); + } else { + messageSender = new RemoteKerConnection(messageDispatcher, this.myExposedUrl, null, null, + knowledgeEngineRuntime); + } remoteKerConnections.put(knowledgeEngineRuntime.getId(), messageSender); messageSender.start(); messageSender.sendMyKerDetailsToPeer(messageDispatcher.getMyKnowledgeEngineRuntimeDetails()); @@ -179,7 +261,8 @@ public RemoteKerConnection getRemoteKerConnection(URI toKnowledgeBase) { * Another KER would like to know our {@link KnowledgeEngineRuntimeDetails}. */ @Override - public Response runtimedetailsGet(SecurityContext securityContext) throws NotFoundException { + public Response runtimedetailsGet(String authorizationToken, SecurityContext securityContext) + throws NotFoundException { KnowledgeEngineRuntimeDetails runtimeDetails = messageDispatcher.getMyKnowledgeEngineRuntimeDetails(); return Response.status(200).entity(runtimeDetails).build(); } @@ -189,8 +272,9 @@ public Response runtimedetailsGet(SecurityContext securityContext) throws NotFou * {@link KnowledgeEngineRuntimeDetails} have changed. */ @Override - public Response runtimedetailsPost(KnowledgeEngineRuntimeDetails knowledgeEngineRuntimeDetails, - SecurityContext securityContext) throws NotFoundException { + public Response runtimedetailsPost(String authorizationToken, + KnowledgeEngineRuntimeDetails knowledgeEngineRuntimeDetails, SecurityContext securityContext) + throws NotFoundException { RemoteKerConnection remoteKerConnection = remoteKerConnections .get(knowledgeEngineRuntimeDetails.getRuntimeId()); if (remoteKerConnection == null) { @@ -210,7 +294,8 @@ public Response runtimedetailsPost(KnowledgeEngineRuntimeDetails knowledgeEngine * Another KER lets us know it will leave. */ @Override - public Response runtimedetailsKerIdDelete(String kerId, SecurityContext securityContext) throws NotFoundException { + public Response runtimedetailsKerIdDelete(String authorizationToken, String kerId, SecurityContext securityContext) + throws NotFoundException { RemoteKerConnection kerConnection = remoteKerConnections.remove(kerId); if (kerConnection == null) { // That one didn't exist @@ -227,6 +312,25 @@ public Response runtimedetailsKerIdDelete(String kerId, SecurityContext security } } + @Override + public Response tokenPost(String body, SecurityContext securityContext) throws NotFoundException { + + LOG.info("Token JSON received: {}", body); + // TODO Change runtimeexception from new Token to something we can use? + if (tokenManager != null) { + tokenManager.tokenReceived(new Token(body)); + Token t = new Token(body); + + for (RemoteKerConnection ker : this.remoteKerConnections.values()) { + if (ker.getTransferId().equals(t.id()) && ker.getContractAgreementId().equals(t.contractId())) { + ker.setToken(t.authCode()); + } + } + } + + return Response.status(200).build(); + } + /** * Notify other KnowledgeEngineRuntimes that something changed locally. Called * directly by the {@link LocalSmartConnectorConnectionManager} after it made @@ -254,4 +358,14 @@ public List getRemoteSmartConnectorIds() { return list; } + public boolean isTokenValid(String authorizationToken, URI fromKnowledgeBase) { + if (getRemoteKerConnection(fromKnowledgeBase) != null) { + return getRemoteKerConnection(fromKnowledgeBase).checkAuthorizationToken(authorizationToken); + } + return false; + } + + URI getEdcConnectorUrl() { + return this.myEdcConnectorUrl; + } } diff --git a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteMessageReceiver.java b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteMessageReceiver.java index 5a471f959..3a6115b2e 100644 --- a/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteMessageReceiver.java +++ b/smart-connector/src/main/java/eu/knowledge/engine/smartconnector/runtime/messaging/RemoteMessageReceiver.java @@ -33,12 +33,29 @@ public RemoteMessageReceiver(MessageDispatcher messageDispatcher) { this.messageDispatcher = messageDispatcher; } - private Response handleMessage(KnowledgeMessage message) { + private Response handleMessage(String authorizationToken, KnowledgeMessage message) { try { LOG.trace("Received {} {} from KnowledgeDirectory for KnowledgeBase {} from remote SmartConnector", message.getClass().getSimpleName(), message.getMessageId(), message.getToKnowledgeBase()); - messageDispatcher.deliverToLocalSmartConnector(message); - return Response.status(202).build(); + //TODO: Hacky way to determine meta-ness of KI is not great, replace by more robust solution + if (message.getToKnowledgeInteraction().toURL().toString().contains("meta")) { + messageDispatcher.deliverToLocalSmartConnector(message); + return Response.status(202).build(); + } else { + if (this.messageDispatcher.usesEdc()) { + if (messageDispatcher.getRemoteSmartConnectorConnectionsManager().isTokenValid(authorizationToken, message.getFromKnowledgeBase())) { + LOG.info("Authorization token has been validated"); + messageDispatcher.deliverToLocalSmartConnector(message); + return Response.status(202).build(); + } else { + LOG.warn("Could not validate the authorization token"); + return Response.status(403).entity("Invalid authorization token").build(); + } + } else { + messageDispatcher.deliverToLocalSmartConnector(message); + return Response.status(202).build(); + } + } } catch (IOException e) { // Was not able to deliver message to the SmartConnector return createErrorResponse(e); @@ -51,20 +68,20 @@ private Response createErrorResponse(Exception e) { } @Override - public Response messagingAskmessagePost(AskMessage askMessage, SecurityContext securityContext) + public Response messagingAskmessagePost(String authorizationToken, AskMessage askMessage, SecurityContext securityContext) throws NotFoundException { try { - return handleMessage(MessageConverter.fromJson(askMessage)); + return handleMessage(authorizationToken, MessageConverter.fromJson(askMessage)); } catch (URISyntaxException | IllegalArgumentException e) { return createErrorResponse(e); } } @Override - public Response messagingAnswermessagePost(AnswerMessage answerMessage, SecurityContext securityContext) + public Response messagingAnswermessagePost(String authorizationToken, AnswerMessage answerMessage, SecurityContext securityContext) throws NotFoundException { try { - return handleMessage(MessageConverter.fromJson(answerMessage)); + return handleMessage(authorizationToken, MessageConverter.fromJson(answerMessage)); } catch (URISyntaxException | IllegalArgumentException e) { // Could not parse message return createErrorResponse(e); @@ -72,30 +89,30 @@ public Response messagingAnswermessagePost(AnswerMessage answerMessage, Security } @Override - public Response messagingPostmessagePost(PostMessage postMessage, SecurityContext securityContext) + public Response messagingPostmessagePost(String authorizationToken, PostMessage postMessage, SecurityContext securityContext) throws NotFoundException { try { - return handleMessage(MessageConverter.fromJson(postMessage)); + return handleMessage(authorizationToken, MessageConverter.fromJson(postMessage)); } catch (URISyntaxException | IllegalArgumentException e) { return createErrorResponse(e); } } @Override - public Response messagingReactmessagePost(ReactMessage reactMessage, SecurityContext securityContext) + public Response messagingReactmessagePost(String authorizationToken, ReactMessage reactMessage, SecurityContext securityContext) throws NotFoundException { try { - return handleMessage(MessageConverter.fromJson(reactMessage)); + return handleMessage(authorizationToken, MessageConverter.fromJson(reactMessage)); } catch (URISyntaxException | IllegalArgumentException e) { return createErrorResponse(e); } } @Override - public Response messagingErrormessagePost(ErrorMessage errorMessage, SecurityContext securityContext) + public Response messagingErrormessagePost(String authorizationToken, ErrorMessage errorMessage, SecurityContext securityContext) throws NotFoundException { try { - return handleMessage(MessageConverter.fromJson(errorMessage)); + return handleMessage(authorizationToken, MessageConverter.fromJson(errorMessage)); } catch (URISyntaxException | IllegalArgumentException e) { return createErrorResponse(e); } diff --git a/smart-connector/src/main/resources/META-INF/microprofile-config.properties b/smart-connector/src/main/resources/META-INF/microprofile-config.properties index c65161db5..bbf90fa7c 100644 --- a/smart-connector/src/main/resources/META-INF/microprofile-config.properties +++ b/smart-connector/src/main/resources/META-INF/microprofile-config.properties @@ -6,4 +6,11 @@ ke.http.timeout = 5 kd.url = http://localhost:8080 sc.validate.outgoing.bindings.wrt.incoming.bindings = true ke.runtime.hostname = localhost -ke.reasoner.level = 2 \ No newline at end of file +ke.reasoner.level = 2 +ke.runtime.use.edc = false + +ke.edc.protocol.url = +ke.edc.management.url = +ke.edc.dataplane.control.url = +ke.edc.dataplane.public.url = +ke.edc.token.validation.endpoint = diff --git a/smart-connector/src/main/resources/openapi-inter-ker.yaml b/smart-connector/src/main/resources/openapi-inter-ker.yaml index 2a7aa8ba5..8a0b8d122 100644 --- a/smart-connector/src/main/resources/openapi-inter-ker.yaml +++ b/smart-connector/src/main/resources/openapi-inter-ker.yaml @@ -11,6 +11,13 @@ paths: summary: List all the Smart Connectors represented by THIS Smart Connector Runtime tags: - "smart connector management" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string responses: '200': description: A list of Smart Connector Runtimes @@ -18,6 +25,8 @@ paths: application/json; charset=UTF-8: schema: $ref: '#/components/schemas/KnowledgeEngineRuntimeDetails' + '403': + description: Invalid authorization token. '500': description: If a problem occurred. content: @@ -28,6 +37,13 @@ paths: summary: Push a list of all the Smart Connectors that ANOTHER runtime represents. This can be done because the runtime is new or becaus its details have changed. tags: - "smart connector management" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -41,6 +57,8 @@ paths: text/plan; charset=UTF-8: schema: type: string + '403': + description: Invalid authorization token. '500': description: Provided data was not valid content: @@ -54,6 +72,12 @@ paths: tags: - "smart connector management" parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string - name: "ker_id" in: "path" description: ID of the Knowledge Engine Runtime @@ -63,18 +87,45 @@ paths: responses: '204': description: Understood. + '403': + description: Invalid authorization token. '404': description: The Smart Connector Runtime not known content: text/plain; charset=UTF-8: schema: type: string + /token: + post: + summary: Token to use when communicating with that particular KE Runtime (for usage with Eclipse Dataspace Components (EDC)) + tags: + - "smart connector management" + requestBody: + required: true + content: + application/json; charset=UTF-8: + schema: + $ref: '#/components/schemas/Token' + responses: + '200': + description: Successfully received token. + '400': + description: Incorrect token. + '500': + description: Error while processing token. /messaging/askmessage: post: summary: Handle an AskMessage tags: - "messaging" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -86,6 +137,8 @@ paths: description: Successfully received message '400': description: Messages not understood, don't try to resend message + '403': + description: Invalid authorization token. '500': description: Server was not able to receive message at this time, try again later /messaging/answermessage: @@ -93,6 +146,13 @@ paths: summary: Handle an AnswerMessage tags: - "messaging" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -104,6 +164,8 @@ paths: description: Successfully received message '400': description: Messages not understood, don't try to resend message + '403': + description: Invalid authorization token. '500': description: Server was not able to receive message at this time, try again later /messaging/postmessage: @@ -111,6 +173,13 @@ paths: summary: Handle a PostMessage tags: - "messaging" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -122,6 +191,8 @@ paths: description: Successfully received message '400': description: Messages not understood, don't try to resend message + '403': + description: Invalid authorization token. '500': description: Server was not able to receive message at this time, try again later /messaging/reactmessage: @@ -129,6 +200,13 @@ paths: summary: Handle a ReactMessage tags: - "messaging" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -140,6 +218,8 @@ paths: description: Successfully received message '400': description: Messages not understood, don't try to resend message + '403': + description: Invalid authorization token. '500': description: Server was not able to receive message at this time, try again later /messaging/errormessage: @@ -147,6 +227,13 @@ paths: summary: Handle an ErrorMessage tags: - "messaging" + parameters: + - name: Authorization + in: header + required: true + description: Authorization token (EDC-IDS) + schema: + type: string requestBody: required: true content: @@ -158,6 +245,8 @@ paths: description: Successfully received message '400': description: Messages not understood, don't try to resend message + '403': + description: Invalid authorization token. '500': description: Server was not able to receive message at this time, try again later @@ -270,4 +359,6 @@ components: errorMessage: type: string required: - - replyToMessage \ No newline at end of file + - replyToMessage + Token: + type: string \ No newline at end of file diff --git a/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/misc/WireMockFirstConfigurationTest.java b/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/misc/WireMockFirstConfigurationTest.java index 7073d9d14..9909a5c9d 100644 --- a/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/misc/WireMockFirstConfigurationTest.java +++ b/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/misc/WireMockFirstConfigurationTest.java @@ -62,7 +62,7 @@ public void testConfigHttpConnectTimeout() throws Exception { MessageDispatcher messageDispatcher = mock(MessageDispatcher.class); - var ker = new RemoteKerConnection(messageDispatcher, + var ker = new RemoteKerConnection(messageDispatcher, new URI(SmartConnectorConfig.CONF_KEY_KE_RUNTIME_EXPOSED_URL), null, null, new KnowledgeEngineRuntimeConnectionDetails().exposedUrl(URI.create("http://10.255.255.1/"))); ker.start(); @@ -77,4 +77,4 @@ public static void after() { System.clearProperty(SmartConnectorConfig.CONF_KEY_KE_RUNTIME_EXPOSED_URL); } -} +} \ No newline at end of file diff --git a/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnectionManagerTest.java b/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnectionManagerTest.java index 6751eedb7..4a648b71f 100644 --- a/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnectionManagerTest.java +++ b/smart-connector/src/test/java/eu/knowledge/engine/smartconnector/runtime/messaging/KnowledgeDirectoryConnectionManagerTest.java @@ -27,7 +27,7 @@ public void testSuccess() throws Exception { kd.start(); KnowledgeDirectoryConnection cm = new KnowledgeDirectoryConnection(new URI("http://localhost:8080"), - new URI("http://localhost:8081")); + new URI("http://localhost:8081"), null); assertEquals(KnowledgeDirectoryConnection.State.UNREGISTERED, cm.getState()); @@ -59,7 +59,7 @@ public void testSuccess() throws Exception { public void testNoKd() throws Exception { KnowledgeDirectoryConnection cm = new KnowledgeDirectoryConnection(new URI("http://localhost:8080"), - new URI("http://localhost:8081")); + new URI("http://localhost:8081"), null); assertEquals(KnowledgeDirectoryConnection.State.UNREGISTERED, cm.getState()); @@ -84,7 +84,7 @@ public void testInterrupted() throws Exception { kd = new KnowledgeDirectory(8080); kd.start(); - cm = new KnowledgeDirectoryConnection(new URI("http://localhost:8080"), new URI("http://localhost:8081")); + cm = new KnowledgeDirectoryConnection(new URI("http://localhost:8080"), new URI("http://localhost:8081"), null); assertEquals(KnowledgeDirectoryConnection.State.UNREGISTERED, cm.getState());