Implement code header for interactions through use of EDC by DaviddeBest-TNO · Pull Request #723 · TNO/knowledge-engine

DaviddeBest-TNO · 2025-08-27T09:00:26Z

Users can toggle the EDC functionality through setting the KE_RUNTIME_USE_EDC environment variables.
This sets an authorization code header of messages between KERs by using features from EDC.
A working examples is in the examples folder.

… with remote runtimes

…a-info & initial verification of authorization token

Callback not received. Not sure why yet.

Note that you need to start the EDC Connectors first manually and afterwards start the rest of the docker containers. Done: - When KER starts, it configures its EDC connector with the standard TKE asset. - When a new RemoteKER is found, we negotiate a contract and start a transfer. Then we receive a authentication token that we use for all communication with that particular KER. Todo: - remove hardcoded otherKER from edc.properties file. - include kb3 in example. - automatically revalidate token when it becomes invalid.

…mple

…PI and with simplified connector configuration

…le to work

bnouwt · 2025-08-29T07:34:06Z

+These Knowledge Interactions are first registered at the Smart Connector. 
+After they have been registered, they can be executed.
+
+For more information on the Knowledge Engine, check out the [documentation](../../docs/00_home.md).


Link is no longer valid. Replace with: https://docs.knowledge-engine.eu/

bnouwt · 2025-08-29T07:38:26Z

+- `connector/configuration/ker-configuration.properties` contains settings for the EDC-IDS Connector
+- `connector/configuration/ker-vault.properties` contains a public key
+- A new directory for this specific KER. Currently, they are named `ker1`, `ker2`, ..., containing:
+  - `edc.properties` file containing several properties that are used in the communication between KERs


This one is probably no longer necessary?

bnouwt · 2025-08-29T07:40:47Z

+For each additional KER with an EDC-IDS Connector, we need the following files in the `examples/edc-example` directory:
+- `connector/configuration/ker-configuration.properties` contains settings for the EDC-IDS Connector
+- `connector/configuration/ker-vault.properties` contains a public key
+- A new directory for this specific KER. Currently, they are named `ker1`, `ker2`, ..., containing:


A separate directory is no longer necessary now the edc.properties file is replaced by the default KE configuration that is being modified using environment variables in the docker-compose.yml file.

bnouwt · 2025-08-29T07:41:27Z

+
+The `docker-compose.yml` in `examples/edc-example/` should also be modified to include:
+- An additional KER (currently named `runtime-1`, `runtime-2`, ...)
+  - The `build` setting refers to the directory for the new KER.


build: can probably be replaced with image:.

bnouwt · 2025-08-29T07:42:02Z

+- An additional KER (currently named `runtime-1`, `runtime-2`, ...)
+  - The `build` setting refers to the directory for the new KER.
+  - The `depends_on` setting refers to the Docker component for the EDC-IDS Connector
+  - The `KE_RUNTIME_EXPOSED_URL` is a unique URL for the new KER.


Include configuration environment variables that used to be in edc.properties.

bnouwt · 2025-08-29T07:49:27Z

I think it is not necessary to add documentation about EDC functionality to https://docs.knowledge-engine.eu/, but maybe we can add a short section in the README.md about using EDC? Just mention the configuration key to enable it and refer to the edc-example for more info.

bnouwt · 2025-08-29T07:52:59Z

Like I mentioned above, this file can probably be removed.

bnouwt · 2025-08-29T08:06:00Z

I hope this file can be deleted, because we do not want it there.

bnouwt · 2025-08-29T08:22:08Z

+2. In the `knowledge-directory` directory in this project, execute `docker build . -t testkd:1.2.5-SNAPSHOT`.
+3. In the `smart-connector-rest-dist` directory in this project, execute `docker build . -t testsc:1.2.5-SNAPSHOT`.


Check if we still need this.

Also check the version numbers, because these are probably no longer correct.

I updated the version numbers. For now we still need this, but I agree that a more elegant solution is better in the future.

bnouwt · 2025-08-29T08:34:22Z

+  # This is the knowledge directory, facilitating discovery between different
+  # runtimes. It exposes its service over port 8282.
+  knowledge-directory:
+    image: testkd:1.3.2-SNAPSHOT


This version number should be 1.3.3-SNAPSHOT I think.

bnouwt · 2025-08-29T08:34:44Z

@@ -0,0 +1,2 @@
+FROM docker.io/library/testsc:1.3.2-SNAPSHOT


This version number should be 1.3.3-SNAPSHOT, I think.

bnouwt · 2025-08-29T08:34:57Z

@@ -0,0 +1,2 @@
+FROM docker.io/library/testsc:1.3.2-SNAPSHOT


This version number should be 1.3.3-SNAPSHOT, I think.

bnouwt · 2025-08-29T08:35:13Z

@@ -0,0 +1,2 @@
+FROM docker.io/library/testsc:1.3.2-SNAPSHOT


This version number should be 1.3.3-SNAPSHOT, I think.

bnouwt · 2025-08-29T08:40:27Z

+
+# authCode token expiration in seconds
+#edc.transfer.proxy.token.validity.seconds=999999
+edc.transfer.proxy.token.validity.seconds=120


Because this token validity is set to 120 seconds, the edc-example stops working (i.e. one of the KBs stops returning data) after 2 minutes. Maybe change this to the same value as the other connectors?

bnouwt · 2025-08-29T08:43:27Z

+            "tke-dataplane", 
+            "", 
+            "", 
+            "http://tke.tno.nl", 


Maybe change this to https://www.knowledge-engine.eu/?

bnouwt · 2025-08-29T08:44:54Z

+	// these are all static for every connector
+	public final static String DATA_PLANE_ID = "tke-dataplane";
+	public final static String ASSET_NAME = "TNO Knowledge Engine Runtime";
+	public final static String ASSET_URL = "https://www.knowledge-engine.eu/";


What is the difference between ASSET_URL and the assetUrl in the previous file? Probably nothing.

Something to clean up later

bnouwt · 2025-08-29T09:02:42Z

I have also tested the behavior of the empty configuration options when using EDC and this does not give any problems when ke.runtime.use.edc is false. However, when we set ke.runtime.use.edc to true and we do not correctly set these other configuration options, startup op the KE fails with an unclear exception. If we think this is a problem, we should think about how to deal with this. For example, give a better error message than:

2025-08-29 11:02:14:656 +0200 [qtp575335780-26] ERROR KeRuntime - Could not start HTTP server, cannot start Knowledge Engine
java.lang.IllegalArgumentException: URI with undefined scheme
	at java.net.http/jdk.internal.net.http.common.Utils.newIAE(Utils.java:286)
	at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.checkURI(HttpRequestBuilderImpl.java:79)
	at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.uri(HttpRequestBuilderImpl.java:71)
	at java.net.http/jdk.internal.net.http.HttpRequestBuilderImpl.uri(HttpRequestBuilderImpl.java:43)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorClient.httpPost(EdcConnectorClient.java:279)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorClient.httpPost(EdcConnectorClient.java:272)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorClient.catalogRequest(EdcConnectorClient.java:157)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorService.catalogRequest(EdcConnectorService.java:107)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorService.getAssetIdFromCatalogForAssetName(EdcConnectorService.java:85)
	at eu.knowledge.engine.smartconnector.edc.EdcConnectorService.configureConnector(EdcConnectorService.java:55)
	at eu.knowledge.engine.smartconnector.runtime.messaging.RemoteKerConnectionManager.start(RemoteKerConnectionManager.java:129)
	at eu.knowledge.engine.smartconnector.runtime.messaging.MessageDispatcher.start(MessageDispatcher.java:126)
	at eu.knowledge.engine.smartconnector.runtime.KeRuntime.getMessageDispatcher(KeRuntime.java:144)
	at eu.knowledge.engine.smartconnector.runtime.KeRuntime.<clinit>(KeRuntime.java:99)
	at eu.knowledge.engine.smartconnector.impl.OtherKnowledgeBaseStoreImpl.<init>(OtherKnowledgeBaseStoreImpl.java:29)
	at eu.knowledge.engine.smartconnector.impl.SmartConnectorImpl.<init>(SmartConnectorImpl.java:76)
	at eu.knowledge.engine.smartconnector.impl.SmartConnectorBuilder.create(SmartConnectorBuilder.java:15)
	at eu.knowledge.engine.rest.api.impl.RestKnowledgeBase.<init>(RestKnowledgeBase.java:235)
	at eu.knowledge.engine.rest.api.impl.RestKnowledgeBaseManager.createKB(RestKnowledgeBaseManager.java:86)
	at eu.knowledge.engine.rest.api.impl.SmartConnectorLifeCycleApiServiceImpl.scPost(SmartConnectorLifeCycleApiServiceImpl.java:162)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
	at java.base/java.lang.Thread.run(Thread.java:833)

bnouwt · 2025-08-29T09:12:49Z

 kd.url = http://localhost:8080
 sc.validate.outgoing.bindings.wrt.incoming.bindings = true
 ke.runtime.hostname = localhost
-ke.reasoner.level = 2
+ke.reasoner.level = 2
+ke.runtime.use.edc = false


These red ^M character mean that this file uses windows line endings, which is wrong. I've already create issue #724, because some other files in the repo also use this, but maybe for this file we can already change it to using LF (line feed).

bnouwt · 2025-08-29T09:21:48Z

+        edcConnectorUrl:
+          type: string
+          format: uri


I wondered what happened with this property when we did not configure it via the configuration options and it looks like this:

[ { "id": "http://localhost:8081", "exposedUrl": "http://localhost:8081", "protocolVersion": "1.0.0", "lastRenew": "2025-08-29T11:16:39.984815+02:00", "edcConnectorUrl": null } ]

This is OK I guess, but I do find it a pity that it is necessary. We could think about whether we could leave it out if not applicable. This might also break backwards compatibility of the Knowledge Directory and KE Runtimes, so we should at least mention this in the next release of the KE.

Now edcConnectorUrl is only set if not null

bnouwt · 2025-08-29T09:22:32Z

 		KnowledgeEngineRuntimeConnectionDetails ker = new KnowledgeEngineRuntimeConnectionDetails();
 		ker.setExposedUrl(myExposedUrl);
 		ker.setProtocolVersion(PROTOCOL_VERSION);
+		ker.setEdcConnectorUrl(myEdcConnectorUrl); //TODO: make optional, or can just be null?


This TODO from (probably) @Sophietje is spot on 😉

Now made optional!

bnouwt · 2025-08-29T09:38:33Z

I am a bit scared for this class, since its error handling has been improved the last year or so. I assume those improvements were correctly merge, but it would be a pity if we lost them.

I've quickly double checked and compared the master branch (and its recent history) of this file and compared it with this version and it seems to be fine.

Are there certain things that you would like me to check? We can also discuss in person

Maybe double check if the error handling cases that currently happens in the master branch are also handled in this branch? If you also think they are similar (like I suggest above) then it is fine by me.👍

bnouwt · 2025-08-29T09:43:40Z

What is this empty class doing here? The naming Tken... is also quite strange. If I remove it locally everything still builds, but I did not test whether the EDC example also still works.

I truly have no idea what this is. Will remove it!

bnouwt

Hi @DaviddeBest-TNO, please don't be alarmed by the many comments😉. When reviewing we should just mention as many things as possible and afterwards determine which ones are actually worthwhile/necessary to address.

I have tested the edc-example and it also works for me locally, so that is good news!

I think some of the suggestions are worthwhile and easy to address, but we can discuss the others. Maybe go over the comments yourself and fix those that are easy and you agree with. After that we can discuss the remaining ones. Thanks for the effort👍

Sophietje · 2025-09-01T13:44:56Z

I think it is not necessary to add documentation about EDC functionality to https://docs.knowledge-engine.eu/, but maybe we can add a short section in the README.md about using EDC? Just mention the configuration key to enable it and refer to the edc-example for more info.

Personally I don't think the main README of this repo would be the right place for that. It's already quite long and it's not necessary information for most (?) users. Perhaps we can find a better spot? Maybe a README elsewhere in the repo (near the example?). Otherwise the docs.knowledge-engine.eu could be a good spot if we add something like a header saying 'experimental'

DaviddeBest-TNO · 2025-09-25T15:26:27Z

I think it is not necessary to add documentation about EDC functionality to https://docs.knowledge-engine.eu/, but maybe we can add a short section in the README.md about using EDC? Just mention the configuration key to enable it and refer to the edc-example for more info.

Personally I don't think the main README of this repo would be the right place for that. It's already quite long and it's not necessary information for most (?) users. Perhaps we can find a better spot? Maybe a README elsewhere in the repo (near the example?). Otherwise the docs.knowledge-engine.eu could be a good spot if we add something like a header saying 'experimental'

I made an attempt to add a small text on current developments to the README, also referring to the README of the EDC example for more info. This something we think is suitable for now? And then change to something more elaborate later in the project?

bnouwt · 2025-09-25T15:32:24Z

I made an attempt to add a small text on current developments to the README, also referring to the README of the EDC example for more info. This something we think is suitable for now? And then change to something more elaborate later in the project?

For me this is fine👍 What do you think @Sophietje?

bnouwt

Thanks for processing the comments @DaviddeBest-TNO . You can merge if you think it is finished 👍

Sophietje and others added 19 commits January 22, 2024 09:52

Adds Authorization header to remote messages & sets up Docker example…

9caac48

… with remote runtimes

Adds Authorization header to all messages, skip authorization for met…

6e14338

…a-info & initial verification of authorization token

Added working (manual) example with edc integration

dcf3424

Small improvements to the edc example.

2e77a95

Clean up log statements & avoid HTML response printed in logs

a3ab594

Merge branch 'master' into add-authorization-header

d878f2e

Works until starting data transfer.

1557f5a

Callback not received. Not sure why yet.

Remove quick 'n dirty path fix.

5ee3632

Also remove environment variable from RemoteKerConnection prop file.

4dc87c6

Made authorization optional and extended documentation of the EDC exa…

133fcc0

…mple

Refer to edc-example directory as examples/edc-example in README

28ddf93

Fix merge conflicts from master

de64174

Merge branch 'master' into add-authorization-header

ae3f1b8

Merge branch 'master' into add-authorization-header

e609727

Add authorization header to remote ker request if EDC service is set

5f4af37

Change tests to use reasonerLevel instead of reasonerEnabled

a6a03ea

Add TKE EDC manager library code to this repository, without Spring A…

573232e

…PI and with simplified connector configuration

Add EDC config parameters to smart connector config and fix edc examp…

16f0883

…le to work

DaviddeBest-TNO requested a review from bnouwt August 27, 2025 09:00

DaviddeBest-TNO self-assigned this Aug 27, 2025

DaviddeBest-TNO added 2 commits August 27, 2025 11:04

Remove tke-edc-manager as a dependency

4fbdbb9

Fix dependencies from tke-edc-manager

2fcc4d2

bnouwt reviewed Aug 29, 2025

View reviewed changes

Comment thread smart-connector/pom.xml

bnouwt reviewed Aug 29, 2025

View reviewed changes

bnouwt requested changes Aug 29, 2025

View reviewed changes

Fix for comments on merge request

c74b6d1

bnouwt approved these changes Sep 25, 2025

View reviewed changes

bnouwt mentioned this pull request Sep 26, 2025

TestAskWithGapsEnabled still uses reasonerEnabled instead of reasonerLevel #705

Closed

bnouwt and others added 4 commits September 29, 2025 16:45

Make sure edcConnectionUrl only shows up when non-null.

1eaf6e8

Merge branch 'master' into add-authorization-header

3c54f3c

Undo incorrect merge conflict solve'

b139da0

Undo another incorrect merge conflict solve'

b8fae42

DaviddeBest-TNO merged commit fd63a00 into master Oct 2, 2025
2 checks passed

bnouwt deleted the add-authorization-header branch October 3, 2025 06:27

Sophietje mentioned this pull request Oct 24, 2025

Fix test that still uses reasonerEnabled #737

Closed

		2. In the `knowledge-directory` directory in this project, execute `docker build . -t testkd:1.2.5-SNAPSHOT`.
		3. In the `smart-connector-rest-dist` directory in this project, execute `docker build . -t testsc:1.2.5-SNAPSHOT`.

Conversation

DaviddeBest-TNO commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnouwt commented Aug 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnouwt commented Aug 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!