512 improve configuration of the knowledge engine

README.md

@@ -44,6 +44,7 @@ The rest of this README is structured as follows:
   - [(advanced) Administering a Knowledge Engine runtime](#advanced-administering-a-knowledge-engine-runtime)
     - [Starting the Knowledge Engine in local mode](#starting-the-knowledge-engine-in-local-mode)
     - [Starting the Knowledge Engine in distributed mode](#starting-the-knowledge-engine-in-distributed-mode)
+    - [Configuration](#configuration)
 
 # Demonstration videos and tutorials
 
@@ -82,10 +83,10 @@ However, running it in the above way **does not support** data exchange with rem
 To interact with other runtimes, it needs additional configuration:
 
 - An additional port mapping for the socket that listens for communication from other runtimes.
-- `KE_RUNTIME_EXPOSED_URL`: The URL via which the above socket is available for communication from the other runtime(s). You need to make sure the traffic is correctly routed to the container's port 8081 with a reverse proxy.
-- `KD_URL`: The URL on which to find the knowledge directory (to discover peers).
+- `ke.runtime.exposed.url`: The URL via which the above socket is available for communication from the other runtime(s). You need to make sure the traffic is correctly routed to the container's port 8081 with a reverse proxy.
+- `kd.url`: The URL on which to find the knowledge directory (to discover peers).
 
-The configuration can be set as follows:
+The configuration can be set as follows (note that the configuration properties below use [underscores and capital letters](#configuration))
 
 ```bash
 docker run \
@@ -258,12 +259,15 @@ nohup java -cp "smart-connector-rest-dist-1.2.5.jar:dependency/*" eu.knowledge.e
 ### Starting the Knowledge Engine in distributed mode
 The Knowledge Engine can also start in distributed mode, where it connects with a remote knowledge directory and where different instances of the Knowledge Engine (each instance hosting one or more smart connectors) can communicate with each other. More information about starting the Knowledge Engine in distributed mode can be found in the [documentation](docs/docs/distributed_mode.md).
 
-### Additional configuration environment variables
+### Configuration
+TNO Knowledge Engine uses the [MicroProfile Config 3.1](https://microprofile.io/specifications/config/) specification to configure its behaviour and we use [SmallRye](https://smallrye.io/smallrye-config/) as the implementation of this specification. The default configuration values can be found in the [microprofile-config.properties](./smart-connector/src/main/resources/META-INF/microprofile-config.properties) configuration file. And, as described in the specification, these configuration values can be overridden by [environment variables and system properties](https://download.eclipse.org/microprofile/microprofile-config-3.1/microprofile-config-spec-3.1.html#default_configsources). Note that environment variables can use underscores and capital letters to adhere to their naming conventions and the MicroProfile Config automatically maps those to corresponding configuration properties using [specific rules](https://download.eclipse.org/microprofile/microprofile-config-3.1/microprofile-config-spec-3.1.html#default_configsources.env.mapping).
+
+A description of all configuration properties can be found in the [`SmartConnectorConfig`](./smart-connector/src/main/java/eu/knowledge/engine/smartconnector/impl/SmartConnectorConfig.java) class. The rest of this section highlights some of these configuration properties.
 
 *Increasing the wait time for other KBs to respond*
 
-By default, a Smart Connector waits `10` seconds max for a reply from another Smart Connector when sending an ASK/POST message. This time is configurable via the `KE_KB_WAIT_TIMEOUT` environment variable and setting it to `0` means the Smart Connector will wait indefinitely (this can be useful when dealing with Human KBs).
+By default, a Smart Connector waits `10` seconds max for a reply from another Smart Connector when sending an ASK/POST message. This time is configurable via the `ke.kb.wait.timeout` property and setting it to `0` means the Smart Connector will wait indefinitely (this can be useful when dealing with Human KBs).
 
 *Increasing the HTTP timeouts*
 
-By default, a KER waits `5` seconds max for a HTTP response from another KER when sending a message via the inter-KER protocol. The time is configurable via the `KE_HTTP_TIMEOUT` environment variable.
+By default, a KER waits `5` seconds max for a HTTP connection response from another KER when sending a message via the inter-KER protocol. The time is configurable via the `ke.http.timeout` property

admin-ui/src/main/java/eu/knowledge/engine/admin/AdminUI.java

@@ -16,6 +16,7 @@
 import org.apache.jena.update.UpdateFactory;
 import org.apache.jena.update.UpdateRequest;
 import org.apache.jena.vocabulary.RDF;
+import org.eclipse.microprofile.config.ConfigProvider;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -43,8 +44,6 @@ public class AdminUI implements KnowledgeBase {
 
 	private static final String META_GRAPH_PATTERN_STR = "?kb rdf:type ke:KnowledgeBase . ?kb ke:hasName ?name . ?kb ke:hasDescription ?description . ?kb ke:hasKnowledgeInteraction ?ki . ?ki rdf:type ?kiType . ?ki ke:isMeta ?isMeta . ?ki ke:hasCommunicativeAct ?act . ?act rdf:type ke:CommunicativeAct . ?act ke:hasRequirement ?req . ?act ke:hasSatisfaction ?sat . ?req rdf:type ?reqType . ?sat rdf:type ?satType . ?ki ke:hasGraphPattern ?gp . ?gp rdf:type ?patternType . ?gp ke:hasPattern ?pattern .";
 
-	private static final String CONF_KEY_INITIAL_ADMIN_UI_DELAY = "INITIAL_ADMIN_UI_DELAY";
-
 	private SmartConnector sc;
 	private final PrefixMapping prefixes;
 
@@ -141,7 +140,8 @@ public void smartConnectorReady(SmartConnector aSC) {
 		// to receive the initial state, we do a single Ask (after sleeping for a
 		// specific amount of time)
 		try {
-			Thread.sleep(Integer.parseInt(AdminUI.getConfigProperty(CONF_KEY_INITIAL_ADMIN_UI_DELAY, "5000")));
+			Thread.sleep(
+					ConfigProvider.getConfig().getValue(AdminUIConfig.CONF_KEY_INITIAL_ADMIN_UI_DELAY, Integer.class));
 		} catch (InterruptedException e) {
 			LOG.info("{}", e);
 		}
@@ -184,7 +184,8 @@ public BindingSet handleChangedKnowledgeBaseKnowledge(ReactExchangeInfo ei) {
 			// - insert the *new* data about that knowledge base
 
 			Resource kb = model.listSubjectsWithProperty(RDF.type, Vocab.KNOWLEDGE_BASE).next();
-			String query = String.format("DELETE { %s } WHERE { %s FILTER (?kb = <%s>) } ", this.metaGraphPattern.getPattern(), this.metaGraphPattern.getPattern(), kb.toString());
+			String query = String.format("DELETE { %s } WHERE { %s FILTER (?kb = <%s>) } ",
+					this.metaGraphPattern.getPattern(), this.metaGraphPattern.getPattern(), kb.toString());
 
 			UpdateRequest updateRequest = UpdateFactory.create(query);
 			UpdateAction.execute(updateRequest, this.model);
@@ -215,7 +216,8 @@ public BindingSet handleRemovedKnowledgeBaseKnowledge(ReactExchangeInfo ei) {
 			// - delete all old data about that knowledge base
 
 			Resource kb = model.listSubjectsWithProperty(RDF.type, Vocab.KNOWLEDGE_BASE).next();
-			String query = String.format("DELETE { %s } WHERE { %s FILTER (?kb = <%s>) } ", this.metaGraphPattern.getPattern(), this.metaGraphPattern.getPattern(), kb.toString());
+			String query = String.format("DELETE { %s } WHERE { %s FILTER (?kb = <%s>) } ",
+					this.metaGraphPattern.getPattern(), this.metaGraphPattern.getPattern(), kb.toString());
 
 			UpdateRequest updateRequest = UpdateFactory.create(query);
 			UpdateAction.execute(updateRequest, this.model);
@@ -328,15 +330,4 @@ public void close() {
 	public Model getModel() {
 		return model;
 	}
-
-	public static String getConfigProperty(String key, String defaultValue) {
-		// We might replace this with something a bit more fancy in the future...
-		String value = System.getenv(key);
-		if (value == null) {
-			value = defaultValue;
-			LOG.info("No value for the configuration parameter '" + key + "' was provided, using the default value '"
-					+ defaultValue + "'");
-		}
-		return value;
-	}
 }

admin-ui/src/main/java/eu/knowledge/engine/admin/AdminUIConfig.java

@@ -0,0 +1,13 @@
+package eu.knowledge.engine.admin;
+
+public class AdminUIConfig {
+
+	/**
+	 * The key to configure how long (in milliseconds) should the Admin UI wait
+	 * until it tries to ask for all KBs in the network. This value should probably
+	 * be higher in distributed mode, to allow the participants to reach equilibrium
+	 * with respect to knowledge about each other.
+	 */
+	public static final String CONF_KEY_INITIAL_ADMIN_UI_DELAY = "initial.admin.ui.delay";
+
+}

admin-ui/src/main/resources/META-INF/microprofile-config.properties

@@ -0,0 +1 @@
+initial.admin.ui.delay = 5000

admin-ui/src/main/resources/openapi-admin-ui.yaml

@@ -36,6 +36,17 @@ paths:
             text/plain; charset=UTF-8:
               schema:
                 type: string
+  /rest/admin/reload:
+    get:
+      summary: Manually reload the admin-ui's smart connectors within the network. This is sometimes necessary when the initial load did not pick up all SCs correctly.
+      tags:
+        - admin API
+      operationId: reloadSCs
+      responses:
+          '200':
+            description: If the SC were reloaded.
+          '500':
+            description: If a problem occurred.
 
 components:
   schemas:

smart-connector/pom.xml

@@ -47,6 +47,25 @@
 			<version>5.11.3</version>
 			<scope>test</scope>
 		</dependency>
+		<dependency>
+			<groupId>org.mockito</groupId>
+			<artifactId>mockito-core</artifactId>
+			<version>5.14.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.mockito</groupId>
+			<artifactId>mockito-junit-jupiter</artifactId>
+			<version>5.14.2</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.wiremock</groupId>
+			<artifactId>wiremock</artifactId>
+			<version>3.10.0</version>
+			<scope>test</scope>
+		</dependency>
+
 
 		<!-- jena -->
 		<dependency>
@@ -189,6 +208,16 @@
 				</exclusion>
 			</exclusions>
 		</dependency>
+		<dependency>
+			<groupId>org.eclipse.microprofile.config</groupId>
+			<artifactId>microprofile-config-api</artifactId>
+			<version>${version.eclipse.microprofile.config}</version>
+		</dependency>
+		<dependency>
+			<groupId>io.smallrye.config</groupId>
+			<artifactId>smallrye-config</artifactId>
+			<version>3.10.1</version>
+		</dependency>
 	</dependencies>
 
 	<properties>
@@ -197,6 +226,7 @@
 		<jersey3-version>3.1.9</jersey3-version>
 		<jackson-version>2.18.2</jackson-version>
 		<servlet-api-version>6.1.0</servlet-api-version>
+		<version.eclipse.microprofile.config>3.1</version.eclipse.microprofile.config>
 	</properties>
 
 	<build>

smart-connector/src/main/java/eu/knowledge/engine/smartconnector/impl/InteractionProcessorImpl.java

@@ -28,6 +28,7 @@
 import org.apache.jena.sparql.syntax.ElementData;
 import org.apache.jena.sparql.syntax.ElementGroup;
 import org.apache.jena.vocabulary.RDF;
+import org.eclipse.microprofile.config.ConfigProvider;
 import org.slf4j.Logger;
 
 import eu.knowledge.engine.reasoner.Rule;
@@ -76,8 +77,6 @@ public class InteractionProcessorImpl implements InteractionProcessor {
 	 */
 	private boolean reasonerEnabled = false;
 
-	private static boolean VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS_DEFAULT = true;
-
 	private static final Query query = QueryFactory.create(
 			"ASK WHERE { ?req <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?someClass . FILTER NOT EXISTS {?sat <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?someClass .} VALUES (?req ?sat) {} }");
 
@@ -388,9 +387,8 @@ public CompletableFuture<ReactMessage> processPostFromMessageRouter(PostMessage
 	}
 
 	private boolean shouldValidateInputOutputBindings() {
-		return SmartConnectorConfig.getBoolean(
-				SmartConnectorConfig.CONF_KEY_VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS,
-				VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS_DEFAULT);
+		return ConfigProvider.getConfig().getValue(
+				SmartConnectorConfig.CONF_KEY_VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS, Boolean.class);
 	}
 
 	@Override

smart-connector/src/main/java/eu/knowledge/engine/smartconnector/impl/MessageRouterImpl.java

@@ -10,6 +10,7 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
+import org.eclipse.microprofile.config.ConfigProvider;
 import org.slf4j.Logger;
 
 import eu.knowledge.engine.smartconnector.messaging.AnswerMessage;
@@ -30,14 +31,6 @@ public class MessageRouterImpl implements MessageRouter, SmartConnectorEndpoint
 	 */
 	private static final int MAX_ENTRIES = 5000;
 
-	/**
-	 * How many seconds should the MessageRouter wait for ANSWER/REACT Message when
-	 * sending a ASK/POST Message? 0 means wait forever (useful when working with a
-	 * human KB)
-	 */
-	private static final String CONF_KEY_WAIT_TIMEOUT = "KE_KB_WAIT_TIMEOUT";
-	private static final int DEFAULT_WAIT_TIMEOUT = 10;
-
 	private final SmartConnectorImpl smartConnector;
 	private final Map<UUID, CompletableFuture<AnswerMessage>> openAskMessages = Collections
 			.synchronizedMap(new LinkedHashMap<UUID, CompletableFuture<AnswerMessage>>() {
@@ -87,7 +80,7 @@ public MessageRouterImpl(SmartConnectorImpl smartConnector) {
 	}
 
 	private int getWaitTimeout() {
-		return Integer.parseInt(this.getConfigProperty(CONF_KEY_WAIT_TIMEOUT, Integer.toString(DEFAULT_WAIT_TIMEOUT)));
+		return ConfigProvider.getConfig().getValue(SmartConnectorConfig.CONF_KEY_KE_KB_WAIT_TIMEOUT, Integer.class);
 	}
 
 	@Override
@@ -286,21 +279,6 @@ public URI getKnowledgeBaseId() {
 		return this.smartConnector.getKnowledgeBaseId();
 	}
 
-	public String getConfigProperty(String key, String defaultValue) {
-		// We might replace this with something a bit more fancy in the future...
-		String value = System.getenv(key);
-		if (value == null) {
-			value = defaultValue;
-			LOG.trace("No value for the configuration parameter '{}' was provided, using the default value '{}'", key,
-					defaultValue);
-		}
-		return value;
-	}
-
-	public boolean hasConfigProperty(String key) {
-		return System.getenv(key) != null;
-	}
-
 	@Override
 	public void setMessageDispatcher(MessageDispatcherEndpoint messageDispatcherEndpoint) {
 		assert this.messageDispatcherEndpoint == null;

smart-connector/src/main/java/eu/knowledge/engine/smartconnector/impl/SmartConnectorConfig.java

@@ -1,14 +1,53 @@
 package eu.knowledge.engine.smartconnector.impl;
 
 public class SmartConnectorConfig {
-	public static final String CONF_KEY_VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS = "SC_VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS";
-
-	public static boolean getBoolean(String key, boolean defaultValue) {
-		String valueString = System.getenv(key);
-		if (valueString == null) {
-			return defaultValue;
-		} else {
-			return Boolean.parseBoolean(valueString);
-		}
-	}
+
+	/**
+	 * Key to set whether this KER should strictly validate whether outgoing
+	 * bindings are compatible with incoming bindings.
+	 */
+	public static final String CONF_KEY_VALIDATE_OUTGOING_BINDINGS_WRT_INCOMING_BINDINGS = "sc.validate.outgoing.bindings.wrt.incoming.bindings";
+
+	/**
+	 * Key to configure the hostname of the machine this Knowledge Engine Runtime
+	 * (KER) runs on.
+	 * 
+	 * @deprecated Replaced by
+	 *             {@link SmartConnectorConfig#CONF_KEY_KE_RUNTIME_EXPOSED_URL}
+	 */
+	@Deprecated
+	public static final String CONF_KEY_KE_RUNTIME_HOSTNAME = "ke.runtime.hostname";
+
+	/**
+	 * Key to configure the URL of the Knowledge Directory where this KER can find
+	 * other KERs in the network. Note that overriding this configuration property
+	 * will run this KER in distributed mode.
+	 */
+	public static final String CONF_KEY_KD_URL = "kd.url";
+
+	/**
+	 * Key to configure the time in seconds the SCs in this KER wait for a HTTP
+	 * connection response from another KER. Only used in distributed mode.
+	 */
+	public static final String CONF_KEY_KE_HTTP_TIMEOUT = "ke.http.timeout";
+
+	/**
+	 * Key to configure the how many seconds the MessageRouter should wait for
+	 * ANSWER/REACT Message when sending a ASK/POST Message? 0 means wait forever
+	 * (useful when working with a human KB).
+	 */
+	public static final String CONF_KEY_KE_KB_WAIT_TIMEOUT = "ke.kb.wait.timeout";
+
+	/**
+	 * Key to configure the URL that is advertised to other KERs. Other KERs can use
+	 * this URL to reach this KER. Note that this configuration property is only
+	 * used in distributed mode.
+	 */
+	public static final String CONF_KEY_KE_RUNTIME_EXPOSED_URL = "ke.runtime.exposed.url";
+
+	/**
+	 * Key to configure the port at which the KER's peer-to-peer communication
+	 * should happen. Only used in distributed mode.
+	 */
+	public static final String CONF_KEY_KE_RUNTIME_PORT = "ke.runtime.port";
 }