Emulated vault

CHANGELOG.md

@@ -69,6 +69,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added validation to prevent assigning servers to delivery services without required capabilities.
 - Added deep coverage zone routing percentage to the Traffic Portal dashboard.
 - Added a `traffic_ops/app/bin/osversions-convert.pl` script to convert the `osversions.cfg` file from Perl to JSON as part of the `/osversions` endpoint rewrite.
+- Added [Experimental] - Emulated Vault suppling a HTTP server mimicking RIAK behavior for usage as traffic-control vault.
 
 ### Changed
 - Traffic Router:  TR will now allow steering DSs and steering target DSs to have RGB enabled. (fixes #3910)

experimental/emulated_vault/README.md

@@ -0,0 +1,51 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Emulated Vault - Background
+
+The emulated_vault module supplies a HTTP server mimicking RIAK behavior for usage as traffic-control vault.
+It may be used in order to replace RIAK traffic_vault, as it is much more simple to install.
+The server may use different type of persistent storage (e.g. file-system), using the proper adapter. 
+The resiliency of the stored keys is derived from the resiliency of the underlying storage.
+
+# Installation
+
+Basic requirements: Centos ver >= 7; Python >= 2.7
+
+In order to install the module on a server please:
+1. Copy the module files to the server's root
+2. Add the certificate and key to your favorite path
+3. Adjust /opt/emulated_vault/conf/cfg.json - pointing at your certificate and key
+4. "systemctl enable" the service
+
+Logs may be found under /opt/emulated_vault/var/log
+
+# Developer's Notes
+
+If you just want to play around with the module, you may of course run the server script on its own.
+Before doing that, you would probably need to adjust the opt/emulated_vault/conf/cfg.json:
+1. Changing the db-path to one you have access to
+2. Disable ssl (just to make it easier)
+
+Additionally, the vault-debug script is also available to work against the DB with command line. 
+It is mostly useful when developing a new adapter.
+
+# Contact
+
+For additional information, questions or assistance, please approach [Nir B. Sopher](mailto:nir@apache.org)

experimental/emulated_vault/etc/systemd/system/emulated-vault.service

@@ -0,0 +1,29 @@
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+#
+[Unit]
+Description=Emulated Vault Service
+After=network.target
+
+[Service]
+Type=simple
+User=root
+ExecStart=/opt/emulated-vault/vault-server
+Restart=on-failure
+
+
+[Install]
+WantedBy=multi-user.target

experimental/emulated_vault/opt/emulated-vault/conf/cfg.ini

@@ -0,0 +1,30 @@
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+#
+
+[general]
+storage-adapter-type = fs
+# Optional: log-dir = /var/log/messages/emulated-vault
+[http-server]
+ssl-key-path = path/to/cert/key.pem
+ssl-cert-path = path/to/cert/cert.pem
+# Optional: use-ssl = False
+# Optional: listen-ip = 1.2.3.4
+# Optional: listen-port = 12345
+
+[fs-adapter]
+db-base-os-path = /opt/emulated-vault/db
+ping-os-path = /opt/emulated-vault/ping

experimental/emulated_vault/opt/emulated-vault/storage/__init__.py

@@ -0,0 +1,16 @@
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#	 http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+#

experimental/emulated_vault/opt/emulated-vault/storage/adapter_base.py

@@ -0,0 +1,163 @@
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#	 http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+#
+
+import sys
+if sys.version_info >= (3, 0):
+	from abc import ABC, abstractmethod	
+else:	
+	ABC = object
+	abstractmethod = lambda f: f
+
+
+
+class AdapterBase(ABC):
+	"""
+	Base adapter class.
+	This class implements the API required for storing and retriving the content kept in the vault.
+
+	Methods to be implemented at derived classes:
+	:meth:`get_parameter_storage_path` given a url-key of a parameter return the storage path
+	:meth:`get_parameter_storage_path_from_url_path` given a storage path of a parameter
+	retrun the url-key
+	:meth:`init_cfg` given a config-parser object, read the parameters required for 
+	adapter's operation. Return "success" bool value
+	:meth:`init` prepare the adapter - connecting to storgae / setting the conntent for 
+	"ping" requests etc.. Return "success" bool value
+	:meth:`ping` test the 'ping' status with the adapter. 
+	Return a tuple: "success" bool & "value" kept as ping variable
+	:meth:`read_parameter_by_storage_path` given a storage path retrieve the parameter value.
+	Return a tuple: "success" bool & "value" kept in the parameter
+	:meth:`read_parameters_by_storage_path` given a storage and and a key holding 
+	filters on the key and values.
+	Return "success" bool indicating a sucessful write, and a key->value dictionary 
+	for the relevant parameters
+	:meth:`write_parameter_by_storage_path` given a storage path and a value string,
+	keep the parameter value. Return "success" bool indicating a sucessful write
+	:meth:`remove_parameter_by_storage_path` given a storage path, delete the parameter
+	from the DB. Return "success" bool indicating a sucessful deletion
+	"""
+
+	@abstractmethod
+	def init_cfg(self, fullConfig):# -> bool:
+		"""
+		Method to be implemented at derived classes
+		Initialize the class basic parameters. Part of Adapter required API.
+		:param fullConfig: configuration to operate upon.
+		:type fullConfig: configparser.ConfigParser class
+		:return: 'True' for successful initialization
+		:rtype: bool
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def init(self):# -> bool:
+		"""
+		Method to be implemented at derived classes
+		Initialize the class - e.g. connection to storage & ability to answer for ping. 
+		:return: 'True' for successful initialization
+		:rtype: bool
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def get_parameter_storage_path(self, parameterUrlPath):# -> (bool, str):
+		"""
+		Method to be implemented at derived classes
+		Conversion function - taking a key's path and translate to a file path on the file system
+		:param parameterUrlPath: the "url-path" like key of the variable
+		:type parameterUrlPath: str
+		:return: "success" bool and a file path of where the value is be kept
+		:rtype: Tuple[bool, str]
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def get_parameter_url_path_from_storage_path(self, parameterStoragePath):# -> (bool, str):
+		"""
+		Method to be implemented at derived classes
+		Conversion function - taking file path on the file system and translate to key's path
+		:param parameterStoragePath: the file name holding a value
+		:type parameterUrlPath: str
+		:return: "success" bool and the matching variable url-path like key
+		:rtype: Tuple[bool, str]
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def ping(self):# -> bool:
+		"""
+		Method to be implemented at derived classes
+		Check ping connection
+		:return: 'True' for successful connection with the storage layer
+		:rtype: bool
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def read_parameter_by_storage_path(self, parameterStoragePath):# -> (bool, str):
+		"""
+		Method to be implemented at derived classes
+		Reading the value from the provided file name.
+		:param parameterStoragePath: the file name 
+		:type parameterUrlPath: str
+		:return: 'True' for successful retrivaland the retrieved value
+		:rtype: Tuple[bool, str]
+		"""
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def read_parameters_by_storage_path(self, parameterStoragePathPrefix, keyFilters):# -> (bool, dict(str,str)):
+		"""
+		Method to be implemented at derived classes
+		Reading the values of the parameters the provided directory.
+		:param parameterStoragePathPrefix: the directory to look into
+		:type parameterStoragePathPrefix: str
+		:param keyFilters: filter-name/filter-func dict, holding functions that get a key as 
+		input and retunn "true" if key should be included in the result
+		:type keyFilters: Dict[str,function[str]]
+		:return: 'True' for successful retrival and a dict for key-name/value 
+		:rtype: Tuple[bool, Dict[str, str]]
+		"""
+		raise NotImplementedError()#...
+
+
+	@abstractmethod	
+	def write_parameter_by_storage_path(self, parameterStoragePath, value):# -> bool:
+		"""
+		Method to be implemented at derived classes
+		Writing the value to the provided file name.
+		:param parameterStoragePath: the file name 
+		:type parameterUrlPath: str
+		:param value: value to be writen
+		:type parameterUrlPath: str
+		:return: 'True' for successful writing
+		:rtype: bool		
+	   """
+		raise NotImplementedError()#...
+
+	@abstractmethod	
+	def remove_parameter_by_storage_path(self, parameterStoragePath):# -> bool:
+		"""
+		Method to be implemented at derived classes
+		Deleting the the provided file.
+		:param parameterStoragePath: the file name 
+		:type parameterUrlPath: str
+		:return: 'True' for successful deletion
+		:rtype: bool		
+		"""
+		raise NotImplementedError()#...
+