Plugin reload

doc/developer-guide/api/functions/TSVConnCreate.en.rst

@@ -0,0 +1,32 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed
+   with this work for additional information regarding copyright
+   ownership.  The ASF licenses this file to you under the Apache
+   License, Version 2.0 (the "License"); you may not use this file
+   except in compliance with the License.  You may obtain a copy of
+   the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+   implied.  See the License for the specific language governing
+   permissions and limitations under the License.
+
+.. include:: ../../../common.defs
+
+.. default-domain:: c
+
+TSVConnCreate
+*************
+
+Synopsis
+========
+
+`#include <ts/ts.h>`
+
+.. function:: TSCont TSVConnCreate(TSEventFunc funcp, TSMutex mutexp)
+
+Description
+===========

doc/developer-guide/design-documents/index.en.rst

@@ -0,0 +1,28 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+.. include:: ../../common.defs
+
+.. _developer-design-documents:
+
+Design Documents
+****************
+
+.. toctree::
+   :maxdepth: 1
+
+   reloading-plugins.en

doc/developer-guide/design-documents/reloading-plugins.en.rst

@@ -0,0 +1,178 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+.. include:: ../../common.defs
+
+.. _developer-plugins-reloading-plugins:
+
+Reloading Plugins
+*****************
+
+Reloading plugin allows new versions of a plugin code to be loaded and executed and old versions to be unloaded without
+restarting the traffic server process.
+
+Plugins are Dynamic Shared Objects (DSO), new versions of the plugins are currently loaded by using a traffic server
+configuration reload, i.e.::
+
+  traffic_ctl config reload
+
+Although this feature should be transparent to there plugin developers, the following are some design considerations
+and implementation details.
+
+
+Design Considerations
+=====================
+
+1. The mechanism of the plugin reload should be transparent to the plugin developers, plugin developers should be
+   concerned only with properly initializing and cleaning up after the plugin and its instances.
+
+2. With the current traffic server implementation new version plugin (re)load is only triggered by a configuration
+   (re)load hence naturally the configuration should be always coupled with the set of plugins it loaded.
+
+3. Due to its asynchronouse nature traffic server should allow running different newer and older versions of the same plugin at the same time.
+
+4. Old plugin versions should be unloaded after the traffic server process no longer needs them after reload.
+
+5. Running different versions of the configuration and plugin versions at the same time requires maintaining
+   a "current active set" to be used by new transactions, new continuations, etc. and also multiple "previous inactive" sets as well.
+
+6. The result of the plugin reloading should be consistent across operating systems, file systems, dynamic loader
+   implementations.
+
+
+Currently only loading of "remap" plugins (`remap.config`) is supported but (re)loading of "global" plugins
+(`plugin.config`) could use same ideas and reuse some of the classes below.
+
+
+Consistent (re)loading behavior
+-------------------------------
+
+The following are some of the problems noticed during the initial experimentation:
+
+  a. There is an internal reference counting of the DSOs implemented inside the dynamic loader.
+     If an older version of the plugin DSO is still loaded then loading of a newer version of the DSO by using
+     the same filename does not load the new version.
+
+  b. If the filename used by the dynamic loader reference counting contains symbolic links the results are not
+     consistent across different operating/file systems and dynamic loader implementations.
+
+The following possible solutions were considered:
+
+  a. maintaining different plugin filenames for each version - this would put unnecessary burden on the
+     configuration management tools
+
+  b. experiments with Linux specific `dlmopen <http://man7.org/linux/man-pages/man3/dlopen.3.html>`_ yielded
+     good results but it was not available on all supported platforms
+
+
+A less efficient but more reliable solution was chosen - DSO files are temporarily copied to and consequently
+loaded from a runtime location and the copies is kept until plugin is unloaded.
+
+Each configuration / plugin reload would use a different runtime location, ``ATSUuid`` is used to create unique
+runtime directories.
+
+
+Reference counting against DSOs
+-------------------------------
+
+During the initial analysis a common sense solution was considered - to add instrumentation around handling
+of registered hooks in order to unload plugins safely. This would be more involved and not sufficient since hooks
+are not the only mechanism that relies on the plugin DSO being loaded. This design / implementation proposes
+a different approach.
+
+Plugin code can be called from HTTP state machine (1) while handling HTTP transactions or (2) while calling
+event handling functions of continuations created by the plugin code.
+The plugin reload mechanism should guarantee that all necessary plugin DSOs are still loaded when those calls
+are performed.
+
+Those continuations are created by :c:func:`TSContCreate` and :c:func:`TSVConnCreate` and
+could be used for registering hooks (i.e. registered by :c:func:`TSHttpHookAdd`) or for
+scheduling events in the future (i.e. :c:func:`TSContScheduleOnPool`).
+
+
+Registering hooks always requires creating continuations from inside the plugin code and a separate
+instrumentation around handling of hooks is not necessary.
+
+There is an existing reference counting around ``UrlRewrite`` which makes sure it stays loaded until the HTTP state
+machine (the last HTTP transaction) stops using it. By making all plugins loaded by a single configuration reload
+a part of ``UrlRewrite`` (see `PluginFactory`_ below), we could guarantee the plugins are always loaded while
+in use by the HTTP transactions.
+
+
+Plugin context
+--------------
+
+Reference counting and managing different configuration and plugin sets require the continuation creation and
+destruction to know in which plugin context they are running.
+
+Traffic server API change was considered for ``TSCreateCont``, ``TSVConnCreate`` and ``TSDestroyCont`` but
+it was decided to keep things hidden from the plugin developer by using thread local plugin context which
+would be set/switched accordingly before executing the plugin code.
+
+The continuations created by the plugin will have a context member added to them which will be used by
+the reference counting and when continuations are destroyed or handle events.
+
+
+TSHttpArgs
+----------
+
+Traffic Server sessions and transactions provide a fixed array of void pointers that can be used by plugins
+to store information. To avoid collisions between plugins a plugin should first *reserve* an index in the array.
+
+Since :c:func:`TSHttpTxnArgIndexReserve` and :c:func:`TSHttpSsnArgIndexReserve` are meant to be called during plugin
+initialization we could end up "leaking" indices during plugins reload.
+Hence it is necessary to make sure only one index is allocated per "plugin identifying name", current
+:c:func:`TSHttpTxnArgIndexNameLookup` and :c:func:`TSHttpTxnArgIndexNameLookup` implementation assumes 1-1
+index-to-name relationship as well.
+
+
+PluginFactory
+-------------
+
+`PluginFactory` - creates and holds all plugin instances corresponding to a single configuration (re)load.
+
+#. Instantiates and initializes 'remap' plugins, eventually signals plugin unload/destruction, makes sure each plugin
+   version is loaded only once per configuration (re)load by maintaining a list of DSOs already loaded.
+
+#. Initializes, keeps track of all resulting plugin instances and eventually signals each instance destruction.
+
+#. Handles multiple plugin search paths.
+
+#. Sets a common runtime path for all plugins loaded in a single configuration (re)load to guarantee
+   `consistent (re)loading behavior`_.
+
+
+
+RemapPluginInfo
+---------------
+
+`RemapPluginInfo` - a class representing a 'remap' plugin, derived from `PluginDso`, and handling 'remap' plugin specific
+initialization and destruction and also sets up the right plugin context when its methods are called.
+
+
+
+PluginDso
+---------
+
+`PluginDso` - a class performing the actual DSO loading and unloading and all related initialization and
+cleanup plus related error handling. Its functionality is modularized into a separate class in hopes to
+be reused by 'global' plugins in the future.
+
+
+To make sure plugins are still loaded while their code is still in use there is reference counting done around ``PluginDso``
+which inherits ``RefCountObj`` and implements ``aqcuire()`` and ``release()`` methods which are called by ``TSCreateCont``,
+``TSVConnCreate`` and ``TSDestroyCont``.

doc/developer-guide/index.en.rst

@@ -55,4 +55,5 @@ duplicate bugs is encouraged, but not required.
    host-resolution-proposal.en
    client-session-architecture.en
    core-architecture/index.en
-   layout/index.en
+   design-documents/index.en
+   layout/index.en

include/ts/InkAPIPrivateIOCore.h

@@ -43,7 +43,7 @@ class INKContInternal : public DummyVConnection
   INKContInternal();
   INKContInternal(TSEventFunc funcp, TSMutex mutexp);
 
-  void init(TSEventFunc funcp, TSMutex mutexp);
+  void init(TSEventFunc funcp, TSMutex mutexp, void *context = 0);
   virtual void destroy();
 
   void handle_event_count(int event);
@@ -60,6 +60,7 @@ class INKContInternal : public DummyVConnection
   int m_closed;
   int m_deletable;
   int m_deleted;
+  void *m_context;
   // INKqa07670: Nokia memory leak bug fix
   INKContInternalMagic_t m_free_magic;
 };

include/tscore/ts_file.h

@@ -104,6 +104,12 @@ namespace file
     /// Get a copy of the path.
     std::string string() const;
 
+    /// Get relative path
+    self_type relative_path();
+
+    /// Get parent path
+    path parent_path();
+
   protected:
     std::string _path; ///< File path.
   };
@@ -120,6 +126,7 @@ namespace file
     friend self_type status(const path &, std::error_code &) noexcept;
 
     friend int file_type(const self_type &);
+    friend time_t modification_time(const file_status &fs);
     friend uintmax_t file_size(const self_type &);
     friend bool is_regular_file(const file_status &);
     friend bool is_dir(const file_status &);
@@ -141,6 +148,9 @@ namespace file
   /// Return the file type value.
   int file_type(const file_status &fs);
 
+  /// Return modification time
+  time_t modification_time(const file_status &fs);
+
   /// Check if the path is to a regular file.
   bool is_regular_file(const file_status &fs);
 
@@ -159,7 +169,28 @@ namespace file
   /// Check if file is readable.
   bool is_readable(const path &s);
 
-  /** Load the file at @a p into a @c std::string.
+  // Get directory location suitable for temporary files
+  path temp_directory_path();
+
+  // Returns current path.
+  path current_path();
+
+  // Returns return the canonicalized absolute pathname
+  path canonical(const path &p, std::error_code &ec);
+
+  // Checks if the file/directory exists
+  bool exists(const path &p);
+
+  // Create directories
+  bool create_directories(const path &p, std::error_code &ec, mode_t mode = 0775) noexcept;
+
+  // Copy files ("from" cannot be directory). @todo make it more generic
+  bool copy(const path &from, const path &to, std::error_code &ec);
+
+  // Removes files and directories recursively
+  bool remove(const path &path, std::error_code &ec);
+
+  /** Load the file at @a p into a @c std::string
    *
    * @param p Path to file
    * @return The contents of the file.
@@ -222,6 +253,28 @@ namespace file
     return *this /= std::string_view(that._path);
   }
 
+  inline path
+  path::relative_path()
+  {
+    return (this->is_absolute()) ? path(_path.substr(sizeof(preferred_separator))) : *this;
+  }
+
+  inline path
+  path::parent_path()
+  {
+    if (this->is_absolute() && _path.substr(sizeof(preferred_separator)).empty()) {
+      // No relative path
+      return *this;
+    }
+
+    const size_t last_slash_idx = _path.find_last_of(preferred_separator);
+    if (std::string::npos != last_slash_idx) {
+      return path(_path.substr(0, last_slash_idx));
+    } else {
+      return path();
+    }
+  }
+
   /** Combine two strings as file paths.
 
        @return A @c path with the combined path.
@@ -250,6 +303,18 @@ namespace file
     return path(std::move(lhs)) /= rhs;
   }
 
+  inline bool
+  operator==(const path &lhs, const path &rhs)
+  {
+    return lhs.string() == rhs.string();
+  }
+
+  inline bool
+  operator!=(const path &lhs, const path &rhs)
+  {
+    return lhs.string() != rhs.string();
+  }
+
   /* ------------------------------------------------------------------- */
 } // namespace file
 } // namespace ts

proxy/ReverseProxy.cc

@@ -43,7 +43,8 @@
 
 // Global Ptrs
 static Ptr<ProxyMutex> reconfig_mutex;
-UrlRewrite *rewrite_table = nullptr;
+UrlRewrite *rewrite_table                             = nullptr;
+thread_local PluginThreadContext *pluginThreadContext = nullptr;
 
 // Tokens for the Callback function
 #define FILE_CHANGED 0
@@ -150,6 +151,7 @@ reloadUrlRewrite()
     ink_assert(oldTable != nullptr);
 
     // Release the old one
+    oldTable->pluginFactory.indicateReload();
     oldTable->release();
 
     Debug("url_rewrite", "%s", msg);