TP Delivery Service Generate SSL update, new letsencrypt generate and renew API endpoints

CHANGELOG.md

@@ -9,6 +9,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Updated Traffic Router to read new EDSN0 client subnet field and route accordingly only for enabled delivery services. When enabled and a subnet is present in the request, the subnet appears in the `chi` field and the resolver address is in the `rhi` field.
 - Added an optimistic quorum feature to Traffic Monitor to prevent false negative states from propagating to downstream components in the event of network isolation.
 - Added the ability to fetch users by role
+- Added an API 1.5 endpoint to generate delivery service certificates using Let's Encrypt
+- Added an API 1.5 endpoint to GET a single or all records for Let's Encrypt DNS challenge
+- Added an API 1.5 endpoint to renew certificates
 - Traffic Ops Golang Endpoints
   - /api/2.0 for all of the most recent route versions
   - /api/1.1/cachegroupparameters/{{cachegroupID}}/{{parameterID}} `(DELETE)`
@@ -17,6 +20,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
   - /api/1.5/stats_summary `(POST)`
   - /api/1.1/cdns/routing
   - /api/1.1/cachegroupparameters/ `(GET, POST)`
+  - /api/1.5/deliveryservices/sslkeys/generate/letsencrypt `POST`
+  - /api/1.5/letsencrypt/autorenew `POST`
+  - /api/1.5/letsencrypt/dnsrecords `GET`
 
 ### Changed
 - Fix to traffic_ops_ort.pl to strip specific comment lines before checking if a file has changed.  Also promoted a changed file message from DEBUG to ERROR for report mode.

LICENSE

@@ -452,3 +452,7 @@ For the ogier/pflag (commit 45c278a) component:
 For the errors (commit 27936f6) component:
 @vendor/github.com/pkg/errors/*
 ./vendor/github.com/pkg/errors/LICENSE
+
+For the go-acme/lego (version 2.7.2) component:
+@traffic_ops/traffic_ops_golang/vendor/github.com/go-acme/lego/*
+./traffic_ops/traffic_ops_golang/vendor/github.com/go-acme/lego/LICENSE

docs/source/admin/traffic_ops.rst

@@ -337,6 +337,16 @@ This file deals with the configuration parameters of running Traffic Ops itself.
 
 	.. warning:: While relative paths are allowed, they are discouraged, as the path will be relative to the working directory of the `traffic_ops_golang`_ process itself, not relative to the ``cdn.conf`` configuration file, which can be confusing.
 
+:lets_encrypt:
+
+	.. versionadded:: 4.1
+
+	:user_email: An optional email address to create an account with Let's Encrypt or to receive expiration updates
+	:send_expiration_email: A boolean option to send email summarizing certificate expiration status
+	:convert_self_signed: A boolean option to convert self signed to Let's Encrypt certificates as they expire. This only works for certificates labeled as Self Signed in the Certificate Source field.
+	:renew_days_before_expiration: Set the number of days before expiration date to renew certificates.
+	:environment: This specifies which Let's Encrypt environment to use: 'staging' or 'production'. It defaults to 'production'.
+
 :portal: This section provides information regarding a connected UI with which users interact, so that emails can include links to it.
 
 	:base_url: This URL should be the root and/or landing page of the UI. For Traffic Portal instances, this should include the fragment part of the URL, e.g. ``https://trafficportal.infra.ciab.test/#!/``.

docs/source/admin/traffic_router.rst

@@ -691,8 +691,70 @@ The ordering of certificates within the certificate bundle matters. It must be:
 
 To see the ordering of certificates you may have to manually split up your certificate chain and use :manpage:`openssl(1ssl)` on each individual certificate
 
-Suggested Way of Setting up an HTTPS Delivery Service
------------------------------------------------------
+Let's Encrypt
+-------------
+Let’s Encrypt is a free, automated :abbr:`CA (Certificate Authority)` using :abbr:`ACME (Automated Certificate Management Environment)` protocol. Let's Encrypt performs a domain validation before issuing or renewing a certificate. There are several options for domain validation but for this application the DNS challenge is used in order to receive wildcard certificates. Let's Encrypt sends a token to be used as a TXT record at ``_acme-challenge.domain.example.com`` and after verifying that the token is accessible there, will return the newly generated and signed certificate and key. The basic workflow implemented is:
+
+#. ``POST`` to Let's Encrypt and receive the DNS challenge token.
+#. Traffic Ops stores the DNS challenge.
+#. Traffic Router has a watcher which checks with Traffic Ops for any new challenges or deleted challenges.
+#. When a new record appears, Traffic Router temporarily adds a static route for the specified :term:`Delivery Service` with the token from Let's Encrypt at ``_acme-challenge.domain.example.com``.
+#. Let's Encrypt continuously attempts to resolve it as a TXT record to verify ownership of the domain.
+
+.. Note:: DNSSec should be turned on for any CDN using Let's Encrypt to guard against a 'Man in the Middle' interference with this transaction.
+
+#. Let's Encrypt returns the signed certificate and key to Traffic Ops.
+#. Traffic Ops stores the certificate and key in Traffic Vault and removes the DNS challenge record.
+#. The Traffic Router watcher removes the TXT record.
+
+Let's Encrypt can be set up through :ref:`cdn.conf` by updating the following fields:
+
+.. table:: Fields to update for Let's Encrypt under `lets_encrypt`
+
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+	| Name                         | Type    | Description                                                                                                                                                      |
+	+==============================+=========+==================================================================================================================================================================+
+	| user_email                   | string  | Optional. Email to create account with Let's Encrypt or to receive expiration updates                                                                            |
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+	| send_expiration_email        | boolean | Option to send email summarizing certificate expiration status                                                                                                   |
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+	| convert_self_signed          | boolean | Option to convert self signed to Let's Encrypt certificates as they expire. Only works for certificates labeled as Self Signed in the Certificate Source field.  |
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+	| renew_days_before_expiration | int     | Number of days before expiration date to renew certificates                                                                                                      |
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+	| environment                  | string  | Let's Encrypt environment to use.  Options are 'staging' or 'production'. Defaults to 'production'                                                               |
+	+------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+.. table:: Fields to update for sending emails under `smtp`
+
+	+------------+------------------+----------------------------------------------------------------------+
+	| Name       | Type             | Description                                                          |
+	+============+==================+======================================================================+
+	| enabled    | boolean          | Enable sending emails through Simple Mail Transfer Protocol (SMTP)   |
+	+------------+------------------+----------------------------------------------------------------------+
+	| user       | string           | User for SMTP server access                                          |
+	+------------+------------------+----------------------------------------------------------------------+
+	| password   | string           | Password for SMTP server access                                      |
+	+------------+------------------+----------------------------------------------------------------------+
+	| address    | string           | SMTP server address including port                                   |
+	+------------+------------------+----------------------------------------------------------------------+
+
+
+Suggested Way of Setting up an HTTPS Delivery Service With Let's Encrypt Automation
+-----------------------------------------------------------------------------------
+Assuming you have already created a :term:`Delivery Service` which you plan to modify to use HTTPS, do the following in Traffic Portal:
+
+#. Select one of '1 - HTTPS', '2 - HTTP AND HTTPS', or '3 - HTTP TO HTTPS' for the protocol field of a :term:`Delivery Service` and click the :guilabel:`Update` button
+#. Go to :menuselection:`More --> Manage SSL Keys`
+#. Click on :menuselection:`More --> Generate SSL Keys`
+#. Click on the :guilabel:`Use Let's Encrypt` slider, click on the green :guilabel:`Generate Keys` button, then confirm that you want to make these changes
+#. Take a new CDN :term:`Snapshot`
+
+Once this is done you should be able to verify that you are being correctly redirected by Traffic Router using e.g. :manpage:`curl(1)` commands to HTTPS destinations on your :term:`Delivery Service`.
+
+
+Suggested Way of Setting up an HTTPS Delivery Service With Certificate Authority
+--------------------------------------------------------------------------------
 Assuming you have already created a :term:`Delivery Service` which you plan to modify to use HTTPS, do the following in Traffic Portal:
 
 #. Select one of '1 - HTTPS', '2 - HTTP AND HTTPS', or '3 - HTTP TO HTTPS' for the protocol field of a :term:`Delivery Service` and click the :guilabel:`Update` button

docs/source/api/deliveryservices_hostname_name_sslkeys.rst

@@ -46,11 +46,11 @@ Request Structure
 .. code-block:: http
 	:caption: Request Example
 
-GET /api/1.4/deliveryservices/hostname/video.demo1.mycdn.ciab.test/sslkeys HTTP/1.1
-Host: trafficops.infra.ciab.test
-User-Agent: curl/7.47.0
-Accept: */*
-Cookie: mojolicious=...
+	GET /api/1.4/deliveryservices/hostname/video.demo1.mycdn.ciab.test/sslkeys HTTP/1.1
+	Host: trafficops.infra.ciab.test
+	User-Agent: curl/7.47.0
+	Accept: */*
+	Cookie: mojolicious=...
 
 Response Structure
 ------------------
@@ -70,9 +70,13 @@ Response Structure
 :hostname:        The hostname generated by Traffic Ops that is used as the common name when generating the certificate - this will be an :abbr:`FQDN (Fully Qualified Domain Name)` for DNS :term:`Delivery Services` and a wildcard URL for HTTP :term:`Delivery Services`
 :organization:    An optional field which, if present, contains the organization entered by the user when generating certificate\ [#optional]_
 :state:           An optional field which, if present, contains the state entered by the user when generating certificate\ [#optional]_
-:version:         The version of the certificate record in Traffic Vault
+:version:         An integer that defines the "version" of the key - which may be thought of as the sequential generation; that is, the higher the number the more recent the key
 
-.. code- block:: http
+:expiration:      The expiration date of the certificate for the :term:`Delivery Service` in :rfc:`3339` format
+
+	.. versionadded:: 1.5
+
+.. code-block:: http
 	:caption: Response Example
 
 	HTTP/1.1 200 OK
@@ -97,7 +101,8 @@ Response Structure
 			"crt": "...",
 			"key": "...",
 			"csr": "..."
-		}
+		},
+		"expiration": "2020-08-18T13:53:06Z"
 	}}
 
 .. note:: The response example uses abbreviated values for the ``crt``, ``key``, and ``csr``, as these will generally be very large, base64-encoded SSL keys and certificates. Note that in general the output of this request should **not** be made available, as the ``key`` field contains the *private* SSL key corresponding to the certificate.

docs/source/api/deliveryservices_sslkeys_generate.rst

@@ -39,7 +39,7 @@ Request Structure
 :organization: An optional field which, if present, will represent the organization for which the SSL certificate was generated
 :state:        An optional field which, if present, will represent the resident state or province of the generated SSL certificate
 :businessUnit: An optional field which, if present, will represent the business unit for which the SSL certificate was generated
-:version:      version of the keys being generated
+:version:      An integer that defines the "version" of the key - which may be thought of as the sequential generation; that is, the higher the number the more recent the key
 
 .. code-block:: http
 	:caption: Request Example

docs/source/api/deliveryservices_sslkeys_generate_letsencrypt.rst

@@ -0,0 +1,64 @@
+..
+..
+.. Licensed under the Apache License, Version 2.0 (the "License");
+.. you may not use this file except in compliance with the License.
+.. You may obtain a copy of the License at
+..
+..     http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+..
+
+.. _to-api-deliveryservices-sslkeys-generate-letsencrypt:
+
+*************************************************
+``deliveryservices/sslkeys/generate/letsencrypt``
+*************************************************
+
+.. versionadded:: 1.5
+
+``POST``
+========
+Generates an SSL certificate and private key using Let's Encrypt for a :term:`Delivery Service`
+
+:Auth. Required: Yes
+:Roles Required: "admin" or "operations"
+:Response Type:  Object (string)
+
+Request Structure
+-----------------
+:key:             The :ref:`ds-xmlid` of the :term:`Delivery Service` for which keys will be generated
+:version:         An integer that defines the "version" of the key - which may be thought of as the sequential generation; that is, the higher the number the more recent the key
+:hostname:        The desired hostname of the :term:`Delivery Service`
+
+	.. note:: In most cases, this must be the same as the :ref:`ds-example-urls`.
+
+:cdn:             The name of the CDN of the :term:`Delivery Service` for which the certs will be generated
+
+.. code-block:: http
+	:caption: Request Example
+
+	POST /api/1.5/deliveryservices/sslkeys/generate/letsencrypt HTTP/1.1
+	Content-Type: application/json
+
+	{
+		"key": "ds-01",
+		"version": "3",
+		"hostname": "tr.ds-01.ott.kabletown.com",
+		"cdn":"test-cdn"
+	}
+
+
+Response Structure
+------------------
+.. code-block:: json
+	:caption: Response Example
+
+	{ "alerts": [{
+		"level": "success",
+		"text": "Beginning async call to Let's Encrypt for ds-01.  This may take a few minutes."
+	}]}

docs/source/api/deliveryservices_xmlid_xmlid_sslkeys.rst

@@ -71,9 +71,13 @@ Response Structure
 :hostname:        The hostname generated by Traffic Ops that is used as the common name when generating the certificate - this will be a FQDN for DNS :term:`Delivery Services` and a wildcard URL for HTTP :term:`Delivery Services`
 :organization:    An optional field which, if present, contains the organization entered by the user when generating certificate\ [1]_
 :state:           An optional field which, if present, contains the state entered by the user when generating certificate\ [1]_
-:version:         The version of the certificate record in Riak
+:version:         An integer that defines the "version" of the key - which may be thought of as the sequential generation; that is, the higher the number the more recent the key
 
-.. code- block:: http
+:expiration:      The expiration date of the certificate for the :term:`Delivery Service` in :rfc:`3339` format
+
+	.. versionadded:: 1.5
+
+.. code-block:: http
 	:caption: Response Example
 
 	HTTP/1.1 200 OK
@@ -93,7 +97,9 @@ Response Structure
 		"hostname": "foober.com",
 		"country": "US",
 		"state": "Colorado",
-		"version": "1"
+		"version": "1",
+		"expiration": "2020-08-18T13:53:06Z"
 	}}
 
+
 .. [1] These optional fields will be present in the response if and only if they were specified during key generation; they are optional during key generation and thus cannot be guaranteed to exist or not exist.