Ciab atstccfg

docs/source/tools/atstccfg.rst

@@ -20,49 +20,84 @@
 ********
 atstccfg
 ********
-:program:`atstccfg` is a tool for generating configuration files server-side on :abbr:`ATC (Apache Traffic Control)` cache servers. It stores its generated/cached files in ``/tmp/atstccfg_cache/`` for re-use.
+:program:`atstccfg` is a tool for generating configuration files server-side on :abbr:`ATC (Apache Traffic Control)` cache servers.
 
 .. warning:: :program:`atstccfg` does not have a stable command-line interface, it can and will change without warning. Scripts should avoid calling it for the time being, as its only intended caller is :term:`ORT`.
 
 The source code for :program:`atstccfg` may be found in :atc-file:`traffic_ops/ort/atstccfg`, and the Go module reference is :atc-godoc:`traffic_ops/ort/atstccfg`.
 
 Usage
 =====
-``atstccfg [-u TO_URL] [-U TO_USER] [-P TO_PASSWORD] [-n] [-r N] [-e ERROR_LOCATION] [-w WARNING_LOCATION] [-i INFO_LOCATION] [-g] [-s] [-t TIMEOUT] [-a MAX_AGE] [-l] [-h] [-v]``
+- ``atstccfg -h``
+- ``atstccfg -v``
+- ``atstccfg -l``
+- ``atstccfg [-e ERROR_LOCATION] [-i INFO_LOCATION] [-p] [-P TO_PASSWORD] [-r N] [-s] [-t TIMEOUT] [-u TO_URL] [-U TO_USER] [-w WARNING_LOCATION] [-y] -n CACHE_NAME``
+- ``atstccfg [-e ERROR_LOCATION] [-i INFO_LOCATION] [-p] [-P TO_PASSWORD] [-r N] [-s] [-t TIMEOUT] [-u TO_URL] [-U TO_USER] [-w WARNING_LOCATION] -n CACHE_NAME -d DATA``
+- ``atstccfg [-e ERROR_LOCATION] [-i INFO_LOCATION] [-p] [-P TO_PASSWORD] [-r N] [-s] [-t TIMEOUT] [-u TO_URL] [-U TO_USER] [-w WARNING_LOCATION] -n CACHE_NAME -a REVAL_STATUS -q QUEUE_STATUS``
+
+When called using the fourth form, :program:`atstccfg` provides its normal output. This is the entirety of all configuration files necessary for the server, all provided at once. The output is in :mimetype:`mixed/multipart` format, defined by :rfc:`1521`. Each chunk of the message comes with the proprietary ``Path`` header, which specifies the exact location on disk of the file whose contents are contained in that chunk.
 
 Options
 -------
-.. option:: -a AGE, --cache-file-max-age-seconds AGE
+.. option:: -a REVAL_STATUS, --set-reval-status REVAL_STATUS
 
-	Sets the maximum age - in seconds - a cached response can be in order to be considered "fresh" - older files will be re-generated and cached. Default: 60
+	Sets the ``reval_pending`` property of the server in Traffic Ops. Must be 'true' or 'false'. Requires :option:`--set-queue-status` also be set. This disables normal output.
 
 .. option:: -e ERROR_LOCATION, --log-location-error ERROR_LOCATION
 
 	The file location to which to log errors. Respects the special string constants of :atc-godoc:`lib/go-log`. Default: 'stderr'
 
-.. option:: -g, --print-generated-files
+.. option:: -d DATA, --get-data DATA
 
-	If given, the names of files generated (and not proxied to Traffic Ops) will be printed to stdout, then :program:`atstccfg` will exit.
+	Specifies non-configuration-file data to retrieve from Traffic Ops. This disables normal output. Valid values are:
 
-.. option:: -i INFO_LOCATION, --log-location-info INFO_LOCATION
+	chkconfig
+		Retrieves information about the services which should be running on the :term:`cache server`. The output will be in JSON-encoded format as an array of objects with these fields:
 
-	The file location to which to log information messages. Respects the special string constants of :atc-godoc:`lib/go-log`. Default: 'stderr'
+		:name:  The name of the service. This should correspond to an existing systemd service unit file.
+		:value: A "chkconfig" line describing on which "run-levels" the services should be running. See the :manpage:`chkconfig(8)` manual pages for details on what this field means.
+
+	packages
+		Retrieves information about the packages which should exist on the :term:`cache server`. The output will be in JSON-encoded format as an array of
+		objects with these fields:
+
+		:name:    The name of the package. This should hopefully be a meaningful package name for the :term:`cache server`'s package management system.
+		:version: The version of the package which should be installed. This might also be an empty string which means "any version will do".
+
+	statuses
+		Retrieves all statuses from Traffic Ops. This is defined to be exactly the ``response`` object from the response to a GET request made to the :ref:`to-api-statuses` Traffic Ops API endpoint.
+	system-info
+		Retrieves generic information about the Traffic Control system from the :ref:`to-api-system-info` API endpoint. The output is the ``parameters`` object of the responses from GET requests to that endpoint (still JSON-encoded).
+	update-status
+		Retrieves information about the current update status using :ref:`to-api-servers-hostname-update_status`. The response is in the same format as the responses for that endpoint's GET method handler - except that that endpoint returns an array and this :program:`atstccfg` call signature returns a single one of those elements. Which one is chosen is arbitrary (hence undefined behavior when more than one server with the same hostname exists).
 
 .. option:: -h, --help
 
 	Print usage information and exit.
 
+.. option:: -i INFO_LOCATION, --log-location-info INFO_LOCATION
+
+	The file location to which to log information messages. Respects the special string constants of :atc-godoc:`lib/go-log`. Default: 'stderr'
+
 .. option:: -l, --list-plugins
 
-	List the loaded plugins and then exit.
+	List the loaded plug-ins and then exit.
+
+.. option:: -n NAME, --cache-host-name NAME
+
+	Required. Specifies the (short) hostname of the :term:`cache server` for which output will be generated. Must be the server hostname in Traffic Ops, not a URL, or :abbr:`FQDN (Fully Qualified Domain Name)`. Behavior when more than one server exists with the passed hostname is undefined.
 
-.. option:: -n, --no-cache
+.. option:: -p, --traffic-ops-disable-proxy
 
-	If given, existing cache files will not be used. Cache files will still be created, existing ones just won't be used.
+	Bypass the Traffic Ops caching proxy and make requests directly to Traffic Ops. Has no effect if no such proxy exists.
 
 .. option:: -P TO_PASSWORD, --traffic-ops-password TO_PASSWORD
 
-	Authenticate using this password - if not given, atstccfg will attempt to use the value of the :envvar:`TO_PASS` environment variable.
+	Authenticate using this password - if not given, :program:`atstccfg` will attempt to use the value of the :envvar:`TO_PASSWORD` environment variable.
+
+.. option:: -q QUEUE_STATUS, --set-queue-status QUEUE_STATUS
+
+	Sets the ``upd_pending`` property of the server identified by :option:`--cache-host-name` to the specified value, which must be 'true' or 'false'. Requires :option:`--set-reval-status` to also be set.
 
 .. option:: -r N, --num-retries N
 
@@ -76,7 +111,7 @@ Options
 
 .. option:: -t TIMEOUT, --traffic-ops-timeout-milliseconds TIMEOUT
 
-	Sets the timeout - in milliseconds - for requests made to Traffic Ops. Default: 10000
+	Sets the timeout - in milliseconds - for requests made to Traffic Ops. Default: 30000
 
 .. option:: -u TO_URL, --traffic-ops-url TO_URL
 
@@ -94,14 +129,18 @@ Options
 
 	The file location to which to log warnings. Respects the special string constants of :atc-godoc:`lib/go-log`. Default: 'stderr'
 
+.. option:: -y, --revalidate-only
+
+	When given, :program:`atstccfg` will only emit files relevant for updating content invalidation jobs. for Apache Traffic Server implementations, this limits the output to be only files named ``regex_revalidate.config``. Has no effect if :option:`--get-data` or :option:`--set-queue-status`/:option:`--set-reval-status` is/are used.
+
 Environment Variables
 ---------------------
 
 .. envvar:: TO_USER
 
 	Defines the user as whom to authenticate with Traffic Ops. This is only used if :option:`-U`/:option:`--traffic-ops-user` is not given.
 
-.. envvar:: TO_PASS
+.. envvar:: TO_PASSWORD
 
 	Defines the password to use when authenticating with Traffic Ops. This is only used if :option:`-P`/:option:`--traffic-ops-password` is not given.
 

infrastructure/cdn-in-a-box/Makefile

@@ -41,8 +41,9 @@ SPECIAL_SAUCE := $(TC_VERSION)-$(BUILD_NUMBER).$(RHEL_VERSION).x86_64.rpm
 SPECIAL_SEASONING := $(TOMCAT_VERSION).$(TOMCAT_RELEASE)-$(BUILD_NUMBER).$(RHEL_VERSION).x86_64.rpm
 
 TO_SOURCE := $(wildcard ../../traffic_ops/**/*)
-ORT_SOURCE:= $(wildcard ../../traffic_ops/bin/*)
 TO_SOURCE += $(wildcard ../../traffic_ops_db/**/*)
+ORT_SOURCE:= $(wildcard ../../traffic_ops_ort/atstccfg/**/*.go)
+ORT_SOURCE+= $(wildcard ../../lib/**/*.go)
 TM_SOURCE := $(wildcard ../../traffic_monitor/**/*)
 TP_SOURCE := $(wildcard ../../traffic_portal/**/*)
 TR_SOURCE := $(wildcard ../../traffic_router/**/*)
@@ -51,7 +52,7 @@ TS_SOURCE := $(wildcard ../../traffic_stats/**/*)
 .PHONY: clean very-clean all nearly-all debug
 
 # Default target; builds all pre-requisite rpms from source trees
-all: traffic_monitor/traffic_monitor.rpm traffic_portal/traffic_portal.rpm traffic_ops/traffic_ops.rpm traffic_router/traffic_router.rpm traffic_router/tomcat.rpm traffic_stats/traffic_stats.rpm
+all: cache/traffic_ops_ort.rpm traffic_monitor/traffic_monitor.rpm traffic_portal/traffic_portal.rpm traffic_ops/traffic_ops.rpm traffic_router/traffic_router.rpm traffic_router/tomcat.rpm traffic_stats/traffic_stats.rpm
 
 debug: PKG_FLAGS += -d
 ifneq ($(MAKECMDGOALS), debug)
@@ -73,6 +74,8 @@ traffic_router/tomcat.rpm: ../../dist/tomcat-$(SPECIAL_SEASONING)
 	cp -f $? $@
 traffic_stats/traffic_stats.rpm: ../../dist/traffic_stats-$(SPECIAL_SAUCE)
 	cp -f $? $@
+cache/traffic_ops_ort.rpm: ../../dist/traffic_ops_ort-$(SPECIAL_SAUCE)
+	cp -f $? $@
 
 # Dist rpms
 ../../dist/traffic_monitor-$(SPECIAL_SAUCE): $(TM_SOURCE)
@@ -90,8 +93,11 @@ traffic_stats/traffic_stats.rpm: ../../dist/traffic_stats-$(SPECIAL_SAUCE)
 ../../dist/traffic_stats-$(SPECIAL_SAUCE): $(TS_SOURCE)
 	../../pkg $(PKG_FLAGS) traffic_stats_build
 
+../../dist/traffic_ops_ort-$(SPECIAL_SAUCE): $(ORT_SOURCE)
+	../../pkg $(PKG_FLAGS) traffic_ops_ort_build
+
 clean:
-	$(RM) traffic_monitor/traffic_monitor.rpm traffic_ops/traffic_ops.rpm traffic_portal/traffic_portal.rpm traffic_router/traffic_router.rpm traffic_router/tomcat.rpm edge/traffic_ops_ort.rpm mid/traffic_ops_ort.rpm traffic_stats/traffic_stats.rpm
+	$(RM) traffic_monitor/traffic_monitor.rpm traffic_ops/traffic_ops.rpm traffic_portal/traffic_portal.rpm traffic_router/traffic_router.rpm traffic_router/tomcat.rpm cache/traffic_ops_ort.rpm traffic_stats/traffic_stats.rpm
 
 very-clean: clean
 	$(warning This will destroy ALL OUTPUT RPMS IN 'dist'. Please be sure this is what you want)

infrastructure/cdn-in-a-box/cache/Dockerfile

@@ -25,17 +25,21 @@ FROM centos:7
 
 EXPOSE 80
 
+
 ADD https://ci.trafficserver.apache.org/RPMS/CentOS7/trafficserver-7.1.4-2.el7.x86_64.rpm /trafficserver.rpm
 ADD https://ci.trafficserver.apache.org/RPMS/CentOS7/trafficserver-devel-7.1.4-2.el7.x86_64.rpm /trafficserver-devel.rpm
 
 RUN yum install -y bind-utils kyotocabinet-libs epel-release initscripts iproute net-tools nmap-ncat gettext autoconf automake libtool gcc-c++ cronie glibc-devel openssl-devel
 RUN yum install -y /trafficserver.rpm /trafficserver-devel.rpm jq python36-psutil python36-typing python36-setuptools python36-pip logrotate && yum clean all
 RUN python3 -m pip install --upgrade pip && python3 -m pip install requests urllib3 distro
 
+
 ADD traffic_server/plugins/astats_over_http/astats_over_http.c traffic_server/plugins/astats_over_http/Makefile.am /
 
 RUN tsxs -v -c astats_over_http.c -o astats_over_http.so
-RUN mkdir -p /usr/libexec/trafficserver /opt/ort && tsxs -v -o astats_over_http.so -i
+
+# The symbolic link here is a shim for broken atstccfg behavior - remove when it's fixed.
+RUN mkdir -p /usr/libexec/trafficserver /opt/ort /opt/trafficserver/etc/trafficserver/ /opt/init.d && ln -s /opt/trafficserver/etc/trafficserver/ssl /etc/trafficserver/ssl && tsxs -v -o astats_over_http.so -i
 
 RUN yum remove -y gcc-c++ glibc-devel autoconf automake libtool && rm -f /astats_over_http.c /Makefile.am
 
@@ -48,11 +52,23 @@ RUN mkdir -p /var/trafficserver /opt/ort && \
 
 RUN setcap CAP_NET_BIND_SERVICE=+eip /bin/traffic_server && setcap CAP_NET_BIND_SERVICE=+eip /bin/traffic_manager && setcap CAP_NET_BIND_SERVICE=+eip /bin/trafficserver && setcap CAP_NET_BIND_SERVICE=+eip /bin/traffic_cop
 
-ADD infrastructure/cdn-in-a-box/ort /opt/ort/
-
 WORKDIR /opt
 
-RUN touch /var/log/ort.log && pip3 install Apache-TrafficControl==1.1.3 && pip3 install ./ort && cp ort/traffic_ops_ort.crontab /etc/cron.d/traffic_ops_ort-cron-template && mkdir -p /opt/init.d && cp ort/traffic_ops_ort.logrotate /etc/logrotate.d/ort
+ADD infrastructure/cdn-in-a-box/ort /opt/ort/
+ADD traffic_control/clients/python /opt/Apache-TrafficControl/
+
+RUN touch /var/log/ort.log && \
+	pip3 install ./Apache-TrafficControl && \
+	pip3 install ./ort && \
+	cp ort/traffic_ops_ort.crontab /etc/cron.d/traffic_ops_ort-cron-template && \
+	cp ort/traffic_ops_ort.logrotate /etc/logrotate.d/ort
 
 ADD infrastructure/cdn-in-a-box/cache/run.sh infrastructure/cdn-in-a-box/traffic_ops/to-access.sh infrastructure/cdn-in-a-box/enroller/server_template.json /
+
+ARG ORT_RPM=infrastructure/cdn-in-a-box/cache/traffic_ops_ort.rpm
+ADD $ORT_RPM /
+RUN yum install -y /$(basename $ORT_RPM) &&\
+    rm /$(basename $ORT_RPM) && \
+    yum -y clean all
+
 CMD /run.sh

infrastructure/cdn-in-a-box/edge/run.sh

@@ -75,7 +75,7 @@ until [[ $(to-get api/2.0/cdns/name/$CDN_NAME/sslkeys | jq '.response | length')
 done
 
 # Leaves the container hanging open in the event of a failure for debugging purposes
-traffic_ops_ort -kl ALL BADASS || { echo "Failed"; }
+PATH="$PATH:/opt/ort" traffic_ops_ort -kl ALL BADASS || { echo "Failed"; }
 
 envsubst < "/etc/cron.d/traffic_ops_ort-cron-template" > "/var/spool/cron/root" && rm -f "/etc/cron.d/traffic_ops_ort-cron-template"
 crontab "/var/spool/cron/root"
@@ -87,4 +87,4 @@ until grep -q demo1 /etc/trafficserver/remap.config; do
 done
 
 touch /var/log/trafficserver/diags.log
-tail -Fn +1 /var/log/trafficserver/diags.log
+tail -Fn +1 /var/log/trafficserver/diags.log /var/log/ort.log

infrastructure/cdn-in-a-box/mid/run.sh

@@ -69,12 +69,12 @@ while [[ -z "$(testenrolled)" ]]; do
 done
 
 # Leaves the container hanging open in the event of a failure for debugging purposes
-traffic_ops_ort -k BADASS ALL "https://$TO_FQDN:$TO_PORT" "$TO_ADMIN_USER:$TO_ADMIN_PASSWORD" || { echo "Failed"; }
+PATH="$PATH:/opt/ort" traffic_ops_ort -k BADASS ALL "https://$TO_FQDN:$TO_PORT" "$TO_ADMIN_USER:$TO_ADMIN_PASSWORD" || { echo "Failed"; }
 
 envsubst < "/etc/cron.d/traffic_ops_ort-cron-template" > "/etc/cron.d/traffic_ops_ort-cron" && rm -f "/etc/cron.d/traffic_ops_ort-cron-template"
 chmod "0644" "/etc/cron.d/traffic_ops_ort-cron" && crontab "/etc/cron.d/traffic_ops_ort-cron"
 
 crond -im off
 
 touch /var/log/trafficserver/diags.log
-tail -Fn +1 /var/log/trafficserver/diags.log
+tail -Fn +1 /var/log/trafficserver/diags.log /var/log/ort.log

infrastructure/cdn-in-a-box/ort/README.rst

@@ -29,4 +29,4 @@ Running Tests
 
 To run tests:
 
-    python3 -m unittest discover -s tests
+    ./doctest-runner.py

infrastructure/cdn-in-a-box/ort/tests/__init__.py → infrastructure/cdn-in-a-box/ort/doctest-runner.py

@@ -1,3 +1,5 @@
+#!/usr/bin/python3
+
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -14,3 +16,14 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+
+if __name__ == "__main__":
+	from traffic_ops_ort import config_files, configuration, main_routines, packaging, services, to_api, utils
+	import doctest
+	doctest.testmod(config_files)
+	doctest.testmod(configuration)
+	doctest.testmod(main_routines)
+	doctest.testmod(packaging)
+	doctest.testmod(services)
+	doctest.testmod(to_api)
+	doctest.testmod(utils)

infrastructure/cdn-in-a-box/ort/setup.py

@@ -60,8 +60,7 @@
 	],
 	keywords='network connection configuration TrafficControl',
 	packages=find_packages(exclude=['contrib', 'docs', 'tests']),
-	install_requires=['setuptools', 'typing', 'requests', 'urllib3', 'distro', 'psutil', 'Apache-TrafficControl==1.1.3'],
-	extras_require={'dev': ['unittest']},
+	install_requires=['setuptools', 'typing', 'requests', 'urllib3', 'distro', 'psutil', 'Apache-TrafficControl'],
 	# data_files=[('etc/crontab', ['traffic_ops_ort.crontab'])],
 	entry_points={
 		'console_scripts': [

infrastructure/cdn-in-a-box/ort/traffic_ops_ort.crontab

@@ -14,4 +14,4 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-*/1 * * * * /usr/local/bin/traffic_ops_ort -k --dispersion 0 SYNCDS ALL $TO_URL $TO_USER:$TO_PASSWORD >> /var/log/ort.log 2>&1
+*/1 * * * * PATH=$PATH:/opt/ort /usr/local/bin/traffic_ops_ort -k --dispersion 0 SYNCDS ALL $TO_URL $TO_USER:$TO_PASSWORD >> /var/log/ort.log 2>&1

infrastructure/cdn-in-a-box/ort/traffic_ops_ort/__init__.py

@@ -70,6 +70,10 @@
 
 	Print version information and exit
 
+.. option:: -t, --timeout
+
+	Sets the timeout in milliseconds for connections to Traffic Ops.
+
 .. option:: -k, --insecure
 
 	An optional flag which, when used, disables the checking of SSL certificates for validity
@@ -330,11 +334,12 @@
 ===============
 """
 
-__version__ = "0.1.1"
+__version__ = "0.2.0"
 __author__  = "Brennan Fieck"
 
 import argparse
 import datetime
+from distutils.spawn import find_executable
 import logging
 import os
 import random
@@ -484,6 +489,10 @@ def main() -> int:
 	                    help="Skip verification of SSL certificates for Traffic Ops connections. "\
 	                         "DON'T use this in production!",
 	                    action="store_true")
+	parser.add_argument("-t", "--timeout",
+	                    help="Sets the timeout in milliseconds for requests made to Traffic Ops.",
+	                    type=int,
+	                    default=None)
 	parser.add_argument("-v", "--version",
 	                    action="version",
 	                    version="%(prog)s v"+__version__,
@@ -530,5 +539,8 @@ def main() -> int:
 			print("(Hint: use -h/--help for usage)", file=sys.stderr)
 			return 1
 
+	if not find_executable("atstccfg"):
+		print("Could not find atstccfg executable - this is required to run ORT!", file=sys.stderr)
+		return 1
 
 	return doMain(args)