From cb45aa1d8a5bfa7beb6fb1141344d5e2d8b81f9f Mon Sep 17 00:00:00 2001 From: Rawlin Peters Date: Mon, 7 Mar 2022 17:21:07 -0700 Subject: [PATCH 1/4] Reduce TM dependency on CRConfig Do not use CRConfig data when the same data already exists in the TMConfig. For data that TM needs but doesn't exist in TMConfig, add it to TMConfig. This way, TM only uses data in the TMConfig and can become simply a shuttle for the CRConfig without actually using any CRConfig data. Closes: #6512 Related: #1739 Related: #5367 --- CHANGELOG.md | 1 + .../api/v2/cdns_name_configs_monitoring.rst | 68 ++++++--- .../api/v3/cdns_name_configs_monitoring.rst | 66 ++++++--- .../api/v4/cdns_name_configs_monitoring.rst | 66 ++++++--- lib/go-tc/enum.go | 8 ++ lib/go-tc/traffic_monitor.go | 32 +++-- lib/go-tc/traffic_monitor_test.go | 2 +- lib/go-tc/traffic_router.go | 123 ++--------------- lib/go-tc/traffic_router_test.go | 2 +- traffic_monitor/health/cache.go | 8 +- traffic_monitor/health/cache_test.go | 1 - traffic_monitor/manager/monitorconfig.go | 2 +- traffic_monitor/manager/opsconfig.go | 14 -- .../tests/_integration/snapshot.json | 2 +- traffic_monitor/todata/todata.go | 130 ++++++++++++------ traffic_monitor/todata/todata_test.go | 16 +-- traffic_monitor/towrap/towrap.go | 96 +------------ .../traffic_ops_golang/crconfig/crconfig.go | 3 +- .../crconfig/deliveryservice.go | 11 +- .../crconfig/edgelocations_test.go | 4 +- .../traffic_ops_golang/crconfig/servers.go | 53 +------ .../crconfig/servers_test.go | 39 ++---- .../dbhelpers/db_helpers.go | 33 +++++ .../monitoring/monitoring.go | 71 +++++++--- .../monitoring/monitoring_test.go | 48 ++++++- traffic_ops/traffic_ops_golang/test/rand.go | 38 +++++ .../topologies.go => topology/snapshot.go} | 6 +- .../snapshot_test.go} | 13 +- 28 files changed, 494 insertions(+), 462 deletions(-) create mode 100644 traffic_ops/traffic_ops_golang/test/rand.go rename traffic_ops/traffic_ops_golang/{crconfig/topologies.go => topology/snapshot.go} (91%) rename traffic_ops/traffic_ops_golang/{crconfig/topologies_test.go => topology/snapshot_test.go} (94%) diff --git a/CHANGELOG.md b/CHANGELOG.md index a217fa3c3e..28ea5e0c2e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). - Added Rocky Linux 8 support - Traffic Monitors now peer with other Traffic Monitors of the same status (e.g. ONLINE with ONLINE, OFFLINE with OFFLINE), instead of all peering with ONLINE. - Changed the Traffic Ops user last_authenticated update query to only update once per minute to avoid row-locking when the same user logs in frequently. +- Added new fields to the monitoring.json snapshot and made Traffic Monitor prefer data in monitoring.json to the CRConfig snapshot - Added permissions to the role form in traffic portal - Updated the Cache Stats Traffic Portal page to use a more performant AG-Grid-based table. - Updated the CDNs Traffic Portal page to use a more performant AG-Grid-based table. diff --git a/docs/source/api/v2/cdns_name_configs_monitoring.rst b/docs/source/api/v2/cdns_name_configs_monitoring.rst index e4d6590185..13dc3ae855 100644 --- a/docs/source/api/v2/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v2/cdns_name_configs_monitoring.rst @@ -58,9 +58,12 @@ Response Structure :deliveryServices: An array of objects representing each :term:`Delivery Service` provided by this CDN + :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status + :topology: A string that is the name of the Delivery Service's topology (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle + :type: A string that is the Delivery Service's type category (HTTP or DNS) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -80,29 +83,33 @@ Response Structure :type: A string that names the :ref:`Profile's Type ` +:topologies: A map of :term:`Topology` names to objects + + :nodes: An array of strings which are the names of the EDGE_LOC-type cache groups in the topology + :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance - :hostname: The hostname of the server running this Traffic Monitor instance - :ip6: The IPv6 address of this Traffic Monitor - when applicable - :ip: The IPv4 address of this Traffic Monitor - :port: The port on which this Traffic Monitor listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor - :status: The status of the server running this Traffic Monitor instance + :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance + :hostname: The hostname of the server running this Traffic Monitor instance + :ip6: The IPv6 address of this Traffic Monitor - when applicable + :ip: The IPv4 address of this Traffic Monitor + :port: The port on which this Traffic Monitor listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor + :status: The status of the server running this Traffic Monitor instance :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN - :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address - :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons - :hostName: The (short) hostname of the :term:`cache server` - :interfacename: The name of the network interface device being used by the :term:`cache server`'s HTTP proxy - :ip6: The :term:`cache server`'s IPv6 address - when applicable - :ip: The :term:`cache server`'s IPv4 address - :port: The port on which the :term:`cache server` listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` - :status: The status of the :term:`cache server` - :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` + :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs + :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address + :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons + :hostName: The (short) hostname of the :term:`cache server` + :port: The port on which the :term:`cache server` listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` + :status: The status of the :term:`cache server` + :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` + :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. .. code-block:: http :caption: Response Example @@ -120,6 +127,13 @@ Response Structure Transfer-Encoding: chunked { "response": { + "topologies": { + "example-topology": { + "nodes": [ + "CDN_in_a_Box_Edge" + ] + } + }, "trafficServers": [ { "profile": "ATS_EDGE_TIER_CACHE", @@ -127,7 +141,7 @@ Response Structure "ip": "172.16.239.100", "ip6": "fc01:9400:1000:8::100", "port": 80, - "cachegroup": "CDN_in_a_Box_Edge", + "cacheGroup": "CDN_in_a_Box_Edge", "hostname": "edge", "fqdn": "edge.infra.ciab.test", "interfacename": "eth0", @@ -140,7 +154,7 @@ Response Structure "ip": "172.16.239.120", "ip6": "fc01:9400:1000:8::120", "port": 80, - "cachegroup": "CDN_in_a_Box_Mid", + "cacheGroup": "CDN_in_a_Box_Mid", "hostname": "mid", "fqdn": "mid.infra.ciab.test", "interfacename": "eth0", @@ -207,7 +221,19 @@ Response Structure } } ], - "deliveryServices": [], + "deliveryServices": [ + { + "xmlId": "example-ds", + "totalTpsThreshold": 0, + "status": "REPORTED", + "totalKbpsThreshold": 0, + "type": "DNS", + "topology": "example-topology", + "hostRegexes": [ + ".*\\.example-ds\\..*" + ] + } + ], "config": { "health.polling.interval": 6000, "heartbeat.polling.interval": 3000, diff --git a/docs/source/api/v3/cdns_name_configs_monitoring.rst b/docs/source/api/v3/cdns_name_configs_monitoring.rst index e4b9a136bc..1b95ad38a5 100644 --- a/docs/source/api/v3/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v3/cdns_name_configs_monitoring.rst @@ -58,9 +58,12 @@ Response Structure :deliveryServices: An array of objects representing each :term:`Delivery Service` provided by this CDN + :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status + :topology: A string that is the name of the Delivery Service's topology (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle + :type: A string that is the Delivery Service's type category (HTTP or DNS) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -80,27 +83,33 @@ Response Structure :type: A string that names the :ref:`Profile's Type ` +:topologies: A map of :term:`Topology` names to objects + + :nodes: An array of strings which are the names of the EDGE_LOC-type cache groups in the topology + :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance - :hostname: The hostname of the server running this Traffic Monitor instance - :ip6: The IPv6 address of this Traffic Monitor - when applicable - :ip: The IPv4 address of this Traffic Monitor - :port: The port on which this Traffic Monitor listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor - :status: The status of the server running this Traffic Monitor instance + :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance + :hostname: The hostname of the server running this Traffic Monitor instance + :ip6: The IPv6 address of this Traffic Monitor - when applicable + :ip: The IPv4 address of this Traffic Monitor + :port: The port on which this Traffic Monitor listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor + :status: The status of the server running this Traffic Monitor instance :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN - :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address - :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons - :hostName: The (short) hostname of the :term:`cache server` - :port: The port on which the :term:`cache server` listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` - :status: The status of the :term:`cache server` - :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` - :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. + :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs + :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address + :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons + :hostName: The (short) hostname of the :term:`cache server` + :port: The port on which the :term:`cache server` listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` + :status: The status of the :term:`cache server` + :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` + :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. :ipAddresses: A set of objects representing IP Addresses assigned to this network interface. In most scenarios, only one or two (usually one IPv4 address and one IPv6 address) will be present, but it is illegal for this set to be an empty collection. @@ -132,12 +141,19 @@ Response Structure Transfer-Encoding: chunked { "response": { + "topologies": { + "example-topology": { + "nodes": [ + "CDN_in_a_Box_Edge" + ] + } + }, "trafficServers": [ { "profile": "ATS_EDGE_TIER_CACHE", "status": "REPORTED", "port": 80, - "intefaces": [ + "interfaces": [ { "ipAddresses": [ { @@ -167,7 +183,7 @@ Response Structure "profile": "ATS_MID_TIER_CACHE", "status": "REPORTED", "port": 80, - "intefaces": [ + "interfaces": [ { "ipAddresses": [ { @@ -253,7 +269,19 @@ Response Structure } } ], - "deliveryServices": [], + "deliveryServices": [ + { + "xmlId": "example-ds", + "totalTpsThreshold": 0, + "status": "REPORTED", + "totalKbpsThreshold": 0, + "type": "DNS", + "topology": "example-topology", + "hostRegexes": [ + ".*\\.example-ds\\..*" + ] + } + ], "config": { "health.polling.interval": 6000, "heartbeat.polling.interval": 3000, diff --git a/docs/source/api/v4/cdns_name_configs_monitoring.rst b/docs/source/api/v4/cdns_name_configs_monitoring.rst index 34d0770ce9..cb4229824d 100644 --- a/docs/source/api/v4/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v4/cdns_name_configs_monitoring.rst @@ -59,9 +59,12 @@ Response Structure :deliveryServices: An array of objects representing each :term:`Delivery Service` provided by this CDN + :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status + :topology: A string that is the name of the Delivery Service's topology (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle + :type: A string that is the Delivery Service's type category (HTTP or DNS) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -81,27 +84,33 @@ Response Structure :type: A string that names the :ref:`Profile's Type ` +:topologies: A map of :term:`Topology` names to objects + + :nodes: An array of strings which are the names of the EDGE_LOC-type cache groups in the topology + :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance - :hostname: The hostname of the server running this Traffic Monitor instance - :ip6: The IPv6 address of this Traffic Monitor - when applicable - :ip: The IPv4 address of this Traffic Monitor - :port: The port on which this Traffic Monitor listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor - :status: The status of the server running this Traffic Monitor instance + :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance + :hostname: The hostname of the server running this Traffic Monitor instance + :ip6: The IPv6 address of this Traffic Monitor - when applicable + :ip: The IPv4 address of this Traffic Monitor + :port: The port on which this Traffic Monitor listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this Traffic Monitor + :status: The status of the server running this Traffic Monitor instance :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN - :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address - :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons - :hostName: The (short) hostname of the :term:`cache server` - :port: The port on which the :term:`cache server` listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` - :status: The status of the :term:`cache server` - :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` - :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. + :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs + :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address + :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons + :hostName: The (short) hostname of the :term:`cache server` + :port: The port on which the :term:`cache server` listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` + :status: The status of the :term:`cache server` + :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` + :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. :ipAddresses: A set of objects representing IP Addresses assigned to this network interface. In most scenarios, only one or two (usually one IPv4 address and one IPv6 address) will be present, but it is illegal for this set to be an empty collection. @@ -133,12 +142,19 @@ Response Structure Transfer-Encoding: chunked { "response": { + "topologies": { + "example-topology": { + "nodes": [ + "CDN_in_a_Box_Edge" + ] + } + }, "trafficServers": [ { "profile": "ATS_EDGE_TIER_CACHE", "status": "REPORTED", "port": 80, - "intefaces": [ + "interfaces": [ { "ipAddresses": [ { @@ -168,7 +184,7 @@ Response Structure "profile": "ATS_MID_TIER_CACHE", "status": "REPORTED", "port": 80, - "intefaces": [ + "interfaces": [ { "ipAddresses": [ { @@ -254,7 +270,19 @@ Response Structure } } ], - "deliveryServices": [], + "deliveryServices": [ + { + "xmlId": "example-ds", + "totalTpsThreshold": 0, + "status": "REPORTED", + "totalKbpsThreshold": 0, + "type": "DNS", + "topology": "example-topology", + "hostRegexes": [ + ".*\\.example-ds\\..*" + ] + } + ], "config": { "health.polling.interval": 6000, "heartbeat.polling.interval": 3000, diff --git a/lib/go-tc/enum.go b/lib/go-tc/enum.go index 3bb9da6316..014a624485 100644 --- a/lib/go-tc/enum.go +++ b/lib/go-tc/enum.go @@ -315,6 +315,14 @@ func DSTypeCategoryFromString(s string) DSTypeCategory { } } +// GetDSTypeCategory returns the delivery service type category (either HTTP or DNS) of the given delivery service type. +func GetDSTypeCategory(dsType string) string { + if strings.HasPrefix(dsType, "DNS") { + return "DNS" + } + return "HTTP" +} + // These are the allowable values for the Signing Algorithm property of a // Delivery Service. const ( diff --git a/lib/go-tc/traffic_monitor.go b/lib/go-tc/traffic_monitor.go index f3c08247dd..04e3f15055 100644 --- a/lib/go-tc/traffic_monitor.go +++ b/lib/go-tc/traffic_monitor.go @@ -84,6 +84,9 @@ type TrafficMonitorConfig struct { // servers (those given in TrafficServers), which are stored here to // avoid potentially lengthy reiteration. Profiles []TMProfile `json:"profiles,omitempty"` + // Topologies is the set of topologies defined in Traffic Ops, consisting + // of just the EDGE_LOC-type cachegroup nodes. + Topologies map[string]CRConfigTopology `json:"topologies"` } const healthThresholdAvailableBandwidthInKbps = "availableBandwidthInKbps" @@ -123,6 +126,7 @@ func (tmc *TrafficMonitorConfig) ToLegacyConfig() LegacyTrafficMonitorConfig { Profiles: tmc.Profiles, DeliveryServices: tmc.DeliveryServices, TrafficServers: servers, + Topologies: tmc.Topologies, } return legacy } @@ -130,12 +134,13 @@ func (tmc *TrafficMonitorConfig) ToLegacyConfig() LegacyTrafficMonitorConfig { // LegacyTrafficMonitorConfig represents TrafficMonitorConfig for ATC versions // before 5.0. type LegacyTrafficMonitorConfig struct { - TrafficServers []LegacyTrafficServer `json:"trafficServers,omitempty"` - CacheGroups []TMCacheGroup `json:"cacheGroups,omitempty"` - Config map[string]interface{} `json:"config,omitempty"` - TrafficMonitors []TrafficMonitor `json:"trafficMonitors,omitempty"` - DeliveryServices []TMDeliveryService `json:"deliveryServices,omitempty"` - Profiles []TMProfile `json:"profiles,omitempty"` + TrafficServers []LegacyTrafficServer `json:"trafficServers,omitempty"` + CacheGroups []TMCacheGroup `json:"cacheGroups,omitempty"` + Config map[string]interface{} `json:"config,omitempty"` + TrafficMonitors []TrafficMonitor `json:"trafficMonitors,omitempty"` + DeliveryServices []TMDeliveryService `json:"deliveryServices,omitempty"` + Profiles []TMProfile `json:"profiles,omitempty"` + Topologies map[string]CRConfigTopology `json:"topologies,omitempty"` } // Upgrade converts a legacy TM Config to the newer structure. @@ -150,6 +155,7 @@ func (s *LegacyTrafficMonitorConfig) Upgrade() *TrafficMonitorConfig { Profiles: s.Profiles, TrafficMonitors: s.TrafficMonitors, TrafficServers: make([]TrafficServer, 0, len(s.TrafficServers)), + Topologies: s.Topologies, } for _, ts := range s.TrafficServers { upgraded.TrafficServers = append(upgraded.TrafficServers, ts.Upgrade()) @@ -190,6 +196,8 @@ type TrafficMonitorConfigMap struct { DeliveryService map[string]TMDeliveryService // Profile is a map of Profile Names to TMProfile objects. Profile map[string]TMProfile + // Topology is a map of Topology names to CRConfigTopology structs. + Topology map[string]CRConfigTopology } // ToLegacy converts a Stats to a LegacyStats. @@ -461,10 +469,13 @@ type MonitoringCoordinates struct { // necessary for Traffic Monitor to do its job of monitoring health and // statistics. type TMDeliveryService struct { - XMLID string `json:"xmlId"` - TotalTPSThreshold int64 `json:"TotalTpsThreshold"` - ServerStatus string `json:"status"` - TotalKbpsThreshold int64 `json:"TotalKbpsThreshold"` + XMLID string `json:"xmlId"` + TotalTPSThreshold int64 `json:"TotalTpsThreshold"` + ServerStatus string `json:"status"` + TotalKbpsThreshold int64 `json:"TotalKbpsThreshold"` + Topology string `json:"topology"` + Type string `json:"type"` + HostRegexes []string `json:"hostRegexes"` } // TMProfile is primarily a collection of the Parameters with special meaning @@ -634,6 +645,7 @@ func TrafficMonitorTransformToMap(tmConfig *TrafficMonitorConfig) (*TrafficMonit tm.TrafficMonitor = make(map[string]TrafficMonitor, len(tmConfig.TrafficMonitors)) tm.DeliveryService = make(map[string]TMDeliveryService, len(tmConfig.DeliveryServices)) tm.Profile = make(map[string]TMProfile, len(tmConfig.Profiles)) + tm.Topology = tmConfig.Topologies for _, trafficServer := range tmConfig.TrafficServers { tm.TrafficServer[trafficServer.HostName] = trafficServer diff --git a/lib/go-tc/traffic_monitor_test.go b/lib/go-tc/traffic_monitor_test.go index 343603db0f..8b7e5c86a6 100644 --- a/lib/go-tc/traffic_monitor_test.go +++ b/lib/go-tc/traffic_monitor_test.go @@ -291,7 +291,7 @@ func ExampleLegacyTrafficMonitorConfigMap_Upgrade() { TrafficServer: map[string]LegacyTrafficServer{ "test": { CacheGroup: "test", - DeliveryServices: []tsdeliveryService{}, + DeliveryServices: []string{}, FQDN: "test.quest", HashID: "test", HostName: "test", diff --git a/lib/go-tc/traffic_router.go b/lib/go-tc/traffic_router.go index 5421644790..3f025e4eba 100644 --- a/lib/go-tc/traffic_router.go +++ b/lib/go-tc/traffic_router.go @@ -71,13 +71,6 @@ type SOA struct { RetrySecondsTime time.Time `json:"-"` } -// MissLocation is a geographic location that will be used in the event that a -// client request cannot be localized (a "Geo Miss"). -type MissLocation struct { - Latitude float64 `json:"latitude"` - Longitude float64 `json:"longitude"` -} - // MatchSet structures are a list of MatchList structures with an associated // Protocol. type MatchSet struct { @@ -91,92 +84,6 @@ type MatchList struct { MatchType string `json:"match-type"` } -// A BypassDestination is an FQDN, IP address, or CNAME to use for redirection -// in the event that bypass becomes necessary. -type BypassDestination struct { - FQDN string `json:"fqdn"` - Type string `json:"type"` - Port int `json:"Port"` -} - -// TTLS is a structure that contains Time-To-Live values for different types of -// DNS records (in seconds). -type TTLS struct { - Arecord int `json:"A"` - SoaRecord int `json:"SOA"` - NsRecord int `json:"NS"` - AaaaRecord int `json:"AAAA"` -} - -// TrafficRouter is a subset of a server that contains all the information a -// Traffic Router instance may need to know about one of its peers. -type TrafficRouter struct { - Port int `json:"port"` - IP6 string `json:"ip6"` - IP string `json:"ip"` - FQDN string `json:"fqdn"` - Profile string `json:"profile"` - Location string `json:"location"` - ServerStatus string `json:"status"` - APIPort int `json:"apiPort"` -} - -// TrafficRouterConfig is the json unmarshalled without any changes -// note all structs are local to this file _except_ the TrafficRouterConfig struct. -type TrafficRouterConfig struct { - TrafficServers []TrafficServer `json:"trafficServers,omitempty"` - TrafficMonitors []TrafficMonitor `json:"trafficMonitors,omitempty"` - TrafficRouters []TrafficRouter `json:"trafficRouters,omitempty"` - CacheGroups []TMCacheGroup `json:"cacheGroups,omitempty"` - DeliveryServices []TRDeliveryService `json:"deliveryServices,omitempty"` - Stats map[string]interface{} `json:"stats,omitempty"` - Config map[string]interface{} `json:"config,omitempty"` -} - -// TrafficRouterConfigMap is a set of mappings of various objects important to -// Traffic Router to the structural data a Traffic Router needs to know about -// them. -// -// In general, the objects herein contained do not strictly correspond to other -// structures that represent the same concept defined in this package - Traffic -// Router only gets exactly what information it needs, as keeping the size of -// this structure low is important for performance. -// -// TODO: Why isn't this used instead of legacy-named things like CRConfig? -type TrafficRouterConfigMap struct { - TrafficServer map[string]TrafficServer - TrafficMonitor map[string]TrafficMonitor - TrafficRouter map[string]TrafficRouter - CacheGroup map[string]TMCacheGroup - DeliveryService map[string]TRDeliveryService - Config map[string]interface{} - Stat map[string]interface{} -} - -// TRDeliveryService structures contain all the information about a Delivery -// Service that are important to Traffic Routers. -type TRDeliveryService struct { - XMLID string `json:"xmlId"` - Domains []string `json:"domains"` - RoutingName string `json:"routingName"` - MissLocation MissLocation `json:"missCoordinates"` - CoverageZoneOnly bool `json:"coverageZoneOnly"` - MatchSets []MatchSet `json:"matchSets"` - TTL int `json:"ttl"` - TTLs TTLS `json:"ttls"` - BypassDestination BypassDestination `json:"bypassDestination"` - StatcDNSEntries []StaticDNS `json:"statitDnsEntries"` - Soa SOA `json:"soa"` -} - -// A StaticDNS is a somewhat arbitrary static DNS entry of some kind. -type StaticDNS struct { - Value string `json:"value"` - TTL int `json:"ttl"` - Name string `json:"name"` - Type string `json:"type"` -} - // A LegacyTrafficServer is a representation of a cache server containing a // subset of the information available in a server structure that conveys all // the information important for Traffic Router and Traffic Monitor to handle @@ -185,19 +92,19 @@ type StaticDNS struct { // Deprecated: The configuration versions that use this structure to represent // a cache server are deprecated, new code should use TrafficServer instead. type LegacyTrafficServer struct { - CacheGroup string `json:"cacheGroup"` - DeliveryServices []tsdeliveryService `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids - FQDN string `json:"fqdn"` - HashID string `json:"hashId"` - HostName string `json:"hostName"` - HTTPSPort int `json:"httpsPort,omitempty"` - InterfaceName string `json:"interfaceName"` - IP string `json:"ip"` - IP6 string `json:"ip6"` - Port int `json:"port"` - Profile string `json:"profile"` - ServerStatus string `json:"status"` - Type string `json:"type"` + CacheGroup string `json:"cacheGroup"` + DeliveryServices []string `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids + FQDN string `json:"fqdn"` + HashID string `json:"hashId"` + HostName string `json:"hostName"` + HTTPSPort int `json:"httpsPort,omitempty"` + InterfaceName string `json:"interfaceName"` + IP string `json:"ip"` + IP6 string `json:"ip6"` + Port int `json:"port"` + Profile string `json:"profile"` + ServerStatus string `json:"status"` + Type string `json:"type"` } // Upgrade upgrades the LegacyTrafficServer into its modern-day equivalent. @@ -299,8 +206,8 @@ func (ts *TrafficServer) ToLegacyServer() LegacyTrafficServer { // TrafficServer represents a cache server for use by Traffic Monitor and // Traffic Router instances. type TrafficServer struct { - CacheGroup string `json:"cacheGroup"` - DeliveryServices []tsdeliveryService `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids + CacheGroup string `json:"cachegroup"` + DeliveryServices []string `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids FQDN string `json:"fqdn"` HashID string `json:"hashId"` HostName string `json:"hostName"` diff --git a/lib/go-tc/traffic_router_test.go b/lib/go-tc/traffic_router_test.go index ed8af64388..7090253865 100644 --- a/lib/go-tc/traffic_router_test.go +++ b/lib/go-tc/traffic_router_test.go @@ -88,7 +88,7 @@ func ExampleTrafficServer_IPv6() { func ExampleLegacyTrafficServer_Upgrade() { lts := LegacyTrafficServer{ CacheGroup: "testCG", - DeliveryServices: []tsdeliveryService{}, + DeliveryServices: []string{}, FQDN: "test.quest", HashID: "test", HostName: "test", diff --git a/traffic_monitor/health/cache.go b/traffic_monitor/health/cache.go index 5247eeda73..8ce26311ec 100644 --- a/traffic_monitor/health/cache.go +++ b/traffic_monitor/health/cache.go @@ -388,7 +388,7 @@ func CalcAvailability( localCacheStatuses[result.ID] = availStatus } - calculateDeliveryServiceState(toData.DeliveryServiceServers, localStates) + calculateDeliveryServiceState(localStates) localCacheStatusThreadsafe.Set(localCacheStatuses) } @@ -433,13 +433,9 @@ func eventDesc(status tc.CacheStatus, message string) string { } //calculateDeliveryServiceState calculates the state of delivery services from the new cache state data `cacheState` and the CRConfig data `deliveryServiceServers` and puts the calculated state in the outparam `deliveryServiceStates` -func calculateDeliveryServiceState(deliveryServiceServers map[tc.DeliveryServiceName][]tc.CacheName, states peer.CRStatesThreadsafe) { +func calculateDeliveryServiceState(states peer.CRStatesThreadsafe) { deliveryServices := states.GetDeliveryServices() for deliveryServiceName, deliveryServiceState := range deliveryServices { - if _, ok := deliveryServiceServers[deliveryServiceName]; !ok { - log.Infof("CRConfig does not have delivery service %s, but traffic monitor poller does; skipping\n", deliveryServiceName) - continue - } // NOTE: DisabledLocations is always empty, and it's important that it isn't nil, so it serialises to the JSON `[]` instead of `null`. // It's no longer populated due to it being an unnecessary optimization for Traffic Router, but the field is kept for compatibility. deliveryServiceState.DisabledLocations = []tc.CacheGroupName{} diff --git a/traffic_monitor/health/cache_test.go b/traffic_monitor/health/cache_test.go index 013847b8ff..847d82931a 100644 --- a/traffic_monitor/health/cache_test.go +++ b/traffic_monitor/health/cache_test.go @@ -412,7 +412,6 @@ func TestCalcAvailabilityThresholds(t *testing.T) { ServerCachegroups: map[tc.CacheName]tc.CacheGroupName{}, } toData.ServerTypes[tc.CacheName(result.ID)] = tc.CacheTypeEdge - toData.DeliveryServiceServers["myDS"] = []tc.CacheName{tc.CacheName(result.ID)} toData.ServerCachegroups[tc.CacheName(result.ID)] = "myCG" localCacheStatusThreadsafe := threadsafe.NewCacheAvailableStatus() diff --git a/traffic_monitor/manager/monitorconfig.go b/traffic_monitor/manager/monitorconfig.go index c2103de1aa..93d66091da 100644 --- a/traffic_monitor/manager/monitorconfig.go +++ b/traffic_monitor/manager/monitorconfig.go @@ -225,7 +225,7 @@ func monitorConfigListen( monitorConfig := pollerMonitorCfg.Cfg cdn := pollerMonitorCfg.CDN monitorConfigTS.Set(monitorConfig) - if err := toData.Update(toSession, cdn); err != nil { + if err := toData.Update(toSession, cdn, monitorConfig); err != nil { log.Errorln("Updating Traffic Ops Data: " + err.Error()) } diff --git a/traffic_monitor/manager/opsconfig.go b/traffic_monitor/manager/opsconfig.go index 64a8d97b0e..dda98119d2 100644 --- a/traffic_monitor/manager/opsconfig.go +++ b/traffic_monitor/manager/opsconfig.go @@ -201,20 +201,6 @@ func StartOpsConfigManager( newOpsConfig.CdnName = cdn } - // fixed an issue when traffic_monitor receives corrupt data, CRConfig, from traffic_ops. - // Will loop and retry until a good CRConfig is received from traffic_ops - backoff.Reset() - for { - if err := toData.Fetch(toSession, newOpsConfig.CdnName); err != nil { - handleErr(fmt.Errorf("Error getting Traffic Ops data: %v\n", err)) - duration := backoff.BackoffDuration() - log.Errorf("retrying in %v\n", duration) - time.Sleep(duration) - continue - } - break - } - // These must be in a goroutine, because the monitorConfigPoller tick sends to a channel this select listens for. Thus, if we block on sends to the monitorConfigPoller, we have a livelock race condition. // More generically, we're using goroutines as an infinite chan buffer, to avoid potential livelocks for _, subscriber := range opsConfigChangeSubscribers { diff --git a/traffic_monitor/tests/_integration/snapshot.json b/traffic_monitor/tests/_integration/snapshot.json index e720abf389..2e6dc7fd0c 100644 --- a/traffic_monitor/tests/_integration/snapshot.json +++ b/traffic_monitor/tests/_integration/snapshot.json @@ -89,7 +89,7 @@ "matchlist": [ { "regex": "\\.*ds0\\.*", - "match-type": "regex" + "match-type": "HOST" } ] } diff --git a/traffic_monitor/todata/todata.go b/traffic_monitor/todata/todata.go index 0148a03f63..9548ea0c5f 100644 --- a/traffic_monitor/todata/todata.go +++ b/traffic_monitor/todata/todata.go @@ -29,7 +29,7 @@ import ( "github.com/apache/trafficcontrol/lib/go-tc" "github.com/apache/trafficcontrol/traffic_monitor/towrap" - "github.com/json-iterator/go" + jsoniter "github.com/json-iterator/go" ) // Regexes maps Delivery Service Regular Expressions to delivery services. @@ -132,6 +132,7 @@ type CRConfig struct { Protocol string `json:"protocol"` MatchList []struct { Regex string `json:"regex"` + Type string `json:"match-type"` } `json:"matchlist"` } `json:"matchsets"` } `json:"deliveryServices"` @@ -140,20 +141,11 @@ type CRConfig struct { } } -// Fetch gets the CRConfig from Traffic Ops, creates the TOData maps, and atomically sets the TOData. -// TODO since the session is threadsafe, each TOData get func below could be put in a goroutine, if performance mattered -func (d TODataThreadsafe) Fetch(to towrap.TrafficOpsSessionThreadsafe, cdn string) error { - if _, err := to.CRConfigRaw(cdn); err != nil { - return fmt.Errorf("Error getting CRconfig from Traffic Ops: %v", err) - } - return d.Update(to, cdn) -} - // Update updates the TOData data with the last fetched CDN -func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn string) error { +func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn string, mc tc.TrafficMonitorConfigMap) error { crConfigBytes, _, err := to.LastCRConfig(cdn) if err != nil { - return fmt.Errorf("Error getting last CRConfig: %v", err) + return fmt.Errorf("getting last CRConfig: %v", err) } newTOData := TOData{} @@ -162,32 +154,29 @@ func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn stri json := jsoniter.ConfigFastest err = json.Unmarshal(crConfigBytes, &crConfig) if err != nil { - return fmt.Errorf("Error unmarshalling CRconfig: %v", err) + return fmt.Errorf("unmarshalling CRconfig: %v", err) } - newTOData.DeliveryServiceServers, newTOData.ServerDeliveryServices, err = getDeliveryServiceServers(crConfig) - if err != nil { - return err - } + // TODO: remove the fallback behavior on the CRConfig in ATC 8.0 (https://github.com/apache/trafficcontrol/issues/6627) + newTOData.DeliveryServiceServers, newTOData.ServerDeliveryServices = getDeliveryServiceServers(crConfig, mc) - newTOData.DeliveryServiceTypes, err = getDeliveryServiceTypes(crConfig) + // TODO: remove the fallback behavior on the CRConfig in ATC 8.0 (https://github.com/apache/trafficcontrol/issues/6627) + newTOData.DeliveryServiceTypes, err = getDeliveryServiceTypes(crConfig, mc) if err != nil { - return fmt.Errorf("Error getting delivery service types from Traffic Ops: %v\n", err) + return fmt.Errorf("getting delivery service types from Traffic Ops: %v\n", err) } - newTOData.DeliveryServiceRegexes, err = getDeliveryServiceRegexes(crConfig) + // TODO: remove the fallback behavior on the CRConfig in ATC 8.0 (https://github.com/apache/trafficcontrol/issues/6627) + newTOData.DeliveryServiceRegexes, err = getDeliveryServiceRegexes(crConfig, mc) if err != nil { - return fmt.Errorf("Error getting delivery service regexes from Traffic Ops: %v\n", err) + return fmt.Errorf("getting delivery service regexes from Traffic Ops: %v\n", err) } - newTOData.ServerCachegroups, err = getServerCachegroups(crConfig) - if err != nil { - return fmt.Errorf("Error getting server cachegroups from Traffic Ops: %v\n", err) - } + newTOData.ServerCachegroups = getServerCachegroups(mc) - newTOData.ServerTypes, err = getServerTypes(crConfig) + newTOData.ServerTypes, err = getServerTypes(mc) if err != nil { - return fmt.Errorf("Error getting server types from Traffic Ops: %v\n", err) + return fmt.Errorf("getting server types from monitoring config: %v\n", err) } d.set(newTOData) @@ -195,11 +184,41 @@ func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn stri } // getDeliveryServiceServers gets the servers on each delivery services, for the given CDN, from Traffic Ops. -func getDeliveryServiceServers(crc CRConfig) (map[tc.DeliveryServiceName][]tc.CacheName, map[tc.CacheName][]tc.DeliveryServiceName, error) { +func getDeliveryServiceServers(crc CRConfig, mc tc.TrafficMonitorConfigMap) (map[tc.DeliveryServiceName][]tc.CacheName, map[tc.CacheName][]tc.DeliveryServiceName) { dsServers := map[tc.DeliveryServiceName][]tc.CacheName{} serverDses := map[tc.CacheName][]tc.DeliveryServiceName{} topologyCacheGroupDses := map[string][]tc.DeliveryServiceName{} + + if canUseMonitorConfig(mc) { + for deliveryServiceName, deliveryService := range mc.DeliveryService { + if deliveryService.Topology == "" { + continue + } + dsName := tc.DeliveryServiceName(deliveryServiceName) + for _, cacheGroup := range mc.Topology[deliveryService.Topology].Nodes { + topologyCacheGroupDses[cacheGroup] = append(topologyCacheGroupDses[cacheGroup], dsName) + } + } + + for serverName, serverData := range mc.TrafficServer { + srvName := tc.CacheName(serverName) + if cacheGroupDses, inTopology := topologyCacheGroupDses[serverData.CacheGroup]; inTopology { + for _, deliveryServiceName := range cacheGroupDses { + dsServers[deliveryServiceName] = append(dsServers[deliveryServiceName], srvName) + } + serverDses[srvName] = append(serverDses[srvName], cacheGroupDses...) + } + for _, deliveryServiceName := range serverData.DeliveryServices { + dsName := tc.DeliveryServiceName(deliveryServiceName) + dsServers[dsName] = append(dsServers[dsName], srvName) + serverDses[srvName] = append(serverDses[srvName], dsName) + } + + } + return dsServers, serverDses + } + for deliveryServiceName, deliveryService := range crc.DeliveryServices { if deliveryService.Topology == "" { continue @@ -222,20 +241,34 @@ func getDeliveryServiceServers(crc CRConfig) (map[tc.DeliveryServiceName][]tc.Ca } } - return dsServers, serverDses, nil + return dsServers, serverDses } // getDeliveryServiceRegexes gets the regexes of each delivery service, for the given CDN, from Traffic Ops. // Returns a map[deliveryService][]regex. -func getDeliveryServiceRegexes(crc CRConfig) (Regexes, error) { +func getDeliveryServiceRegexes(crc CRConfig, mc tc.TrafficMonitorConfigMap) (Regexes, error) { dsRegexes := map[tc.DeliveryServiceName][]string{} + if canUseMonitorConfig(mc) { + for dsName, dsData := range mc.DeliveryService { + if len(dsData.HostRegexes) < 1 { + log.Warnln("TMConfig missing regex for delivery service " + string(dsName)) + continue + } + dsRegexes[tc.DeliveryServiceName(dsName)] = dsData.HostRegexes + } + return createRegexes(dsRegexes) + } + for dsName, dsData := range crc.DeliveryServices { for _, matchset := range dsData.Matchsets { if len(matchset.MatchList) < 1 { log.Warnln("CRConfig missing regex for delivery service '" + string(dsName) + "' matchset protocol '" + matchset.Protocol + "'") continue } + if matchset.MatchList[0].Type != "HOST" { + continue + } dsRegexes[dsName] = append(dsRegexes[dsName], matchset.MatchList[0].Regex) } if len(dsRegexes[dsName]) == 0 { @@ -286,31 +319,50 @@ func createRegexes(dsToRegex map[tc.DeliveryServiceName][]string) (Regexes, erro // getServerCachegroups gets the cachegroup of each ATS Edge+Mid Cache server, for the given CDN, from Traffic Ops. // Returns a map[server]cachegroup. -func getServerCachegroups(crc CRConfig) (map[tc.CacheName]tc.CacheGroupName, error) { +func getServerCachegroups(mc tc.TrafficMonitorConfigMap) map[tc.CacheName]tc.CacheGroupName { serverCachegroups := map[tc.CacheName]tc.CacheGroupName{} - for server, serverData := range crc.ContentServers { - serverCachegroups[server] = tc.CacheGroupName(serverData.CacheGroup) + for server, serverData := range mc.TrafficServer { + serverCachegroups[tc.CacheName(server)] = tc.CacheGroupName(serverData.CacheGroup) } - return serverCachegroups, nil + return serverCachegroups } // getServerTypes gets the cache type of each ATS Edge+Mid Cache server, for the given CDN, from Traffic Ops. -func getServerTypes(crc CRConfig) (map[tc.CacheName]tc.CacheType, error) { +func getServerTypes(mc tc.TrafficMonitorConfigMap) (map[tc.CacheName]tc.CacheType, error) { serverTypes := map[tc.CacheName]tc.CacheType{} - for server, serverData := range crc.ContentServers { + for server, serverData := range mc.TrafficServer { t := tc.CacheTypeFromString(serverData.Type) if t == tc.CacheTypeInvalid { - return nil, fmt.Errorf("getServerTypes CRConfig unknown type for '%s': '%s'", server, serverData.Type) + return nil, fmt.Errorf("getServerTypes monitoring config unknown type for '%s': '%s'", server, serverData.Type) } - serverTypes[server] = t + serverTypes[tc.CacheName(server)] = t } return serverTypes, nil } -func getDeliveryServiceTypes(crc CRConfig) (map[tc.DeliveryServiceName]tc.DSTypeCategory, error) { +// canUseMonitorConfig returns true if we can prefer monitor config data to crconfig data. +func canUseMonitorConfig(mc tc.TrafficMonitorConfigMap) bool { + for _, dsData := range mc.DeliveryService { + return dsData.Type != "" + } + return false +} + +func getDeliveryServiceTypes(crc CRConfig, mc tc.TrafficMonitorConfigMap) (map[tc.DeliveryServiceName]tc.DSTypeCategory, error) { dsTypes := map[tc.DeliveryServiceName]tc.DSTypeCategory{} + if canUseMonitorConfig(mc) { + for dsName, dsData := range mc.DeliveryService { + dsType := tc.DSTypeCategoryFromString(dsData.Type) + if dsType == tc.DSTypeCategoryInvalid { + log.Warnln("monitoring config invalid DS type category for delivery service '" + string(dsName) + "' matchset protocol '" + dsData.Type + "'; skipping") + continue + } + dsTypes[tc.DeliveryServiceName(dsName)] = dsType + } + return dsTypes, nil + } for dsName, dsData := range crc.DeliveryServices { if len(dsData.Matchsets) < 1 { diff --git a/traffic_monitor/todata/todata_test.go b/traffic_monitor/todata/todata_test.go index 5f4900f076..43e354316a 100644 --- a/traffic_monitor/todata/todata_test.go +++ b/traffic_monitor/todata/todata_test.go @@ -48,6 +48,7 @@ func TestGetDeliveryServiceServersWithTopologyBasedDeliveryService(t *testing.T) Protocol string `json:"protocol"` MatchList []struct { Regex string `json:"regex"` + Type string `json:"match-type"` } `json:"matchlist"` } `json:"matchsets"` }{ @@ -57,11 +58,13 @@ func TestGetDeliveryServiceServersWithTopologyBasedDeliveryService(t *testing.T) Protocol string `json:"protocol"` MatchList []struct { Regex string `json:"regex"` + Type string `json:"match-type"` } `json:"matchlist"` }{{ Protocol: "HTTP", MatchList: []struct { Regex string `json:"regex"` + Type string `json:"match-type"` }{{Regex: `.*\.demo1\..*`}}, }}, }}, @@ -77,10 +80,7 @@ func TestGetDeliveryServiceServersWithTopologyBasedDeliveryService(t *testing.T) } topologiesTOData := TOData{} - var err error - if topologiesTOData.DeliveryServiceServers, topologiesTOData.ServerDeliveryServices, err = getDeliveryServiceServers(topologyCrConfig); err != nil { - t.Fatalf("unable to get deliveryservice servers for crconfig with topology-based delivery service: %s", err) - } + topologiesTOData.DeliveryServiceServers, topologiesTOData.ServerDeliveryServices = getDeliveryServiceServers(topologyCrConfig, tc.TrafficMonitorConfigMap{}) if !reflect.DeepEqual(expectedTopologiesTOData, topologiesTOData) { t.Fatalf("getDeliveryServiceServers with topology-based delivery service expected: %+v actual: %+v", expectedTopologiesTOData, topologiesTOData) } @@ -110,6 +110,7 @@ func TestGetDeliveryServiceServersWithNonTopologyBasedDeliveryService(t *testing Protocol string `json:"protocol"` MatchList []struct { Regex string `json:"regex"` + Type string `json:"match-type"` } `json:"matchlist"` } `json:"matchsets"` }{ @@ -118,11 +119,13 @@ func TestGetDeliveryServiceServersWithNonTopologyBasedDeliveryService(t *testing Protocol string `json:"protocol"` MatchList []struct { Regex string `json:"regex"` + Type string `json:"match-type"` } `json:"matchlist"` }{{ Protocol: "HTTP", MatchList: []struct { Regex string `json:"regex"` + Type string `json:"match-type"` }{{Regex: `.*\.demo2\..*`}}, }}, }, @@ -134,10 +137,7 @@ func TestGetDeliveryServiceServersWithNonTopologyBasedDeliveryService(t *testing } nonTopologiesTOData := TOData{} - var err error - if nonTopologiesTOData.DeliveryServiceServers, nonTopologiesTOData.ServerDeliveryServices, err = getDeliveryServiceServers(nonTopologyCrConfig); err != nil { - t.Fatalf("unable to get deliveryservice servers for crconfig with non-topology-based delivery service: %s", err) - } + nonTopologiesTOData.DeliveryServiceServers, nonTopologiesTOData.ServerDeliveryServices = getDeliveryServiceServers(nonTopologyCrConfig, tc.TrafficMonitorConfigMap{}) if !reflect.DeepEqual(expectedNonTopologiesTOData, nonTopologiesTOData) { t.Fatalf("getDeliveryServiceServers with non-topology-based delivery service expected: %+v actual: %+v", expectedNonTopologiesTOData, nonTopologiesTOData) } diff --git a/traffic_monitor/towrap/towrap.go b/traffic_monitor/towrap/towrap.go index a425eff5d3..9ea4eda9e0 100644 --- a/traffic_monitor/towrap/towrap.go +++ b/traffic_monitor/towrap/towrap.go @@ -457,7 +457,7 @@ func (s TrafficOpsSessionThreadsafe) CRConfigRaw(cdn string) ([]byte, error) { // to try to get the CRConfig from Traffic Ops. func (s TrafficOpsSessionThreadsafe) LastCRConfig(cdn string) ([]byte, time.Time, error) { crConfig, crConfigTime, _ := s.lastCRConfig.Get(cdn) - if crConfig == nil { + if len(crConfig) == 0 { b, err := s.CRConfigRaw(cdn) return b, time.Now(), err } @@ -552,23 +552,6 @@ func (s TrafficOpsSessionThreadsafe) TrafficMonitorConfigMap(cdn string) (*tc.Tr if err != nil { return nil, fmt.Errorf("getting monitor config map: %v", err) } - - crcData, err := s.CRConfigRaw(cdn) - if err != nil { - return nil, fmt.Errorf("getting CRConfig: %v", err) - } - - crConfig := tc.CRConfig{} - json := jsoniter.ConfigFastest - if err := json.Unmarshal(crcData, &crConfig); err != nil { - return nil, fmt.Errorf("unmarshalling CRConfig JSON : %v", err) - } - - mc, err = CreateMonitorConfig(crConfig, mc) - if err != nil { - return nil, fmt.Errorf("creating Traffic Monitor Config: %v", err) - } - return mc, nil } @@ -675,80 +658,3 @@ func (s TrafficOpsSessionThreadsafe) MonitorCDN(hostName string) (string, error) // return an error in that case return *server.CDNName, nil } - -// CreateMonitorConfig modifies the passed TrafficMonitorConfigMap to add the -// Traffic Monitors and Delivery Services found in a CDN Snapshot -func CreateMonitorConfig(crConfig tc.CRConfig, mc *tc.TrafficMonitorConfigMap) (*tc.TrafficMonitorConfigMap, error) { - // Dump the "live" monitoring.json monitors, and populate with the - // "snapshotted" CRConfig - if mc == nil { - return mc, errors.New("no TM configmap data") - } - // TODO: in the next major/minor release following 6.1, we should be able to remove this - // fallback dependency on the CRConfig entirely (https://github.com/apache/trafficcontrol/issues/6512) - for name, mon := range crConfig.Monitors { - if tmData, ok := mc.TrafficMonitor[name]; !ok || tmData.IP != "" || tmData.IP6 != "" { - continue - } - mc.TrafficMonitor[name] = tc.TrafficMonitor{} - m := mc.TrafficMonitor[name] - if mon.Port != nil { - m.Port = *mon.Port - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing Port field\n", name) - } - if mon.IP6 != nil { - m.IP6 = *mon.IP6 - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing IP6 field\n", name) - } - if mon.IP != nil { - m.IP = *mon.IP - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing IP field\n", name) - } - m.HostName = name - if mon.FQDN != nil { - m.FQDN = *mon.FQDN - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing FQDN field\n", name) - } - if mon.Profile != nil { - m.Profile = *mon.Profile - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing Profile field\n", name) - } - if mon.Location != nil { - m.Location = *mon.Location - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing Location field\n", name) - } - if mon.ServerStatus != nil { - m.ServerStatus = string(*mon.ServerStatus) - } else { - log.Warnf("Creating monitor config: CRConfig monitor %s missing ServerStatus field\n", name) - } - mc.TrafficMonitor[name] = m - } - - // Dump the "live" monitoring.json DeliveryServices, and populate with the - // "snapshotted" CRConfig but keep using the monitoring.json thresholds, - // because they're not in the CRConfig. - rawDeliveryServices := mc.DeliveryService - mc.DeliveryService = map[string]tc.TMDeliveryService{} - for name, _ := range crConfig.DeliveryServices { - if rawDS, ok := rawDeliveryServices[name]; ok { - // use the raw DS if it exists, because the CRConfig doesn't have - // thresholds or statuses - mc.DeliveryService[name] = rawDS - } else { - mc.DeliveryService[name] = tc.TMDeliveryService{ - XMLID: name, - TotalTPSThreshold: 0, - ServerStatus: "REPORTED", - TotalKbpsThreshold: 0, - } - } - } - return mc, nil -} diff --git a/traffic_ops/traffic_ops_golang/crconfig/crconfig.go b/traffic_ops/traffic_ops_golang/crconfig/crconfig.go index f252b9c20f..456feaef76 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/crconfig.go +++ b/traffic_ops/traffic_ops_golang/crconfig/crconfig.go @@ -27,6 +27,7 @@ import ( "github.com/apache/trafficcontrol/lib/go-log" "github.com/apache/trafficcontrol/lib/go-tc" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/topology" ) // Make creates and returns the CRConfig from the database. @@ -52,7 +53,7 @@ func Make(tx *sql.Tx, cdn, user, toHost, toVersion string, useClientReqHost bool if crc.DeliveryServices, err = makeDSes(cdn, cdnDomain, tx); err != nil { return nil, errors.New("Error getting Delivery Services: " + err.Error()) } - if crc.Topologies, err = makeTopologies(tx); err != nil { + if crc.Topologies, err = topology.MakeTopologies(tx); err != nil { return nil, errors.New("Error getting Topologies: " + err.Error()) } diff --git a/traffic_ops/traffic_ops_golang/crconfig/deliveryservice.go b/traffic_ops/traffic_ops_golang/crconfig/deliveryservice.go index 028ee7e9fc..91b556f449 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/deliveryservice.go +++ b/traffic_ops/traffic_ops_golang/crconfig/deliveryservice.go @@ -226,7 +226,7 @@ AND d.active = true ds.TTL = &ttl } - protocolStr := getProtocolStr(ttype) + protocolStr := tc.GetDSTypeCategory(ttype) ds.Protocol = protocolDefault if protocol.Valid { @@ -454,13 +454,6 @@ and d.active = true return entries, nil } -func getProtocolStr(dsType string) string { - if strings.HasPrefix(dsType, "DNS") { - return "DNS" - } - return "HTTP" -} - func getDSRegexesDomains(cdn string, domain string, tx *sql.Tx) (map[string][]*tc.MatchSet, map[string][]string, error) { dsmatchsets := map[string][]*tc.MatchSet{} domains := map[string][]string{} @@ -498,7 +491,7 @@ order by dr.set_number asc } else { dsNameOrderMap[dsname] = dsNameOrderMap[dsname] + 1 } - protocolStr := getProtocolStr(dstype) + protocolStr := tc.GetDSTypeCategory(dstype) dsmatchsets[dsname] = append(dsmatchsets[dsname], nil) diff --git a/traffic_ops/traffic_ops_golang/crconfig/edgelocations_test.go b/traffic_ops/traffic_ops_golang/crconfig/edgelocations_test.go index c52dc7fff6..217f3b78d3 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/edgelocations_test.go +++ b/traffic_ops/traffic_ops_golang/crconfig/edgelocations_test.go @@ -64,10 +64,10 @@ func MockMakeLocations(mock sqlmock.Sqlmock, expectedEdgeLocs map[string]tc.CRCo rows := sqlmock.NewRows([]string{"name", "id", "type", "latitude", "longitude", "fallback_to_closest", "localization_methods"}) for s, l := range expectedEdgeLocs { - rows = rows.AddRow(s, 1, EdgeTypePrefix, l.Lat, l.Lon, false, []byte("{CZ}")) + rows = rows.AddRow(s, 1, tc.EdgeTypePrefix, l.Lat, l.Lon, false, []byte("{CZ}")) } for s, l := range expectedRouterLocs { - rows = rows.AddRow(s, 1, RouterTypeName, l.Lat, l.Lon, false, nil) + rows = rows.AddRow(s, 1, tc.RouterTypeName, l.Lat, l.Lon, false, nil) } mock.ExpectQuery("SELECT").WithArgs(cdn).WillReturnRows(rows) diff --git a/traffic_ops/traffic_ops_golang/crconfig/servers.go b/traffic_ops/traffic_ops_golang/crconfig/servers.go index 9a5ed7c775..28f17b8195 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/servers.go +++ b/traffic_ops/traffic_ops_golang/crconfig/servers.go @@ -26,18 +26,12 @@ import ( "strconv" "strings" - "github.com/lib/pq" - "github.com/apache/trafficcontrol/lib/go-log" "github.com/apache/trafficcontrol/lib/go-tc" - "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/dbhelpers" -) -const RouterTypeName = "CCR" -const MonitorTypeName = "RASCAL" -const EdgeTypePrefix = "EDGE" -const MidTypePrefix = "MID" + "github.com/lib/pq" +) func makeCRConfigServers(cdn string, tx *sql.Tx, cdnDomain string) ( map[string]tc.CRConfigTrafficOpsServer, @@ -259,39 +253,6 @@ func getAllServers(cdn string, tx *sql.Tx) (map[string]ServerUnion, error) { return hostToServerMap, nil } -func getServerDSNames(cdn string, tx *sql.Tx) (map[tc.CacheName][]tc.DeliveryServiceName, error) { - q := ` -select s.host_name, ds.xml_id -from deliveryservice_server as dss -inner join server as s on dss.server = s.id -inner join deliveryservice as ds on ds.id = dss.deliveryservice -inner join type as dt on dt.id = ds.type -inner join profile as p on p.id = s.profile -inner join status as st ON st.id = s.status -where ds.cdn_id = (select id from cdn where name = $1) -and ds.active = true` + - fmt.Sprintf(" and dt.name != '%s' ", tc.DSTypeAnyMap) + ` -and p.routing_disabled = false -and (st.name = 'REPORTED' or st.name = 'ONLINE' or st.name = 'ADMIN_DOWN') -` - rows, err := tx.Query(q, cdn) - if err != nil { - return nil, errors.New("Error querying server deliveryservice names: " + err.Error()) - } - defer rows.Close() - - serverDSes := map[tc.CacheName][]tc.DeliveryServiceName{} - for rows.Next() { - ds := "" - server := "" - if err := rows.Scan(&server, &ds); err != nil { - return nil, errors.New("Error scanning server deliveryservice names: " + err.Error()) - } - serverDSes[tc.CacheName(server)] = append(serverDSes[tc.CacheName(server)], tc.DeliveryServiceName(ds)) - } - return serverDSes, nil -} - type DSRouteInfo struct { IsDNS bool IsRaw bool @@ -299,7 +260,7 @@ type DSRouteInfo struct { } func getServerDSes(cdn string, tx *sql.Tx, domain string) (map[tc.CacheName]map[string][]string, error) { - serverDSNames, err := getServerDSNames(cdn, tx) + serverDSNames, err := dbhelpers.GetServerDSNamesByCDN(tx, cdn) if err != nil { return nil, errors.New("Error getting server deliveryservices: " + err.Error()) } @@ -361,10 +322,10 @@ order by dsr.set_number asc serverDSPatterns := map[tc.CacheName]map[string][]string{} for server, dses := range serverDSNames { for _, dsName := range dses { - dsInfList, ok := dsInfs[string(dsName)] + dsInfList, ok := dsInfs[dsName] if !ok { - if !dsHasTopology[string(dsName)] { - log.Warnln("Creating CRConfig: deliveryservice " + string(dsName) + " has no regexes, skipping") + if !dsHasTopology[dsName] { + log.Warnln("creating CRConfig: deliveryservice " + dsName + " has no regexes, skipping") } continue } @@ -375,7 +336,7 @@ order by dsr.set_number asc if _, ok := serverDSPatterns[server]; !ok { serverDSPatterns[server] = map[string][]string{} } - serverDSPatterns[server][string(dsName)] = append(serverDSPatterns[server][string(dsName)], dsInf.Remap) + serverDSPatterns[server][dsName] = append(serverDSPatterns[server][dsName], dsInf.Remap) } } } diff --git a/traffic_ops/traffic_ops_golang/crconfig/servers_test.go b/traffic_ops/traffic_ops_golang/crconfig/servers_test.go index a30d30cf76..1fa51984bf 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/servers_test.go +++ b/traffic_ops/traffic_ops_golang/crconfig/servers_test.go @@ -22,7 +22,6 @@ package crconfig import ( "context" "fmt" - "github.com/lib/pq" "math/rand" "net" "reflect" @@ -32,32 +31,22 @@ import ( "github.com/apache/trafficcontrol/lib/go-tc" "github.com/apache/trafficcontrol/lib/go-util" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/dbhelpers" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/test" "gopkg.in/DATA-DOG/go-sqlmock.v1" ) -type StringArray pq.StringArray - +// TODO: move all these rand functions to traffic_ops_golang/test/rand.go func randBool() *bool { b := rand.Int()%2 == 0 return &b } func randStr() *string { - chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_" - num := 100 - s := "" - for i := 0; i < num; i++ { - s += string(chars[rand.Intn(len(chars))]) - } - return &s + return test.RandStr() } func randStrArray() []string { - num := 100 - sArray := make([]string, num) - for i := 0; i < num; i++ { - sArray[i] = *randStr() - } - return sArray + return test.RandStrArray() } func randInt() *int { i := rand.Int() @@ -490,21 +479,21 @@ func TestGetAllServersNonService(t *testing.T) { compare(expected, actual, t) } -func ExpectedGetServerDSNames() map[tc.CacheName][]tc.DeliveryServiceName { - return map[tc.CacheName][]tc.DeliveryServiceName{ - "cache0": []tc.DeliveryServiceName{"ds0", "ds1"}, - "cache1": []tc.DeliveryServiceName{"ds0", "ds1"}, +func ExpectedGetServerDSNames() map[tc.CacheName][]string { + return map[tc.CacheName][]string{ + "cache0": {"ds0", "ds1"}, + "cache1": {"ds0", "ds1"}, } } -func MockGetServerDSNames(mock sqlmock.Sqlmock, expected map[tc.CacheName][]tc.DeliveryServiceName, cdn string) { +func MockGetServerDSNames(mock sqlmock.Sqlmock, expected map[tc.CacheName][]string, cdn string) { rows := sqlmock.NewRows([]string{"host_name", "xml_id"}) for cache, dses := range expected { for _, ds := range dses { rows = rows.AddRow(cache, ds) } } - mock.ExpectQuery("select").WithArgs(cdn).WillReturnRows(rows) + mock.ExpectQuery("SELECT").WithArgs(cdn).WillReturnRows(rows) } func TestGetServerDSNames(t *testing.T) { @@ -529,7 +518,7 @@ func TestGetServerDSNames(t *testing.T) { } defer tx.Commit() - actual, err := getServerDSNames(cdn, tx) + actual, err := dbhelpers.GetServerDSNamesByCDN(tx, cdn) if err != nil { t.Fatalf("getServerDSNames expected: nil error, actual: %v", err) @@ -544,12 +533,12 @@ func TestGetServerDSNames(t *testing.T) { } } -func ExpectedGetServerDSes(expectedGetServerDSNames map[tc.CacheName][]tc.DeliveryServiceName) map[tc.CacheName]map[string][]string { +func ExpectedGetServerDSes(expectedGetServerDSNames map[tc.CacheName][]string) map[tc.CacheName]map[string][]string { e := map[tc.CacheName]map[string][]string{} for cache, dses := range expectedGetServerDSNames { e[cache] = map[string][]string{} for _, ds := range dses { - e[cache][string(ds)] = []string{string(ds) + "regex0", string(ds) + "regex1"} + e[cache][ds] = []string{ds + "regex0", ds + "regex1"} } } return e diff --git a/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go b/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go index efd1e06057..e90471e578 100644 --- a/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go +++ b/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go @@ -543,6 +543,39 @@ func GetDSTenantIDFromXMLID(tx *sql.Tx, xmlid string) (int, bool, error) { return id, true, nil } +func GetServerDSNamesByCDN(tx *sql.Tx, cdn string) (map[tc.CacheName][]string, error) { + q := ` +SELECT s.host_name, ds.xml_id +FROM deliveryservice_server AS dss +INNER JOIN server AS s ON dss.server = s.id +INNER JOIN deliveryservice AS ds ON ds.id = dss.deliveryservice +INNER JOIN type AS dt ON dt.id = ds.type +INNER JOIN profile AS p ON p.id = s.profile +INNER JOIN status AS st ON st.id = s.status +WHERE ds.cdn_id = (SELECT id FROM cdn WHERE name = $1) +AND ds.active = true +AND dt.name != '` + tc.DSTypeAnyMap.String() + `' +AND p.routing_disabled = false +AND (st.name = '` + tc.CacheStatusOnline.String() + `' OR st.name = '` + tc.CacheStatusReported.String() + `' OR st.name = '` + tc.CacheStatusAdminDown.String() + `') +` + rows, err := tx.Query(q, cdn) + if err != nil { + return nil, errors.New("querying server deliveryservice names by CDN: " + err.Error()) + } + defer log.Close(rows, "closing rows after querying server deliveryservice names by CDN") + + serverDSes := map[tc.CacheName][]string{} + for rows.Next() { + ds := "" + server := "" + if err := rows.Scan(&server, &ds); err != nil { + return nil, errors.New("scanning server deliveryservice names: " + err.Error()) + } + serverDSes[tc.CacheName(server)] = append(serverDSes[tc.CacheName(server)], ds) + } + return serverDSes, nil +} + // returns returns the delivery service name and cdn, whether it existed, and any error. func GetDSNameAndCDNFromID(tx *sql.Tx, id int) (tc.DeliveryServiceName, tc.CDNName, bool, error) { name := tc.DeliveryServiceName("") diff --git a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go index be7db0e8ee..bca24ae7fc 100644 --- a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go +++ b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go @@ -30,6 +30,8 @@ import ( "github.com/apache/trafficcontrol/lib/go-log" "github.com/apache/trafficcontrol/lib/go-tc" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/dbhelpers" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/topology" "github.com/lib/pq" ) @@ -38,7 +40,6 @@ const CacheMonitorConfigFile = "rascal.properties" const MonitorType = "RASCAL" const RouterType = "CCR" -const MonitorProfilePrefix = "RASCAL" const MonitorConfigFile = "rascal-config.txt" const KilobitsPerMegabit = 1000 const DeliveryServiceStatus = "REPORTED" @@ -72,9 +73,10 @@ type LegacyCache struct { type Cache struct { CommonServerProperties - Interfaces []tc.ServerInterfaceInfo `json:"interfaces"` - Type string `json:"type"` - HashID string `json:"hashid"` + Interfaces []tc.ServerInterfaceInfo `json:"interfaces"` + Type string `json:"type"` + HashID string `json:"hashid"` + DeliveryServices []string `json:"deliveryServices,omitempty"` } type Cachegroup struct { @@ -104,12 +106,13 @@ type LegacyMonitoring struct { } type Monitoring struct { - TrafficServers []Cache `json:"trafficServers"` - TrafficMonitors []Monitor `json:"trafficMonitors"` - Cachegroups []Cachegroup `json:"cacheGroups"` - Profiles []Profile `json:"profiles"` - DeliveryServices []DeliveryService `json:"deliveryServices"` - Config map[string]interface{} `json:"config"` + TrafficServers []Cache `json:"trafficServers"` + TrafficMonitors []Monitor `json:"trafficMonitors"` + Cachegroups []Cachegroup `json:"cacheGroups"` + Profiles []Profile `json:"profiles"` + DeliveryServices []DeliveryService `json:"deliveryServices"` + Config map[string]interface{} `json:"config"` + Topologies map[string]tc.CRConfigTopology `json:"topologies"` } // LegacyMonitoringResponse represents MontiroingResponse for ATC versions before 5.0. @@ -127,10 +130,13 @@ type Router struct { } type DeliveryService struct { - XMLID string `json:"xmlId"` - TotalTPSThreshold float64 `json:"totalTpsThreshold"` - Status string `json:"status"` - TotalKBPSThreshold float64 `json:"totalKbpsThreshold"` + XMLID string `json:"xmlId"` + TotalTPSThreshold float64 `json:"totalTpsThreshold"` + Status string `json:"status"` + TotalKBPSThreshold float64 `json:"totalKbpsThreshold"` + Type string `json:"type"` + Topology string `json:"topology"` + HostRegexes []string `json:"hostRegexes"` } func GetMonitoringJSON(tx *sql.Tx, cdnName string) (*Monitoring, error) { @@ -158,6 +164,10 @@ func GetMonitoringJSON(tx *sql.Tx, cdnName string) (*Monitoring, error) { if err != nil { return nil, fmt.Errorf("error getting config: %v", err) } + topologies, err := topology.MakeTopologies(tx) + if err != nil { + return nil, fmt.Errorf("getting topologies: %s", err.Error()) + } return &Monitoring{ TrafficServers: caches, @@ -166,6 +176,7 @@ func GetMonitoringJSON(tx *sql.Tx, cdnName string) (*Monitoring, error) { Profiles: profiles, DeliveryServices: deliveryServices, Config: config, + Topologies: topologies, }, nil } @@ -268,6 +279,11 @@ AND cdn.name = $3 interfacesByNameAndServer[serverID][interfaceName] = interf } + serverDSNames, err := dbhelpers.GetServerDSNamesByCDN(tx, cdn) + if err != nil { + return nil, nil, nil, err + } + rows, err := tx.Query(serversQuery, cdn) if err != nil { return nil, nil, nil, err @@ -300,6 +316,7 @@ AND cdn.name = $3 if _, ok := interfacesByNameAndServer[int(serverID.Int64)]; ok { for _, interf := range interfacesByNameAndServer[int(serverID.Int64)] { for _, ipAddress := range interf.IPAddresses { + ipAddress.Address = strings.Split(ipAddress.Address, "/")[0] ip := net.ParseIP(ipAddress.Address) if ip == nil { continue @@ -353,9 +370,10 @@ AND cdn.name = $3 HostName: hostName.String, FQDN: fqdn.String, }, - Interfaces: cacheInterfaces, - Type: ttype.String, - HashID: hashID.String, + Interfaces: cacheInterfaces, + Type: ttype.String, + HashID: hashID.String, + DeliveryServices: serverDSNames[tc.CacheName(hostName.String)], } caches = append(caches, cache) } else if ttype.String == tc.RouterTypeName { @@ -470,17 +488,22 @@ WHERE pr.config_file = $2; func getDeliveryServices(tx *sql.Tx, cdnName string) ([]DeliveryService, error) { query := ` - SELECT ds.xml_id, ds.global_max_tps, ds.global_max_mbps + SELECT ds.xml_id, ds.global_max_tps, ds.global_max_mbps, t.name AS ds_type, ds.topology, ARRAY_AGG(r.pattern) FROM deliveryservice ds - JOIN cdn on cdn.id = ds.cdn_id + JOIN type t ON ds.type = t.id + JOIN cdn ON cdn.id = ds.cdn_id + JOIN deliveryservice_regex dsr ON dsr.deliveryservice = ds.id + JOIN regex r ON r.id = dsr.regex WHERE ds.active = true AND cdn.name=$1 + AND r.type = (SELECT id FROM type WHERE name = 'HOST_REGEXP') + GROUP BY ds.xml_id, ds.global_max_tps, ds.xml_id, ds.global_max_mbps, t.name, ds.topology ` rows, err := tx.Query(query, cdnName) if err != nil { return nil, err } - defer rows.Close() + defer log.Close(rows, "closing deliveryservice rows") dses := []DeliveryService{} @@ -488,7 +511,10 @@ func getDeliveryServices(tx *sql.Tx, cdnName string) ([]DeliveryService, error) var xmlid sql.NullString var tps sql.NullFloat64 var mbps sql.NullFloat64 - if err := rows.Scan(&xmlid, &tps, &mbps); err != nil { + var dsType string + var topology sql.NullString + var hostRegexes []string + if err := rows.Scan(&xmlid, &tps, &mbps, &dsType, &topology, pq.Array(&hostRegexes)); err != nil { return nil, err } dses = append(dses, DeliveryService{ @@ -496,6 +522,9 @@ func getDeliveryServices(tx *sql.Tx, cdnName string) ([]DeliveryService, error) TotalTPSThreshold: tps.Float64, Status: DeliveryServiceStatus, TotalKBPSThreshold: mbps.Float64 * KilobitsPerMegabit, + Type: tc.GetDSTypeCategory(dsType), + Topology: topology.String, + HostRegexes: hostRegexes, }) } return dses, nil diff --git a/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go b/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go index e3b21a3430..f87207d806 100644 --- a/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go +++ b/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go @@ -5,6 +5,7 @@ import ( "reflect" "sort" "strconv" + "strings" "testing" "time" @@ -337,14 +338,18 @@ func TestGetDeliveryServices(t *testing.T) { TotalTPSThreshold: 42.42, Status: DeliveryServiceStatus, TotalKBPSThreshold: 24.24, + Type: "HTTP", + Topology: "foo", + HostRegexes: []string{`.*\.example\..*`}, } deliveryservices := []DeliveryService{deliveryservice} mock.ExpectBegin() - rows := sqlmock.NewRows([]string{"xml_id", "global_max_tps", "global_max_mbps"}) + rows := sqlmock.NewRows([]string{"xml_id", "global_max_tps", "global_max_mbps", "ds_type", "topology", "host_regexes"}) for _, deliveryservice := range deliveryservices { - rows = rows.AddRow(deliveryservice.XMLID, deliveryservice.TotalTPSThreshold, deliveryservice.TotalKBPSThreshold/KilobitsPerMegabit) + rows = rows.AddRow(deliveryservice.XMLID, deliveryservice.TotalTPSThreshold, deliveryservice.TotalKBPSThreshold/KilobitsPerMegabit, + deliveryservice.Type, deliveryservice.Topology, "{"+strings.Join(deliveryservice.HostRegexes, ",")+"}") } mock.ExpectQuery("SELECT").WillReturnRows(rows) @@ -367,7 +372,7 @@ func TestGetDeliveryServices(t *testing.T) { for i, sqlDeliveryservice := range sqlDeliveryservices { deliveryservice := deliveryservices[i] - if deliveryservice != sqlDeliveryservice { + if !reflect.DeepEqual(deliveryservice, sqlDeliveryservice) { t.Errorf("getDeliveryServices expected: %v, actual: %v", deliveryservice, sqlDeliveryservice) } } @@ -513,14 +518,18 @@ func TestGetMonitoringJSON(t *testing.T) { TotalTPSThreshold: 42.42, Status: DeliveryServiceStatus, TotalKBPSThreshold: 24.24, + Type: "HTTP", + Topology: "foo", + HostRegexes: []string{`.*\.example\..*`}, } deliveryservices := []DeliveryService{deliveryservice} // routers := []Router{router} - rows := sqlmock.NewRows([]string{"xml_id", "global_max_tps", "global_max_mbps"}) + rows := sqlmock.NewRows([]string{"xml_id", "global_max_tps", "global_max_mbps", "ds_type", "topology", "host_regexes"}) for _, deliveryservice := range deliveryservices { - rows = rows.AddRow(deliveryservice.XMLID, deliveryservice.TotalTPSThreshold, deliveryservice.TotalKBPSThreshold/KilobitsPerMegabit) + rows = rows.AddRow(deliveryservice.XMLID, deliveryservice.TotalTPSThreshold, deliveryservice.TotalKBPSThreshold/KilobitsPerMegabit, + deliveryservice.Type, deliveryservice.Topology, "{"+strings.Join(deliveryservice.HostRegexes, ",")+"}") } mock.ExpectQuery("SELECT").WillReturnRows(rows) @@ -543,6 +552,21 @@ func TestGetMonitoringJSON(t *testing.T) { mock.ExpectQuery("SELECT").WillReturnRows(rows) resp.Response.Config = config } + { + // + // topologies + // + topologies := map[string]tc.CRConfigTopology{ + "foo": {Nodes: []string{"cg1"}}, + } + + rows := sqlmock.NewRows([]string{"name", "nodes"}) + for name, nodes := range topologies { + rows = rows.AddRow(name, "{"+strings.Join(nodes.Nodes, ",")+"}") + } + mock.ExpectQuery("SELECT").WillReturnRows(rows) + resp.Response.Topologies = topologies + } dbCtx, f := context.WithTimeout(context.TODO(), time.Duration(10)*time.Second) defer f() @@ -590,6 +614,9 @@ func TestGetMonitoringJSON(t *testing.T) { if !reflect.DeepEqual(sqlResp.Config, resp.Response.Config) { t.Errorf("GetMonitoringJSON expected Config: %+v actual: %+v", resp.Response.Config, sqlResp.Config) } + if !reflect.DeepEqual(sqlResp.Topologies, resp.Response.Topologies) { + t.Errorf("GetMonitoringJSON expected Topologies: %+v actual: %+v", resp.Response.Topologies, sqlResp.Topologies) + } } type SortableProfiles []Profile @@ -715,9 +742,11 @@ func setupMockGetMonitoringServersWithoutIPv4(mock sqlmock.Sqlmock, monitor Moni serverRows := sqlmock.NewRows([]string{"hostName", "fqdn", "status", "cachegroup", "port", "profile", "type", "hashId", "serverID"}) interfaceRows := sqlmock.NewRows([]string{"name", "max_bandwidth", "mtu", "monitor", "server"}) ipAddressRows := sqlmock.NewRows([]string{"address", "gateway", "service_address", "server", "interface"}) + dssRows := sqlmock.NewRows([]string{"host_name", "xml_id"}) serverRows = serverRows.AddRow(monitor.HostName, monitor.FQDN, monitor.Status, monitor.Cachegroup, monitor.Port, monitor.Profile, MonitorType, "noHash", 5) for index, cache := range caches { serverRows = serverRows.AddRow(cache.HostName, cache.FQDN, cache.Status, cache.Cachegroup, cache.Port, cache.Profile, cache.Type, cache.HashID, cacheIDs[index]) + dssRows = dssRows.AddRow(cache.HostName, "xml_id_foo") interfaceRows = interfaceRows.AddRow("none", nil, 1500, false, 0) for _, interf := range cache.Interfaces { @@ -737,6 +766,7 @@ func setupMockGetMonitoringServersWithoutIPv4(mock sqlmock.Sqlmock, monitor Moni serverRows = serverRows.AddRow("noHostname", "noFqdn", "noStatus", "noGroup", 0, router.Profile, RouterType, "noHashid", 3) mock.ExpectQuery("SELECT (.+) FROM interface i (.+)").WithArgs(cdn).WillReturnRows(interfaceRows) mock.ExpectQuery("SELECT (.+) FROM ip_address ip (.+)").WillReturnRows(ipAddressRows) + mock.ExpectQuery("SELECT (.+) FROM deliveryservice_server AS dss (.+)").WillReturnRows(dssRows) mock.ExpectQuery("SELECT (.+) FROM server me (.+)").WithArgs(cdn).WillReturnRows(serverRows) } @@ -744,9 +774,11 @@ func setupMockGetMonitoringServers(mock sqlmock.Sqlmock, monitor Monitor, router serverRows := sqlmock.NewRows([]string{"hostName", "fqdn", "status", "cachegroup", "port", "profile", "type", "hashId", "serverID"}) interfaceRows := sqlmock.NewRows([]string{"name", "max_bandwidth", "mtu", "monitor", "server"}) ipAddressRows := sqlmock.NewRows([]string{"address", "gateway", "service_address", "server", "interface"}) + dssRows := sqlmock.NewRows([]string{"host_name", "xml_id"}) serverRows = serverRows.AddRow(monitor.HostName, monitor.FQDN, monitor.Status, monitor.Cachegroup, monitor.Port, monitor.Profile, MonitorType, "noHash", 5) for index, cache := range caches { serverRows = serverRows.AddRow(cache.HostName, cache.FQDN, cache.Status, cache.Cachegroup, cache.Port, cache.Profile, cache.Type, cache.HashID, cacheIDs[index]) + dssRows = dssRows.AddRow(cache.HostName, "xml_id_foo") interfaceRows = interfaceRows.AddRow("none", nil, 1500, false, 0) for _, interf := range cache.Interfaces { @@ -767,6 +799,7 @@ func setupMockGetMonitoringServers(mock sqlmock.Sqlmock, monitor Monitor, router serverRows = serverRows.AddRow("noHostname", "noFqdn", "noStatus", "noGroup", 0, router.Profile, RouterType, "noHashid", 3) mock.ExpectQuery("SELECT (.+) FROM interface i (.+)").WithArgs(cdn).WillReturnRows(interfaceRows) mock.ExpectQuery("SELECT (.+) FROM ip_address ip (.+)").WillReturnRows(ipAddressRows) + mock.ExpectQuery("SELECT (.+) FROM deliveryservice_server AS dss (.+)").WillReturnRows(dssRows) mock.ExpectQuery("SELECT (.+) FROM server me (.+)").WithArgs(cdn).WillReturnRows(serverRows) } @@ -848,8 +881,9 @@ func createMockCache(interfaceName string) Cache { Name: interfaceName + "2", }, }, - Type: "EDGE", - HashID: "cacheHash", + Type: "EDGE", + HashID: "cacheHash", + DeliveryServices: []string{"xml_id_foo"}, } } diff --git a/traffic_ops/traffic_ops_golang/test/rand.go b/traffic_ops/traffic_ops_golang/test/rand.go new file mode 100644 index 0000000000..ab0456765d --- /dev/null +++ b/traffic_ops/traffic_ops_golang/test/rand.go @@ -0,0 +1,38 @@ +package test + +/* + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +import ( + "math/rand" +) + +func RandStr() *string { + chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_" + num := 100 + s := "" + for i := 0; i < num; i++ { + s += string(chars[rand.Intn(len(chars))]) + } + return &s +} +func RandStrArray() []string { + num := 100 + sArray := make([]string, num) + for i := 0; i < num; i++ { + sArray[i] = *RandStr() + } + return sArray +} diff --git a/traffic_ops/traffic_ops_golang/crconfig/topologies.go b/traffic_ops/traffic_ops_golang/topology/snapshot.go similarity index 91% rename from traffic_ops/traffic_ops_golang/crconfig/topologies.go rename to traffic_ops/traffic_ops_golang/topology/snapshot.go index d86cdc1fb6..5a86a4b279 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/topologies.go +++ b/traffic_ops/traffic_ops_golang/topology/snapshot.go @@ -1,4 +1,4 @@ -package crconfig +package topology /* * Licensed to the Apache Software Foundation (ASF) under one @@ -22,12 +22,14 @@ package crconfig import ( "database/sql" "errors" + "github.com/apache/trafficcontrol/lib/go-log" "github.com/apache/trafficcontrol/lib/go-tc" "github.com/lib/pq" ) -func makeTopologies(tx *sql.Tx) (map[string]tc.CRConfigTopology, error) { +// MakeTopologies makes the topologies data for the crconfig and tmconfig snapshots. +func MakeTopologies(tx *sql.Tx) (map[string]tc.CRConfigTopology, error) { query := ` SELECT t.name, diff --git a/traffic_ops/traffic_ops_golang/crconfig/topologies_test.go b/traffic_ops/traffic_ops_golang/topology/snapshot_test.go similarity index 94% rename from traffic_ops/traffic_ops_golang/crconfig/topologies_test.go rename to traffic_ops/traffic_ops_golang/topology/snapshot_test.go index 3a8a6324d1..bbf62993a9 100644 --- a/traffic_ops/traffic_ops_golang/crconfig/topologies_test.go +++ b/traffic_ops/traffic_ops_golang/topology/snapshot_test.go @@ -1,4 +1,4 @@ -package crconfig +package topology /* * Licensed to the Apache Software Foundation (ASF) under one @@ -22,17 +22,20 @@ package crconfig import ( "context" "encoding/json" - "github.com/apache/trafficcontrol/lib/go-tc" - "gopkg.in/DATA-DOG/go-sqlmock.v1" "reflect" "strings" "testing" "time" + + "github.com/apache/trafficcontrol/lib/go-tc" + "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/test" + + "gopkg.in/DATA-DOG/go-sqlmock.v1" ) func randTopology() tc.CRConfigTopology { return tc.CRConfigTopology{ - Nodes: randStrArray(), + Nodes: test.RandStrArray(), } } @@ -75,7 +78,7 @@ func TestMakeTops(t *testing.T) { t.Fatal("creating transaction: ", err) } - actual, err := makeTopologies(tx) + actual, err := MakeTopologies(tx) if err != nil { t.Fatal("makeTopologies expected: nil error, actual: ", err) } From bf8dac27837d6ed92fe479df4762aa5a443475e1 Mon Sep 17 00:00:00 2001 From: Rawlin Peters Date: Tue, 15 Mar 2022 11:41:33 -0600 Subject: [PATCH 2/4] Make trafficServer deliveryService field backwards-compatible --- .../api/v2/cdns_name_configs_monitoring.rst | 5 ++- .../api/v3/cdns_name_configs_monitoring.rst | 5 ++- .../api/v4/cdns_name_configs_monitoring.rst | 5 ++- lib/go-tc/traffic_monitor_test.go | 2 +- lib/go-tc/traffic_router.go | 33 +++++++++---------- lib/go-tc/traffic_router_test.go | 2 +- traffic_monitor/todata/todata.go | 4 +-- .../monitoring/monitoring.go | 12 +++++-- .../monitoring/monitoring_test.go | 2 +- 9 files changed, 43 insertions(+), 27 deletions(-) diff --git a/docs/source/api/v2/cdns_name_configs_monitoring.rst b/docs/source/api/v2/cdns_name_configs_monitoring.rst index 13dc3ae855..dba252ac0b 100644 --- a/docs/source/api/v2/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v2/cdns_name_configs_monitoring.rst @@ -101,7 +101,10 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + + :xmlId: A string which is the XML ID of the delivery service + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons :hostName: The (short) hostname of the :term:`cache server` diff --git a/docs/source/api/v3/cdns_name_configs_monitoring.rst b/docs/source/api/v3/cdns_name_configs_monitoring.rst index 1b95ad38a5..53e016903c 100644 --- a/docs/source/api/v3/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v3/cdns_name_configs_monitoring.rst @@ -101,7 +101,10 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + + :xmlId: A string which is the XML ID of the delivery service + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons :hostName: The (short) hostname of the :term:`cache server` diff --git a/docs/source/api/v4/cdns_name_configs_monitoring.rst b/docs/source/api/v4/cdns_name_configs_monitoring.rst index cb4229824d..011c7a6449 100644 --- a/docs/source/api/v4/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v4/cdns_name_configs_monitoring.rst @@ -102,7 +102,10 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of strings which are the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + + :xmlId: A string which is the XML ID of the delivery service + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons :hostName: The (short) hostname of the :term:`cache server` diff --git a/lib/go-tc/traffic_monitor_test.go b/lib/go-tc/traffic_monitor_test.go index 8b7e5c86a6..1cd4fb8dfc 100644 --- a/lib/go-tc/traffic_monitor_test.go +++ b/lib/go-tc/traffic_monitor_test.go @@ -291,7 +291,7 @@ func ExampleLegacyTrafficMonitorConfigMap_Upgrade() { TrafficServer: map[string]LegacyTrafficServer{ "test": { CacheGroup: "test", - DeliveryServices: []string{}, + DeliveryServices: []TSDeliveryService{}, FQDN: "test.quest", HashID: "test", HostName: "test", diff --git a/lib/go-tc/traffic_router.go b/lib/go-tc/traffic_router.go index 3f025e4eba..bf6dd11098 100644 --- a/lib/go-tc/traffic_router.go +++ b/lib/go-tc/traffic_router.go @@ -92,19 +92,19 @@ type MatchList struct { // Deprecated: The configuration versions that use this structure to represent // a cache server are deprecated, new code should use TrafficServer instead. type LegacyTrafficServer struct { - CacheGroup string `json:"cacheGroup"` - DeliveryServices []string `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids - FQDN string `json:"fqdn"` - HashID string `json:"hashId"` - HostName string `json:"hostName"` - HTTPSPort int `json:"httpsPort,omitempty"` - InterfaceName string `json:"interfaceName"` - IP string `json:"ip"` - IP6 string `json:"ip6"` - Port int `json:"port"` - Profile string `json:"profile"` - ServerStatus string `json:"status"` - Type string `json:"type"` + CacheGroup string `json:"cacheGroup"` + DeliveryServices []TSDeliveryService `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids + FQDN string `json:"fqdn"` + HashID string `json:"hashId"` + HostName string `json:"hostName"` + HTTPSPort int `json:"httpsPort,omitempty"` + InterfaceName string `json:"interfaceName"` + IP string `json:"ip"` + IP6 string `json:"ip6"` + Port int `json:"port"` + Profile string `json:"profile"` + ServerStatus string `json:"status"` + Type string `json:"type"` } // Upgrade upgrades the LegacyTrafficServer into its modern-day equivalent. @@ -207,7 +207,7 @@ func (ts *TrafficServer) ToLegacyServer() LegacyTrafficServer { // Traffic Router instances. type TrafficServer struct { CacheGroup string `json:"cachegroup"` - DeliveryServices []string `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids + DeliveryServices []TSDeliveryService `json:"deliveryServices,omitempty"` // the deliveryServices key does not exist on mids FQDN string `json:"fqdn"` HashID string `json:"hashId"` HostName string `json:"hostName"` @@ -249,7 +249,6 @@ func (ts *TrafficServer) IPv6() string { return *lid.IP6Address } -type tsdeliveryService struct { - Xmlid string `json:"xmlId"` - Remaps []string `json:"remaps"` +type TSDeliveryService struct { + XmlId string `json:"xmlId"` } diff --git a/lib/go-tc/traffic_router_test.go b/lib/go-tc/traffic_router_test.go index 7090253865..a60df23d48 100644 --- a/lib/go-tc/traffic_router_test.go +++ b/lib/go-tc/traffic_router_test.go @@ -88,7 +88,7 @@ func ExampleTrafficServer_IPv6() { func ExampleLegacyTrafficServer_Upgrade() { lts := LegacyTrafficServer{ CacheGroup: "testCG", - DeliveryServices: []string{}, + DeliveryServices: []TSDeliveryService{}, FQDN: "test.quest", HashID: "test", HostName: "test", diff --git a/traffic_monitor/todata/todata.go b/traffic_monitor/todata/todata.go index 9548ea0c5f..96bb7e9bfc 100644 --- a/traffic_monitor/todata/todata.go +++ b/traffic_monitor/todata/todata.go @@ -209,8 +209,8 @@ func getDeliveryServiceServers(crc CRConfig, mc tc.TrafficMonitorConfigMap) (map } serverDses[srvName] = append(serverDses[srvName], cacheGroupDses...) } - for _, deliveryServiceName := range serverData.DeliveryServices { - dsName := tc.DeliveryServiceName(deliveryServiceName) + for _, serverDS := range serverData.DeliveryServices { + dsName := tc.DeliveryServiceName(serverDS.XmlId) dsServers[dsName] = append(dsServers[dsName], srvName) serverDses[srvName] = append(serverDses[srvName], dsName) } diff --git a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go index bca24ae7fc..b5ac2a8f4c 100644 --- a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go +++ b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go @@ -76,7 +76,7 @@ type Cache struct { Interfaces []tc.ServerInterfaceInfo `json:"interfaces"` Type string `json:"type"` HashID string `json:"hashid"` - DeliveryServices []string `json:"deliveryServices,omitempty"` + DeliveryServices []tc.TSDeliveryService `json:"deliveryServices,omitempty"` } type Cachegroup struct { @@ -283,6 +283,14 @@ AND cdn.name = $3 if err != nil { return nil, nil, nil, err } + serverDSes := make(map[tc.CacheName][]tc.TSDeliveryService, len(serverDSNames)) + for c, dsNames := range serverDSNames { + tsDS := make([]tc.TSDeliveryService, 0, len(dsNames)) + for _, n := range dsNames { + tsDS = append(tsDS, tc.TSDeliveryService{XmlId: n}) + } + serverDSes[c] = tsDS + } rows, err := tx.Query(serversQuery, cdn) if err != nil { @@ -373,7 +381,7 @@ AND cdn.name = $3 Interfaces: cacheInterfaces, Type: ttype.String, HashID: hashID.String, - DeliveryServices: serverDSNames[tc.CacheName(hostName.String)], + DeliveryServices: serverDSes[tc.CacheName(hostName.String)], } caches = append(caches, cache) } else if ttype.String == tc.RouterTypeName { diff --git a/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go b/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go index f87207d806..af8bbcb6c7 100644 --- a/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go +++ b/traffic_ops/traffic_ops_golang/monitoring/monitoring_test.go @@ -883,7 +883,7 @@ func createMockCache(interfaceName string) Cache { }, Type: "EDGE", HashID: "cacheHash", - DeliveryServices: []string{"xml_id_foo"}, + DeliveryServices: []tc.TSDeliveryService{{XmlId: "xml_id_foo"}}, } } From d298ad7c84b47bd3789487894aec182d682a6db9 Mon Sep 17 00:00:00 2001 From: Rawlin Peters Date: Tue, 15 Mar 2022 15:23:45 -0600 Subject: [PATCH 3/4] Address review comments --- .../api/v2/cdns_name_configs_monitoring.rst | 22 ++++++++++--------- .../api/v3/cdns_name_configs_monitoring.rst | 4 ++-- .../api/v4/cdns_name_configs_monitoring.rst | 4 ++-- lib/go-tc/traffic_router.go | 1 + traffic_monitor/todata/todata.go | 8 +++---- .../dbhelpers/db_helpers.go | 3 +++ .../monitoring/monitoring.go | 2 +- 7 files changed, 25 insertions(+), 19 deletions(-) diff --git a/docs/source/api/v2/cdns_name_configs_monitoring.rst b/docs/source/api/v2/cdns_name_configs_monitoring.rst index dba252ac0b..e8242bf3cf 100644 --- a/docs/source/api/v2/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v2/cdns_name_configs_monitoring.rst @@ -60,7 +60,7 @@ Response Structure :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status - :topology: A string that is the name of the Delivery Service's topology (if assigned one) + :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle :type: A string that is the Delivery Service's type category (HTTP or DNS) @@ -89,7 +89,7 @@ Response Structure :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :cachegroup: The name of the :term:`Cache Group` to which this Traffic Monitor belongs :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance :hostname: The hostname of the server running this Traffic Monitor instance :ip6: The IPv6 address of this Traffic Monitor - when applicable @@ -105,14 +105,16 @@ Response Structure :xmlId: A string which is the XML ID of the delivery service - :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address - :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons - :hostName: The (short) hostname of the :term:`cache server` - :port: The port on which the :term:`cache server` listens for incoming connections - :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` - :status: The status of the :term:`cache server` - :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` - :interfaces: A set of the network interfaces in use by the server. In most scenarios, only one will be present, but it is illegal for this set to be an empty collection. + :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the :term:`cache server`'s IPv4 (or IPv6) address + :hashId: The (short) hostname for the :term:`cache server` - named "hashId" for legacy reasons + :hostName: The (short) hostname of the :term:`cache server` + :interfacename: The name of the network interface device being used by the :term:`cache server`'s HTTP proxy + :ip6: The :term:`cache server`'s IPv6 address - when applicable + :ip: The :term:`cache server`'s IPv4 address + :port: The port on which the :term:`cache server` listens for incoming connections + :profile: A string that is the :ref:`profile-name` of the :term:`Profile` assigned to this :term:`cache server` + :status: The status of the :term:`cache server` + :type: A string that names the :term:`Type` of the :term:`cache server` - should (ideally) be either ``EDGE`` or ``MID`` .. code-block:: http :caption: Response Example diff --git a/docs/source/api/v3/cdns_name_configs_monitoring.rst b/docs/source/api/v3/cdns_name_configs_monitoring.rst index 53e016903c..a689464fd6 100644 --- a/docs/source/api/v3/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v3/cdns_name_configs_monitoring.rst @@ -60,7 +60,7 @@ Response Structure :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status - :topology: A string that is the name of the Delivery Service's topology (if assigned one) + :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle :type: A string that is the Delivery Service's type category (HTTP or DNS) @@ -89,7 +89,7 @@ Response Structure :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :cachegroup: The name of the :term:`Cache Group` to which this Traffic Monitor belongs :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance :hostname: The hostname of the server running this Traffic Monitor instance :ip6: The IPv6 address of this Traffic Monitor - when applicable diff --git a/docs/source/api/v4/cdns_name_configs_monitoring.rst b/docs/source/api/v4/cdns_name_configs_monitoring.rst index 011c7a6449..f4083f09c4 100644 --- a/docs/source/api/v4/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v4/cdns_name_configs_monitoring.rst @@ -61,7 +61,7 @@ Response Structure :hostRegexes: An array of strings which are the Delivery Service's HOST_REGEXP-type regexes :status: The :term:`Delivery Service`'s status - :topology: A string that is the name of the Delivery Service's topology (if assigned one) + :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle :type: A string that is the Delivery Service's type category (HTTP or DNS) @@ -90,7 +90,7 @@ Response Structure :trafficMonitors: An array of objects representing each Traffic Monitor that monitors this CDN (this is used by Traffic Monitor's "peer polling" function) - :cachegroup: The :term:`Cache Group` to which this Traffic Monitor belongs + :cachegroup: The name of the :term:`Cache Group` to which this Traffic Monitor belongs :fqdn: An :abbr:`FQDN (Fully Qualified Domain Name)` that resolves to the IPv4 (and/or IPv6) address of the server running this Traffic Monitor instance :hostname: The hostname of the server running this Traffic Monitor instance :ip6: The IPv6 address of this Traffic Monitor - when applicable diff --git a/lib/go-tc/traffic_router.go b/lib/go-tc/traffic_router.go index bf6dd11098..3d3dcbaeea 100644 --- a/lib/go-tc/traffic_router.go +++ b/lib/go-tc/traffic_router.go @@ -249,6 +249,7 @@ func (ts *TrafficServer) IPv6() string { return *lid.IP6Address } +// TSDeliveryService represents a delivery service as assigned to a TrafficServer in the TMConfig. type TSDeliveryService struct { XmlId string `json:"xmlId"` } diff --git a/traffic_monitor/todata/todata.go b/traffic_monitor/todata/todata.go index 96bb7e9bfc..8991e5b2d2 100644 --- a/traffic_monitor/todata/todata.go +++ b/traffic_monitor/todata/todata.go @@ -163,20 +163,20 @@ func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn stri // TODO: remove the fallback behavior on the CRConfig in ATC 8.0 (https://github.com/apache/trafficcontrol/issues/6627) newTOData.DeliveryServiceTypes, err = getDeliveryServiceTypes(crConfig, mc) if err != nil { - return fmt.Errorf("getting delivery service types from Traffic Ops: %v\n", err) + return fmt.Errorf("getting delivery service types from Traffic Ops: %v", err) } // TODO: remove the fallback behavior on the CRConfig in ATC 8.0 (https://github.com/apache/trafficcontrol/issues/6627) newTOData.DeliveryServiceRegexes, err = getDeliveryServiceRegexes(crConfig, mc) if err != nil { - return fmt.Errorf("getting delivery service regexes from Traffic Ops: %v\n", err) + return fmt.Errorf("getting delivery service regexes from Traffic Ops: %v", err) } newTOData.ServerCachegroups = getServerCachegroups(mc) newTOData.ServerTypes, err = getServerTypes(mc) if err != nil { - return fmt.Errorf("getting server types from monitoring config: %v\n", err) + return fmt.Errorf("getting server types from monitoring config: %v", err) } d.set(newTOData) @@ -252,7 +252,7 @@ func getDeliveryServiceRegexes(crc CRConfig, mc tc.TrafficMonitorConfigMap) (Reg if canUseMonitorConfig(mc) { for dsName, dsData := range mc.DeliveryService { if len(dsData.HostRegexes) < 1 { - log.Warnln("TMConfig missing regex for delivery service " + string(dsName)) + log.Warnln("TMConfig missing regex for delivery service " + dsName) continue } dsRegexes[tc.DeliveryServiceName(dsName)] = dsData.HostRegexes diff --git a/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go b/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go index e90471e578..09a1c9e187 100644 --- a/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go +++ b/traffic_ops/traffic_ops_golang/dbhelpers/db_helpers.go @@ -543,6 +543,9 @@ func GetDSTenantIDFromXMLID(tx *sql.Tx, xmlid string) (int, bool, error) { return id, true, nil } +// GetServerDSNamesByCDN returns a map of ONLINE/REPORTED/ADMIN_DOWN cache names to slice of +// strings which are the XML IDs of the active, non-ANYMAP delivery services to which the cache +// is assigned in the given CDN. func GetServerDSNamesByCDN(tx *sql.Tx, cdn string) (map[tc.CacheName][]string, error) { q := ` SELECT s.host_name, ds.xml_id diff --git a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go index b5ac2a8f4c..d9be7e5eaa 100644 --- a/traffic_ops/traffic_ops_golang/monitoring/monitoring.go +++ b/traffic_ops/traffic_ops_golang/monitoring/monitoring.go @@ -166,7 +166,7 @@ func GetMonitoringJSON(tx *sql.Tx, cdnName string) (*Monitoring, error) { } topologies, err := topology.MakeTopologies(tx) if err != nil { - return nil, fmt.Errorf("getting topologies: %s", err.Error()) + return nil, fmt.Errorf("getting topologies: %w", err) } return &Monitoring{ From 96625bcc09d46d2e256e2430759263f207d45e1b Mon Sep 17 00:00:00 2001 From: Rawlin Peters Date: Tue, 15 Mar 2022 16:12:45 -0600 Subject: [PATCH 4/4] Fix grave (accent) mistakes, clarify field omission --- docs/source/api/v2/cdns_name_configs_monitoring.rst | 4 ++-- docs/source/api/v3/cdns_name_configs_monitoring.rst | 4 ++-- docs/source/api/v4/cdns_name_configs_monitoring.rst | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/api/v2/cdns_name_configs_monitoring.rst b/docs/source/api/v2/cdns_name_configs_monitoring.rst index e8242bf3cf..1e7cf8181a 100644 --- a/docs/source/api/v2/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v2/cdns_name_configs_monitoring.rst @@ -63,7 +63,7 @@ Response Structure :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle - :type: A string that is the Delivery Service's type category (HTTP or DNS) + :type: A string that is the Delivery Service's type category (``"HTTP"`` or ``"DNS"``) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -101,7 +101,7 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cacheGroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned (this field is omitted entirely if no delivery services are assigned to this cache server) :xmlId: A string which is the XML ID of the delivery service diff --git a/docs/source/api/v3/cdns_name_configs_monitoring.rst b/docs/source/api/v3/cdns_name_configs_monitoring.rst index a689464fd6..9bc775c354 100644 --- a/docs/source/api/v3/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v3/cdns_name_configs_monitoring.rst @@ -63,7 +63,7 @@ Response Structure :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle - :type: A string that is the Delivery Service's type category (HTTP or DNS) + :type: A string that is the Delivery Service's type category (``"HTTP"`` or ``"DNS"``) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -101,7 +101,7 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned (this field is omitted entirely if no delivery services are assigned to this cache server) :xmlId: A string which is the XML ID of the delivery service diff --git a/docs/source/api/v4/cdns_name_configs_monitoring.rst b/docs/source/api/v4/cdns_name_configs_monitoring.rst index f4083f09c4..552be5f98c 100644 --- a/docs/source/api/v4/cdns_name_configs_monitoring.rst +++ b/docs/source/api/v4/cdns_name_configs_monitoring.rst @@ -64,7 +64,7 @@ Response Structure :topology: A string that is the name of the Delivery Service's :term:`Topology` (if assigned one) :totalKbpsThreshold: A threshold rate of data transfer this :term:`Delivery Service` is configured to handle, in Kilobits per second :totalTpsThreshold: A threshold amount of transactions per second that this :term:`Delivery Service` is configured to handle - :type: A string that is the Delivery Service's type category (HTTP or DNS) + :type: A string that is the Delivery Service's type category (``"HTTP"`` or ``"DNS"``) :xmlId: A string that is the :ref:`Delivery Service's XMLID ` :profiles: An array of the :term:`Profiles` in use by the :term:`cache servers` and :term:`Delivery Services` belonging to this CDN @@ -102,7 +102,7 @@ Response Structure :trafficServers: An array of objects that represent the :term:`cache servers` being monitored within this CDN :cachegroup: The :term:`Cache Group` to which this :term:`cache server` belongs - :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned + :deliveryServices: An array of objects which contain the XML IDs of the delivery services to which this cache server is assigned (this field is omitted entirely if no delivery services are assigned to this cache server) :xmlId: A string which is the XML ID of the delivery service