diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index 174a0962e..788bee9f4 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -22,10 +22,10 @@ #### Description - + #### Deployment - - - + + + diff --git a/.gitignore b/.gitignore index 76373e102..29a7a38ad 100644 --- a/.gitignore +++ b/.gitignore @@ -115,3 +115,4 @@ logs # vim *~ *.swp +*.iml diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 48a25d422..7c1e3d5f3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -58,7 +58,7 @@ $ py.test --cov ./ --cov-report=html $ open htmlcov/index.html ``` -If you are running our functional tests you will need a real BIG-IP® to run +If you are running our functional tests you will need a real BIG-IP to run them against; you can get one of those pretty easily in [Amazon EC2](https://aws.amazon.com/marketplace/pp/B00JL3UASY/ref=srh_res_product_title?ie=UTF8&sr=0-10&qid=1449332167461). ## License @@ -77,4 +77,4 @@ See the License for the specific language governing permissions and limitations under the License. ### Contributor License Agreement -Individuals or business entities who contribute to this project must have completed and submitted the [F5® Contributor License Agreement](http://f5-openstack-docs.readthedocs.org/en/latest/cla_landing.html#cla-landing) to Openstack_CLA@f5.com prior to their code submission being included in this project. +Individuals or business entities who contribute to this project must have completed and submitted the [F5 Contributor License Agreement](http://f5-openstack-docs.readthedocs.org/en/latest/cla_landing.html#cla-landing) to Openstack_CLA@f5.com prior to their code submission being included in this project. diff --git a/README.rst b/README.rst index 5e21abfd0..c9b7a5b16 100644 --- a/README.rst +++ b/README.rst @@ -24,7 +24,7 @@ f5-openstack-agent Introduction ************ -The F5® agent translates from 'OpenStack' to 'F5®'. It uses the `f5-sdk `_ to translate OpenStack messaging calls -- such as those from the Neutron RPC messaging queue -- into iControl® REST calls to F5® technologies, such as BIG-IP®. +The F5 agent translates from 'OpenStack' to 'F5'. It uses the `f5-sdk `_ to translate OpenStack messaging calls -- such as those from the Neutron RPC messaging queue -- into iControl REST calls to F5 technologies, such as BIG-IP. Documentation ************* @@ -34,9 +34,9 @@ Documentation is published on Read the Docs, at http://f5-openstack-agent.readth Compatibility ************* -The F5® OpenStack agent is compatible with OpenStack releases from Liberty forward. If you are using Kilo or earlier, you'll need the `LBaaSv1 plugin `_. +The F5 OpenStack agent is compatible with OpenStack releases from Liberty forward. If you are using Kilo or earlier, you'll need the `LBaaSv1 plugin `_. -See the `F5® OpenStack Releases and Support Matrix `_ for more information. +See the `F5 OpenStack Releases and Support Matrix `_ for more information. Installation ************ @@ -59,7 +59,7 @@ Test **** Before you open a pull request, your code must have passing `pytest `__ unit tests. In addition, you should -include a set of functional tests written to use a real BIG-IP® device +include a set of functional tests written to use a real BIG-IP device for testing. Information on how to run our set of tests is included below. @@ -160,7 +160,7 @@ limitations under the License. Contributor License Agreement ============================= -Individuals or business entities who contribute to this project must have completed and submitted the `F5® Contributor License Agreement `_ to Openstack\_CLA@f5.com prior to their code submission being included in this project. +Individuals or business entities who contribute to this project must have completed and submitted the `F5 Contributor License Agreement `_ to Openstack\_CLA@f5.com prior to their code submission being included in this project. .. |Build Status| image:: https://travis-ci.org/F5Networks/f5-openstack-agent.svg?branch=liberty diff --git a/dev_install b/dev_install new file mode 100755 index 000000000..c179ee0b9 --- /dev/null +++ b/dev_install @@ -0,0 +1,6 @@ +#git init +python setup.py install + +python /var/lib/openstack/bin/f5-oslbaasv2-agent --config-file /etc/neutron/f5-oslbaasv2-agent.ini --config-file /etc/neutron/neutron.conf --log-file /var/log/neutron/f5-agent.log + +#python /var/lib/kolla/venv/bin/f5-oslbaasv2-purge --config-file /etc/neutron/f5-oslbaasv2-agent.ini --partition 234 \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 252cd882f..d60465e14 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,6 +1,6 @@ .. _home: -F5® OpenStack Agent +F5 OpenStack Agent =================== |Build Status| |Docs Build Status| @@ -28,13 +28,13 @@ Installation .. include:: topic_install-f5-agent.rst :start-line: 3 -For more information about using F5® technologies in OpenStack with Neutron LBaaSv2, please see the :ref:`f5-openstack-lbaasv2-driver documentation `. +For more information about using F5 technologies in OpenStack with Neutron LBaaSv2, please see the :ref:`f5-openstack-lbaasv2-driver documentation `. Configuration and Usage ----------------------- -See the :ref:`F5® OpenStack LBaaSv2 documentation `. +See the :ref:`F5 OpenStack LBaaSv2 documentation `. .. |Build Status| image:: https://travis-ci.org/F5Networks/f5-openstack-agent.svg?branch=liberty diff --git a/docs/ref_agent-config-file.rst b/docs/ref_agent-config-file.rst index 62944b424..d87e8b5a2 100644 --- a/docs/ref_agent-config-file.rst +++ b/docs/ref_agent-config-file.rst @@ -4,7 +4,7 @@ Agent Configuration File ------------------------ -A sample F5® OpenStack agent configuration file is shown below. The file can be found at ``/etc/neutron/services/f5/f5-openstack-agent.ini``. When setting up your own F5® agent(s), be sure to use the correct information for your environment. +A sample F5 OpenStack agent configuration file is shown below. The file can be found at ``/etc/neutron/services/f5/f5-openstack-agent.ini``. When setting up your own F5 agent(s), be sure to use the correct information for your environment. .. literalinclude:: ../etc/neutron/services/f5/f5-openstack-agent.ini diff --git a/etc/init.d/f5-oslbaasv2-agent b/etc/init.d/f5-oslbaasv2-agent index 5959aad0a..1e10b26e6 100755 --- a/etc/init.d/f5-oslbaasv2-agent +++ b/etc/init.d/f5-oslbaasv2-agent @@ -8,7 +8,7 @@ # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: f5-openstack-agent -# Description: Provides the F5® OpenStack agent to configure BIG-IP® +# Description: Provides the F5 OpenStack agent to configure BIG-IP ### END INIT INFO PROJECT_NAME=neutron diff --git a/etc/neutron/services/f5/esd/esd.json b/etc/neutron/services/f5/esd/esd.json new file mode 100644 index 000000000..e1e2914aa --- /dev/null +++ b/etc/neutron/services/f5/esd/esd.json @@ -0,0 +1,47 @@ +{ + "proxy_protocol_2edF_v1_0": { + "lbaas_fastl4": "", + "lbaas_ctcp": "tcp", + "lbaas_irule": ["proxy_protocol_2edF_v1_0"], + "lbaas_one_connect": "" + }, + "proxy_protocol_V2_e8f6_v1_0": { + "lbaas_fastl4": "", + "lbaas_ctcp": "tcp", + "lbaas_irule": ["cc_proxy_protocol_V2_e8f6_v1_0"], + "lbaas_one_connect": "" + }, + "fastl4_protocol_keepalive_dd2b_v1_0": { + "lbaas_fastl4": "cc_fastl4", + "lbaas_one_connect": "" + }, + "standard_tcp_a3de_v1_0": { + "lbaas_fastl4": "", + "lbaas_ctcp": "tcp", + "lbaas_one_connect": "" + }, + "x_forward_5b6e_v1_0": { + "lbaas_irule": ["cc_x_forward_5b6e_v1_0"] + }, + "one_connect_dd5c_v1_0": { + "lbaas_one_connect": "oneconnect" + }, + "no_one_connect_3caB_v1_0": { + "lbaas_one_connect": "" + }, + "http_compression_e4a2_v1_0": { + "lbaas_http_compression": "cc_http_compression_e4a2_v1_0" + }, + "cookie_encryption_b82a_v1_0": { + "lbaas_irule": ["cc_cookie_encryption_b82a_v1_0"] + }, + "sso_22b0_v1_0": { + "lbaas_irule": ["cc_sso_22b0_v1_0"] + }, + "sso_required_f544_v1_0": { + "lbaas_irule": ["cc_sso_required_f544_v1_0"] + }, + "http_redirect_a26c_v1_0": { + "lbaas_irule": ["cc_http_redirect_a26c_v1_0"] + } +} \ No newline at end of file diff --git a/etc/neutron/services/f5/f5-openstack-agent.ini b/etc/neutron/services/f5/f5-openstack-agent.ini index 0a95754f7..a63ac03e9 100644 --- a/etc/neutron/services/f5/f5-openstack-agent.ini +++ b/etc/neutron/services/f5/f5-openstack-agent.ini @@ -50,17 +50,17 @@ periodic_interval = 10 # Environment Settings ############################################################################### # -# Since many TMOS® object names must start with an alpha character +# Since many TMOS object names must start with an alpha character # the environment_prefix is used to prefix all service objects. # -# Objects created on the BIG-IP® by this agent will have their names prefixed +# Objects created on the BIG-IP by this agent will have their names prefixed # by an environment string. This allows you set this string. The default is # 'Project'. # # WARNING - you should only set this before creating any objects. If you change # it with established objects, the objects created with an alternative prefix, # will no longer be associated with this agent and all objects in neutron -# and on the the BIG-IP® associated with the old environment will need to be managed +# and on the the BIG-IP associated with the old environment will need to be managed # manually. # # environment_prefix = 'Project' @@ -183,10 +183,10 @@ f5_external_physical_mappings = default:1.1:True # Some systems require the need to bind and prune VLANs ids # allowed to specific ports, often for security. # -# An example would be if a LBaaS iControl® endpoint is using +# An example would be if a LBaaS iControl endpoint is using # tagged VLANs. When a VLAN tagged network is added to a -# specific BIG-IP® device, the facing switch port will need -# to allow traffic for that VLAN tag through to the BIG-IP®'s +# specific BIG-IP device, the facing switch port will need +# to allow traffic for that VLAN tag through to the BIG-IP's # port for traffic to flow. # # What is required is a software hook which allows the binding. @@ -197,16 +197,16 @@ f5_external_physical_mappings = default:1.1:True # vlan_binding_driver = f5.oslbaasv1agent.drivers.bigip.vlan_binding.NullBinding # # The interface_port_static_mappings allows for a JSON encoded dictionary -# mapping BIG-IP® devices and interfaces to corresponding ports. A port id can be +# mapping BIG-IP devices and interfaces to corresponding ports. A port id can be # any string which is meaningful to a vlan_binding_driver. It can be a # switch_id and port, or it might be a neutron port_id. # -# In addition to any static mappings, when the iControl® endpoints +# In addition to any static mappings, when the iControl endpoints # are initialized, all their TMM interfaces will be collect # for each device and neutron will be queried to see if which # device port_ids correspond to known neutron ports. If they do, # automatic entries for all mapped port_ids will be made referencing -# the BIG-IP® device name and interface and the neutron port_ids. +# the BIG-IP device name and interface and the neutron port_ids. # # interface_port_static_mappings = {"device_name_1":{"interface_ida":"port_ida","interface_idb":"port_idb"}, {"device_name_2":{"interface_ida":"port_ida","interface_idb":"port_idb"}} # @@ -216,7 +216,7 @@ f5_external_physical_mappings = default:1.1:True # # Device Tunneling (VTEP) Self IPs # -# This is the name of a BIG-IP® self IP address to use for VTEP addresses. +# This is the name of a BIG-IP self IP address to use for VTEP addresses. # # If no gre or vxlan tunneling is required, these settings should be # commented out or set to None. @@ -265,10 +265,10 @@ f5_populate_static_arp = False # # Device Tunneling (VTEP) self IPs # -# This is a boolean entry which determines if the BIG-IP® will use +# This is a boolean entry which determines if the BIG-IP will use # L2 Population service to update its fdb tunnel entries. This needs # to be set up in accordance with the way the other tunnel agents are -# set up. If the BIG-IP® agent and other tunnel agents don't match +# set up. If the BIG-IP agent and other tunnel agents don't match # the tunnel setup will not work properly. # l2_population = True @@ -303,13 +303,13 @@ l2_population = True # L3 Segmentation Mode Settings ############################################################################### # -# Global Routed Mode - No L2 or L3 Segmentation on BIG-IP® +# Global Routed Mode - No L2 or L3 Segmentation on BIG-IP # # This setting will cause the agent to assume that all VIPs # and pool members will be reachable via global device -# L3 routes, which must be already provisioned on the BIG-IP®s. +# L3 routes, which must be already provisioned on the BIG-IPs. # -# In f5_global_routed_mode, BIG-IP® will not assume L2 +# In f5_global_routed_mode, BIG-IP will not assume L2 # adjacentcy to any neutron network, therefore no # L2 segementation between tenant services in the data plane # will be provisioned by the agent. Because the routing @@ -320,22 +320,22 @@ l2_population = True # # WARNING: setting this mode to True will override # the use_namespaces, setting it to False, because only -# one global routing space will used on the BIG-IP®. This +# one global routing space will used on the BIG-IP. This # means overlapping IP addresses between tenants is no # longer supported. # # WARNING: setting this mode to True will override # the f5_snat_mode, setting it to True, because pool members -# will never be considered L2 adjacent to the BIG-IP® by +# will never be considered L2 adjacent to the BIG-IP by # the agent. All member access will be via L3 routing, which -# will need to be set up on the BIG-IP® before LBaaS provisions +# will need to be set up on the BIG-IP before LBaaS provisions # resources on behalf of tenants. # # WARNING: setting this mode to True will override the # f5_snat_addresses_per_subnet, setting it to 0 (zero). # This will force all VIPs to use AutoMap SNAT for which # enough Self IP will need to be pre-provisioned on the -# BIG-IP® to handle all pool member connections. The SNAT, +# BIG-IP to handle all pool member connections. The SNAT, # an L3 mechanism, will all be global without reference # to any specific tenant SNAT pool. # @@ -344,7 +344,7 @@ l2_population = True # because no L2 information will be taken from # neutron, thus making the assumption that all VIP # L3 addresses will be globally routable without -# segmentation at L2 on the BIG-IP®. +# segmentation at L2 on the BIG-IP. # f5_global_routed_mode = True # @@ -399,14 +399,14 @@ f5_route_domain_strictness = False # This setting will force the use of SNATs. # # If this is set to False, a SNAT will not -# be created (routed mode) and the BIG-IP® +# be created (routed mode) and the BIG-IP # will attempt to set up a floating self IP # as the subnet's default gateway address. # and a wild card IP forwarding virtual # server will be set up on member's network. # Setting this to False will mean Neutron # floating self IPs will not longer work -# if the same BIG-IP® device is not being used +# if the same BIG-IP device is not being used # as the Neutron Router implementation. # # This setting will be forced to True if @@ -444,16 +444,16 @@ f5_common_external_networks = True # separated list where if the name is a neutron # network id used for a vip or a pool member, # the network should not be created or deleted -# on the BIG-IP®, but rather assumed that the value +# on the BIG-IP, but rather assumed that the value # is the name of the network already created in # the Common partition with all L3 addresses # assigned to route domain 0. This is useful # for shared networks which are already defined -# on the BIG-IP® prior to LBaaS configuration. The +# on the BIG-IP prior to LBaaS configuration. The # network should not be managed by the LBaaS agent, # but can be used for VIPs or pool members # -# If your Internet VLAN on your BIG-IP® is named +# If your Internet VLAN on your BIG-IP is named # /Common/external, and that corresponds to # Neutron uuid: 71718972-78e2-449e-bb56-ce47cc9d2680 # then the entry would look like: @@ -472,7 +472,7 @@ f5_common_external_networks = True # Some systems require the need to bind L3 addresses # to specific ports, often for security. # -# An example would be if a LBaaS iControl® endpoint is using +# An example would be if a LBaaS iControl endpoint is using # untagged VLANs and is a nova guest instance. By # default, neutron will attempt to apply security rule # for anti-spoofing which will not allow just any L3 @@ -492,7 +492,7 @@ f5_common_external_networks = True # vary between providers. They may look like a neutron port id # and a nova guest instance id. # -# In addition to any static mappings, when the iControl® endpoints +# In addition to any static mappings, when the iControl endpoints # are initialized, all their TMM MAC addresses will be collect # and neutron will be queried to see if the MAC addresses # correspond to known neutron ports. If they do, automatic entries @@ -511,7 +511,7 @@ f5_bigip_lbaas_device_driver = f5_openstack_agent.lbaasv2.drivers.bigip.icontrol # # ############################################################################### -# Device Driver - iControl® Driver Setting +# Device Driver - iControl Driver Setting ############################################################################### # # icontrol_hostname is valid for external device type only. @@ -524,17 +524,17 @@ f5_bigip_lbaas_device_driver = f5_openstack_agent.lbaasv2.drivers.bigip.icontrol # is not standalone, all devices in the sync failover # device group for the hostname specified must have # their management IP address reachable to the agent. -# If order to access devices' iControl® interfaces via +# If order to access devices' iControl interfaces via # self IPs, you should specify them as a comma # separated list below. # icontrol_hostname = 10.190.7.232 # -# If you are using vCMP® with VLANs, you will need to configure -# your vCMP® host addresses, in addition to the guests addresses. -# vCMP® Host access is necessary for provisioning VLANs to a guest. -# Use icontrol_hostname for vCMP® guests and icontrol_vcmp_hostname -# for vCMP® hosts. The plug-in will automatically determine +# If you are using vCMP with VLANs, you will need to configure +# your vCMP host addresses, in addition to the guests addresses. +# vCMP Host access is necessary for provisioning VLANs to a guest. +# Use icontrol_hostname for vCMP guests and icontrol_vcmp_hostname +# for vCMP hosts. The plug-in will automatically determine # which host corresponds to each guest. # # icontrol_vcmp_hostname = 192.168.1.245 @@ -585,4 +585,5 @@ os_project_domain_name = default # inherit settings from the parent you define. This must be an existing profile, # and if it does not exist on your BIG-IP system the agent will use the default # profile, clientssl. -f5_parent_ssl_profile = clientssl +f5_parent_ssl_profile = cc_clientssl +f5_parent_https_monitor = /Common/cc_https \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py b/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py index b3859f7ad..648425001 100755 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py @@ -17,8 +17,16 @@ import errno import inspect import sys +import urllib3 +import requests import f5_openstack_agent.lbaasv2.drivers.bigip.exceptions as exceptions +from requests.packages.urllib3.exceptions import InsecureRequestWarning + + +requests.packages.urllib3.disable_warnings(InsecureRequestWarning) +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + try: from oslo_config import cfg @@ -64,7 +72,7 @@ def start(self): self.manager.run_periodic_tasks, None, None - ) # Hmmm.... "tg"? + ) # tg = olso_service thread group to run periodic tasks super(F5AgentService, self).start() diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py b/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py index 8159b852c..f5f71f880 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py @@ -16,7 +16,8 @@ # import datetime -import uuid +import sys +from random import randint from oslo_config import cfg from oslo_log import helpers as log_helpers @@ -27,21 +28,32 @@ from oslo_utils import importutils from neutron.agent import rpc as agent_rpc -from neutron.common import constants as plugin_const -from neutron.common import exceptions as q_exception from neutron.common import topics from neutron import context as ncontext from neutron.plugins.ml2.drivers.l2pop import rpc as l2pop_rpc from neutron_lbaas.services.loadbalancer import constants as lb_const +from neutron_lib import constants as plugin_const +from neutron_lib import exceptions as q_exception from f5_openstack_agent.lbaasv2.drivers.bigip import constants_v2 from f5_openstack_agent.lbaasv2.drivers.bigip import plugin_rpc - +from f5_openstack_agent.lbaasv2.drivers.bigip import utils +from f5_openstack_agent.lbaasv2.drivers.bigip import exceptions as f5_ex LOG = logging.getLogger(__name__) # XXX OPTS is used in (at least) agent.py Maybe move/rename to agent.py OPTS = [ + cfg.IntOpt( + 'periodic_interval', + default=10, + help='Seconds between periodic task runs' + ), + cfg.BoolOpt( + 'start_agent_admin_state_up', + default=True, + help='Should the agent force its admin_state_up to True on boot' + ), cfg.StrOpt( # XXX should we use this with internal classes? 'f5_bigip_lbaas_device_driver', # XXX maybe remove "device" and "f5"? default=('f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver.' @@ -114,8 +126,27 @@ help=( 'Amount of time to wait for a pending service to become active') ), + cfg.IntOpt( + 'f5_errored_services_timeout', + default=60, + help=( + 'Amount of time to wait for a errored service to become active') + ), + cfg.FloatOpt( + 'ccloud_orphans_cleanup_interval', + default=0.0, + help=( + 'Rescheduling interval for orphan cleanup in hours') + ), + cfg.BoolOpt( + 'ccloud_orphans_cleanup_testrun', + default=True, + help='Simulate orphan cleaning without real deletion if set to True' + ) ] +PERIODIC_TASK_INTERVAL = 60 + class LogicalServiceCache(object): """Manage a cache of known services.""" @@ -208,34 +239,80 @@ class LbaasAgentManager(periodic_task.PeriodicTasks): # b --> B target = oslo_messaging.Target(version='1.0') - def __init__(self, conf): + def __init__(self, conf, cli_sync=False): """Initialize LbaasAgentManager.""" super(LbaasAgentManager, self).__init__(conf) LOG.debug("Initializing LbaasAgentManager") + LOG.debug("runtime environment: %s" % sys.version) + self.cli_sync = cli_sync self.conf = conf self.context = ncontext.get_admin_context_without_session() self.serializer = None + global PERIODIC_TASK_INTERVAL + PERIODIC_TASK_INTERVAL = self.conf.periodic_interval + # Create the cache of provisioned services self.cache = LogicalServiceCache() self.last_resync = datetime.datetime.now() + self.service_resync_interval = conf.service_resync_interval + LOG.debug('setting service resync interval to %d seconds' % self.service_resync_interval) + + + # calculate last resync date in a way that not all the agents do it at a same time when they got redeployed + # with that first agent will resync after start_delay seconds, second after start_delay*2 secs, ... + max_grps = 3 + if self.conf.environment_group_number: + grp_nr = int(self.conf.environment_group_number) + else: + grp_nr = randint(1, max_grps) + # Hack for QA with more than 3 env_grps + if grp_nr > max_grps: + max_grps = max_grps*2 + + rsi = self.service_resync_interval + start_delay = int(rsi / max_grps) + self.last_resync = datetime.datetime.now() - datetime.timedelta(seconds=(start_delay*(max_grps-grp_nr)+max_grps)) + + LOG.info('ccloud: Periodic resync interval = %s', self.service_resync_interval) + LOG.info('ccloud: Periodic resync triggered by timer of ALL objects will be done latest after %s UTC', self.last_resync + datetime.timedelta(seconds=self.service_resync_interval)) + + + # get orphan cleanup interval and set to a value between 0 and 24 if nonsense given + orphans_interval = float(self.conf.ccloud_orphans_cleanup_interval) + if orphans_interval < 0.0: + orphans_interval = 0.0 + elif orphans_interval > 24.0: + orphans_interval = 24.0 + + self.orphans_cleanup_interval = 3600 * orphans_interval + orphan_delay = int(self.orphans_cleanup_interval / max_grps) + self.last_clean_orphans = self.last_resync - datetime.timedelta(seconds=(orphan_delay*(max_grps - grp_nr )+max_grps)) + + LOG.info('ccloud: Orphan cleanup testrun = %s', self.conf.ccloud_orphans_cleanup_testrun) + LOG.info('ccloud: Orphan cleanup interval = %s', self.orphans_cleanup_interval) + LOG.info('ccloud: Orphan cleanup first run will start at %s UTC', self.last_clean_orphans + datetime.timedelta(seconds=self.orphans_cleanup_interval)) + self.needs_resync = False + # used after recovering of errored devices + self.forced_resync = False + self.forced_resync_tries = 0 self.plugin_rpc = None + self.tunnel_rpc = None + self.l2_pop_rpc = None + self.state_rpc = None self.pending_services = {} - self.service_resync_interval = conf.service_resync_interval - LOG.debug('setting service resync intervl to %d seconds' % - self.service_resync_interval) # Set the agent ID if self.conf.agent_id: self.agent_host = self.conf.agent_id - LOG.debug('setting agent host to %s' % self.agent_host) else: self.agent_host = conf.host + LOG.debug('setting agent host to %s' % self.agent_host) - # Load the iControl® driver. + # Load the iControl driver. self._load_driver(conf) # Initialize agent configurations @@ -251,7 +328,9 @@ def __init__(self, conf): if len(nv) > 1: agent_configurations[nv[0]] = nv[1] - # Initialize agent-state + # Initialize agent-state to a default values + self.admin_state_up = self.conf.start_agent_admin_state_up + self.agent_state = { 'binary': constants_v2.AGENT_BINARY_NAME, 'host': self.agent_host, @@ -262,19 +341,36 @@ def __init__(self, conf): 'start_flag': True } - self.admin_state_up = True + # Setup RPC for communications to and from controller + self._setup_rpc() - # Set iControl® driver context for RPC. + # Set driver context for RPC. self.lbdriver.set_context(self.context) + # Allow the driver to make callbacks to the LBaaS driver plugin + self.lbdriver.set_plugin_rpc(self.plugin_rpc) + # Allow the driver to update tunnel endpoints + self.lbdriver.set_tunnel_rpc(self.tunnel_rpc) + # Allow the driver to update forwarding records in the SDN + self.lbdriver.set_l2pop_rpc(self.l2_pop_rpc) - # Setup RPC: - self._setup_rpc() + # Disable state monitoring for utils calls like druckhammer, ... + if not self.cli_sync: + # Allow the driver to force and agent state report to the controller + self.lbdriver.set_agent_report_state(self._report_state) - # Allow driver to run post init process not that the RPC is all setup. - self.lbdriver.post_init() + # Set the flag to resync tunnels/services + self.needs_resync = True - # Set the flag to resync tunnels/services - self.needs_resync = True + # Mark this agent admin_state_up per startup policy + if(self.admin_state_up): + self.plugin_rpc.set_agent_admin_state(self.admin_state_up) + + # Start state reporting of agent to Neutron + report_interval = self.conf.AGENT.report_interval + if report_interval: + heartbeat = loopingcall.FixedIntervalLoopingCall( + self._report_state) + heartbeat.start(interval=report_interval) def _load_driver(self, conf): self.lbdriver = None @@ -285,21 +381,7 @@ def _load_driver(self, conf): self.lbdriver = importutils.import_object( conf.f5_bigip_lbaas_device_driver, self.conf) - - if self.lbdriver.initialized: - if not self.conf.agent_id: - # If not set statically, add the driver agent env hash - agent_hash = str( - uuid.uuid5(uuid.NAMESPACE_DNS, - self.conf.environment_prefix + - '.' + self.lbdriver.hostnames[0]) - ) - self.agent_host = conf.host + ":" + agent_hash - LOG.debug('setting agent host to %s' % self.agent_host) - else: - LOG.error('Driver did not initialize. Fix the driver config ' - 'and restart the agent.') - return + return except ImportError as ie: msg = ('Error importing loadbalancer device driver: %s error %s' % (conf.f5_bigip_lbaas_device_driver, repr(ie))) @@ -308,12 +390,20 @@ def _load_driver(self, conf): def _setup_rpc(self): - # LBaaS Plugin API + # + # Setting up outbound (callbacks) communications from agent + # + + # setup the topic to send oslo messages RPC calls + # from this agent to the controller topic = constants_v2.TOPIC_PROCESS_ON_HOST_V2 if self.conf.environment_specific_plugin: topic = topic + '_' + self.conf.environment_prefix LOG.debug('agent in %s environment will send callbacks to %s' % (self.conf.environment_prefix, topic)) + + # create our class we will use to send callbacks to the controller + # for processing by the driver plugin self.plugin_rpc = plugin_rpc.LBaaSv2PluginRPC( topic, self.context, @@ -322,67 +412,91 @@ def _setup_rpc(self): self.agent_host ) - # Allow driver to make callbacks using the - # same RPC proxy as the manager - self.lbdriver.set_plugin_rpc(self.plugin_rpc) + # + # Setting up outbound communcations with the neutron agent extension + # + self.state_rpc = agent_rpc.PluginReportStateAPI(topic) - self._setup_state_rpc(topic) + # + # Setting up all inbound notifications and outbound callbacks + # for standard neutron agent services: + # + # tunnel_sync - used to advertise the driver VTEP endpoints + # and optionally learn about other VTEP endpoints + # + # update - used to get updates to agent state triggered by + # the controller, like setting admin_state_up + # the agent + # + # l2_populateion - used to get updates on neutron SDN topology + # changes + # + # We only establish notification if we care about L2/L3 updates + # - # Setup message queues to listen for updates from - # Neutron. if not self.conf.f5_global_routed_mode: - # Core plugin - self.lbdriver.set_tunnel_rpc(agent_rpc.PluginApi(topics.PLUGIN)) + # notifications when tunnel endpoints get added + self.tunnel_rpc = agent_rpc.PluginApi(topics.PLUGIN) + + # define which controler notifications the agent comsumes consumers = [[constants_v2.TUNNEL, topics.UPDATE]] + + # if we are dynamically changing tunnel peers, + # register to recieve and send notificatoins via RPC if self.conf.l2_population: - # L2 Populate plugin Callbacks API - self.lbdriver.set_l2pop_rpc( - l2pop_rpc.L2populationAgentNotifyAPI()) + # communications of notifications from the + # driver to neutron for SDN topology changes + self.l2_pop_rpc = l2pop_rpc.L2populationAgentNotifyAPI() + # notification of SDN topology updates from the + # controller by adding to the general consumer list consumers.append( [topics.L2POPULATION, topics.UPDATE, self.agent_host] ) + # kick off the whole RPC process by creating + # a connection to the message bus self.endpoints = [self] - self.connection = agent_rpc.create_consumers( self.endpoints, topics.AGENT, consumers ) - def _setup_state_rpc(self, topic): - # Agent state API - self.state_rpc = agent_rpc.PluginReportStateAPI(topic) - report_interval = self.conf.AGENT.report_interval - if report_interval: - heartbeat = loopingcall.FixedIntervalLoopingCall( - self._report_state) - heartbeat.start(interval=report_interval) - - def _report_state(self): - + def _report_state(self, force_resync=False): try: - # assure the agent is connected - # FIXME: what happens if we can't connect. - if not self.lbdriver.connected: - self.lbdriver.connect() - - service_count = self.cache.size - self.agent_state['configurations']['services'] = service_count - if hasattr(self.lbdriver, 'service_queue'): - self.agent_state['configurations']['request_queue_depth'] = ( - len(self.lbdriver.service_queue) - ) + if force_resync: + self.needs_resync = True + self.cache.services = {} + self.lbdriver.flush_cache() + # use the admin_state_up to notify the + # controller if all backend devices + # are functioning properly. If not + # automatically set the admin_state_up + # for this agent to False + if self.lbdriver: + if not self.lbdriver.backend_integrity(): + self.needs_resync = True + self.cache.services = {} + self.lbdriver.flush_cache() + self.plugin_rpc.set_agent_admin_state(False) + self.admin_state_up = False + else: + # if we are transitioning from down to up, + # change the controller state for this agent + if not self.admin_state_up: + self.plugin_rpc.set_agent_admin_state(True) + self.admin_state_up = True - # Add configuration from icontrol_driver. - if self.lbdriver.agent_configurations: + if self.lbdriver: self.agent_state['configurations'].update( - self.lbdriver.agent_configurations + self.lbdriver.get_agent_configurations() ) - # Compute the capacity score. + # add the capacity score, used by the scheduler + # for horizontal scaling of an environment, from + # the driver if self.conf.capacity_policy: env_score = ( self.lbdriver.generate_capacity_score( @@ -390,7 +504,7 @@ def _report_state(self): ) ) self.agent_state['configurations'][ - 'environment_capaciy_score'] = env_score + 'environment_capacity_score'] = env_score else: self.agent_state['configurations'][ 'environment_capacity_score'] = 0 @@ -398,9 +512,14 @@ def _report_state(self): LOG.debug("reporting state of agent as: %s" % self.agent_state) self.state_rpc.report_state(self.context, self.agent_state) self.agent_state.pop('start_flag', None) + LOG.debug("ccloud: reporting state of agent succesfully done") + except Exception as e: LOG.exception(("Failed to report state: " + str(e.message))) + # callback from oslo messaging letting us know we are properly + # connected to the message bus so we can register for inbound + # messages to this agent def initialize_service_hook(self, started_by): """Create service hook to listen for messanges on agent topic.""" node_topic = "%s_%s.%s" % (constants_v2.TOPIC_LOADBALANCER_AGENT_V2, @@ -410,37 +529,59 @@ def initialize_service_hook(self, started_by): endpoints = [started_by.manager] started_by.conn.create_consumer( node_topic, endpoints, fanout=False) - self.sync_state() - @periodic_task.periodic_task(spacing=10) - def periodic_resync(self, context): - """Resync tunnels/service state.""" - now = datetime.datetime.now() - LOG.debug("%s: periodic_resync called." % now) + @periodic_task.periodic_task(spacing=PERIODIC_TASK_INTERVAL, run_immediately=True) + def connect_driver(self, context): - # Only force resync if the agent thinks it is - # synchronized and the resync timer has exired - if (now - self.last_resync).seconds > self.service_resync_interval: - if not self.needs_resync: - self.needs_resync = True - LOG.debug( - 'Forcing resync of services on resync timer (%d seconds).' - % self.service_resync_interval) - self.cache.services = {} - self.last_resync = now - self.lbdriver.flush_cache() - LOG.debug("periodic_sync: service_resync_interval expired: %s" - % str(self.needs_resync)) - # resync if we need to - if self.needs_resync: - self.needs_resync = False - if self.tunnel_sync(): - self.needs_resync = True - if self.sync_state(): + if self.cli_sync: + return + + """Trigger driver connect attempts to all devices.""" + if self.lbdriver: + self.lbdriver.connect() + + @periodic_task.periodic_task(spacing=(PERIODIC_TASK_INTERVAL/2)) + def recover_errored_devices(self, context): + + if self.cli_sync: + return + + """Try to reconnect to errored devices.""" + if self.lbdriver: + LOG.debug("running periodic task to recover disconnected BIG-IPs") + recovered = self.lbdriver.recover_errored_devices() + # clear the cache to resync everything in case of a recovery + if recovered: self.needs_resync = True + self.forced_resync = True + + # Taken from actual mitaka for documentation purpose. Functionality also disabled in f5 neutron driver + # Disabled because it makes no sense to move all Objects from a device group to another only because agent is down. + # Changes can't be made to objects hosted by agents but traffic won't be affected. + # Movement would affect traffic because of ARP update issues in ASR + @periodic_task.periodic_task( + spacing=constants_v2.UPDATE_OPERATING_STATUS_INTERVAL) + def scrub_dead_agents_in_env_and_group(self, context): - @periodic_task.periodic_task(spacing=30) + if self.cli_sync: + return + + """Triggering a dead agent scrub on the controller.""" + LOG.debug("ccloud: scrubbing - running periodic scrub_dead_agents_in_env_and_group for EnvGroup %s", self.conf.environment_group_number) + if not self.plugin_rpc: + return + + self.plugin_rpc.scrub_dead_agents(self.conf.environment_prefix, + self.conf.environment_group_number) + + @periodic_task.periodic_task( + spacing=constants_v2.UPDATE_OPERATING_STATUS_INTERVAL) def update_operating_status(self, context): + + if self.cli_sync: + return + + """Update pool member operational status from devices to controller.""" if not self.plugin_rpc: return @@ -459,142 +600,310 @@ def update_operating_status(self, context): except Exception as e: LOG.exception('Error updating status %s.', e.message) + # setup a period task to decide if it is time empty the local service + # cache and resync service definitions form the controller + @periodic_task.periodic_task(spacing=PERIODIC_TASK_INTERVAL) + def periodic_resync(self, context): + + if self.cli_sync: + return + + """Determine if it is time to resync services from controller.""" + try: + now = datetime.datetime.now() + + # Only force resync if the agent thinks it is + # synchronized and the resync timer has exired + # use forced resync switch which is only set by recovering of errored F5 to guarantee sync + if self.forced_resync: + self.forced_resync_tries += 1 + self.needs_resync = True + self.cache.services = {} + self.lbdriver.flush_cache() + self.last_resync = self.last_resync + datetime.timedelta(seconds=self.service_resync_interval) + LOG.debug("ccloud - periodic_resync: Forcing resync of ALL services because of a recovered F5 device") + elif (now - self.last_resync).seconds > self.service_resync_interval: + if not self.needs_resync: + self.needs_resync = True + self.cache.services = {} + self.lbdriver.flush_cache() + self.last_resync = self.last_resync + datetime.timedelta(seconds=self.service_resync_interval) + LOG.debug('ccloud - periodic_resync: Forcing resync of ALL services on resync timer (%d seconds).' % self.service_resync_interval) + else: + LOG.debug('ccloud - periodic_resync: Forcing resync of NON CACHED services on resync timer (%d seconds).' % self.service_resync_interval) + elif self.needs_resync: + LOG.debug('ccloud - periodic_resync: Starting requested resync of NON CACHED services.') + else: + LOG.debug("ccloud - periodic_resync: Waiting minimum {0} seconds for next timer triggered resync".format((self.service_resync_interval - (now - self.last_resync ).seconds))) + + # resync if we need to + if self.needs_resync: + LOG.info('ccloud: periodic_resync: Starting resync ...') + self.needs_resync = False + if self.tunnel_sync(): + self.needs_resync = True + if self.sync_state(): + self.needs_resync = True + if not self.needs_resync or self.forced_resync_tries > 2: + self.forced_resync = False + self.forced_resync_tries = 0 + try: + self.clean_orphaned_snat_objects() + except Exception as e: + LOG.warning("ccloud - Couldn't clear orphan snat objects because of : " + str(e.message)) + else: + LOG.debug("ccloud - periodic_resync: Resync not needed! Discarding ...") + + if self.orphans_cleanup_interval > 0: + if (now - self.last_clean_orphans).seconds > self.orphans_cleanup_interval: + LOG.debug("ccloud - periodic_resync - orphans: Start cleaning orphan objects from F5 device") + self.last_clean_orphans = self.last_clean_orphans + datetime.timedelta(seconds=self.orphans_cleanup_interval) + if self.clean_orphaned_objects_and_save_device_config(): + self.needs_resync = True + orphan_cache = self.lbdriver.get_orphans_cache() + LOG.debug("ccloud - periodic_resync - orphans: Finished cleaning orphan objects from F5 device. {0} objects remaining --> {1}".format(len(orphan_cache), orphan_cache)) + else: + LOG.debug("ccloud - periodic_resync - orphans: Skipping cleaning orphan objects because cleanup interval not expired. Waiting another {0} seconds" + .format((self.last_clean_orphans + datetime.timedelta(seconds=self.orphans_cleanup_interval) - now).seconds)) + else: + LOG.debug("ccloud - periodic_resync - orphans: No orphan cleaning enabled. Only SNAT pool orphan handling will be done") + + LOG.info("ccloud - periodic_resync: Resync took {0} seconds".format((datetime.datetime.now() - now).seconds)) + + except Exception as e: + LOG.exception("ccloud - periodic_resync: Exception in periodic resync happend: " + str(e.message)) + pass + + # ccloud: clean orphaned snat pools + @log_helpers.log_method_call + def clean_orphaned_snat_objects(self): + try: + virtual_addresses = self.lbdriver.get_all_virtual_addresses() + snat_pools = self.lbdriver.get_all_snat_pools() + + for va in virtual_addresses: + snat_obj = self.find_in_collection(va.name.replace('Project_', 'lb_'), snat_pools) + if snat_obj is not None: + snat_pools.remove(snat_obj) + + for orphaned_snat in snat_pools: + LOG.debug("sapcc: purging orphaned snat pool %s" % orphaned_snat.name) + try: + orphaned_snat.delete() + except Exception as e: + LOG.warning("sapcc: attempt made to purge orphaned snat pool which might be in use: " + str(e.message)) + + except Exception as e: + LOG.warning("Unable to clean snat objects: %s" % e.message) + + + def find_in_collection(self, name, collection): + for item in collection: + if item is not None and item.name == name: + return item + return None + def tunnel_sync(self): - """Call into driver to advertise tunnels.""" + """Call into driver to advertise device tunnel endpoints.""" LOG.debug("manager:tunnel_sync: calling driver tunnel_sync") return self.lbdriver.tunnel_sync() @log_helpers.log_method_call + @utils.instrument_execution_time def sync_state(self): - """Sync state of BIG-IP with that of the neutron database.""" + """Synchronize device configuration from controller state.""" resync = False - known_services = set() - owned_services = set() - for lb_id, service in self.cache.services.iteritems(): - known_services.add(lb_id) - if self.agent_host == service.agent_host: - owned_services.add(lb_id) - now = datetime.datetime.now() + if hasattr(self, 'lbdriver'): + if not self.lbdriver.backend_integrity(): + return resync + + known_services, owned_services = self._all_vs_known_services() try: # Get loadbalancers from the environment which are bound to # this agent. - active_loadbalancers = ( - self.plugin_rpc.get_active_loadbalancers(host=self.agent_host) - ) - active_loadbalancer_ids = set( - [lb['lb_id'] for lb in active_loadbalancers] - ) - - all_loadbalancers = ( - self.plugin_rpc.get_all_loadbalancers(host=self.agent_host) - ) - all_loadbalancer_ids = set( - [lb['lb_id'] for lb in all_loadbalancers] - ) + active_loadbalancers, active_loadbalancer_ids = \ + self._get_remote_loadbalancers('get_active_loadbalancers', + host=self.agent_host) + all_loadbalancers, all_loadbalancer_ids = \ + self._get_remote_loadbalancers('get_all_loadbalancers', + host=self.agent_host) LOG.debug("plugin produced the list of active loadbalancer ids: %s" % list(active_loadbalancer_ids)) LOG.debug("currently known loadbalancer ids before sync are: %s" % list(known_services)) + LOG.debug("ccloud: plugin got all loadbalancer ids as: %s" + % list(all_loadbalancer_ids)) + # ccloud: Get rid of 'Cached service not found in neutron database' message + # Clear cache entry if not found in neutron. In case of a temp issue + # lb will be added again with next sync for deleted_lb in owned_services - all_loadbalancer_ids: - LOG.error("Cached service not found in neutron database") + self.cache.remove_by_loadbalancer_id(deleted_lb) + LOG.info("ccloud: Cached service not found in neutron database. Clearing cache for LB_id %s" % deleted_lb) # self.destroy_service(deleted_lb) # Validate each service we own, i.e. loadbalancers to which this # agent is bound, that does not exist in our service cache. - for lb_id in active_loadbalancer_ids: - if not self.cache.get_by_loadbalancer_id(lb_id): - self.validate_service(lb_id) - - # This produces a list of loadbalancers with pending tasks to - # be performed. - pending_loadbalancers = ( - self.plugin_rpc.get_pending_loadbalancers(host=self.agent_host) - ) - pending_lb_ids = set( - [lb['lb_id'] for lb in pending_loadbalancers] - ) - LOG.debug( - "plugin produced the list of pending loadbalancer ids: %s" - % list(pending_lb_ids)) - - for lb_id in pending_lb_ids: - lb_pending = self.refresh_service(lb_id) - if lb_pending: - if lb_id not in self.pending_services: - self.pending_services[lb_id] = now - - time_added = self.pending_services[lb_id] - time_expired = ((now - time_added).seconds > - self.conf.f5_pending_services_timeout) + self._validate_services(all_loadbalancer_ids) - if time_expired: - lb_pending = False - self.service_timeout(lb_id) - - if not lb_pending: - del self.pending_services[lb_id] - - # If there are services in the pending cache resync - if self.pending_services: - resync = True + resync = self._refresh_pending_services() # Get a list of any cached service we now know after # refreshing services - known_services = set() - for (lb_id, service) in self.cache.services.iteritems(): - if self.agent_host == service.agent_host: - known_services.add(lb_id) + owned_services, known_services = self._all_vs_known_services() LOG.debug("currently known loadbalancer ids after sync: %s" % list(known_services)) except Exception as e: - LOG.error("Unable to retrieve ready service: %s" % e.message) + LOG.exception("Unable to sync state: %s" % e.message) resync = True return resync + def _all_vs_known_services(self): + all_services = set() + known_services = set() + for lb_id, service in self.cache.services.iteritems(): + all_services.add(lb_id) + if self.agent_host == service.agent_host: + known_services.add(lb_id) + return all_services, known_services + + def _refresh_pending_services(self): + now = datetime.datetime.now() + resync = False + # This produces a list of loadbalancers with pending tasks to + # be performed. + pending_loadbalancers, pending_lb_ids = \ + self._get_remote_loadbalancers('get_pending_loadbalancers', + host=self.agent_host) + LOG.debug( + "plugin produced the list of pending loadbalancer ids: %s" + % list(pending_lb_ids)) + + for lb_id in list(pending_lb_ids): + lb_pending = self.refresh_service(lb_id) + if lb_pending: + if lb_id not in self.pending_services: + self.pending_services[lb_id] = now + + time_added = self.pending_services[lb_id] + has_expired = bool((now - time_added).seconds > + self.conf.f5_pending_services_timeout) + + if has_expired: + lb_pending = False + self.service_timeout(lb_id) + + if not lb_pending: + try: + del self.pending_services[lb_id] + except KeyError as e: + # ccloud: message makes no sense if lb got deleted in between of self._get_remote_loadbalancers and + # self.refresh_service(lb_id) + pass + #LOG.error("LB not found in pending services: {0}".format( + # e.message)) + + # If there are services in the pending cache resync + if self.pending_services: + resync = True + return resync + + def _get_remote_loadbalancers(self, plugin_rpc_attr, host=None): + loadbalancers = getattr(self.plugin_rpc, plugin_rpc_attr)(host=host) + lb_ids = [lb['lb_id'] for lb in loadbalancers] + return tuple(loadbalancers), set(lb_ids) + + def _validate_services(self, lb_ids): + for lb_id in lb_ids: + if not self.cache.get_by_loadbalancer_id(lb_id): + self.validate_service(lb_id) + @log_helpers.log_method_call + @utils.instrument_execution_time def validate_service(self, lb_id): - try: service = self.plugin_rpc.get_service_by_loadbalancer_id( lb_id ) - self.cache.put(service, self.agent_host) - if not self.lbdriver.service_exists(service): - LOG.error('active loadbalancer %s is not on BIG-IP...syncing' - % lb_id) - - if self.lbdriver.service_rename_required(service): - self.lbdriver.service_object_teardown(service) - LOG.error('active loadbalancer %s is configured with ' - 'non-unique names on BIG-IP...rename in ' - 'progress.' - % lb_id) - LOG.error('removing the service objects that are ' - 'incorrectly named') + try: + found = True + if (not self.lbdriver.service_exists(service)) or self.has_provisioning_status_of_error(service): + LOG.warning("Active loadbalancer '{}' is not on BIG-IP" + " or has error state".format(lb_id)) + found = False else: - LOG.debug('service rename not required') - + LOG.debug("Found service definition for '{}', state is ACTIVE" + " move on.".format(lb_id)) + except Exception as ex: + #iControlUnexpectedHTTPError + LOG.warning("ccloud: Service %s not found on BIGip because of exception %s " % (lb_id, ex.message)) + found = False + # really not found or Exception happend: Try to fix it + if not found: + LOG.info("ccloud: Start syncing loadbalancer '{}'".format(lb_id)) self.lbdriver.sync(service) - else: - LOG.debug("Found service definition for %s" % (lb_id)) + LOG.info("ccloud: Finished syncing loadbalancer '{}'".format(lb_id)) + if service: + self.cache.put(service, self.agent_host) + except f5_ex.InvalidNetworkType as exc: + LOG.warning(exc.msg) except q_exception.NeutronException as exc: LOG.error("NeutronException: %s" % exc.msg) except Exception as exc: LOG.exception("Service validation error: %s" % exc.message) - @log_helpers.log_method_call + + @staticmethod + def has_provisioning_status_of_error(service): + """Determine if a service is in an ERROR/DEGRADED status. + + This staticmethod will go through a service object and determine if it + has an ERROR status anywhere within the object. + """ + expected_tree = dict(loadbalancer=dict, members=list, pools=list, + listeners=list, healthmonitors=list, + l7policies=list, l7policy_rules=list) + error_status = False # assume we're in the clear unless otherwise... + loadbalancer = service.get('loadbalancer', dict()) + + def handle_error(error_status, obj): + provisioning_status = obj.get('provisioning_status') + if provisioning_status == plugin_const.ERROR: + obj_id = obj.get('id', 'unknown') + LOG.warning("Service object has object of type(id) {}({})" + " that is in '{}' status.".format( + item, obj_id, plugin_const.ERROR)) + error_status = True + return error_status + + for item in expected_tree: + obj = service.get(item, expected_tree[item]()) + if expected_tree[item] == dict and isinstance(service[item], dict): + error_status = handle_error(error_status, obj) + elif expected_tree[item] == list and \ + isinstance(obj, list): + for item in obj: + if len(item) == 1: + # {'networks': [{'id': {}}]} + item = item[item.keys()[0]] + error_status = handle_error(error_status, item) + if error_status: + loadbalancer['provisioning_status'] = plugin_const.ERROR + return error_status + + @utils.instrument_execution_time def refresh_service(self, lb_id): try: service = self.plugin_rpc.get_service_by_loadbalancer_id( lb_id ) self.cache.put(service, self.agent_host) + LOG.info("ccloud: refresh_service - get service from rpc '{}' for sync".format(service)) if self.lbdriver.sync(service): self.needs_resync = True except q_exception.NeutronException as exc: @@ -641,12 +950,259 @@ def destroy_service(self, lb_id): self.cache.remove_by_loadbalancer_id(lb_id) @log_helpers.log_method_call - def remove_orphans(self, all_loadbalancers): + def clean_orphaned_objects_and_save_device_config(self): + + cleaned = False try: - self.lbdriver.remove_orphans(all_loadbalancers) - except Exception as exc: - LOG.error("Exception: removing orphans: %s" % exc.message) + + unbound_loadbalancers = self.plugin_rpc.get_loadbalancers_without_agent_binding() + + if len(unbound_loadbalancers) > 0: + # create list with tenant and id for error logging + ubl = [[lb['tenant_id'], lb['id']] for lb in unbound_loadbalancers] + # verify if unbound lb's reside on this agent to give an idea for correction + # the host or unknown will be added to the unbounds + all_lbs = self.lbdriver.get_all_deployed_loadbalancers(purge_orphaned_folders=False) + hosted = 0 + for lbu in ubl: + if lbu[1] in all_lbs: + hosted += 1 + lbu.append("{0}".format(self.agent_host)) + + if hosted > 0: + LOG.error("ccloud: MissingAgentBinding: {0} Loadbalancers with a missing neutron agent binding are hosted on this agent. " + "The binding to this agent {1} has to be repaired in neutron DB".format(hosted, self.agent_host)) + else: + LOG.warning("ccloud: MissingAgentBinding: NO Loadbalancers with a missing neutron agent binding are hosted on this agent {1}. " + "These LBs might be deleted in neutron DB if no other agent is hosting them.".format(hosted, self.agent_host)) + + # Abort cleanup in case of non testrun, otherwise report errors and continue with testrun + if not self.conf.ccloud_orphans_cleanup_testrun: + LOG.error("ccloud: MissingAgentBinding: {2} Loadbalancers without an agent binding found. Orphan cleanup process aborted!!! Agent name: {1}. " + "Manual intervention needed to clarify state of unbound loadbalancers and where they should belong to. " + "If an agent name is given as 3rd argument the agent has detected that it is hosting the LB but binding in neutron DB is missing. " + "If no agent name is given, this agent doesn't host the LB, but maybe another one." + "The following loadbalancers without binding were found [tenant.id, lb.id, ]: {0}".format(ubl, self.agent_host, len(unbound_loadbalancers))) + return False + else: + LOG.error("ccloud: MissingAgentBinding: {2} Loadbalancers without an agent binding found.Orphan cleanup testrun will continue. Agent name: {1}. " + "Manual intervention needed to clarify state of unbound loadbalancers and where they should belong to. " + "If an agent name is given as 3rd argument the agent has detected that it is hosting the LB but binding in neutron DB is missing. " + "If no agent name is given, this agent doesn't host the LB, but maybe another one." + "The following loadbalancers without binding were found [tenant.id, lb.id, ]: {0}".format(ubl, self.agent_host, len(unbound_loadbalancers))) + + # + # Global cluster refresh tasks + # + + # global_agent = self.plugin_rpc.get_clusterwide_agent( + # self.conf.environment_prefix, + # self.conf.environment_group_number + # ) + # + # if 'host' not in global_agent: + # LOG.debug('No global agent available to sync config') + # return True + + # ccloud: Set the agent to the cluster wide one as we have onley one per cluster at the moment + + global_agent = {} + global_agent['host'] = self.agent_host + + if global_agent['host'] == self.agent_host: + LOG.debug('this agent is the global config agent') + # We're the global agent perform global cluster tasks + + # Ask BIG-IP for all deployed loadbalancers (virtual addresses) + lbs = self.lbdriver.get_all_deployed_loadbalancers( + purge_orphaned_folders=True) + if lbs: + self.purge_orphaned_loadbalancers(lbs) + + # Ask the BIG-IP for all deployed listeners to make + # sure we are not orphaning listeners which have + # valid loadbalancers in a OK state + listeners = self.lbdriver.get_all_deployed_listeners() + if listeners: + self.purge_orphaned_listeners(listeners) + + policies = self.lbdriver.get_all_deployed_l7_policys() + if policies: + self.purge_orphaned_l7_policys(policies) + + # Ask the BIG-IP for all deployed pools not associated + # to a virtual server + pools = self.lbdriver.get_all_deployed_pools() + if pools: + self.purge_orphaned_pools(pools) + self.purge_orphaned_nodes(pools) + + # Ask the BIG-IP for all deployed monitors not associated + # to a pool + monitors = self.lbdriver.get_all_deployed_health_monitors() + if monitors: + self.purge_orphaned_health_monitors(monitors) + + else: + LOG.debug('the global agent is %s' % (global_agent['host'])) + cleaned = False + + cleaned = True + # serialize config and save to disk + self.lbdriver.backup_configuration() + except Exception as e: + LOG.error("Unable to clean_orphaned_objects_and_save_device_config: %s" % e.message) + cleaned = True + + return cleaned + + @log_helpers.log_method_call + def purge_orphaned_loadbalancers(self, lbs): + """Gets 'unknown' loadbalancers from Neutron and purges them + + Provisioning status of 'unknown' on loadbalancers means that the object + does not exist in Neutron. These should be deleted to consolidate + hanging objects. + """ + lbs_status = self.plugin_rpc.validate_loadbalancers_state( + list(lbs.keys())) + LOG.debug('validate_loadbalancers_state returned: %s' + % lbs_status) + lbs_removed = False + for lbid in lbs_status: + # If the statu is Unknown, it no longer exists + # in Neutron and thus should be removed from the BIG-IP + if lbs_status[lbid] in ['Unknown']: + LOG.debug('removing orphaned loadbalancer %s' + % lbid) + # This will remove pools, virtual servers and + # virtual addresses + self.lbdriver.purge_orphaned_loadbalancer( + tenant_id=lbs[lbid]['tenant_id'], + loadbalancer_id=lbid, + hostnames=lbs[lbid]['hostnames']) + lbs_removed = True + if lbs_removed: + # If we have removed load balancers, then scrub + # for tenant folders we can delete because they + # no longer contain loadbalancers. + self.lbdriver.get_all_deployed_loadbalancers( + purge_orphaned_folders=True) + + @log_helpers.log_method_call + def purge_orphaned_listeners(self, listeners): + """Deletes the hanging listeners from the deleted loadbalancers""" + listener_status = self.plugin_rpc.validate_listeners_state( + list(listeners.keys())) + LOG.debug('validated_listeners_state returned: %s' + % listener_status) + for listenerid in listener_status: + # If the listener status is Unknown, it no longer exists + # in Neutron and thus should be removed from BIG-IP + if listener_status[listenerid] in ['Unknown']: + LOG.debug('removing orphaned listener %s' + % listenerid) + self.lbdriver.purge_orphaned_listener( + tenant_id=listeners[listenerid]['tenant_id'], + listener_id=listenerid, + hostnames=listeners[listenerid]['hostnames']) + + @log_helpers.log_method_call + def purge_orphaned_l7_policys(self, policies): + """Deletes hanging l7_policies from the deleted listeners""" + policies_used = set() + listeners = self.lbdriver.get_all_deployed_listeners( + expand_subcollections=True) + for li_id in listeners: + policy = listeners[li_id]['l7_policy'] + if policy: + policy = policy.split('/')[2] + policies_used.add(policy) + has_l7policies = \ + self.plugin_rpc.validate_l7policys_state_by_listener( + listeners.keys()) + # Ask Neutron for the status of all deployed l7_policys + for policy_key in policies: + policy = policies.get(policy_key) + purged = False + if policy_key not in policies_used: + LOG.debug("policy '{}' no longer referenced by a listener: " + "({})".format(policy_key, policies_used)) + self.lbdriver.purge_orphaned_l7_policy( + tenant_id=policy['tenant_id'], + l7_policy_id=policy_key, + hostnames=policy['hostnames']) + purged = True + elif not has_l7policies.get(policy['id'], False): + # should always be present on Neutron DB! + LOG.debug("policy '{}' no longer present in Neutron's DB: " + "({})".format(policy_key, has_l7policies)) + self.lbdriver.purge_orphaned_l7_policy( + tenant_id=policy['tenant_id'], + l7_policy_id=policy_key, + hostnames=policy['hostnames'], + listener_id=li_id) + purged = True + if purged: + LOG.info("purging orphaned l7policy {} as it's no longer in " + "Neutron".format(policy_key)) + + @log_helpers.log_method_call + def purge_orphaned_nodes(self, pools): + """Deletes hanging nodes from the deleted listeners""" + pools_members = self.plugin_rpc.get_pools_members( + list(pools.keys())) + + tenant_members = dict() + for pool_id, pool in pools.iteritems(): + tenant_id = pool['tenant_id'] + members = pools_members.get(pool_id, list()) + + if tenant_id not in tenant_members: + tenant_members[tenant_id] = members + else: + tenant_members[tenant_id].extend(members) + + self.lbdriver.purge_orphaned_nodes(tenant_members) + + @log_helpers.log_method_call + def purge_orphaned_pools(self, pools): + """Deletes hanging pools from the deleted listeners""" + # Ask Neutron for the status of all deployed pools + pools_status = self.plugin_rpc.validate_pools_state( + list(pools.keys())) + LOG.debug('validated_pools_state returned: %s' + % pools_status) + for poolid in pools_status: + # If the pool status is Unknown, it no longer exists + # in Neutron and thus should be removed from BIG-IP + if pools_status[poolid] in ['Unknown']: + LOG.debug('removing orphaned pool %s' % poolid) + self.lbdriver.purge_orphaned_pool( + tenant_id=pools[poolid]['tenant_id'], + pool_id=poolid, + hostnames=pools[poolid]['hostnames']) + + @log_helpers.log_method_call + def purge_orphaned_health_monitors(self, monitors): + """Deletes hanging Health Monitors from the deleted Pools""" + # ask Neutron for for the status of all deployed monitors... + monitors_used = set() + pools = self.lbdriver.get_all_deployed_pools() + LOG.debug("pools found: {}".format(pools)) + for pool_id in pools: + monitorid = pools.get(pool_id).get('monitors', 'None') + monitors_used.add(monitorid) + LOG.debug('health monitors in use: {}'.format(monitors_used)) + for monitorid in monitors: + if monitorid not in monitors_used: + LOG.debug("purging healthmonitor {} as it is not " + "in ({})".format(monitorid, monitors_used)) + self.lbdriver.purge_orphaned_health_monitor( + tenant_id=monitors[monitorid]['tenant_id'], + monitor_id=monitorid, + hostnames=monitors[monitorid]['hostnames']) @log_helpers.log_method_call def create_loadbalancer(self, context, loadbalancer, service): @@ -879,15 +1435,15 @@ def delete_health_monitor(self, context, health_monitor, service): def agent_updated(self, context, payload): """Handle the agent_updated notification event.""" if payload['admin_state_up'] != self.admin_state_up: + LOG.info("agent administration status updated %s!", payload) self.admin_state_up = payload['admin_state_up'] - if self.admin_state_up: - # FIXME: This needs to be changed back to True - self.needs_resync = False + # the agent transitioned to down to up and the + # driver reports healthy, trash the cache + # and force an update to update agent scheduler + if self.lbdriver.backend_integrity() and self.admin_state_up: + self._report_state(True) else: - for loadbalancer_id in self.cache.get_loadbalancer_ids(): - LOG.debug("DESTROYING loadbalancer: " + loadbalancer_id) - # self.destroy_service(loadbalancer_id) - LOG.info("agent_updated by server side %s!", payload) + self._report_state(False) @log_helpers.log_method_call def tunnel_update(self, context, **kwargs): diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/barbican_cert.py b/f5_openstack_agent/lbaasv2/drivers/bigip/barbican_cert.py index af4b4fb00..a4644e030 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/barbican_cert.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/barbican_cert.py @@ -103,7 +103,7 @@ def _init_barbican_client(self): # NOTE: Session is deprecated in keystoneclient 2.1.0 # and will be removed in a future keystoneclient release. - sess = Session(auth=auth) + sess = Session(auth=auth, verify=False) self.barbican = Client(session=sess) # test barbican service @@ -147,6 +147,17 @@ def get_private_key(self, container_ref): container = self.barbican.containers.get(container_ref) return container.private_key.payload + def get_private_key_passphrase(self, container_ref): + """Retrieves key passphrase from certificate manager. + + :param string container_ref: Reference to key stored in a + certificate manager. + :returns string: passphrase. + This method MUST be implemented, in agent-compliant cert managers. + """ + container = self.barbican.containers.get(container_ref) + return container.private_key_passphrase.payload + def get_name(self, container_ref, prefix): """Returns a name that uniquely identifies cert/key pair. @@ -165,3 +176,25 @@ def get_name(self, container_ref, prefix): i = container_ref.rindex("/") + 1 return prefix + container_ref[i:] + + def get_container(self, container_ref): + """Retrieves container object from certificate manager. + + :param string container_ref: Reference to container stored in a + certificate manager. + :returns string: Container Object + """ + return self.barbican.containers.get(container_ref) + + def get_intermediates(self, container_ref): + """Retrieves intermediates from barbican certificate. + + :param string container_ref: Reference to container stored in a + certificate manager. + :returns string: Intermediate payload data. + """ + container = self.barbican.containers.get(container_ref) + if (container.intermediates and container.intermediates.payload): + return container.intermediates.payload + else: + return None \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/__init__.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/__init__.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/base_action.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/base_action.py new file mode 100644 index 000000000..e702acdd8 --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/base_action.py @@ -0,0 +1,88 @@ +from oslo_config import cfg +from oslo_utils import importutils +from oslo_log import log as logging +import socket +import collections +import sys + +import errno +import inspect +import sys + +import f5_openstack_agent.lbaasv2.drivers.bigip.exceptions as exceptions + +from oslo_config import cfg +from oslo_log import log as oslo_logging +from oslo_service import service + +from neutron.agent.linux import interface +from neutron.agent.common import config +from neutron.common import config as common_config +from neutron.common import rpc as n_rpc + +import f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver as driver +import f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager as manager +import f5_openstack_agent.lbaasv2.drivers.bigip.agent as agent +import f5_openstack_agent.lbaasv2.drivers.bigip.constants_v2 as f5constants + + + + +LOG = logging.getLogger(__name__) + + +class BaseAction(object): + + + def __init__(self,namespace): + # append appends config paths to defaults... not what we intend + if len(namespace.config) > 2: + self.config_files = namespace.config[2:] + else: + self.config_files = namespace.config + + self.conf = cfg.CONF + + config_files = [] + + for s in self.config_files: + config_files.append("--config-file") + config_files.append(s) + + common_config.init(config_files) + + cfg.CONF.register_opts(manager.OPTS) + cfg.CONF.register_opts(interface.OPTS) + cfg.CONF.register_opts(agent.OPTS) + cfg.CONF.register_opts(driver.OPTS) + config.register_agent_state_opts_helper(cfg.CONF) + config.register_root_helper(cfg.CONF) + + self.host = socket.gethostname() + + if namespace.log: + common_config.setup_logging() + + self.manager = manager.LbaasAgentManager(cfg.CONF, cli_sync=True) + self.manager.lbdriver.make_bigips_operational() + self.driver = self.manager.lbdriver + + + def replace_dict_value(self,obj,key,new_value): + if isinstance(obj,dict): + for k, v in obj.iteritems(): + if k == key or isinstance(v,dict) or isinstance(v,list): + obj[k] = self.replace_dict_value(v,key,new_value) + result = obj + elif isinstance(obj,list): + result = [] + for v in obj: + result.append(self.replace_dict_value(v, key, new_value)) + + else: + result= new_value + + return result + + + diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/delete.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/delete.py new file mode 100644 index 000000000..8b7367488 --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/delete.py @@ -0,0 +1,38 @@ + +import base_action + + +from oslo_log import log as logging +from neutron.plugins.common import constants as plugin_const + +LOG = logging.getLogger(__name__) + + +class Delete(base_action.BaseAction): + + def __init__(self, namespace): + self.lb_id = namespace.lb_id + super(Delete, self).__init__(namespace) + + def execute(self): + if self.lb_id is None : + print("Please specify an LB id with --lb_id") + exit(1) + + print("Starting delete attempt for load balancer {}".format(self.lb_id)) + + service = self.manager.plugin_rpc.get_service_by_loadbalancer_id( + self.lb_id + ) + + if not bool(service): + print("Loadbalancer {} not found".format(self.lb_id)) + exit(1) + + service = self.replace_dict_value(service, 'provisioning_status', plugin_const.PENDING_DELETE) + + + self.driver._common_service_handler(service,cli_sync=True) + + print("All device configuration forloadbalancer {} has been removed".format(self.lb_id)) + diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/druckhammer.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/druckhammer.py new file mode 100644 index 000000000..01c3b888b --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/druckhammer.py @@ -0,0 +1,111 @@ +import base_action + +from sync_all import SyncAll +from f5_openstack_agent.lbaasv2.drivers.bigip import system_helper, resource_helper +from oslo_log import log as logging + +LOG = logging.getLogger(__name__) + +class Druckhammer(base_action.BaseAction): + + def __init__(self, namespace): + self.exempt_folders = ['/', 'Common', 'Drafts'] + self.sure = namespace.sure + self.sync = namespace.sync + self.project_id = None + self.sh = system_helper.SystemHelper() + self.rd_manager = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.route_domain) + self.si_manager = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.selfip) + self.vlan_manger = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.vlan) + super(Druckhammer, self).__init__(namespace) + + def execute(self): + if not self.sure: + print("Please be sure by appending --i-am-sure-what-i-am-doing") + exit(1) + + for bigip in self.driver.get_all_bigips(): + for folder in self.sh.get_folders(bigip): + if folder in self.exempt_folders: + continue + + try: + print("Purging Folder \"%s\"" % folder) + self.sh.purge_folder_contents(bigip, folder) + self.sh.purge_folder(bigip, folder) + except Exception as err: + print("Failed purging folder %s: %s" % (folder, err.message)) + + for route in self.get_routes(bigip): + try: + print("Purging route \"%s\"" % route.name) + route.delete() + except Exception as err: + print("Failed purging route %s: %s" % (route.name, err.message)) + + + for selfip in self.get_selfips(bigip): + try: + print("Purging selfip \"%s\"" % selfip.name) + selfip.delete() + except Exception as err: + print("Failed purging selfip %s: %s" % (selfip.name, err.message)) + + for route_domain in self.get_routedomains(bigip): + try: + print("Purging route_domain \"%s\"" % route_domain.name) + route_domain.delete() + except Exception as err: + print("Failed purging route_domain %s: %s" % (route_domain.name, err.message)) + + + for vlan in self.get_vlans(bigip): + try: + print("Purging vlan \"%s\"" % vlan.name) + vlan.delete() + except Exception as err: + print("Failed purging vlan %s: %s" % (vlan.name, err.message)) + + if self.sync: + # Crude hack, but it works :D + # I wanted to reuse the code of SyncAll but don't want to initalize a new SyncAll class instance, + # which would connect to F5 and reinitalize objects etc... + # Instead, I just cast this druckhammer instance to the SyncAll Class and re-execute meself :) + Syncer = self + self.__class__ = SyncAll + Syncer.execute() + + def get_selfips(self, bigip): + selfips = [] + for selfip in self.si_manager.get_resources(bigip, "Common"): + if selfip.name.startswith("local-" + bigip.device_name): + selfips.append(selfip) + + return selfips + + def get_routedomains(self, bigip): + routedomains = [] + for routedomain in self.rd_manager.get_resources(bigip, "Common"): + if routedomain.name.startswith("rd-"): + routedomains.append(routedomain) + + return routedomains + + def get_routes(self, bigip): + routes = [] + for route in bigip.tm.net.routes.get_collection(): + if route.name.startswith("rt-"): + routes.append(route) + + return routes + + def get_vlans(self, bigip): + vlans = [] + for vlan in self.vlan_manger.get_resources(bigip, "Common"): + if vlan.name.startswith("vlan-"): + vlans.append(vlan) + + return vlans diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/holzhammer.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/holzhammer.py new file mode 100644 index 000000000..bfaf2a9e6 --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/holzhammer.py @@ -0,0 +1,36 @@ +import base_action + +from sync_all import SyncAll +from f5_openstack_agent.lbaasv2.drivers.bigip import system_helper +from oslo_log import log as logging + +LOG = logging.getLogger(__name__) + +class Holzhammer(base_action.BaseAction): + + def __init__(self, namespace): + self.sync = namespace.sync + self.project_id = namespace.project_id + self.sh = system_helper.SystemHelper() + super(Holzhammer, self).__init__(namespace) + + def execute(self): + if self.project_id is None: + print("Please specify an Project id with --project-id") + exit(1) + + for bigip in self.driver.get_all_bigips(): + try: + print("Cleaning Partition %s" % "Project_" + self.project_id) + self.sh.purge_folder_contents(bigip, "Project_" + self.project_id) + except Exception as err: + print(err.message) + + if self.sync: + # Crude hack, but it works :D + # I wanted to reuse the code of SyncAll but don't want to initalize a new SyncAll class instance, + # which would connect to F5 and reinitalize objects etc... + # Instead, I just cast this Holzhammer instance to the SyncAll Class and re-execute meself :) + Syncer = self + self.__class__ = SyncAll + Syncer.execute() diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync.py new file mode 100644 index 000000000..6995cab4c --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync.py @@ -0,0 +1,38 @@ + +import base_action + +from neutron.plugins.common import constants as plugin_const + +from oslo_log import log as logging + +LOG = logging.getLogger(__name__) + + +class Sync(base_action.BaseAction): + + def __init__(self, namespace): + self.lb_id = namespace.lb_id + super(Sync, self).__init__(namespace) + + def execute(self): + if self.lb_id is None: + print("Please specify an LB id with --lb_id") + exit(1) + + print("Starting sync attempt for load balancer {}".format(self.lb_id)) + + service = self.manager.plugin_rpc.get_service_by_loadbalancer_id( + self.lb_id + ) + + if not bool(service): + print("Loadbalancer {} not found".format(self.lb_id)) + exit(1) + + service = self.replace_dict_value(service, 'provisioning_status', plugin_const.PENDING_CREATE) + + self.driver._common_service_handler(service) + + print("The device state of loadbalancer {} has been synced with Neutron".format(self.lb_id)) + + diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync_all.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync_all.py new file mode 100644 index 000000000..db133def1 --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/actions/sync_all.py @@ -0,0 +1,40 @@ + +import base_action + +from neutron.plugins.common import constants as plugin_const + +from oslo_log import log as logging + +LOG = logging.getLogger(__name__) + + +class SyncAll(base_action.BaseAction): + + def __init__(self, namespace): + self.project_id = namespace.project_id + super(SyncAll, self).__init__(namespace) + + def execute(self): + + services = self.manager.plugin_rpc.get_all_loadbalancers(host=self.manager.agent_host) + + if self.project_id is not None: + print("Syncing all LBs in project {}".format(self.project_id)) + else: + print("Syncing all LBs hosted on agent {}".format(self.manager.agent_host)) + + for service in services: + + if self.project_id is None or service['tenant_id'] == self.project_id : + + detailed_service = self.manager.plugin_rpc.get_service_by_loadbalancer_id(service['lb_id']) + + print("Starting sync attempt for load balancer {}".format(service['lb_id'])) + + + detailed_service = self.replace_dict_value(detailed_service, 'provisioning_status', plugin_const.PENDING_CREATE) + self.driver._common_service_handler(detailed_service) + + print("The device state of loadbalancer {} has been synced with Neutron".format(service['lb_id'])) + + diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cli/f5_cli_utils.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/f5_cli_utils.py new file mode 100644 index 000000000..abd9f68b5 --- /dev/null +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cli/f5_cli_utils.py @@ -0,0 +1,82 @@ +import argparse + +import sys +import urllib3 +import requests +from requests.packages.urllib3.exceptions import InsecureRequestWarning +import warnings +warnings.filterwarnings("ignore") + +requests.packages.urllib3.disable_warnings(InsecureRequestWarning) +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +from oslo_utils import importutils + +ACTION_MODULE = 'f5_openstack_agent.lbaasv2.drivers.bigip.cli.actions.' + +class Execute(argparse._SubParsersAction): + def __init__(self, option_strings, **kwargs): + super(Execute, self).__init__(option_strings, **kwargs) + self.actions = { + "sync":"sync.Sync", + "sync-all":"sync_all.SyncAll", + "delete":"delete.Delete", + "holzhammer":"holzhammer.Holzhammer", + "druckhammer":"druckhammer.Druckhammer" + } + + def __call__(self, parser, namespace, values, option_string=None): + super(Execute, self).__call__(parser, namespace, values, option_string) + action = self.actions.get(values[0]) + if action: + + instance = importutils.import_object(ACTION_MODULE+action,namespace) + instance.execute() + +def main(): + + parser = argparse.ArgumentParser(prog='f5_utils', description='Operations utilities for F5 LBAAS driver.') + + parser.add_argument('--config-file', dest='config', action='append', + default=["/etc/neutron/f5-oslbaasv2-agent.ini", "/etc/neutron/neutron.conf"], + help='Configuration files') + + parser.add_argument('--log',dest='log', action='store_true', + help='Enable openstack log output') + + subparsers = parser.add_subparsers(title='command', description='valid subcommands', + help='command to execute', action=Execute, dest='subcommand') + + parser_sync = subparsers.add_parser('sync', help='sync a specific load balancer') + parser_sync.add_argument('--lb-id',dest='lb_id', + help='router id',action='store') + + parser_sync_all = subparsers.add_parser('sync-all', help='sync all load balancer') + parser_sync_all.add_argument('--project-id',dest='project_id', + help='project id',action='store') + + + parser_delete = subparsers.add_parser('delete', help='delete a specific load balancer') + parser_delete.add_argument('--lb-id',dest='lb_id', + help='router id',action='store') + + parser_holzhammer = subparsers.add_parser('holzhammer', help='purge and resync project') + parser_holzhammer.add_argument('--i-am-sure-what-i-am-doing', action='store_true', dest='sure', + help='declaration of liability') + parser_holzhammer.add_argument('--no-sync', action='store_false', dest='sync', + help='disable resync of project') + parser_holzhammer.add_argument('--project-id',dest='project_id', + help='project id',action='store') + + parser_druckhammer = subparsers.add_parser('druckhammer', help='purge all bigip ' + 'partitions/vlans/selfip/routes/routedomains') + parser_druckhammer.add_argument('--i-am-sure-what-i-am-doing', action='store_true', dest='sure', + help='declaration of liability') + parser_druckhammer.add_argument('--sync', action='store_true', dest='sync', + help='resync all LB from agent/neutron') + + parser.parse_args() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/cluster_manager.py b/f5_openstack_agent/lbaasv2/drivers/bigip/cluster_manager.py index e102e1dfe..f0969efdb 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/cluster_manager.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/cluster_manager.py @@ -61,8 +61,13 @@ def get_traffic_groups(self, bigip): def save_config(self, bigip): try: - c = bigip.tm.sys.config - c.save() + # invalid for the version of f5-sdk in requirements + # c = bigip.tm.sys.config + # c.save() + bigip.tm.util.bash.exec_cmd( + command='run', + utilCmdArgs="-c 'tmsh save sys config'" + ) except HTTPError as err: LOG.error("Error saving config." "Repsponse status code: %s. Response " diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/constants_v2.py b/f5_openstack_agent/lbaasv2/drivers/bigip/constants_v2.py index b6c002046..bd8d797b8 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/constants_v2.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/constants_v2.py @@ -16,6 +16,8 @@ # Service resync interval RESYNC_INTERVAL = 300 +UPDATE_OPERATING_STATUS_INTERVAL = 30 + # Topic for tunnel notifications between the plugin and agent TUNNEL = 'tunnel' @@ -55,3 +57,5 @@ DEVICE_HEALTH_SCORE_CPS_WEIGHT = 1 DEVICE_HEALTH_SCORE_CPS_PERIOD = 5 DEVICE_HEALTH_SCORE_CPS_MAX = 100 + +DEVICE_CONNECTION_TIMEOUT = 5 diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/esd_filehandler.py b/f5_openstack_agent/lbaasv2/drivers/bigip/esd_filehandler.py index bfdb81cb8..1a7988b46 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/esd_filehandler.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/esd_filehandler.py @@ -157,7 +157,7 @@ def verify_esd(self, bigip, name, esd): valid_esd[tag] = esd[tag] LOG.debug("Tag {0} is valid for ESD {1}.".format(tag, name)) except f5_ex.esdJSONFileInvalidException as err: - LOG.error('Tag {0} failed validation for ESD {1} and was not ' + LOG.info('Tag {0} failed validation for ESD {1} and was not ' 'added to ESD. Error: {2}'. format(tag, name, err.message)) @@ -184,6 +184,9 @@ def verify_value(self, bigip, tag, value): # verify value exists on BIG-IP if isinstance(value, list): is_valid = self.is_valid_value_list(bigip, value, resource_type) + elif value=='': + # ESD Processing we will handle this as a special case and use this to toggle things like fastl4 + is_valid = True else: is_valid = self.is_valid_value(bigip, value, resource_type) @@ -202,6 +205,9 @@ def verify_tag(self, tag): # we are implementing the tags that can be applied only to listeners valid_esd_tags = { + 'lbaas_fastl4': { + 'resource_type': ResourceType.fastl4_profile, + 'value_type': types.StringTypes}, 'lbaas_ctcp': { 'resource_type': ResourceType.tcp_profile, 'value_type': types.StringTypes}, @@ -210,6 +216,18 @@ def verify_tag(self, tag): 'resource_type': ResourceType.tcp_profile, 'value_type': types.StringTypes}, + 'lbaas_http': { + 'resource_type': ResourceType.http_profile, + 'value_type': types.StringTypes}, + + 'lbaas_one_connect': { + 'resource_type': ResourceType.one_connect_profile, + 'value_type': types.StringTypes}, + + 'lbaas_http_compression': { + 'resource_type': ResourceType.http_compression_profile, + 'value_type': types.StringTypes}, + 'lbaas_cssl_profile': { 'resource_type': ResourceType.client_ssl_profile, 'value_type': types.StringTypes}, diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py b/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py index 4ab1a9ead..cea23d2e2 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py @@ -20,14 +20,14 @@ import json import logging as std_logging import os -import urllib2 +import urllib +from requests import HTTPError from eventlet import greenthread from time import strftime from time import time from neutron.common.exceptions import InvalidConfigurationOption -from neutron.common.exceptions import NeutronException from neutron.plugins.common import constants as plugin_const from neutron_lbaas.services.loadbalancer import constants as lb_const @@ -63,13 +63,14 @@ from f5_openstack_agent.lbaasv2.drivers.bigip.virtual_address import \ VirtualAddress +import re LOG = logging.getLogger(__name__) NS_PREFIX = 'qlbaas-' __VERSION__ = '0.1.1' -# configuration objects specific to iControl® driver +# configuration objects specific to iControl driver # XXX see /etc/neutron/services/f5/f5-openstack-agent.ini OPTS = [ # XXX maybe we should make this a dictionary cfg.StrOpt( @@ -280,6 +281,36 @@ help='Parent profile used when creating client SSL profiles ' 'for listeners with TERMINATED_HTTPS protocols.' ), + cfg.StrOpt( + 'f5_default_http_profile', + default='/Common/http', + help='Default profiles to use for HTTP Protocol VS' + ), + + cfg.StrOpt( + 'f5_default_https_profile', + default='/Common/http', + help='Default profiles to use for HTTPS Protocol VS' + ), + + cfg.StrOpt( + 'f5_default_terminated_https_profile', + default='/Common/http', + help='Default profiles to use for TERMINATED_HTTPS Protocol VS' + ), + + cfg.StrOpt( + 'f5_default_oneconnect_profile', + default='/Common/oneconnect', + help='Default oneconnect profile for HTTP virtual servers' + ), + # ccloud: new parent monitor profile for https monitors for tmos > v13.1.1.2 + # default parent is standard https monitor. New onle will be /Common/cc_https + cfg.StrOpt( + 'f5_parent_https_monitor', + default='/Common/https', + help='Parent monitor for https monitors.' + ), cfg.StrOpt( 'os_tenant_name', default=None, @@ -293,21 +324,21 @@ ] -def is_connected(method): - # Decorator to check we are connected before provisioning. +def is_operational(method): + # Decorator to check we are operational before provisioning. def wrapper(*args, **kwargs): instance = args[0] - if instance.connected: + if instance.operational: try: return method(*args, **kwargs) except IOError as ioe: LOG.error('IO Error detected: %s' % method.__name__) - instance.connect_bigips() # what's this do? + LOG.error(str(ioe)) raise ioe else: - LOG.error('Cannot execute %s. Not connected. Connecting.' + LOG.error('Cannot execute %s. Not operational. Re-initializing ...' % method.__name__) - instance.connect_bigips() + instance._init_bigips() return wrapper @@ -326,14 +357,30 @@ def __init__(self, conf, registerOpts=True): self.hostnames = None self.device_type = conf.f5_device_type self.plugin_rpc = None # overrides base, same value - self.__last_connect_attempt = None - self.connected = False # overrides base, same value + self.agent_report_state = None # overrides base, same value + self.operational = False # overrides base, same value self.driver_name = 'f5-lbaasv2-icontrol' - # BIG-IP® containers + # + # BIG-IP containers + # + + # BIG-IPs which currectly active self.__bigips = {} + self.__last_connect_attempt = None + + # HA and traffic group validation + self.ha_validated = False + self.tg_initialized = False + # traffic groups discovered from BIG-IPs for service placement self.__traffic_groups = [] + + # base configurations to report to Neutron agent state reports self.agent_configurations = {} # overrides base, same value + self.agent_configurations['device_drivers'] = [self.driver_name] + self.agent_configurations['icontrol_endpoints'] = {} + + # service component managers self.tenant_manager = None self.cluster_manager = None self.system_helper = None @@ -342,103 +389,81 @@ def __init__(self, conf, registerOpts=True): self.vlan_binding = None self.l3_binding = None self.cert_manager = None # overrides register_OPTS + + # server helpers self.stat_helper = stat_helper.StatHelper() self.network_helper = network_helper.NetworkHelper() + # f5-sdk helpers self.vs_manager = resource_helper.BigIPResourceHelper( resource_helper.ResourceType.virtual) self.pool_manager = resource_helper.BigIPResourceHelper( resource_helper.ResourceType.pool) + self.va_manager = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual_address) - if self.conf.trace_service_requests: - path = '/var/log/neutron/service/' - if not os.path.exists(path): - os.makedirs(path) - self.file_name = path + strftime("%H%M%S-%m%d%Y") + '.json' - with open(self.file_name, 'w') as fp: - fp.write('[{}] ') - - if self.conf.f5_global_routed_mode: - LOG.info('WARNING - f5_global_routed_mode enabled.' - ' There will be no L2 or L3 orchestration' - ' or tenant isolation provisioned. All vips' - ' and pool members must be routable through' - ' pre-provisioned SelfIPs.') - self.conf.use_namespaces = False - self.conf.f5_snat_mode = True - self.conf.f5_snat_addresses_per_subnet = 0 - self.agent_configurations['tunnel_types'] = [] - self.agent_configurations['bridge_mappings'] = {} - else: - self.agent_configurations['tunnel_types'] = \ - self.conf.advertised_tunnel_types - for net_id in self.conf.common_network_ids: - LOG.debug('network %s will be mapped to /Common/%s' - % (net_id, self.conf.common_network_ids[net_id])) - - self.agent_configurations['common_networks'] = \ - self.conf.common_network_ids - LOG.debug('Setting static ARP population to %s' - % self.conf.f5_populate_static_arp) - self.agent_configurations['f5_common_external_networks'] = \ - self.conf.f5_common_external_networks - f5const.FDB_POPULATE_STATIC_ARP = self.conf.f5_populate_static_arp - - self.agent_configurations['device_drivers'] = [self.driver_name] - self._init_bigip_hostnames() - self._init_bigip_managers() - self.connect_bigips() - - # After we have a connection to the BIG-IPs, initialize vCMP - if self.network_builder: - self.network_builder.initialize_vcmp() - - self.agent_configurations['network_segment_physical_network'] = \ - self.conf.f5_network_segment_physical_network - - LOG.info('iControlDriver initialized to %d bigips with username:%s' - % (len(self.__bigips), self.conf.icontrol_username)) - LOG.info('iControlDriver dynamic agent configurations:%s' - % self.agent_configurations) + self.orphan_cache = {} + self.orphan_cache_last_reset = datetime.datetime.now() + self.orphan_cleanup_testrun = self.conf.ccloud_orphans_cleanup_testrun - # read enhanced services definitions - esd_dir = os.path.join(self.get_config_dir(), 'esd') - esd = EsdTagProcessor(esd_dir) try: - esd.process_esd(self.get_all_bigips()) - self.lbaas_builder.init_esd(esd) - except f5ex.esdJSONFileInvalidException as err: - LOG.error("Unable to initialize ESD. Error: %s.", err.message) - - self.initialized = True - - def connect_bigips(self): - self._init_bigips() - if self.conf.f5_global_routed_mode: - local_ips = [] - else: - try: - local_ips = self.network_builder.initialize_tunneling() - except Exception: - LOG.error("Error creating BigIP VTEPs in connect_bigips") - raise - - self._init_agent_config(local_ips) - - def post_init(self): - # run any post initialized tasks, now that the agent - # is fully connected - if self.vlan_binding: - LOG.debug( - 'Getting BIG-IP device interface for VLAN Binding') - self.vlan_binding.register_bigip_interfaces() - if self.l3_binding: - LOG.debug('Getting BIG-IP MAC Address for L3 Binding') - self.l3_binding.register_bigip_mac_addresses() + # debug logging of service requests recieved by driver + if self.conf.trace_service_requests: + path = '/var/log/neutron/service/' + if not os.path.exists(path): + os.makedirs(path) + self.file_name = path + strftime("%H%M%S-%m%d%Y") + '.json' + with open(self.file_name, 'w') as fp: + fp.write('[{}] ') + + # driver mode settings - GRM vs L2 adjacent + if self.conf.f5_global_routed_mode: + LOG.info('WARNING - f5_global_routed_mode enabled.' + ' There will be no L2 or L3 orchestration' + ' or tenant isolation provisioned. All vips' + ' and pool members must be routable through' + ' pre-provisioned SelfIPs.') + self.conf.use_namespaces = False + self.conf.f5_snat_mode = True + self.conf.f5_snat_addresses_per_subnet = 0 + self.agent_configurations['tunnel_types'] = [] + self.agent_configurations['bridge_mappings'] = {} + else: + self.agent_configurations['tunnel_types'] = \ + self.conf.advertised_tunnel_types + for net_id in self.conf.common_network_ids: + LOG.debug('network %s will be mapped to /Common/%s' + % (net_id, self.conf.common_network_ids[net_id])) + + self.agent_configurations['common_networks'] = \ + self.conf.common_network_ids + LOG.debug('Setting static ARP population to %s' + % self.conf.f5_populate_static_arp) + self.agent_configurations['f5_common_external_networks'] = \ + self.conf.f5_common_external_networks + f5const.FDB_POPULATE_STATIC_ARP = \ + self.conf.f5_populate_static_arp + + # parse the icontrol_hostname setting + self._init_bigip_hostnames() + # instantiate the managers + self._init_bigip_managers() + + self.initialized = True + LOG.debug('iControlDriver loaded successfully') + except Exception as exc: + LOG.error("exception in intializing driver %s" % str(exc)) + self._set_agent_status(False) - if self.network_builder: - self.network_builder.post_init() + def connect(self): + # initialize communications wiht BIG-IP via iControl + try: + self._init_bigips() + except Exception as exc: + LOG.error("exception in intializing communications to BIG-IPs %s" + % str(exc)) + self._set_agent_status(False) def _init_bigip_managers(self): @@ -512,19 +537,36 @@ def _init_bigip_hostnames(self): self.hostnames = [item.strip() for item in self.hostnames] self.hostnames = sorted(self.hostnames) + # initialize per host agent_configurations + for hostname in self.hostnames: + self.__bigips[hostname] = bigip = type('', (), {})() + bigip.hostname = hostname + bigip.status = 'creating' + bigip.status_message = 'creating BIG-IP from iControl hostnames' + bigip.device_interfaces = dict() + self.agent_configurations[ + 'icontrol_endpoints'][hostname] = {} + self.agent_configurations[ + 'icontrol_endpoints'][hostname]['failover_state'] = \ + 'undiscovered' + self.agent_configurations[ + 'icontrol_endpoints'][hostname]['status'] = 'unknown' + self.agent_configurations[ + 'icontrol_endpoints'][hostname]['status_message'] = '' + def _init_bigips(self): - # Connect to all BIG-IP®s - if self.connected: + # Connect to all BIG-IPs + if self.operational: + LOG.debug('iControl driver reports connection is operational') return + LOG.debug('initializing communications to BIG-IPs') try: + # setup logging options if not self.conf.debug: - sudslog = std_logging.getLogger('suds.client') - sudslog.setLevel(std_logging.FATAL) requests_log = std_logging.getLogger( "requests.packages.urllib3") requests_log.setLevel(std_logging.ERROR) requests_log.propagate = False - else: requests_log = std_logging.getLogger( "requests.packages.urllib3") @@ -533,172 +575,430 @@ def _init_bigips(self): self.__last_connect_attempt = datetime.datetime.now() - first_bigip = self._open_bigip(self.hostnames[0]) - self._init_bigip(first_bigip, self.hostnames[0], None) - self.__bigips[self.hostnames[0]] = first_bigip - - device_group_name = self._validate_ha(first_bigip) - self._init_traffic_groups(first_bigip) - - # connect to the rest of the devices - for hostname in self.hostnames[1:]: - bigip = self._open_bigip(hostname) - self._init_bigip(bigip, hostname, device_group_name) - self.__bigips[hostname] = bigip + min_one_initialized = False + for hostname in self.hostnames: + if self._init_bigip_from_hostname(hostname): + min_one_initialized = True - self.connected = True + if min_one_initialized: + self._set_agent_status(force_resync=True) + else: + self._set_agent_status(force_resync=False) - except NeutronException as exc: - LOG.error('Could not communicate with all ' + - 'iControl devices: %s' % exc.msg) - greenthread.sleep(5) # this should probably go away + except Exception as exc: + LOG.error('Invalid agent configuration: %s' % exc.message) + self._set_agent_status(force_resync=False) raise + + def _init_errored_bigips(self): + min_one_initialized = False + try: + errored_bigips = self.get_errored_bigips_hostnames() + if errored_bigips: + LOG.debug('attempting to recover %s BIG-IPs' % + len(errored_bigips)) + for hostname in errored_bigips: + # try to connect and set status + if self._init_bigip_from_hostname(hostname): + min_one_initialized = True + if min_one_initialized: + self._set_agent_status(force_resync=True) + else: + self._set_agent_status(force_resync=False) + else: + LOG.debug('there are no disconnected BIG-IPs to recover') except Exception as exc: - LOG.error('Could not communicate with all ' + - 'iControl devices: %s' % exc.message) - greenthread.sleep(5) # this should probably go away + LOG.error('Invalid agent configuration: %s' % exc.message) raise + return min_one_initialized + + def _init_bigip_from_hostname(self, hostname): + initialized = True + LOG.debug('ccloud: _init_bigip_from_hostname: %s' % hostname) + try: + # connect to each BIG-IP and set it status + bigip = self._open_bigip(hostname) + if bigip.status == 'connected': + # set the status down until we assure initialized + bigip.status = 'initializing' + bigip.status_message = 'initializing HA viability' + LOG.debug('initializing HA viability %s' % hostname) + device_group_name = None + if not self.ha_validated: + device_group_name = self._validate_ha(bigip) + LOG.debug('HA validated from %s with DSG %s' % + (hostname, device_group_name)) + self.ha_validated = True + if not self.tg_initialized: + self._init_traffic_groups(bigip) + LOG.debug('learned traffic groups from %s as %s' % + (hostname, self.__traffic_groups)) + self.tg_initialized = True + LOG.debug('initializing bigip %s' % hostname) + self._init_bigip(bigip, hostname, device_group_name) + LOG.debug('initializing agent configurations %s' + % hostname) + self._init_agent_config(bigip) + # Assure basic BIG-IP HA is operational + LOG.debug('validating HA state for %s' % hostname) + bigip.status = 'validating_HA' + bigip.status_message = 'validating the current HA state' + if self._validate_ha_operational(bigip): + LOG.debug('setting status to active for %s' % hostname) + bigip.status = 'active' + bigip.status_message = 'BIG-IP ready for provisioning' + self._post_init() + else: + LOG.debug('setting status to error for %s' % hostname) + bigip.status = 'error' + bigip.status_message = 'BIG-IP is not operational' + initialized = False + else: + LOG.error('error opening BIG-IP %s - %s:%s' + % (hostname, bigip.status, bigip.status_message)) + initialized = False + except Exception as e: + bigip.status = 'error' + bigip.status_message = 'BIG-IP is not operational' + LOG.error('ccloud: Invalid agent configuration: BIG-IP %s - %s:%s - %s' + % (hostname, bigip.status, bigip.status_message, e.message)) + initialized = False + finally: + return initialized def _open_bigip(self, hostname): # Open bigip connection """ - LOG.info('Opening iControl connection to %s @ %s' % - (self.conf.icontrol_username, hostname)) - - return ManagementRoot(hostname, - self.conf.icontrol_username, - self.conf.icontrol_password) + try: + bigip = self.__bigips[hostname] + if bigip.status not in ['creating', 'error']: + LOG.debug('BIG-IP %s status invalid %s to open a connection' + % (hostname, bigip.status)) + return bigip + bigip.status = 'connecting' + bigip.status_message = 'requesting iControl endpoint' + LOG.info('opening iControl connection to %s @ %s' % + (self.conf.icontrol_username, hostname)) + bigip = ManagementRoot(hostname, + self.conf.icontrol_username, + self.conf.icontrol_password, + timeout=f5const.DEVICE_CONNECTION_TIMEOUT) + bigip.status = 'connected' + bigip.status_message = 'connected to BIG-IP' + self.__bigips[hostname] = bigip + return bigip + except Exception as exc: + LOG.error('could not communicate with ' + + 'iControl device: %s' % hostname) + # since no bigip object was created, create a dummy object + # so we can store the status and status_message attributes + errbigip = type('', (), {})() + errbigip.hostname = hostname + errbigip.status = 'error' + errbigip.status_message = str(exc)[:80] + self.__bigips[hostname] = errbigip + return errbigip def _init_bigip(self, bigip, hostname, check_group_name=None): # Prepare a bigip for usage - - major_version, minor_version = self._validate_bigip_version( - bigip, hostname) - - device_group_name = None - extramb = self.system_helper.get_provision_extramb(bigip) - if int(extramb) < f5const.MIN_EXTRA_MB: - raise f5ex.ProvisioningExtraMBValidateFailed( - 'Device %s BIG-IP not provisioned for ' - 'management LARGE.' % hostname) - - if self.conf.f5_ha_type == 'pair' and \ - self.cluster_manager.get_sync_status(bigip) == 'Standalone': - raise f5ex.BigIPClusterInvalidHA( - 'HA mode is pair and bigip %s in standalone mode' - % hostname) - - if self.conf.f5_ha_type == 'scalen' and \ - self.cluster_manager.get_sync_status(bigip) == 'Standalone': - raise f5ex.BigIPClusterInvalidHA( - 'HA mode is scalen and bigip %s in standalone mode' - % hostname) - - if self.conf.f5_ha_type != 'standalone': - device_group_name = self.cluster_manager.get_device_group(bigip) - if not device_group_name: + try: + major_version, minor_version = self._validate_bigip_version( + bigip, hostname) + + device_group_name = None + extramb = self.system_helper.get_provision_extramb(bigip) + if int(extramb) < f5const.MIN_EXTRA_MB: + raise f5ex.ProvisioningExtraMBValidateFailed( + 'Device %s BIG-IP not provisioned for ' + 'management LARGE.' % hostname) + + if self.conf.f5_ha_type == 'pair' and \ + self.cluster_manager.get_sync_status(bigip) == \ + 'Standalone': raise f5ex.BigIPClusterInvalidHA( - 'HA mode is %s and no sync failover ' - 'device group found for device %s.' - % (self.conf.f5_ha_type, hostname)) - if check_group_name and device_group_name != check_group_name: + 'HA mode is pair and bigip %s in standalone mode' + % hostname) + + if self.conf.f5_ha_type == 'scalen' and \ + self.cluster_manager.get_sync_status(bigip) == \ + 'Standalone': raise f5ex.BigIPClusterInvalidHA( - 'Invalid HA. Device %s is in device group' - ' %s but should be in %s.' - % (hostname, device_group_name, check_group_name)) - bigip.device_group_name = device_group_name + 'HA mode is scalen and bigip %s in standalone mode' + % hostname) + + if self.conf.f5_ha_type != 'standalone': + device_group_name = \ + self.cluster_manager.get_device_group(bigip) + if not device_group_name: + raise f5ex.BigIPClusterInvalidHA( + 'HA mode is %s and no sync failover ' + 'device group found for device %s.' + % (self.conf.f5_ha_type, hostname)) + if check_group_name and device_group_name != check_group_name: + raise f5ex.BigIPClusterInvalidHA( + 'Invalid HA. Device %s is in device group' + ' %s but should be in %s.' + % (hostname, device_group_name, check_group_name)) + bigip.device_group_name = device_group_name - if self.network_builder: - for network in self.conf.common_network_ids.values(): - if not self.network_builder.vlan_exists(bigip, - network, - folder='Common'): - raise f5ex.MissingNetwork( - 'Common network %s on %s does not exist' - % (network, bigip.hostname)) - - bigip.device_name = self.cluster_manager.get_device_name(bigip) - bigip.mac_addresses = self.system_helper.get_mac_addresses(bigip) - LOG.debug("Initialized BIG-IP %s with MAC addresses %s" % - (bigip.device_name, ', '.join(bigip.mac_addresses))) - bigip.device_interfaces = \ - self.system_helper.get_interface_macaddresses_dict(bigip) - bigip.assured_networks = {} - bigip.assured_tenant_snat_subnets = {} - bigip.assured_gateway_subnets = [] - - if self.conf.f5_ha_type != 'standalone': - self.cluster_manager.disable_auto_sync(device_group_name, bigip) - - # Turn off tunnel syncing... our VTEPs are local SelfIPs - if self.system_helper.get_tunnel_sync(bigip) == 'enable': - self.system_helper.set_tunnel_sync(bigip, enabled=False) - - LOG.debug('Connected to iControl %s @ %s ver %s.%s' - % (self.conf.icontrol_username, hostname, - major_version, minor_version)) + if self.network_builder: + for network in self.conf.common_network_ids.values(): + if not self.network_builder.vlan_exists(bigip, + network, + folder='Common'): + raise f5ex.MissingNetwork( + 'Common network %s on %s does not exist' + % (network, bigip.hostname)) + bigip.device_name = self.cluster_manager.get_device_name(bigip) + bigip.mac_addresses = self.system_helper.get_mac_addresses(bigip) + LOG.debug("Initialized BIG-IP %s with MAC addresses %s" % + (bigip.device_name, ', '.join(bigip.mac_addresses))) + bigip.device_interfaces = \ + self.system_helper.get_interface_macaddresses_dict(bigip) + bigip.assured_networks = {} + bigip.assured_tenant_snat_subnets = {} + bigip.assured_gateway_subnets = [] + + if self.conf.f5_ha_type != 'standalone': + self.cluster_manager.disable_auto_sync( + device_group_name, bigip) + + # validate VTEP SelfIPs + if not self.conf.f5_global_routed_mode: + self.network_builder.initialize_tunneling(bigip) + + # Turn off tunnel syncing between BIG-IP + # as our VTEPs properly use only local SelfIPs + if self.system_helper.get_tunnel_sync(bigip) == 'enable': + self.system_helper.set_tunnel_sync(bigip, enabled=False) + + LOG.debug('connected to iControl %s @ %s ver %s.%s' + % (self.conf.icontrol_username, hostname, + major_version, minor_version)) + except Exception as exc: + bigip.status = 'error' + bigip.status_message = str(exc)[:80] + raise return bigip - def _validate_ha(self, first_bigip): + def _post_init(self): + # After we have a connection to the BIG-IPs, initialize vCMP + # on all connected BIG-IPs + if self.network_builder: + self.network_builder.initialize_vcmp() + + self.agent_configurations['network_segment_physical_network'] = \ + self.conf.f5_network_segment_physical_network + + LOG.info('iControlDriver initialized to %d bigips with username:%s' + % (len(self.get_active_bigips()), + self.conf.icontrol_username)) + LOG.info('iControlDriver dynamic agent configurations:%s' + % self.agent_configurations) + + if self.vlan_binding: + LOG.debug( + 'getting BIG-IP device interface for VLAN Binding') + self.vlan_binding.register_bigip_interfaces() + + if self.l3_binding: + LOG.debug('getting BIG-IP MAC Address for L3 Binding') + self.l3_binding.register_bigip_mac_addresses() + + # endpoints = self.agent_configurations['icontrol_endpoints'] + # for ic_host in endpoints.keys(): + for hostbigip in self.get_all_bigips(): + + # hostbigip = self.__bigips[ic_host] + mac_addrs = [mac_addr for interface, mac_addr in + hostbigip.device_interfaces.items() + if interface != "mgmt"] + ports = self.plugin_rpc.get_ports_for_mac_addresses( + mac_addresses=mac_addrs) + if ports: + self.agent_configurations['nova_managed'] = True + else: + self.agent_configurations['nova_managed'] = False + + if self.network_builder: + self.network_builder.post_init() + + # read enhanced services definitions + esd_dir = os.path.join(self.get_config_dir(), 'esd') + esd = EsdTagProcessor(esd_dir) + try: + esd.process_esd(self.get_all_bigips()) + self.lbaas_builder.init_esd(esd) + #ccloud: self.service_adapter.init_esd(esd) + except f5ex.esdJSONFileInvalidException as err: + LOG.error("unable to initialize ESD. Error: %s.", err.message) + self._set_agent_status(False) + + def _validate_ha(self, bigip): # if there was only one address supplied and # this is not a standalone device, get the # devices trusted by this device. """ device_group_name = None if self.conf.f5_ha_type == 'standalone': if len(self.hostnames) != 1: + bigip.status = 'error' + bigip.status_message = \ + 'HA mode is standalone and %d hosts found.'\ + % len(self.hostnames) raise f5ex.BigIPClusterInvalidHA( 'HA mode is standalone and %d hosts found.' % len(self.hostnames)) + device_group_name = 'standalone' elif self.conf.f5_ha_type == 'pair': device_group_name = self.cluster_manager.\ - get_device_group(first_bigip) + get_device_group(bigip) if len(self.hostnames) != 2: mgmt_addrs = [] - devices = self.cluster_manager.devices(first_bigip, - device_group_name) + devices = self.cluster_manager.devices(bigip) for device in devices: mgmt_addrs.append( - self.cluster_manager.get_mgmt_addr_by_device(device)) + self.cluster_manager.get_mgmt_addr_by_device( + bigip, device)) self.hostnames = mgmt_addrs if len(self.hostnames) != 2: + bigip.status = 'error' + bigip.status_message = 'HA mode is pair and %d hosts found.' \ + % len(self.hostnames) raise f5ex.BigIPClusterInvalidHA( 'HA mode is pair and %d hosts found.' % len(self.hostnames)) elif self.conf.f5_ha_type == 'scalen': device_group_name = self.cluster_manager.\ - get_device_group(first_bigip) + get_device_group(bigip) if len(self.hostnames) < 2: mgmt_addrs = [] - devices = self.cluster_manager.devices(first_bigip, - device_group_name) + devices = self.cluster_manager.devices(bigip) for device in devices: mgmt_addrs.append( self.cluster_manager.get_mgmt_addr_by_device( - first_bigip, device)) + bigip, device) + ) self.hostnames = mgmt_addrs + if len(self.hostnames) < 2: + bigip.status = 'error' + bigip.status_message = 'HA mode is scale and 1 hosts found.' + raise f5ex.BigIPClusterInvalidHA( + 'HA mode is pair and 1 hosts found.') return device_group_name - def _init_agent_config(self, local_ips): - # Init agent config - icontrol_endpoints = {} - for host in self.__bigips: - hostbigip = self.__bigips[host] - ic_host = {} - ic_host['version'] = self.system_helper.get_version(hostbigip) - ic_host['device_name'] = hostbigip.device_name - ic_host['platform'] = self.system_helper.get_platform(hostbigip) - ic_host['serial_number'] = self.system_helper.get_serial_number( - hostbigip) - icontrol_endpoints[host] = ic_host - - self.agent_configurations['tunneling_ips'] = local_ips - self.agent_configurations['icontrol_endpoints'] = icontrol_endpoints + def _validate_ha_operational(self, bigip): + if self.conf.f5_ha_type == 'standalone': + return True + else: + # how many active BIG-IPs are there? + active_bigips = self.get_active_bigips() + if active_bigips: + sync_status = self.cluster_manager.get_sync_status(bigip) + if sync_status in ['Disconnected', 'Sync Failure']: + if len(active_bigips) > 1: + # the device should not be in the disconnected state + return False + if len(active_bigips) > 1: + # it should be in the same sync-failover group + # as the rest of the active bigips + device_group_name = \ + self.cluster_manager.get_device_group(bigip) + for active_bigip in active_bigips: + adgn = self.cluster_manager.get_device_group( + active_bigip) + if not adgn == device_group_name: + return False + return True + else: + return True + def _init_agent_config(self, bigip): + # Init agent config + ic_host = {} + ic_host['version'] = self.system_helper.get_version(bigip) + ic_host['device_name'] = bigip.device_name + ic_host['platform'] = self.system_helper.get_platform(bigip) + ic_host['serial_number'] = self.system_helper.get_serial_number(bigip) + ic_host['status'] = bigip.status + ic_host['status_message'] = bigip.status_message + ic_host['failover_state'] = self.get_failover_state(bigip) + if hasattr(bigip, 'local_ip') and bigip.local_ip: + ic_host['local_ip'] = bigip.local_ip + else: + ic_host['local_ip'] = 'VTEP disabled' + self.agent_configurations['tunnel_types'] = list() + self.agent_configurations['icontrol_endpoints'][bigip.hostname] = \ + ic_host if self.network_builder: self.agent_configurations['bridge_mappings'] = \ self.network_builder.interface_mapping + def _set_agent_status(self, force_resync=False): + for hostname in self.__bigips: + bigip = self.__bigips[hostname] + self.agent_configurations[ + 'icontrol_endpoints'][bigip.hostname][ + 'status'] = bigip.status + self.agent_configurations[ + 'icontrol_endpoints'][bigip.hostname][ + 'status_message'] = bigip.status_message + # Policy - if any BIG-IP are active we're operational + if self.get_active_bigips(): + self.operational = True + else: + self.operational = False + if self.agent_report_state: + self.agent_report_state(force_resync=force_resync) + + def get_failover_state(self, bigip): + try: + if hasattr(bigip, 'tm'): + fs = bigip.tm.sys.dbs.db.load(name='failover.state') + bigip.failover_state = fs.value + return bigip.failover_state + else: + return 'error' + except Exception as exc: + LOG.exception('Error getting %s failover state' % bigip.hostname) + bigip.status = 'error' + bigip.status_message = str(exc)[:80] + self._set_agent_status(False) + return 'error' + + def get_agent_configurations(self): + for hostname in self.__bigips: + bigip = self.__bigips[hostname] + if bigip.status == 'active': + failover_state = self.get_failover_state(bigip) + self.agent_configurations[ + 'icontrol_endpoints'][bigip.hostname][ + 'failover_state'] = failover_state + else: + self.agent_configurations[ + 'icontrol_endpoints'][bigip.hostname][ + 'failover_state'] = 'unknown' + self.agent_configurations['icontrol_endpoints'][ + bigip.hostname]['status'] = bigip.status + self.agent_configurations['icontrol_endpoints'][ + bigip.hostname]['status_message'] = bigip.status_message + self.agent_configurations['operational'] = \ + self.operational + LOG.debug('agent configurations are: %s' % self.agent_configurations) + return dict(self.agent_configurations) + + def recover_errored_devices(self): + # trigger a retry on errored BIG-IPs + try: + return self._init_errored_bigips() + except Exception as exc: + LOG.error("Could not recover BIG-IPs: %s" % exc.message) + + def backend_integrity(self): + if self.operational: + return True + return False + def generate_capacity_score(self, capacity_policy=None): - """Generate the capacity score of connected devices """ + """Generate the capacity score of connected devices.""" if capacity_policy: highest_metric = 0.0 highest_metric_name = None @@ -711,14 +1011,17 @@ def generate_capacity_score(self, capacity_policy=None): metric_func = getattr(self, func_name) metric_value = 0 for bigip in bigips: - global_stats = \ - self.stat_helper.get_global_statistics(bigip) - value = int( - metric_func(bigip=bigip, - global_statistics=global_stats) - ) - LOG.debug('calling capacity %s on %s returned: %s' - % (func_name, bigip.hostname, value)) + if bigip.status == 'active': + global_stats = \ + self.stat_helper.get_global_statistics(bigip) + value = int( + metric_func(bigip=bigip, + global_statistics=global_stats) + ) + LOG.debug('calling capacity %s on %s returned: %s' + % (func_name, bigip.hostname, value)) + else: + value = 0 if value > metric_value: metric_value = value metric_capacity = float(metric_value) / float(max_capacity) @@ -753,6 +1056,10 @@ def set_l2pop_rpc(self, l2pop_rpc): if self.network_builder: self.network_builder.set_l2pop_rpc(l2pop_rpc) + def set_agent_report_state(self, report_state_callback): + """Set Agent Report State.""" + self.agent_report_state = report_state_callback + def service_exists(self, service): return self._service_exists(service) @@ -763,21 +1070,562 @@ def flush_cache(self): bigip.assured_tenant_snat_subnets = {} bigip.assured_gateway_subnets = [] + # method is only needed for f5-utils cli calls like druckhammer, ... + @is_operational + def make_bigips_operational(self): + return + + + @serialized('get_all_deployed_loadbalancers') + @is_operational + def get_all_deployed_loadbalancers(self, purge_orphaned_folders=False): + LOG.debug('getting all deployed loadbalancers on BIG-IPs') + deployed_lb_dict = {} + for bigip in self.get_all_bigips(): + folders = self.system_helper.get_folders(bigip) + for folder in folders: + tenant_id = folder[len(self.service_adapter.prefix):] + if str(folder).startswith(self.service_adapter.prefix): + resource = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual_address) + deployed_lbs = resource.get_resources(bigip, folder) + if deployed_lbs: + for lb in deployed_lbs: + if lb.name.startswith(self.service_adapter.prefix): + lb_id = lb.name[len(self.service_adapter.prefix):] + if lb_id in deployed_lb_dict: + deployed_lb_dict[lb_id][ + 'hostnames'].append(bigip.hostname) + else: + deployed_lb_dict[lb_id] = { + 'id': lb_id, + 'tenant_id': tenant_id, + 'hostnames': [bigip.hostname] + } + else: + # delay to assure we are not in the tenant creation + # process before a virtual address is created. + greenthread.sleep(10) + deployed_lbs = resource.get_resources(bigip, folder) + if deployed_lbs: + for lb in deployed_lbs: + if lb.name.startswith(self.service_adapter.prefix): + lb_id = lb.name[len(self.service_adapter.prefix):] + deployed_lb_dict[lb_id] = \ + {'id': lb_id, 'tenant_id': tenant_id} + else: + # Orphaned folder! + if purge_orphaned_folders: + try: + if self._is_orphan(bigip.device_name, folder): + self.system_helper.purge_folder_contents(bigip, folder) + self.system_helper.purge_folder(bigip, folder) + self._remove_from_orphan_cache(bigip.device_name, folder) + LOG.warning('ccloud: orphan folder purged %s on %s' % (folder, bigip.hostname)) + except Exception as exc: + LOG.error('Error purging folder %s: %s' % (folder, str(exc))) + return deployed_lb_dict + + @serialized('get_all_deployed_listeners') + @is_operational + def get_all_deployed_listeners(self, expand_subcollections=False): + LOG.debug('getting all deployed listeners on BIG-IPs') + deployed_virtual_dict = {} + for bigip in self.get_all_bigips(): + folders = self.system_helper.get_folders(bigip) + for folder in folders: + tenant_id = folder[len(self.service_adapter.prefix):] + if str(folder).startswith(self.service_adapter.prefix): + resource = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual) + deployed_listeners = resource.get_resources( + bigip, folder, expand_subcollections) + if deployed_listeners: + for virtual in deployed_listeners: + virtual_id = \ + virtual.name[len(self.service_adapter.prefix):] + l7_policy = '' + if hasattr(virtual, 'policiesReference') and \ + 'items' in virtual.policiesReference: + l7_policy = \ + virtual.policiesReference['items'][0] + l7_policy = l7_policy['fullPath'] + if virtual_id in deployed_virtual_dict: + deployed_virtual_dict[virtual_id][ + 'hostnames'].append(bigip.hostname) + else: + deployed_virtual_dict[virtual_id] = { + 'id': virtual_id, + 'tenant_id': tenant_id, + 'hostnames': [bigip.hostname], + 'l7_policy': l7_policy + } + return deployed_virtual_dict + + def _maintain_orphan_cache(self): + # clear cache every 24 hours to delete outdated objects + if (self.orphan_cache_last_reset + datetime.timedelta(hours=24)) < datetime.datetime.now(): + LOG.info('ccloud: Orphan objects cache cleared to avoid orphan orphans :-)') + self.orphan_cache_last_reset = datetime.datetime.now() + self.orphan_cache.clear() + + def _is_orphan(self, device_name, id): + + self._maintain_orphan_cache() + # check if orphan can be deleted or rise counter by 1 + if not id or not device_name: + return False + else: + key = device_name + '-' + id + + if key in self.orphan_cache: + if self.orphan_cache[key] >= 2: + if self.orphan_cleanup_testrun: + LOG.info('ccloud: Orphan TESTRUN: object %s marked for deletion %d times. Object would have been deleted NOW' % (key, self.orphan_cache[key])) + del self.orphan_cache[key] + return False + else: + LOG.info('ccloud: Orphan object %s marked for deletion %d times. Object will be deleted NOW' % (key, self.orphan_cache[key])) + return True + else: + self.orphan_cache[key] += 1 + if self.orphan_cleanup_testrun: + LOG.info('ccloud: Orphan TESTRUN %s marked for deletion %d times' % (key, self.orphan_cache[key])) + else: + LOG.info('ccloud: Orphan object %s marked for deletion %d times' % (key, self.orphan_cache[key])) + else: + self.orphan_cache[key] = 1 + if self.orphan_cleanup_testrun: + LOG.info('ccloud: Orphan TESTRUN object %s marked for deletion %d times' % (key, self.orphan_cache[key])) + else: + LOG.info('ccloud: Orphan object %s marked for deletion %d times' % (key, self.orphan_cache[key])) + return False + + def _remove_from_orphan_cache(self, device_name, id): + if id and device_name: + key = device_name + '-' + id + if key in self.orphan_cache: + LOG.info('ccloud: Orphan object %s got deleted and is removed from orphan cache' % key) + try: + del self.orphan_cache[key] + except Exception: + pass + return + + + def get_orphans_cache(self): + self._maintain_orphan_cache() + return self.orphan_cache + + @serialized('purge_orphaned_nodes') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_nodes(self, tenant_members): + # This algotithm is not able to determine nodes and members with an rd which isn't right fot the tenant, but + # it detects at least nodes and members without rd in case of a non global routed scenario + node_helper = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.node) + pool_helper = resource_helper.BigIPResourceHelper(resource_helper.ResourceType.pool) + for bigip in self.get_all_bigips(): + for tenant_id, members in tenant_members.iteritems(): + + partition = self.service_adapter.prefix + tenant_id + try: + nodes = node_helper.get_resources(bigip, partition=partition) + pools = pool_helper.get_resources(bigip, partition=partition) + except Exception as err: + LOG.info('ccloud: Error in node or pool retrieval for partition %s: %s', (partition, err.response)) + continue + + allf5members = [] + orphan_members = [] + orphan_nodes = [] + # All f5 members without rd or which are not part of os members are orphan + for pool in pools: + f5members = pool.members_s.get_collection() + # create cross pool membership list for verifying if node is used somewhere as member + allf5members.extend(f5members) + for f5member in f5members: + orphan_members.append(f5member) + for member in members: + # check with or without rd + if (not self.conf.f5_global_routed_mode): + if re.match(r"({0})(%\d+)(:{1})".format(member['address'], member['protocol_port']), f5member.name): + orphan_members.remove(f5member) + break + else: + if re.match(r"({0})(:{1})".format(member['address'], member['protocol_port']), f5member.name): + orphan_members.remove(f5member) + break + # All f5 nodes without rd or which are not used inside any membership are orphan + for node in nodes: + orphan_nodes.append(node.address) + # Node with no route id is orphan + if (not self.conf.f5_global_routed_mode) and ('%' not in node.address): + continue + for f5member in allf5members: + if node.address == f5member.address: + orphan_nodes.remove(node.address) + break + + # Log the determined orphans + orphan_member_names = [] + for omember in orphan_members: + orphan_member_names.append(omember.name) + if len(orphan_nodes) > 0: + LOG.debug('ccloud: Deleting orphan nodes --> {0}'.format(orphan_nodes)) + if len(orphan_member_names) > 0: + LOG.debug('ccloud: Deleting orphan members --> {0}'.format(orphan_member_names)) + + # Delete orphan members + for member in orphan_members: + member_name = member.name + if self._is_orphan(bigip.device_name, member_name): + try: + member.delete() + self._remove_from_orphan_cache(bigip.device_name, member_name) + except HTTPError as error: + LOG.warning("ccloud: Failed to delete orphan member %s: %s", (member_name, error.response)) + except Exception as err: + LOG.error("ccloud: Error - Failed to delete orphan member %s: %s", (member_name, err.response)) + # Delete orphan nodes + for node in orphan_nodes: + if self._is_orphan(bigip.device_name, node): + try: + node_helper.delete(bigip, name=urllib.quote(node), partition=partition) + self._remove_from_orphan_cache(bigip.device_name, node) + except HTTPError as error: + LOG.warning("ccloud: Failed to delete orphan node %s: %s", (node, error.response)) + except Exception as err: + LOG.error("ccloud: Error - Failed to delete orphan member %s: %s", (node, err.response)) + return True + + @serialized('get_all_deployed_pools') + @is_operational + def get_all_deployed_pools(self): + LOG.debug('getting all deployed pools on BIG-IPs') + deployed_pool_dict = {} + for bigip in self.get_all_bigips(): + folders = self.system_helper.get_folders(bigip) + for folder in folders: + tenant_id = folder[len(self.service_adapter.prefix):] + if str(folder).startswith(self.service_adapter.prefix): + resource = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.pool) + deployed_pools = resource.get_resources(bigip, folder) + if deployed_pools: + for pool in deployed_pools: + pool_id = \ + pool.name[len(self.service_adapter.prefix):] + monitor_id = '' + if hasattr(pool, 'monitor'): + monitor = pool.monitor.split('/')[2].strip() + monitor_id = \ + monitor[len(self.service_adapter.prefix):] + LOG.debug( + 'pool {} has monitor {}'.format( + pool.name, monitor)) + else: + LOG.debug( + 'pool {} has no healthmonitors'.format( + pool.name)) + if pool_id in deployed_pool_dict: + deployed_pool_dict[pool_id][ + 'hostnames'].append(bigip.hostname) + else: + deployed_pool_dict[pool_id] = { + 'id': pool_id, + 'tenant_id': tenant_id, + 'hostnames': [bigip.hostname], + 'monitors': monitor_id + } + return deployed_pool_dict + + @serialized('purge_orphaned_pool') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_pool(self, tenant_id=None, pool_id=None, + hostnames=list()): + for bigip in self.get_all_bigips(): + if bigip.hostname in hostnames: + try: + node_helper = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.node) + pool_name = self.service_adapter.prefix + pool_id + partition = self.service_adapter.prefix + tenant_id + pool = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.pool).load( + bigip, pool_name, partition) + members = pool.members_s.get_collection() + if self._is_orphan(bigip.device_name, pool_id): + pool.delete() + self._remove_from_orphan_cache(bigip.device_name, pool_id) + for member in members: + node_name = member.address + try: + if self._is_orphan(bigip.device_name, node_name): + node_helper.delete(bigip, + name=urllib.quote(node_name), + partition=partition) + self._remove_from_orphan_cache(bigip.device_name, node_name) + except HTTPError as e: + if e.response.status_code == 404: + pass + if e.response.status_code == 400: + LOG.warn("Failed to delete node -- in use") + else: + LOG.exception("Failed to delete node") + except HTTPError as err: + if err.response.status_code == 404: + LOG.debug('pool %s not on BIG-IP %s.' + % (pool_id, bigip.hostname)) + except Exception as exc: + LOG.exception('Exception purging pool %s' % str(exc)) + + @serialized('get_all_deployed_monitors') + @is_operational + def get_all_deployed_health_monitors(self): + """Retrieve a list of all Health Monitors deployed""" + LOG.debug('getting all deployed monitors on BIG-IP\'s') + monitor_types = ['http_monitor', 'https_monitor', 'tcp_monitor', + 'ping_monitor'] + deployed_monitor_dict = {} + adapter_prefix = self.service_adapter.prefix + for bigip in self.get_all_bigips(): + folders = self.system_helper.get_folders(bigip) + for folder in folders: + tenant_id = folder[len(adapter_prefix):] + if str(folder).startswith(adapter_prefix): + resources = map( + lambda x: resource_helper.BigIPResourceHelper( + getattr(resource_helper.ResourceType, x)), + monitor_types) + for resource in resources: + deployed_monitors = resource.get_resources( + bigip, folder) + if deployed_monitors: + for monitor in deployed_monitors: + monitor_id = monitor.name[len(adapter_prefix):] + if monitor_id in deployed_monitor_dict: + deployed_monitor_dict[monitor_id][ + 'hostnames'].append(bigip.hostname) + else: + deployed_monitor_dict[monitor_id] = { + 'id': monitor_id, + 'tenant_id': tenant_id, + 'hostnames': [bigip.hostname] + } + return deployed_monitor_dict + + @serialized('purge_orphaned_health_monitor') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_health_monitor(self, tenant_id=None, monitor_id=None, + hostnames=list()): + """Purge all monitors that exist on the BIG-IP but not in Neutron""" + resource_types = [ + resource_helper.BigIPResourceHelper(x) for x in [ + resource_helper.ResourceType.http_monitor, + resource_helper.ResourceType.https_monitor, + resource_helper.ResourceType.ping_monitor, + resource_helper.ResourceType.tcp_monitor]] + for bigip in self.get_all_bigips(): + if bigip.hostname in hostnames: + try: + monitor_name = self.service_adapter.prefix + monitor_id + partition = self.service_adapter.prefix + tenant_id + monitor = None + for monitor_type in resource_types: + try: + monitor = monitor_type.load(bigip, monitor_name, + partition) + break + except HTTPError as err: + if err.response.status_code == 404: + continue + if self._is_orphan(bigip.device_name, monitor_id): + monitor.delete() + self._remove_from_orphan_cache(bigip.device_name, monitor_id) + except TypeError as err: + if 'NoneType' in err: + LOG.exception("Could not find monitor {}".format( + monitor_name)) + except Exception as exc: + LOG.exception('Exception purging monitor %s' % str(exc)) + + @serialized('get_all_deployed_l7_policys') + @is_operational + def get_all_deployed_l7_policys(self): + """Retrieve a dict of all l7policies deployed + + The dict returned will have the following format: + {policy_bigip_id_0: {'id': policy_id_0, + 'tenant_id': tenant_id, + 'hostnames': [hostnames_0]} + ... + } + Where hostnames is the list of BIG-IP hostnames impacted, and the + policy_id is the policy_bigip_id without 'wrapper_policy_' + """ + LOG.debug('getting all deployed l7_policys on BIG-IP\'s') + deployed_l7_policys_dict = {} + for bigip in self.get_all_bigips(): + folders = self.system_helper.get_folders(bigip) + for folder in folders: + tenant_id = folder[len(self.service_adapter.prefix):] + if str(folder).startswith(self.service_adapter.prefix): + resource = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.l7policy) + deployed_l7_policys = resource.get_resources( + bigip, folder) + if deployed_l7_policys: + for l7_policy in deployed_l7_policys: + l7_policy_id = l7_policy.name + if l7_policy_id in deployed_l7_policys_dict: + my_dict = \ + deployed_l7_policys_dict[l7_policy_id] + my_dict['hostnames'].append(bigip.hostname) + else: + po_id = l7_policy_id.replace( + 'wrapper_policy_', '') + deployed_l7_policys_dict[l7_policy_id] = { + 'id': po_id, + 'tenant_id': tenant_id, + 'hostnames': [bigip.hostname] + } + return deployed_l7_policys_dict + + @serialized('purge_orphaned_l7_policy') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_l7_policy(self, tenant_id=None, l7_policy_id=None, + hostnames=list(), listener_id=None): + """Purge all l7_policys that exist on the BIG-IP but not in Neutron""" + for bigip in self.get_all_bigips(): + if bigip.hostname in hostnames: + error = None + try: + l7_policy_name = l7_policy_id + partition = self.service_adapter.prefix + tenant_id + if listener_id and partition: + if self.service_adapter.prefix not in listener_id: + listener_id = \ + self.service_adapter.prefix + listener_id + li_resource = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual).load( + bigip, listener_id, partition) + li_resource.update(policies=[]) + l7_policy = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.l7policy).load( + bigip, l7_policy_name, partition) + if self._is_orphan(bigip.device_name, l7_policy_id): + l7_policy.delete() + self._remove_from_orphan_cache(bigip.device_name, l7_policy_id) + except HTTPError as err: + if err.response.status_code == 404: + LOG.debug('l7_policy %s not on BIG-IP %s.' + % (l7_policy_id, bigip.hostname)) + else: + error = err + except Exception as exc: + error = err + if error: + kwargs = dict( + tenant_id=tenant_id, l7_policy_id=l7_policy_id, + hostname=bigip.hostname, listener_id=listener_id) + LOG.exception('Exception: purge_orphaned_l7_policy({}) ' + '"{}"'.format(kwargs, exc)) + + @serialized('purge_orphaned_loadbalancer') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_loadbalancer(self, tenant_id=None, + loadbalancer_id=None, hostnames=list()): + for bigip in self.get_all_bigips(): + if bigip.hostname in hostnames: + try: + va_name = self.service_adapter.prefix + loadbalancer_id + partition = self.service_adapter.prefix + tenant_id + va = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual_address).load( + bigip, va_name, partition) + # get virtual services (listeners) + # referencing this virtual address + vses = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual).get_resources( + bigip, partition) + vs_dest_compare = '/' + partition + '/' + va.name + for vs in vses: + if str(vs.destination).startswith(vs_dest_compare): + if hasattr(vs, 'pool'): + pool = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.pool).load( + bigip, os.path.basename(vs.pool), + partition) + vs_name = vs.name + if self._is_orphan(bigip.device_name, vs_name): + vs.delete() + self._remove_from_orphan_cache(bigip.device_name, vs_name) + pool_name = pool.name + if self._is_orphan(bigip.device_name, pool_name): + pool.delete() + self._remove_from_orphan_cache(bigip.device_name, pool_name) + else: + vs_name = vs.name + if self._is_orphan(bigip.device_name, vs_name): + vs.delete() + self._remove_from_orphan_cache(bigip.device_name, vs_name) + if self._is_orphan(bigip.device_name, va_name): + resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual_address).delete( + bigip, va_name, partition) + self._remove_from_orphan_cache(bigip.device_name, va_name) + except HTTPError as err: + if err.response.status_code == 404: + LOG.debug('loadbalancer %s not on BIG-IP %s.' + % (loadbalancer_id, bigip.hostname)) + except Exception as exc: + LOG.exception('Exception purging loadbalancer %s' + % str(exc)) + + @serialized('purge_orphaned_listener') + @is_operational + @log_helpers.log_method_call + def purge_orphaned_listener( + self, tenant_id=None, listener_id=None, hostnames=[]): + for bigip in self.get_all_bigips(): + if bigip.hostname in hostnames: + try: + listener_name = self.service_adapter.prefix + listener_id + partition = self.service_adapter.prefix + tenant_id + listener = resource_helper.BigIPResourceHelper( + resource_helper.ResourceType.virtual).load( + bigip, listener_name, partition) + if self._is_orphan(bigip.device_name, listener_id): + listener.delete() + self._remove_from_orphan_cache(bigip.device_name, listener_id) + except HTTPError as err: + if err.response.status_code == 404: + LOG.debug('listener %s not on BIG-IP %s.' + % (listener_id, bigip.hostname)) + except Exception as exc: + LOG.exception('Exception purging listener %s' % str(exc)) + @serialized('create_loadbalancer') - @is_connected + @is_operational def create_loadbalancer(self, loadbalancer, service): """Create virtual server""" return self._common_service_handler(service) @serialized('update_loadbalancer') - @is_connected + @is_operational def update_loadbalancer(self, old_loadbalancer, loadbalancer, service): """Update virtual server""" # anti-pattern three args unused. return self._common_service_handler(service) @serialized('delete_loadbalancer') - @is_connected + @is_operational def delete_loadbalancer(self, loadbalancer, service): """Delete loadbalancer""" LOG.debug("Deleting loadbalancer") @@ -787,14 +1635,14 @@ def delete_loadbalancer(self, loadbalancer, service): delete_event=True) @serialized('create_listener') - @is_connected + @is_operational def create_listener(self, listener, service): """Create virtual server""" LOG.debug("Creating listener") return self._common_service_handler(service) @serialized('update_listener') - @is_connected + @is_operational def update_listener(self, old_listener, listener, service): """Update virtual server""" LOG.debug("Updating listener") @@ -802,77 +1650,103 @@ def update_listener(self, old_listener, listener, service): return self._common_service_handler(service) @serialized('delete_listener') - @is_connected + @is_operational def delete_listener(self, listener, service): """Delete virtual server""" LOG.debug("Deleting listener") - return self._common_service_handler(service) + return self._common_service_handler(service, delete_event=True) @serialized('create_pool') - @is_connected + @is_operational def create_pool(self, pool, service): """Create lb pool""" LOG.debug("Creating pool") return self._common_service_handler(service) @serialized('update_pool') - @is_connected + @is_operational def update_pool(self, old_pool, pool, service): """Update lb pool""" LOG.debug("Updating pool") return self._common_service_handler(service) @serialized('delete_pool') - @is_connected + @is_operational def delete_pool(self, pool, service): """Delete lb pool""" LOG.debug("Deleting pool") - return self._common_service_handler(service) + return self._common_service_handler(service, delete_event=True) @serialized('create_member') - @is_connected + @is_operational def create_member(self, member, service): """Create pool member""" LOG.debug("Creating member") return self._common_service_handler(service) @serialized('update_member') - @is_connected + @is_operational def update_member(self, old_member, member, service): """Update pool member""" LOG.debug("Updating member") return self._common_service_handler(service) @serialized('delete_member') - @is_connected + @is_operational def delete_member(self, member, service): """Delete pool member""" LOG.debug("Deleting member") return self._common_service_handler(service, delete_event=True) @serialized('create_health_monitor') - @is_connected + @is_operational def create_health_monitor(self, health_monitor, service): """Create pool health monitor""" LOG.debug("Creating health monitor") return self._common_service_handler(service) @serialized('update_health_monitor') - @is_connected - def update_health_monitor(self, old_health_monitor, - health_monitor, service): + @is_operational + def update_health_monitor(self, old_health_monitor, health_monitor, service): """Update pool health monitor""" LOG.debug("Updating health monitor") return self._common_service_handler(service) @serialized('delete_health_monitor') - @is_connected + @is_operational def delete_health_monitor(self, health_monitor, service): """Delete pool health monitor""" LOG.debug("Deleting health monitor") - return self._common_service_handler(service) + return self._common_service_handler(service, delete_event=True) + + # sapcc: get all snat pools + @serialized('get_all_snat_pools') + @is_operational + def get_all_snat_pools(self, partition=None): + LOG.debug('getting all snat pools on BIG-IPs') + + snat_pools = [] + if self.network_builder: + for bigip in self.get_all_bigips(): + snat_pools += self.network_builder.bigip_snat_manager.get_snats(bigip, partition) + + return snat_pools + + # sapcc: get all virtual_addresses + @serialized('get_all_virtual_addresses') + @is_operational + def get_all_virtual_addresses(self): + LOG.debug('getting all virtual addresses on BIG-IPs') + + virtual_address_s = [] + if self.network_builder: + for bigip in self.get_all_bigips(): + virtual_address_s += self.va_manager.get_resources(bigip) - @is_connected + return virtual_address_s + + + @is_operational def get_stats(self, service): lb_stats = {} stats = ['clientside.bitsIn', @@ -904,23 +1778,6 @@ def get_stats(self, service): finally: return lb_stats - @serialized('remove_orphans') - def remove_orphans(self, all_loadbalancers): - """Remove out-of-date configuration on big-ips """ - existing_tenants = [] - existing_lbs = [] - for loadbalancer in all_loadbalancers: - existing_tenants.append(loadbalancer['tenant_id']) - existing_lbs.append(loadbalancer['lb_id']) - - for bigip in self.get_all_bigips(): - bigip.pool.purge_orphaned_pools(existing_lbs) - for bigip in self.get_all_bigips(): - bigip.system.purge_orphaned_folders_contents(existing_tenants) - - for bigip in self.get_all_bigips(): - bigip.system.purge_orphaned_folders(existing_tenants) - def fdb_add(self, fdb): # Add (L2toL3) forwarding database entries self.remove_ips_from_fdb_update(fdb) @@ -972,22 +1829,26 @@ def tunnel_sync(self): return False @serialized('sync') - @is_connected + @is_operational def sync(self, service): """Sync service defintion to device""" + + load_balancer = service.get('loadbalancer',None) + # plugin_rpc may not be set when unit testing - if self.plugin_rpc: + if self.plugin_rpc and load_balancer: # Get the latest service. It may have changed. service = self.plugin_rpc.get_service_by_loadbalancer_id( - service['loadbalancer']['id'] + load_balancer.get('id') ) - if service['loadbalancer']: + + if service.get('loadbalancer',None): return self._common_service_handler(service) else: - LOG.debug("Attempted sync of deleted pool") + LOG.debug("Attempted sync of deleted load balancer") @serialized('backup_configuration') - @is_connected + @is_operational def backup_configuration(self): # Save Configuration on Devices for bigip in self.get_all_bigips(): @@ -1096,7 +1957,7 @@ def service_object_teardown(self, service): m_obj.delete() def _service_exists(self, service): - # Returns whether the bigip has a pool for the service + # Returns whether the bigip has the service defined if not service['loadbalancer']: return False loadbalancer = service['loadbalancer'] @@ -1105,11 +1966,15 @@ def _service_exists(self, service): loadbalancer['tenant_id'] ) + if self.network_builder: + # append route domain to member address + self.network_builder._annotate_service_route_domains(service) + # Foreach bigip in the cluster: for bigip in self.get_config_bigips(): # Does the tenant folder exist? if not self.system_helper.folder_exists(bigip, folder_name): - LOG.error("Folder %s does not exists on bigip: %s" % + LOG.warning("Folder %s does not exists on bigip: %s" % (folder_name, bigip.hostname)) return False @@ -1117,7 +1982,7 @@ def _service_exists(self, service): virtual_address = VirtualAddress(self.service_adapter, loadbalancer) if not virtual_address.exists(bigip): - LOG.error("Virtual address %s(%s) does not " + LOG.warning("Virtual address %s(%s) does not " "exists on bigip: %s" % (virtual_address.name, virtual_address.address, bigip.hostname)) @@ -1132,12 +1997,12 @@ def _service_exists(self, service): if not self.vs_manager.exists(bigip, name=virtual_server['name'], partition=folder_name): - LOG.error("Virtual /%s/%s not found on bigip: %s" % + LOG.warning("Virtual /%s/%s not found on bigip: %s" % (virtual_server['name'], folder_name, bigip.hostname)) return False - # Ensure that each virtual service exists. + # Ensure that each pool exists. for pool in service['pools']: svc = {"loadbalancer": loadbalancer, "pool": pool} @@ -1146,11 +2011,42 @@ def _service_exists(self, service): bigip, name=bigip_pool['name'], partition=folder_name): - LOG.error("Pool /%s/%s not found on bigip: %s" % + LOG.warning("Pool /%s/%s not found on bigip: %s" % (bigip_pool['name'], folder_name, bigip.hostname)) return False - + else: + deployed_pool = self.pool_manager.load( + bigip, + name=bigip_pool['name'], + partition=folder_name) + deployed_members = \ + deployed_pool.members_s.get_collection() + + # First check that number of members deployed + # is equal to the number in the service. + if len(deployed_members) != len(pool['members']): + LOG.warning("Pool %s members member count mismatch " + "match: deployed %d != service %d" % + (bigip_pool['name'], len(deployed_members), + len(pool['members']))) + return False + + # Ensure each pool member exists + for member in service['members']: + if member['pool_id'] == pool['id']: + lb = self.lbaas_builder + pool = lb.get_pool_by_id( + service, member["pool_id"]) + svc = {"loadbalancer": loadbalancer, + "member": member, + "pool": pool} + if not lb.pool_builder.member_exists(svc, bigip): + LOG.warning("Pool member not found: %s" % + svc['member']) + return False + + # Ensure that each health monitor exists. for healthmonitor in service['healthmonitors']: svc = {"loadbalancer": loadbalancer, "healthmonitor": healthmonitor} @@ -1158,7 +2054,7 @@ def _service_exists(self, service): monitor_ep = self._get_monitor_endpoint(bigip, svc) if not monitor_ep.exists(name=monitor['name'], partition=folder_name): - LOG.error("Monitor /%s/%s not found on bigip: %s" % + LOG.warning("Monitor /%s/%s not found on bigip: %s" % (monitor['name'], folder_name, bigip.hostname)) return False @@ -1172,7 +2068,7 @@ def get_loadbalancers_in_tenant(self, tenant_id): def _common_service_handler(self, service, delete_partition=False, - delete_event=False): + delete_event=False,cli_sync=False): # Assure that the service is configured on bigip(s) start_time = time() @@ -1193,6 +2089,8 @@ def _common_service_handler(self, service, try: try: self.tenant_manager.assure_tenant_created(service) + LOG.debug(" _assure_tenant_created took %.5f secs" % + (time() - start_time)) except Exception as e: LOG.error("Tenant folder creation exception: %s", e.message) @@ -1201,9 +2099,6 @@ def _common_service_handler(self, service, plugin_const.ERROR raise e - LOG.debug(" _assure_tenant_created took %.5f secs" % - (time() - start_time)) - traffic_group = self.service_to_traffic_group(service) loadbalancer['traffic_group'] = traffic_group @@ -1243,18 +2138,20 @@ def _common_service_handler(self, service, {'check_for_delete_subnets': {}, 'do_not_delete_subnets': []} - LOG.debug("XXXXXXXXX: Pre assure service") - # pdb.set_trace() + LOG.debug("ccloud: Pre assure service ***********************************************") self.lbaas_builder.assure_service(service, traffic_group, - all_subnet_hints) - LOG.debug("XXXXXXXXX: Post assure service") + all_subnet_hints, + delete_event) + LOG.debug("ccloud: Post assure service **********************************************") if self.network_builder: start_time = time() try: + LOG.debug("ccloud: Pre post_service_networking ***********************************************") self.network_builder.post_service_networking( service, all_subnet_hints) + LOG.debug("ccloud: Post post_service_networking **********************************************") except Exception as error: LOG.error("Post-network exception: icontrol_driver: %s", error.message) @@ -1278,7 +2175,7 @@ def _common_service_handler(self, service, self.tenant_manager.assure_tenant_cleanup(service, all_subnet_hints) - if do_service_update: + if do_service_update and not cli_sync: self.update_service_status(service) lb_provisioning_status = loadbalancer.get("provisioning_status", @@ -1290,8 +2187,7 @@ def _common_service_handler(self, service, return lb_pending def update_service_status(self, service, timed_out=False): - """Update status of objects in OpenStack """ - + """Update status of objects in controller.""" LOG.debug("_update_service_status") if not self.plugin_rpc: @@ -1323,13 +2219,13 @@ def update_service_status(self, service, timed_out=False): self._update_loadbalancer_status(service, timed_out) def _update_member_status(self, members, timed_out): - """Update member status in OpenStack """ + """Update member status in OpenStack.""" for member in members: if 'provisioning_status' in member: provisioning_status = member['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): if timed_out: member['provisioning_status'] = plugin_const.ERROR @@ -1358,7 +2254,7 @@ def _update_health_monitor_status(self, health_monitors): if 'provisioning_status' in health_monitor: provisioning_status = health_monitor['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): self.plugin_rpc.update_health_monitor_status( health_monitor['id'], plugin_const.ACTIVE, @@ -1373,14 +2269,14 @@ def _update_health_monitor_status(self, health_monitors): self.plugin_rpc.update_health_monitor_status( health_monitor['id']) - @log_helpers.log_method_call + def _update_pool_status(self, pools): """Update pool status in OpenStack """ for pool in pools: if 'provisioning_status' in pool: provisioning_status = pool['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): self.plugin_rpc.update_pool_status( pool['id'], plugin_const.ACTIVE, @@ -1393,7 +2289,7 @@ def _update_pool_status(self, pools): elif provisioning_status == plugin_const.ERROR: self.plugin_rpc.update_pool_status(pool['id']) - @log_helpers.log_method_call + def _update_listener_status(self, service): """Update listener status in OpenStack """ listeners = service['listeners'] @@ -1401,7 +2297,7 @@ def _update_listener_status(self, service): if 'provisioning_status' in listener: provisioning_status = listener['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): self.plugin_rpc.update_listener_status( listener['id'], plugin_const.ACTIVE, @@ -1418,14 +2314,14 @@ def _update_listener_status(self, service): provisioning_status, lb_const.OFFLINE) - @log_helpers.log_method_call + def _update_l7rule_status(self, l7rules): """Update l7rule status in OpenStack """ for l7rule in l7rules: if 'provisioning_status' in l7rule: provisioning_status = l7rule['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): self.plugin_rpc.update_l7rule_status( l7rule['id'], l7rule['policy_id'], @@ -1439,7 +2335,7 @@ def _update_l7rule_status(self, l7rules): self.plugin_rpc.update_l7rule_status( l7rule['id'], l7rule['policy_id']) - @log_helpers.log_method_call + def _update_l7policy_status(self, l7policies): LOG.debug("_update_l7policy_status") """Update l7policy status in OpenStack """ @@ -1447,7 +2343,7 @@ def _update_l7policy_status(self, l7policies): if 'provisioning_status' in l7policy: provisioning_status = l7policy['provisioning_status'] if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): self.plugin_rpc.update_l7policy_status( l7policy['id'], plugin_const.ACTIVE, @@ -1460,7 +2356,7 @@ def _update_l7policy_status(self, l7policies): elif provisioning_status == plugin_const.ERROR: self.plugin_rpc.update_l7policy_status(l7policy['id']) - @log_helpers.log_method_call + def _update_loadbalancer_status(self, service, timed_out=False): """Update loadbalancer status in OpenStack """ loadbalancer = service.get('loadbalancer', {}) @@ -1468,7 +2364,7 @@ def _update_loadbalancer_status(self, service, timed_out=False): plugin_const.ERROR) if (provisioning_status == plugin_const.PENDING_CREATE or - provisioning_status == plugin_const.PENDING_UPDATE): + provisioning_status == plugin_const.PENDING_UPDATE or provisioning_status == plugin_const.ACTIVE): if timed_out: operating_status = (lb_const.OFFLINE) if provisioning_status == plugin_const.PENDING_CREATE: @@ -1500,12 +2396,17 @@ def _update_loadbalancer_status(self, service, timed_out=False): else: LOG.error('Loadbalancer provisioning status is invalid') - @is_connected + @is_operational def update_operating_status(self, service): if 'members' in service: if self.network_builder: # append route domain to member address - self.network_builder._annotate_service_route_domains(service) + try: + self.network_builder._annotate_service_route_domains( + service) + except f5ex.InvalidNetworkType as exc: + LOG.warning(exc.msg) + return # get currrent member status self.lbaas_builder.update_operating_status(service) @@ -1526,8 +2427,9 @@ def get_active_bigip(self): return bigips[0] for bigip in bigips: - if self.cluster_manager.is_device_active(bigip): - return bigip + if hasattr(bigip, 'failover_state'): + if bigip.failover_state == 'active': + return bigip # if can't determine active, default to first one return bigips[0] @@ -1544,31 +2446,46 @@ def tenant_to_traffic_group(self, tenant_id): tg_index = int(hexhash, 16) % len(self.__traffic_groups) return self.__traffic_groups[tg_index] + # these functions should return only active BIG-IP + # not errored BIG-IPs. def get_bigip(self): - # Get one consistent big-ip - # As implemented I think this always returns the "first" bigip - # without any HTTP traffic? CONFIRMED: __bigips are mgmt_rts - hostnames = sorted(self.__bigips) - for i in range(len(hostnames)): # C-style make Pythonic. - try: - bigip = self.__bigips[hostnames[i]] # Calling devices?! - return bigip - except urllib2.URLError: - pass - raise urllib2.URLError('cannot communicate to any bigips') + hostnames = sorted(list(self.__bigips)) + for host in hostnames: + if hasattr(self.__bigips[host], 'status') and \ + self.__bigips[host].status == 'active': + return self.__bigips[host] def get_bigip_hosts(self): - # Get all big-ips hostnames under management - return self.__bigips + return_hosts = [] + for host in list(self.__bigips): + if hasattr(self.__bigips[host], 'status') and \ + self.__bigips[host].status == 'active': + return_hosts.append(host) + return sorted(return_hosts) def get_all_bigips(self): - # Get all big-ips under management - return self.__bigips.values() + return_bigips = [] + for host in list(self.__bigips): + if hasattr(self.__bigips[host], 'status') and \ + self.__bigips[host].status == 'active': + return_bigips.append(self.__bigips[host]) + return return_bigips def get_config_bigips(self): - # Return a list of big-ips that need to be configured. return self.get_all_bigips() + # these are the refactored methods + def get_active_bigips(self): + return self.get_all_bigips() + + def get_errored_bigips_hostnames(self): + return_hostnames = [] + for host in list(self.__bigips): + bigip = self.__bigips[host] + if hasattr(bigip, 'status') and bigip.status == 'error': + return_hostnames.append(host) + return return_hostnames + def get_inbound_throughput(self, bigip, global_statistics=None): return self.stat_helper.get_inbound_throughput( bigip, global_stats=global_statistics) @@ -1592,6 +2509,12 @@ def get_ssltps(self, bigip=None, global_statistics=None): def get_node_count(self, bigip=None, global_statistics=None): return len(bigip.tm.ltm.nodes.get_collection()) + def get_virtual_address_count(self, bigip=None, global_statistics=None): + return len(bigip.tm.ltm.virtual_address_s.get_collection()) + + def get_virtual_server_count(self, bigip=None, global_statistics=None): + return len(bigip.tm.ltm.virtuals.get_collection()) + def get_clientssl_profile_count(self, bigip=None, global_statistics=None): return ssl_profile.SSLProfileHelper.get_client_ssl_profile_count(bigip) @@ -1608,13 +2531,24 @@ def get_route_domain_count(self, bigip=None, global_statistics=None): return self.network_helper.get_route_domain_count(bigip) def _init_traffic_groups(self, bigip): - self.__traffic_groups = self.cluster_manager.get_traffic_groups(bigip) - if 'traffic-group-local-only' in self.__traffic_groups: - self.__traffic_groups.remove('traffic-group-local-only') - self.__traffic_groups.sort() + try: + LOG.debug('retrieving traffic groups from %s' % bigip.hostname) + self.__traffic_groups = \ + self.cluster_manager.get_traffic_groups(bigip) + if 'traffic-group-local-only' in self.__traffic_groups: + LOG.debug('removing reference to non-floating traffic group') + self.__traffic_groups.remove('traffic-group-local-only') + self.__traffic_groups.sort() + LOG.debug('service placement will done on traffic group(s): %s' + % self.__traffic_groups) + except Exception: + bigip.status = 'error' + bigip.status_message = \ + 'could not determine traffic groups for service placement' + raise def _validate_bigip_version(self, bigip, hostname): - # Ensure the BIG-IP® has sufficient version + # Ensure the BIG-IP has sufficient version major_version = self.system_helper.get_major_version(bigip) if major_version < f5const.MIN_TMOS_MAJOR_VERSION: raise f5ex.MajorVersionValidateFailed( @@ -1630,46 +2564,46 @@ def _validate_bigip_version(self, bigip, hostname): return major_version, minor_version @serialized('create_l7policy') - @is_connected + @is_operational def create_l7policy(self, l7policy, service): """Create lb l7policy""" LOG.debug("Creating l7policy") self._common_service_handler(service) @serialized('update_l7policy') - @is_connected + @is_operational def update_l7policy(self, old_l7policy, l7policy, service): """Update lb l7policy""" LOG.debug("Updating l7policy") self._common_service_handler(service) @serialized('delete_l7policy') - @is_connected + @is_operational def delete_l7policy(self, l7policy, service): """Delete lb l7policy""" LOG.debug("Deleting l7policy") - self._common_service_handler(service) + self._common_service_handler(service, delete_event=True) @serialized('create_l7rule') - @is_connected + @is_operational def create_l7rule(self, pool, service): """Create lb l7rule""" LOG.debug("Creating l7rule") self._common_service_handler(service) @serialized('update_l7rule') - @is_connected + @is_operational def update_l7rule(self, old_l7rule, l7rule, service): """Update lb l7rule""" LOG.debug("Updating l7rule") self._common_service_handler(service) @serialized('delete_l7rule') - @is_connected + @is_operational def delete_l7rule(self, l7rule, service): """Delete lb l7rule""" LOG.debug("Deleting l7rule") - self._common_service_handler(service) + self._common_service_handler(service, delete_event=True) def trace_service_requests(self, service): with open(self.file_name, 'r+') as fp: @@ -1679,7 +2613,7 @@ def trace_service_requests(self, service): fp.write(']') def get_config_dir(self): - """Determines F5 agent configuration directory. + """Determine F5 agent configuration directory. Oslo cfg has a config_dir option, but F5 agent is not currently started with this option. To be complete, the code will check if diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/l2_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/l2_service.py index 3549dc54f..280c33519 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/l2_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/l2_service.py @@ -34,7 +34,7 @@ def _get_tunnel_name(network): - # BIG-IP® object name for a tunnel + # BIG-IP object name for a tunnel tunnel_type = network['provider:network_type'] tunnel_id = network['provider:segmentation_id'] return 'tunnel-' + str(tunnel_type) + '-' + str(tunnel_id) @@ -130,11 +130,12 @@ def set_context(self, context): def is_common_network(self, network): # Does this network belong in the /Common folder? - return network['shared'] or \ - (network['id'] in self.conf.common_network_ids) or \ - ('router:external' in network and - network['router:external'] and - self.conf.f5_common_external_networks) + return True + # return network['shared'] or \ + # (network['id'] in self.conf.common_network_ids) or \ + # ('router:external' in network and + # network['router:external'] and + # self.conf.f5_common_external_networks) def get_vlan_name(self, network, hostname): # Construct a consistent vlan name @@ -254,6 +255,9 @@ def _assure_device_network_flat(self, network, bigip, network_folder): 'partition': network_folder, 'description': network['id'], 'route_domain_id': network['route_domain_id']} + if network['mtu']: + model['mtu'] = network['mtu'] + self.network_helper.create_vlan(bigip, model) except Exception as err: LOG.exception("%s", err.message) @@ -306,6 +310,9 @@ def _assure_device_network_vlan(self, network, bigip, network_folder): 'partition': network_folder, 'description': network['id'], 'route_domain_id': network['route_domain_id']} + if network['mtu']: + model['mtu'] = network['mtu'] + self.network_helper.create_vlan(bigip, model) except Exception as err: LOG.exception("%s", err.message) @@ -402,6 +409,9 @@ def _assure_vcmp_device_network(self, bigip, vlan): 'interface': vlan['interface'], 'description': vlan['network']['id'], 'route_domain_id': vlan['network']['route_domain_id']} + if vlan['network']['mtu']: + model['mtu'] = vlan['network']['mtu'] + try: self.network_helper.create_vlan(vcmp_host['bigip'], model) LOG.debug(('Created VLAN %s on vCMP Host %s' % diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/l3_binding.py b/f5_openstack_agent/lbaasv2/drivers/bigip/l3_binding.py index f5e9a36ee..76a6fde11 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/l3_binding.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/l3_binding.py @@ -58,8 +58,8 @@ def __init__(self, conf, driver): LOG.debug('l3_binding_static_mappings not configured') def register_bigip_mac_addresses(self): - # Delayed binding BIG-IP® ports will be called - # after BIG-IP® endpoints are registered. + # Delayed binding BIG-IP ports will be called + # after BIG-IP endpoints are registered. if not self.__initialized__bigip_ports: for bigip in self.driver.get_all_bigips(): LOG.debug('Request Port information for MACs: %s' diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_adapter.py b/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_adapter.py index bc078db76..8c1227c26 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_adapter.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_adapter.py @@ -159,15 +159,22 @@ def _check_if_adapted_rules_empty(self): 'wrapper_policy.' raise PolicyHasNoRules(msg) - def _adapt_policy(self): + def _adapt_policy(self,service): '''Setup the wrapper policy, which will contain rules.''' if not self.service['l7rules']: msg = 'No Rules given to implement. A Policy cannot be attached ' \ 'to a Virtual until it has one or more Rules.' raise PolicyHasNoRules(msg) + + listener_id = '' + for l7_policy in service['l7policies']: + listener_id = l7_policy.get('listener_id','') + break + + self.policy_dict = {} - self.policy_dict['name'] = 'wrapper_policy' + self.policy_dict['name'] = 'wrapper_policy_'+listener_id self.policy_dict['partition'] = self.folder self.policy_dict['strategy'] = 'first-match' self.policy_dict['rules'] = [] @@ -181,9 +188,9 @@ def translate(self, service): self.service = service self.folder = self.get_folder_name( self.service['l7policies'][0]['tenant_id']) - self._adapt_policy() + self._adapt_policy(service) return self.policy_dict def translate_name(self, l7policy): - return {'name': 'wrapper_policy', + return {'name': 'wrapper_policy_'+l7policy.get('listener_id',''), 'partition': self.get_folder_name(l7policy['tenant_id'])} diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_service.py index 3fe394637..f600b5b8d 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/l7policy_service.py @@ -29,12 +29,15 @@ from f5_openstack_agent.lbaasv2.drivers.bigip.vs_builder import \ VirtualServerBuilder +#import pdb + LOG = logging.getLogger(__name__) class L7PolicyService(object): """Handles requests to create, update, delete L7 policies on BIG-IPs.""" - def __init__(self, conf): + def __init__(self, lbaas_builder, conf): + self.lbaas_builder = lbaas_builder self.conf = conf def create_l7policy(self, l7policy, service_object, bigips): @@ -55,7 +58,7 @@ def create_l7policy(self, l7policy, service_object, bigips): # create L7 policy try: l7policy_adapter = L7PolicyServiceAdapter(self.conf) - policies = self.build_policy(l7policy, lbaas_service) + policies = self.build_l7policy(l7policy, lbaas_service) if policies['l7policies']: f5_l7policy = l7policy_adapter.translate(policies) stack.append(L7PolicyBuilder(event, f5_l7policy)) @@ -109,7 +112,7 @@ def update_l7policy(self, l7policy, service_object, bigips): try: l7policy_adapter = L7PolicyServiceAdapter(self.conf) - policies = self.build_policy(l7policy, lbaas_service) + policies = self.build_l7policy(l7policy, lbaas_service) if policies['l7policies']: f5_l7policy = l7policy_adapter.translate(policies) stack.append(L7PolicyBuilder(event, f5_l7policy)) @@ -157,14 +160,13 @@ def update_l7rule(self, l7rule, service_object, bigips): # re-create policy with updated rule self.update_l7policy(l7policy, service_object, bigips) - @staticmethod - def build_policy(l7policy, lbaas_service): + def build_l7policy(self, l7policy, lbaas_service): # build data structure for service adapter input LOG.debug("L7PolicyService: service") - import pprint - LOG.debug(pprint.pformat(lbaas_service.service_object, indent=4)) - LOG.debug("L7PolicyService: l7policy") - LOG.debug(pprint.pformat(l7policy, indent=4)) + #import pprint + #LOG.debug(pprint.pformat(lbaas_service.service_object, indent=4)) + #LOG.debug("L7PolicyService: l7policy") + #LOG.debug(pprint.pformat(l7policy, indent=4)) os_policies = {'l7rules': [], 'l7policies': []} @@ -174,13 +176,18 @@ def build_policy(l7policy, lbaas_service): for policy_id in listener['l7_policies']: policy = lbaas_service.get_l7policy(policy_id['id']) if policy: - os_policies['l7policies'].append(policy) - for rule in policy['rules']: - l7rule = lbaas_service.get_l7rule(rule['id']) - if l7rule: - os_policies['l7rules'].append(l7rule) - - LOG.debug(pprint.pformat(os_policies, indent=4)) + is_esd = False + if policy['name'] and self.lbaas_builder.is_esd(policy['name']): + is_esd = True + if not is_esd: + os_policies['l7policies'].append(policy) + for rule in policy['rules']: + l7rule = lbaas_service.get_l7rule(rule['id']) + if l7rule: + os_policies['l7rules'].append(l7rule) + + #LOG.debug(pprint.pformat(os_policies, indent=4)) + LOG.debug(os_policies) return os_policies @staticmethod diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py b/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py index 5a84c91f6..afb690b0d 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py @@ -27,63 +27,77 @@ from f5_openstack_agent.lbaasv2.drivers.bigip import pool_service from f5_openstack_agent.lbaasv2.drivers.bigip import virtual_address +from f5_openstack_agent.lbaasv2.drivers.bigip import utils + from requests import HTTPError +#import pdb + LOG = logging.getLogger(__name__) class LBaaSBuilder(object): - # F5® LBaaS Driver using iControl® for BIG-IP® to - # create objects (vips, pools) - not using an iApp®.""" + # F5 LBaaS Driver using iControl for BIG-IP to + # create objects (vips, pools) - not using an iApp.""" def __init__(self, conf, driver, l2_service=None): self.conf = conf self.driver = driver self.l2_service = l2_service self.service_adapter = driver.service_adapter - self.listener_builder = listener_service.ListenerServiceBuilder( - self.service_adapter, - driver.cert_manager, - conf.f5_parent_ssl_profile) + self.listener_builder = listener_service.ListenerServiceBuilder(self, + self.service_adapter, + driver.cert_manager, + conf.f5_parent_ssl_profile) self.pool_builder = pool_service.PoolServiceBuilder( - self.service_adapter - ) - self.l7service = l7policy_service.L7PolicyService(conf) + self.service_adapter, + conf.f5_parent_https_monitor) + self.l7service = l7policy_service.L7PolicyService(self, conf) self.esd = None - def assure_service(self, service, traffic_group, all_subnet_hints): + @utils.instrument_execution_time + def assure_service(self, service, traffic_group, all_subnet_hints, delete_event=False): """Assure that a service is configured on the BIGIP.""" start_time = time() LOG.debug("Starting assure_service") + # Needed also for delete events because of subnet hints self._assure_loadbalancer_created(service, all_subnet_hints) + # Create and update + if not delete_event: + self._assure_pools_created(service) - self._assure_listeners_created(service) + self._assure_listeners_created(service) - self._assure_pools_created(service) + self._assure_monitors_created(service) - self._assure_monitors(service) + self._assure_members_created(service, all_subnet_hints) - self._assure_members(service, all_subnet_hints) + self._assure_pools_configured(service) - self._assure_pools_deleted(service) + self._assure_l7policies_created(service) - self._assure_l7policies_created(service) + self._assure_l7rules_created(service) + else: # delete + self._assure_monitors_deleted(service) - self._assure_l7rules_created(service) + self._assure_members_deleted(service, all_subnet_hints) - self._assure_l7rules_deleted(service) + self._assure_l7rules_deleted(service) - self._assure_l7policies_deleted(service) + self._assure_l7policies_deleted(service) - self._assure_listeners_deleted(service) + self._assure_pools_deleted(service) - self._assure_loadbalancer_deleted(service) + self._assure_listeners_deleted(service) + + self._assure_loadbalancer_deleted(service) LOG.debug(" _assure_service took %.5f secs" % (time() - start_time)) return all_subnet_hints + @utils.instrument_execution_time def _assure_loadbalancer_created(self, service, all_subnet_hints): if 'loadbalancer' not in service: return @@ -107,7 +121,10 @@ def _assure_loadbalancer_created(self, service, all_subnet_hints): loadbalancer["network_id"], all_subnet_hints, False) + if loadbalancer['provisioning_status'] != plugin_const.PENDING_DELETE: + loadbalancer['provisioning_status'] = plugin_const.ACTIVE + @utils.instrument_execution_time def _assure_listeners_created(self, service): if 'listeners' not in service: return @@ -116,11 +133,23 @@ def _assure_listeners_created(self, service): loadbalancer = service["loadbalancer"] networks = service["networks"] bigips = self.driver.get_config_bigips() - + old_listener = service.get('old_listener') for listener in listeners: - svc = {"loadbalancer": loadbalancer, - "listener": listener, - "networks": networks} + if (old_listener != None and old_listener.get('id') == listener.get('id')): + svc = {"loadbalancer": loadbalancer, + "listener": listener, + "old_listener": old_listener, + "networks": networks} + else: + svc = {"loadbalancer": loadbalancer, + "listener": listener, + "networks": networks} + + default_pool_id = listener.get('default_pool_id', '') + if default_pool_id: + pool = self.get_pool_by_id(service, default_pool_id) + if pool: + svc['pool'] = pool if listener['provisioning_status'] == plugin_const.PENDING_UPDATE: try: @@ -128,6 +157,7 @@ def _assure_listeners_created(self, service): except Exception as err: loadbalancer['provisioning_status'] = plugin_const.ERROR listener['provisioning_status'] = plugin_const.ERROR + LOG.exception(err) raise f5_ex.VirtualServerUpdateException(err.message) elif listener['provisioning_status'] != \ @@ -142,6 +172,10 @@ def _assure_listeners_created(self, service): listener['provisioning_status'] = plugin_const.ERROR raise f5_ex.VirtualServerCreationException(err.message) + if listener['provisioning_status'] != plugin_const.PENDING_DELETE: + listener['provisioning_status'] = plugin_const.ACTIVE + + @utils.instrument_execution_time def _assure_pools_created(self, service): if "pools" not in service: return @@ -153,8 +187,7 @@ def _assure_pools_created(self, service): for pool in pools: if pool['provisioning_status'] != plugin_const.PENDING_DELETE: - svc = {"loadbalancer": loadbalancer, - "pool": pool} + svc = {"loadbalancer": loadbalancer, "pool": pool} svc['members'] = self._get_pool_members(service, pool['id']) try: @@ -163,34 +196,78 @@ def _assure_pools_created(self, service): plugin_const.PENDING_CREATE: self.pool_builder.create_pool(svc, bigips) else: - self.pool_builder.update_pool(svc, bigips) + try: + self.pool_builder.update_pool(svc, bigips) + except HTTPError as err: + if err.response.status_code == 404: + self.pool_builder.create_pool(svc, bigips) + + except HTTPError as err: + if err.response.status_code != 409: + pool['provisioning_status'] = plugin_const.ERROR + loadbalancer['provisioning_status'] = ( + plugin_const.ERROR) + raise f5_ex.PoolCreationException(err.message) + + except Exception as err: + pool['provisioning_status'] = plugin_const.ERROR + loadbalancer['provisioning_status'] = plugin_const.ERROR + raise f5_ex.PoolCreationException(err.message) + pool['provisioning_status'] = plugin_const.ACTIVE + + @utils.instrument_execution_time + def _assure_pools_configured(self, service): + if "pools" not in service: + return + + pools = service["pools"] + loadbalancer = service["loadbalancer"] + + bigips = self.driver.get_config_bigips() + + for pool in pools: + if pool['provisioning_status'] != plugin_const.PENDING_DELETE: + svc = {"loadbalancer": loadbalancer, "pool": pool} + svc['members'] = self._get_pool_members(service, pool['id']) + + try: # assign pool name to virtual pool_name = self.service_adapter.init_pool_name( loadbalancer, pool) # get associated listeners for pool for listener in pool['listeners']: - svc['listener'] = \ - self.get_listener_by_id(service, listener['id']) - self.listener_builder.update_listener_pool( - svc, pool_name["name"], bigips) + listener = self.get_listener_by_id(service, listener['id']) + + if listener: + svc['listener'] = listener + self.listener_builder.update_listener_pool( + svc, pool_name["name"], bigips) + # update virtual sever pool name, session persistence + self.listener_builder.update_session_persistence( + svc, bigips) + + # ccloud: update pool to set lb_method right + self.pool_builder.update_pool(svc, bigips) + + pool['provisioning_status'] = plugin_const.ACTIVE - # update virtual sever pool name, session persistence - self.listener_builder.update_session_persistence( - svc, bigips) except HTTPError as err: if err.response.status_code != 409: pool['provisioning_status'] = plugin_const.ERROR loadbalancer['provisioning_status'] = ( plugin_const.ERROR) - raise f5_ex.PoolCreationException(err.message) + LOG.exception(err) + raise f5_ex.PoolCreationException("ccloud: Error #1" + err.message) except Exception as err: pool['provisioning_status'] = plugin_const.ERROR loadbalancer['provisioning_status'] = plugin_const.ERROR - raise f5_ex.PoolCreationException(err.message) + LOG.exception(err) + raise f5_ex.PoolCreationException("ccloud: Error #2" + err.message) + @utils.instrument_execution_time def _get_pool_members(self, service, pool_id): '''Return a list of members associated with given pool.''' @@ -200,6 +277,7 @@ def _get_pool_members(self, service, pool_id): members.append(member) return members + @utils.instrument_execution_time def _update_listener_pool(self, service, listener_id, pool_name, bigips): listener = self.get_listener_by_id(service, listener_id) if listener is not None: @@ -213,7 +291,9 @@ def _update_listener_pool(self, service, listener_id, pool_name, bigips): listener['provisioning_status'] = plugin_const.ERROR raise f5_ex.VirtualServerUpdateException(err.message) - def _assure_monitors(self, service): + @utils.instrument_execution_time + def _assure_monitors_deleted(self, service): + if not (("pools" in service) and ("healthmonitors" in service)): return @@ -231,7 +311,22 @@ def _assure_monitors(self, service): except Exception as err: monitor['provisioning_status'] = plugin_const.ERROR raise f5_ex.MonitorDeleteException(err.message) - else: + + @utils.instrument_execution_time + def _assure_monitors_created(self, service): + + if not (("pools" in service) and ("healthmonitors" in service)): + return + + monitors = service["healthmonitors"] + loadbalancer = service["loadbalancer"] + bigips = self.driver.get_config_bigips() + + for monitor in monitors: + svc = {"loadbalancer": loadbalancer, + "healthmonitor": monitor, + "pool": self.get_pool_by_id(service, monitor["pool_id"])} + if monitor['provisioning_status'] != plugin_const.PENDING_DELETE: try: self.pool_builder.create_healthmonitor(svc, bigips) except HTTPError as err: @@ -247,7 +342,10 @@ def _assure_monitors(self, service): monitor['provisioning_status'] = plugin_const.ERROR raise f5_ex.MonitorCreationException(err.message) - def _assure_members(self, service, all_subnet_hints): + monitor['provisioning_status'] = plugin_const.ACTIVE + + @utils.instrument_execution_time + def _assure_members_created(self, service, all_subnet_hints): if not (("pools" in service) and ("members" in service)): return @@ -262,20 +360,15 @@ def _assure_members(self, service, all_subnet_hints): "pool": pool} if 'port' not in member and \ - member['provisioning_status'] != plugin_const.PENDING_DELETE: + member['provisioning_status'] != plugin_const.PENDING_DELETE: LOG.warning("Member definition does not include Neutron port") # delete member if pool is being deleted - if member['provisioning_status'] == plugin_const.PENDING_DELETE or\ - pool['provisioning_status'] == plugin_const.PENDING_DELETE: - try: - self.pool_builder.delete_member(svc, bigips) - except Exception as err: - member['provisioning_status'] = plugin_const.ERROR - raise f5_ex.MemberDeleteException(err.message) - else: + if not (member['provisioning_status'] == plugin_const.PENDING_DELETE or \ + pool['provisioning_status'] == plugin_const.PENDING_DELETE): try: self.pool_builder.create_member(svc, bigips) + member['provisioning_status'] = plugin_const.ACTIVE except HTTPError as err: if err.response.status_code != 409: # FIXME(RB) @@ -290,16 +383,54 @@ def _assure_members(self, service, all_subnet_hints): except Exception as err: member['provisioning_status'] = plugin_const.ERROR raise f5_ex.MemberUpdateException(err.message) + + except Exception as err: member['provisioning_status'] = plugin_const.ERROR raise f5_ex.MemberCreationException(err.message) - self._update_subnet_hints(member["provisioning_status"], - member["subnet_id"], - member["network_id"], - all_subnet_hints, - True) + self._update_subnet_hints(member["provisioning_status"], + member["subnet_id"], + member["network_id"], + all_subnet_hints, + True) + + @utils.instrument_execution_time + def _assure_members_deleted(self, service, all_subnet_hints): + if not (("pools" in service) and ("members" in service)): + return + + members = service["members"] + loadbalancer = service["loadbalancer"] + bigips = self.driver.get_config_bigips() + for member in members: + pool = self.get_pool_by_id(service, member["pool_id"]) + svc = {"loadbalancer": loadbalancer, + "member": member, + "pool": pool} + + if 'port' not in member and \ + member['provisioning_status'] != plugin_const.PENDING_DELETE: + LOG.warning("Member definition does not include Neutron port") + + # delete member if pool is being deleted + if member['provisioning_status'] == plugin_const.PENDING_DELETE or \ + pool['provisioning_status'] == plugin_const.PENDING_DELETE: + try: + self.pool_builder.delete_member(svc, bigips) + except Exception as err: + member['provisioning_status'] = plugin_const.ERROR + raise f5_ex.MemberDeleteException(err.message) + + self._update_subnet_hints(member["provisioning_status"], + member["subnet_id"], + member["network_id"], + all_subnet_hints, + True) + + + @utils.instrument_execution_time def _assure_loadbalancer_deleted(self, service): if (service['loadbalancer']['provisioning_status'] != plugin_const.PENDING_DELETE): @@ -320,6 +451,7 @@ def _assure_loadbalancer_deleted(self, service): for bigip in bigips: vip_address.assure(bigip, delete=True) + @utils.instrument_execution_time def _assure_pools_deleted(self, service): if 'pools' not in service: return @@ -335,7 +467,6 @@ def _assure_pools_deleted(self, service): "pool": pool} try: - # update listeners for pool for listener in pool['listeners']: svc['listener'] = \ @@ -355,6 +486,7 @@ def _assure_pools_deleted(self, service): pool['provisioning_status'] = plugin_const.ERROR raise f5_ex.PoolDeleteException(err.message) + @utils.instrument_execution_time def _assure_listeners_deleted(self, service): if 'listeners' not in service: return @@ -367,6 +499,14 @@ def _assure_listeners_deleted(self, service): if listener['provisioning_status'] == plugin_const.PENDING_DELETE: svc = {"loadbalancer": loadbalancer, "listener": listener} + # ccloud: try to delete persistence which might be attached to listener + # ignore errors, persistence might be used somewhere else if pool is used more than once as default + try: + self.listener_builder.remove_session_persistence( + svc, bigips) + except Exception: + pass + # delete the listener try: self.listener_builder.delete_listener(svc, bigips) except Exception as err: @@ -387,7 +527,7 @@ def _check_monitor_delete(service): @staticmethod def get_pool_by_id(service, pool_id): - if "pools" in service: + if pool_id and "pools" in service: pools = service["pools"] for pool in pools: if pool["id"] == pool_id: @@ -423,6 +563,7 @@ def _update_subnet_hints(self, status, subnet_id, 'subnet_id': subnet_id, 'is_for_member': is_member} + @utils.instrument_execution_time def listener_exists(self, bigip, service): """Test the existence of the listener defined by service.""" try: @@ -435,6 +576,7 @@ def listener_exists(self, bigip, service): return True + @utils.instrument_execution_time def _assure_l7policies_created(self, service): if 'l7policies' not in service: return @@ -446,13 +588,7 @@ def _assure_l7policies_created(self, service): try: name = l7policy.get('name', None) if name and self.is_esd(name): - listener = self.get_listener_by_id( - service, l7policy.get('listener_id', '')) - - svc = {"loadbalancer": service["loadbalancer"], - "listener": listener} - esd = self.get_esd(name) - self.listener_builder.apply_esd(svc, esd, bigips) + continue else: self.l7service.create_l7policy( l7policy, service, bigips) @@ -462,6 +598,9 @@ def _assure_l7policies_created(self, service): plugin_const.ERROR raise f5_ex.L7PolicyCreationException(err.message) + l7policy['provisioning_status'] = plugin_const.ACTIVE + + @utils.instrument_execution_time def _assure_l7policies_deleted(self, service): if 'l7policies' not in service: return @@ -473,19 +612,7 @@ def _assure_l7policies_deleted(self, service): try: name = l7policy.get('name', None) if name and self.is_esd(name): - listener = self.get_listener_by_id( - service, l7policy.get('listener_id', '')) - svc = {"loadbalancer": service["loadbalancer"], - "listener": listener} - - # pool is needed to reset session persistence - if listener['default_pool_id']: - pool = self.get_pool_by_id( - service, listener.get('default_pool_id', '')) - if pool: - svc['pool'] = pool - esd = self.get_esd(name) - self.listener_builder.remove_esd(svc, esd, bigips) + continue else: # Note: use update_l7policy because a listener can have # multiple policies @@ -497,6 +624,7 @@ def _assure_l7policies_deleted(self, service): plugin_const.ERROR raise f5_ex.L7PolicyDeleteException(err.message) + @utils.instrument_execution_time def _assure_l7rules_created(self, service): if 'l7policy_rules' not in service: return @@ -522,6 +650,8 @@ def _assure_l7rules_created(self, service): plugin_const.ERROR raise f5_ex.L7PolicyCreationException(err.message) + l7rule['provisioning_status'] = plugin_const.ACTIVE + @utils.instrument_execution_time def _assure_l7rules_deleted(self, service): if 'l7policy_rules' not in service: return @@ -544,7 +674,7 @@ def _assure_l7rules_deleted(self, service): service['loadbalancer']['provisioning_status'] = \ plugin_const.ERROR raise f5_ex.L7PolicyDeleteException(err.message) - + @utils.instrument_execution_time def get_listener_stats(self, service, stats): """Get statistics for a loadbalancer service. @@ -578,7 +708,7 @@ def get_listener_stats(self, service, stats): collected_stats[stat] += vs_stats[stat] return collected_stats - + @utils.instrument_execution_time def update_operating_status(self, service): bigip = self.driver.get_active_bigip() loadbalancer = service["loadbalancer"] @@ -644,4 +774,4 @@ def get_esd(self, name): return None def is_esd(self, name): - return self.esd.get_esd(name) is not None + return self.esd.get_esd(name) is not None \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_driver.py b/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_driver.py index a5738cc0e..becfad643 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_driver.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_driver.py @@ -15,154 +15,188 @@ class LBaaSBaseDriver(object): - """Abstract base LBaaS Driver class for interfacing with Agent Manager """ + """Abstract base LBaaS Driver class for interfacing with Agent Manager.""" def __init__(self, conf): # XXX 'conf' appears to be unused - '''Maybe we can remove this method altogether? Or maybe it's for future + """Maybe we can remove this method altogether? Or maybe it's for future. subclassing... - ''' + """ self.agent_id = None self.plugin_rpc = None # XXX overridden in the only known subclass self.connected = False # XXX overridden in the only known subclass self.service_queue = [] + self.queues = {} self.agent_configurations = {} # XXX overridden in subclass def set_context(self, context): - """Set the global context object for the lbaas driver """ + """Set the global context object for the lbaas driver.""" raise NotImplementedError() - def post_init(self): - """Run after agent is fully connected """ - raise NotImplementedError() + def set_plugin_rpc(self, plugin_rpc): + """Provide LBaaS Plugin RPC access.""" - def set_tunnel_rpc(self, tunnel_rpc): # XXX into this class? - """Provide FDB Connector RPC access """ + def set_agent_report_state(self, report_state_callback): + """Set Agent Report State.""" raise NotImplementedError() - def set_l2pop_rpc(self, l2pop_rpc): - """Provide FDB Connector with L2 Population RPC access """ + def set_tunnel_rpc(self, tunnel_rpc): + """Provide FDB Connector RPC access.""" raise NotImplementedError() - def connect(self): - """Connect backend API endpoints """ + def set_l2pop_rpc(self, l2pop_rpc): + """Provide FDB Connector with L2 Population RPC access.""" raise NotImplementedError() def flush_cache(self): - """Remove all cached items """ + """Remove all cached items.""" raise NotImplementedError() + def backend_integrity(self): + """Return True, if the agent is be considered viable for services.""" + raise NotImplemented() + def backup_configuration(self): - """Persist backend configuratoins """ + """Persist backend configuratoins.""" raise NotImplementedError() - def get_stats(self, service): - """Get Stats for a loadbalancer Service """ - raise NotImplementedError() + def generate_capacity_score(self, capacity_policy): + """Generate the capacity score of connected devices.""" + raise NotImplemented - def exists(self, service): - """Check If LBaaS Service is Defined on Driver Target """ - raise NotImplementedError() + def update_operating_status(self): + """Update pool member operational status from devices to controller.""" + raise NotImplemented - def sync(self, service): - """Force Sync a Service on Driver Target """ - raise NotImplementedError() + def recover_errored_devices(self): + """Trigger attempt to reconnect any errored devices.""" + raise NotImplemented - def remove_orphans(self, known_services): - """Remove Unknown Service from Driver Target """ + def get_stats(self, service): + """Get Stats for a loadbalancer Service.""" raise NotImplementedError() - def create_vip(self, vip, service): - """LBaaS Create VIP """ - raise NotImplementedError() + def get_all_deployed_loadbalancers(self, purge_orphaned_folders=True): + """Get all Loadbalancers defined on devices.""" + raise NotImplemented + + def purge_orphaned_loadbalancer(self, tenant_id, loadbalancer_id, + hostnames): + """Remove all loadbalancers without references in Neutron.""" + raise NotImplemented - def update_vip(self, old_vip, vip, service): - """LBaaS Update VIP """ + def service_exists(self, service): + """Check If LBaaS Service is Defined on Driver Target.""" raise NotImplementedError() - def delete_vip(self, vip, service): - """LBaaS Delete VIP """ + def sync(self, service): + """Force Sync a Service on Driver Target.""" raise NotImplementedError() def create_pool(self, pool, service): - """LBaaS Delete VIP """ + """LBaaS Create Pool.""" raise NotImplementedError() def update_pool(self, old_pool, pool, service): - """LBaaS Update Pool """ + """LBaaS Update Pool.""" raise NotImplementedError() def delete_pool(self, pool, service): - """LBaaS Delete Pool """ + """LBaaS Delete Pool.""" raise NotImplementedError() def create_member(self, member, service): - """LBaaS Create Member """ + """LBaaS Create Member.""" raise NotImplementedError() def update_member(self, old_member, member, service): - """LBaaS Update Member """ + """LBaaS Update Member.""" raise NotImplementedError() def delete_member(self, member, service): - """LBaaS Delete Member """ + """LBaaS Delete Member.""" raise NotImplementedError() def create_pool_health_monitor(self, health_monitor, pool, service): - """LBaaS Create Pool Health Monitor """ + """LBaaS Create Pool Health Monitor.""" raise NotImplementedError() def update_health_monitor(self, old_health_monitor, health_monitor, pool, service): - """LBaaS Update Health Monitor """ + """LBaaS Update Health Monitor.""" + raise NotImplementedError() + + def delete_health_monitor(self, health_monitor, pool, service): + """LBaaS Delete Health Monior.""" raise NotImplementedError() def delete_pool_health_monitor(self, health_monitor, pool, service): - """LBaaS Delete Health Monitor """ + """LBaaS Delete Health Monitor.""" + raise NotImplementedError() + + def get_all_deployed_health_monitors(self): + """Get listing of all deployed Health Monitors""" + raise NotImplementedError() + + def purge_orphaned_health_monitor(self, tenant_id=None, monitor_id=None, + hostnames=list()): + """LBaaS Purge Health Monitor.""" + raise NotImplementedError() + + def get_all_deployed_l7_policys(self): + """Get listing of all deployed Health Monitors""" + raise NotImplementedError() + + def purge_orphaned_l7_policy(self, tenant_id=None, monitor_id=None, + hostnames=list()): + """LBaaS Purge Health Monitor.""" raise NotImplementedError() def tunnel_update(self, **kwargs): - """Neutron Core Tunnel Update """ + """Neutron Core Tunnel Update.""" raise NotImplementedError() def tunnel_sync(self): - """Neutron Core Tunnel Sync Messages """ + """Neutron Core Tunnel Sync Messages.""" raise NotImplementedError() def fdb_add(self, fdb_entries): - """L2 Population FDB Add """ + """L2 Population FDB Add.""" raise NotImplementedError() def fdb_remove(self, fdb_entries): - """L2 Population FDB Remove """ + """L2 Population FDB Remove.""" raise NotImplementedError() def fdb_update(self, fdb_entries): - """L2 Population FDB Update """ + """L2 Population FDB Update.""" raise NotImplementedError() def create_l7policy(self, l7policy, service): - """LBaaS Create l7policy """ + """LBaaS Create l7policy.""" raise NotImplementedError() def update_l7policy(self, old_l7policy, l7policy, service): - """LBaaS Update l7policy """ + """LBaaS Update l7policy.""" raise NotImplementedError() def delete_l7policy(self, l7policy, service): - """LBaaS Delete l7policy """ + """LBaaS Delete l7policy.""" raise NotImplementedError() def create_l7rule(self, l7rule, service): - """LBaaS Create l7rule """ + """LBaaS Create l7rule.""" raise NotImplementedError() def update_l7rule(self, old_l7rule, l7rule, service): - """LBaaS Update l7rule """ + """LBaaS Update l7rule.""" raise NotImplementedError() def delete_l7rule(self, l7rule, service): - """LBaaS Delete l7rule """ + """LBaaS Delete l7rule.""" + raise NotImplementedError() + + def get_orphans_cache(self): raise NotImplementedError() diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/listener_adapter.py b/f5_openstack_agent/lbaasv2/drivers/bigip/listener_adapter.py index 596862c04..a97bded18 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/listener_adapter.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/listener_adapter.py @@ -25,6 +25,6 @@ def translate(self, service, listener, l7policy=None): listener.get('tenant_id', ''))} if l7policy: - f5_vs['l7policy_name'] = "wrapper_policy" + f5_vs['l7policy_name'] = "wrapper_policy_"+listener.get('id','') return f5_vs diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/listener_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/listener_service.py index 48c1ea9cb..9e103adf4 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/listener_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/listener_service.py @@ -14,25 +14,33 @@ # limitations under the License. # +#import pdb + from oslo_log import log as logging +from neutron.plugins.common import constants as plugin_const + from f5_openstack_agent.lbaasv2.drivers.bigip import resource_helper from f5_openstack_agent.lbaasv2.drivers.bigip import ssl_profile from neutron_lbaas.services.loadbalancer import constants as lb_const from requests import HTTPError +from f5_openstack_agent.lbaasv2.drivers.bigip import utils + LOG = logging.getLogger(__name__) class ListenerServiceBuilder(object): - u"""Create LBaaS v2 Listener on BIG-IP®s. + u"""Create LBaaS v2 Listener on BIG-IPs. Handles requests to create, update, delete LBaaS v2 listener - objects on one or more BIG-IP® systems. Maps LBaaS listener - defined in service object to a BIG-IP® virtual server. + objects on one or more BIG-IP systems. Maps LBaaS listener + defined in service object to a BIG-IP virtual server. """ - def __init__(self, service_adapter, cert_manager, parent_ssl_profile=None): + def __init__(self, lbaas_builder, service_adapter, cert_manager, parent_ssl_profile=None): + + self.lbaas_builder = lbaas_builder self.cert_manager = cert_manager self.parent_ssl_profile = parent_ssl_profile self.vs_helper = resource_helper.BigIPResourceHelper( @@ -41,10 +49,11 @@ def __init__(self, service_adapter, cert_manager, parent_ssl_profile=None): LOG.debug("ListenerServiceBuilder: using parent_ssl_profile %s ", parent_ssl_profile) + def create_listener(self, service, bigips): - u"""Create listener on set of BIG-IP®s. + u"""Create listener on set of BIG-IPs. - Create a BIG-IP® virtual server to represent an LBaaS + Create a BIG-IP virtual server to represent an LBaaS Listener object. :param service: Dictionary which contains a both a listener @@ -60,22 +69,42 @@ def create_listener(self, service, bigips): service['listener']['operating_status'] = lb_const.ONLINE network_id = service['loadbalancer']['network_id'] + error = None for bigip in bigips: self.service_adapter.get_vlan(vip, bigip, network_id) try: self.vs_helper.create(bigip, vip) except HTTPError as err: if err.response.status_code == 409: - LOG.debug("Virtual server already exists") + LOG.debug("Virtual server already exists updating") + try: + self.update_listener(service, [bigip]) + #self.vs_helper.update(bigip, vip) + except Exception as e: + LOG.warn("Update triggered in create failed, this could be due to timing issues in assure_service") + LOG.warn('VS info %s',service['listener']) + LOG.exception(e) + LOG.warn('Exception %s',e) + raise e else: LOG.exception("Virtual server creation error: %s" % err.message) raise if tls: - self.add_ssl_profile(tls, bigip) + # Don't stop processing in case of errors. Otherwise the other F5's won't get the same vs + try: + self.add_ssl_profile(tls, bigip) + except Exception as err: + LOG.error("Error adding SSL Profile to listener: {0}".format(err)) + error = err if error is None else error + + if error: + service['listener']['provisioning_status'] = 'ERROR' + raise error + def get_listener(self, service, bigip): - u"""Retrieve BIG-IP® virtual from a single BIG-IP® system. + u"""Retrieve BIG-IP virtual from a single BIG-IP system. :param service: Dictionary which contains a both a listener and load balancer definition. @@ -88,7 +117,7 @@ def get_listener(self, service, bigip): return obj def delete_listener(self, service, bigips): - u"""Delete Listener from a set of BIG-IP® systems. + u"""Delete Listener from a set of BIG-IP systems. Delete virtual server that represents a Listener object. @@ -101,36 +130,60 @@ def delete_listener(self, service, bigips): if tls: tls['name'] = vip['name'] tls['partition'] = vip['partition'] - + error = None for bigip in bigips: self.vs_helper.delete(bigip, name=vip["name"], partition=vip["partition"]) # delete ssl profiles - self.remove_ssl_profiles(tls, bigip) + # Don't stop processing in case of errors. Otherwise the other F5's might have a different configuration + try: + self.remove_ssl_profiles(tls, bigip) + except Exception as err: + LOG.error("Error adding SSL Profile to listener: {0}".format(err)) + error = err if error is None else error + + if error: + raise error - def add_ssl_profile(self, tls, bigip): + def add_ssl_profile(self, tls, bigip, add_to_vip=True): # add profile to virtual server vip = {'name': tls['name'], 'partition': tls['partition']} - + error = None if "default_tls_container_id" in tls: container_ref = tls["default_tls_container_id"] self.create_ssl_profile( - container_ref, bigip, vip, True) + container_ref, bigip, vip, True, add_to_vip) if "sni_containers" in tls and tls["sni_containers"]: for container in tls["sni_containers"]: - container_ref = container["tls_container_id"] - self.create_ssl_profile(container_ref, bigip, vip, False) + try: + container_ref = container["tls_container_id"] + self.create_ssl_profile(container_ref, bigip, vip, False, add_to_vip) + except Exception as err: + LOG.error("Error creating SSL Profile for listener: {0}".format(err)) + error = err if error is None else error + if error: + raise error + - def create_ssl_profile(self, container_ref, bigip, vip, sni_default=False): + def create_ssl_profile(self, container_ref, bigip, vip, sni_default=False, add_to_vip=True): cert = self.cert_manager.get_certificate(container_ref) key = self.cert_manager.get_private_key(container_ref) + key_passphrase = self.cert_manager.get_private_key_passphrase(container_ref) + intermediate = self.cert_manager.get_intermediates(container_ref) name = self.cert_manager.get_name(container_ref, self.service_adapter.prefix) + container = self.cert_manager.get_container(container_ref) + caClientTrust = bool(container.name and container.name.startswith('CATrust')) + ex = None + error_default = False + error_not_default = False + # Add 2 client ssl profiles. Try to add both even if first one has errors + # try: # upload cert/key and create SSL profile ssl_profile.SSLProfileHelper.create_client_ssl_profile( @@ -138,17 +191,58 @@ def create_ssl_profile(self, container_ref, bigip, vip, sni_default=False): name, cert, key, - sni_default=sni_default, - parent_profile=self.parent_ssl_profile) + intermediate, + sni_default=True, + parent_profile=self.parent_ssl_profile, + caClientTrust=caClientTrust, + key_passphrase=key_passphrase + ) + except HTTPError as err: + if err.response.status_code != 409: + ex = err + error_default = True + except Exception as e: + ex = e + error_default = True + + try: + # upload cert/key and create SSL profile + ssl_profile.SSLProfileHelper.create_client_ssl_profile( + bigip, + name, + cert, + key, + intermediate, + sni_default=False, + parent_profile=self.parent_ssl_profile, + caClientTrust=caClientTrust, + key_passphrase=key_passphrase + ) + except HTTPError as err: + if err.response.status_code != 409: + ex = err + error_not_default = True + except Exception as e: + ex = e + error_not_default = True finally: del cert del key + del intermediate # add ssl profile to virtual server - self._add_profile(vip, name, bigip, context='clientside') + if add_to_vip: + f5name = name + if sni_default and not error_default: + self._add_profile(vip, f5name, bigip, context='clientside') + elif not sni_default and not error_not_default: + f5name += '_NotDefault' + self._add_profile(vip, f5name, bigip, context='clientside') + if ex: + raise ex def update_listener(self, service, bigips): - u"""Update Listener from a single BIG-IP® system. + u"""Update Listener from a single BIG-IP system. Updates virtual servers that represents a Listener object. @@ -156,10 +250,107 @@ def update_listener(self, service, bigips): and load balancer definition. :param bigips: Array of BigIP class instances to update. """ + + u""" + ATTENTION: The hole impl. is a hack. + For ssl profile settings the order is very important: + 1. A new ssl profile is created but not applied to the listener + 2. The esd_apply configures the listener with the new profile (so the old one will be detached) + 3. The update will apply the changes to the listener + 4. The remove_ssl is than be able to remove unneeded ssl profiles because they got detached in 3. + """ + + # check for ssl client cert changes + old_default = None + old_sni_containers = None + new_default = None + new_sni_containers = None vip = self.service_adapter.get_virtual(service) + #pdb.set_trace() + + listener = service.get('listener') + if listener.get('protocol') == 'TERMINATED_HTTPS': + old_listener = service.get('old_listener') + if old_listener != None: + listener = service.get('listener') + if old_listener.get('default_tls_container_id') != listener.get('default_tls_container_id'): + old_default = old_listener.get('default_tls_container_id') + new_default = listener.get('default_tls_container_id') + + # determine sni delta with set substraction + old_snis = old_listener.get('sni_containers') + new_snis = listener.get('sni_containers') + old_ids = [] + new_ids = [] + for old in old_snis: + old_ids.append(old.get('tls_container_id')) + for new in new_snis: + new_ids.append(new.get('tls_container_id')) + new_sni_containers = self._make_sni_tls(vip, list(set(new_ids) - set(old_ids))) + old_sni_containers = self._make_sni_tls(vip, list(set(old_ids) - set(new_ids))) + + # create old and new tls listener configurations + # create new ssl-profiles on F5 BUT DO NOT APPLY them to listener + old_tls = None + if (new_default != None or (new_sni_containers != None and new_sni_containers['sni_containers'])): + new_tls = self.service_adapter.get_tls(service) + new_tls = self._make_default_tls(vip, new_tls.get('default_tls_container_id')) + + if old_default != None: + old_tls = self._make_default_tls(vip, old_default) + + for bigip in bigips: + # create ssl profile but do not apply + if new_tls != None: + try: + self.add_ssl_profile(new_tls, bigip, False) + except: + pass + if new_sni_containers != None and new_sni_containers['sni_containers']: + try: + self.add_ssl_profile(new_sni_containers, bigip, False) + except: + pass + + + # process esd's AND create new client ssl config for listener + self.apply_esds(service, vip) + + # apply changes to listener AND remove not needed ssl profiles on F5 + error = None + network_id = service['loadbalancer']['network_id'] for bigip in bigips: - self.vs_helper.update(bigip, vip) + self.service_adapter.get_vlan(vip, bigip, network_id) + try: + self.vs_helper.update(bigip, vip) + except Exception as err: + LOG.error("Error changing listener: {0}".format(err)) + error = err if error is None else error + # delete ssl profiles + if listener.get('protocol') == 'TERMINATED_HTTPS': + if old_tls != None: + try: + self.remove_ssl_profiles(old_tls, bigip) + except: + pass + if old_sni_containers != None and old_sni_containers['sni_containers']: + try: + self.remove_ssl_profiles(old_sni_containers, bigip) + except: + pass + + if error: + raise error + + def _make_default_tls(self, vip, id): + return {'name': vip['name'], 'partition': vip['partition'], 'default_tls_container_id': id} + + def _make_sni_tls(self, vip, ids): + containers = {'name': vip['name'], 'partition': vip['partition'], 'sni_containers': []} + for id in ids: + containers['sni_containers'].append({'tls_container_id': id}) + return containers def update_listener_pool(self, service, name, bigips): """Update virtual server's default pool attribute. @@ -200,17 +391,21 @@ def update_session_persistence(self, service, bigips): persistence = pool['session_persistence'] persistence_type = persistence['type'] vip_persist = self.service_adapter.get_session_persistence(service) - listener = service['listener'] for bigip in bigips: # For TCP listeners, must remove fastL4 profile before adding # adding http/oneconnect profiles. - if persistence_type != 'SOURCE_IP': - if listener['protocol'] == 'TCP': - self._remove_profile(vip, 'fastL4', bigip) + # if persistence_type != 'SOURCE_IP': + # #if listener['protocol'] == 'TCP': + # #self._remove_profile(vip, 'fastL4', bigip) + # + # # Add default profiles + # + # profiles = utils.get_default_profiles(self.service_adapter.conf, listener['protocol']) + # + # + # for profile in profiles.values(): + # self._add_profile(vip, profile.get('name'), bigip) - # HTTP listeners should have http and oneconnect profiles - self._add_profile(vip, 'http', bigip) - self._add_profile(vip, 'oneconnect', bigip) if persistence_type == 'APP_COOKIE' and \ 'cookie_name' in persistence: @@ -293,6 +488,66 @@ def _create_app_cookie_persist_rule(self, cookiename): rule_text += "}\n\n" return rule_text + def _cc_create_app_cookie_persist_rule(self, cookiename): + """Create cookie persistence rule. + + :param cookiename: Name to substitute in rule. + """ + rule_text = """ + when RULE_INIT { + + # Cookie name prefix + set static::ck_pattern BIGipServer*, %s + + # Log debug to /var/log/ltm? 1=yes, 0=no) + set static::ck_debug 1 + + # Cookie encryption passphrase + # Change this to a custom string! + set static::ck_pass "abc123" + } + when HTTP_REQUEST { + + if {$static::ck_debug}{log local0. "Request cookie names: [HTTP::cookie names]"} + + # Check if the cookie names in the request match our string glob pattern + if {[set cookie_names [lsearch -all -inline [HTTP::cookie names] $static::ck_pattern]] ne ""}{ + + # We have at least one match so loop through the cookie(s) by name + if {$static::ck_debug}{log local0. "Matching cookie names: [HTTP::cookie names]"} + foreach cookie_name $cookie_names { + + # Decrypt the cookie value and check if the decryption failed (null return value) + if {[HTTP::cookie decrypt $cookie_name $static::ck_pass] eq ""}{ + + # Cookie wasn't encrypted, delete it + if {$static::ck_debug}{log local0. "Removing cookie as decryption failed for $cookie_name"} + HTTP::cookie remove $cookie_name + } + } + if {$static::ck_debug}{log local0. "Cookie header(s): [HTTP::header values Cookie]"} + } + } + when HTTP_RESPONSE { + + if {$static::ck_debug}{log local0. "Response cookie names: [HTTP::cookie names]"} + + # Check if the cookie names in the request match our string glob pattern + if {[set cookie_names [lsearch -all -inline [HTTP::cookie names] $static::ck_pattern]] ne ""}{ + + # We have at least one match so loop through the cookie(s) by name + if {$static::ck_debug}{log local0. "Matching cookie names: [HTTP::cookie names]"} + foreach cookie_name $cookie_names { + + # Encrypt the cookie value + HTTP::cookie encrypt $cookie_name $static::ck_pass + } + if {$static::ck_debug}{log local0. "Set-Cookie header(s): [HTTP::header values Set-Cookie]"} + } + } + """ % (cookiename) + return rule_text + def remove_session_persistence(self, service, bigips): """Resest persistence for virtual server instance. @@ -312,7 +567,7 @@ def remove_session_persistence(self, service, bigips): # Revert VS back to fastL4. Must do an update to replace # profiles instead of using add/remove profile. Leave http # profiles in place for non-TCP listeners. - vip['profiles'] = ['/Common/fastL4'] + vip['profiles'] = ['/Common/cc_fastL4'] for bigip in bigips: # Check for custom app_cookie profile. @@ -341,6 +596,8 @@ def remove_ssl_profiles(self, tls, bigip): i = container_ref.rindex("/") + 1 name = self.service_adapter.prefix + container_ref[i:] self._remove_ssl_profile(name, bigip) + self._remove_ssl_profile(name + '_NotDefault', bigip) + if "sni_containers" in tls and tls["sni_containers"]: for container in tls["sni_containers"]: @@ -348,6 +605,7 @@ def remove_ssl_profiles(self, tls, bigip): i = container_ref.rindex("/") + 1 name = self.service_adapter.prefix + container_ref[i:] self._remove_ssl_profile(name, bigip) + self._remove_ssl_profile(name + '_NotDefault', bigip) def _remove_ssl_profile(self, name, bigip): """Delete profile. @@ -359,14 +617,14 @@ def _remove_ssl_profile(self, name, bigip): ssl_client_profile = bigip.tm.ltm.profile.client_ssls.client_ssl if ssl_client_profile.exists(name=name, partition='Common'): obj = ssl_client_profile.load(name=name, partition='Common') - obj.delete() + if obj: + obj.delete() + LOG.info("ccloud: SSL Profile deleted: %s" % name) except Exception as err: # Not necessarily an error -- profile might be referenced # by another virtual server. - LOG.warn( - "Unable to delete profile %s. " - "Response message: %s." % (name, err.message)) + LOG.warn("Unable to delete profile %s . Response message: %s ." % (name, err.message)) def _remove_profile(self, vip, profile_name, bigip): """Delete profile. @@ -558,108 +816,170 @@ def _remove_irule(self, vs, irule_name, bigip, rule_partition='Common'): LOG.debug("Removed iRule {0} for virtual sever {1}". format(irule_name, vs_name)) - def apply_esd(self, svc, esd, bigips): - profiles = [] + def apply_esds(self, service, vip): + + + listener = service['listener'] + default_profiles = utils.get_default_profiles(self.service_adapter.conf, listener['protocol']) + l7policies = listener.get('l7_policies') + + if l7policies is None: + return + fastl4 = {'partition':'Common','name':'cc_fastL4','context':'all'} + stcp_profiles = [] + ctcp_profiles = [] + cssl_profiles = [] + sssl_profiles = [] + http_profile = {} + oneconnect_profile = default_profiles.get('oneconnect') + compression_profile = {} + persistence_profiles = [] + + policies = [] + irules = [] # get virtual server name - update_attrs = self.service_adapter.get_virtual_name(svc) - - # start with server tcp profile - if 'lbaas_stcp' in esd: - # set serverside tcp profile - profiles.append({'name': esd['lbaas_stcp'], - 'partition': 'Common', - 'context': 'serverside'}) - # restrict client profile - ctcp_context = 'clientside' - else: - # no serverside profile; use client profile for both - ctcp_context = 'all' - # must define client profile; default to tcp if not in ESD - if 'lbaas_ctcp' in esd: - ctcp_profile = esd['lbaas_ctcp'] + # get ssl certificates for listener + tls = self.service_adapter.get_tls(service) + # initialize client ssl profile with already existing certificates + if bool(tls): + if "default_tls_container_id" in tls: + container_ref = tls["default_tls_container_id"] + def_name = self.cert_manager.get_name(container_ref, + self.service_adapter.prefix) + cssl_profiles.append({'name': def_name, + 'partition': 'Common', + 'context': 'clientside'}) + + if "sni_containers" in tls and tls["sni_containers"]: + for container in tls["sni_containers"]: + if 'tls_container_id' in container: + sni_ref = container['tls_container_id'] + sni_name = self.cert_manager.get_name(sni_ref, + self.service_adapter.prefix) + '_NotDefault' + cssl_profiles.append({'name': sni_name, + 'partition': 'Common', + 'context': 'clientside'}) + + + #pdb.set_trace() + for l7policy in l7policies: + name = l7policy.get('name', None) + if name and self.lbaas_builder.is_esd(name) and l7policy.get('provisioning_status')!= plugin_const.PENDING_DELETE: + esd = self.lbaas_builder.get_esd(name) + if esd is not None: + + # start with server tcp profile, only add if not already got some + ctcp_context = 'all' + + if 'lbaas_fastl4' in esd: + if esd['lbaas_fastl4']=='': + fastl4= {} + else: + fastl4 = {'partition': 'Common', 'name': esd['lbaas_fastl4'], 'context': 'all'} + + if len(stcp_profiles)==0: + if 'lbaas_stcp' in esd: + # set serverside tcp profile + stcp_profiles.append({'name': esd['lbaas_stcp'], + 'partition': 'Common', + 'context': 'serverside'}) + # restrict client profile + ctcp_context = 'clientside' + + if len(ctcp_profiles)==0: + # must define client profile; default to tcp if not in ESD + if 'lbaas_ctcp' in esd: + ctcp_profile = esd['lbaas_ctcp'] + else: + ctcp_profile = 'tcp' + ctcp_profiles.append({'name': ctcp_profile, + 'partition': 'Common', + 'context': ctcp_context}) + # http profiles + if 'lbaas_http' in esd and not bool(http_profile): + if esd['lbaas_http'] == '': + http_profile = {} + else: + http_profile = {'name': esd['lbaas_http'], + 'partition': 'Common', + 'context': 'all'} + + # one connect profiles if not already set + if 'lbaas_one_connect' in esd: + if esd['lbaas_one_connect'] == '': + oneconnect_profile = {} + else: + oneconnect_profile = {'name': esd['lbaas_one_connect'], + 'partition': 'Common', + 'context': 'all'} + + # http compression profiles + if 'lbaas_http_compression' in esd and not bool(compression_profile): + if esd['lbaas_http_compression'] == '': + compression_profile = {} + else: + compression_profile = {'name': esd['lbaas_http_compression'], + 'partition': 'Common', + 'context': 'all'} + + # SSL profiles + if 'lbaas_cssl_profile' in esd: + cssl_profiles.append({'name': esd['lbaas_cssl_profile'], + 'partition': 'Common', + 'context': 'clientside'}) + if 'lbaas_sssl_profile' in esd: + sssl_profiles.append({'name': esd['lbaas_sssl_profile'], + 'partition': 'Common', + 'context': 'serverside'}) + + # persistence + if 'lbaas_persist' in esd: + vip['persist'] = [{'name': esd['lbaas_persist']}] + if 'lbaas_fallback_persist' in esd: + vip['fallbackPersistence'] = esd['lbaas_fallback_persist'] + + # iRules + if 'lbaas_irule' in esd: + for irule in esd['lbaas_irule']: + irules.append('/Common/' + irule) + + # L7 policies + if 'lbaas_policy' in esd: + for policy in esd['lbaas_policy']: + policies.append({'name': policy, 'partition': 'Common'}) + + profiles=[] + + if listener['protocol'] == lb_const.PROTOCOL_TCP: + if bool(fastl4): + profiles.append(fastl4) + oneconnect_profile = None + else: + profiles = stcp_profiles + ctcp_profiles + + if bool(http_profile): + profiles.append(http_profile) else: - ctcp_profile = 'tcp' - profiles.append({'name': ctcp_profile, - 'partition': 'Common', - 'context': ctcp_context}) - - # SSL profiles - if 'lbaas_cssl_profile' in esd: - profiles.append({'name': esd['lbaas_cssl_profile'], - 'partition': 'Common', - 'context': 'clientside'}) - if 'lbaas_sssl_profile' in esd: - profiles.append({'name': esd['lbaas_sssl_profile'], - 'partition': 'Common', - 'context': 'serverside'}) - - # persistence - if 'lbaas_persist' in esd: - update_attrs['persist'] = [{'name': esd['lbaas_persist']}] - if 'lbaas_fallback_persist' in esd: - update_attrs['fallbackPersistence'] = esd['lbaas_fallback_persist'] - - if profiles: - # always use http and oneconnect - profiles.append({'name': 'http', - 'partition': 'Common', - 'context': 'all'}) - profiles.append({'name': 'oneconnect', - 'partition': 'Common', - 'context': 'all'}) - update_attrs['profiles'] = profiles - - # iRules - if 'lbaas_irule' in esd: - irules = [] - for irule in esd['lbaas_irule']: - irules.append('/Common/' + irule) - update_attrs['rules'] = irules - - # L7 policies - if 'lbaas_policy' in esd: - policies = [] - for policy in esd['lbaas_policy']: - policies.append({'name': policy, 'partition': 'Common'}) - update_attrs['policies'] = policies - - # udpate BIG-IPs - for bigip in bigips: - self.vs_helper.update(bigip, update_attrs) + if listener['protocol'] != lb_const.PROTOCOL_TCP: + profiles.append( default_profiles['http']) - def remove_esd(self, svc, esd, bigips): - # original service object definition of listener - vs = self.service_adapter.get_virtual(svc) + if bool(cssl_profiles): + for cssl_profile in cssl_profiles: + profiles.append(cssl_profile) - # add back SSL profile for TLS? - tls = self.service_adapter.get_tls(svc) - if tls: - tls['name'] = vs['name'] - tls['partition'] = vs['partition'] + if bool(oneconnect_profile): + profiles.append(oneconnect_profile) - # remove iRules - if 'lbaas_irule' in esd: - vs['rules'] = [] + if bool(compression_profile): + profiles.append(compression_profile) - # remove policies - if 'lbaas_policy' in esd: - vs['policies'] = [] + if profiles: + vip['profiles'] = profiles - # reset persistence to original definition - if 'pool' in svc: - vip_persist = self.service_adapter.get_session_persistence(svc) - vs.update(vip_persist) + vip['rules'] = vip.get('rules',[])+irules - for bigip in bigips: - try: - # update VS back to original listener definition - self.vs_helper.update(bigip, vs) + vip['policies'] = vip.get('policies',[])+policies - # add back SSL profile for TLS - if tls: - self.add_ssl_profile(tls, bigip) - except Exception as err: - LOG.exception("Virtual server update error: %s" % err.message) - raise + LOG.info("APPLY_ESD: Listener after ESDs got applied: %s", vip) \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/loadbalancer_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/loadbalancer_service.py index 6b8dc1342..27c820550 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/loadbalancer_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/loadbalancer_service.py @@ -27,16 +27,16 @@ class LoadBalancerServiceBuilder(object): - """Create loadbalancer related objects on BIG-IP®s + """Create loadbalancer related objects on BIG-IPs Handles requests to create and delete LBaaS v2 tenant partition - folders on one or more BIG-IP® systems. + folders on one or more BIG-IP systems. """ def __init__(self): self.folder_helper = BigIPResourceHelper(ResourceType.folder) def create_partition(self, service, bigips): - """Create tenant partition on set of BIG-IP®s. + """Create tenant partition on set of BIG-IPs. Creates a partition if it is not named "Common". @@ -50,7 +50,7 @@ def create_partition(self, service, bigips): self.folder_helper.create(bigip, folder) def delete_partition(self, service, bigips): - """Deletes partition from a set of BIG-IP® systems. + """Deletes partition from a set of BIG-IP systems. Deletes partition if it is not named "Common". diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/network_helper.py b/f5_openstack_agent/lbaasv2/drivers/bigip/network_helper.py index 4c8693f41..ee0e7164d 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/network_helper.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/network_helper.py @@ -23,6 +23,7 @@ from oslo_log import helpers as log_helpers from oslo_log import log as logging from requests.exceptions import HTTPError +#from f5_openstack_agent.lbaasv2.drivers.bigip.utils import strip_domain_address LOG = logging.getLogger(__name__) @@ -59,6 +60,11 @@ class NetworkHelper(object): 'strict': 'disabled', } + route_defaults = { + 'name': None, + 'partition': '/' + const.DEFAULT_PARTITION, + } + @log_helpers.log_method_call def create_l2gre_multipoint_profile(self, bigip, name, partition=const.DEFAULT_PARTITION): @@ -174,24 +180,35 @@ def get_selfip_addr(self, bigip, name, partition=const.DEFAULT_PARTITION): err.message)) return None - def route_domain_exists(self, bigip, partition=const.DEFAULT_PARTITION, + def route_domain_exists(self, bigip, partition=const.DEFAULT_PARTITION, name=None, domain_id=None): - if partition == 'Common': - return True + # if partition == 'Common': + # return True + + if name: + name = self._get_route_domain_name(name) + r = bigip.tm.net.route_domains.route_domain - name = partition + if domain_id: name += '_aux_' + str(domain_id) - return r.exists(name=name, partition=partition) + + if r.exists(name=name, partition=partition): + return r.load(name=name, partition=partition) + else: + return None + @log_helpers.log_method_call - def get_route_domain(self, bigip, partition=const.DEFAULT_PARTITION): + def get_route_domain(self, bigip, partition=const.DEFAULT_PARTITION, name=None): # this only works when the domain was created with is_aux=False, # same as the original code. - if partition == 'Common': - name = '0' + + if name: + name = self._get_route_domain_name(name) else: name = partition + r = bigip.tm.net.route_domains.route_domain return r.load(name=name, partition=partition) @@ -231,11 +248,21 @@ def _get_next_domain_id(self, bigip): return lowest_available_index @log_helpers.log_method_call - def create_route_domain(self, bigip, partition=const.DEFAULT_PARTITION, - strictness=False, is_aux=False): + def create_route_domain(self, bigip, partition=const.DEFAULT_PARTITION, name=None, + strictness=False, is_aux=False, rd_id=None): + + name = self._get_route_domain_name(name) + rd = bigip.tm.net.route_domains.route_domain - name = partition - id = self._get_next_domain_id(bigip) + if not name: + name = partition + + # ccloud: use an given id to avoid inconsistencies across bigip pair members + if rd_id is None: + id = self._get_next_domain_id(bigip) + else: + id = rd_id + if is_aux: name += '_aux_' + str(id) payload = NetworkHelper.route_domain_defaults @@ -252,8 +279,11 @@ def create_route_domain(self, bigip, partition=const.DEFAULT_PARTITION, def delete_route_domain(self, bigip, partition=const.DEFAULT_PARTITION, name=None): r = bigip.tm.net.route_domains.route_domain - if not name: + if name: + name = self._get_route_domain_name(name) + else: name = partition + obj = r.load(name=name, partition=partition) obj.delete() @@ -285,6 +315,60 @@ def get_route_domain_names(self, bigip, partition=const.DEFAULT_PARTITION): rd_names_list.append(rd.name) return rd_names_list + + @log_helpers.log_method_call + def route_exists(self, bigip, partition=const.DEFAULT_PARTITION, name=None): + rc = bigip.tm.net.routes.route + + if name: + name = self._get_route_name(name) + + return rc.exists(name=name, partition=partition) + + + + @log_helpers.log_method_call + def get_route(self, bigip, partition=const.DEFAULT_PARTITION, name=None): + rc = bigip.tm.net.routes.route + + if name: + name = self._get_route_name(name) + + return rc.load(name=name, partition=partition) + + @log_helpers.log_method_call + def create_route(self, bigip, partition=const.DEFAULT_PARTITION, name=None, gateway_ip='0.0.0.0', rd_id=0, destination_ip='0.0.0.0',netmask=0): + if self.route_exists(bigip, name=name, partition=partition): + LOG.info("Skipping create of route %s route already exists" % name) + return + + rc = bigip.tm.net.routes.route + + if name: + name = self._get_route_name(name) + + destination_ip+= '%' + str(rd_id)+ '/'+str(netmask) + gateway_ip+= '%' + str(rd_id) + + payload = NetworkHelper.route_defaults + + + + payload['name'] = name + payload['partition'] = partition + payload['gw'] = gateway_ip + payload['network'] = destination_ip + + + rc.create(**payload) + + @log_helpers.log_method_call + def delete_route(self ,bigip, partition=const.DEFAULT_PARTITION, name=None): + + if self.route_exists(bigip, partition=partition, name=name): + obj = self.get_route(bigip, partition, name) + obj.delete() + @log_helpers.log_method_call def get_vlans_in_route_domain(self, bigip, @@ -293,6 +377,21 @@ def get_vlans_in_route_domain(self, rd = self.get_route_domain(bigip, partition) return getattr(rd, 'vlans', []) + @log_helpers.log_method_call + def _get_route_domain_name(self, name): + if not name or name.startswith('rd-'): + return name + + return "rd-%s" % (name) + + @log_helpers.log_method_call + def _get_route_name(self, name): + if not name or name.startswith('rt-'): + return name + + return "rt-%s" % name + + @log_helpers.log_method_call def create_vlan(self, bigip, model): name = model.get('name', None) @@ -301,25 +400,30 @@ def create_vlan(self, bigip, model): description = model.get('description', None) route_domain_id = model.get('route_domain_id', const.DEFAULT_ROUTE_DOMAIN_ID) + mtu = model.get('mtu', None) if not name: return None v = bigip.tm.net.vlans.vlan if v.exists(name=name, partition=partition): obj = v.load(name=name, partition=partition) else: + # ccloud: Enable SYN Flood protection payload = {'name': name, 'partition': partition, - 'tag': tag} + 'tag': tag, + 'hardwareSyncookie': 'enabled'} if description: payload['description'] = description + if mtu: + payload['mtu'] = mtu + obj = v.create(**payload) interface = model.get('interface', None) if interface: payload = {'name': interface} if tag: payload['tagged'] = True - payload['tagMode'] = "service" else: payload['untagged'] = True @@ -331,6 +435,12 @@ def create_vlan(self, bigip, model): LOG.warn(e.message) payload.pop('tagMode') i.create(**payload) + # ccloud: 12.1.3 throws a different exception in case QinQ isn't allowed + except Exception as ie: + # Providing the tag-mode is not supported + LOG.info(ie.message) + payload.pop('tagMode') + i.create(**payload) if not partition == const.DEFAULT_PARTITION: self.add_vlan_to_domain_by_id(bigip, name, partition, @@ -354,9 +464,15 @@ def add_vlan_to_domain( self, bigip, name, - partition=const.DEFAULT_PARTITION): + partition=const.DEFAULT_PARTITION, rd_name=None): + + if rd_name: + rd_name = self._get_route_domain_name(rd_name) + else: + rd_name = partition + """Add VLANs to Domain.""" - rd = self.get_route_domain(bigip, partition) + rd = self.get_route_domain(bigip, partition, rd_name) existing_vlans = getattr(rd, 'vlans', []) if name in existing_vlans: return False @@ -513,14 +629,21 @@ def get_virtual_service_insertion( partition=const.DEFAULT_PARTITION): """Returns list of virtual server addresses""" vs = bigip.tm.ltm.virtuals - virtual_servers = vs.get_collection(partition=partition) + filter = "$filter=partition%20eq%20" + partition + # The filtering for partition of origin call below doesn't work. Therefore a new filtering is used + #virtual_servers = vs.get_collection(partition=partition) + virtual_servers = vs.get_collection(requests_params={'params': filter}) virtual_services = [] for virtual_server in virtual_servers: name = virtual_server.name virtual_address = {name: {}} dest = os.path.basename(virtual_server.destination) - (vip_addr, vip_port) = self.split_addr_port(dest) + # Don't take vs with snap pools instead of real ip's + if (virtual_server.sourceAddressTranslation and virtual_server.sourceAddressTranslation['type'] == 'snat'): + continue + else: + (vip_addr, vip_port) = self.split_addr_port(dest) virtual_address[name]['address'] = vip_addr virtual_address[name]['netmask'] = virtual_server.mask @@ -530,6 +653,26 @@ def get_virtual_service_insertion( return virtual_services + @log_helpers.log_method_call + def get_snat_addresses( + self, + bigip, + partition=const.DEFAULT_PARTITION): + """Returns list of snat addresses""" + filter = "$filter=partition%20eq%20" + partition + + snat_addrs = [] + try: + snats = bigip.tm.ltm.snat_translations.get_collection(requests_params={'params': filter}) + for snat in snats: + snat_addrs.append(snat.address) + + except Exception as e: + LOG.error('get_snat_addresses', + 'could not get addresses due to: %s' + % e.message) + return snat_addrs + @log_helpers.log_method_call def get_node_addresses(self, bigip, partition=const.DEFAULT_PARTITION): """Get the addresses of nodes within the partition.""" @@ -541,6 +684,82 @@ def get_node_addresses(self, bigip, partition=const.DEFAULT_PARTITION): return node_addrs + # Dummy method to check functionality from a standalone python script. + # The origin is in network_service as privta emethod descared + # def ips_exist_on_subnet(self, bigip, service, subnet, route_domain): + # # Does the big-ip have any IP addresses on this subnet? + # LOG.debug("_ips_exist_on_subnet entry %s rd %s" + # % (str(subnet['cidr']), route_domain)) + # route_domain = str(route_domain) + # ipsubnet = netaddr.IPNetwork(subnet['cidr']) + # + # # Are there any virtual addresses on this subnet? + # folder = service['loadbalancer']['tenant_id'] + # virtual_services = self.get_virtual_service_insertion( + # bigip, + # partition=folder + # ) + # for virt_serv in virtual_services: + # print virt_serv + # (_, dest) = virt_serv.items()[0] + # LOG.debug(" _ips_exist_on_subnet: checking vip %s" + # % str(dest['address'])) + # if len(dest['address'].split('%')) > 1: + # vip_route_domain = dest['address'].split('%')[1] + # else: + # vip_route_domain = '0' + # if vip_route_domain != route_domain: + # continue + # vip_addr = strip_domain_address(dest['address']) + # if netaddr.IPAddress(vip_addr) in ipsubnet: + # LOG.debug(" _ips_exist_on_subnet: found") + # return True + # + # # If there aren't any virtual addresses, are there + # # snat addresses on this subnet? + # snats = self.get_snat_addresses( + # bigip, + # partition=folder + # ) + # for snat in snats: + # LOG.debug(" _ips_exist_on_subnet: checking snat %s" + # % str(snat)) + # if len(snat.split('%')) > 1: + # snat_route_domain = snat.split('%')[1] + # else: + # snat_route_domain = '0' + # if snat_route_domain != route_domain: + # continue + # snat_addr = strip_domain_address(snat) + # if netaddr.IPAddress(snat_addr) in ipsubnet: + # LOG.debug(" _ips_exist_on_subnet: found") + # return True + # + # # If there aren't any virtual addresses and snats, are there + # # node addresses on this subnet? + # nodes = self.get_node_addresses( + # bigip, + # partition=folder + # ) + # for node in nodes: + # LOG.debug(" _ips_exist_on_subnet: checking node %s" + # % str(node)) + # if len(node.split('%')) > 1: + # node_route_domain = node.split('%')[1] + # else: + # node_route_domain = '0' + # if node_route_domain != route_domain: + # continue + # node_addr = strip_domain_address(node) + # if netaddr.IPAddress(node_addr) in ipsubnet: + # LOG.debug(" _ips_exist_on_subnet: found") + # return True + # + # LOG.debug(" _ips_exist_on_subnet exit %s" + # % str(subnet['cidr'])) + # # nothing found + # return False + @log_helpers.log_method_call def add_fdb_entry( self, diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/network_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/network_service.py index ac3f27b11..b0dcc5664 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/network_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/network_service.py @@ -13,9 +13,13 @@ # limitations under the License. # +#import pdb + import itertools import netaddr +import re +import constants_v2 as const from neutron.common.exceptions import NeutronException from neutron.plugins.common import constants as plugin_const from oslo_log import log as logging @@ -29,7 +33,7 @@ from f5_openstack_agent.lbaasv2.drivers.bigip.selfips import BigipSelfIpManager from f5_openstack_agent.lbaasv2.drivers.bigip.snats import BigipSnatManager from f5_openstack_agent.lbaasv2.drivers.bigip.utils import strip_domain_address - +from f5_openstack_agent.lbaasv2.drivers.bigip import utils LOG = logging.getLogger(__name__) @@ -74,51 +78,46 @@ def set_l2pop_rpc(self, l2pop_rpc): def initialize_vcmp(self): self.l2_service.initialize_vcmp_manager() - def initialize_tunneling(self): + def initialize_tunneling(self, bigip): # setup tunneling vtep_folder = self.conf.f5_vtep_folder vtep_selfip_name = self.conf.f5_vtep_selfip_name - local_ips = [] - - for bigip in self.driver.get_all_bigips(): - bigip.local_ip = None + bigip.local_ip = None - if not vtep_folder or vtep_folder.lower() == 'none': - vtep_folder = 'Common' + if not vtep_folder or vtep_folder.lower() == 'none': + vtep_folder = 'Common' - if vtep_selfip_name and \ - not vtep_selfip_name.lower() == 'none': + if vtep_selfip_name and \ + not vtep_selfip_name.lower() == 'none': - # profiles may already exist - # create vxlan_multipoint_profile` - self.network_helper.create_vxlan_multipoint_profile( - bigip, - 'vxlan_ovs', - partition='Common') - # create l2gre_multipoint_profile - self.network_helper.create_l2gre_multipoint_profile( - bigip, - 'gre_ovs', - partition='Common') + # profiles may already exist + # create vxlan_multipoint_profile` + self.network_helper.create_vxlan_multipoint_profile( + bigip, + 'vxlan_ovs', + partition='Common') + # create l2gre_multipoint_profile + self.network_helper.create_l2gre_multipoint_profile( + bigip, + 'gre_ovs', + partition='Common') - # find the IP address for the selfip for each box - local_ip = self.bigip_selfip_manager.get_selfip_addr( - bigip, - vtep_selfip_name, - partition=vtep_folder - ) + # find the IP address for the selfip for each box + local_ip = self.bigip_selfip_manager.get_selfip_addr( + bigip, + vtep_selfip_name, + partition=vtep_folder + ) - if local_ip: - bigip.local_ip = local_ip - local_ips.append(local_ip) - else: - raise f5_ex.MissingVTEPAddress( - 'device %s missing vtep selfip %s' - % (bigip.device_name, - '/' + vtep_folder + '/' + - vtep_selfip_name)) - return local_ips + if local_ip: + bigip.local_ip = local_ip + else: + raise f5_ex.MissingVTEPAddress( + 'device %s missing vtep selfip %s' + % (bigip.device_name, + '/' + vtep_folder + '/' + + vtep_selfip_name)) def is_service_connected(self, service): networks = service.get('networks', {}) @@ -136,30 +135,34 @@ def is_service_connected(self, service): segmentation_id = \ network.get('provider:segmentation_id', None) if not segmentation_id: - if network_type in supported_net_types and \ - self.conf.f5_network_segment_physical_network: + if network_type in supported_net_types and self.conf.f5_network_segment_physical_network: return False - - LOG.error("Misconfiguration: Segmentation ID is " - "missing from the service definition. " - "Please check the setting for " - "f5_network_segment_physical_network in " - "f5-openstack-agent.ini in case neutron " - "is operating in Hierarchical Port Binding " - "mode.") - raise f5_ex.InvalidNetworkDefinition( - "Network segment ID %s not defined" % network_id) + else: + LOG.error("Misconfiguration: Segmentation ID is " + "missing from the service definition. " + "Please check the setting for " + "f5_network_segment_physical_network in " + "f5-openstack-agent.ini in case neutron " + "is operating in Hierarchical Port Binding " + "mode.") + raise f5_ex.InvalidNetworkDefinition( + "Network segment ID %s not defined" % network_id) return True + @utils.instrument_execution_time def prep_service_networking(self, service, traffic_group): """Assure network connectivity is established on all bigips.""" if self.conf.f5_global_routed_mode: return - if not self.is_service_connected(service): + try: + if not self.is_service_connected(service): + raise f5_ex.NetworkNotReady( + "Network segment(s) definition incomplete") + except f5_ex.InvalidNetworkDefinition as exc: raise f5_ex.NetworkNotReady( - "Network segment(s) definition incomplete") + "Network segment(s) definition invalid %s", exc.message) if self.conf.use_namespaces: try: @@ -195,6 +198,8 @@ def prep_service_networking(self, service, traffic_group): for subnetinfo in subnetsinfo: if self.conf.f5_snat_addresses_per_subnet > 0: self._assure_subnet_snats(assure_bigips, service, subnetinfo) + elif self.conf.f5_snat_addresses_per_subnet == -1: + self._assure_lb_snats(assure_bigips, service, subnetinfo) if subnetinfo['is_for_member'] and not self.conf.f5_snat_mode: try: @@ -208,11 +213,29 @@ def prep_service_networking(self, service, traffic_group): self.bigip_selfip_manager.assure_gateway_on_subnet( assure_bigip, subnetinfo, traffic_group) + self._assure_subnet_gateway(service) + + def _assure_subnet_gateway(self,service): + network_id = service['loadbalancer']['network_id'] + + for bigip in self.driver.get_all_bigips(): + rd = self.network_helper.get_route_domain(bigip, partition=const.DEFAULT_PARTITION, name=network_id) + + for subnet_id, subnet in service['subnets'].iteritems(): + + if not self.network_helper.route_exists(bigip, const.DEFAULT_PARTITION,subnet_id): + try: + self.network_helper.create_route(bigip, const.DEFAULT_PARTITION,subnet_id, subnet['gateway_ip'], rd.id) + except Exception as err: + LOG.error("Failed to create default gateway route for network %s subnet %s" % (network_id, subnet_id)) + LOG.exception(err) + def _annotate_service_route_domains(self, service): + # wtn : subnet for member has to be subnet for vip # Add route domain notation to pool member and vip addresses. + # ccloud: don't allow creation of members without route domain in case of NOT global routed mode setting tenant_id = service['loadbalancer']['tenant_id'] self.update_rds_cache(tenant_id) - if 'members' in service: for member in service['members']: if 'address' in member: @@ -229,37 +252,58 @@ def _annotate_service_route_domains(self, service): member['subnet_id'] )) if member_network: - self.assign_route_domain( - tenant_id, member_network, member_subnet) - rd_id = ( - '%' + str(member_network['route_domain_id']) - ) - member['address'] += rd_id + self.assign_route_domain(tenant_id, member_network, member_subnet) + if 'route_domain_id' in member_network and member_network['route_domain_id']: + rd_id = ( + '%' + str(member_network['route_domain_id']) + ) + if rd_id != '%0': + member['address'] += rd_id + else: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK1 Global routing disabled but route domain ID 0 was found. Discarding ...') + else: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK2 Global routing disabled but route domain ID could not be found for pool member. Discarding ...') + else: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK3 Global routing disabled but NO member network can be found for pool member. Discarding ...') else: - member['address'] += '%0' + if not self.conf.f5_global_routed_mode: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK4 Global routing disabled but NO member network ID given for pool member. Discarding ...') + else: + member['address'] += '%0' + LOG.info("ccloud: NETWORK-RDCHECK5 Using default Route Domain because of global routing %s" % member['address']) if 'vip_address' in service['loadbalancer']: loadbalancer = service['loadbalancer'] - if 'network_id' in loadbalancer: - lb_network = self.service_adapter.get_network_from_service( - service, loadbalancer['network_id']) - vip_subnet = self.service_adapter.get_subnet_from_service( - service, loadbalancer['vip_subnet_id']) - self.assign_route_domain( - tenant_id, lb_network, vip_subnet) - rd_id = '%' + str(lb_network['route_domain_id']) - service['loadbalancer']['vip_address'] += rd_id + if 'network_id' in loadbalancer and loadbalancer['network_id']: + lb_network = self.service_adapter.get_network_from_service(service, loadbalancer['network_id']) + vip_subnet = self.service_adapter.get_subnet_from_service(service, loadbalancer['vip_subnet_id']) + self.assign_route_domain(tenant_id, lb_network, vip_subnet) + if 'route_domain_id' in lb_network and lb_network['route_domain_id']: + rd_id = '%' + str(lb_network['route_domain_id']) + if rd_id != '%0': + loadbalancer['vip_address'] += rd_id + else: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK5 Global routing disabled but route domain ID 0 was found. Discarding ...') + else: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK6 Global routing disabled but route domain ID could not be found for virtual_address member. Discarding ...') else: - service['loadbalancer']['vip_address'] += '%0' + if not self.conf.f5_global_routed_mode: + raise f5_ex.RouteDomainQueryException('ccloud: NETWORK-RDCHECK7 Global routing disabled but NO vip_address network ID given. Discarding ...') + else: + loadbalancer['vip_address'] += '%0' + LOG.info("ccloud: NETWORK-RDCHECK8 Using default Route Domain because of global routing %s" % loadbalancer['vip_address']) + def is_common_network(self, network): return self.l2_service.is_common_network(network) def assign_route_domain(self, tenant_id, network, subnet): # Assign route domain for a network - if self.l2_service.is_common_network(network): - network['route_domain_id'] = 0 - return + + + # if self.l2_service.is_common_network(network): + # network['route_domain_id'] = 0 + # return LOG.debug("Assign route domain get from cache %s" % network) route_domain_id = self.get_route_domain_from_cache(network) @@ -274,10 +318,13 @@ def assign_route_domain(self, tenant_id, network, subnet): if self.conf.max_namespaces_per_tenant == 1: bigip = self.driver.get_bigip() LOG.debug("bigip before get_domain: %s" % bigip) - partition_id = self.service_adapter.get_folder_name( - tenant_id) + # partition_id = self.service_adapter.get_folder_name( + # tenant_id) + + partition_id='Common' + tenant_rd = self.network_helper.get_route_domain( - bigip, partition=partition_id) + bigip, partition=partition_id, name=network['id']) network['route_domain_id'] = tenant_rd.id return @@ -538,12 +585,17 @@ def get_neutron_net_short_name(network): net_type = network.get('provider:network_type', None) net_seg_key = network.get('provider:segmentation_id', None) if not net_type or not net_seg_key: - raise f5_ex.InvalidNetworkType + raise f5_ex.InvalidNetworkType( + 'Provider network attributes not complete:' + 'provider: network_type - {0} ' + 'and provider:segmentation_id - {1}' + .format(net_type, net_seg_key)) return net_type + '-' + str(net_seg_key) def _assure_subnet_snats(self, assure_bigips, service, subnetinfo): # Ensure snat for subnet exists on bigips + lb_id = service['loadbalancer']['id'] tenant_id = service['loadbalancer']['tenant_id'] subnet = subnetinfo['subnet'] snats_per_subnet = self.conf.f5_snat_addresses_per_subnet @@ -567,7 +619,38 @@ def _assure_subnet_snats(self, assure_bigips, service, subnetinfo): (snats_per_subnet, len(snat_addrs))) for assure_bigip in assure_bigips: self.bigip_snat_manager.assure_bigip_snats( - assure_bigip, subnetinfo, snat_addrs, tenant_id) + assure_bigip, subnetinfo, snat_addrs, tenant_id, lb_id) + + def _assure_lb_snats(self, assure_bigips, service, subnetinfo): + # Ensure snat for loadbalancer exists on bigips + tenant_id = service['loadbalancer']['tenant_id'] + + lb_id = service['loadbalancer']['id'] + + + assure_bigips = \ + [bigip for bigip in assure_bigips + if tenant_id not in bigip.assured_tenant_snat_subnets or + lb_id not in + bigip.assured_tenant_snat_subnets[tenant_id]] + + LOG.debug("_assure_subnet_snats: getting snat addrs for: %s" % + lb_id) + if len(assure_bigips): + + ip_address = service['loadbalancer']["vip_address"] + + match = re.search("%[0-9]+$", str(ip_address)) + + if match is not None: + ip_address = ip_address[:-len(match.group(0))] + + snat_addrs = [ip_address] + for assure_bigip in assure_bigips: + self.bigip_snat_manager.assure_bigip_snats( + assure_bigip, subnetinfo, snat_addrs, tenant_id, lb_id) + + pass def _allocate_gw_addr(self, subnetinfo): # Create a name for the port and for the IP Forwarding @@ -609,6 +692,7 @@ def _allocate_gw_addr(self, subnetinfo): LOG.exception(ermsg) return True + @utils.instrument_execution_time def post_service_networking(self, service, all_subnet_hints): # Assure networks are deleted from big-ips if self.conf.f5_global_routed_mode: @@ -620,35 +704,28 @@ def post_service_networking(self, service, all_subnet_hints): # Delete shared config objects deleted_names = set() + lb_is_last_on_network = self._is_last_on_network(service) + for bigip in self.driver.get_config_bigips(): LOG.debug('post_service_networking: calling ' - '_assure_delete_networks del nets sh for bigip %s %s' + '_assure_delete_networks del nets shared for bigip %s %s' % (bigip.device_name, all_subnet_hints)) subnet_hints = all_subnet_hints[bigip.device_name] - deleted_names = deleted_names.union( - self._assure_delete_nets_shared(bigip, service, - subnet_hints)) + deleted_names = deleted_names.union(self._assure_delete_nets_shared(bigip, service, subnet_hints, lb_is_last_on_network)) # Delete non shared config objects for bigip in self.driver.get_all_bigips(): LOG.debug(' post_service_networking: calling ' - ' _assure_delete_networks del nets ns for bigip %s' + ' _assure_delete_networks del nets NONshared for bigip %s' % bigip.device_name) - subnet_hints = all_subnet_hints[bigip.device_name] - - deleted_names = deleted_names.union( - self._assure_delete_nets_nonshared( - bigip, service, subnet_hints) - ) + deleted_names = deleted_names.union(self._assure_delete_nets_nonshared(bigip, service, subnet_hints, lb_is_last_on_network)) for port_name in deleted_names: - LOG.debug(' post_service_networking: calling ' - ' del port %s' - % port_name) - self.driver.plugin_rpc.delete_port_by_name( - port_name=port_name) + LOG.debug(' post_service_networking: calling del port %s' % port_name) + self.driver.plugin_rpc.delete_port_by_name(port_name=port_name) + @utils.instrument_execution_time def update_bigip_l2(self, service): # Update fdb entries on bigip loadbalancer = service['loadbalancer'] @@ -703,7 +780,8 @@ def update_bigip_member_l2(self, bigip, loadbalancer, member): self.l2_service.add_bigip_fdbs( bigip, net_folder, fdb_info, member) else: - LOG.warning('LBaaS member, %s, is not associated with Neutron ' + #ccloud: reduced to info, external(non project) member IP's never get an port in neutron + LOG.info('LBaaS member, %s, is not associated with Neutron ' 'port. No fdb entries will be created for this ' 'member.' % member['address']) @@ -762,48 +840,67 @@ def delete_bigip_vip_l2(self, bigip, loadbalancer): self.l2_service.delete_bigip_fdbs( bigip, net_folder, fdb_info, loadbalancer) - def _assure_delete_nets_shared(self, bigip, service, subnet_hints): + def _assure_delete_nets_shared(self, bigip, service, subnet_hints, lb_is_last_on_network): # Assure shared configuration (which syncs) is deleted deleted_names = set() tenant_id = service['loadbalancer']['tenant_id'] + lb_id = service['loadbalancer']['id'] + # delete all snats for a subnet id subnet doesn't hold any ip's anymore delete_gateway = self.bigip_selfip_manager.delete_gateway_on_subnet - for subnetinfo in self._get_subnets_to_delete(bigip, - service, - subnet_hints): + subnet_to_delete, subnet_with_deletion = self._get_subnets_to_delete(bigip, service, subnet_hints) + for subnetinfo in subnet_to_delete: try: + my_deleted_names, my_in_use_subnets = self.bigip_snat_manager.delete_bigip_snats(bigip, subnetinfo, tenant_id, lb_id) + deleted_names = deleted_names.union(my_deleted_names) + for in_use_subnetid in my_in_use_subnets: + subnet_hints['check_for_delete_subnets'].pop(in_use_subnetid, None) + if not self.conf.f5_snat_mode: gw_name = delete_gateway(bigip, subnetinfo) deleted_names.add(gw_name) - my_deleted_names, my_in_use_subnets = \ - self.bigip_snat_manager.delete_bigip_snats( - bigip, subnetinfo, tenant_id) + elif lb_is_last_on_network: + self.network_helper.delete_route(bigip, const.DEFAULT_PARTITION,subnetinfo['subnet_id']) + + except NeutronException as exc: + LOG.error("assure_delete_nets_shared: exception #1: %s" + % str(exc.msg)) + except Exception as exc: + LOG.error("assure_delete_nets_shared: exception #2: %s" + % str(exc.message)) + + # delete one snat for a loadbalancer if a load balancer deletion happend + for subnetinfo in subnet_with_deletion: + try: + my_deleted_names, my_in_use_subnets = self.bigip_snat_manager.delete_bigip_snats(bigip, subnetinfo, tenant_id, lb_id) deleted_names = deleted_names.union(my_deleted_names) - for in_use_subnetid in my_in_use_subnets: - subnet_hints['check_for_delete_subnets'].pop( - in_use_subnetid, None) + except NeutronException as exc: - LOG.error("assure_delete_nets_shared: exception: %s" + LOG.error("assure_delete_nets_shared: exception #3: %s" % str(exc.msg)) except Exception as exc: - LOG.error("assure_delete_nets_shared: exception: %s" + LOG.error("assure_delete_nets_shared: exception #4: %s" % str(exc.message)) return deleted_names - def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): + @utils.instrument_execution_time + def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints, lb_is_last_on_network): # Delete non shared base objects for networks deleted_names = set() - for subnetinfo in self._get_subnets_to_delete(bigip, - service, - subnet_hints): + + if not lb_is_last_on_network: + return deleted_names + + + subnet_to_delete, subnet_with_deletion = self._get_subnets_to_delete(bigip, service, subnet_hints) + for subnetinfo in subnet_to_delete: try: network = subnetinfo['network'] if self.l2_service.is_common_network(network): network_folder = 'Common' else: - network_folder = self.service_adapter.get_folder_name( - service['loadbalancer']['tenant_id']) + network_folder = self.service_adapter.get_folder_name(service['loadbalancer']['tenant_id']) subnet = subnetinfo['subnet'] if self.conf.f5_populate_static_arp: @@ -814,8 +911,11 @@ def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): partition=network_folder ) - local_selfip_name = "local-" + bigip.device_name + \ - "-" + subnet['id'] + + if lb_is_last_on_network: + self.network_helper.delete_route(bigip, const.DEFAULT_PARTITION,subnetinfo['subnet_id']) + + local_selfip_name = "local-" + bigip.device_name + "-" + subnet['id'] selfip_address = self.bigip_selfip_manager.get_selfip_addr( bigip, @@ -824,8 +924,7 @@ def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): ) if not selfip_address: - LOG.error("Failed to get self IP address %s in cleanup.", - local_selfip_name) + LOG.error("Failed to get self IP address %s in cleanup.", local_selfip_name) self.bigip_selfip_manager.delete_selfip( bigip, @@ -834,8 +933,7 @@ def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): ) if self.l3_binding and selfip_address: - self.l3_binding.unbind_address(subnet_id=subnet['id'], - ip_address=selfip_address) + self.l3_binding.unbind_address(subnet_id=subnet['id'], ip_address=selfip_address) deleted_names.add(local_selfip_name) @@ -847,8 +945,7 @@ def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): self.remove_from_rds_cache(network, subnet) tenant_id = service['loadbalancer']['tenant_id'] if tenant_id in bigip.assured_tenant_snat_subnets: - tenant_snat_subnets = \ - bigip.assured_tenant_snat_subnets[tenant_id] + tenant_snat_subnets = bigip.assured_tenant_snat_subnets[tenant_id] if subnet['id'] in tenant_snat_subnets: tenant_snat_subnets.remove(subnet['id']) except NeutronException as exc: @@ -860,10 +957,25 @@ def _assure_delete_nets_nonshared(self, bigip, service, subnet_hints): return deleted_names - def _get_subnets_to_delete(self, bigip, service, subnet_hints): + def _is_last_on_network(self, service): + network_id= service['loadbalancer']['network_id'] + + lb_id = service['loadbalancer']['id'] + + loadbalancers = self.driver.plugin_rpc.get_loadbalancers_by_network(network_id) + + for lb in loadbalancers: + if lb['lb_id'] != lb_id: + return False + + return True + + @utils.instrument_execution_time + def _get_subnets_to_delete(self, bigip, service, subnet_hints, whole_subnet=True): # Clean up any Self IP, SNATs, networks, and folder for # services items that we deleted. subnets_to_delete = [] + subnets_with_deletion = [] for subnetinfo in subnet_hints['check_for_delete_subnets'].values(): subnet = self.service_adapter.get_subnet_from_service( service, subnetinfo['subnet_id']) @@ -880,9 +992,12 @@ def _get_subnets_to_delete(self, bigip, service, subnet_hints): subnet, route_domain): subnets_to_delete.append(subnetinfo) + else: + subnets_with_deletion.append(subnetinfo) - return subnets_to_delete + return subnets_to_delete, subnets_with_deletion + @utils.instrument_execution_time def _ips_exist_on_subnet(self, bigip, service, subnet, route_domain): # Does the big-ip have any IP addresses on this subnet? LOG.debug("_ips_exist_on_subnet entry %s rd %s" @@ -958,7 +1073,6 @@ def _get_subnets_to_assure(self, service): networks = dict() loadbalancer = service['loadbalancer'] service_adapter = self.service_adapter - lb_status = loadbalancer['provisioning_status'] if lb_status != plugin_const.PENDING_DELETE: if 'network_id' in loadbalancer: diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/plugin_rpc.py b/f5_openstack_agent/lbaasv2/drivers/bigip/plugin_rpc.py index 7f243482f..399bc6fe4 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/plugin_rpc.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/plugin_rpc.py @@ -23,7 +23,7 @@ from neutron_lbaas.services.loadbalancer import constants as lb_const from f5_openstack_agent.lbaasv2.drivers.bigip import constants_v2 as constants - +from f5_openstack_agent.lbaasv2.drivers.bigip import utils LOG = logging.getLogger @@ -418,6 +418,7 @@ def delete_port(self, port_id=None, mac_address=None): ) @log_helpers.log_method_call + @utils.instrument_execution_time def get_service_by_loadbalancer_id(self, loadbalancer_id=None): """Retrieve the service definition for this loadbalancer.""" @@ -437,6 +438,7 @@ def get_service_by_loadbalancer_id(self, return service @log_helpers.log_method_call + @utils.instrument_execution_time def get_all_loadbalancers(self, env=None, group=None, host=None): """Retrieve a list of loadbalancers in Neutron.""" loadbalancers = [] @@ -460,6 +462,7 @@ def get_all_loadbalancers(self, env=None, group=None, host=None): return loadbalancers @log_helpers.log_method_call + @utils.instrument_execution_time def get_active_loadbalancers(self, env=None, group=None, host=None): """Retrieve a list of active loadbalancers for this agent.""" loadbalancers = [] @@ -478,11 +481,12 @@ def get_active_loadbalancers(self, env=None, group=None, host=None): ) except messaging.MessageDeliveryFailure: LOG.error("agent->plugin RPC exception caught: ", - "get_all_loadbalancers") + "get_active_loadbalancers") return loadbalancers @log_helpers.log_method_call + @utils.instrument_execution_time def get_pending_loadbalancers(self, env=None, group=None, host=None): """Retrieve a list of pending loadbalancers for this agent.""" loadbalancers = [] @@ -501,6 +505,221 @@ def get_pending_loadbalancers(self, env=None, group=None, host=None): ) except messaging.MessageDeliveryFailure: LOG.error("agent->plugin RPC exception caught: ", - "get_all_loadbalancers") + "get_pending_loadbalancers") + + return loadbalancers + + @log_helpers.log_method_call + @utils.instrument_execution_time + def get_loadbalancers_without_agent_binding(self, env=None, group=None): + """Retrieve a list of loadbalancers without an agent binding in Neutron.""" + unbound_loadbalancers = [] + + if not env: + env = self.env + + try: + unbound_loadbalancers = self._call( + self.context, + self._make_msg('get_loadbalancers_without_agent_binding', + env=env, + group=group), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "get_loadbalancers_without_agent_binding") + + return unbound_loadbalancers + + @log_helpers.log_method_call + def get_errored_loadbalancers(self, env=None, group=None, host=None): + """Retrieve a list of errored loadbalancers for this agent.""" + loadbalancers = [] + + if not env: + env = self.env + + try: + loadbalancers = self._call( + self.context, + self._make_msg('get_errored_loadbalancers', + env=env, + group=group, + host=host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "get_errored_loadbalancers") return loadbalancers + + @log_helpers.log_method_call + def get_loadbalancers_by_network(self, network_id, env=None,group=None,host=None): + """Retrieve a list of loadbalancers for a network.""" + loadbalancers = [] + + if not env: + env = self.env + + try: + loadbalancers = self._call( + self.context, + self._make_msg('get_loadbalancers_by_network', + env=env, + network_id=network_id, + group=group, + host=host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "get_loadbalancers_by_network") + + return loadbalancers + + @log_helpers.log_method_call + def set_agent_admin_state(self, admin_state_up): + """Set the admin_state_up of for this agent""" + succeeded = False + try: + succeeded = self._call( + self.context, + self._make_msg('set_agent_admin_state', + admin_state_up=admin_state_up, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "set_agent_admin_state") + + return succeeded + + @log_helpers.log_method_call + def scrub_dead_agents(self, env, group): + """Set the admin_state_up of for this agent""" + service = {} + try: + service = self._call( + self.context, + self._make_msg('scrub_dead_agents', + env=env, + group=group, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "scrub_dead_agents") + + return service + + @log_helpers.log_method_call + def get_clusterwide_agent(self, env, group): + """Determin which agent performce global tasks for the cluster""" + agent = {} + try: + agent = self._call( + self.context, + self._make_msg('get_clusterwide_agent', + env=env, + group=group, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "scrub_dead_agents") + + return agent + + @log_helpers.log_method_call + def validate_loadbalancers_state(self, loadbalancers): + """Get the status of a list of loadbalancers IDs in Neutron""" + lb_status = {} + try: + lb_status = self._call( + self.context, + self._make_msg('validate_loadbalancers_state', + loadbalancers=loadbalancers, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "validate_loadbalancers_state") + + return lb_status + + @log_helpers.log_method_call + def validate_listeners_state(self, listeners): + """Get the status of a list of listener IDs in Neutron""" + listener_status = {} + try: + listener_status = self._call( + self.context, + self._make_msg('validate_listeners_state', + listeners=listeners, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "validate_pool_state") + + return listener_status + + @log_helpers.log_method_call + def validate_pools_state(self, pools): + """Get the status of a list of pools IDs in Neutron""" + pool_status = {} + try: + pool_status = self._call( + self.context, + self._make_msg('validate_pools_state', + pools=pools, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "validate_pool_state") + + return pool_status + + @log_helpers.log_method_call + def get_pools_members(self, pools): + """Get the members of a list of pools IDs in Neutron.""" + pools_members = {} + try: + pools_members = self._call( + self.context, + self._make_msg('get_pools_members', + pools=pools, + host=self.host), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "get_pools_members") + + return pools_members + + @log_helpers.log_method_call + def validate_l7policys_state_by_listener(self, listeners): + """Get the status of a list of l7policys IDs in Neutron""" + l7policy_status = {} + try: + l7policy_status = self._call( + self.context, + self._make_msg('validate_l7policys_state_by_listener', + listeners=listeners), + topic=self.topic + ) + except messaging.MessageDeliveryFailure: + LOG.error("agent->plugin RPC exception caught: ", + "validate_l7policys_state_by_listener") + + return l7policy_status diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/pool_service.py b/f5_openstack_agent/lbaasv2/drivers/bigip/pool_service.py index 8c2f7d1ee..a1f4d375e 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/pool_service.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/pool_service.py @@ -28,13 +28,13 @@ class PoolServiceBuilder(object): - """Create LBaaS v2 pools and related objects on BIG-IP®s. + """Create LBaaS v2 pools and related objects on BIG-IPs. Handles requests to create, update, delete LBaaS v2 pools, - health monitors, and members on one or more BIG-IP® systems. + health monitors, and members on one or more BIG-IP systems. """ - def __init__(self, service_adapter): + def __init__(self, service_adapter, f5_parent_https_monitor=None): self.service_adapter = service_adapter self.http_mon_helper = BigIPResourceHelper(ResourceType.http_monitor) self.https_mon_helper = BigIPResourceHelper(ResourceType.https_monitor) @@ -42,58 +42,117 @@ def __init__(self, service_adapter): self.ping_mon_helper = BigIPResourceHelper(ResourceType.ping_monitor) self.pool_helper = BigIPResourceHelper(ResourceType.pool) self.node_helper = BigIPResourceHelper(ResourceType.node) + self.f5_parent_https_monitor = f5_parent_https_monitor def create_pool(self, service, bigips): - """Create a pool on set of BIG-IP®s. + """Create a pool on set of BIG-IPs. - Creates a BIG-IP® pool to represent an LBaaS pool object. + Creates a BIG-IP pool to represent an LBaaS pool object. :param service: Dictionary which contains a both a pool and load balancer definition. :param bigips: Array of BigIP class instances to create pool. """ pool = self.service_adapter.get_pool(service) + ex = None for bigip in bigips: - self.pool_helper.create(bigip, pool) + try: + self.pool_helper.create(bigip, pool) + LOG.info("Pool created: %s", pool['name']) + except HTTPError as err: + if err.response.status_code == 409: + LOG.info("Pool already exists...updating") + try: + self.pool_helper.update(bigip, pool) + LOG.info("Pool updated: %s", pool['name']) + except Exception as err: + ex = err + LOG.error("Pool creation/update FAILED for pool %s on %s: %s", + pool['name'], bigip, err.message) + else: + ex = err + LOG.error("Pool creation FAILED for pool %s on %s: %s", + pool['name'], bigip, err.message) + + if ex: + raise ex + def delete_pool(self, service, bigips): - """Delete a pool on set of BIG-IP®s. + """Delete a pool on set of BIG-IPs. - Deletes a BIG-IP® pool defined by LBaaS pool object. + Deletes a BIG-IP pool defined by LBaaS pool object. :param service: Dictionary which contains a both a pool and load balancer definition. :param bigips: Array of BigIP class instances to delete pool. """ pool = self.service_adapter.get_pool(service) - + ex = None for bigip in bigips: - self.pool_helper.delete(bigip, - name=pool["name"], - partition=pool["partition"]) + try: + self.pool_helper.delete(bigip, + name=pool["name"], + partition=pool["partition"]) + LOG.info("Pool deleted: %s", pool['name']) + except HTTPError as err: + LOG.info("Pool deletion FAILED: %s", pool['name']) + ex = err + if ex: + raise ex + def update_pool(self, service, bigips): - """Update BIG-IP® pool. + """Update BIG-IP pool. :param service: Dictionary which contains a both a pool and load balancer definition. :param bigips: Array of BigIP class instances to create pool. """ pool = self.service_adapter.get_pool(service) + ex = None for bigip in bigips: - self.pool_helper.update(bigip, pool) + try: + self.pool_helper.update(bigip, pool) + LOG.info("Pool updated DONE: %s", pool['name']) + except HTTPError as err: + LOG.debug("Pool update FAILED: %s", pool['name']) + ex = err + if ex: + raise ex + def create_healthmonitor(self, service, bigips): # create member hm = self.service_adapter.get_healthmonitor(service) + #ccloud: set additional attributes like parent monitor in case of creation, might be ignored for update + self._set_monitor_attributes(service, hm) hm_helper = self._get_monitor_helper(service) pool = self.service_adapter.get_pool(service) + ex = None for bigip in bigips: - hm_helper.create(bigip, hm) + try: + hm_helper.create(bigip, hm) + # update pool with new health monitor + self.pool_helper.update(bigip, pool) + LOG.info("Health Monitor created: %s", hm['name']) + except HTTPError as err: + if err.response.status_code == 409: + try: + hm_helper.update(bigip, hm) + LOG.info("Health Monitor upserted: %s", hm['name']) + except Exception as err: + ex = err + LOG.error("Failed to upsert monitor %s on %s: %s", + hm['name'], bigip, err.message) + else: + ex = err + LOG.error("Failed to upsert monitor %s on %s: %s", + hm['name'], bigip, err.message) + if ex: + raise ex - # update pool with new health monitor - self.pool_helper.update(bigip, pool) def delete_healthmonitor(self, service, bigips): # delete health monitor @@ -104,44 +163,75 @@ def delete_healthmonitor(self, service, bigips): pool = self.service_adapter.get_pool(service) pool["monitor"] = "" + ex = None for bigip in bigips: - # need to first remove monitor reference from pool - self.pool_helper.update(bigip, pool) - - # after updating pool, delete monitor - hm_helper.delete(bigip, - name=hm["name"], - partition=hm["partition"]) + try: + # need to first remove monitor reference from pool + self.pool_helper.update(bigip, pool) + # after updating pool, delete monitor + hm_helper.delete(bigip, + name=hm["name"], + partition=hm["partition"]) + LOG.info("Health Monitor deleted: %s", hm['name']) + except HTTPError as err: + LOG.info("Health Monitor deletion FAILED: %s", hm['name']) + ex = err + if ex: + raise ex def update_healthmonitor(self, service, bigips): hm = self.service_adapter.get_healthmonitor(service) hm_helper = self._get_monitor_helper(service) pool = self.service_adapter.get_pool(service) + ex = None for bigip in bigips: - hm_helper.update(bigip, hm) - - # update pool with new health monitor - self.pool_helper.update(bigip, pool) + try: + hm_helper.update(bigip, hm) + # update pool with new health monitor + self.pool_helper.update(bigip, pool) + LOG.info("Health Monitor updated: %s", hm['name']) + except HTTPError as err: + LOG.info("Health Monitor update FAILED: %s", hm['name']) + ex = err + if ex: + raise ex # Note: can't use BigIPResourceHelper class because members # are created within pool objects. Following member methods - # use the F5® SDK directly. + # use the F5 SDK directly. def create_member(self, service, bigips): pool = self.service_adapter.get_pool(service) member = self.service_adapter.get_member(service) + if '%' not in member['address'] or '%0' in member['address']: + LOG.error("ccloud: POOL-RDCHECK1 - trying to create member with address: %s", member['address']) + + ex = None for bigip in bigips: - part = pool["partition"] - p = self.pool_helper.load(bigip, - name=pool["name"], - partition=part) - m = p.members_s.members - m.create(**member) + try: + part = pool["partition"] + p = self.pool_helper.load(bigip, + name=pool["name"], + partition=part) + m = p.members_s.members + m.create(**member) + LOG.info("Member created: %s", member['address']) + except HTTPError as err: + # ccloud: Do not log failure because create is always called and updates are made in case of failure + # create member method logs it's own method in case of failure + #LOG.info("Member creation FAILED: %s", member['address']) + ex = err + if ex: + raise ex def delete_member(self, service, bigips): pool = self.service_adapter.get_pool(service) member = self.service_adapter.get_member(service) + if '%' not in member['address'] or '%0' in member['address']: + LOG.error("ccloud: POOL-RDCHECK2 - trying to create member with address: %s", member['address']) part = pool["partition"] + + ex = None for bigip in bigips: p = self.pool_helper.load(bigip, name=pool["name"], @@ -154,36 +244,52 @@ def delete_member(self, service, bigips): m = m.load(name=urllib.quote(member["name"]), partition=part) - m.delete() try: + m.delete() + LOG.info("Member deleted: %s", member['address']) + node = self.service_adapter.get_member_node(service) self.node_helper.delete(bigip, name=urllib.quote(node["name"]), partition=node["partition"]) + LOG.info("Node deleted: %s", node["name"]) except HTTPError as err: # Possilbe error if node is shared with another member. # If so, ignore the error. if err.response.status_code == 400: - LOG.debug(err.message) + LOG.debug("ccloud: Node %s not deleted because it's referenced as member somewhere else" % node['name']) else: - raise + LOG.info("Member or Node deletion FAILED: %s", member['address']) + ex = err + if ex: + raise ex def update_member(self, service, bigips): pool = self.service_adapter.get_pool(service) member = self.service_adapter.get_member(service) - + if '%' not in member['address'] or '%0' in member['address']: + LOG.error("ccloud: POOL-RDCHECK3 - trying to create member with address: %s", member['address']) part = pool["partition"] - for bigip in bigips: - p = self.pool_helper.load(bigip, - name=pool["name"], - partition=part) - m = p.members_s.members - if m.exists(name=urllib.quote(member["name"]), partition=part): - m = m.load(name=urllib.quote(member["name"]), - partition=part) - member.pop("address", None) - m.modify(**member) + ex = None + for bigip in bigips: + try: + p = self.pool_helper.load(bigip, + name=pool["name"], + partition=part) + + m = p.members_s.members + if m.exists(name=urllib.quote(member["name"]), partition=part): + m = m.load(name=urllib.quote(member["name"]), + partition=part) + member.pop("address", None) + m.modify(**member) + #LOG.info("Member updated: %s", member['address']) + except HTTPError as err: + #LOG.info("Member update FAILED: %s", member['address']) + ex = err + if ex: + raise ex def _get_monitor_helper(self, service): monitor_type = self.service_adapter.get_monitor_type(service) @@ -197,6 +303,35 @@ def _get_monitor_helper(self, service): hm = self.http_mon_helper return hm + def _set_monitor_attributes(self, service, monitor): + monitor_type = self.service_adapter.get_monitor_type(service) + if monitor_type == "HTTPS": + if self.f5_parent_https_monitor: + monitor['defaultsFrom'] = self.f5_parent_https_monitor + + def member_exists(self, service, bigip): + """Return True if a member exists in a pool. + + :param service: Has pool and member name/partition + :param bigip: BIG-IP to get member status from. + :return: Boolean + """ + pool = self.service_adapter.get_pool(service) + member = self.service_adapter.get_member(service) + part = pool["partition"] + try: + p = self.pool_helper.load(bigip, + name=pool["name"], + partition=part) + + m = p.members_s.members + if m.exists(name=urllib.quote(member["name"]), partition=part): + return True + except Exception as e: + # log error but continue on + LOG.error("Error checking member exists: %s", e.message) + return False + def get_member_status(self, service, bigip, status_keys): """Return status values for a single pool. @@ -225,8 +360,7 @@ def get_member_status(self, service, bigip, status_keys): member_status = self.pool_helper.collect_stats( m, stat_keys=status_keys) else: - LOG.error("Unable to get member status. " - "Member %s does not exist.", member["name"]) + LOG.warning("Unable to get member status. Member %s does not exist.", member["name"]) except Exception as e: # log error but continue on diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/resource_helper.py b/f5_openstack_agent/lbaasv2/drivers/bigip/resource_helper.py index f3406f695..5bc50a228 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/resource_helper.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/resource_helper.py @@ -22,7 +22,7 @@ class ResourceType(Enum): - u"""Defines supported BIG-IP® resource types.""" + u"""Defines supported BIG-IP resource types.""" nat = 1 pool = 2 @@ -59,12 +59,16 @@ class ResourceType(Enum): ssl_persistence = 33 universal_persistence = 34 ssl_cert_file = 35 + http_profile = 36 + one_connect_profile = 37 + http_compression_profile = 38 + fastl4_profile = 39 class BigIPResourceHelper(object): - u"""Helper class for creating, updating and deleting BIG-IP® resources. + u"""Helper class for creating, updating and deleting BIG-IP resources. - Reduces some of the boilerplate that surrounds using the F5® SDK. + Reduces some of the boilerplate that surrounds using the F5 SDK. Example usage: bigip = BigIP("10.1.1.1", "admin", "admin") pool = {"name": "pool1", @@ -80,13 +84,13 @@ def __init__(self, resource_type): self.resource_type = resource_type def create(self, bigip, model): - u"""Create/update resource (e.g., pool) on a BIG-IP® system. + u"""Create/update resource (e.g., pool) on a BIG-IP system. First checks to see if resource has been created and creates it if not. :param bigip: BigIP instance to use for creating resource. - :param model: Dictionary of BIG-IP® attributes to add resource. Must + :param model: Dictionary of BIG-IP attributes to add resource. Must include name and partition. :returns: created or updated resource object. """ @@ -101,7 +105,7 @@ def exists(self, bigip, name=None, partition=None): return resource.exists(name=name, partition=partition) def delete(self, bigip, name=None, partition=None): - u"""Delete a resource on a BIG-IP® system. + u"""Delete a resource on a BIG-IP system. Checks if resource exists and deletes it. Returns without error if resource does not exist. @@ -116,10 +120,10 @@ def delete(self, bigip, name=None, partition=None): obj.delete() def load(self, bigip, name=None, partition=None): - u"""Retrieve a BIG-IP® resource from a BIG-IP®. + u"""Retrieve a BIG-IP resource from a BIG-IP. Populates a resource object with attributes for instance on a - BIG-IP® system. + BIG-IP system. :param bigip: BigIP instance to use for creating resource. :param name: Name of resource to load. @@ -130,13 +134,13 @@ def load(self, bigip, name=None, partition=None): return resource.load(name=name, partition=partition) def update(self, bigip, model): - u"""Update a resource (e.g., pool) on a BIG-IP® system. + u"""Update a resource (e.g., pool) on a BIG-IP system. - Modifies a resource on a BIG-IP® system using attributes + Modifies a resource on a BIG-IP system using attributes defined in the model object. :param bigip: BigIP instance to use for creating resource. - :param model: Dictionary of BIG-IP® attributes to update resource. + :param model: Dictionary of BIG-IP attributes to update resource. Must include name and partition in order to identify resource. """ partition = None @@ -147,10 +151,11 @@ def update(self, bigip, model): return resource - def get_resources(self, bigip, partition=None): - u"""Retrieve a collection BIG-IP® of resources from a BIG-IP®. + def get_resources(self, bigip, partition=None, + expand_subcollections=False): + u"""Retrieve a collection BIG-IP of resources from a BIG-IP. - Generates a list of resources objects on a BIG-IP® system. + Generates a list of resources objects on a BIG-IP system. :param bigip: BigIP instance to use for creating resource. :param name: Name of resource to load. @@ -165,10 +170,15 @@ def get_resources(self, bigip, partition=None): raise err if collection: + params = {'params': ''} if partition: - params = { - 'params': get_filter(bigip, 'partition', 'eq', partition) - } + params['params'] = get_filter( + bigip, 'partition', 'eq', partition) + if expand_subcollections and \ + isinstance(params['params'], dict): + params['params']['expandSubcollections'] = 'true' + elif expand_subcollections: + params['params'] += '&expandSubCollections=true' resources = collection.get_collection(requests_params=params) else: resources = collection.get_collection() @@ -246,7 +256,16 @@ def _resource(self, bigip): ResourceType.universal_persistence: lambda bigip: bigip.tm.ltm.persistence.universal, ResourceType.ssl_cert_file: - lambda bigip: bigip.tm.sys.file.ssl_certs.ssl_cert + lambda bigip: bigip.tm.sys.file.ssl_certs.ssl_cert, + ResourceType.http_profile: + lambda bigip: bigip.tm.ltm.profile.https.http, + ResourceType.one_connect_profile: + lambda bigip: bigip.tm.ltm.profile.one_connects.one_connect, + ResourceType.http_compression_profile: + lambda bigip: bigip.tm.ltm.profile.http_compressions.http_compression, + ResourceType.fastl4_profile: + lambda bigip: bigip.tm.ltm.profile.fastl4s.fastl4, + }[self.resource_type](bigip) def _collection(self, bigip): @@ -312,7 +331,16 @@ def _collection(self, bigip): ResourceType.universal_persistence: lambda bigip: bigip.tm.ltm.persistence.universals, ResourceType.ssl_cert_file: - lambda bigip: bigip.tm.sys.file.ssl_certs + lambda bigip: bigip.tm.sys.file.ssl_certs, + ResourceType.http_profile: + lambda bigip: bigip.tm.ltm.profile.https, + ResourceType.one_connect_profile: + lambda bigip: bigip.tm.ltm.profile.one_connects, + ResourceType.http_compression_profile: + lambda bigip: bigip.tm.ltm.profile.http_compressions, + ResourceType.fastl4_profile: + lambda bigip: bigip.tm.ltm.profile.fastl4s, + } if self.resource_type in collection_map: diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/selfips.py b/f5_openstack_agent/lbaasv2/drivers/bigip/selfips.py index cfb72ee73..4b453f889 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/selfips.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/selfips.py @@ -40,7 +40,7 @@ def __init__(self, driver, l2_service, l3_binding): self.selfip_manager = BigIPResourceHelper(ResourceType.selfip) self.network_helper = NetworkHelper() - def _create_bigip_selfip(self, bigip, model): + def _create_bigip_selfip(self, bigip, model, network_id): created = False if self.selfip_manager.exists(bigip, name=model['name'], partition=model['partition']): @@ -59,7 +59,8 @@ def _create_bigip_selfip(self, bigip, model): self.network_helper.add_vlan_to_domain( bigip, name=model['vlan'], - partition=model['partition']) + partition=model['partition'], + rd_name=network_id) self.selfip_manager.create(bigip, model) created = True except HTTPError as err: @@ -132,7 +133,10 @@ def assure_bigip_selfip(self, bigip, service, subnetinfo): "floating": "disabled", "partition": network_folder } - self._create_bigip_selfip(bigip, model) + + network_id = network['id'] + + self._create_bigip_selfip(bigip, model, network_id) if self.l3_binding: self.l3_binding.bind_address(subnet_id=subnet['id'], @@ -140,7 +144,7 @@ def assure_bigip_selfip(self, bigip, service, subnetinfo): def _get_bigip_selfip_address(self, bigip, subnet): u"""Ensure a selfip address is allocated on Neutron network.""" - # Get ip address for selfip to use on BIG-IP®. + # Get ip address for selfip to use on BIG-IP. selfip_address = "" selfip_name = "local-" + bigip.device_name + "-" + subnet['id'] ports = self.driver.plugin_rpc.get_port_by_name(port_name=selfip_name) @@ -203,7 +207,9 @@ def assure_gateway_on_subnet(self, bigip, subnetinfo, traffic_group): 'partition': network_folder } - if not self._create_bigip_selfip(bigip, model): + network_id = network['id'] + + if not self._create_bigip_selfip(bigip, model, network_id): LOG.error("failed to create gateway selfip") if self.l3_binding: diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/service_adapter.py b/f5_openstack_agent/lbaasv2/drivers/bigip/service_adapter.py index a78788e23..f18af5ac7 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/service_adapter.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/service_adapter.py @@ -28,9 +28,9 @@ class UnsupportedProtocolException(Exception): class ServiceModelAdapter(object): - """Class to translate LBaaS service objects to BIG-IP® model objects. + """Class to translate LBaaS service objects to BIG-IP model objects. - Creates BIG-IP® model objects (dictionary of resource attributes) given + Creates BIG-IP model objects (dictionary of resource attributes) given an LBaaS service objet. """ @@ -92,13 +92,19 @@ def get_virtual(self, service): if listener["use_snat"] and self.snat_count() > 0: listener["snat_pool_name"] = self.get_folder_name( loadbalancer["tenant_id"]) + elif listener["use_snat"] and self.snat_count() == -1: + listener["snat_pool_name"] = "lb_" + loadbalancer["id"] + + # transfer session_persistence from pool to listener if "pool" in service and "session_persistence" in service["pool"]: listener["session_persistence"] = \ service["pool"]["session_persistence"] - vip = self._map_virtual(loadbalancer, listener) + vip = self._map_virtual( + loadbalancer, listener, service.get('pool', None)) + self._add_bigip_items(listener, vip) return vip @@ -206,27 +212,38 @@ def _map_healthmonitor(self, loadbalancer, lbaas_healthmonitor): lbaas_healthmonitor["type"] == "HTTPS"): # url path - if "url_path" in lbaas_healthmonitor: - healthmonitor["send"] = ("GET " + - lbaas_healthmonitor["url_path"] + - " HTTP/1.0\\r\\n\\r\\n") - else: - healthmonitor["send"] = "GET / HTTP/1.0\\r\\n\\r\\n" + if "url_path" not in lbaas_healthmonitor or not lbaas_healthmonitor["url_path"]: + lbaas_healthmonitor["url_path"] = "/" + if "http_method" not in lbaas_healthmonitor or not lbaas_healthmonitor["http_method"]: + lbaas_healthmonitor["http_method"] = "GET" + + healthmonitor["send"] = (lbaas_healthmonitor["http_method"] + " " + + lbaas_healthmonitor["url_path"] + + " HTTP/1.0\\r\\n\\r\\n") # expected codes healthmonitor["recv"] = self._get_recv_text( lbaas_healthmonitor) # interval - delay - if "delay" in lbaas_healthmonitor: - healthmonitor["interval"] = lbaas_healthmonitor["delay"] - - # timeout - if "timeout" in lbaas_healthmonitor: - if "max_retries" in lbaas_healthmonitor: - timeout = (int(lbaas_healthmonitor["max_retries"]) * - int(lbaas_healthmonitor["timeout"])) - healthmonitor["timeout"] = timeout + if "delay" not in lbaas_healthmonitor or not lbaas_healthmonitor["delay"]: + lbaas_healthmonitor["delay"] = "5" + healthmonitor["interval"] = lbaas_healthmonitor["delay"] + # ccloud : ignore OS timeout because F5 treats stuff different + # timeout = delay * interval + 1 second + if "max_retries" not in lbaas_healthmonitor or not lbaas_healthmonitor["max_retries"]: + lbaas_healthmonitor["max_retries"] = "3" + + timeout = (int(lbaas_healthmonitor["max_retries"]) * + int(lbaas_healthmonitor["delay"])) + 1 + healthmonitor["timeout"] = timeout + + # timeout OLD logic + # if "timeout" in lbaas_healthmonitor: + # if "max_retries" in lbaas_healthmonitor: + # timeout = (int(lbaas_healthmonitor["max_retries"]) * + # int(lbaas_healthmonitor["timeout"])) + # healthmonitor["timeout"] = timeout return healthmonitor @@ -342,9 +359,15 @@ def _get_lb_method(self, method): else: return 'round-robin' - def _map_virtual(self, loadbalancer, listener): + def _map_virtual(self, loadbalancer, listener, pool=None): vip = self._init_virtual_name(loadbalancer, listener) + if pool: + p = self.init_pool_name(loadbalancer, pool) + vip["pool"] = p["name"] + else: + vip["pool"] = None + vip["description"] = self.get_resource_description(listener) if "protocol" in listener: @@ -380,9 +403,6 @@ def _map_virtual(self, loadbalancer, listener): else: vip["disabled"] = True - if "pool" in listener: - vip["pool"] = listener["pool"] - return vip def get_vlan(self, vip, bigip, network_id): @@ -407,7 +427,7 @@ def _add_bigip_items(self, listener, vip): if 'session_persistence' in listener: persistence_type = listener['session_persistence'] if persistence_type == 'APP_COOKIE': - virtual_type = 'standard' + #virtual_type = 'standard' vip['persist'] = [{'name': 'app_cookie_' + vip['name']}] elif persistence_type == 'SOURCE_IP': @@ -421,10 +441,17 @@ def _add_bigip_items(self, listener, vip): vip['persist'] = [] if virtual_type == 'fastl4': - vip['profiles'] = ['/Common/fastL4'] + vip['profiles'] = ['/Common/cc_fastL4'] else: # add profiles for HTTP, HTTPS, TERMINATED_HTTPS protocols - vip['profiles'] = ['/Common/http', '/Common/oneconnect'] + + default_profiles = utils.get_default_profiles(self.conf,listener['protocol']) + profiles=[] + for profile in default_profiles.values(): + if listener['protocol'] != 'TCP': + profiles.append('/{}/{}'.format(profile.get('partition'), profile.get('name'))) + + vip['profiles'] = profiles # mask if "ip_address" in vip: diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/snats.py b/f5_openstack_agent/lbaasv2/drivers/bigip/snats.py index 70d09eb0f..6d1758cb9 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/snats.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/snats.py @@ -26,6 +26,8 @@ ResourceType from oslo_log import log as logging +#import pdb + LOG = logging.getLogger(__name__) @@ -39,16 +41,19 @@ def __init__(self, driver, l2_service, l3_binding): ResourceType.snat_translation) self.network_helper = NetworkHelper() - def _get_snat_name(self, subnet, tenant_id): + def _get_snat_name(self, subnet, tenant_id, lb_id=None): + + key = self._get_pool_uuid(subnet,lb_id) + # Get the snat name based on HA type if self.driver.conf.f5_ha_type == 'standalone': - return 'snat-traffic-group-local-only-' + subnet['id'] + return 'snat-traffic-group-local-only-' + key elif self.driver.conf.f5_ha_type == 'pair': - return 'snat-traffic-group-1-' + subnet['id'] + return 'snat-traffic-group-1-' + key elif self.driver.conf.f5_ha_type == 'scalen': traffic_group = self.driver.tenant_to_traffic_group(tenant_id) base_traffic_group = os.path.basename(traffic_group) - return 'snat-' + base_traffic_group + '-' + subnet['id'] + return 'snat-' + base_traffic_group + '-' + key LOG.error('Invalid f5_ha_type:%s' % self.driver.conf.f5_ha_type) return '' @@ -65,6 +70,9 @@ def _get_snat_traffic_group(self, tenant_id): LOG.error('Invalid f5_ha_type:%s' % self.driver.conf.f5_ha_type) return '' + def get_snats(self, bigip, partition=None): + return self.snatpool_manager.get_resources(bigip, partition) + def get_snat_addrs(self, subnetinfo, tenant_id, snat_count): # Get the ip addresses for snat """ subnet = subnetinfo['subnet'] @@ -98,41 +106,67 @@ def get_snat_addrs(self, subnetinfo, tenant_id, snat_count): return snat_addrs - def assure_bigip_snats(self, bigip, subnetinfo, snat_addrs, tenant_id): + def assure_bigip_snats(self, bigip, subnetinfo, snat_addrs, tenant_id,lb_id=None): # Ensure Snat Addresses are configured on a bigip. # Called for every bigip only in replication mode. # otherwise called once and synced. network = subnetinfo['network'] snat_info = {} - if self.l2_service.is_common_network(network): + + + # if we config to -1 for VIP pool IP then we use tenant partition + + if self.driver.conf.f5_snat_addresses_per_subnet > 0: snat_info['network_folder'] = 'Common' - else: + elif self.driver.conf.f5_snat_addresses_per_subnet == -1: snat_info['network_folder'] = ( self.driver.service_adapter.get_folder_name(tenant_id) ) - snat_info['pool_name'] = self.driver.service_adapter.get_folder_name( - tenant_id - ) - snat_info['pool_folder'] = self.driver.service_adapter.get_folder_name( - tenant_id - ) + + snat_info['pool_name'] = self._get_snat_pool_name(tenant_id,lb_id) + + + snat_info['pool_folder'] = self.driver.service_adapter.get_folder_name(tenant_id) + snat_info['addrs'] = snat_addrs - self._assure_bigip_snats(bigip, subnetinfo, snat_info, tenant_id) + self._assure_bigip_snats(bigip, subnetinfo, snat_info, tenant_id,lb_id) + + # try to delete any incorrectly named SNAT pools + if self.driver.conf.f5_snat_addresses_per_subnet == -1: + pool_name = self._get_snat_pool_name(tenant_id) + elif self.driver.conf.f5_snat_addresses_per_subnet > 0: + pool_name = "lb_"+lb_id + + try: + if self.snatpool_manager.exists(bigip,name=pool_name, partition=snat_info['pool_folder']): + snatpool = self.snatpool_manager.load(bigip, pool_name, snat_info['pool_folder']) + if snatpool is not None: + snatpool.delete() + + except Exception as exc: + pass + - def _assure_bigip_snats(self, bigip, subnetinfo, snat_info, tenant_id): + + + + def _assure_bigip_snats(self, bigip, subnetinfo, snat_info, tenant_id,lb_id=None): # Configure the ip addresses for snat network = subnetinfo['network'] subnet = subnetinfo['subnet'] + key = self._get_pool_uuid(subnet,lb_id) + + if tenant_id not in bigip.assured_tenant_snat_subnets: bigip.assured_tenant_snat_subnets[tenant_id] = [] - if subnet['id'] in bigip.assured_tenant_snat_subnets[tenant_id]: + if key in bigip.assured_tenant_snat_subnets[tenant_id]: return - snat_name = self._get_snat_name(subnet, tenant_id) + snat_name = self._get_snat_name(subnet, tenant_id, lb_id) for i, snat_address in enumerate(snat_info['addrs']): ip_address = snat_address + \ '%' + str(network['route_domain_id']) @@ -198,9 +232,10 @@ def _assure_bigip_snats(self, bigip, subnetinfo, snat_info, tenant_id): self.l3_binding.bind_address(subnet_id=subnet['id'], ip_address=ip_address) - bigip.assured_tenant_snat_subnets[tenant_id].append(subnet['id']) - def delete_bigip_snats(self, bigip, subnetinfo, tenant_id): + bigip.assured_tenant_snat_subnets[tenant_id].append(key) + + def delete_bigip_snats(self, bigip, subnetinfo, tenant_id, lb_id=None): # Assure shared snat configuration (which syncs) is deleted. # if not subnetinfo['network']: @@ -208,46 +243,58 @@ def delete_bigip_snats(self, bigip, subnetinfo, tenant_id): 'for missing network ... skipping.') return set() - return self._delete_bigip_snats(bigip, subnetinfo, tenant_id) + return self._delete_bigip_snats(bigip, subnetinfo, tenant_id, lb_id) - def _remove_assured_tenant_snat_subnet(self, bigip, tenant_id, subnet): + def _remove_assured_tenant_snat_subnet(self, bigip, tenant_id, subnet,lb_id=None): # Remove ref for the subnet for this tenant""" + + key = self._get_pool_uuid(subnet,lb_id) + if tenant_id in bigip.assured_tenant_snat_subnets: tenant_snat_subnets = \ bigip.assured_tenant_snat_subnets[tenant_id] - if tenant_snat_subnets and subnet['id'] in tenant_snat_subnets: + if tenant_snat_subnets and key in tenant_snat_subnets: LOG.debug( 'Remove subnet id %s from ' 'bigip.assured_tenant_snat_subnets for tenant %s' % (subnet['id'], tenant_id)) - tenant_snat_subnets.remove(subnet['id']) + tenant_snat_subnets.remove(key) else: LOG.debug( 'Subnet id %s does not exist in ' 'bigip.assured_tenant_snat_subnets for tenant %s' % - (subnet['id'], tenant_id)) + (key, tenant_id)) else: LOG.debug( 'Tenant id %s does not exist in ' 'bigip.assured_tenant_snat_subnets' % tenant_id) - def _delete_bigip_snats(self, bigip, subnetinfo, tenant_id): + def _delete_bigip_snats(self, bigip, subnetinfo, tenant_id, lb_id=None): # Assure snats deleted in standalone mode """ subnet = subnetinfo['subnet'] - network = subnetinfo['network'] - if self.l2_service.is_common_network(network): - partition = 'Common' - else: - partition = self.driver.service_adapter.get_folder_name(tenant_id) + key = self._get_pool_uuid(subnet,lb_id) + + if self.driver.conf.f5_snat_addresses_per_subnet == -1 and lb_id is not None: + partition = ( + self.driver.service_adapter.get_folder_name(tenant_id) + ) - snat_pool_name = self.driver.service_adapter.get_folder_name(tenant_id) - snat_pool_folder = snat_pool_name + + snat_pool_name = self._get_snat_pool_name(tenant_id,lb_id) + snat_pool_folder = self.driver.service_adapter.get_folder_name(tenant_id) deleted_names = set() in_use_subnets = set() # Delete SNATs on traffic-group-local-only - snat_name = self._get_snat_name(subnet, tenant_id) - for i in range(self.driver.conf.f5_snat_addresses_per_subnet): + snat_name = self._get_snat_name(subnet, tenant_id,key) + + count = 0 + if self.driver.conf.f5_snat_addresses_per_subnet > 0: + count = self.driver.conf.f5_snat_addresses_per_subnet + elif self.driver.conf.f5_snat_addresses_per_subnet == -1: + count = 1 + + for i in range(count): index_snat_name = snat_name + "_" + str(i) tmos_snat_name = index_snat_name @@ -303,22 +350,25 @@ def _delete_bigip_snats(self, bigip, subnetinfo, tenant_id): except HTTPError as err: LOG.error("Update SNAT pool failed %s" % err.message) except HTTPError as err: - LOG.error("Failed to load SNAT pool %s" % err.message) + if err.response.status_code == 404: + LOG.info("ccloud: Failed to load SNAT pool for deletion %s" % err.message) + else: + LOG.error("Failed to load SNAT pool %s" % err.message) # Check if subnet in use by any tenants/snatpools. If in use, # add subnet to hints list of subnets in use. - self._remove_assured_tenant_snat_subnet(bigip, tenant_id, subnet) + self._remove_assured_tenant_snat_subnet(bigip, tenant_id, subnet,lb_id) LOG.debug( - 'Check cache for subnet %s in use by other tenant' % - subnet['id']) + 'Check cache for pool %s in use by other tenant' % + key) in_use_count = 0 for loop_tenant_id in bigip.assured_tenant_snat_subnets: tenant_snat_subnets = \ bigip.assured_tenant_snat_subnets[loop_tenant_id] - if subnet['id'] in tenant_snat_subnets: + if key in tenant_snat_subnets: LOG.debug( - 'Subnet %s in use (tenant %s)' % - (subnet['id'], loop_tenant_id)) + 'Pool %s in use (tenant %s)' % + (key, loop_tenant_id)) in_use_count += 1 if in_use_count: @@ -327,7 +377,7 @@ def _delete_bigip_snats(self, bigip, subnetinfo, tenant_id): LOG.debug('Check subnet in use by any tenant') member_use_count = \ self.get_snatpool_member_use_count( - bigip, subnet['id']) + bigip, key) if member_use_count: LOG.debug('Subnet in use - do not delete') in_use_subnets.add(subnet['id']) @@ -356,3 +406,17 @@ def get_snatpool_member_use_count(self, bigip, member_name): if member_name == os.path.basename(member): snat_count += 1 return snat_count + + def _get_snat_pool_name(self, tenant_id, lb_id=None): + if lb_id is not None and self.driver.conf.f5_snat_addresses_per_subnet == -1: + return "lb_"+lb_id + else: + return self.driver.service_adapter.get_folder_name( + tenant_id + ) + + def _get_pool_uuid(self,subnet, lb_id=None): + if lb_id is not None and self.driver.conf.f5_snat_addresses_per_subnet == -1: + return lb_id + else: + return subnet['id'] \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/ssl_profile.py b/f5_openstack_agent/lbaasv2/drivers/bigip/ssl_profile.py index 18018b5e3..ccabc11a9 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/ssl_profile.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/ssl_profile.py @@ -28,14 +28,20 @@ class SSLProfileHelper(object): @staticmethod def create_client_ssl_profile( - bigip, name, cert, key, sni_default=False, parent_profile=None): + bigip, name, cert, key, intermediate=None, sni_default=False, parent_profile=None, caClientTrust=False, + key_passphrase=None): uploader = bigip.shared.file_transfer.uploads cert_registrar = bigip.tm.sys.crypto.certs + intermediate_registrar = bigip.tm.sys.crypto.certs key_registrar = bigip.tm.sys.crypto.keys ssl_client_profile = bigip.tm.ltm.profile.client_ssls.client_ssl + profilename = name + if not sni_default: + profilename = name + '_NotDefault' + # No need to create if it exists - if ssl_client_profile.exists(name=name, partition='Common'): + if ssl_client_profile.exists(name=profilename, partition='Common'): return # Check that parent profile exists; use default if not. @@ -45,12 +51,17 @@ def create_client_ssl_profile( certfilename = name + '.crt' keyfilename = name + '.key' + # we need both names because uploader fiddles around with names + intermediatefilename = name + '.chain' + intermediatecrtfilename = intermediatefilename + '.crt' try: # In-memory upload -- data not written to local file system but # is saved as a file on the BIG-IP. uploader.upload_bytes(cert, certfilename) uploader.upload_bytes(key, keyfilename) + if intermediate: + uploader.upload_bytes(intermediate, intermediatefilename) # import certificate param_set = {} @@ -65,15 +76,55 @@ def create_client_ssl_profile( '/var/config/rest/downloads/', keyfilename) key_registrar.exec_cmd('install', **param_set) + if intermediate: + # import intermediates + param_set = {} + param_set['name'] = intermediatefilename + param_set['from-local-file'] = os.path.join( + '/var/config/rest/downloads/', intermediatefilename) + intermediate_registrar.exec_cmd('install', **param_set) + # create ssl-client profile from cert/key pair - chain = [{'name': name, + if intermediate: + chain = [{'name': name, + 'cert': '/Common/' + certfilename, + 'key': '/Common/' + keyfilename, + 'chain': '/Common/' + intermediatecrtfilename}] + # create ssl-client profile from cert/key pair + else: + chain = [{'name': name, 'cert': '/Common/' + certfilename, 'key': '/Common/' + keyfilename}] - ssl_client_profile.create(name=name, + + if key_passphrase: + chain[0]['passphrase'] = key_passphrase + + if caClientTrust and intermediate: + ssl_client_profile.create(name=profilename, + partition='Common', + certKeyChain=chain, + sniDefault=sni_default, + defaultsFrom=parent_profile, + clientCertCa=intermediatecrtfilename, + caFile=intermediatecrtfilename) + LOG.info("Creating SSL profile WITH caClientTrust and WITH intermediate %s", chain) + elif (not caClientTrust) and intermediate: + ssl_client_profile.create(name=profilename, partition='Common', certKeyChain=chain, sniDefault=sni_default, defaultsFrom=parent_profile) + LOG.info("Creating SSL profile WITHOUT caClientTrust and WITH intermediate %s", chain) + elif (not caClientTrust) and (not intermediate): + ssl_client_profile.create(name=profilename, + partition='Common', + certKeyChain=chain, + sniDefault=sni_default, + defaultsFrom=parent_profile) + LOG.info("Creating SSL profile WITHOUT caClientTrust and WITHOUT intermediate %s", chain) + else: + LOG.error("ERROR: Cannot create a SSL profile WITH caClientTrust and WITHOUT intermediate") + raise SSLProfileError("ERROR: Cannot create a SSL profile WITH caClientTrust and WITHOUT intermediate") except Exception as err: LOG.error("Error creating SSL profile: %s" % err.message) raise SSLProfileError(err.message) diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/system_helper.py b/f5_openstack_agent/lbaasv2/drivers/bigip/system_helper.py index 2bbaa1b00..49f6dc53c 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/system_helper.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/system_helper.py @@ -48,9 +48,11 @@ def folder_exists(self, bigip, folder): def get_folders(self, bigip): f_collection = [] - folders = bigip.tm.sys.folders.get_collection() + # example: bigip.tm.sys.folders.get_collection(requests_params={'params': '$select=name'}) + folders = bigip.tm.sys.folders.get_collection(requests_params={'params': '$select=name'}) for folder in folders: - f_collection.append(folder.name) + if 'name' in folder: + f_collection.append(folder['name']) return f_collection @@ -140,6 +142,7 @@ def purge_folder_contents(self, bigip, folder): ltm_types = [ ResourceType.virtual, ResourceType.virtual_address, + ResourceType.l7policy, ResourceType.pool, ResourceType.http_monitor, ResourceType.https_monitor, @@ -149,6 +152,7 @@ def purge_folder_contents(self, bigip, folder): ResourceType.snat, ResourceType.snatpool, ResourceType.snat_translation, + ResourceType.universal_persistence, ResourceType.rule ] for ltm_type in ltm_types: diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/tenants.py b/f5_openstack_agent/lbaasv2/drivers/bigip/tenants.py index 62abb82d0..45cd29b1c 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/tenants.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/tenants.py @@ -14,6 +14,7 @@ # limitations under the License. # +import constants_v2 as const from oslo_log import log as logging from f5_openstack_agent.lbaasv2.drivers.bigip import exceptions as f5ex @@ -47,6 +48,7 @@ def assure_tenant_created(self, service): adaptations shouldn't we be using ServiceModelAdapter... though I suppose this is the other way. """ + network_id = service['loadbalancer']['network_id'] tenant_id = service['loadbalancer']['tenant_id'] traffic_group = self.driver.service_to_traffic_group(service) traffic_group = '/Common/' + traffic_group @@ -71,20 +73,54 @@ def assure_tenant_created(self, service): (tenant_id)) # create tenant route domain + + # ccloud: change of rd creartion to avoid different id's on the bigip pair members if self.conf.use_namespaces: + # Determine which bigip needs a rd and if the rd is already created somewhere else so that id should be used + route_domain_id = None + bigiprds = [] for bigip in self.driver.get_all_bigips(): - if not self.network_helper.route_domain_exists(bigip, - folder_name): - try: - self.network_helper.create_route_domain( - bigip, - folder_name, - self.conf.f5_route_domain_strictness) - except Exception as err: - LOG.exception(err.message) - raise f5ex.RouteDomainCreationException( - "Failed to create route domain for " - "tenant in %s" % (folder_name)) + bigip_route_domain = self.network_helper.route_domain_exists(bigip, const.DEFAULT_PARTITION, network_id) + bigip_route_domain_id = bigip_route_domain.id if bigip_route_domain else None + # rd already created but not different between bigips (maybe not created on all of them) + if bigip_route_domain_id and route_domain_id is None: + route_domain_id = bigip_route_domain_id + # rd already created on different bigips with DIFFERENT id --> ERROR + elif bigip_route_domain_id and route_domain_id and bigip_route_domain_id != bigip_route_domain_id: + LOG.error("Route Domain Failure: RD for network %s is defined with ID %s on one and with %s on another Bigip" + % (network_id, bigip_route_domain_id, route_domain_id)) + raise f5ex.RouteDomainCreationException("Route Domain Failure: RD for network %s is defined with ID %s on one and with %s on another Bigip" + % (network_id, bigip_route_domain_id, route_domain_id)) + # rd not created anywhere until now + elif bigip_route_domain_id is None: + bigiprds.append(bigip) + # now we have the bigip's with the missing rd and an rd id in case it's created on one of the bigip's + # create rd in bigips where it's missing either with the given id or a new one to be determined + for bigip in bigiprds: + try: + bigip_route_domain = self.network_helper.create_route_domain( + bigip, + partition=const.DEFAULT_PARTITION, + name=network_id, + strictness=self.conf.f5_route_domain_strictness, + is_aux=False, + rd_id=route_domain_id) + # use newly created id for next bigip + if route_domain_id is None: + route_domain_id = bigip_route_domain.id + # something went wrong + elif bigip_route_domain.id != route_domain_id: + LOG.error("Route Domain Failure: Attempt to create RD for network %s with ID %s on one and with %s on another Bigip" + % (network_id, bigip_route_domain_id, route_domain_id)) + raise f5ex.RouteDomainCreationException("Route Domain Failure: RD for network %s is defined with ID %s on one and with %s on another Bigip" + % (network_id, bigip_route_domain_id, route_domain_id)) + # error within rd creation procedure + except Exception as err: + LOG.exception(err.message) + raise f5ex.RouteDomainCreationException("Failed to create route domain for network %s in tenant %s" % (network_id, const.DEFAULT_PARTITION)) + + LOG.debug("Allocated route domain for network %s for tenant %s" + % (network_id, tenant_id)) def assure_tenant_cleanup(self, service, all_subnet_hints): """Delete tenant partition.""" @@ -98,28 +134,26 @@ def assure_tenant_cleanup(self, service, all_subnet_hints): # otherwise called once def _assure_bigip_tenant_cleanup(self, bigip, service, subnet_hints): tenant_id = service['loadbalancer']['tenant_id'] + network_id = service['loadbalancer']['network_id'] - self._remove_tenant_replication_mode(bigip, tenant_id) + self._remove_tenant_replication_mode(bigip, tenant_id, network_id) - def _remove_tenant_replication_mode(self, bigip, tenant_id): + def _remove_tenant_replication_mode(self, bigip, tenant_id, network_id): # Remove tenant in replication sync-mode partition = self.service_adapter.get_folder_name(tenant_id) - domain_names = self.network_helper.get_route_domain_names(bigip, - partition) - for domain_name in domain_names: - try: - self.network_helper.delete_route_domain(bigip, - partition, - domain_name) - except Exception as err: - LOG.error("Failed to delete route domain %s. " - "%s. Manual intervention might be required." - % (domain_name, err.message)) + + + try: + self.network_helper.delete_route_domain(bigip, + "Common", + network_id) + except Exception as err: + LOG.info("Failed to delete route domain %s. " + "Manual intervention might be required." % (network_id)) try: self.system_helper.delete_folder(bigip, partition) except Exception as err: - LOG.error( - "Folder deletion exception for tenant partition %s occurred. " - "Manual cleanup might be required." % (tenant_id)) - LOG.exception("%s" % err.message) + LOG.info( + "Folder deletion failed for tenant partition %s. " + "Manual cleanup might be required." % (tenant_id)) \ No newline at end of file diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/utils.py b/f5_openstack_agent/lbaasv2/drivers/bigip/utils.py index 238ef3db6..5f26f5109 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/utils.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/utils.py @@ -15,10 +15,15 @@ # from time import time import uuid +import eventlet from distutils.version import LooseVersion from eventlet import greenthread + from oslo_log import log as logging +from neutron_lbaas.services.loadbalancer import constants as lb_const + +from threading import Thread, current_thread LOG = logging.getLogger(__name__) OBJ_PREFIX = 'uuid_' @@ -28,6 +33,44 @@ class IpNotInCidrNotation(Exception): pass +def get_default_profiles(conf, listener_protocol): + http_defaults = {lb_const.PROTOCOL_HTTP:conf.f5_default_http_profile,lb_const.PROTOCOL_HTTPS:conf.f5_default_https_profile,lb_const.PROTOCOL_TERMINATED_HTTPS:conf.f5_default_terminated_https_profile} + + http_profile = http_defaults.get(listener_protocol,'/Common/http') + oneconnect_profile= conf.f5_default_oneconnect_profile + + result = {} + + + + l = http_profile[1:].split("/") + if len(l)==2: + result['http']={'partition': l[0], 'name': l[1],'context':'all'} + + l = oneconnect_profile[1:].split("/") + if len(l)==2: + result['oneconnect']={'partition': l[0], 'name': l[1],'context':'all'} + + + return result + + +def instrument_execution_time(method): + + def timed(*args, **kw): + ts = time() + result = method(*args, **kw) + te = time() + + LOG.debug('******* PROFILE %r : %r %2.2f sec' % \ + (current_thread().name, method.__name__, te-ts)) + return result + + return timed + + + + def strip_domain_address(ip_address): """Return the address or address/netmask from a route domain address. @@ -45,7 +88,6 @@ def strip_domain_address(ip_address): else: return ip_address.split('%')[0] - def serialized(method_name): """Outer wrapper in order to specify method name.""" def real_serialized(method): @@ -56,13 +98,13 @@ def wrapper(*args, **kwargs): service_queue = args[0].service_queue my_request_id = uuid.uuid4() - service = None - if len(args) > 0: - last_arg = args[-1] - if isinstance(last_arg, dict) and ('loadbalancer' in last_arg): - service = last_arg - if 'service' in kwargs: - service = kwargs['service'] + # service = None + # if len(args) > 0: + # last_arg = args[-1] + # if isinstance(last_arg, dict) and ('loadbalancer' in last_arg): + # service = last_arg + # if 'service' in kwargs: + # service = kwargs['service'] # Consolidate create_member requests for the same pool. # @@ -74,7 +116,8 @@ def wrapper(*args, **kwargs): # To avoid race conditions, DO NOT add logging to this code # block. - req = (my_request_id, method_name, service) + #req = (my_request_id, method_name, service) + req = my_request_id service_queue.append(req) reqs_ahead_of_us = request_index(service_queue, my_request_id) while reqs_ahead_of_us != 0: @@ -112,7 +155,15 @@ def wrapper(*args, **kwargs): def request_index(request_queue, request_id): """Get index of request in request queue. + If we are not in the queue return the length of the list. + """ + try: + return request_queue.index(request_id) + except Exception: + return len(request_queue) +def request_index_xxx(request_queue, request_id): + """Get index of request in request queue. If we are not in the queue return the length of the list. """ for request in request_queue: @@ -121,6 +172,68 @@ def request_index(request_queue, request_id): return len(request_queue) +def serialized_ccloud (method_name): + """Outer wrapper in order to specify method name.""" + + def real_serialized(method): + """Decorator to serialize calls to configure via iControl.""" + def wrapper(*args, **kwargs): + # args[0] must be an instance of iControlDriver + my_request_id = uuid.uuid4() + + service = None + if len(args) > 0: + last_arg = args[-1] + if isinstance(last_arg, dict) and ('loadbalancer' in last_arg): + service = last_arg + if 'service' in kwargs: + service = kwargs['service'] + + lb_id = 'generic' + + if service is not None: + lb = service.get('loadbalancer') + if lb is not None: + lb_id = lb.get('id') + + queue = args[0].queues.get(lb_id) + if queue is None: + queue = eventlet.queue.Queue(1) + args[0].queues[lb_id] = queue + + wait_start = time() + queue.put(my_request_id) + LOG.debug('Waited %.2f secs to put request %s on to queue %s %s' + % (time() - wait_start, my_request_id, lb_id, queue)) + + + try: + + start_time = time() + + result = method(*args, **kwargs) + LOG.debug('%s request %s took %.5f secs' + % (str(method_name), my_request_id, + time() - start_time)) + + except Exception: + LOG.error('%s request %s FAILED' + % (str(method_name), my_request_id)) + raise + finally: + wait_start = time() + wait_request = queue.get() + LOG.debug('Waited %.2f secs to get request %s' + % (time() - wait_start,wait_request)) + + + return result + + return wrapper + + return real_serialized + + def get_filter(bigip, key, op, value): if LooseVersion(bigip.tmos_version) < LooseVersion('11.6.0'): return '$filter=%s+%s+%s' % (key, op, value) diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/virtual_address.py b/f5_openstack_agent/lbaasv2/drivers/bigip/virtual_address.py index 4bb073e01..aa41267b7 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/virtual_address.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/virtual_address.py @@ -24,9 +24,9 @@ class VirtualAddress(object): - u"""Class to translate LBaaS loadbalancer objects to BIG-IP® virtual address. + u"""Class to translate LBaaS loadbalancer objects to BIG-IP virtual address. - Creates BIG-IP® virtual address objects given an LBaaS service object. + Creates BIG-IP virtual address objects given an LBaaS service object. """ def __init__(self, adapter, loadbalancer): @@ -101,12 +101,20 @@ def load(self, bigip): def update(self, bigip): - # Get the model object, pop immutables and update - model = self.model() - model.pop("address") - va = self.virtual_address.update(bigip, model) - return va + model = self.model() + remote = self.load(bigip) + if remote.address != model["address"]: + # could be route domain or IP has changed + try: + self.delete(bigip) + except: + LOG.error("Failed to deleted redundant virtual address %s", remote) + return self.create(bigip) + else: + # pop immutables and update + model.pop("address") + return self.virtual_address.update(bigip, model) def assure(self, bigip, delete=False): diff --git a/f5_openstack_agent/lbaasv2/drivers/bigip/vlan_binding.py b/f5_openstack_agent/lbaasv2/drivers/bigip/vlan_binding.py index d13796150..562d96563 100644 --- a/f5_openstack_agent/lbaasv2/drivers/bigip/vlan_binding.py +++ b/f5_openstack_agent/lbaasv2/drivers/bigip/vlan_binding.py @@ -44,8 +44,8 @@ def __init__(self, conf, driver): LOG.debug('interface_port_static_mappings not configured') def register_bigip_interfaces(self): - # Delayed binding BIG-IP® ports will be called - # after BIG-IP® endpoints are registered. + # Delayed binding BIG-IP ports will be called + # after BIG-IP endpoints are registered. if not self.__initialized__bigip_ports: for bigip in self.driver.get_all_bigips(): diff --git a/requirements.functest.txt b/requirements.functest.txt index 83f12d919..8ab33f95c 100644 --- a/requirements.functest.txt +++ b/requirements.functest.txt @@ -16,4 +16,4 @@ pytest-cov>=2.4.0,<3 responses==0.5.1 coverage==4.2 python-coveralls==2.8.0 -requests<=2.12.5 +requests>=2.20.0 diff --git a/setup.py b/setup.py index 3ae72b4b5..79d332a92 100644 --- a/setup.py +++ b/setup.py @@ -24,6 +24,7 @@ author_email="f5_openstack_agent@f5.com", data_files=[('/etc/neutron/services/f5', ['etc/neutron/services/f5/f5-openstack-agent.ini']), ('/etc/neutron/services/f5/esd', ['etc/neutron/services/f5/esd/demo.json']), + ('/etc/neutron/services/f5/esd', ['etc/neutron/services/f5/esd/esd.json']), ('/etc/init.d', ['etc/init.d/f5-oslbaasv2-agent']), ('/usr/lib/systemd/system', ['lib/systemd/system/f5-openstack-agent.service']), ('/usr/bin/f5', ['bin/debug_bundler.py'])], @@ -40,7 +41,8 @@ ], entry_points={ 'console_scripts': [ - 'f5-oslbaasv2-agent = f5_openstack_agent.lbaasv2.drivers.bigip.agent:main' + 'f5-oslbaasv2-agent = f5_openstack_agent.lbaasv2.drivers.bigip.agent:main', + 'f5-utils = f5_openstack_agent.lbaasv2.drivers.bigip.cli.f5_cli_utils:main' ] }, install_requires=['f5-sdk==2.3.3'] diff --git a/test/send_to_driver/test_listener.py b/test/send_to_driver/test_listener.py index 60fd642ee..79448a4e3 100644 --- a/test/send_to_driver/test_listener.py +++ b/test/send_to_driver/test_listener.py @@ -37,7 +37,7 @@ def test_create_listener(bigip): # create partition lb_service.prep_service(service, bigips) - # create BIG-IP® virtual servers + # create BIG-IP virtual servers listeners = service["listeners"] loadbalancer = service["loadbalancer"]