Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions data/data/bootstrap/baremetal/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@ internal to the cluster as possible.
There is a DNS VIP managed by `keepalived` in a manner similar to the API VIP
discussed above.

`coredns` runs with a custom `mdns` plugin (`coredns-mdns`).

Relevant files:
* **[files/etc/dhcp/dhclient.conf](files/etc/dhcp/dhclient.conf)** - Specify
that the bootstrap VM should use `localhost` as its primary DNS server.
2 changes: 0 additions & 2 deletions data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ CLUSTER_BOOTSTRAP_IMAGE=$(image_for cluster-bootstrap)

KEEPALIVED_IMAGE=$(image_for keepalived-ipfailover || echo "no-keepalived-image")
COREDNS_IMAGE=$(image_for coredns)
MDNS_PUBLISHER_IMAGE=$(image_for mdns-publisher)
HAPROXY_IMAGE=$(image_for haproxy-router)
BAREMETAL_RUNTIMECFG_IMAGE=$(image_for baremetal-runtimecfg)

Expand Down Expand Up @@ -299,7 +298,6 @@ then
--infra-image="${MACHINE_CONFIG_INFRA_IMAGE}" \
--keepalived-image="${KEEPALIVED_IMAGE}" \
--coredns-image="${COREDNS_IMAGE}" \
--mdns-publisher-image="${MDNS_PUBLISHER_IMAGE}" \
--haproxy-image="${HAPROXY_IMAGE}" \
--baremetal-runtimecfg-image="${BAREMETAL_RUNTIMECFG_IMAGE}" \
--release-image="${RELEASE_IMAGE_DIGEST}" \
Expand Down
11 changes: 0 additions & 11 deletions data/data/openstack/topology/sg-master.tf
Original file line number Diff line number Diff line change
Expand Up @@ -62,17 +62,6 @@ resource "openstack_networking_secgroup_rule_v2" "master_ingress_dns_udp" {
description = local.description
}

resource "openstack_networking_secgroup_rule_v2" "master_ingress_mdns_udp" {
direction = "ingress"
ethertype = "IPv4"
protocol = "udp"
port_range_min = 5353
port_range_max = 5353
remote_ip_prefix = var.cidr_block
security_group_id = openstack_networking_secgroup_v2.master.id
description = local.description
}

resource "openstack_networking_secgroup_rule_v2" "master_ingress_https" {
direction = "ingress"
ethertype = "IPv4"
Expand Down
11 changes: 0 additions & 11 deletions data/data/openstack/topology/sg-worker.tf
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,6 @@ resource "openstack_networking_secgroup_rule_v2" "worker_ingress_ssh" {
description = local.description
}

resource "openstack_networking_secgroup_rule_v2" "worker_ingress_mdns_udp" {
direction = "ingress"
ethertype = "IPv4"
protocol = "udp"
port_range_min = 5353
port_range_max = 5353
remote_ip_prefix = var.cidr_block
security_group_id = openstack_networking_secgroup_v2.worker.id
description = local.description
}

resource "openstack_networking_secgroup_rule_v2" "worker_ingress_http" {
direction = "ingress"
ethertype = "IPv4"
Expand Down
47 changes: 12 additions & 35 deletions docs/design/baremetal/networking-infrastructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,54 +94,31 @@ before starting the installation process.
In a `baremetal` environment, we do not know the IP addresses of all hosts in
advance. Those will come from an organization’s DHCP server. Further, we can
not rely on being able to program an organization’s DNS infrastructure in all
cases. We address these challenges in two ways:

1. Self host some DNS infrastructure to provide DNS resolution for records only
needed internal to the cluster.
2. Make use of mDNS (Multicast DNS) to dynamically discover the addresses of
hosts that we must resolve records for.
cases. We address these challenges by self hosting a DNS server to provide DNS
resolution for records internal to the cluster.

### api-int hostname resolution

The CoreDNS server performing our internal DNS resolution includes
configuration to resolve the `api-int` hostname. `api-int` will be resolved to
the API VIP.

### mdns-publisher

https://github.com/openshift/mdns-publisher

The `mdns-publisher` is the component that runs on each host to make itself
discoverable by other hosts in the cluster. Control plane hosts currently
advertise `master-NN` names, and the worker nodes advertise
`worker-NN` names. *Note: The `master-NN` and `worker-NN` names were based on records
previously published on AWS clusters, but may no longer be necessary.*

On masters and workers it is run by the `machine-config-operator`.

`mdns-publisher` is not run on the bootstrap node, as there is no need for any
other host to discover the IP address that the bootstrap VM gets from DHCP.

### coredns-mdns

https://github.com/openshift/coredns-mdns/
### nodes hostname resolution

The `mdns` plugin for `coredns` was developed to perform DNS lookups
based on discoverable information from mDNS. The plugin will resolve the
`master-NN` and `worker-NN` records in the cluster domain.
The same CoreDNS server also resolves the `master-NN` and `worker-NN` records
in the cluster domain.

The IP addresses that the `master-NN` host records resolve to comes from the
mDNS advertisement sent out by the `mdns-publisher` on that control plane node.
The IP addresses that the `master-NN` and `worker-NN` host records resolve to
comes from querying the OpenShift API.

### DNS Resolution

Because the baremetal platform does not have a cloud DNS service available to
provide internal DNS records, it instead uses a coredns static pod configured
with the `coredns-mdns` plugin discussed above. There is one of these pods
running on every node in a deployment, and a NetworkManager dispatcher script
is used to configure resolv.conf to point at the node's public IP address.
`localhost` can't be used because `resolv.conf` is propagated into some
containers where that won't resolve to the actual host.
provide internal DNS records, it instead uses a coredns static pod. There is
one of these pods running on every node in a deployment, and a NetworkManager
dispatcher script is used to configure resolv.conf to point at the node's
public IP address. `localhost` can't be used because `resolv.conf` is
propagated into some containers where that won't resolve to the actual host.

### Bootstrap Asset Details

Expand Down
23 changes: 4 additions & 19 deletions docs/design/openstack/networking-infrastructure.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# OpenStack IPI Networking Infrastructure

The `OpenStack` platform installer uses an internal networking solution that
is based heavily on the [baremetal networking infrastructure](../baremetal/networking-infrastructure.md).
The `OpenStack` platform installer uses an internal networking solution identical to
the [baremetal networking infrastructure](../baremetal/networking-infrastructure.md).
For an overview of the quotas required, and the entrypoints created when
you build an OpenStack IPI cluster, see the [user docs](../../user/openstack/README.md).

Expand All @@ -15,21 +15,6 @@ These services are initially hosted by the bootstrap node until the control
plane is up. Then, control is pivoted to the control plane machines. We will go into further detail on
that process in the [Virtual IPs section](#virtual-ips).

## CoreDNS-mDNS

https://github.com/openshift/CoreDNS-mdns/

The `mDNS` plugin for `CoreDNS` was developed to perform DNS lookups
based on discoverable information from mDNS. This plugin will resolve both the
`etcd-NNN` records, as well as the `_etcd-server-ssl._tcp.` SRV record. It is also
able to resolve the name of the nodes.

The list of `etcd` hosts included in the SRV record is based on the list of
control plane nodes currently running.

The IP addresses that the `etcd-NNN` host records resolve to comes from the
mDNS advertisement sent out by the `mDNS-publisher` on that control plane node.

## Virtual IPs

We use virtual IP addresses, VIPs, managed by Keepalived to provide high
Expand Down Expand Up @@ -61,9 +46,9 @@ The bootstrap node is responsible for running temporary networking infrastructur
nodes are still coming up. The bootstrap node will run a CoreDNS instance, as well as
Keepalived. While the bootstrap node is up, it will have priority running the API VIP.

The Master nodes run dhcp, HAProxy, CoreDNS, mDNS-publisher, and Keepalived. Haproxy loadbalances incoming requests
The Master nodes run dhcp, HAProxy, CoreDNS, and Keepalived. Haproxy loadbalances incoming requests
to the API across all running masters. It also runs a stats and healthcheck server. Keepalived manages both VIPs on the master, where each
master has an equal chance of being assigned one of the VIPs. Initially, the bootstrap node has the highest priority for hosting the API VIP, so they will point to addresses there at startup. Meanwhile, the master nodes will try to get the control plane, and the OpenShift API up. Keepalived implements periodic health checks for each VIP that are used to determine the weight assigned to each server. The server with the highest weight is assigned the VIP. Keepalived has two seperate healthchecks that attempt to reach the OpenShift API and CoreDNS on the localhost of each master node. When the API on a master node is reachable, Keepalived substantially increases it's weight for that VIP, making its priority higher than that of the bootstrap node and any node that does not yet have the that service running. This ensures that nodes that are incapable of serving DNS records or the OpenShift API do not get assigned the respective VIP. The Ingress VIP is also managed by a healthcheck that queries for an OCP Router HAProxy healthcheck, not the HAProxy we stand up in static pods for the API. This makes sure that the Ingress VIP is pointing to a server that is running the necessary OpenShift Ingress Operator resources to enable external access to the node.

The Worker Nodes run dhcp, CoreDNS, mDNS-publisher, and Keepalived. On workers, Keepalived is only responsible for managing
The Worker Nodes run dhcp, CoreDNS, and Keepalived. On workers, Keepalived is only responsible for managing
the Ingress VIP. It's algorithm is the same as the one run on the masters.
4 changes: 0 additions & 4 deletions docs/user/openstack/known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,6 @@

We have been tracking a few issues and FAQs from our users, and are documenting them here along with the known workarounds and solutions. For issues that still have open bugs, we have attached the links to where the engineering team is tracking their progress. As changes occur, we will update both this document and the issue trackers with the latest information.

## Long Cluster Names

If the mDNS service name of a server is too long, it will exceed the character limit and cause the installer to fail. To prevent this from happening, please restrict the `metadata.name` field in the `install-config.yaml` to 14 characters. The installer validates this in your install config and throws an error to prevent you from triggering this install time bug. This is being tracked in this [github issue](https://github.com/openshift/installer/issues/2243).

## Resources With Duplicate Names

Since the installer requires the *Name* of your external network and Red Hat Core OS image, if you have other networks or images with the same name, it will choose one randomly from the set. This is not a reliable way to run the installer. We highly recommend that you resolve this with your cluster administrator by creating unique names for your resources in openstack.
Expand Down
16 changes: 0 additions & 16 deletions upi/openstack/security-groups.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,6 @@
port_range_min: 53
port_range_max: 53

- name: 'Create master-sg rule "mDNS"'
os_security_group_rule:
security_group: "{{ os_sg_master }}"
remote_ip_prefix: "{{ os_subnet_range }}"
protocol: udp
port_range_min: 5353
port_range_max: 5353

- name: 'Create master-sg rule "OpenShift API"'
os_security_group_rule:
security_group: "{{ os_sg_master }}"
Expand Down Expand Up @@ -202,14 +194,6 @@
port_range_min: 22
port_range_max: 22

- name: 'Create worker-sg rule "mDNS"'
os_security_group_rule:
security_group: "{{ os_sg_worker }}"
protocol: udp
remote_ip_prefix: "{{ os_subnet_range }}"
port_range_min: 5353
port_range_max: 5353

- name: 'Create worker-sg rule "Ingress HTTP"'
os_security_group_rule:
security_group: "{{ os_sg_worker }}"
Expand Down