Skip to content

[baremetal] Change to unicast for keepalived #1504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bcrochet wants to merge 2 commits into from
Closed

[baremetal] Change to unicast for keepalived #1504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
219ab0f
Change to unicast for keepalived
bcrochet Jan 27, 2020
9e4bbe8
Make unicast enablement optional
bcrochet Mar 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 34 additions & 12 deletions manifests/baremetal/keepalived.conf.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,54 @@
# For more information, see installer/data/data/bootstrap/baremetal/README.md
# in the installer repo.

vrrp_instance {{`{{.Cluster.Name}}`}}_API {
{{`{{$nonVirtualIP := .NonVirtualIP}}`}}

{{`vrrp_instance {{.Cluster.Name}}_API {
state BACKUP
interface {{`{{.VRRPInterface}}`}}
virtual_router_id {{`{{.Cluster.APIVirtualRouterID }}`}}
interface {{.VRRPInterface}}
virtual_router_id {{.Cluster.APIVirtualRouterID }}
priority 50
advert_int 1
{{ if .EnableUnicast }}
unicast_src_ip {{.NonVirtualIP}}
unicast_peer {
{{range .LBConfig.Backends}}
{{if ne $nonVirtualIP .Address}}{{.Address}}{{end}}
{{else}}
{{.NonVirtualIP}}
{{end}}
}
{{end}}
authentication {
auth_type PASS
auth_pass {{`{{.Cluster.Name}}`}}_api_vip
auth_pass {{.Cluster.Name}}_api_vip
}
virtual_ipaddress {
{{`{{ .Cluster.APIVIP }}`}}/{{`{{ .Cluster.VIPNetmask }}`}}
{{ .Cluster.APIVIP }}/{{ .Cluster.VIPNetmask }}
}
}
}`}}

vrrp_instance {{`{{.Cluster.Name}}`}}_DNS {
{{`vrrp_instance {{.Cluster.Name}}_DNS {
state MASTER
interface {{`{{.VRRPInterface}}`}}
virtual_router_id {{`{{.Cluster.DNSVirtualRouterID }}`}}
interface {{.VRRPInterface}}
virtual_router_id {{.Cluster.DNSVirtualRouterID }}
priority 140
advert_int 1
{{if .EnableUnicast }}
unicast_src_ip {{.NonVirtualIP}}
unicast_peer {
{{range .LBConfig.Backends}}
{{if ne $nonVirtualIP .Address}}{{.Address}}{{end}}
{{else}}
{{.NonVirtualIP}}
{{end}}
}
{{end}}
authentication {
auth_type PASS
auth_pass {{`{{.Cluster.Name}}`}}_dns_vip
auth_pass {{.Cluster.Name}}_dns_vip
}
virtual_ipaddress {
{{`{{ .Cluster.DNSVIP }}`}}/{{`{{ .Cluster.VIPNetmask }}`}}
{{ .Cluster.DNSVIP }}/{{ .Cluster.VIPNetmask }}
}
}
}`}}
113 changes: 82 additions & 31 deletions manifests/baremetal/keepalived.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,61 +21,112 @@ spec:
- name: manifests
hostPath:
path: "/opt/openshift/manifests"
initContainers:
- name: render-config
- name: run-dir
empty-dir: {}
containers:
- name: keepalived-unicast
image: {{ .Images.BaremetalRuntimeCfgBootstrap }}
command:
- runtimecfg
- render
- "/etc/kubernetes/kubeconfig"
- unicastipserver
- "--api-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.APIServerInternalIP }}"
- "--dns-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.NodeDNSIP }}"
- "--ingress-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.IngressIP }}"
- "/config"
- "--out-dir"
- "/etc/keepalived"
- "--cluster-config"
- "/opt/openshift/manifests/cluster-config.yaml"
resources: {}
volumeMounts:
- name: resource-dir
mountPath: "/config"
- name: kubeconfig
mountPath: "/etc/kubernetes/kubeconfig"
- name: conf-dir
mountPath: "/etc/keepalived"
- name: manifests
mountPath: "/opt/openshift/manifests"
imagePullPolicy: IfNotPresent
containers:
- name: keepalived
securityContext:
privileged: true
image: {{ .Images.KeepalivedBootstrap }}
image: {{.Images.KeepalivedBootstrap}}
env:
- name: NSS_SDB_USE_CACHE
value: "no"
command:
- /usr/sbin/keepalived
args:
- "-f"
- "/etc/keepalived/keepalived.conf"
- "--dont-fork"
- "--vrrp"
- "--log-detail"
- "--log-console"
- /bin/bash
Copy link
Copy Markdown
Contributor

@yboaron yboaron Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resources in manifest directory run in bootstrap node, right?
Do we need the keepalived-monitor in bootstrap?

Copy link
Copy Markdown
Member Author

@bcrochet bcrochet Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. And yes we do. We are writing the configuration dynamically.

Copy link
Copy Markdown
Contributor

@yboaron yboaron Feb 27, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, got it - 10x

- -c
- |
#/bin/bash
reload_keepalived()
{
if pid=$(pgrep -o keepalived); then
kill -s SIGHUP "$pid"
else
/usr/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --vrrp --log-detail --log-console &
fi
}
msg_handler()
{
while read -r line; do
echo "The client sent: $line" >&2
# currently only 'reload' msg is supported
if [ "$line" = reload ]; then
reload_keepalived
fi
done
}
set -ex
declare -r keepalived_sock="/var/run/keepalived/keepalived.sock"
export -f msg_handler
export -f reload_keepalived
if [ -s "/etc/keepalived/keepalived.conf" ]; then
/usr/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --vrrp --log-detail --log-console &
fi
rm -f "$keepalived_sock"
socat UNIX-LISTEN:${keepalived_sock},fork system:'bash -c msg_handler'
resources:
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: conf-dir
mountPath: "/etc/keepalived"
- name: run-dir
mountPath: "/var/run/keepalived"
livenessProbe:
exec:
command:
- /bin/sh
- -c
- |
[[ -s /etc/keepalived/keepalived.conf ]] || \
kill -s SIGUSR1 "$(pgrep -o keepalived)" && ! grep -q "State = FAULT" /tmp/keepalived.data
initialDelaySeconds: 10
terminationMessagePolicy: FallbackToLogsOnError
imagePullPolicy: IfNotPresent
- name: keepalived-monitor
image: {{ .Images.BaremetalRuntimeCfgBootstrap }}
env:
- name: ENABLE_UNICAST
value: "yes"
command:
- dynkeepalived
- "/etc/kubernetes/kubeconfig"
- "/config/keepalived.conf.tmpl"
- "/etc/keepalived/keepalived.conf"
- "--api-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.APIServerInternalIP }}"
- "--dns-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.NodeDNSIP }}"
- "--ingress-vip"
- "{{ .ControllerConfig.Infra.Status.PlatformStatus.BareMetal.IngressIP }}"
- "--cluster-config"
Copy link
Copy Markdown
Contributor

@yboaron yboaron Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to specify 'cluster-config' parameter for the bootstrap case?

Copy link
Copy Markdown
Member Author

@bcrochet bcrochet Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly so I don't have to do a special case. It didn't work otherwise.

- "/opt/openshift/manifests/cluster-config.yaml"
resources:
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: resource-dir
mountPath: "/config"
- name: kubeconfig
mountPath: "/etc/kubernetes/kubeconfig"
- name: conf-dir
mountPath: "/etc/keepalived"
- name: run-dir
mountPath: "/var/run/keepalived"
- name: manifests
mountPath: "/opt/openshift/manifests"
imagePullPolicy: IfNotPresent
hostNetwork: true
tolerations:
- operator: Exists
Expand Down
Loading