[DO NOT MERGE] Router HA and failover support

[ramr]

HA and failover support for the router. Adding virtual-ips automatically generates a privileged container running keepalived. There's also a unicast option for environments without multicast support.

Example usage:
openshift ex router --credentials="${KUBECONFIG}" --virtual-ips="10.0.2.100-102,10.0.2.42" --latest-images=true --create

@pweil- @smarterclayton PTAL and comments would be appreciated. Thx.

Still has 1 pending issue - will log a separate comment on that.

[ramr]

I still have an issue here with this configuration. For some reason, the containers that get started are not started using the HostNetwork.

However my test json script works fine - the docker containers use the host network.
See: https://gist.github.com/ramr/b6389a7bfbade8df8cdc for details.
This works if created with osc create -f foo.json

So something in this API interaction is causing issues. Spent a good day plus trying to figure this out - thought originally it was an issue w/ api version v1beta3 to v1beta1 conversions but the above mentioned json config script works fine. So its something in here that am not doing correct. Any help/pointers would be greatly appreciated. thx

[smarterclayton]

Does the kubelet see the host network setting? You can query the kubelet directly to see if it's getting the right response.

[ramr]

No it doesn't - think it is probably something in the interaction via the DeployConfig (via the deployer pod).

[smarterclayton]

To be honest, I'm a bit concerned with all of this complexity being directly coupled with a router (or it appearing to be coupled). First, I'm not sure sufficient number of organizations would choose this solution or want to tweak it for it to be as coupled as it is. Second, the abstraction should really be "add HA to a general deployment config, not just to routers". Third, this is pretty dependent on upfront configuration and doesn't appear to be flexible to the cluster changing, which is a serious issue.

The solution we want to get to is general virtualized IPs across the cluster (whether provided by the external infrastructure or something we emulate with VIPs internally). I don't want to bind it closely to routers and then have to rip it out again.

Before we dive too much into the details of this pull, I'd like to see the following

1. Move the keepalived code into a new command that adds a keepalived sidecar to an existing deployment config (similar to what you have, but something like openshift admin enable-keepalive <deployment config name> with the keepalive options there).
2. A design proposal on how dynamic cluster membership is going to be handled (i.e. if I'm reading this right the unicast depends on reading the ips of the cluster up front - when new nodes come in what happens)
3. A description of what an admin needs to do to configure HA IP outside the cluster, and why they would or wouldn't choose that (I'm an admin, I know how to set up keepalived, why is this solution better)
4. A description of how keepalived could be moved to run on a subset of hosts (as containers) that would then provide HA IPs for pods that they host (and what topologies are reasonable there). How would keepalived react to dynamic membership changes?

I'm looking to dramatically reduce the operational costs associated with managing VIPs - while this is a good starting point, it doesn't really reduce costs enough for it to be something we want to bless ootb.

[ramr]

Hmm, ok -- the trello card I had read router failover and not really VIP management across the cluster. That would seem to be a bit of scope creep here!! :)
I do agree its complex and there is benefit in making it generic ... imho, the wrappers around this are more complex than they need to be!! ;^) Also to your other point, this is actually flexible to the cluster changing and not fixed on the cluster - the workflow is similar to a node going down and a new one taking over the VIP or in a case of a small cluster, implicitly having a few nodes short/down - so in a way, its no different than a failure case.

As re: the points you raised:

1. Will address that in a separate PR .. if that's the way you want this to proceed.
2. unicast is a bit of an issue in any case - as the number of nodes scales, the unicast is not going to be a good solution or a scalable one in any case - imagine send the same advertisement packets to n hosts - would increase the number of cross host packets ... n * (n-1).
   Now as re: new nodes coming into a cluster, this really only has an effect if a router instance is migrated to that node - but then so would a keepalived instance - its part of the same pod and in that case the cluster membership is automatically managed (all peers + the node itself form that group). Otherwise a new node joining the cluster but not running a router - really has no effect. There's probably some more work to still do there - not complete by any means but in all fairness, I threw in unicast support because I have seen environments ala Amazon EC2 where multicast is not supported. Didn't think that would really be a big driver in an enterprise case as it does have limitations of scale.
3. Not sure I understood this point - A description of what an admin needs to do to configure HA IP outside the cluster, and why they would or wouldn't choose that - if you are looking for a HA solution from outside the cluster, you still have to manage how traffic gets router to one or more router instances and handle cases where the router pod migrates between nodes.
4. In general, I can add more documentation around this and the other bits once this get flushed out a wee bit more. This point is however really easy to address in that - its would just be a selector on the ReplicationControllerSpec, which would define what set of nodes this gets run on (router + the failover component). Again the dynamic membership bits are as mentioned at the top of this comment.

And finally, as re: the last part, reducing cost - I don't know about that. There's a fair deal of complexity, understanding, layout - physical/network, migrations etc which need to be done by hand - so there is definitely automation and reduction of cost here. Now that said, we have a lot lot of assumptions across the board on how the nodes are connected - the network topologies of how the whole shebang is deployed - whether its cross availability zones/networks/WANs etc - and I don't think that's something specific to this PR though - its something that needs to be addressed outside the scope of this crazy lil' world where all the keepalived's know each other!! ;^)

[smarterclayton]

On Apr 2, 2015, at 2:55 AM, ramr notifications@github.com wrote:

Hmm, ok -- the card I had read router failover and not really VIP management across the cluster. That would seem to be a bit of scope creep here!! :)

My concern is that all solutions like this become beautiful snowflakes unless we have a path to them either becoming fundamental or being removed. I think this is a fundamental cluster problem (I want an external ip allocated for my pod, and I want it HA) which has many implications wrt IaaS (GCE and AWS already does this for us - we need that to be automated eventually).

You don't have to do the actual work to solve the generic problem, but I want to see that path before we merge it.

I do agree its complex and there is benefit in making it generic ... imho, the wrappers around this are more complex than they need to be!! ;^) Also to your other point, this is actually flexible to the cluster changing and not fixed on the cluster - the workflow is similar to a node going down and a new one taking over the VIP or in a case of a small cluster, implicitly having a few nodes short/down - so in a way, its no different than a failure case.

As re: the points you raised:

1. Will address that in a separate PR .. if that's the way you want this to proceed.
2. unicast is a bit of an issue in any case - as the number of nodes scales, the unicast is not going to be a good solution or a scalable one in any case - imagine send the same advertisement packets to n hosts - would increase the number of cross host packets ... n * (n-1).
   Now as re: new nodes coming into a cluster, this really only has an effect if a router instance is migrated to that node - but then so would a keepalived instance - its part of the same pod and in that case the cluster membership is automatically managed (all peers + the node itself form that group). Otherwise a new node joining the cluster but not running a router - really has no effect. There's probably some more work to still do there - not complete by any means but in all fairness, I threw in unicast support because I have seen environments ala Amazon EC2 where multicast is not supported. Didn't think that would really be a big driver in an enterprise case as it does have limitations of scale.

I think it can definitely be valuable, but don't want to put it in unless your confident it's a long term solution.

1. Not sure I understood this point - A description of what an admin needs to do to configure HA IP outside the cluster, and why they would or wouldn't choose that - if you are looking for a HA solution from outside the cluster, you still have to manage how traffic gets router to one or more router instances and handle cases where the router pod migrates between nodes.

So this is a "we make pods HA" solution internal to the cluster. If an admin says "I know how to do that", what are the things they have to know to properly enable it from the outside in a normal, enterprisey ops shop. Ideally the outside directions are a clear subset of the work you're doing here, so that shared bit of explanation is important for us to support this.

1. In general, I can add more documentation around this and the other bits once this get flushed out a wee bit more. This point is however really easy to address in that - its would just be a selector on the ReplicationControllerSpec, which would define what set of nodes this gets run on (router + the failover component). Again the dynamic membership bits are as mentioned at the top of this comment.

And finally, as re: the last part, reducing cost - I don't know about that. There's a fair deal of complexity, understanding, layout - physical/network, migrations etc which need to be done by hand - so there is definitely automation and reduction of cost here. Now that said, we have a lot lot of assumptions across the board on how the nodes are connected - the network topologies of how the whole shebang is deployed - whether its cross availability zones/networks/WANs etc - and I don't think that's something specific to this PR though - its something that needs to be addressed outside the scope of this crazy lil' world where all the keepalived's know each other!! ;^)

—
Reply to this email directly or view it on GitHub.

[ramr]

Thanks for the clarifications/comments - I will take a lot at this when I am back next week. I will probably disable/not expose the unicast functionality for now - there's obvious mentioned scale issues therein. Meanwhile, let me mull on this a bit more. Closing this PR out for now. Thanks.

[smarterclayton]

Also can you look over the external ip discussion upstream (and the service public ip) and get familiar with the implications?

External IP at least is a request by an app consumer to get an IP provisioned by the infra that is bound to a pod (or the pods in a service?). That overlaps with services a great deal. And headless services having a pair of virtual ips also maps to this problem.

On Apr 2, 2015, at 1:12 PM, ramr notifications@github.com wrote:

Thanks for the clarifications/comments - I will take a lot at this when I am back next week. I will probably disable/not expose the unicast functionality for now - there's obvious mentioned scale issues therein. Meanwhile, let me mull on this a bit more. Closing this PR out for now. Thanks.

—
Reply to this email directly or view it on GitHub.

[ramr]

Sure - will take a look at the upstream external ip discussion as well next week when I am back.