Multiple OpenStack providers throwing the same cluster creation errors

[kciredor]

1. What kops version are you running?
master

2. What Kubernetes version are you running?
1.11.7

3. What cloud provider are you using?
openstack

4. What commands did you run? What is the simplest way to reproduce this issue?
kops create cluster (see https://github.com/kubernetes/kops/blob/master/docs/tutorial/openstack.md)

5. What happened after the commands executed?
Deploying kops on two different "OpenStack as a service" providers, both times I get the same errors:

• executor.go:130] error running task "LBPool/api.dev.k8s.local-https" (9m30s remaining to succeed): error creating LB pool: Failed to create pool: Bad request with: [POST https://host/v2.0/lbaas/pools], error message: {"NeutronError": {"message": "Unrecognized attribute(s) 'loadbalancer_id'", "type": "HTTPBadRequest", "detail": ""}}
• executor.go:130] error running task "SecurityGroupRule/*openstacktasks.SecurityGroupRule {"ID":null,"Direction":"ingress","EtherType":"IPv4","SecGroup":{"ID":null,"Name":"masters.dev.k8s.local","Description":null,"Lifecycle":"Sync"},"PortRangeMin":53,"PortRangeMax":53,"Protocol":"udp","RemoteIPPrefix":null,"RemoteGroup":{"ID":null,"Name":"masters.dev.k8s.local","Description":null,"Lifecycle":"Sync"},"Lifecycle":"Sync"}" (6m1s remaining to succeed): found multiple SecurityGroupRules
• executor.go:130] error running task "RouterInterface/ri-xxx-1" (5m47s remaining to succeed): Error creating router interface: error creating router interface: Resource not found

On a Devstack setup running on my laptop I did not see any errors, but ofcourse it was much too slow to ever become a healthy cluster ;-)

I have specifically tried two different OpenStack providers, to ensure it's not a provider specific issue.

/sig openstack
/kind bug

[drekle]

Thanks @kciredor. To clarify your devstack deployment was successful?

There are quite a bit of differences between openstack deployments in terms of loadbalancers. Would you be able to tell me if you are using octavia on the openstack as a service offerings? You might try passing -os-octavia if so. @zetaab knows more in this regard as I believe he is using octavia.

@zetaab I believe we should flex the library we are using in this regard. Consider the differences:
https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/openstacktasks/lbpool.go#L23
is using:
https://github.com/gophercloud/gophercloud/blob/master/openstack/networking/v2/extensions/lbaas_v2/pools/requests.go#L19
and never using:
https://github.com/gophercloud/gophercloud/blob/master/openstack/loadbalancer/v2/pools/requests.go#L26
I would imagine this is a bug for octavia deployments.

I have seen this SecurityGroupRule complaint before in my testing. I believe this error might be more of a warning and successfully complete after some time. My assumption is that this might be caused by network slowness. @kciredor any truth to this? I am also looking at resolving this on my side, however I also do not experience this in devstack and have logged a ticket against my Openstack team.

creating router interface: error creating router interface: Resource not found
https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/openstacktasks/routerinterface.go#L131
https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/openstack/router.go#L78
It seems like the router or subnet do not exist by the time the router interface is attempted to be added to its parent router. Can you confirm this @kciredor? This can be queried with openstack router list.
These tasks have dependencies however openstack is some times slow to create objects and the objects have to be in an ACTIVE status UP state for further actions to be taken on them. In the event that it is slow these tasks will fail until they succeed or reach their timeouts, which is intentional. This may also be why you see such fewer errors using a faster local devstack deployment.

[kciredor]

Tnx for your fast response @drekle.

My Devstack setup was succesful up to and including update cluster --yes. Kops rolls out fine, no errors. In my specific Devstack setup I cannot enable nested vt-x meaning the master/node instances are running emulated, which is too slow to ever become fully operational. It was good enough for me to proceed testing on a real openstack provider though.

One of the providers uses LBaaSv2 with Tungsten Fabric networking and a neutron api layer afaik. Octavia was not compatible so I stepped back and removed that toggle from my cluster creation. The other provider is yet to respond to my question about their stack, octavia did not work either.

Note that the error mentioned is about the LBPool, the loadbalancer itself is already succesfully created (neutron lbaas-loadbalancer-list shows an ACTIVE status with provider 'opencontrail' which is Tungsten). The pool does not know what a loadbalancer_id is?

About the SecurityGroupRule error: I'm getting many, e.g. 10-20 in a row of these, for different ports. They don't seem to eventually resolve.

RouterInterface errors occur even though I can see an ACTIVE+UP dev-k8s-local router (using openstack router list) for my project almost instantly after kops update cluster --yes.

All three errors keep on repeating, not a single tasks completes anymor, until kops reaches too many timeouts/max retries and gives up.

[namsral]

I manage one of the mentioned OpenStack providers, if you need any help please let me know.

[drekle]

About the SecurityGroupRule error: I'm getting many, e.g. 10-20 in a row of these, for different ports. They don't seem to eventually resolve.

I cant relate to the Security Group Rule issue. Is there any way that you can prove that these are indeed duplicates? The error logs should output the rule, ports, direction, etc...

$ kops --name derek.k8s.local --state swift://derek update cluster --yes
W0327 10:03:39.050585   84918 urls.go:81] Using base url from KOPS_BASE_URL env var: "http://build.example.com/.build/dist/"
W0327 10:03:39.613743   84918 urls.go:136] Using nodeup location from NODEUP_URL env var: "http://build.example.com/.build/dist/nodeup"
W0327 10:03:39.890661   84918 urls.go:173] Using protokube location from PROTOKUBE_IMAGE env var: "http://build.example.com/.build/dist/images/protokube.tar.gz"
I0327 10:03:49.942276   84918 apply_cluster.go:559] Gossip DNS: skipping DNS validation
I0327 10:04:06.020810   84918 executor.go:103] Tasks: 0 done / 111 total; 42 can run
I0327 10:04:13.853715   84918 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator-ca"
I0327 10:04:14.499170   84918 vfs_castore.go:729] Issuing new certificate: "ca"
I0327 10:04:36.944878   84918 executor.go:103] Tasks: 42 done / 111 total; 44 can run
I0327 10:04:44.728392   84918 vfs_castore.go:729] Issuing new certificate: "kube-proxy"
I0327 10:04:44.927354   84918 vfs_castore.go:729] Issuing new certificate: "kube-controller-manager"
I0327 10:04:45.365123   84918 vfs_castore.go:729] Issuing new certificate: "kubecfg"
I0327 10:04:45.503650   84918 vfs_castore.go:729] Issuing new certificate: "kops"
I0327 10:04:45.618470   84918 vfs_castore.go:729] Issuing new certificate: "apiserver-proxy-client"
I0327 10:04:45.843664   84918 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator"
I0327 10:04:46.187104   84918 vfs_castore.go:729] Issuing new certificate: "kubelet-api"
I0327 10:04:46.235392   84918 vfs_castore.go:729] Issuing new certificate: "kubelet"
I0327 10:04:46.397551   84918 vfs_castore.go:729] Issuing new certificate: "kube-scheduler"
I0327 10:05:02.610799   84918 executor.go:103] Tasks: 86 done / 111 total; 11 can run
I0327 10:05:26.702107   84918 executor.go:103] Tasks: 97 done / 111 total; 5 can run
I0327 10:05:32.326419   84918 executor.go:103] Tasks: 102 done / 111 total; 1 can run
I0327 10:05:37.100770   84918 executor.go:103] Tasks: 103 done / 111 total; 4 can run
I0327 10:05:55.391239   84918 vfs_castore.go:729] Issuing new certificate: "master"
I0327 10:06:02.552044   84918 executor.go:103] Tasks: 107 done / 111 total; 4 can run
I0327 10:06:18.406021   84918 executor.go:103] Tasks: 111 done / 111 total; 0 can run
I0327 10:06:21.569557   84918 update_cluster.go:290] Exporting kubecfg for cluster
kops has set your kubectl context to derek.k8s.local

Cluster is starting.  It should be ready in a few minutes.

I am not personally familiar with Tungsten. If you are interested in contributing to the project it would great to have yet another openstack environment, mine might be closer to a vanilla deployment.

Note that the error mentioned is about the LBPool, the loadbalancer itself is already succesfully created (neutron lbaas-loadbalancer-list shows an ACTIVE status with provider 'opencontrail' which is Tungsten). The pool does not know what a loadbalancer_id is?

The error frankly does not make much sense to me, it is not something I have seen. I do know that we are sending this struct as json
https://github.com/gophercloud/gophercloud/blob/master/openstack/loadbalancer/v2/pools/requests.go#L19
to your networking endpoint, and your networking endpoint does not like the loadbalancer_id member. Something is choosing the wrong endpoint on your stack to send this to, or this is not the correct API for the endpoint which you have.

The openstack support is still fairly young and alpha. We can probably add --debug logging to dump the raw http requests to openstack if this would help. Unfortunately I wouldn't have access to many openstack environments that are being deployed, and am aiming to support a common, out of the box, openstack.

[drekle]

@kciredor, @namsral - I should correct myself and say that I no longer experience the security group rule issue. There was some network changes made below me at the openstack level which I do not manage myself. I have been told the upstream commit is below:
https://review.openstack.org/#/c/575268/
which adds atomicity to security group rule transactions preventing an ambiguous state from being reached. Just a shot in the dark here, but if this sounds right I might be able to help, but this issue wouldn't be in Kops.

[kciredor]

Honestly quite new to OpenStack myself, big fan of kops though (on AWS).

Would it be a fitting suggestion when I ask if it's possible for @drekle to work with an OpenStack account provided by @namsral?

It seems you two might be the key to broadening kops on OpenStack support (and maybe even terraform as well: @namsral told me they share the gophercloud dep).

[drekle]

@kciredor I have looked at the openstack cloud provider for Kubernetes and I appear to be creating both the client for neutron as well as the security group in an indistinguishable way. I don't have much more to go on here. I would be curious if you've successfully used the manage-security-groups option in your current tooling with any success?
https://github.com/kubernetes/cloud-provider-openstack/blob/master/pkg/cloudprovider/providers/openstack/openstack.go#L114

[kciredor]

@drekle I only learned about 'manage-security-groups' yesterday and have not been able to test it with kubespray yet, will let you know (early next week).

Really appreciate you looking into these issues. Would it be worthwhile for you to get an OpenStack account from @namsral?

[drekle]

hey @kciredor. I apologize but I missed this comments. I am not sure how much help it would be, perhaps I would be able to assist better with deeper insight into whats going on.

You might have better luck reaching me in the kubernetes slack.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten