Redeploying Functions leaves many old revisions in the namespace

[utherp0]

I'm doing some development on functions that require me to rebuild and redeploy a lot; every time I redeploy my function the namespace retains the services (one normal, one 'private') for previous versions; currently I have three active functions and 83 service objects

Check out the creation date shown below:

oc get services | grep attack
attack ExternalName kourier-internal.knative-serving-ingress.svc.cluster.local 80/TCP 15d
attack-00010 ClusterIP 172.30.37.117 80/TCP 13d
attack-00010-private ClusterIP 172.30.170.6 80/TCP,9090/TCP,9091/TCP,8022/TCP 13d
attack-00011 ClusterIP 172.30.205.153 80/TCP 13d
attack-00011-private ClusterIP 172.30.28.109 80/TCP,9090/TCP,9091/TCP,8022/TCP 13d
attack-00012 ClusterIP 172.30.2.103 80/TCP 13d
attack-00012-private ClusterIP 172.30.21.55 80/TCP,9090/TCP,9091/TCP,8022/TCP 13d
attack-00013 ClusterIP 172.30.0.112 80/TCP 12d
attack-00013-private ClusterIP 172.30.185.217 80/TCP,9090/TCP,9091/TCP,8022/TCP 12d
attack-00014 ClusterIP 172.30.129.229 80/TCP 12d
attack-00014-private ClusterIP 172.30.92.155 80/TCP,9090/TCP,9091/TCP,8022/TCP 12d
attack-00015 ClusterIP 172.30.168.239 80/TCP 12d
attack-00015-private ClusterIP 172.30.47.144 80/TCP,9090/TCP,9091/TCP,8022/TCP 12d
attack-00016 ClusterIP 172.30.238.71 80/TCP 12d
attack-00016-private ClusterIP 172.30.163.130 80/TCP,9090/TCP,9091/TCP,8022/TCP 12d
attack-00017 ClusterIP 172.30.93.143 80/TCP 11d
attack-00017-private ClusterIP 172.30.209.112 80/TCP,9090/TCP,9091/TCP,8022/TCP 11d
attack-00018 ClusterIP 172.30.11.207 80/TCP 7d2h
attack-00018-private ClusterIP 172.30.158.247 80/TCP,9090/TCP,9091/TCP,8022/TCP 7d2h
attack-00019 ClusterIP 172.30.167.56 80/TCP 7d1h
attack-00019-private ClusterIP 172.30.197.111 80/TCP,9090/TCP,9091/TCP,8022/TCP 7d1h
attack-00020 ClusterIP 172.30.114.214 80/TCP 7d1h
attack-00020-private ClusterIP 172.30.15.14 80/TCP,9090/TCP,9091/TCP,8022/TCP 7d1h
attack-00021 ClusterIP 172.30.170.60 80/TCP 7d
attack-00021-private ClusterIP 172.30.131.180 80/TCP,9090/TCP,9091/TCP,8022/TCP 7d
attack-00022 ClusterIP 172.30.156.117 80/TCP 7d
attack-00022-private ClusterIP 172.30.10.71 80/TCP,9090/TCP,9091/TCP,8022/TCP 7d
attack-00023 ClusterIP 172.30.96.109 80/TCP 6d23h
attack-00023-private ClusterIP 172.30.243.53 80/TCP,9090/TCP,9091/TCP,8022/TCP 6d23h
attack-00024 ClusterIP 172.30.252.102 80/TCP 6d20h
attack-00024-private ClusterIP 172.30.107.178 80/TCP,9090/TCP,9091/TCP,8022/TCP 6d20h
attack-00025 ClusterIP 172.30.227.199 80/TCP 23h
attack-00025-private ClusterIP 172.30.5.34 80/TCP,9090/TCP,9091/TCP,8022/TCP 23h
attack-00026 ClusterIP 172.30.161.83 80/TCP 22h
attack-00026-private ClusterIP 172.30.73.89 80/TCP,9090/TCP,9091/TCP,8022/TCP 22h
attack-00027 ClusterIP 172.30.26.204 80/TCP 22h
attack-00027-private ClusterIP 172.30.0.131 80/TCP,9090/TCP,9091/TCP,8022/TCP 22h
attack-00028 ClusterIP 172.30.13.73 80/TCP 22h
attack-00028-private ClusterIP 172.30.116.247 80/TCP,9090/TCP,9091/TCP,8022/TCP 22h
attack-00029 ClusterIP 172.30.191.122 80/TCP 22h
attack-00029-private ClusterIP 172.30.240.33 80/TCP,9090/TCP,9091/TCP,8022/TCP 22h
attack-00030 ClusterIP 172.30.10.187 80/TCP 3h19m
attack-00030-private ClusterIP 172.30.229.51 80/TCP,9090/TCP,9091/TCP,8022/TCP 3h19m

[lance]

@utherp0 I believe this is simply how Knative works. You will also see a Deployment for each revision as well. You can see this by using the kn CLI.

❯ kn service create test285 --image docker.io/lanceball/myfunc
Creating service 'test285' in namespace 'default':

  0.064s The Configuration is still working to reflect the latest desired specification.
  0.144s The Route is still working to reflect the latest desired specification.
  0.202s Configuration "test285" is waiting for a Revision to become ready.
 15.208s ...
 15.333s Ingress has not yet been reconciled.
 15.530s Waiting for load balancer to be ready
 16.014s Ready to serve.

Service 'test285' created to latest revision 'test285-cdykp-1' is available at URL:
http://test285-default.apps.lball.devcluster.openshift.com

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3 took 16s
 kn service list
NAME      URL                                                          LATEST            AGE     CONDITIONS   READY   REASON
test285   http://test285-default.apps.lball.devcluster.openshift.com   test285-cdykp-1   21s     3 OK / 3     True

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3
❯ kubectl get services
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP                                                  PORT(S)                             AGE
kubernetes                ClusterIP      172.30.0.1       <none>                                                       443/TCP                             56m
openshift                 ExternalName   <none>           kubernetes.default.svc.cluster.local                         <none>                              45m
test285                   ExternalName   <none>           kourier-internal.knative-serving-ingress.svc.cluster.local   80/TCP                              22s
test285-cdykp-1           ClusterIP      172.30.166.87    <none>                                                       80/TCP                              37s
test285-cdykp-1-private   ClusterIP      172.30.120.147   <none>                                                       80/TCP,9090/TCP,9091/TCP,8022/TCP   37s

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3
❯ kn service update test285 --env KEY=VALUE
Updating Service 'test285' in namespace 'default':

  0.039s The Configuration is still working to reflect the latest desired specification.
 10.366s Traffic is not yet migrated to the latest revision.
 10.429s Ingress has not yet been reconciled.
 10.553s Waiting for load balancer to be ready
 11.162s Ready to serve.

Service 'test285' updated to latest revision 'test285-wmpfz-2' is available at URL:
http://test285-default.apps.lball.devcluster.openshift.com

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3 took 11s
❯ kn service list
NAME      URL                                                          LATEST            AGE     CONDITIONS   READY   REASON
test285   http://test285-default.apps.lball.devcluster.openshift.com   test285-wmpfz-2   90s     3 OK / 3     True

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3
❯ kubectl get services
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP                                                  PORT(S)                             AGE
kubernetes                ClusterIP      172.30.0.1       <none>                                                       443/TCP                             57m
/TCP,8022/TCP   7m57s
openshift                 ExternalName   <none>           kubernetes.default.svc.cluster.local                         <none>                              46m
test285                   ExternalName   <none>           kourier-internal.knative-serving-ingress.svc.cluster.local   80/TCP                              78s
test285-cdykp-1           ClusterIP      172.30.166.87    <none>                                                       80/TCP                              93s
test285-cdykp-1-private   ClusterIP      172.30.120.147   <none>                                                       80/TCP,9090/TCP,9091/TCP,8022/TCP   93s
test285-wmpfz-2           ClusterIP      172.30.191.86    <none>                                                       80/TCP                              29s
test285-wmpfz-2-private   ClusterIP      172.30.162.239   <none>                                                       80/TCP,9090/TCP,9091/TCP,8022/TCP   29s

[lance]

If you don't have the kn CLI and just want to list your current Knative revisions, you can do this with kubectl get ksvc.

~/demo/myfunc is 📦 v0.1.0 via ⬢ v12.18.3
❯ kubectl get ksvc
NAME      URL                                                          LATESTCREATED     LATESTREADY       READY   REASON
test285   http://test285-default.apps.lball.devcluster.openshift.com   test285-wmpfz-2   test285-wmpfz-2   True

[utherp0]

Aye, but these are the SDN services not the ksvc; I have a single ksvc service (attack) which I have redeployed around 30 times. I'm just slightly confused that the system is retaining so many of the old services. The knative stuff creates revisions for every change you make to a service (ksvc) but functions don't; even though they are expressed as knative services the original images and config is not retained for revisions, so the legacy services are just clutter.

If you check the attached screenshot you will see what I mean; the namespace creates and retains SDN services for each of the revisions (hence the revision UID) and these are never cleared up even though they are not revisions you can target with traffic.

It's just messing with my tidiness OCD :-)

[zroubalik]

The knative stuff creates revisions for every change you make to a service (ksvc) but functions don't; even though they are expressed as knative services the original images and config is not retained for revisions

I don't think I follow this, function is deployed as ksvc, there shouldn't be any difference in the behavior. Could you please give us more details on this?

[utherp0]

It might just be me misunderstanding; with 'normal' knative services the UI renders the revisions. With functions you get a single revision active at once which is fine, I'm only interested in that one, but it appears it retains the last twenty deployments as revisions (without rendering them as such in the UI). Hence the existence of the 2x kubernetes services per revision.

To check this I did a 'kn revision list' and got:

NAME SERVICE TRAFFIC TAGS GENERATION AGE CONDITIONS READY REASON
attack-00030 attack 100% 30 2d2h 3 OK / 4 True
attack-00029 attack 29 2d21h 3 OK / 4 True
attack-00028 attack 28 2d21h 3 OK / 4 True
attack-00027 attack 27 2d21h 3 OK / 4 True
attack-00026 attack 26 2d22h 3 OK / 4 True
attack-00025 attack 25 2d22h 3 OK / 4 True
attack-00024 attack 24 8d 3 OK / 4 True
attack-00023 attack 23 8d 3 OK / 4 True
attack-00022 attack 22 8d 3 OK / 4 True
attack-00021 attack 21 8d 3 OK / 4 True
attack-00020 attack 20 9d 3 OK / 4 True
attack-00019 attack 19 9d 3 OK / 4 True
attack-00018 attack 18 9d 3 OK / 4 True
attack-00017 attack 17 13d 3 OK / 4 True
attack-00016 attack 16 14d 3 OK / 4 True
attack-00015 attack 15 14d 3 OK / 4 True
attack-00014 attack 14 14d 3 OK / 4 True
attack-00013 attack 13 14d 3 OK / 4 True
attack-00012 attack 12 14d 3 OK / 4 True
attack-00011 attack 11 15d 3 OK / 4 True
attack-00010 attack 10 15d 3 OK / 4 True

I think the question really is whether functions should have revisions? Each of these was a new build with code changes and from my perspective I'm not interested in previous versions (in terms of development).

[zroubalik]

It might just be me misunderstanding; with 'normal' knative services the UI renders the revisions. With functions you get a single revision active at once

With both ksvc and functions you have only one active revision, but you can select older revisions or set traffic distribution across them in the UI. There is no real difference, here you can see deployed function with 2 revisions:

I think the question really is whether functions should have revisions? Each of these was a new build with code changes and from my perspective I'm not interested in previous versions (in terms of development).

You are not forced to use revisions if you don't want to. But there might be reasons for some users to have them. And IMHO both Serverless Apps (ksvc) and functions should have the same behavior, ie. support revisions.

[utherp0]

Is there an easy way to throttle the number of revisions created? Would be nice if I could define as part of the function that I didn't want revisions created, or if so that it only 'remembered' x of them - seeing 83 services running in my namespace gave me a heart attack :-)

[lance]

Is there an easy way to throttle the number of revisions created?

I'm not sure, but if so it would be via the kn CLI (probably). @rhuss ?

seeing 83 services running in my namespace gave me a heart attack :-)

To be clear, only one revision is actually "running" at any given time, unless you have manually done some traffic shaping/distribution. All of the earlier revisions are instances of a Revision resource, but they are not actively running anything and should not consume a lot of resources.

[zroubalik]

IIRC there's global property for specifying the number of stale revisions and other similar options (for how long should Knative keep the Revisions etc). Not sure if this is applicable on single KService or Namespace?

That property is in ConfigMap in knative-serving namespace, probably config-gc or something like that.

[rhuss]

Is there an easy way to throttle the number of revisions created?

I'm not sure, but if so it would be via the kn CLI (probably). @rhuss ?

Actually everytime you change the image (which happens for every function build) a new revision is created, that's the way how Knative works. There are some global configurations parameters that can be used to limited the overall amount of revisions (those are stored in the a global configmap, see knative/serving#8208 for the possible knobs. There is also a responsive revision garbage collector which cleans up revisions, that are not part of a route. However this is also a global flag, but I wonder whether a per-service options would be possible, too ? (pinging @markusthoemmes as I'm not really an expert here). I think it really depends more on the use case (per service) e.g. to switch this on for services during development, but for other services the less aggressive GC would be better (both running on the same Knative installation)

[markusthoemmes]

It'd be nice to qualify whether or not the services are an actual problem here. The assumption is that those are fairly cheap at steady state and shouldn't impact your performance too much.

However, as others have pointed out, there is the possibility to make the garbage collector more aggressive. https://github.com/knative/serving/blob/main/config/core/configmaps/gc.yaml is the configmap in Knative that controls how the garbage collector works. In addition, there's the responsive GC that Roland mentioned above.

[rhuss]

Actually I wonder whether having a service-individual setting would make sense, too? For use case where you create tons of revisions (like a fsnotify based watched regeneration of images, or during the interactive development phased for a specific service) you probably look for a much more aggressive GC than for services that you rollout as application. I expect both types of services can occur on the same cluster, so global settings won't be able to cover both scenarios. I think about allowing the GC parameters also as Service annotations.

Would this make sense ?

[markusthoemmes]

The other possible angle is to have a more aggressive cleanup for inactive revisions by default. I.e. remove the services and deployments once no route points at a specific revisions anymore. Let's create a discussion for this in Knative Serving.

[rhuss]

I'm also going to open a docs issue as i feel the garbage collector with all variant deserve a dedicated documentation outside the source code. Just not sure, where to open the issue, so started this meta-discussion at https://knative.slack.com/archives/C01JBD1LSF3/p1618471556150700

[zroubalik]

Actually I wonder whether having a service-individual setting would make sense, too? For use case where you create tons of revisions (like a fsnotify based watched regeneration of images, or during the interactive development phased for a specific service) you probably look for a much more aggressive GC than for services that you rollout as application. I expect both types of services can occur on the same cluster, so global settings won't be able to cover both scenarios. I think about allowing the GC parameters also as Service annotations.

Would this make sense ?

+1

And/or allowing the GC parameters on a Namespace, would be benefitial as well. This would allow the admins to set this properties individually a teams or projects.