Deletion of Mapping does not actually delete it

[cn-mayank]

I created a mapping, did a telnet to the service and used escape character to quit the telnet connection.
After that when i tried to delete the mapping, the kubectl returned message that successfully deleted but I could still get/describe the mapping in next kubectl command.
Logs on smartNat service showed following error message:

Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
 to delete system configuration" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
 to delete DNAT rules" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
 to delete per-service DNAT with chain name MAP-WQVIHMDAZHKRBDJBYMBE3KFU" name=mapping-sample9 namespace=devspaces-
deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Flushing
 chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Executin
g iptables in table nat and chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU; rule option -F, selector: , action: , comment: "
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Destroyi
ng ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU"
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Error re
moving ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU: exit status 1, stdErr: ipset v6.30: Set cannot be destroyed: it is in u
se by a kernel component\n"
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=error msg="Error re
moving ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU: exit status 1" name=mapping-sample9 namespace=devspaces-deploy-dev type
=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=error msg="Error de
leting DNAT: exit status 1" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=warning msg="Failed
 to cleanup mapping from the operating system: &{%!c(*os.ProcessState=&{8187 256 0xc4202fc990}) [i p s e t   v 6 . 
3 0 :   S e t   c a n n o t   b e   d e s t r o y e d :   i t   i s   i n   u s e   b y   a   k e r n e l   c o m p
 o n e n t \n]}" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping

[cn-mayank]

output of some iptables command

$ sudo iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 2 packets, 76 bytes)
 pkts bytes target     prot opt in     out     source               destination         
88298 4795K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
88328 4801K SNM-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* "for SNM" */

Chain INPUT (policy ACCEPT 2 packets, 76 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
41809 2541K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
41825 2541K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
41861 2544K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
41860 2544K SNM-POSTROUTING-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* "for SNM" */

Chain KUBE-FIREWALL (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-DROP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-LOAD-BALANCER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes service load balancer ip + port with externalTrafficPolicy=local */ match-set KUBE-LOAD-BALANCER-LOCAL dst,dst
    0     0 KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-MARK-DROP (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-MARK-MASQ (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODE-PORT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes nodeport TCP port with externalTrafficPolicy=local */ match-set KUBE-NODE-PORT-LOCAL-TCP dst
    0     0 KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-POSTROUTING (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose */ match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-LOAD-BALANCER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes service lb portal */ match-set KUBE-LOAD-BALANCER dst,dst
    0     0 KUBE-NODE-PORT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes nodeport TCP port for masquerade purpose */ match-set KUBE-NODE-PORT-TCP dst
    0     0 KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes service cluster ip + port for masquerade purpose */ match-set KUBE-CLUSTER-IP dst,dst
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst

Chain MAP-KJI62Y2IAW63Q3D6QBAYI3AV (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAP-LO7TEVUCNQ3B7Q6Q3CQ5RG4B (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAP-OSDFTRYVHZ7TSEKNWVJUQZZO (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAP-VMEYKBHWZV2RLE7XCT25UIPS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* "mark for masquerade in SNM-POSTROUTING-MASQ" */ MARK or 0x100000
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9097 /* "for mapping devspaces-deploy-dev/mapping-sample10 [tcp:9097:80]" */ to:10.19.249.9:80

Chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain SNM-POSTROUTING-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   16   840 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x100000/0x100000 /* "masquerade traffic marked in PREROUTING rules as destined for services" */

Chain SNM-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   532 MAP-KJI62Y2IAW63Q3D6QBAYI3AV  all  --  *      *       0.0.0.0/0            10.128.0.7           match-set DNAT-KJI62Y2IAW63Q3D6QBAYI3AV src,dst /* "for mapping devspaces-deploy-dev/mapping-sample7" */
    8   396 MAP-OSDFTRYVHZ7TSEKNWVJUQZZO  all  --  *      *       0.0.0.0/0            10.128.0.7           match-set DNAT-OSDFTRYVHZ7TSEKNWVJUQZZO src,dst /* "for mapping devspaces-deploy-dev/mapping-sample8" */
    0     0 MAP-LO7TEVUCNQ3B7Q6Q3CQ5RG4B  all  --  *      *       0.0.0.0/0            10.128.0.7           match-set DNAT-LO7TEVUCNQ3B7Q6Q3CQ5RG4B src,dst /* "for mapping devspace-dev/devspaces-8ckhdxyp-service-mp" */
    1    64 MAP-WQVIHMDAZHKRBDJBYMBE3KFU  all  --  *      *       0.0.0.0/0            10.128.0.7           match-set DNAT-WQVIHMDAZHKRBDJBYMBE3KFU src,dst /* "for mapping devspaces-deploy-dev/mapping-sample9" */
    0     0 MAP-VMEYKBHWZV2RLE7XCT25UIPS  all  --  *      *       0.0.0.0/0            10.128.0.7           match-set DNAT-VMEYKBHWZV2RLE7XCT25UIPS src,dst /* "for mapping devspaces-deploy-dev/mapping-sample10" */

[cn-mayank]

$ sudo iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 7261 packets, 3729K bytes)
 pkts bytes target     prot opt in     out     source               destination         
4791K 2611M SNM-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* "for SNM" */

Chain INPUT (policy ACCEPT 7261 packets, 3729K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 6920 packets, 488K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 6920 packets, 488K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain SNM-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   51  5738 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set KJI62Y2IAW63Q3D6QBAYI3AV src /* "for Maping devspaces-deploy-dev/mapping-sample7 [KJI62Y2IAW63Q3D6QBAYI3AV] mark 0x803" */ MARK or 0x803
   36  4004 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set OSDFTRYVHZ7TSEKNWVJUQZZO src /* "for Maping devspaces-deploy-dev/mapping-sample8 [OSDFTRYVHZ7TSEKNWVJUQZZO] mark 0x803" */ MARK or 0x803
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set LO7TEVUCNQ3B7Q6Q3CQ5RG4B src /* "for Maping devspace-dev/devspaces-8ckhdxyp-service-mp [LO7TEVUCNQ3B7Q6Q3CQ5RG4B] mark 0x803" */ MARK or 0x803
    3   164 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set WQVIHMDAZHKRBDJBYMBE3KFU src /* "for Maping devspaces-deploy-dev/mapping-sample9 [WQVIHMDAZHKRBDJBYMBE3KFU] mark 0x803" */ MARK or 0x803
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set VMEYKBHWZV2RLE7XCT25UIPS src /* "for Maping devspaces-deploy-dev/mapping-sample10 [VMEYKBHWZV2RLE7XCT25UIPS] mark 0x803" */ MARK or 0x803

[cn-mayank]

@piontec Any update on this issue please ?