Set 53/111 LoadBalancer rule may kill haproxy and cause loadbalancer down completely.

[echoidcf]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR/Loadbalancer

CLOUDSTACK VERSION

All version after 4.11 (maybe)
4.7 are not affected

CONFIGURATION

VR with network service offering which having DNS(dnsmasq daemon) and LB(haproxy daemon) services.

OS / ENVIRONMENT

SUMMARY

Because dnsmasq daemon is using tcp53 port, haproxy can not use that port again.
Set a loadbalancer rule for public port 53 will result in error while start/reload/restarting haproxy.
After 4.11, haproxy is reload/restart via python script (CsHelper.py)
But CsHelper.py nevery check if reload/restart is ok, haproxy is killed permanently after setting this loadbalancer rule.

The rule can be set without any error/warning, and loadbalancer service is down after that.

STEPS TO REPRODUCE

1. create/use a network offerings with DNS and LB services.
2. create a LB rule with public 53

After that rule is added without error, but haproxy down completely.

EXPECTED RESULTS

ACTUAL RESULTS

[yadvr]

I think the fix would be to disallow users to from creating rule for port 53 if network offers DNS

[echoidcf]

Same with port 111, opened by rpcbind.

[yadvr]

cc @Spaceman1984

[Spaceman1984]

Should adding port 111 on the load balancer always throw an error? Is there a check that needs to be done or should it be hardcoded?

[andrijapanicsb]

More proper solution would be to NOT bind those ports on public interfaces (dumb idea anyway) - so that haproxy will not fail to bind those ports later on public interfaces
But the quick one would be to give an error early:

Ideally: "netstat-antup | grep "0.0.0.0" should be used to see all ports that are binded on all intefaces - and make list of all those ports that can NOT be binded by haproxy - and give descriptive error for all of them, in the UI - better in API

[yadvr]

@echoidcf I checked port 111 is for rpcbind/NFS, do you think there is any internal use for port 111 at all for VRs?

[yadvr]

Agree @andrijapanicsb however there may be a case for port 53 (allow DNS externally, there is a setting for that I think?)

[andrijapanicsb]

I think that's true @rhtyd , but only valid for Shared networks, or something - I recall the issue we fix for 4.13.1/4.14

In general, the design around those should be considered (i..e even if we give option to expose 53 publicly, we do so only on the default public IP, but with additional public IPs (that are needed for PF/LB anyways) we should allow LB on any port)

Not sure what's the story with port 111, shouldn't be there IMO.

[Spaceman1984]

I checked on the VR, nothing was listening on port 111.

[Spaceman1984]

This is the output running DefaultIsolatedNetworkOffering,

tcp        0      0 10.1.1.1:80             0.0.0.0:*               LISTEN      1080/apache2        
tcp        0      0 10.1.1.1:8080           0.0.0.0:*               LISTEN      897/python          
tcp        0      0 169.254.25.109:3922     0.0.0.0:*               LISTEN      1492/sshd           
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      1946/dnsmasq        
tcp        0      0 10.1.1.1:443            0.0.0.0:*               LISTEN      1080/apache2        
tcp        0      0 0.0.0.0:35999           0.0.0.0:*               LISTEN      1202/haproxy        
udp        0      0 0.0.0.0:53              0.0.0.0:*                           1946/dnsmasq        
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1946/dnsmasq        
udp        0      0 0.0.0.0:69              0.0.0.0:*                           1946/dnsmasq        
udp        0      0 0.0.0.0:3914            0.0.0.0:*                           287/rsyslogd        
udp        0      0 192.168.43.34:123       0.0.0.0:*                           804/ntpd            
udp        0      0 169.254.25.109:123      0.0.0.0:*                           804/ntpd            
udp        0      0 10.1.1.1:123            0.0.0.0:*                           804/ntpd            
udp        0      0 127.0.0.1:123           0.0.0.0:*                           804/ntpd            
udp        0      0 0.0.0.0:123             0.0.0.0:*                           804/ntpd            
udp        0      0 0.0.0.0:45297           0.0.0.0:*                           1200/haproxy 

Would it be a good idea to throw an error if any of these ports are passed as a param to create an lb rule?

[Spaceman1984]

I'll check which ports actually prevent haproxy from starting, I think most of them are not breaking the startup.