password for VMs and VR health check not working - ACS 4.14 VMware with Basic Networking

[ccycv]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR

CLOUDSTACK VERSION

4.14 

CONFIGURATION

Upgraded to 4.14 from 4.11, Cloudstack + VMware 6.0 configured with Basic Networking

OS / ENVIRONMENT

CentOS 7

SUMMARY

By default management server is able to establish ssh connection only via local IP with VR: eth1 172.11.0.167/24, but in order to run health check it trying to connect via public IPs of the VR, this is not possible because of this:

sshd config :
Port 3922
#AddressFamily any
ListenAddress 172.11.0.167, here i changed to 0.0.0.0

iptables :
-A INPUT -i eth1 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j ACCEPT  ( rule for eth0 is missing ) in basic network it will not work without this. I have added a rule to allow also for eth0

Regarding password issue:
in VR iptables there is only this rule :
-A INPUT -s 158.xx.xx.224/28 -i eth0 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT, only for the first, main public IP, not for all the IPs, so i have added a rule to allow 8080 on each public IP from this router.

oot@r-3480-VM:~#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:91:00:00:33 brd ff:ff:ff:ff:ff:ff
    inet 158.xx.xx.226/28 brd 158.xx.xx.239 scope global eth0
       valid_lft forever preferred_lft forever
    inet 167.xxx.xx.246/28 brd 167.xxx.xx.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 149.xx.xxx.80/27 brd 149.xx.xxx.95 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.xx.xxx.79/26 brd 192.xx.xxx.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet 198.xx.xxx.162/27 brd 198.xx.xxx.191 scope global eth0
       valid_lft forever preferred_lft forever
    inet 149.xx.xxx.99/27 brd 149.xx.xxx.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet 144.xxx.xx.199/27 brd 144.xxx.xx.223 scope global eth0
       valid_lft forever preferred_lft forever
    inet 144.xxx.xxx.177/27 brd 144.xxx.xxx.191 scope global eth0
       valid_lft forever preferred_lft forever
    inet 66.xxx.xxx.133/27 brd 66.xx.xxx.159 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:57:d0:02:14 brd ff:ff:ff:ff:ff:ff
    inet 172.11.0.167/24 brd 172.11.0.255 scope global eth1
       valid_lft forever preferred_lft forever
root@r-3480-VM:~#

VR Log : https://gist.github.com/ccycv/027f31366afe909772bf0592df7b1030
Management log : https://gist.github.com/ccycv/1990d18d0d970fb4e90b0e8f96415c12

Everything works now, till i destroy the router and i have to reconfigure again.

STEPS TO REPRODUCE

Destroy router, redeploy.

EXPECTED RESULTS

No issue, generated password for VM to work and health check (VR) to be performed without issue.

ACTUAL RESULTS

password for VMs not working, didn't updated the password, health check for VR failing. 

[DaanHoogland]

I haven't taken the trouble of upgrading but on a vmware 6.7 ACS 4.14 env:

healthcheck fails initially
deploy of vm works with generated password functioning on login
destroy works but router doesn't autoregenerate
restart network starts a new router
logon still works
new router passes healthcheck
stop vm -> reset password -> restarting (takes long time) -> login succeeds.

I wonder how an upgrade could have caused this. I don't think the vmware version has any influence.
Can you check your upgraded db scheme against a fresh install?

[nvazquez]

Upgrades should not make any difference, @weizhouapache any input about this issue?

[weizhouapache]

@nvazquez

VR health check error is caused by commit 15eab82
https://github.com/apache/cloudstack/blob/main/server/src/main/java/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L1922-L1932

no idea why the author made the changes.

I think we should use _routerControlHelper.getRouterControlIp
https://github.com/apache/cloudstack/blob/main/server/src/main/java/com/cloud/network/router/RouterControlHelper.java#L46-L56

[weizhouapache]

@nvazquez password issue should be fixed by #5013
network/VR: fix dhcp/password/metadata issues on shared networks with multiple subnets #5013