Multiple Public Subnets Single Shared Network ( Metadata Error )

[BenjaminM007]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR

CLOUDSTACK VERSION

4.15

CONFIGURATION

Advanced Networking with SG
One defaultGuestNetwork ( 2 x subnets setup using legacy interface.
Example: 188.165.185.64/28 51.255.101.128/26

The first subnet /28 was added when the guestnetwork was created. We can manually add an IP from the /26 subnet ( using the legacy interface and DHCP is working and IP is assigned to the instance. When querying the metadata server for the password the instance seeks out the /28 VR gateway which fails.

OS / ENVIRONMENT

SUMMARY

Creating more than one public CIDR in the DefaultGuestNetwork fails to obtain password from cloud-init metadata server.

STEPS TO REPRODUCE

2021-04-16 13:29:47,489 - DataSourceCloudStack.py[DEBUG]: Crawl of metadata service took 0 seconds
2021-04-16 13:29:47,489 - util.py[DEBUG]: Running command ['wget', '--quiet', '--tries', '3', '--timeout', '20', '--output-document', '-', '--header', 'DomU_Request: send_my_password', ' xxx.xxx.xxx..100:8080'] with allowed return codes [0] (shell=False, capture=True)
2021-04-16 13:30:51,568 - util.py[WARNING]: Failed to fetch password from virtual router 188.165.185.100
2021-04-16 13:30:51,570 - util.py[DEBUG]: Failed to fetch password from virtual router  188.165.185.100
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceCloudStack.py", line 131, in _get_data
    set_password = password_client.get_password()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceCloudStack.py", line 58, in get_password
    password = self._do_request('send_my_password')
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceCloudStack.py", line 53, in _do_request
    '{0}:8080'.format(self.virtual_router_address)
  File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 2084, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['wget', '--quiet', '--tries', '3', '--timeout', '20', '--output-document', '-', '--header', 'DomU_Request: send_my_password', ' xxx.xxx.xxx.:8080']
Exit code: 4




1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:52:01:02:43 brd ff:ff:ff:ff:ff:ff
    inet 188.165.185.100/27 brd 188.165.185.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet 51.255.101.129/26 brd 51.255.101.191 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:00:a9:fe:7f:ef brd ff:ff:ff:ff:ff:ff
    inet 169.254.127.239/16 brd 169.254.255.255 scope global eth1
       valid_lft forever preferred_lft forever

EXPECTED RESULTS

• Obtain password from VR
• instance should query correct gateway of VR for the additional subnet.

ACTUAL RESULTS

• Failed to fetch password from virtual router 188.165.185.100 |  
  -- | --

• Instance is using gateway from the subnet that was originally created when the defaultnetwork was setup.
  Get request to the VR using both IP and DNS name is working as per below. Curl request inside the instance works. When booting a Ubuntu or CentOS instance cloud-init times out when accessing the server_password.

curl http://data-server./latest/meta-data/local-hostname
Test51[root@test51 cloud.cfg.d]# 

[DaanHoogland]

can you add 'real' rfc1918 examples? from this we can not really see where you wnet wrong, @benmcguire79. I'm not saying you are wrong. I've seen a similar issue recently and it might even duplicate that one. .

[BenjaminM007]

Hi Dan

I have updated the initial report with the actual IPs. In addition here is the full cloud-init.log from the test VM. As stated, ssh keys with but password does not.

https://mega.nz/file/ArwlTY7Q#I0a7C30mXhf7YfSpAjG3IwmXX8VtRAfEN5LVzRcwPCk
Also here are the screenshots if they help.

[BenjaminM007]

from the screenshots above it appears the CentOS7 vm is trying to use the first address of the subnet ( 51.255.101.131 ) which Im unsure why it chooses that 131 address and not 129 which is the first usable IP in the subnet.

BUT, the cloudinit log says it is trying to use the 188.165.185.100 which is the VR. Im assuming it is becuase of the DNS entry in the VR for data-server.

Read from http://188.165.185.100/latest/meta-data/availability-zone (200, 6b) after 1 attempts
2021-04-19 09:43:09,840 - url_helper.py[DEBUG]: [0/6] open 'http://188.165.185.100/latest/meta-data/instance-id' with {'url': 'http://188.165.185.100/latest/meta-data/instance-id', 'headers': {'User-Agent': 'Cloud-Init/19.4'}, 'allow_redirects': True, 'method': 'GET', 'timeout': 5.0} configuration

Let me know if I can provide further information.

[BenjaminM007]

Just tested with Ubuntu 16.04 and it tries 51.255.101.129 which still fails.Unsure why the first one ( CentOS 7 ) uses 51.255.101.131 and Ubuntu tries 51.255.101.129 ???

[BenjaminM007]

HI.

Is there any update on this. It would be great to get this fix included in the 4.15.1 release as it is now impossible to use multiple subnets in a shared network and have the password server working.

[yadvr]

I'm tentatively assigning the 4.15.1.0 milestone (doesn't guarantee a fix will make into the release) for investigation.
cc @wido @GabrielBrascher @weizhouapache have you seen something in your SG-based environments?

[wido]

I'm tentatively assigning the 4.15.1.0 milestone (doesn't guarantee a fix will make into the release) for investigation.
cc @wido @GabrielBrascher @weizhouapache have you seen something in your SG-based environments?

We have a strict one-subnet-per-network policy on our advanced zones. Each VXLAN network has one IPv4 and one IPv6 subnet.

In the past we have seen too many issues with multiple subnets per network and therefor we chose to not implement this anymore.

[BenjaminM007]

I'm tentatively assigning the 4.15.1.0 milestone (doesn't guarantee a fix will make into the release) for investigation.
cc @wido @GabrielBrascher @weizhouapache have you seen something in your SG-based environments?

Thank you @rhtyd @wido I guess we will await 4.15.1.0 and see if any changes inthat makes any difference. Should the issue still persist I guess we will go with @wido and use one subnet per network....a pain but if there's no other way :) I was thinking it maybe a simialr issue to #4865

[weizhouapache]

I'm tentatively assigning the 4.15.1.0 milestone (doesn't guarantee a fix will make into the release) for investigation.
cc @wido @GabrielBrascher @weizhouapache have you seen something in your SG-based environments?

We have a strict one-subnet-per-network policy on our advanced zones. Each VXLAN network has one IPv4 and one IPv6 subnet.

In the past we have seen too many issues with multiple subnets per network and therefor we chose to not implement this anymore.

@rhtyd @wido
We have multiple subnets many years ago (cloudstack 2.2.14 ?), when we upgraded to a major version, we created separated shared network for each subnet. for now each shared network have only 1 subnet on our productions (same as pcextreme).

[yadvr]

Thanks @weizhouapache @wido. @benmcguire79 can you help test the latest 4.15.1 and report your findings, or are you happy to accept the general prod use-case (i.e. use single subnet per network) and close the issue?

[BenjaminM007]

Hi @rhtyd . I am happy to test 4.15.1 once it is released as Im patiently waiting for it. For our use case setting up multiple networks for each subnet is not ideal as we have 20+ subnets and having 20+ networks again is not ideal. I see someone else raised the same issue ( albeit with vmware ) but from what I read it is the same issue ( #4943 ) with the data-server and hosts file. In any event as soon as 4.15.1 is released ill immediately test and advise you of my findings. FYI i have tested with an isolated network with multiple internal subnets and these work with no issues....it appears only muliple public subnets on a single network is causing the issue.

[yadvr]

cc @DaanHoogland

[DaanHoogland]

@wei @wido @rhtyd @benmcguire79 I'll create a doc pr to describe the way of working mentioned and do a short test to confirm @benmcguire79 's limitation.