cloud-init fails to fetch metadata when shared networks are present

[onitake]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR
metadata
cloud-init

CLOUDSTACK VERSION

4.11.2
likely others affected as well

CONFIGURATION

Advanced networking, with isolated and shared networks.

OS / ENVIRONMENT

CentOS 7, but likely other VM operating systems affected as well.

SUMMARY

As reported here: https://bugs.launchpad.net/cloud-init/+bug/1839854

cloud-init is longer able to fetch metadata on VMs that have both isolated and shared networks attached. In older cloud-init versions (0.7.5), acquiring metadata would still succeed if network manager was used, because cloud-init would then ignore the DHCP lease files and use the default gateway.

The logic in cloud-init is to use the latest created DHCP lease file for obtaining the metadata server IP, but if that is

STEPS TO REPRODUCE

• Deploy a VM that has cloud-init newer than 0.7.5 and attach an isolated and a shared network to the VM
• Configure cloud-init to use CloudStack metadata
• Depending on the OS, ensure that the shared network obtains a DHCP lease last
• Apply metadata to the VM

EXPECTED RESULTS

• cloud-init obtains and applies metadata on boot

ACTUAL RESULTS

• The VM takes much longer to boot than normal,
• fails to obtain metadata because it isn't offered by the shared network router
• and doesn't apply metadata to the host

[ustcweizhou]

@onitake have you removed the old lease files in "/var/lib/dhclient", "/var/lib/dhcp" and "/var/lib/NetworkManager" when you created the template ?

[onitake]

@ustcweizhou I don't think this has any influence on the issue here.
cloud-init lists all lease files and consults only the newest one for determining the metadata server.

If the DHCP client obtains a lease from the shared network last, cloud-init will use that. And since the shared network VR doesn't start the metadata server, cloud-init will not be able to obtain metadata and fail.

[ustcweizhou]

@onitake what network offering is used to create the shared network ?
if you use a network offering which support UserData (for example DefaultSharedNetworkOffering /Offering for Shared networks), apache2 service should be running in virtual router as metadata server.

[onitake]

That's a good point. The shared network offering we're using in this case does not have the UserData flag enabled. However, we have a reason for not doing this, and metadata fetching has worked for us fine until cloud-init was updated to consult NetworkManager leases.

As mentioned in the upstream bug report, we would rather have the previous behaviour back, in that cloud-init always uses the default gateway as the metadata IP instead of consulting the last created DHCP lease file.

[ustcweizhou]

@onitake
I think the commit to fix https://bugs.launchpad.net/cloud-init/+bug/1717147 is the root cause of your issue.
canonical/cloud-init@da1db79
However, the bug fix makes more sense in my view.

I have no idea why cloud-init gets metadata from the latest dhcp server. It would be better to get metadata from each dhcp server and stop the process once metadata is fetched (vm metadata are saved on virtual routers of all vm nics if vr supports UserData).

Similar process should be applied on password and sshkey as well.

[onitake]

The bug fix is fine, I think. If the user wishes to rely on DHCP leases, that's all right.

What you describe is one possible solution, and I think cloud-init should behave like this in general.

However, I also think that forcing the use of the default gateway would work best in our environment, because the default gateway will always run the metadata server.

[ustcweizhou]

@onitake for isolated network with single virtual router, yes, the gateway is the metadata server.
for shared network, the ip of virtual router is not gateway.
for isolated network with redundant virtual router, the metadata server is running on guest ip of vr, not gateway ip. see #3409

[onitake]

There are many different ways in which networks can be configured, and I agree that my point of view might not cover all use cases.

But there's also another reason why I'd prefer cloud-init doesn't try to fetch metadata from a shared network: Since such a network is by definition not isolated, there's a higher risk that a malicious party might run a rogue DHCP server and a fake metadata server on the shared network and basically take over instances at startup. With isolated networks, this risk is much lower. This may not be a big risk in all cases, but limiting exposure should always be focused on.

[onitake]

I think we found the "right" way to do this: If the metadata server is enabled on an interface, it will offer the special hostname data-server on its DNS server. This is already the case, and I really don't understand why cloud-init isn't already using that.

This should always be the first host that cloud-init checks.

As a second step, the "only latest lease" heuristic should go be replaced with "try all leases in reverse chronological order".

Ideally, all virtual routers should allow resolving the data-server hostname, returning the IP address of a router that has the metadata server enabled. If none of them do, the hostname shouldn't exist.

[onitake]

@joschi36 has pushed a proposed fix: https://code.launchpad.net/~joschi36/cloud-init/+git/cloud-init/+merge/371807

With this fix, cloud-init will first consult the host data-server before falling back on the previous behaviour.

While this doesn't address the case where multiple default gateways are available with only some running the metadata server, it should at least be much more reliable than just looking at the latest DHCP lease.

@ustcweizhou @rhtyd @DaanHoogland @PaulAngus Please let me know what you think. I'll gladly push an update to http://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines/user-data.html if you think this is the right way to go.

[joschi36]

bump @ustcweizhou @rhtyd @DaanHoogland @PaulAngus
Do you think this is the right way to go. If yes we could merge the "Merge Request" in cloud-init launchpad.
Issue: https://bugs.launchpad.net/cloud-init/+bug/1839854
MR: https://code.launchpad.net/~joschi36/cloud-init/+git/cloud-init/+merge/371807

[ustcweizhou]

@joschi36 I've added a comment on launchpad.

[DaanHoogland]

Do we still considder this a cloudstack issue? @joschi36 I see your cloudinit patch was merged. Can we close this?

[onitake]

It's not a CloudStack issue any more, but there is still a PR open to make the documentation clearer: apache/cloudstack-documentation#132

I think it should be revisited and merged before we close this issue.

[DaanHoogland]

I think it needs a bit of your tlc @onitake ;)