Synchronization of network devices on newly added hosts for Persistent Networks #5977

Pearl1594 · 2022-02-10T06:59:35Z

Description

This PR is a GSoC project: #4814

This PR adds a feature to L2 persitent networks, where resources of the persistent networks are created at hostConnect phase of DefaultHostListener, and cleaned up at hostAboutToBeRemoved phase.

This PR also fixes a bug, where rebooting the hosts removes the persistent network's resources, without readding them when the host is readded to the zone.

Continuation of PR: #5195
Fixes: #5196

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

This has been manually tested with a mbx KVM, XenServer and VmWare setup on a local machine.

Different manual test scenarios:

When the persistent network is created and the host is added afterwards, the newly added host has the persistent network's resources.
When a host within a zone with a persistent network is removed from the zone, its resources will be cleaned up.
When a host with a running VM is removed from the zone, the running VM will migrate before the resources are cleaned up.
When a host is rebooted, it still has access to the persistent network's resources.

A Marvin component test under
test/integration/component/test_persistent_networks.py
added that tests the following steps for a KVM environment.

identify hosts in the zone, and remove the first
create a L2 persistent network
add the host back to the zone

Validation steps:

Persistent network state should be implemented before adding the host
Host should be added back in successfully
Host should have the persistent networks resources after being added

Test output results should be:

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /tmp/MarvinLogs/. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
====Trying SSH Connection: Host:172.20.0.121 User:root Port:22 RetryCnt:10===
===SSH to Host 172.20.0.121 port : 22 SUCCESSFUL===
{Cmd: ip addr | grep breth0-991 > /dev/null 2>&1; echo $? via Host: 172.20.0.121} {returns: ['0']}
=== TestName: test_newly_added_host_for_persistent_network_resources | Status : SUCCESS ===

===final results are now copied to: /tmp//MarvinLogs/test_persistent_networks_===

Pearl1594 · 2022-02-10T07:05:56Z

@blueorangutan package

blueorangutan · 2022-02-10T07:07:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-10T07:09:59Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2559

Pearl1594 · 2022-02-10T07:25:53Z

@blueorangutan package

blueorangutan · 2022-02-10T07:27:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-10T08:10:36Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2561

Pearl1594 · 2022-02-10T12:52:36Z

@blueorangutan test

blueorangutan · 2022-02-10T12:55:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-02-10T22:20:12Z

Trillian test result (tid-3288)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32532 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5977-t3288-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2022-02-21T08:26:53Z

engine/schema/src/main/java/com/cloud/network/dao/NetworkDaoImpl.java


+    @Override
+    public List<NetworkVO> getAllPersistentNetworksFromZone(long dataCenterId) {
+        Object[] guestTypes = {"Isolated", "L2"};


@Pearl1594
cannot Isolated network be persistent ?

they can @weizhouapache hence Isolated & l2 are included in the search criteria

@Pearl1594
oh thanks... stupid me. I thought it contains Shared and L2 ...

why exclude Shared networks ?

We dont have a concept of Persistent shared networks currently @weizhouapache

@Pearl1594 ok.

Pearl1594 · 2022-02-21T09:24:59Z

@blueorangutan package

blueorangutan · 2022-02-21T09:26:02Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

weizhouapache · 2022-02-21T09:11:49Z

engine/schema/src/main/java/com/cloud/network/dao/NetworkDaoImpl.java


+    @Override
+    public List<NetworkVO> getAllPersistentNetworksFromZone(long dataCenterId) {
+        Object[] guestTypes = {"Isolated", "L2"};


@Pearl1594
oh thanks... stupid me. I thought it contains Shared and L2 ...

why exclude Shared networks ?

weizhouapache · 2022-02-21T09:21:36Z

...lume/src/main/java/org/apache/cloudstack/storage/datastore/provider/DefaultHostListener.java

+        to.setIsolationuri(networkVO.getBroadcastUri());
+        to.setNetworkRateMbps(configManager.getNetworkOfferingNetworkRate(networkOfferingVO.getId(), networkVO.getDataCenterId()));
+        to.setSecurityGroupEnabled(networkModel.isSecurityGroupSupportedInNetwork(networkVO));
+        return to;


in PR #5912, Hari added details to nicTO, is it needed here also ?

https://github.com/apache/cloudstack/pull/5912/files#diff-b048f92c96b32da16b08fa92eb30b7fe6d7be51f2e49bdc2579f985080a070e2R1191-R1212

I don't think it is required @weizhouapache

@Pearl1594 ok to me.

...lume/src/main/java/org/apache/cloudstack/storage/datastore/provider/DefaultHostListener.java

weizhouapache · 2022-02-21T09:27:16Z

...lume/src/main/java/org/apache/cloudstack/storage/datastore/provider/DefaultHostListener.java

+        return true;
+    }
+
+    private void setupPersistentNetwork(HostVO host) {


I wonder if it is neccesary to create bridges, vlans on kvm hosts, although it is not harmful.
from my understanding, persist L2/Isolated network means the vlan id will be not released even if there is no vm running.
for vmware/xenserver, it might be useful.

@Pearl1594
can you explain a bit what's the benifit of this feature ?

So, we had persistent networks in ACS - and by definition, these networks are meant to be setup at the time of network created across all hosts. However, while the definition in the docs said that, it didn't actually behave that way. So when a new host is added no bridge/vlan/port-group of the existing persistent network is present. So the idea of this enhancement, is to setup network on the new hosts or when disable hosts are enabled to stay true to the definition of Persistent networks as per docs.

@Pearl1594
I just read the cloudstack document http://docs.cloudstack.apache.org/en/latest/adminguide/networking/persistent_networks.html
which was updated by apache/cloudstack-documentation#183

The only thing different from what I understand is

- Creation of an L2 Persistent network setups up the network devices namely, bridges, VLANs or port-groups across all hosts in a zone.

I do not see the benifit to have permanent linux bridges but without any vm on kvm hosts.
I have no idea it is useful for vmware or xenserver.

in my understanding, persistent network means the network will not be shutdown even if no vm running. therefore vlan will be released/re-allocated, virtual router will not be stopped, which are useful for network administrators and users.

blueorangutan · 2022-02-21T10:05:07Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2669

…ed host is enabled

Pearl1594 · 2022-02-23T13:04:58Z

@blueorangutan package

blueorangutan · 2022-02-23T13:06:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Pearl1594 · 2022-03-21T13:49:59Z

@blueorangutan package

DaanHoogland · 2022-03-21T13:49:59Z

@blueorangutan package

blueorangutan · 2022-03-21T13:51:02Z

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-21T14:48:56Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2947

nvazquez · 2022-03-21T16:10:31Z

@blueorangutan test

blueorangutan · 2022-03-21T16:11:02Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-22T01:36:07Z

Trillian test result (tid-3689)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32526 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5977-t3689-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2022-04-11T13:50:11Z

@blueorangutan package

blueorangutan · 2022-04-11T13:51:02Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

NuxRo

LGTM

blueorangutan · 2022-04-11T14:43:04Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 3120

nvazquez · 2022-04-11T14:43:34Z

@blueorangutan package

blueorangutan · 2022-04-11T14:44:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-04-11T15:27:48Z

Packaging result: ✖️ el7 ✔️ el8 ✖️ debian ✖️ suse15. SL-JID 3123

nvazquez · 2022-04-11T15:44:23Z

@blueorangutan package

blueorangutan · 2022-04-11T15:46:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-04-11T16:22:16Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3126

nvazquez · 2022-04-11T16:23:59Z

@blueorangutan test

blueorangutan · 2022-04-11T16:25:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-04-12T01:52:05Z

Trillian test result (tid-3854)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32585 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5977-t3854-kvm-centos7.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

Pearl1594 mentioned this pull request Feb 10, 2022

Persistent Networks Resources created on host during time of connecti… #5195

Closed

12 tasks

Pearl1594 force-pushed the pers_net_feature-gsoc2021 branch from 61ffcdf to 56bb154 Compare February 10, 2022 07:24

nvazquez added component:networking type:enhancement labels Feb 11, 2022

nvazquez added this to the 4.17.0.0 milestone Feb 11, 2022

weizhouapache reviewed Feb 21, 2022

View reviewed changes

Pearl1594 marked this pull request as ready for review February 21, 2022 09:25

weizhouapache reviewed Feb 21, 2022

View reviewed changes

Pearl1594 self-assigned this Feb 21, 2022

shatoboar and others added 5 commits February 22, 2022 11:47

Persistent Network feature & Marvin component tests

93ecc87

Cleaned up comments and imports

55e5934

fixed small error

9a9d042

add support to add setup persistent networks' resources when a disabl…

aa678bb

…ed host is enabled

small fix

d80623b

Pearl1594 force-pushed the pers_net_feature-gsoc2021 branch from e59c5fb to d80623b Compare February 23, 2022 12:50

Pearl1594 requested a review from weizhouapache February 23, 2022 13:30

NuxRo approved these changes Apr 11, 2022

View reviewed changes

skip test for simulator hypervisor

03c2b7d

boring-cyborg bot added component:agent component:integration-test component:orchestration component:primary-storage component:solidfire component:storage component:vpc Python Warning... Python code Ahead! labels Apr 11, 2022

nvazquez merged commit 431c352 into apache:main Apr 12, 2022

Synchronization of network devices on newly added hosts for Persistent Networks #5977

Synchronization of network devices on newly added hosts for Persistent Networks #5977

Uh oh!

Conversation

Pearl1594 commented Feb 10, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Pearl1594 commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

Pearl1594 commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

Pearl1594 commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

blueorangutan commented Feb 10, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pearl1594 commented Feb 21, 2022

Uh oh!

blueorangutan commented Feb 21, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 21, 2022

Uh oh!

Pearl1594 commented Feb 23, 2022

Uh oh!

blueorangutan commented Feb 23, 2022

Uh oh!

Pearl1594 commented Mar 21, 2022

Uh oh!

DaanHoogland commented Mar 21, 2022

Uh oh!

blueorangutan commented Mar 21, 2022

Uh oh!

blueorangutan commented Mar 21, 2022

Uh oh!

nvazquez commented Mar 21, 2022

Uh oh!

blueorangutan commented Mar 21, 2022

Uh oh!

blueorangutan commented Mar 22, 2022