Interfaces flaps due to DCB message

[liviozanol]

Hi all.

We are having some weird issues with lldpad process.

When the process is running it is causing our physical Intel 10G interfaces to flap. It seems like, but I'm not sure, that the interface gets restarted.

Using strace and comparing with the compute logs, we narrowed the issue to a single message that is sent by the process:

The interface going down, apparently resetting:

08:13:38.968996+0000 my.compute kernel: ixgbe 0000:19:00.0: removed PHC on eno1
08:13:39.261060+0000 my.compute kernel: ixgbe 0000:19:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
08:13:39.525041+0000 my.compute kernel: ixgbe 0000:19:00.0: registered PHC device on eno1
08:13:39.676982+0000 my.compute kernel: bond0: link status definitely down for interface eno1, disabling it

strace showing the process sending a DCB_CMD_IEEE_DEL message just before for this same interface.

08:13:38.968831 sendmsg(69, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=48, type=RTM_SETDCB, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1744791218, pid=9353
54412}, {dcb_family=AF_UNSPEC, cmd=DCB_CMD_IEEE_DEL}, [{{nla_len=9, nla_type=DCB_ATTR_IFNAME}, "\x65\x6e\x6f\x31\x00"}, {{nla_len=16, nla_type=DCB_ATTR_IEEE}, "\x0c\x00\x03\x00\x08\x00\x01\x00\x01\x03\x06\x89"
}]}, iov_len=48}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 48 <0.674729>

We can see that it took 674 ms. So it sent before the interface went down/restarted and the call just had response after the interface went down.

Question: Has anyone seem this before? Can we somehow disable these DCB/PFC/others flow handle on lldpad processes? I just want it to announce neighborhood like compute name and interface.

[penguin359]

Hello @liviozanol, I seemed to have missed the notification about your bug report. Are you still experiencing the issue?

As far as I know, DCB_CMD_IEEE_DEL should not be generated unless DCBX TLVs are configured on that interface. Can you provide the complete output of lldptool -ti $iface and lldptool -tni $iface from the affected interfaces?

Also, please provide details on the Linux kernel version, distribution, OpenLLDP release, and ixgbe driver version to help aid in debugging.

[penguin359]

After some investigation, this appears to be a regression in OpenLLDP 1.1.0 and does not seem to be present in 1.0.1. I found that ETS Configuration and PFC TLVs appear to be forced on despite enableTx being set to no for both of them. After reverting to 1.0.1, I found my system has a previous configuration with PFC, ETS-CFG, ETS-REC, and APP TLVs all set to enableTx=yes and were transmitting, however, I was then able to turn them off and see them no longer transmitting. Under 1.1.0, this same system showed only PFC and ETS-CFG, but they were not able to be turned off.

[penguin359]

I did a clean room install of OpenLLDP 1.1.0 and found that it resolved my issue with disabling the TLVs, however, the ETS-CFG and PFC TLVs were being transmitted despite the initial state of enableTx showing as no, but I was able to still issue the command to set it off and resync it to the actual state.

@liviozanol Can you try the following commands and see if it solves your immediate issue? You may need to adjust the commands for your specific distro:

sudo rm /var/lib/lldpad/lldpad.conf
sudo systemctl restart lldpad
lldptool -Ti $iface -V PFC enableTx=no
lldptool -Ti $iface -V ETS-CFG enableTx=no
lldptool -Ti $iface -V ETS-REC enableTx=no
lldptool -Ti $iface -V APP enableTx=no

[liviozanol]

Hello @penguin359 . I really appreciate your return on this.

We went on a different path on our Ubuntu servers and ran the lldpad daemon with "-R" flag which allowed 802.1Qaz to be on read only mode since we don't use DCBX on our servers and this lldpad behavior was being triggered by external events (received packets from the other end).

I think, but not sure, Ubuntu uses a custom build of the openlldp that includes this flag.