Can't find what you need?


• Ask the Community
• Create a Case
Reset Search
 

 

Article

High CPU utilization for the tbcm_msm_tx1 and bcmRX coupled with CPU congestion on BD8800 IO modules

« Go Back

Information

 
TitleHigh CPU utilization for the tbcm_msm_tx1 and bcmRX coupled with CPU congestion on BD8800 IO modules
Symptoms
High CPU utilization for the tbcm_msm_tx1 and bcmRX is observed as shown in the following "top" output:

top

PID  PPID USER     STAT   RSS %MEM CPU %CPU COMMAND
1127     2 root     RW<      0  0.0   0 44.7 [tbcm_msm_tx1]
1132     2 root     RW       0  0.0   1 31.5 [bcmRX]
1272     1 root     S     3748  0.3   1 10.5 ./emsServer
 1276    1 root     S <  17052  1.6   1  2.6 ./hal


CPU congestion on IO modules is also seen in the "debug hal show congestion" output (after executing the command multiple times), as shown in the following output:

Switch2.7 # debug hal show congestion
Congestion information for slot 1 type 8900-10G8X-xl since last query
  No switch fabric and CPU congestion present
Congestion information for slot 2 type 8900-10G8X-xl since last query
  No switch fabric and CPU congestion present
Congestion information for slot 7 type 8900-10G24X-c since last query
  No switch fabric and CPU congestion present
Congestion information for slot 8 type 8900-10G24X-c since last query
  CPU congestion present: 1742879
Congestion information for slot 9 type 8900-G48T-xl since last query
  CPU congestion present: 1331599
Congestion information for slot 10 type 8900-G48T-xl since last query
  No switch fabric and CPU congestion present
Congestion information for slot MSM-A type 8900-MSM128 since last query
  No switch fabric and CPU congestion present
Environment
BD8800
Summit X450a
Summit X460

EXOS 15.2.2.7-P1-6
EXOS 15.5.5.2.9
Cause
After analyzing the IO module CPU-bound packet capture, the following was noted:
- Packets for the module in slot 8 from specific source MAC/specific source IP are ingressing a specific VLAN  and destined to multicast MAC: "01:00:5e:00:00:fc" which maps to Multicast Group: 224.0.0.252 -->  Link Local Multicast Name Resolution (LLMNR).
This is used by MS Windows systems to provide name resolution in the absence of DNS or NetBIOS name resolution.
 
Multicast packets within the 224.0.0/24 (Local Network Control Block) are always copied to the CPU for packet inspection as this range is mainly used for control packets  traffic like OSPF, VRRP, IGMP etc.  The sever(s) should not generate that many LLMNR packet at a high rate.
Resolution
If the LLMNR name resolution is not required on the network and other methods such as DNS name resolution are in use, you can create an ACL that can block these packets from going up to the CPU.  Please see the example below:
----------------------------------------------------------------
 
entry deny_LLMNR {
if match all {
ethernet-destination-address 01:00:5e:00:00:fc;
} then {
deny-cpu;
count LLMNR-deny;
}
}

 
---------------------------------------------------------------
 
The above needs to be saved to a file with an extension “.pol” and downloaded to the switch from a TFTP server, or alternatively you can use the built-in Vi editor on the switch to create and save the file (e.g. “edit policy mnr1.pol”).
 
Once the policy file is created & saved you need to check the policy file for any syntax errors by using the following command:
  •           “check policy mnr1”  --  use the policy filename without the extension.
 
Secondly, the policy can be applied to port or VLAN. In our scenario we can apply it to the ingress VLAN and monitor to see if the “top” output indicates a reduction in CPU utilization for the “tbcm_msm_tx1and “bcmRX” processes.

To apply the ACL to a specific VLAN the syntax is as follows:
  •           configure access-list <policy file name not including the extension> <VLAN name>
Here is an example using the above mentioned policy file name which is applied to VLAN V728:
  •           configure access-list mnr1 <VLAN Name>
 
After applying the ACL to the VLAN you can use the following command to see if the counter “LLMNR-denymentioned in the last line of the ACL is incrementing
(indicates that the multicast packet destined to 224.0.0.252 are not permitted to be sent to the CPU):
  •           show access-list counter LLMNR-deny
 
  •           You can also check the “top” output to see if there is a reduction in the CPU usage for the “tbcm_msm_tx1" & bcm_RX" processes.
If the LLMNR name resolution is not required on the networks and other methods such as DNS name resolution are in use, they can be disabled on the Microsoft Windows hosts/servers based on the Operating system(s) and version(s) in use, and no further LLMNR packets will be generated.
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255