Reset Search
 

 

Article

High switch packet processing CPU use on N-Series and S-series

« Go Back

Information

 
TitleHigh switch packet processing CPU use on N-Series and S-series
Symptoms
Switching and routing service interruption or degradation
  • Clients may experience slow network response, intermittent connectivity or no connectivity
  • High CPU use.  In a multi-slot system, this condition may be experienced on multiple blades
  • "Switch Packet Processing" process runs at 60% (show support output or CLI command show system utilization process) This symptom is a key indicator
  • Management access via SSH, Telnet and console may fail
  • Network management via SNMP may fail
Environment
  • N-Series
  • S-Series
  • L2 switching 
  • Spanning tree
  • MAC address(es) sourced from same vlan, more than one port
Cause
  • "Switch Packet Processing" task is limited to a max of ~60% total CPU access time
  • If this 60% level is sustained you are generally experiencing a layer 2 network loop or packet reflection behavior
  • The Switch Packet Processing task generates flow data and programs packet forwarding info in hardware
  • Unique MAC address moves cause traffic to be misdirected; system flows are repetitively torn down and re-established
  • Looped flow creation uses all available Switch Packet Processing CPU time - leaving no resources for legitimate traffic flow setups, which will result in service impact
Resolution
Methods for identifying a network loop or identifying the port experiencing reflection behavior. 

1) Enable logging for moved mac addresses to assist debug
  1. set movedaddrtrap enable 
  2. set movedaddrtrap  <port-string> enable (more ports are generally better)
  3. this will print moved MAC addresses to the log
  4. use: show logging buffer to access the information
2) After identifying the high CPU blade
  1. disable half the ports.  
  2. Check the CPU.  
  3. If no change, re-enable the ports
  4. disable the other ports on the blade.  
  5. If the CPU responds positively, enable half of these ports, then half the remaining ports.
  6. Using this method rule out all ports except those causing negative impact.  
3) Using Netsight Compass search for duplicate L2 address entries

4) Issue the command "
show rmon history interval 5min wide" during high switch packet processing to determine which ports have high throughput in the "Util" column

5) Once a flow is learned there is no impact to cpu from traffic forwarded. However, the act of learning and tearing down flows does take cpu time. If there are thousands of flows being learned and aged very quickly on a port these could look like a constant number of flows on first inspection. To get an idea of whether there is flow churn you could use the command below This will repeat the command 10X with 0 seconds between repeating , so repeating continuously. You should see whether a current number of flows on one port fluctuates by 1000s up and down. If so it is likely causing cpu issues due to constant thrashing of flows.

 

Loop 10 0 -r
Show flow stat
 

 

Additional notes

 

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255