Reset Search
 

 

Article

How to troubleshoot mac flapping issue

« Go Back

Information

 
TitleHow to troubleshoot mac flapping issue
Objective
This article is intended to help troubleshoot mac flapping issue in an EXOS switch. 
* This may be only applicable to a switch with having software mac learning enabled by MAC security, MLAG, PVLAN and etc.
Environment
  • EXOS All
Procedure
In the event that a switch experiences one or more mac addresses constantly flapping across multiple ports, the symptoms may vary.
One of the symptoms observed in common is that ping destined for or originated from the switch frequently fails or exhibits an excessive delay in the result.
However, CPU utilization may be seen stable while ping fails.

 
# ping 10.82.190.1 from 10.82.190.252
Ping(ICMP) 10.82.190.1: 4 packets, 8 data bytes, interval 1 second(s).
16 bytes from 10.82.190.1: icmp_seq=1 ttl=255 time=1211 ms
16 bytes from 10.82.190.1: icmp_seq=2 ttl=255 time=689 ms
16 bytes from 10.82.190.1: icmp_seq=3 ttl=255 time=0.592 ms
--- 10.82.190.1 ping statistics ---
4 packets transmitted, 3 packets received, 25% loss
round-trip min/avg/max = 0/633/1211 ms
The other symptom may be the log message warning that async queue is growing at a fast pace.

04/26/2016 02:05:14.85 <Warn:Kern.Card.Warning> Slot-3: async queue is growing (192221) type is 49194, customType is 42

To identify if a switch is experiencing mac flapping, the following commands can be used to determine the issue in an EXOS switch.
  • Check if any vlans show "learned" counter incrementing at an excessive rate.
# show l2stats | in learned
Total number of packets learned = 182.
Total number of packets learned = 3.
Total number of packets learned = 23.
Total number of packets learned = 128898.
Total number of packets learned = 42.
Total number of packets learned = 94.

# show l2stats VXXX-SERVER-01
Bridge interface on VXXX-SERVER-01:
Total number of packets to CPU = 137.
Total number of packets learned = 236975.
Total number of IGMP control packets snooped = 1.
Total number of IGMP data packets switched = 3.
Total number of MLD control packets snooped = 0.
Total number of MLD data packets switched = 0.
  • Check if CPU congestion is constant. In some cases, you may not see noticeable CPU congestion.
# debug hal show congestion 
Congestion information for slot 1 type BDXA-40G24X since last query
  CPU congestion present: 874174442342
Congestion information for slot 3 type BDXA-40G24X since last query
  No switch fabric or CPU congestion present
Congestion information for slot 4 type BDXA-40G24X since last query
  No switch fabric or CPU congestion present
  • Check if MAC Moves counter is high and constantly incrementing.
# debug hal show platform fdb 11 | in Moves
Number of VPLS FDB Moves Ignored  : 0
Number of FDB MAC Moves:            32877
  • Check if CPU queue 3 counter is incrementing, which is assigned packets being lifted to the CPU for software based source mac learning.
# debug hal show emcm cpu-queue-stats slot A
Printing CPU Q statistics for slot=13 unit=0
Cos         PktCount        ByteCount
0               100           16400
1                 0               0
2              1030          168920    
3           2265618       371561352    (Software MAC Learning)
4                 0               0
5               849          139236
6                86           14104
7              1110          202180
  • Run tcpdump in the debug-mode and check if there are packets being received on multiple ports from the same source mac address.
  • In the below output, packets with the source mac address of 00:00:00:00:00:04 are received on port 1:1 and 1:2 alternately, causing constant learning on the switch. 
# !tcpdump -i Broadcom -e -c 10
 RX: 01:01->06:00:38.343327 00:00:00:00:00:04 (oui Ethernet) > 00:00:5e:00:01:01 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 4090, p 0, ethertype IPv4, 10.10.10.10.8080 > 10.4.4.1.49000: UDP, length 18
 RX: 01:02->06:00:38.343338 00:00:00:00:00:04 (oui Ethernet) > 00:00:5e:00:01:01 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 4090, p 0, ethertype IPv4, 10.10.10.10.8080 > 10.4.4.1.49000: UDP, length 18
Once a particular mac address(es) is identified as being flapping across multiple ports, please check the settings of the source device.
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255