Reset Search
 

 

Article

System memory depleted - Process "aaa"

« Go Back

Information

 
TitleSystem memory depleted - Process "aaa"
Symptoms
  • When one or more dot1x clients are authenticating frequently, process AAA is leaking memory.
  • .Due to this unexpected reboot occurred, "System memory depleted" crash.

Switch logs:

10/07/2016 10:23:11.41 <Noti:NM.StrtProc> The Node Manager (NM) has started processing.
10/07/2016 10:23:11.00 <Noti:EPM.start> EPM Started
10/07/2016 10:23:10.99 <Warn:EPM.UnexpctRebootDtect> Booting after System Failure.
10/07/2016 10:23:09.81 <Noti:EPM.wd_warm_reset> Changing to watchdog warm reset mod
e
 
10/07/2016 10:17:17.25 <Warn:EPM.hello_rate> Received hellos from process mrp 2 more often then expected 3
10/07/2016 10:17:05.57 <Warn:EPM.cpu> CPU utilization monitor: process epm consumes 99 % CPU
10/07/2016 10:16:18.21 <Warn:EPM.Msg.hello_rate> Process mcmgr sends hello too often, expected once in 10 secs
10/07/2016 10:16:18.21 <Warn:EPM.hello_rate> Received hellos from process mcmgr 2 more often then expected
3
 
                ->show debug system-dump
===============================================
            Switch system dump information
===============================================
core_dump_info storage: 8/5120 used [EMPTY]
--------------------------------
System memory depleted.
time: Fri Oct  7 10:17:49 2016

Global Memory Stats:
                   MemTotal:       976128 kB
                    MemFree:         3896 kB
                    Buffers:          112 kB
                     Cached:        11036 kB
               Active(anon):       379712 kB
             Inactive(anon):       380228 kB
               Active(file):         5612 kB
             Inactive(file):         5508 kB
                  HighTotal:            0 kB
                   HighFree:            0 kB
                   LowTotal        976128 kB
                    LowFree:         3896 kB
               Committed_AS:      1309216 kB
Top Processes by RSS:
                        aaa:       678980 kB

                  cliMaster:        14640 kB
                        hal:        10328 kB
                       expy:         7112 kB
--------------------------------
System memory depleted.
time: Fri Oct  7 10:17:54 2016

--------------------------------
System memory depleted.
time: Fri Oct  7 10:17:59 2016

--------------------------------
System memory depleted.
time: Fri Oct  7 10:18:04 2016

--------------------------------
System memory depleted.
time: Fri Oct  7 10:18:09 2016


 
Environment
  • X440G2-48t-10G4
  • EXOS 21.1.1.4 patch1-5
Cause
  • Switch memory is leaking slowly during each re-authentication of dot1x client. This will gradually cause memory depletion in customer network.
  • ​From the memory tool and collected outputs, we could see the memory leak for process “aaa” is occurring.

New_3F-2.3 # show memory process aaa
System Memory Information
-------------------------
Total DRAM (KB): 1048576
System     (KB): 72448
User       (KB): 942580
Free       (KB): 33548
Memory Utilization Statistics
-----------------------------
Process Name     Memory (KB)
-----------------------------
aaa              649796           
 
New_3F-2.2 # show memory process aaa
System Memory Information
-------------------------
Total DRAM (KB): 1048576
System     (KB): 72448
User       (KB): 944284
Free       (KB): 31844
Memory Utilization Statistics
-----------------------------
Process Name     Memory (KB)
-----------------------------
aaa              651544     
 
New_3F-2.2 # show memory process aaa
System Memory Information
-------------------------
Total DRAM (KB): 1048576
System     (KB): 72448
User       (KB): 944656
Free       (KB): 31472
Memory Utilization Statistics
-----------------------------
Process Name     Memory (KB)
-----------------------------
aaa              652200  
 
New_3F-2.1 # show memory process aaa
System Memory Information
-------------------------
Total DRAM (KB): 1048576
System     (KB): 72448
User       (KB): 946320
Free       (KB): 29808
Memory Utilization Statistics
-----------------------------
Process Name     Memory (KB)
-----------------------------
aaa              653842


   
Resolution
  • CR#xos0066874 has been created for tracking this issue.
  • ​Kindly follow up in this issue via CR#xos0066874.
  • Please contact GTAC for further assistance.
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255