Reset Search
 

 

Article

Troubleshooting memory issue's

« Go Back

Information

 
TitleTroubleshooting memory issue's
Objective
You want to find out the memory usage on an EXOS switch
Environment
EXOS all
Procedure
EXOS memory is hand out dynamically to processes as they need memory. Normally a process can take as much memory as needed if that is available and will give memory back when it does not need it anymore.
In some rare cases memory usage on the switch keeps increasing leading eventually in memory depletion and possibly a crash.
How can you check what is going on with your switch memory ?

The most easiest command to use is show memory. This command will show first a summary of available memory and then show a detailed list per process.
X460G2-24p-10G4.2 # sh memory
System Memory Information
-------------------------
 Total DRAM (KB): 1048576
 System     (KB): 64388
 User       (KB): 254424
 Free       (KB): 729764

Memory Utilization Statistics
-----------------------------

 Process Name     Memory (KB)
-----------------------------
 aaa              4236            
 acl              3276        
......

If the switch is a stack or chassis it will show the for the master and backup slot. You can take the output of show memory a few times with some regular intervals to see if anything changes. The memory usage will not be constant but a constant increase of used memory could point to a problem.

In EXOS 16.1 and higher a python script is provided with EXOS to assist in troubleshooting memory usage.
The script "mem-stats.py" creates the first time it is used a baseline and every time it runs it will compare the current usage to that baseline.
It will print the differences for each process and the total differences. Below an example.
X460G2-24p-10G4.3 # run script mem-stats.py
##[ Baseline ]##########################################################
- Baseline found     : /usr/local/cfg/mem-stats.pckl
- Current time       : 2016-12-28 12:56:18.527392
- Baseline time      : 2016-11-09 15:20:12.798871
- Min since baseline : 70416.0

##[ Memory overview ]###################################################
[ COUNTER ]                                  [ MEM (KB) ]   [ BASELINE ]

MemFree                                            565736          33012
MemFreeAct                                         728132          26832
MemTotal                                           984188              0
Slab                                                54924          -2236

##[ Processes ]#########################################################
--[ Sorted by memory allocation ]---------------------------------------
[ PID ] [ PROCESS ]                          [ MEM (KB) ]   [ BASELINE ]

1509    cliMaster                                   17740          15640
1505    hal                                         15388          13856
1517    snmpSubagent                                 4712            836
1634    etmon                                        4376           3672
1515    snmpMaster                                   4120         -13940
1521    vlan                                         3536           -688
1525    fdb                                          3288           -916
1572    mcmgr                                        3272            284
1511    cfgmgr                                       2860         -18728
1568    stp                                          2564              0

--[ Sorted by increased memory allocation ]-----------------------------
[ PID ] [ PROCESS ]                          [ MEM (KB) ]   [ BASELINE ]

1509    cliMaster                                   17740          15640
1505    hal                                         15388          13856
1634    etmon                                        4376           3672
1517    snmpSubagent                                 4712            836
1639    ipfix                                        2072            604
1572    mcmgr                                        3272            284
1653    thttpd                                       1556            128
1562    eaps                                         2368             64
1       init                                          660              0
1568    stp                                          2564              0

##[ Kernel slab cache ]#################################################
--[ Sorted by slabs ]---------------------------------------------------
[ NAME ]                         [ SLABS ]  [ SIZE (KB) ]   [ BASELINE ]

size-4096                             1021           4084             47
ext3_inode_cache                       476           1904              0
size-1024                              417           1668              8
dentry                                 374           1496             -9
radix_tree_node                        324           1296             -7
UNIX                                   313           1252              4

The output also shows kernel slab cache, this is memory used by the kernel and buffers.
 
Additional notes
Creating a new baseline for mem-stats.py is done by adding -n to the run script command.
run script mem-stats.py -n

Help is shown with -h
run script mem-stats.py -h
usage: mem-stats [-h] [-n] [-t TOP] [-v]

optional arguments:
  -h, --help         show this help message and exit
  -n, --new
  -t TOP, --top TOP
  -v, --verbose

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255