Reset Search
 

 

Article

A VDX6740 running older NOS 5.0.1a software reports that its flash storage is running out of space.

« Go Back

Information

 
TitleA VDX6740 running older NOS 5.0.1a software reports that its flash storage is running out of space.
Symptoms
A VDX6740 running older NOS 5.0.1a software reports that its flash storage is running out of space.
2019/08/23-07:14:58, [FW-1435], 153169, SW/0 | Active, WARNING, VDX, Switch status change contributing factor Flash: usage out of range.
2019/08/23-07:14:58, [FW-1424], 153168, SW/0 | Active, WARNING, VDX, Switch status changed from HEALTHY to MARGINAL.
2019/08/23-07:14:04, [FW-1425], 153167, SW/0 | Active, INFO, VDX, Switch status changed from MARGINAL to HEALTHY.
2019/08/23-07:13:58, [FW-1435], 153166, SW/0 | Active, WARNING, VDX, Switch status change contributing factor Flash: usage out of range.
2019/08/23-07:13:58, [FW-1424], 153165, SW/0 | Active, WARNING, VDX, Switch status changed from HEALTHY to MARGINAL.
2019/08/23-07:13:58, [FW-1402], 153164, SW/0 | Active, WARNING, VDX, Res Flash 0, is above high boundary(High=90, Low=0). Current value is 91 Percent(%)


VDX# show system monitor
** System Monitor Switch Health Report **
RBridge 14      switch status            : MARGINAL
                Time of Report           : 2019-08-23 09:37:43
                Power supplies monitor   : HEALTHY
                Temperatures monitor     : HEALTHY
                Fans monitor             : HEALTHY
                Flash monitor            : MARGINAL
VDX#

Free flash depletion is severe enough that supportsave collection fails.
Environment
  • VDX6740
  • NOS 5.0.1a
Cause
Some of the most common causes of VDX6740s running out of flash storage space are singular large files in the locations "/var/log" and "/tmp" and large numbers of core dump files listed in "show support". However in older NOS software, it is possible for large numbers of core dump files to be stored in "/var/srmd_data" and not be listed in "show support".

In the example below it is necessary to enter the root password to enable the command "
foscmd". The default root password "fibranne" is listed in each NOS release's version of the Troubleshooting Guide. "foscmd" is extremely powerful. For safety it should be hidden after each use.

 
VDX# unhide foscmd
Password:********
VDX# foscmd "ls /var/srmd_data | wc -l" | nomore 
13764
VDX# 

VDX# foscmd "ls -lrS /var/srmd_data " | nomore 
total 896080
....
-rw-------   1 root     root            0 Aug 26 09:31 core.985
-rw-------   1 root     root            0 Aug 25 12:14 core.9843
-rw-------   1 root     root            0 Aug 26 03:44 core.9829
-rw-------   1 root     root            0 Aug 25 12:14 core.9818
-rw-------   1 root     root            0 Aug 25 12:13 core.9795
-rw-------   1 root     root            0 Aug 26 03:43 core.9781
-rw-------   1 root     root            0 Aug 25 12:13 core.9767
-rw-------   1 root     root            0 Aug 26 03:42 core.9752
-rw-------   1 root     root            0 Aug 25 19:59 core.9747
-rw-------   1 root     root            0 Aug 25 04:29 core.9741
-rw-------   1 root     root            0 Aug 26 03:42 core.9733
-rw-------   1 root     root            0 Aug 25 19:59 core.9720
-rw-------   1 root     root            0 Aug 25 19:58 core.9699
-rw-------   1 root     root            0 Aug 25 12:11 core.9696
-rw-------   1 root     root            0 Aug 26 03:41 core.9685
-rw-------   1 root     root            0 Aug 25 19:58 core.9673
-rw-------   1 root     root            0 Aug 25 12:11 core.9670
-rw-------   1 root     root            0 Aug 26 03:40 core.9659
-rw-------   1 root     root            0 Aug 25 19:57 core.9652
-rw-------   1 root     root            0 Aug 25 12:10 core.9647
-rw-------   1 root     root            0 Aug 26 03:40 core.9637
-rw-------   1 root     root            0 Aug 25 04:27 core.9634
-rw-------   1 root     root            0 Aug 25 19:57 core.9627
....
VDX# hide foscmd

 
Resolution
All of the core files under "/var/srmd_data" can be safely, immediately deleted at any time. Flash usage can be checked before and after with "df".
df
foscmd "ls -l /var/srmd_data/core*" | nomore
foscmd "rm /var/srmd_data/core*" | nomore
df

If flash storage is filling up the most urgent priority is freeing up flash storage space. If core dumps are generated again afterwards they can be investigated at that time. After deleting a large number of them, it should be possible to collect a supportsave or upgrade software. Core dump files under "/var/srmd_data" are not normally collected as part of supportsave, but they can be uploaded separately to an SSH/SCP server for further investigation.

The command for uploading srmd_dara core files would be as follows.
foscmd "scp /var/srmd_data/core* <USERNAME>@<HOSTNAME>:<PATH>" | nomore
As an example of using scp when uploading a single log file:
VDX6# foscmd "scp /var/log/syslog.log extreme@10.170.104.18:." | nomore
The authenticity of host '10.170.104.18 (10.170.104.18)' can't be established.
ECDSA key fingerprint is SHA256:IoUeQmaRACaifyKJ05XJkk5T0IhgaoLgYRofuIJw4j0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.170.104.18' (ECDSA) to the list of known hosts.
extreme@10.170.104.18's password:
syslog.log                                    100%  102   161.0KB/s   00:00
VDX6#


 
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255