Reset Search
 

 

Article

Raslog messages show FW-1402 Res Flash 0 is above high boundary. Syslog.log is several hundred MB in size.

« Go Back

Information

 
TitleRaslog messages show FW-1402 Res Flash 0 is above high boundary. Syslog.log is several hundred MB in size.
Symptoms
Raslog messages show "FW-1402 Res Flash 0 is above high boundary." For example:
2019/07/19-10:23:55, [FW-1402], 21967, SW/0 | Active, WARNING, VDX1, Res Flash 0, is above high boundary(High=90, Low=0). Current value is 91 Percent(%).
2019/07/19-10:23:55, [FW-1424], 21968, SW/0 | Active, WARNING, VDX1, Switch status changed from HEALTHY to MARGINAL.
2019/07/19-10:23:55, [FW-1435], 21969, SW/0 | Active, WARNING, VDX1, Switch status change contributing factor Flash: usage out of range.
2019/07/19-10:24:01, [FW-1425], 21970, SW/0 | Active, INFO, VDX1, Switch status changed from MARGINAL to HEALTHY.

Attempting to collect a supportsave results in "Insufficient CF memory for copy support" and a partial supportsave. For example:
VDX1# copy support ftp directory 20190719 host x.x.x.x user anonymous password xxx rbridge-id all
Rbridge-id 1: Saving support information for chassis:VDX2, module:RAS...

Rbridge-id 2: Insufficient CF memory(169 MB) for copy support.

The output of the df command shows the root partition with utilization of 90% or more. For example:
VDX1# df
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             1.8G  1.6G  170M  90% /
none                     0     0     0   -  /proc
none                     0     0     0   -  /sys
none                     0     0     0   -  /dev/pts
none                     0     0     0   -  /tmp
/dev/sda2             1.8G  704M  1.1G  39% /mnt
VDX1#

Syslog.log is several hundred MB in size. For example:
VDX1# unhide foscmd
Password: ********
VDX1# foscmd ls -l /var/log/syslog.log
-rw-r-----   1 root     root     360048807 Jul 19 11:12 syslog.log
VDX1# hide foscmd
The password used with "unhide foscmd" is the root password. The default root password as listed in each supported version of the Troubleshooting Guide is fibranne.

The most recent messages in syslog.log show "Suspending write operation because of an I/O error", "Error suspend timeout has elapsed, attempting to write again", "and I/O error occurred while writing". For example:
VDX1# unhide foscmd
Password: ********
VDX1# foscmd "tail -n 30 /var/log/syslog.log" | nomore
Jul 22 15:50:03 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:13 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:50:13 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:50:13 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:23 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:50:23 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:50:23 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:26 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='164', error='Socket operation on non-socket (88)'
Jul 22 15:50:26 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='164', time_reopen='10'
Jul 22 15:50:33 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:50:33 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:50:33 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:36 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='164'
Jul 22 15:50:36 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='164', error='Socket operation on non-socket (88)'
Jul 22 15:50:36 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='164', time_reopen='10'
Jul 22 15:50:43 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:50:43 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:50:43 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:46 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='164'
Jul 22 15:50:46 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='164', error='Socket operation on non-socket (88)'
Jul 22 15:50:46 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='164', time_reopen='10'
Jul 22 15:50:53 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:50:53 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:50:53 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
Jul 22 15:50:56 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='164'
Jul 22 15:50:56 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='164', error='Socket operation on non-socket (88)'
Jul 22 15:50:56 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='164', time_reopen='10'
Jul 22 15:51:04 VDX1 syslog-ng[7282]: Error suspend timeout has elapsed, attempting to write again; fd='135'
Jul 22 15:51:04 VDX1 syslog-ng[7282]: I/O error occurred while writing; fd='135', error='Socket operation on non-socket (88)'
Jul 22 15:51:04 VDX1 syslog-ng[7282]: Suspending write operation because of an I/O error; fd='135', time_reopen='10'
VDX1# hide foscmd

 
Environment
  • VDX6740
  • NOS 7.0.1b
Cause
Occasionally the syslog-ng daemon can fail to form an internal connection it needs in order to collect and process messages from the kernel. The syslog.log file can become filled with messages reporting this failure and can grow to consume a significant portion of CF drive space.
Resolution
If the most recent messages at the end of syslog.log consistently mention the same file descriptor over and over again, for example "fd='135'" in the sample above, before trying to delete syslog.log there may be additional value in trying to determine what the file descriptor points to. The following set of commands on the NOS CLI should allow that, where 135 is replaced with the fd number seen in syslog.log.
unhide foscmd
foscmd "find /proc/*/fd -type l -name 135 -exec ls -l {} \;" | nomore
hide foscmd

Before attempting to delete or reduce the size of syslog.log, as in the Symptoms section above the tail command should be used to check the most recent lines and make sure they match the messages in the symptom section above.

As a short term solution, the syslog.log file can be deleted and the VDX can be reloaded. Whenever an active log file is deleted, in order to make its space available to the rest of the system, reload is necessary. For example:
VDX1# unhide foscmd
Password: ********
VDX1# foscmd "rm /var/log/syslog.log"
VDX1# foscmd "ls -l /var/log/syslog.log"
ls: /var/log/syslog.log: No such file or directory
VDX1# hide foscmd
VDX1# reload
Are you sure you want to reload the switch? [y/n]:y
The system is going down for reload NOW !!

Here is an alternate method of reducing the file's size to zero without deleting it.
VDX1# unhide foscmd
Password: ********
VDX1# foscmd "cat /dev/null > /var/log/syslog.log"
VDX1# hide foscmd

It is usually safe to delete or reduce syslog.log and then reload. There is a small possibility that syslog.log consuming all open storage space can trigger internal database corruption, and this corruption can cause subsequent boots to fail. Any attempt to delete or reduce syslog.log should be performed during a maintenance window. If the VDX fails to boot afterwards, it may be possible to recover using  netinstall. For more information on that option in case of failed boot please refer to the following.
How to perform netinstall on VDX 6740 and 6940 via USB?
TFTP/NFS netinstall on 6740/6940/8770

Options less drastic than netinstall do exist, but netinstall is the most certain method.

For a long term solution, software upgrade to any version with the fix for DEFECT000609232, such as NOS 7.2.0c or later, is necessary.
Additional notes
The release notes for NOS 7.2.0c list a workaround for DEFECT000609232 that requires adding "root enable" to the running-config under the affected rbridge-id. When DEFECT000609232 happens on a logical chassis VCS, it can become impossible to change the running config. "Message Generic Error" is returned by most config change attempts. NOS also does not include an editor like "vi" or "emacs" that can be reached with "foscmd". NOS does include a "sed" command that can be reached with "foscmd". Using "foscmd" and "sed", in theory it might be possible to recover from this issue without reloading.

The following example shows that the config lines described in the release note workaround can be found in the files described.
VDX7# foscmd "grep df_kern /etc/syslog-ng/syslog-ng.conf" | nomore
destination df_kern { pipe("/var/log/kmsg"); };
log { source(s_all); filter(f_kern); destination(df_kern); };
VDX7# foscmd "grep df_kern /etc/syslog-ng/syslog-ng.template" | nomore
destination df_kern { pipe("/var/log/kmsg"); };
log { source(s_all); filter(f_kern); destination(df_kern); };
VDX7#

In the lab trial below I output the edited versions to temporary files, check the result, and copy them back to the original filenames.
VDX7# foscmd "cat /etc/syslog-ng/syslog-ng.conf | sed -e '/df_kern/s/^/# /g' > /etc/syslog-ng/syslog-ng.conf.new" | nomore
VDX7# foscmd "grep df_kern /etc/syslog-ng/syslog-ng.conf.new" | nomore
# destination df_kern { pipe("/var/log/kmsg"); };
# log { source(s_all); filter(f_kern); destination(df_kern); };
VDX7# foscmd "cp /etc/syslog-ng/syslog-ng.conf.new /etc/syslog-ng/syslog-ng.conf"
VDX7# foscmd "grep df_kern /etc/syslog-ng/syslog-ng.conf" | nomore
# destination df_kern { pipe("/var/log/kmsg"); };
# log { source(s_all); filter(f_kern); destination(df_kern); };
VDX7#

VDX7# foscmd "cat /etc/syslog-ng/syslog-ng.template | sed -e '/df_kern/s/^/# /g' > /etc/syslog-ng/syslog-ng.template.new" | nomore
VDX7# foscmd "cp /etc/syslog-ng/syslog-ng.template.new /etc/syslog-ng/syslog-ng.template"
VDX7# foscmd "grep df_kern /etc/syslog-ng/syslog-ng.template" | nomore
# destination df_kern { pipe("/var/log/kmsg"); };
# log { source(s_all); filter(f_kern); destination(df_kern); };
VDX7#

"cat" and "sed" work line by line so sending to the original filename is not feasible. The version of "sed" included with NOS does not have the "-i" edit-in-place option, so that is not a possibility. If DEFECT000609232 is consuming all disk space, sending output to temporary files might not be allowed either. One might need to clear out the existing syslog.log with 'foscmd "cat /dev/null > /var/log/syslog.log"'  or 'foscmd "rm /var/log/syslog.log"' first. One might need to free up disk space by deleting old core files with "clear support" first.

The following shows one method of clearing out a log file while it is open and being written to.
VDX7# foscmd "cat /dev/null > /var/log/syslog.log"
VDX7# foscmd "ls -la /var/log/syslog.log"
-rw-r-----   1 root     root            0 Nov 28 16:56 /var/log/syslog.log
VDX7#

The release note workaround shows the "killall" command executed from the root login. That can be executed with "foscmd" as follows.
VDX7# foscmd "/usr/bin/killall -HUP syslog-ng"

Again, if the release note method above is not successful, the simplest most assured method remaining is to delete syslog.log and reload the VDX.
 

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255