Reset Search
 

 

Article

VSP Stuck In Reboot Loop

« Go Back

Information

 
TitleVSP Stuck In Reboot Loop
Symptoms

Following logs were discovered during investigation:

CP1 [11/13/17 00:27:45] rc.appfs.vsp4k: Start mkdir: can't create directory '/intflash/PMEM/1': Read-only file system
CP1 [11/13/17 00:27:45] rc.appfs.vsp4k: /opt/appfs/plat -> /opt/appfs/p4000
CP1 [11/13/17 00:27:45] rc.cgroups_class: Cgroup filesystem (dev/cpuctl) mounted for CFS
CP1 [11/13/17 00:27:45] rc.cgroups_class: Cgroup initialization complete.
mv: can't rename '/intflash/wd_stats/1/wd_stats.cbcp-main.x.1.log': Read-only file system
mv: can't rename '/intflash/runprof/1/runprof.cbcp-main.x.tMainTask.log': Read-only file system
CP1 [11/13/17 00:27:45] rc.appfs.vsp4k: Checking if tap_enet daemon is running ... [NO]
CP1 [11/13/17 00:27:47] ins-klms.SP: Detected card type: cp
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting wdd KLM insmod: can't insert '/opt/appfs/klm/cp/2.6.32-avaya_vsp4k/wdd.ko': File exists mknod: /dev/wdd: File exists
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting TIPC
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting tipc KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Skipping TUN modprobe
CP1 [11/13/17 00:27:47] ins-klms.SP: inserting KLMs from appfs
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting evm KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting i2cs KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting oxate KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting oxide KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting carbonate KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting iodate KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting k2 KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Creating device nodes for K2
CP1 [11/13/17 00:27:47] ins-klms.SP: Creating K2 device files
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting BDE modules
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting linux-kernel-bde KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Inserting linux-user-bde KLM
CP1 [11/13/17 00:27:47] ins-klms.SP: Creating device nodes for bde KLM {dev-id:126}
rm: can't remove '/intflash/qos_profile': Read-only file system
CP1 [11/13/17 04:27:48] Start: Starting Applications
mkdir: can't create directory '/intflash/trace/last_run': Read-only file system
CP1 [11/13/17 04:27:48] Start: Detected vsp4k Platform, disable patching
touch: /intflash/nopatcher: Read-only file system
CP1 [11/13/17 04:27:49.000] LifeCycle: ERROR: Could not create or open file: /intflash/restart_counter
CP1 [11/13/17 04:27:49.000] LifeCycle: INFO: Initial start
CP1 [11/13/17 04:27:49.000] LifeCycle: INFO: Starting all processes
CP1 [11/13/17 04:27:51.000] Image Sync: Primary Release = VOSS4K.5.1.1.6.GA
CP1 [11/13/17 04:27:51.000] Image Sync: Backup Release = VOSS4K.6.1.0.0.GA
[04:27:51]: Slot 1: ERR: IMGSYNCAPP: PutChassisNvram: unable to open chassis NVRAM
BANK1: VOSS4K.5.1.1.6.GA (CG)
BANK2: VOSS4K.6.1.0.0.GA
------------------------------------------------------------
Image Type: Kernel
Bank 1: Release Version = VOSS4K.5.1.1.6int005
Bank 2: Release Version = VOSS4K.6.1.0.0int041
Image Type: Root_FS
Bank 1: Release Version = VOSS4K.5.1.1.6int005
Bank 2: Release Version = VOSS4K.6.1.0.0int041
Image Type: APP_FS
Bank 1: Release Version = VOSS4K.5.1.1.6int005
Bank 2: Release Version = VOSS4K.6.1.0.0int041
CP1 [11/13/17 04:27:51.000] LifeCycle: INFO: All processes started
Attaching network interface lo0... done.
WARNING: Check dummy: modes exec monitor
WARNING: Check dummy: modes exec show debug
WARNING: Check dummy: modes global_configuration debug
WARNING: Check dummy: modes global_configuration default debug
WARNING: Check dummy: modes global_configuration no debug
CP1 [11/13/17 04:27:59.189:GST] 0x00030609 00000000 GlobalRouter SW WARNING Unable to write fabric profile configuration file. Error=30
CP1 [11/13/17 04:27:59.636:GST] 0x000006cc 00000000 GlobalRouter SW INFO rcStart: FIPS Power Up Self Test SUCCESSFUL - 0
CP1 [11/13/17 04:27:59.638:GST] 0x000006c2 00000000 GlobalRouter SW INFO rcStart: Security Stack Init SUCCESSFUL - 0
CP1 [11/13/17 04:27:59.639:GST] 0x000006c3 00000000 GlobalRouter SW INFO rcStart: IPSEC Init SUCCESSFUL
CP1 [11/13/17 04:27:59.639:GST] 0x000006bf 00000000 GlobalRouter SW INFO rcStart: Security Stack Log init SUCCESSFUL - 0
CP1 [11/13/17 04:28:01.819:GST] 0x000005ba 00000000 GlobalRouter SW INFO License Successfully Loaded From License Type: PREMIER+MACSEC
Virtual Services Platform 4000
VSP Operating System Software Release 5.1.1.6
Copyright (c) 2017 Avaya, Inc.
Not able to create LC file for Recovery use
CP1 [11/13/17 04:28:26.324:GST] 0x000006c1 00000000 GlobalRouter SW INFO rcStart: SSH Module Init Successful - 0
CP1 [11/13/17 04:28:26.554:GST] 0x00034594 00000000 GlobalRouter SW INFO System boot
CP1 [11/13/17 04:28:26.554:GST] 0x00034595 00000000 GlobalRouter SW INFO VSP-4000 VSP Operating System Software Release 5.1.1.6
write_file:can't open "/intflash/alarmLog" 0x1e S_errno_EROFS
write_file:can't open "/intflash/alarmLog" 0x1e S_errno_EROFS
CP1 [11/13/17 04:28:27.526:GST] 0x000045f6 00000000 GlobalRouter SNMP ERROR unable to create file: /intflash/engboot.
CP1 [11/13/17 04:28:27.537:GST] 0x0001081c 00400010.1 DYNAMIC SET GlobalRouter HW INFO Slot 1 is initializing.
CP1 [11/13/17 04:28:27.537:GST] 0x00010774 00000000 GlobalRouter HW INFO Detected 4850GTS-PWR+ chassis
CP1 [11/13/17 04:28:27.624:GST] 0x00010729 00000000 GlobalRouter HW INFO Detected 4850GTS-PWR+ Power Supply in slot PS 1. Adding 1000 watts to available power
CP1 [11/13/17 04:28:27.631:GST] 0x00010729 00000000 GlobalRouter HW INFO Detected 4850GTS-PWR+ Power Supply in slot PS 2. Adding 1000 watts to available power
CP1 [11/13/17 04:28:27.652:GST] 0x00010830 00000000 GlobalRouter HW INFO Detected 4850GTS-PWR+ module (Serial#: 17JP3520E333) in slot 1
CP1 [11/13/17 04:28:28.110:GST] 0x0001081d 00400010.1 DYNAMIC CLEAR GlobalRouter HW INFO Slot 1 is finished initialization.
systemDefaults: fail to open build.txt file
/intflash/.shadovnonfedmoc.txt:open failed: CP1 [11/13/17 04:28:29.971:GST] 0x0002475c 00000000 GlobalRouter CPU INFO CPP Rx Qs Initialised num: 8
CP1 [11/13/17 04:28:29.975:GST] 0x000e0598 00000000 GlobalRouter HAL INFO GBIC inserted in slot 1 Port 49 Type:10GbCX Vendor:Avaya
CP1 [11/13/17 04:28:29.976:GST] 0x000e0598 00000000 GlobalRouter HAL INFO GBIC inserted in slot 1 Port 50 Type:10GbCX Vendor:Avaya
CP1 [11/13/17 04:28:30.379:GST] 0x00300604 00000000 GlobalRouter SLAMON INFO Starting SLAMon client initialization
CP1 [11/13/17 04:28:30.728:GST] 0x00300604 00000000 GlobalRouter SLAMON INFO SLAMon client conf.started 1 wait time 33
CP1 [11/13/17 04:28:30.797:GST] 0x00300604 00000000 GlobalRouter SLAMON INFO Starting Slamon Monitor Agent Controller
CP1 [11/13/17 04:28:31.627:GST] 0x00040601 00000000 GlobalRouter WEB INFO HTTPS: Using the existing Server Cert/Key
CP1 [11/13/17 04:28:32.504:GST] 0x00300604 00000000 GlobalRouter SLAMON INFO SLA Monitor Agent Controller started
CP1 [11/13/17 04:28:38.442:GST] 0x00300604 00000000 GlobalRouter SLAMON INFO slamonLoop: Entering main loop
CP1 [11/13/17 04:28:54.584:GST] 0x0008850f 00000000 GlobalRouter SW INFO Waiting for all cards to be ready for configuration download
CP1 [11/13/17 04:28:56.203:GST] 0x00010750 00000000 GlobalRouter HW INFO Module 4850GTS-PWR+ in slot 1 is ready for configuration download
CP1 [11/13/17 04:28:56.203:GST] 0x00010758 00000000 GlobalRouter HW INFO Downloading configuration to all cards
CP1 [11/13/17 04:28:56.208:GST] 0x00088512 00000000 GlobalRouter SW INFO Loading configuration from /intflash/config.cfg
Mon Nov 13 04:28:57 GST 2017
CP1 [11/13/17 04:28:57.000] LifeCycle: WARNING: Critical process is dying (pid:4676)!
CP1 [11/13/17 04:28:57.000] LifeCycle: INFO: Turning off s/w watchdog ...
CP1 [11/13/17 04:28:57.000] LifeCycle: INFO: Stopping all processes
CP1 [11/13/17 04:28:57.000] LifeCycle: INFO: Starting persistent memory archive
CP1 [11/13/17 04:28:57.000] LifeCycle: INFO: Persistent archive lock acquired
CP1 [11/13/17 04:28:58.000] LifeCycle: INFO: Persistent memory archive file created
CP1 [11/13/17 04:28:58.000] LifeCycle: Could not find any flight recorder data files in directory: /intflash/PMEM/1
CP1 [11/13/17 04:28:58.000] LifeCycle: INFO: Persistent memory archive completed
CP1 [11/13/17 04:28:58.000] LifeCycle: INFO: Crash reporter installed for this process, dumping data Lifecycle Crash Reporter: Process Name: cbcp-main.x, Thread Name: tLoadConfig, Signal: 11, Slot: 1, PID 4676, LWP: 5063
[bt] Execution path:
[bt] /opt/appfs/lib/cp/libndlcs.so.1(_Z30nd_lcs_crash_exception_handleriP7siginfoPv+0x184) [0xb396628]
[bt] [0x100350]
[bt] [0x22002422]
[bt] cbcp-main.x(osal_fread+0x38) [0x1122ba38]
[bt] cbcp-main.x(restoreMoc_sshc_host_key+0x58) [0x10c36674]
[bt] cbcp-main.x(sshc_moc_init+0x1cc) [0x10c3698c]
[bt] cbcp-main.x(sshdEnable+0x250) [0x10c42f98]
[bt] cbcp-main.x(rcSshGlobalGrpSetBody+0x1dc) [0x10c2b164]
[bt] /opt/appfs/plat/lib/cp/libnncli.so.1(configsshFunc+0x124) [0xd5ba018]
[bt] /opt/appfs/plat/lib/cp/libnncli.so.1(+0x8bd64) [0xd4f2d64]
[bt] /opt/appfs/plat/lib/cp/libnncli.so.1(nncli_executeTree+0x494) [0xd4f3520]
[bt] /opt/appfs/plat/lib/cp/libnncli.so.1(nncli_execute_line+0xc8) [0xd4f3930]
[bt] /opt/appfs/plat/lib/cp/libnncli.so.1(nncli_execute_cli+0x428) [0xd4f406c]
[bt] /opt/appfs/plat/lib/cp/libnncli_parser.so.1(dm_nncli+0x364) [0xd75c334]
[bt] /opt/appfs/lib/cp/libboot.so.1(+0x25f80) [0xff22f80]
[bt] /opt/appfs/lib/cp/libv2l.so.1(task_wrapper+0x274) [0xb1cb970]
[bt] cbcp-main.x(ckrmThreadStarter+0x6c4) [0x114f22ec]
[bt] /lib/libpthread.so.0(+0x6830) [0xac68830]
INIT: Sending processes the TERM signale1c]
Stopping OpenBSD Secure Shell server: sshdstopped /usr/sbin/sshd (pid 1997)
 



See "Additional Notes" at bottom of article for more data.

Environment
VSP 4850 stuck in reboot loop running software 5.1.1.6
Cause

Issue is due to a hardware defect.

Resolution

Replace faulty switch

Additional notes
VSP 4850 switches running software other than 5.1.1.6 may also experience a boot loop issue with similar errors shown below:
cp: can't create '/boot/flash/board-seep.bin.14': Read-only file system
 
LifeCycle: ERROR: Could not create or open file: /intflash/restart_counter
 
write_file:can't open "/intflash/alarmLog" 0x1e
S_errno_EROFS
CP1  [01/10/70 01:28:57.000] LifeCycle: WARNING: Critical process is dying (pid:4681)!
CP1  [01/10/70 01:28:57.000] LifeCycle: INFO: Turning off s/w watchdog ...
CP1  [01/10/70 01:28:57.000] LifeCycle: INFO: Stopping all processes
CP1  [01/10/70 01:28:57.000] LifeCycle: INFO: Starting persistent memory archive
CP1  [01/10/70 01:28:57.000] LifeCycle: INFO: Persistent archive lock acquired
CFG: Msg could not be sent
 
ERR: IMGSYNCAPP: PutChassisNvram: unable to open chassis NVRAM
 
CP1  [01/10/70 01:28:58.000] LifeCycle: INFO: Process cbcp-main.x (4681) died, exit status: uncaught signal: 11 (core dumped)
CP1  [01/10/70 01:29:04.000] LifeCycle: INFO: Persistent memory archive file created
CP1  [01/10/70 01:29:04.000] LifeCycle: Could not find any flight recorder data files in  directory: /intflash/PMEM/1
CP1  [01/10/70 01:29:04.000] LifeCycle: INFO: Persistent memory archive completed
CP1  [01/10/70 01:29:04.000] LifeCycle: INFO: Crash reporter installed for this process, dumping data
 
Lifecycle Crash Reporter: Process Name: cbcp-main.x, Thread Name: tSnmpd, Signal: 11, Slot: 1, PID 4681, LWP: 4948
[bt] Execution path:
[bt] /opt/appfs/lib/cp/libndlcs.so.1(_Z30nd_lcs_crash_exception_handleriP7siginfoPv+0x1ac)[0xa77b964]
[bt] [0x100350]
[bt] /opt/appfs/lib/cp/libboot.so.1(bfile_remove+0x44)[0xfefe448]
[bt] cbcp-main.x(snmpLoadEngineID+0x104)[0x10eb5e8c]
[bt] cbcp-main.x(snmpIoMain+0x1c)[0x1023b23c]
[bt] cbcp-main.x[0x102394cc]
[bt] /opt/appfs/lib/cp/libv2l.s
INIT: Sending processes the TERM signal

Errors can vary per incident; however, in both cases, if the errors are examined closely, will show evidence of USB / Memory failure(s).

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255