Reset Search
 

 

Article

Switch Cored After Reaching 95% Memory When Flight Recorder Run

« Go Back

Information

 
TitleSwitch Cored After Reaching 95% Memory When Flight Recorder Run
Symptoms
  • One switch in vIST cored after reaching 95% memory utilization
  • Log events leading up to core
2020-05-19T10:54:13.876-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:4056
2020-05-19T10:59:15.041-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:40722020-05-19T11:04:17.198-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:4090
2020-05-19T11:09:17.439-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:41061 2020-05-20T10:25:04.600-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:9049
2020-05-20T10:29:41.744-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:90661 2020-05-20T10:48:11.474-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:9135
2020-05-20T10:52:45.623-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:9150
2020-05-20T10:57:22.853-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:91661 2020-05-21T01:01:12.010-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:12330
2020-05-21T01:05:33.163-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:123451 2020-05-21T01:09:54.392-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:12361
2020-05-21T01:14:15.326-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:12376
<SNIP>
2020-05-21T10:26:18.013-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21732 SSH:X.X.X.X GTAC en
2020-05-21T10:26:19.749-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21733 SSH:X.X.X.X GTAC term mor dis
2020-05-21T10:26:54.049-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21734 SSH:X.X.X.X GTAC show khi performance memory
2020-05-21T10:27:53.230-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:14580
2020-05-21T10:28:09.030-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21735 SSH:X.X.X.X GTAC show khi performance process
2020-05-21T10:28:37.869-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21736 SSH:X.X.X.X GTAC show khi performance pthread
2020-05-21T10:29:39.622-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21737 SSH:X.X.X.X GTAC show fulltech
2020-05-21T10:31:17.934-04:00 VSP CP1 - 0x000d8602 - 00000000 GlobalRouter SSH INFO X509v3 host certificate is unavailable
2020-05-21T10:31:19.145-04:00 VSP CP1 - 0x000a45ec - 00000000 GlobalRouter RADIUS INFO RADIUS authentication successful
2020-05-21T10:31:19.145-04:00 VSP CP1 - 0x000d8602 - 00000000 GlobalRouter SSH INFO SSH user authentication succeeded for user GTAC-1 on host Y.Y.Y.Y
2020-05-21T10:31:19.151-04:00 VSP CP1 - 0x000d8602 - 00000000 GlobalRouter SSH INFO SSH CLI session start: user GTAC-1 on host Y.Y.Y.Y
2020-05-21T10:31:21.177-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21738 SSH:Y.Y.Y.Y GTAC-1 terminal more disable
2020-05-21T10:31:21.657-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21739 SSH:Y.Y.Y.Y GTAC-1 enable
2020-05-21T10:31:22.184-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21740 SSH:Y.Y.Y.Y GTAC-1 show app-telemetry counter
2020-05-21T10:31:22.758-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21741 SSH:Y.Y.Y.Y GTAC-1 show interfaces gigabitEthernet statistics
2020-05-21T10:31:25.277-04:00 VSP CP1 - 0x000d8602 - 00000000 GlobalRouter SSH INFO SSH CLI session end: user GTAC-1 on host Y.Y.Y.Y
2020-05-21T10:31:25.279-04:00 VSP CP1 - 0x000d8602 - 00000000 GlobalRouter SSH INFO SSH session closed by user GTAC-1 on host Y.Y.Y.Y
2020-05-21T10:31:58.366-04:00 VSP IO1 - 0x0027042b - 00000000 GlobalRouter SW INFO Process oom90 started, pid:14598
2020-05-21T10:33:39.428-04:00 VSP CP1 - 0x002c0600 - 00000000 GlobalRouter CLILOG INFO 21742 SSH:X.X.X.X GTAC flight-recorder all all
2020-05-21T10:34:09.095-04:00 VSP IO1 - 0x00270438 - 00000000 GlobalRouter SW INFO LifeCycle: INFO: Starting collecting data for archive
2020-05-21T10:34:10.345-04:00 VSP IO1 - 0x00270439 - 00000000 GlobalRouter SW INFO LifeCycle: INFO: Creating final archive file
2020-05-21T10:34:17.319-04:00 VSP IO1 - 0x00270419 - 00000000 GlobalRouter SW ERROR Memory reached critical level of 95% utilization, generating core dump

 
Environment
  • VSP 8404
  • Release 8.0.8.0
Cause
Product functioning as designed
Resolution
  • Memory was already elevated to 90% based on oom90 messages written to log
  • Memory used by running flight recorder pushed switch to 95% memory and caused crash by design
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255