Reset Search
 

 

Article

Switch Reboot When Reaching 95% Memory Utilization After DBSync Flood In DVR Environment

« Go Back

Information

 
TitleSwitch Reboot When Reaching 95% Memory Utilization After DBSync Flood In DVR Environment
Symptoms
  • One switch in vIST rebooted after reaching 95% memory utilization following flood of DBSync messages in DVR environment
05/06/20 13:31:09.142:EDT] 0x000a45ec 00000000 GlobalRouter RADIUS INFO RADIUS authentication successful
05/06/20 13:31:09.142:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH user authentication succeeded for user <User> on host x.x.x.x
05/06/20 13:31:09.176:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH CLI session start: user <User> on host x.x.x.x
05/06/20 13:31:11.208:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22588 SSH:x.x.x.x <User> terminal more disable
05/06/20 13:31:11.692:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22589 SSH:x.x.x.x <User> enable
05/06/20 13:31:12.204:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22590 SSH:x.x.x.x <User> show app-telemetry counter
05/06/20 13:31:14.290:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22591 SSH:x.x.x.x <User> show interfaces gigabitEthernet statistics
05/06/20 13:31:20.018:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH CLI session end:  user <User> on host x.x.x.x
05/06/20 13:31:20.018:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH session closed by user <User> on host x.x.x.x
05/06/20 13:32:41.858:EDT] 0x00210595 08400002.1 DYNAMIC SET GlobalRouter KHI WARNING CPU utilization 5-minute average is 91, above 90%
05/06/20 13:34:01.139:EDT] 0x0027041a 00900001.1 DYNAMIC SET GlobalRouter SW WARNING Memory reached 89% utilization, module will reboot when 95% utilization is reached
05/06/20 13:35:01.163:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:28793

05/06/20 13:35:13.408:EDT] 0x00390605 00000000 GlobalRouter DBSYNC ERROR Message from tMain to DB Sync not queued, error: 4. Restart PIM-GW and/or DVR subsystem on this node!
05/06/20 13:35:13.408:EDT] 0x00390600 00000000 GlobalRouter DBSYNC ERROR DB Sync software error DBSyncRxPktAPI 325

05/06/20 13:36:53.574:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:28810
05/06/20 13:38:35.883:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:28825
05/06/20 13:39:16.429:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  2 27 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 14 bc e1 43 2c 5f 74 0 0 0 3 0 20 0 0 0 0 1 0
05/06/20 13:39:16.680:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 8 0 11 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:16.681:EDT] 0The previous message repeated 1 time(s).  
05/06/20 13:39:16.683:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 14 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:16.825:EDT] 0The previous message repeated 1 time(s).  
05/06/20 13:39:17.455:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 8 0 11 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.473:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  2 27 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 14 bc e1 43 2c 5f 74 0 0 0 3 0 20 0 0 0 0 1 0
05/06/20 13:39:17.559:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 14 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.560:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 8 0 11 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.566:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 13 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.783:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 14 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.829:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 8 0 11 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.830:EDT] 0x00390602 00000000 GlobalRouter DBSYNC WARNING TLV not found in local DB Sync data base, deletion failed! TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 9 0 13 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.920:EDT] 0x00390601 00000000 GlobalRouter DBSYNC WARNING DB Sync Duplicate TLV:  5 25 0 0 0 0 0 0 0 0 81 bc a 3 d4 59 20 0 0 80 8 0 11 bc e1 43 2c 5f 74 0 0 80 9 0 14 0 0 0 3
05/06/20 13:39:17.921:EDT] 0The previous message repeated 1 time(s).  

05/07/20 01:00:23.892:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH CLI session start: user <User> on host x.x.x.x
05/07/20 01:00:25.904:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22787 SSH:x.x.x.x <User> terminal more disable
05/07/20 01:00:26.412:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22788 SSH:x.x.x.x <User> enable
05/07/20 01:00:26.926:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22789 SSH:x.x.x.x <User> show app-telemetry counter
05/07/20 01:00:27.453:EDT] 0x002c0600 00000000 GlobalRouter CLILOG INFO 22790 SSH:x.x.x.x <User> show interfaces gigabitEthernet statistics
05/07/20 01:00:29.975:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH CLI session end:  user <User> on host x.x.x.x
05/07/20 01:00:29.975:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH session closed by user <User> on host x.x.x.x
05/07/20 01:00:58.398:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4613
05/07/20 01:02:01.481:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4628
05/07/20 01:03:01.549:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4631
05/07/20 01:04:04.622:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4646
05/07/20 01:05:05.993:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4661
05/07/20 01:06:09.089:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4676
05/07/20 01:07:09.139:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4679
05/07/20 01:08:12.646:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4694
05/07/20 01:09:15.724:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4709
05/07/20 01:10:15.885:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4724
05/07/20 01:11:17.956:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4739
05/07/20 01:12:20.040:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4754
05/07/20 01:13:24.117:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4769
05/07/20 01:14:24.198:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4778
05/07/20 01:15:17.054:EDT] 0x00007fff 00000000 GlobalRouter SNMP INFO request  received from x.x.x.x  with invalid authorization  
05/07/20 01:15:21.008:EDT] 0The previous message repeated 484 time(s).
05/07/20 01:15:24.273:EDT] 0x0027042b 00000000 GlobalRouter SW INFO Process oom90 started, pid:4785
05/07/20 01:15:29.038:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO X509v3 host certificate is unavailable
05/07/20 01:15:30.282:EDT] 0x000a45ec 00000000 GlobalRouter RADIUS INFO RADIUS authentication successful
05/07/20 01:15:30.282:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH user authentication succeeded for user <User> on host x.x.x.x
05/07/20 01:15:30.288:EDT] 0x000d8602 00000000 GlobalRouter SSH INFO SSH CLI session start: user <User> on host x.x.x.x
05/07/20 01:15:30.324:EDT] 0x00270419 00000000 GlobalRouter SW ERROR Memory reached critical level of 95% utilization, generating core dump

 
Environment
  • VSP 8404
  • Release 8.0.8.0
  • DVR (Distributed Virtual Routing)
Cause
Software defect VOSS-17820
Resolution
Resolved in 8.1.7 release
Additional notes

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255