Reset Search
 

 

Article

How to configure dynamic buffers for VDX6740 to address shortage of rcv buffer on VDX6740 platform

« Go Back

Information

 
TitleHow to configure dynamic buffers for VDX6740 to address shortage of rcv buffer on VDX6740 platform
Symptoms
As published in TSB 2014-182 -A on March 7, 2014, Enhanced Dynamic Buffering mechanism is introduced to the VDX6740 platform to address the buffer shortage issue.
...
Problem Statement:
During bursty traffic flows, the buffers in the VDX6740 may be over-utilized causing traffic to be tail-dropped as RX Discards in the output of show interface <>. One example of this occurrence is bursty congestion due to traffic going from Servers with 10G NICs towards the Servers with 1G NICs.

Symptom:
In the event of bursty traffic, the switch may exhibit a high number of Rx discards, causing various symptoms such as application timeouts, intermittent packet losses, intermittent ping drops, etc.

Corrective Action:
An enhanced dynamic buffering mechanism has been introduced for VDX6740 series switches which would efficiently utilize the buffering capability in the hardware to manage congestion due to bursty traffic flows.
It is recommended to upgrade to the above versions to utilize the new functionality, which is enabled by default.
The dynamic buffering mechanism offers a shared pool among the interfaces in addition to the statically allocated buffers per port, per priority. This allows the VDX6740 to absorb more bursts than the ports static buffer can handle.

The problem is tracked by Defect000493800, and is patched to releases, NOS 4.0.1b1, 4.01c, 4.0.2, 4.1.1, 5.0.0

See related KB article FAQ 2538for how to read the output of show qos rcv-queue int <>
Environment
Software Release: How to configure dynamic buffers for VDX6740 to address shortage of rcv buffer on VDX6740 platform
Fixed in Version: NOS 4.0.1b1, 4.01c, 4.0.2, 4.1.1, 5.0.0
Cause
During bursty traffic flows, the buffers in the VDX6740 may be over-utilized causing traffic to be tail-dropped as RX Discards in the output of show interface <>. One example of this occurrence is bursty congestion due to traffic going from Servers with 10G NICs towards the Servers with 1G NICs.

Prior to the enhancement of Dynamic Buffering enhancement, Tail drop will start for each TC once it reaches the allocated Byte count of that TC - no sharing, no borrowing is allowed.

With the Dynamic Buffering enhancement, the SW can use the spare buffers available in other TC, that is not handling traffics, for up to the MAX Buffer indicated under the interface for all TC.
Resolution
Despite Dynamic Buffer sharing introduced in 4.x, Tail drops can still continue to occur if the extent of Congestion is high.

For example: In scenarios where multiple 10G servers communicate with multiple 1G server connections >To handle such scenarios, NOS 5.x code has been further enhanced to provide flexibility for users to increase the shared buffer pools for each Traffic class to a maximum of 2MB(285KB in NOS4.x). The CLIfor "QOS RCV-QUEUE LIMIT" is created under Rbrdige configuration to allow increasing the allocation of buffers for each TCup to 2Mb.

Below is the log capturing the steps in setting a 40G ISL port between 2 VDX6740 switches, that is seeing rx Discard during packet burst testing.

We confirm the Rx Discard in SupportSave:

Port 40 CCB Memory Statistics 0/0/40=userport 49 000000000000066a 1,642 Unicast Tail Dropped Packets
CCB Memory Statistics: 1
EMC Block counters
...


The buffering will be more useful during congestion scenarios.

Key points :
1. Must configure from the principal switch, otherwsie you got an error message %Error: This operation is not supported from a secondary node
2. The cli command needs to be configured under the Rbridge(s) of choice, and for each one where you want to tune the rcv-queue limit
3. The configuration can be reverted with no qos rcv-queue limit xxxx to return to the default.

Example of the configuration:

VDX6740T login: admin
Password:
SECURITY WARNING: The default password for at least
one default account (root, admin and user) have not been changed.
Welcome to the Brocade Network Operating System Software
admin connected from 10.110.34.131 using console on VDX6740T
VDX6740T# sh vcs
Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 10
VCS GUID : bc70443c-e996-4db8-90d0-155625da65c9
Total Number of Nodes : 2

Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
12 >10:00:00:27:F8:D4:3A:1C* 10.17.139.30 Online Online VDX6740T
52 10:00:50:EB:1A:08:FE:99 10.17.139.31 Online Online VDX6740T-2

VDX6740T# show qos rcv-q int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T# show qos rcv-q int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T# conf t
Entering configuration mode terminal

VDX6740T(config)# rbridge 12
VDX6740T(config-rbridge-id-12)# qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
< ---- configuration is 2000 , i.e 2MB for each TC for the entire RBridge
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< --- default 285 KB per TC( 285*1025=291840)
1 0 291840
2 0 291840
3 0 148868
< ----TCthat mapped to COS3 with PFC on need less
4 0 291840
5 0 291840
6 0 291840
7 0 148868
< ---- TCthat mapped to COS3 with PFC on need less

VDX6740T(config-rbridge-id-12)# exit
VDX6740T(config)# rbridge 52
VDX6740T(config-rbridge-id-52)# qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-52)# no qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< ----- revert back to default
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T(config-rbridge-id-52)# exit
VDX6740T(config)# rbridge 12
VDX6740T(config-rbridge-id-12)# no qos rcv-queue limit 2000
< ----- revert back to default

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< ------ default 285KB buffers
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868
VDX6740T(config-rbridge-id-12)#

Additional notes
CoS tail drop threshold can be configured to increase the buffering capability in ingress to some extent using CLI :

qos rcv-queue cos-threshold TDT0 {TDT1 | TDT2 | TDT3 | TDT4 | TDT5 | TDT6 | TDT7}

This could marginally improve the performance.

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255