Reset Search
 

 

Article

How to configure dynamic buffers for VDX6740 to address shortage of rcv buffer on VDX6740 platform

« Go Back

Information

 
TitleHow to configure dynamic buffers for VDX6740 to address shortage of rcv buffer on VDX6740 platform
Symptoms
As published in TSB 2014-182-A on March 7, 2014, Enhanced Dynamic Buffering mechanism is introduced to the VDX6740 platform to address the buffer shortage issue.
...
Problem Statement:
During bursty traffic flows, the buffers in the VDX6740 may be over-utilized causing traffic to be tail-dropped as RX Discards in the output of "show interface". One example of this occurrence is bursty congestion due to traffic going from Servers with 10G NICs towards the Servers with 1G NICs.

Symptom:
In the event of bursty traffic, the switch may exhibit a high number of Rx discards, causing various symptoms such as application timeouts, intermittent packet losses, intermittent ping drops, etc.

Corrective Action:
An enhanced dynamic buffering mechanism has been introduced for VDX6740 series switches which would efficiently utilize the buffering capability in the hardware to manage congestion due to bursty traffic flows.
It is recommended to upgrade to the above versions to utilize the new functionality, which is enabled by default.
The dynamic buffering mechanism offers a shared pool among the interfaces in addition to the statically allocated buffers per port, per priority. This allows the VDX6740 to absorb more bursts than the ports static buffer can handle.

The problem is tracked by DEFECT000493800, and is patched to releases, NOS 4.0.1b1, 4.01c, 4.0.2, 4.1.1, 5.0.0. Commands to manually adjust buffering were added in NOS 4.1.3.

See related KCS article 000020494 for how to read the output of show qos rcv-queue int <>.
How to read the output of "show qos rcv-queue int"?
Environment
  • VDX6740, VDX6740T, VDX6740T-1G
  • Enhanced Dynamic Buffering is implemented in versions NOS 4.0.1b1, 4.01c, 4.0.2, 4.1.1, 5.0.0 and later
  • Commands to manually adjust buffering are implemented in NOS 4.1.3 and later
Cause
During bursty traffic flows, the buffers in the VDX6740 may be over-utilized causing traffic to be tail-dropped as RX Discards in the output of show interface <>. One example of this occurrence is bursty congestion due to traffic going from Servers with 10G NICs towards the Servers with 1G NICs.

Prior to the enhancement of Dynamic Buffering enhancement, Tail drop will start for each TC once it reaches the allocated Byte count of that TC - no sharing, no borrowing is allowed.

With the Dynamic Buffering enhancement, the SW can use the spare buffers available in other TC, that is not handling traffics, for up to the MAX Buffer indicated under the interface for all TC.
Resolution
Despite Dynamic Buffer sharing introduced in 4.0.1b1, 4.01c, 4.0.2, 4.1.1, and 5.0.0, Tail drops can still continue to occur if the extent of congestion is high. Versions NOS 4.1.3 and later include settings to further reduce drops dues to congestion.

For example: In scenarios where multiple 10G servers communicate with multiple 1G server connections >To handle such scenarios, NOS 4.1.3 was further enhanced to provide flexibility for users to increase the shared buffer pools for each Traffic class to a maximum of 2MB(285KB in NOS 4.1.3). The CLI command for "qos rcv-queue limit" can be added under Rbrdige settings to allow increasing the allocation of buffers for each TC up to 2Mb.

Below is the log capturing the steps in setting a 40G ISL port between 2 VDX6740 switches, that is seeing rx Discard during packet burst testing.

Extreme TAC is able to confirm Rx Discards in SupportSaves:

Port 40 CCB Memory Statistics 0/0/40=userport 49 000000000000066a 1,642 Unicast Tail Dropped Packets
CCB Memory Statistics: 1
EMC Block counters
...


Manual adjustment of buffering will be more useful during potential congestion scenarios.

Key points :
1. Must configure from the principal switch, otherwsie you got an error message %Error: This operation is not supported from a secondary node
2. The command "qos rcv-queue limit" needs to be configured under the Rbridge(s) of choice, and for each one where you want to tune the rcv-queue limit
3. The configuration can be reverted with  "no qos rcv-queue limit" to return to the default.

Example of the configuration:

VDX6740T login: admin
Password:
SECURITY WARNING: The default password for at least
one default account (root, admin and user) have not been changed.
Welcome to the Brocade Network Operating System Software
admin connected from 10.110.34.131 using console on VDX6740T
VDX6740T# sh vcs
Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 10
VCS GUID : bc70443c-e996-4db8-90d0-155625da65c9
Total Number of Nodes : 2

Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
12 >10:00:00:27:F8:D4:3A:1C* 10.17.139.30 Online Online VDX6740T
52 10:00:50:EB:1A:08:FE:99 10.17.139.31 Online Online VDX6740T-2

VDX6740T# show qos rcv-q int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T# show qos rcv-q int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T# conf t
Entering configuration mode terminal

VDX6740T(config)# rbridge 12
VDX6740T(config-rbridge-id-12)# qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
< ---- configuration is 2000 , i.e 2MB for each TC for the entire RBridge
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< --- default 285 KB per TC( 285*1025=291840)
1 0 291840
2 0 291840
3 0 148868
< ----TCthat mapped to COS3 with PFC on need less
4 0 291840
5 0 291840
6 0 291840
7 0 148868
< ---- TCthat mapped to COS3 with PFC on need less

VDX6740T(config-rbridge-id-12)# exit
VDX6740T(config)# rbridge 52
VDX6740T(config-rbridge-id-52)# qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 8388608 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 2048000
1 0 2048000
2 0 2048000
3 0 148868
4 0 2048000
5 0 2048000
6 0 2048000
7 0 148868

VDX6740T(config-rbridge-id-52)# no qos rcv-queue limit 2000

VDX6740T(config-rbridge-id-52)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< ----- revert back to default
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T(config-rbridge-id-52)# exit
VDX6740T(config)# rbridge 12
VDX6740T(config-rbridge-id-12)# no qos rcv-queue limit 2000
< ----- revert back to default

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 12/0/49
Interface FortyGigabitEthernet 12/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
< ------ default 285KB buffers
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868

VDX6740T(config-rbridge-id-12)# do show qos rcv-queue int fo 52/0/49
Interface FortyGigabitEthernet 52/0/49 Receive Queues
In-use 0 bytes, Max buffer 2048776 bytes
0 packets dropped
In-use Max
TC Bytes Bytes
-------------------------
0 0 291840
1 0 291840
2 0 291840
3 0 148868
4 0 291840
5 0 291840
6 0 291840
7 0 148868
VDX6740T(config-rbridge-id-12)#

Additional notes
CoS tail drop threshold can be configured on individual interfaces to increase the buffering capability in ingress to some extent using CLI :

qos rcv-queue cos-threshold TDT0 {TDT1 | TDT2 | TDT3 | TDT4 | TDT5 | TDT6 | TDT7}

This may marginally improve performance.

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255