Reset Search
 

 

Article

When configuring openflow flows with layer2 match criteria the error 'Flow table is full' is seen in MLXe logs.

« Go Back

Information

 
TitleWhen configuring openflow flows with layer2 match criteria the error 'Flow table is full' is seen in MLXe logs.
Symptoms
On NI 5.8 certain layer 2 flows are rejected with Type=3, Code=0, meaning Table Full when passed to the MLXe, but they would work successfully on NI 5.7.
Environment
Software Release: 5.8
Fixed in Version: N/A
Cause
In NetIron 5.8 and later code releases the layer23 port-types criteria that match on layer2 headers (but do not include ether-type) requires ipv4 & ipv6 CAM allocation. The configuration below only contains ipv4 CAM allocation for openflow entries on layer23 ports and that is why the error is generated.

MLXe openflow configuration:
!
ver V5.9.0aT163
module 1 br-mlx-20-port-10g
module 2 br-mlx-24-port-1gc-x
!
openflow enable ofv130
openflow controller ip-address 192.168.100.100 no-ssl port 6633
!
system-max openflow-flow-entries 10000
system-max openflow-unprotectedvlan-entries 20
system-max np-openflow-flow-entries layer2or3 200 slot 1
system-max np-openflow-flow-entries layer23ipv4 100 slot 1
!
interface ethernet 1/3
openflow enable layer23
enable
gig-default neg-off
!
interface ethernet 1/4
openflow enable layer23
enable
gig-default neg-off
!

MLXe openflow related outputs:
mlxe#show openflow resources
Openflow Resources:
CAM Profile: multi-service
Used - Number of HW entries consumed
Free - Number of Port based flows that can be successfully programmed

Slot: 1 Module: BR-MLX-10Gx20 20-port 1/10GbE Module
Openflow Layer2or3 Flows : MAX: 200 Used: 0 Free: 200
Openflow Layer23IPv4 Flows : MAX: 100 Used: 0 Free: 100

Slot: 2 Module: BR-MLX-1GCx24-X 24-port 10/100/1000Base-T Copper Module
Openflow np-system-max is not configured for this module

Openflow MPLS L2VPN Label Match Flows : MAX: 10000 Used: 0 Free: 10000
Openflow MPLS Tunnel/L3VPN Label Match Flows: MAX: 10000 Used: 0 Free: 10000
Openflow MPLS BOS0 Match Flows : MAX: 1 Used: 0 Free: 1
Openflow MPLS BOS1 Match Flows : MAX: 1 Used: 0 Free: 1


Openflow SELECT group resources available: 256
Openflow ALL/INDIRECT/FF group resources available: 512

Flow action nodes MAX = 2000000
Flow action nodes Used = 0
Flow action nodes Free = 2000000

mlxe#show openflow
Administrative Status: Enabled
SSL Status: Enabled
Source-Interface: Not Configured
Source-Interface Status: NA


Controller Type: ofv130
HELLO Reply: enabled
Number of Controllers: 2

Controller 1:
Connection Mode: active, TCP
Controller Address: 192.168.100.100
Connection Port: 6633
Source IP used: 192.168.100.101
Connection Status: OPENFLOW_ESTABLISHED
Role: MASTER
Packet-in config: no-match, action
Port-status config: add, delete, modify
Flow-removed config: delete, group-delete

Match Capability:
L2 : Port, Source MAC, Destination MAC, Ether type, Vlan, Vlan PCP
MPLS Label, MPLS BoS
L3 : Port, Vlan, Vlan PCP, Ethertype(IP,ARP,LLDP), Source IP, Destination IP, IP Protocol, IP TOS,
IP Src Port, IP Dst Port
Ethertype(IPV6), Source IPv6, Destination IPv6, ICMPv6 Type, ICMPv6 code, ICMPv4 Type, ICMPv4 code, MPLS Label, MPLS BoS
L23: All

Normal Openflow Enabled Ports: e1/3 e1/4
Openflow Hybrid Interfaces:

Default action: drop
Maximum number of flows allowed: 10000
Active flow: 0

Maximum number of Protected Vlans allowed: 0
Maximum number of Unprotected Vlans allowed: 20
Total number of Protected Vlans: 0
Total number of Active Protected Vlans: 0
Total number of Unprotected Vlans: 0
OpenFlow MPLS label range: 368928 - 499999
OpenFlow MPLS L3VPN or tunnel label range: 368928 - 434463
OpenFlow MPLS L2VPN label range: 434464 - 499999
OpenFlow MPLS L3VPN or tunnel remote label range: 368928 - 434463
OpenFlow MPLS L2VPN remote label range: 434464 - 499999
Total number of OpenFlow MPLS labels used: 0
Resolution
To resolve the issue for these types of flows 'system-max np-openflow-entries layer23ipv6' must be added to the configuration. The line-card must be reloaded to apply this change.

Open vSwitch Example:
$ ovs-ofctl add-flow tcp:10.18.24.53:6633 "in_port=4,dl_vlan=4070 action=output:9"

mlxe#sh runn | inc system
system-max ip-tunnels 512
system-max openflow-flow-entries 2000
system-max openflow-pvlan-entries 1000
system-max openflow-unprotectedvlan-entries 2000
system-max np-openflow-flow-entries layer2or3 700 slot 1
system-max np-openflow-flow-entries layer23ipv4 700 slot 1
mlxe#conf t
mlxe(config)#system-max np-openflow-flow-entries layer23ipv6 600 slot 1
Line Card Reload required. Please write memory and then reload or power cycle the Line Card.
mlxe(config)#

mlxe#power-off lp 1
Slot 1 is powered off.
mlxe#power-on lp 1
Slot 1 is powering on.
mlxe#

mlxe#sh run int e 1/4
interface ethernet 1/4
openflow enable layer23 hybrid-mode
enable
phy-mode wan
!

mlxe#sh run int e 1/9
interface ethernet 1/9
openflow enable layer23 hybrid-mode
enable
!

mlxe#show openflow interface

Total number of Openflow interfaces: 3

Port Link Speed Tag MAC OF-portid Name Mode
1/3 Down None Yes 0024.3880.9400 3 Hybrid-Layer23
1/4 Down None No 0024.3880.9403 4 Hybrid-Layer23
1/9 Down None No 0024.3880.9408 9 Hybrid-Layer23
mlxe#sh open flows
Total Number of data packets sent to controller: 0
Total Number of data bytes sent to controller : 0

Total Number of Flows: 1
Total Number of Port based Flows : 1

Total Number of L2 Generic Flows : 0
Total Number of L3 Generic Flows : 0
Total Number of L2+L3 Generic Flows : 0
Total Number of L23 Generic Flows : 0

Flow ID: 1 Priority: 32768 Status: Active
Rule:
In Port: e1/4
In Vlan: Tagged[4070]
Action: FORWARD
Out Port: e1/9
Statistics:
Total Pkts: 0
Total Bytes: 0

mlxe#
mlxe#sh runn | in openflow
openflow enable ofv100
openflow hello-reply disable
openflow controller passive no-ssl ip-address 192.168.100.1005 port 6633
system-max openflow-flow-entries 2000
system-max openflow-pvlan-entries 1000
system-max openflow-unprotectedvlan-entries 2000
system-max np-openflow-flow-entries layer2or3 700 slot 1
system-max np-openflow-flow-entries layer23ipv4 700 slot 1
system-max np-openflow-flow-entries layer23ipv6 600 slot 1
openflow enable layer23 hybrid-mode
openflow enable layer23 hybrid-mode
openflow enable layer23 hybrid-mode
mlxe#
Total Number of Hardware entries for flows: 2
Total Number of Hardware entries for Port flow: 2
Total Number of Hardware entries for Generic flow: 0

Total Number of Openflow interfaces: 3
Total Number of L2 interfaces: 0
Total Number of L3 interfaces: 0
Total Number of L23 interfaces: 3

Flow ID: 1 Priority: 32768 Status: Active
Rule:
In Port: e1/4
In Vlan: Tagged[4070]
Action: FORWARD
Out Port: e1/9
Statistics:
Total Pkts: 0
Total Bytes: 0

mlxe#

Opendaylight Example:
telnet@swn-mlxe-1#show run | i system
system-max openflow-flow-entries 10000
system-max openflow-unprotectedvlan-entries 20
system-max np-openflow-flow-entries layer2or3 200 slot 1
system-max np-openflow-flow-entries layer23ipv4 100 slot 1
system-max np-openflow-flow-entries layer23ipv6 50 slot 1
telnet@swn-mlxe-1#show openflow resources

Openflow Resources:

CAM Profile: ipv4-ipv6
Used - Number of HW entries consumed
Free - Number of Port based flows that can be successfully programmed

Slot: 1 Module: BR-MLX-10Gx20 20-port 1/10GbE Module
Openflow Layer2or3 Flows : MAX: 200 Used: 0 Free: 200
Openflow Layer23IPv4 Flows : MAX: 100 Used: 1 Free: 99
Openflow Layer23IPv6 Flows : MAX: 50 Used: 1 Free: 49 <-- Note two CAM used.

Slot: 2 Module: BR-MLX-1GCx24-X 24-port 10/100/1000Base-T Copper Module
Openflow np-system-max is not configured for this module

Openflow MPLS L2VPN Label Match Flows : MAX: 10000 Used: 0 Free: 10000
Openflow MPLS Tunnel/L3VPN Label Match Flows: MAX: 10000 Used: 0 Free: 10000
Openflow MPLS BOS0 Match Flows : MAX: 1 Used: 0 Free: 1
Openflow MPLS BOS1 Match Flows : MAX: 1 Used: 0 Free: 1

Openflow SELECT group resources available: 256
Openflow ALL/INDIRECT/FF group resources available: 512

Flow action nodes MAX = 2000000
Flow action nodes Used = 1
Flow action nodes Free = 1999999
telnet@swn-mlxe-1#show openflow flows
Total Number of data packets sent to controller: 0
Total Number of data bytes sent to controller : 0

Total Number of Flows: 1
Total Number of Port based Flows : 1

Total Number of L2 Generic Flows : 0
Total Number of L3 Generic Flows : 0
Total Number of L2+L3 Generic Flows : 0
Total Number of L23 Generic Flows : 0
Total Number of MPLS L2VPN Generic Flows : 0
Total Number of MPLS Tunnel/L3VPN Generic Flows : 0

Total Number of Hardware entries for flows: 2
Total Number of Hardware entries for Port flow: 2
Total Number of Hardware entries for Generic flow: 0

Total Number of Openflow interfaces: 2
Total Number of L2 interfaces: 0
Total Number of L3 interfaces: 0
Total Number of L23 interfaces: 2

Flow ID: 1 Priority: 101 Status: Active
Rule:
In Port: e1/3
Source Mac: 0011.43d5.0f92
Destination Mac: 0011.43d5.0f2d
Source Mac Mask: ffff.ffff.ffff
Destination Mac Mask: ffff.ffff.ffff
Idle Timeout : 0 secs
Hard Timeout : 0 secs

Instructions: Apply-Actions
Action: FORWARD
Out Port: e1/4
Statistics:
Total Pkts: 0
Total Bytes: 0

Timing Info:
Time Elapsed(Since Flow Added) : 13 secs
Time Elapsed(Since Last Packet Hit) : 13 secs
Additional notes
1. If the port-type is layer2 (not layer23) the flow will succeed. This is because there is different CAM allocation for layer2 vs. layer23 port-types.

Opendaylight Example:
mlxe#show openflow interface

Total number of Openflow interfaces: 2

Port Link Speed Tag MAC OF-portid Name Mode
1/3 Down None Yes cc4e.2495.3202 3 Layer23
1/4 Down None Yes cc4e.2495.3203 4 Layer23
mlxe#conf t
mlxe(config)# int e 1/3 to 1/4
mlxe(config-mif-1/3-1/4)#no openflow enable layer23
Error: Cannot disable openflow from port 1/4 as this is the last port corresponding to a generic flow
mlxe(config-mif-1/3-1/4)#int e 1/3
mlxe(config-if-e10000-1/3)#openflow enable layer2
mlxe(config-if-e10000-1/3)#int e 1/4
mlxe(config-if-e10000-1/4)#no openflow enable layer23
mlxe(config-if-e10000-1/4)#openflow enable layer2

mlxe(config-if-e10000-1/4)#wr mem
Write startup-config done.

SYSLOG: <14>Apr 14 11:04:38 mlxe Security: startup-config was changed by operator from console
mlxe(config-if-e10000-1/4)#end

SYSLOG: <14>Apr 14 11:04:40 mlxe Security: running-config was changed by operator from console
mlxe# show openflow int


Total number of Openflow interfaces: 2

Port Link Speed Tag MAC OF-portid Name Mode
1/3 Down None Yes cc4e.2495.3202 3 Layer2
1/4 Down None Yes cc4e.2495.3203 4 Layer2
mlxe#
mlxe#show openflow flows
Total Number of data packets sent to controller: 0
Total Number of data bytes sent to controller : 0


Total Number of Flows: 3
Total Number of Port based Flows : 1

Total Number of L2 Generic Flows : 0
Total Number of L3 Generic Flows : 0
Total Number of L2+L3 Generic Flows : 2
Total Number of L23 Generic Flows : 0
Total Number of MPLS L2VPN Generic Flows : 0
Total Number of MPLS Tunnel/L3VPN Generic Flows : 0

Total Number of Hardware entries for flows: 5
Total Number of Hardware entries for Port flow: 1
Total Number of Hardware entries for Generic flow: 4

Total Number of Openflow interfaces: 2
Total Number of L2 interfaces: 2
Total Number of L3 interfaces: 0
Total Number of L23 interfaces: 0

Flow ID: 1 Priority: 100 Status: Active
Rule:
In Port: generic
Ether type: 0x000088cc
Idle Timeout : 0 secs
Hard Timeout : 0 secs

Instructions: Apply-Actions
Action: FORWARD
Out Port: send to controller
Statistics:
Total Pkts: 0
Total Bytes: 0


Timing Info:
Time Elapsed(Since Flow Added) : 1106 secs
Time Elapsed(Since Last Packet Hit) : 1106 secs


Flow ID: 2 Priority: 1 Status: Active
Rule:
In Port: generic
Ether type: 0x00000806
Idle Timeout : 0 secs
Hard Timeout : 0 secs

Instructions: Apply-Actions
Action: FORWARD
Out Port: send to controller
Statistics:
Total Pkts: 0

Total Bytes: 0

Timing Info:
Time Elapsed(Since Flow Added) : 1106 secs
Time Elapsed(Since Last Packet Hit) : 1106 secs


Flow ID: 3 Priority: 101 Status: Active
Rule:
In Port: e1/3
Source Mac: 0011.43d5.0f92
Destination Mac: 0011.43d5.0f2d
Source Mac Mask: ffff.ffff.ffff
Destination Mac Mask: ffff.ffff.ffff
Idle Timeout : 0 secs
Hard Timeout : 0 secs

Instructions: Apply-Actions

Action: FORWARD
Out Port: e1/4
Statistics:
Total Pkts: 0
Total Bytes: 0

Timing Info:
Time Elapsed(Since Flow Added) : 36 secs
Time Elapsed(Since Last Packet Hit) : 36 secs

2. If a value for ether-type (but not IPv6 ether-type) is added to the flow the flow will succeed. (Only IPv4 CAM is used)

Opendaylight Flow:
{"flow":[{"id":"In3_Out4","priority":101,"table_id":0,"match":{"ethernet-match":{"ethernet-type":{"type":2048},"ethernet-destination":{"address":"00:11:43:d5:0f:2d"},"ethernet-source":{"address":"00:11:43:d5:0f:92"}},"in-port":"{{mlxe}}:3"},"instructions":{"instruction":[{"apply-actions":{"action":[{"order":0,"output-action":{"output-node-connector":"{{mlxe}}:4"}}]},"order":0}]}}]}

Logs:
telnet@swn-mlxe-1# show openflow flow flowid 5
Total Number of data packets sent to controller: 0
Total Number of data bytes sent to controller : 0

Total Number of Flows: 3
Total Number of Port based Flows : 1

Total Number of L2 Generic Flows : 0
Total Number of L3 Generic Flows : 0
Total Number of L2+L3 Generic Flows : 2
Total Number of L23 Generic Flows : 0
Total Number of MPLS L2VPN Generic Flows : 0
Total Number of MPLS Tunnel/L3VPN Generic Flows : 0

Total Number of Hardware entries for flows: 5
Total Number of Hardware entries for Port flow: 1
Total Number of Hardware entries for Generic flow: 4

Total Number of Openflow interfaces: 2
Total Number of L2 interfaces: 0
Total Number of L3 interfaces: 0
Total Number of L23 interfaces: 2

Flow ID: 5 Priority: 101 Status: Active
Rule:
In Port: e1/3
Source Mac: 0011.43d5.0f92
Destination Mac: 0011.43d5.0f2d
Source Mac Mask: ffff.ffff.ffff
Destination Mac Mask: ffff.ffff.ffff
Ether type: 0x00000800
Idle Timeout : 0 secs

Hard Timeout : 0 secs

Instructions: Apply-Actions
Action: FORWARD
Out Port: e1/4
Statistics:
Total Pkts: 0
Total Bytes: 0

Timing Info:
Time Elapsed(Since Flow Added) : 460 secs
Time Elapsed(Since Last Packet Hit) : 460 secs


telnet@swn-mlxe-1# show openflow interface

Total number of Openflow interfaces: 2

Port Link Speed Tag MAC OF-portid Name Mode
1/3 Down None Yes cc4e.2495.3202 3 Layer23
1/4 Down None Yes cc4e.2495.3203 4 Layer23

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255