Understanding Cisco Nexus 2000 Series Fabric Extenders

Cisco Nexus 2000 Series Fabric Extenders (FEX) behave and can be considered as remote line cards for Cisco Nexus switches. The fabric extender acts as an extension to the parent Cisco Nexus switch fabric; the fabric extender and the parent Cisco Nexus switch together form a distributed modular system. With this architecture, the fabric extender typically is at top-of-rack (ToR) with the parent Cisco Nexus switch at end-of-row (EoR).

A simple topology with one Cisco Nexus 5548UP switch and one Cisco Nexus 2248PQ Fabric Extender may look like the below with the Fabric Extender at ToR and Nexus 5548UP at EoR.

Here are some absolute basics you need to know about Cisco Nexus 2000 Series Fabric Extenders:

• – No software is included with the Cisco Nexus Fabric Extender; the software is automatically downloaded and upgraded from its parent switch.
• – The parent switch pushes the configuration data to the Fabric Extender. The Fabric Extender does not store any configuration locally.
• – The Fabric Extender can connect to the switch through a number of separate physical Ethernet interfaces or at most one EtherChannel interface.
• – The Fabric Extender is managed by its parent switch over the fabric interfaces. Cisco advertises a zero-touch configuration. The Fabric Extender is discovered by the parent switch by detecting the fabric interfaces of the Fabric Extender.
• – The parent switch establishes in-band IP connectivity with the Fabric Extender. The parent switch assigns the Fabric Extender an IP address in the range of ‘127.0.0.0/8’, commonly known as loopback addresses, to avoid potential conflicts with IP addresses utilized in the network.
• – The Fabric Extender updates the parent switch with its operational status. All Fabric Extender information is displayed using the parent switch commands for monitoring and troubleshooting.
• – There are two types of interfaces: fabric interfaces (uplinks) and host interfaces (server-facing ports).
• – All Fabric Extender host interfaces are set to spanning tree edge ports with “BPDU Guard” enabled and cannot be configured as spanning tree network ports.
• – Servers utilizing active/standby teaming, 802.3ad port channels, or other host-based link redundancy mechanisms can be connected to Fabric Extender host interfaces.
• – Any device running spanning tree connected to a Fabric Extender host interface will result in that host interface being placed in an error-disabled state when a BPDU is received.
• – Any edge switch that leverages a link redundancy mechanism not dependent on spanning tree such as Cisco Flexlink or vPC (with BPDUFilter enabled) may be connected to a Fabric Extender host interface. Since spanning tree is not utilized to eliminate loops, extra precaution must be taken to ensure a loop-free topology below the Fabric Extender host interfaces.
• – The Fabric Extender does not perform any local switching. All traffic is sent to the parent switch where the switching and any policy enforcement is handled.

There are two options for how the uplinks of the Fabric Extender connect to the parent switch: 1. Static Pinning Fabric Interface Connection or 2. EtherChannel Fabric Interface Connection. The type of connection will determine how the traffic from an end host is distributed to the parent switch through the Cisco Nexus Fabric Extender.

The “Static Pinning Fabric Interface Connection” allows for a deterministic relationship between the host interfaces and the parent switch; you can configure the Fabric Extender to use individual fabric interface connections. The “EtherChannel Fabric Interface Connection” provides load balancing between the host interfaces and the parent switch; you can configure the Fabric Extender to use an EtherChannel fabric interface connection. This connection bundles 10-Gigabit Ethernet fabric interfaces into a single logical channel.

In the diagram further above, if using the first first four ports on the Nexus 5548UP with a EtherChannel Fabric Interface Connection to the Fabric Extender, the configuration would be as shown below.

Cisco Nexus 5548UP Config:

configure terminal
interface port-channel 2
switchport mode fex-fabric
fex associate 100
exit

interface ethernet 1/1
switchport mode fex-fabric
fex associate 100
channel-group 2
exit

interface ethernet 1/2
switchport mode fex-fabric
fex associate 100
channel-group 2
exit

interface ethernet 1/3
switchport mode fex-fabric
fex associate 100
channel-group 2
exit

interface ethernet 1/4
switchport mode fex-fabric
fex associate 100
channel-group 2
end
copy running-config startup-config

In the configuration above the “switchport mode fex-fabric” command sets the EtherChannel to support an external Fabric Extender. The “fex associate 100” command associates the chassis ID to the Fabric Extender unit attached to the interface; the range of the chassis ID is from 100 to 199. Once the configuration is complete, you can run the “show interface port-channel 2 fex-intf” command to display the association of a Fabric Extender to the EtherChannel interface.

Although the FEX architecture provides a single point for management and policy enforcement with zero-touch installation on the Fabric Extender, it also introduces the issue of all traffic being switched upstream by the parent switch as the fabric extender is not capable of doing local switching. If you have heavy ‘east-west’ traffic in your network, this may not be the best network architecture for you to deploy. Read my prior blog on ‘east-west’ traffic and network architecture here.