Cisco Nexus 7000 connectivity solutions for Cisco UCS
Last summer I was invited by the Nexus 7000 product management teamat Cisco to help co-author a whitepaper covering general guidelines andbest practices for network integration of Cisco UCS with Cisco Nexus7000.The>Cisco UCS Networking Best Practices (in HD), extract the material most> I am pleased to announce that as of today this whitepaper is now the official Cisco publication:Cisco Nexus 7000 Series Connectivity Solutions for the Cisco Unified Computing System
In summary, this whitepaper discusses the following topics:
[*]Nexus 7000 bandwidth and density complimenting Cisco UCS deployments
[*]Cisco UCS network connectivity overview
[*]Cisco UCS End Host Mode vs. Switch Mode
[*]Why End Host Mode is the preferred (and default) mode of operation
[*]Why vPC uplinks from Cisco UCS to Nexus 7000 are preferred
[*]Traffic patterns and failure scenarios with vPC uplinks to Nexus 7000
[*]Why attaching Cisco UCS without vPC to Nexus 7000′s configured for vPC should be avoided
[*]No vPC? No problem!Best practices when connecting Cisco UCS to Nexus 7000 without vPC
[*]Connecting Cisco UCS to separated Layer 2 networks
[*]Connecting Cisco UCS to networks with Nexus 5000 and Nexus 7000 using vPC
[*]Why connecting Cisco UCS to a Spanning Tree influenced Layer 2 access topology should be avoided.
[*]Summary of Cisco UCS + Nexus 7000 networking best practice recommendations
[*]Cisco Nexus 7000 architectural advantages for Cisco UCS connectivity
[*]Hitless ISSU, Stateful process restarts, Stateful supervisor switchover
[*]N+1 and Grid level power supply redundancy
[*]End of row L2/L3 connectivity for high density compute pods
[*]Scalability for large deployments, 128,000 MAC addresses – hardware learning
[*]Infrastructure consolidation with virtual device contexts (VDC)
[*]Support for next generation switching fabrics with FabricPath, and TRILL
[*]SAN/LAN infrastructure consolidation with future support for FCoE & FCF
What’s NOT covered in this whitepaper:
[*]Connecting Cisco UCS to Nexus 7000 FabricPath networks
[*]Guidance on choosing Nexus 7000 F1 or M1 series linecards for Cisco UCS connectivity
[*]FCoE uplinks from Cisco UCS to Nexus 7000
The above items not covered in this whitepaper may be the subject of future blogs here and/or additional Cisco whitepapers and CVD‘s.However, I will take this opportunity to write a few comments on each subject.
Connecting Cisco UCS to Nexus 7000 FabricPath networks
Nexus 7000 switches configured for FabricPath have a new switch portmode available called, you guessed it, a FabricPath port.These are theports that directly connect to other FabricPath capable switches andmust be explicitly configured as such.
interface Ethernet 1/1
description Connection to FabricPath network
switchport mode fabricpath
All other standard non-FabricPath ports are referred to as “ClassicEthernet” ports that normal switches and servers connect to without anyknowledge or awareness of FabricPath.This is the default port setting.
The Cisco UCS fabric interconnect is not a FabricPath aware switch,and as such should be connected to the Nexus 7000 on a normal “ClassicEthernet” port, in either End Host mode or Switch mode (end host mode isstill preferred).The Nexus 7000 may be participating in a largerFabricPath network upstream, but this fact is completely transparent toCisco UCS or any other device attached to a normal “Classic Ethernet”port.
interface Ethernet 2/1
description Connection to Cisco UCS
switchport mode trunk
spanning-tree port type edge trunk
The Nexus 7000 “Classic Ethernet” ports can still be configured forvPC, so the best practice recommendation of connecting Cisco UCS toNexus 7000 with vPC uplinks in End Host mode still applies, with orwithout FabricPath.
The Nexus 7000 configured for FabricPath has an enhancement to normal vPC, called vPC+ whichbasically makes the Nexus 7000 vPC domain appear as one Switch> In a nutshell, connecting Cisco UCS to a Nexus 7000 FabricPathnetwork has little impact in how you would normally connect Cisco UCS. Just make sure you’re connecting Cisco UCS to a normal “ClassicEthernet” port on the Nexus 7000.
More in this later…
Guidance on choosing Nexus 7000 F1 or M1 series linecards for Cisco UCS connectivity
First lets understand the some of the key differences in terms of price and capabilities…
The Nexus 7000 M1 seriesare the normal Layer 2 and Layer 3capable linecards available sincethe beginning with an 80 Gbps connection to the switch fabric and 4:1oversubscribed at the front panel 32 ports.Additionally, the M1 serieslinecard support hardware learning for 128,000 MAC addresses, androughly 1 million IP routes.The M1 linecard Layer 3 capabilities andMAC scalability provides flexibility that is both simple and scalable,but at twice the price of the F1 linecard for an equivalent 32-ports of10GE.If price is more important than density, an 8-port non-oversubscribed M1 linecard is available for almost half the price of the 32-port card.
The Nexus 7000 F1 series is a new 32-port 10GE linecardthat supports Layer 2 forwarding only with a 230 Gbps connection to theswitch fabric and line rate non-blocking forwarding (320 Gbps) for allLayer 2 flows local to the linecard.Additionally, the F1 linecardsupports FabricPath and is FCoE ready.Every two front panel ports areserviced by a switch on chip (SoC) that supports hardware learning for16,000 MAC addresses.If you simply spread all VLANs across all ports(all SoC), the entire linecard supports 16,000 MAC addresses.Withcareful planning, you can try to isolate VLANs to fewer ports, andtherefore expose the MAC addresses in those VLANs to fewer SoC.Theextreme case would be keeping any given VLAN unique to only one SoC,resulting in the F1 linecard supporting 256,000 unique MAC addresses (16SoC’s each with 16K unique MACs).
Side note: When the F1 linecard receivestraffic that needs Layer 3 switching, it will forward that trafficacross the internal fabric to an M1 linecard (if one exists) for theLayer 3 lookup and forwarding.
Which linecard is best for Cisco UCS connectivity?Each is a goodchoice with pros & cons, so it really depends on what’s moreimportant to you: cost, scalability, flexibility,bandwidth, over-subscription, etc.
You might choose the M1 linecard under these criteria:
[*]Scalability with simplicity, e.g. 128,000 MAC’s with no special planning.
[*]You are linking Cisco UCS to the Aggregation layer Nexus 7000 where Layer 3 switching is required.
[*]Consistency and simplicity of local forwarding for Layer 2 and Layer 3 flows.
[*]Line rate non-oversubscribed forwarding for all Layer 2 and Layer 3 flows (8-port M1)
[*]Low cost & low over-subscription more important than port density (8-port M1)
You might choose the F1 linecard under these criteria:
[*]You are linking Cisco UCS to an Access/Edge Nexus 7000 where only Layer 2 switching is required.
[*]You are linking Cisco UCS to a Nexus 7000 at the Edge of a FabricPath network.
[*]Low over-subscription, low latency, for all end-to-end Layer 2 flows is a concern.
[*]Both port density and cost are key concerns
[*]MAC scalability is not a concern
In my experience, most customers connect their Cisco UCS to theAggregation layer (this makes sense if you view the fabric interconnectas the Access layer).Of those customers, given the choice, most choosethe M1 linecard, except for those where cost, low latency, and lowover-subscription for Pod-to-Pod layer 2 forwarding is a key concern.
Some customers are beginning to deploy Nexus 7000 in both the Access(end of row) and Aggregation layers for density requirements and toprepare themselves for FabricPath.These customers are connecting theirCisco UCS fabric interconnects to the Nexus 7000 Access/Edge switchwhich is Layer 2 only by design, so the F1 linecard there is ano-brainer.
FCoE uplinks from Cisco UCS to Nexus 7000
There isn’t a lot of detail that can be discussed right now becausetwo things still need to happen.But I think I can give you a hint ofwhere this is heading.
[*]Nexus 7000 software (NX-OS) support for Fibre Channel forwarding (FCF)
[*]Cisco UCS Manager software support for FCoE uplinks
The key word in both items is software – Meaning, no new hardware that isn’t already available today will be required.
When these software capabilities arrive, we will begin to see topologieswhere Cisco UCS can link to a common pair of Nexus 7000′s that provideboth the LAN and SAN infrastructure.The holy grail of unified fabricconsolidation at both the access and aggregation layers starts to becomea real world reality.
More on that later too…
Disclaimer:The views and opinions expressed are those of theauthor, and not necessarily the views and opinions of the author’semployer.The author is not an official media spokesperson for CiscoSystems, Inc.For design guidance that best suites your needs, pleaseconsult your local Cisco representative.
Related Posts:
[*]Switchport Configurations Explained
[*]“Jawbreaker”, merchant silicon, QFabric, and flat networks
[*]Top of Rack vs End of Row Data Center Designs
[*]Nexus 5000 & Nexus 2000: New technology requires new thinking
[*]A few words about Nexus 7000
页:
[1]