课件文稿成果cisco nexus 5500 adapterfex configuration example

Nexus5500Adapter−FEXConfigurationExample

ID:117691

ContributedbyCharlesYoussef,CiscoTACEngineer.May09,2014

Contents

IntroductionPrerequisites

RequirementsComponentsUsed

BackgroundInformation

Adapter−FEXOverview

Configure

EthernetvNICsConfigurationvHBAsConfiguration

VerifyTroubleshoot

VirtualEthernetInterfaceDoesNotComeUp

CollecttheAdapterTechnicalSupportInformationfromtheServerSide

Introduction

This describeshowtoconfigure,operate,andtroubleshoottheAdapter−FabricExtender(FEX)featureonNexus5500switches.

Prerequisites

Requirements

Therearenospecificrequirementsforthis .

ComponentsUsed

Theinformationinthis isbasedonthesesoftwareandhardwareversions:

Nexus5548UPwhichrunsVersion5.2(1)N1(4)

UnifiedComputingSystem(UCS)C−SeriesC210M2RackServerwithUCSP81EVirtualInterfaceCard(VIC)thatrunsFirmwareVersion1.4(2)

Theinformationinthis wascreatedfromthedevicesinaspecificlabenvironment.Allofthedevicesusedinthisstartedwithacleared(default)configuration.Ifyournetworkislive,makesurethatyouunderstandthepotentialimpactofanycommandorpacketcapturesetup.

BackgroundInformation

Adapter−FEXOverview

ThisfeatureallowsaNexus5500switchtomanagevirtualinterfaces(bothEthernetvirtualNetworkInterfaceControllers(vNICs)andFibreChannelVirtualHostBusAdapters(FCvHBAs))ontheserver'sVIC.Thisisindependentfromanyhypervisorthatrunsontheserver.WhatevervirtualinterfacesarecreatedwillbevisibletothemainOperatingSystem(OS)installedontheserver(providedthattheOShastheappropriatedrivers).

SupportedtformscanbefoundinthissectionoftheCiscoNexus5000SeriesNX−OSAdapterFEXOperationsGuide,Release5.1(3)N1(1).

SupportedtopologiesforAdapter−FEXcanbefoundinthissectionoftheCiscoNexus5000SeriesNX−OSAdapterFEXOperationsGuide,Release5.1(3)N1(1).

Thesupportedtopologiesare:

Serversingle−homedtoaNexus5500switch

Serversingle−homedtoaStraight−ThroughFEX

Serversingled−homedtoanActive/ActiveFEX

Serverdual−homedviaActive/StandbyuplinkstoapairofNexus5500switches

Serverdual−homedviaActive/StandbyuplinkstoapairofVirtualPortChannel(vPC)Active/ActiveFEXs

Thesubsequentconfigurationsectiondiscusses'Serverdual−homedviaActive/StandbyuplinkstoapairofNexus5500switches'whichisdepictedhere:

EachvNICwillhaveacorrespondingvirtualEthernetinterfaceontheNexus5000.SimilarlyeachvHBAwillhaveacorrespondingVirtualFibreChannel(VFC)interfaceontheNexus5000.

Configure

Note:UsetheCommandLookupTool(registeredcustomersonly)inordertoobtainmoreinformationonthecommandsusedinthissection.

EthernetvNICsConfiguration

CompletethesestepsonbothNexus5000switches:

NormallyvPCisdefinedandoperationalonthetwoNexus5000switches.VerifythatthevPCisdefined,peer−keepaliveisUP,andpeer−linkisUP.

Enterthesecommandsinordertoenablethevirtualizationfeatureset.

(config)#installfeature−setvirtualization

(config)#feature−setvirtualization

(Optional)AllowtheNexus5000toauto−createitsvirtualEthernetinterfaceswhenthecorrespondingvNICsaredefinedontheserver.NotethatthisdoesnotapplytotheVFCinterfaceswhichcanonlybemanuallydefinedontheNexus5000.

(config)#vethernetauto−create

ConfiguretheNexus5000interfacethatconnectstotheserversinVirtualNetworkTag(VNTag)mode.

(config)#interfaceEth1/10(config−if)#switchportmodevntag(config−if)#noshutdown

Configuretheportprofile(s)tobeappliedtothevNICs.

Theportprofilesareconfigurationtemtesthatcanbeapplied(inherited)bytheswitchinterfaces.InthecontextofAdapter−FEX,theportprofilescanbeeitherappliedtothevirtualEthernetinterfacesthataremanuallydefinedortotheonesthatareautomaticallycreatedwhenthevNICsareconfiguredontheUCSC−SeriesCiscoIntegratedManagementController(CIMC)GUIinterface.

Theport−profileisoftype'vethernet'.

Asampleport−profileconfigurationisshownhere:

(config)#port−profiletypevethernetVNIC1(config−port−prof)#switchportmodeaccess(config−port−prof)#switchportaccessvlan10(config−port−prof)#noshutdown(config−port−prof)#stateenabled

CompletethesestepsontheUCSC−Seriesserver:

ConnecttotheCIMCinterfaceviaHTTPandloginwiththeadministratorcredentials.

ChooseInventory>NetworkAdapters>ModifyAdapterProperties.

ChecktheEnableNIVModecheckbox.

ClickSaveChanges.

Poweroffandthenpowerontheserver.

Aftertheservercomesup,chooseInventory>NetworkAdapters>vNICs>AddinordertocreatevNICs.Themostimportantfieldstobedefinedare:

VICUplinkporttobeused(P81Ehas2uplinkportsreferencedas0and1).

ChannelNumber−auniquechannelIDofthevNIContheadapter.ThisisreferencedinthebindcommandunderthevirtualEthernetinterfaceontheNexus5000.ThescopeofthechannelnumberislimitedtotheVNTagphysicallink.Thechannelcanbethoughtofasa'virtuallink'onthephysicallinkbetweentheswitchandserveradapter.

PortProfile−thelistofportprofilesdefinedontheupstreamNexus5000canbeselected.AvirtualEthernetinterfacewillbeautomaticallycreatedontheNexus5000iftheNexus5000isconfiguredwiththevethernetauto−createcommand.NotethatonlythevirtualEthernetportprofilenames(port−profileconfigurationisnot)arepassedtotheserver.ThisoccursaftertheVNTaglinkconnectivityisestablishedandtheinitialhandshakeandnegotiationstepsareperformedbetweentheswitchandserveradapter.

ClickSaveChanges.

Poweroffandthenpowerontheserveragain.

vHBAsConfiguration

WhenyoucreatevHBAsontheserveradapter,thecorrespondingswitchinterfacesarenotautomaticallycreated.Instead,theyshouldbemanuallydefined.Thestepsfortheswitchandserversideareshownhere.

Completethesestepsontheswitchside:

CreateavirtualEthernettrunkinterfacethatisboundtotheVNTaginterface'schanneloftheservervHBAinterface.TheFibreChanneloverEthernet(FCoE)VLANshouldnotbethenativeVLAN.ThevirtualEthernetnumbersshouldbeuniqueacrossthetwoNexus5000switches.

Example:

(config)#interfaceveth10

(config−if)#switchportmodetrunk

(config−if)#switchporttrunkallowedvlan1,100(config−if)#bindinterfaceeth1/1channel3(config−if)#noshutdown

CreateaVFCinterfacethatisboundtothevirtualEthernetinterfacedefinedearlier.

Example:

(config)#interfacevfc10(config−if)#bindinterfaceveth10(config−if)#noshut

TheVirtualStorageAreaNetwork(VSAN)membershipforthisinterfaceisdefinedundertheVSANdatabase:

(config)#vsandatabase

(config−vsan−db)#vsan100interfacevfc10

(config−vsan−db)#vlan100(config−vlan)#fcoevsan100(config−vlan)#showvlanfcoe

Completethesestepsontheserverside:

ChooseInventory>NetworkAdapters>vHBAsinordertocreateavHBAinterface.

Themainfieldstobedefinedare:

PortWorldWideName(pWWN)/NodeWorldWideName(nWWN)

FCOEVLAN

UplinkID

Channelnumber

BootfromStorageAreaNetwork(SAN)ifused

Powercycletheserver.

Verify

Usethissectiontoconfirmthatyourconfigurationworksproperly.

TheOutputInterpreterTool(registeredcustomersonly)supportscertainshowcommands.UsetheOutputInterpreterToolinordertoviewanysisofshowcommandoutput.

ThelistofvirtualEthernetinterfacescanbedisyedwiththesecommands:

n5k1#show

interface

virtualsummary

Veth

Bound

Channel/ Port

Mac

VM

Interface

Interface

DV−Port Profile

Address

Name

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−Veth32770 Eth1/2 1 UPLINK

Total1VethInterfacesn5k1#

n5k1#showinterfacevirtualstatus

InterfaceVIF−index BoundIf Chan Vlan Status Mode Vntag

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

Veth32770VIF−17 Eth1/2 1 10 Up Active 2Total1VethInterfaces

TheautomaticallycreatedvirtualEthernetinterfacesdoappearintherunningconfigurationandwillbesavedtothestartupconfigurationwhencopyrunstartisperformed:

n5k1#showrunintve32770

!Command:showrunning−configinterfaceVethernet32770

!Time:ThuApr1012:56:232014

version5.2(1)N1(4)interfaceVethernet32770

inheritport−profileUPLINK

bindinterfaceEthernet1/2channel1n5k1#showintve32770brief

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

Vethernet VLAN TypeMode Status Reason Speed

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

Veth32770 10 virtaccessup none auton5k1#

Troubleshoot

Thissectionprovidesinformationyoucanusetotroubleshootyourconfiguration.

TheOutputInterpreterTool(registeredcustomersonly)supportscertainshowcommands.UsetheOutputInterpreterToolinordertoviewanysisofshowcommandoutput.

VirtualEthernetInterfaceDoesNotComeUp

VerifytheDenterBridgingCapabilitie changeProtocol(DCBX)informationfortheswitchVNTaginterfacewiththiscommand:

#showsysteminternaldcbxinfointerfaceethernet<>

Verifythat:

D enterBridgingExchange(DCX)protocolisConvergedEthernet(CEE)

CEENetworkIOVirtualization(NIV)extensionisenabled

NIVTypeLengthValue(TLV)ispresent

Ashighlightedbelow:

n5k1#showsysintdcbxinfointerfacee1/2

Interfaceinfoforif_index:0x1a001000(Eth1/2)tx_enabled:TRUE

rx_enabled:TRUEdcbx_enabled:TRUE

DCXProtocol:CEE <<<<<<<

DCXCEENIVextension:enabled <<<<<<<<<

<outputomitted>

FeaturetypeNIV(7) <<<<<<<featuretype7(DCXCEE−NIV)sub_type0

FeatureStateVariables:oper_version0error0localerror0oper_mode1

feature_seq_no0remote_feature_tlv_present1remote_tlv_aged_out0

remote_tlv_not_present_notification_sent0

FeatureRegisterParams:max_version0,enable1,willing0advertise1disruptive_error0mts_addr_node0x2201mts_addr_sap0x193

Otherservermts_addr_node0x2301,mts_addr_sap0x193

Desiredconfigcfglength:8databytes:9f ff 68 ef bd f7 4f c6Operatingconfigcfglength:8databytes:9f ff 68 ef bd f7 4f c6Peerconfigcfglength:8databytes:10 00 00 22 bd d6 66 f8

Commonproblemsinclude:

DCXprotocolisCIN

CheckforL1problems:cables,SFP,portbringup,adapter.

Checktheswitchconfiguration:featureset,switchportVNTag,enableLinkLayerDiscoveryProtocol(LLDP)/DCBX.

NIVTLVisabsent

CheckthatNIVmodeisenabledundertheadapterconfiguration.

CheckVNICInterfaceControl(VIC)communicationhasbeencompleteandtheportprofilesinformationhasbeenexchanged.EnsurethatthecurrentVirtualInterfaceManager(VIM)eventstateisVIM_NIV_PHY_FSM_ST_UP_OPENED_PP.

n5k1#showsysintvimevent−historyinterfacee1/2

>>>>FSM:<Ethernet1/2>has18loggedtransitions<<<<<

FSM:<Ethernet1/2>Transitionat327178usecsafterThuApr1012:22:272014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_PHY_DOWN]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

FSM:<Ethernet1/2>Transitionat327331usecsafterThuApr1012:22:272014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_DOWN_DONE]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

FSM:<Ethernet1/2>Transitionat255216usecsafterThuApr1012:26:152014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_RX_DCBX_CC_NUM]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_3SEC]

FSM:<Ethernet1/2>Transitionat250133usecsafterThuApr1012:26:182014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_3SEC]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_DCX_3SEC_EXP]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_ENCAP]

FSM:<Ethernet1/2>Transitionat262008usecsafterThuApr1012:26:182014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_ENCAP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_RECEIVED]Nextstate:[FSM_ST_NO_CHANGE]

FSM:<Ethernet1/2>Transitionat60944usecsafterThuApr1012:26:192014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_ENCAP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_ENCAP_RESP]Nextstate:[VIM_NIV_PHY_FSM_ST_UP]

FSM:<Ethernet1/2>Transitionat62553usecsafterThuApr1012:26:192014Previousstate:[VIM_NIV_PHY_FSM_ST_UP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_ACKD]Nextstate:[FSM_ST_NO_CHANGE]

FSM:<Ethernet1/2>Transitionat62605usecsafterThuApr1012:26:192014Previousstate:[VIM_NIV_PHY_FSM_ST_UP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_DONE]Nextstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED]

FSM:<Ethernet1/2>Transitionat62726usecsafterThuApr1012:26:192014Previousstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_PP_SEND]Nextstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED_PP]

FSM:<Ethernet1/2>Transitionat475253usecsafterThuApr1012:51:452014Previousstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED_PP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_PHY_DOWN]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_VETH_DN]

FSM:<Ethernet1/2>Transitionat475328usecsafterThuApr1012:51:452014Previousstate:[VIM_NIV_PHY_FSM_ST_WAIT_VETH_DN]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_DOWN_DONE]

Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

12)FSM:<Ethernet1/2>Transitionat983154usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_WAIT_DCBX]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_RX_DCBX_CC_NUM]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_3SEC]

10

12:53:06

2014

13)FSM:<Ethernet1/2>Transitionat992590usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_WAIT_3SEC]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_DCX_3SEC_EXP]Nextstate:[VIM_NIV_PHY_FSM_ST_WAIT_ENCAP]

10

12:53:09

2014

14)FSM:<Ethernet1/2>Transitionat802877usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_WAIT_ENCAP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_ENCAP_RESP]Nextstate:[VIM_NIV_PHY_FSM_ST_UP]

10

12:53:10

2014

15)FSM:<Ethernet1/2>Transitionat804263usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_UP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_ACKD]Nextstate:[FSM_ST_NO_CHANGE]

10

12:53:10

2014

16)FSM:<Ethernet1/2>Transitionat992390usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_UP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_RECEIVED]Nextstate:[FSM_ST_NO_CHANGE]

10

12:53:11

2014

17)FSM:<Ethernet1/2>Transitionat992450usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_UP]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_VIC_OPEN_DONE]Nextstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED]

10

12:53:11

2014

18)FSM:<Ethernet1/2>Transitionat992676usecsafterThuAprPreviousstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED]

Triggeredevent:[VIM_NIV_PHY_FSM_EV_PP_SEND]

Nextstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED_PP]

10

12:53:11

2014

Currstate:[VIM_NIV_PHY_FSM_ST_UP_OPENED_PP] <<<<<<<<<<

n5k1#

IfthevirtualEthernetinterfaceisafixedvirtualEthernet,checktoseeifVIC_CREATEappearsinthiscommand:

#showsysteminternalviminfonivmsglogsfixedinterfacee1/16ch1

Eth1/16(Chan:1)VIFIndex:605

REQMsgId:56630,Type:VICENABLE,CC:SUCCESSRSPMsgId:56630,Type:VICENABLE,CC:SUCCESSREQMsgId:4267,Type:VICSET,CC:SUCCESS

RSPMsgId:4267,Type:VICSET,CC:SUCCESS

REQMsgId:62725,Type:VICCREATE,CC:SUCCESS<<<<<<<RSPMsgId:62725,Type:VICCREATE,CC:SUCCESS<<<<<<<REQMsgId:62789,Type:VICENABLE,CC:SUCCESS

RSPMsgId:62789,Type:VIC