Fibre Channel (SAN)

Reply
Contributor
Posts: 41
Registered: ‎05-24-2010

FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

I have a POC I am trying to setup and am not seeing the issue in what I have.  The connection is a bit "odd" I have a pair of Brocade 8Gb 24 port switches at each "site" (physically linked with 1Gb CAT 5 cables between two racks) with a blade chassis of servers which the customer purchased QLogic FC switches so I put them into NPV mode.  Zones within site work fine.  I have implemented best practices of seperate "A" and "B" fabrics (and for sake of example I call them by colors "red", "blue", "green", "yellow") 

Now I have a storage controller attached directly to the 24port Brocade SAN switches (two controllers with two connections to each switch). I want to link primary site "PSCFC01"  storage ports to remote site "RSCFC01" storage so I can do SAN replication. (and then repeat to join the "PSCFC02" storage to "RSCFC02").

I have the tunnels up and linked and can ping from both directions..

I created LSAN zones which contain the cross connected site WWPNs from each storage controller.  I don't think I have ever done single FCR at each site with expectation to set tunnel to join "GE0" -> Red / blue SANs    and  "GE1" -> yellow and green SANs.  I think their is some step I am missing to set link between the EX to VEX association?

I am likely missing something simple.  It has been a while sense I did FCIP...

FCIP_Diagram.png

Any help and comments would be appreciated.    

***************************

Draft Lab Guide

Fibre Channel Router

Port layout

Port

Switch

Domain ID

Mode

0

lcpscfc01-00

101

EX

1

lcpscfc02-00

102

EX

2

3

16

lcrscfc03-ge0*

121

VEX

17

lcrscfc03-ge1*

122

VEX

18

19

20

21

* Make sure the ports are in the copper (default) ports, if optical is desired mode must be changed.

Step 1: Activate License Keys for FCIP and Routing

Step 2: Login to fibre router and set up port

Select fc port and rename port description

Set mode on port to EX (SAN Routed Port) Set port to "EX-Port" with matching fabric ID to Native mode.

Now enable the "Gi" ports for VEX mode for ports 16, 17 which will bridge FCIP to remote switch.

Choose switch and "FCR"

Set the FCR ID by choosing "Set Fabric ID" (switch will have to be set to "disabled" to change this ID) then select "Enable FCR"  and set Backbone ID to 20

(For lcrscfc03 site, ID will be 20,21,22,23,24  while lcpscfc03 site IDs will be 10,11,12,13,14)

Choose "VE/Vex Ports" and choose the first Ethernet port and set to VEX and set Domain ID

Port Summary of descriptions (upstream connections) as well as Fabric Domain IDs

Step 2:   Setup Zones Aliase and routed Zone

Under Zone Admin

Create aliases for each WWPN. Example is for "lcrscv70001" system Canister 1, port 1

Now create zone for storage

** Repeat above step exactly but this time name zone "LSAN_******".  This creates a Zone that can be routed.

Now add both zones created to the zone database, save to switches

Then activate by clicking "Enable Config" and then select desired version and click ok

Step 3:   Setup FCR IP and routes on Gigibit interfaces

Example: lcrscfc01 with below WAN IP mapping

PSC Ports

RSC Ports

lcpscfc03w1

  1. 10.101.42.1

0/16 (ge0)

lcpscfc03w2

  1. 10.101.42.2

0/17 (ge1)

lcrscfc03w1

  1. 10.101.42.3

0/16 (ge0)

lcrscfc03w1

  1. 10.101.42.4

0/17 (ge1)

Display mapping of ports: portcfgpersistentenable

portcfgpersistentenable SlotO 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+--- Enabled YES YES YES YES YES YES YES YES YES YES YES YES YES YES YES YES Slot 0 16 17 18 19 20 21 22 23 +---+---+---+---+---+---+---+--- Enabled YES YES YES YES YES YES YES YES Slot O geO gel ge2 ge3 ge4 ges Enabled YES YES YES YES YES YES" height="192" width="594">

This sets IP to port "16" which is the first SFP based Ethernet Port.

lcrscfc03:admin> portshow fciptunnel all

No FCIP Tunnels found

lcrscfc03:admin> portshow 0/16

portIndex:  16

portName: lcpscfc03-16

portHealth: No Fabric Watch License

Authentication: None

EX_Port Mode:   Enabled

Fabric ID:      23

Front Phantom:  state = Not OK  Pref Dom ID: 160

Fabric params:  R_A_TOV: 0      E_D_TOV: 0      PID fmt: auto

Authentication Type: None

Hash Algorithm: N/A

DH Group: N/A

Edge fabric's primary wwn: N/A

Edge fabric's version stamp: N/A

portDisableReason: None

portCFlags: 0x0

portFlags: 0x4021        PRESENT VIRTUAL U_PORT EX_PORT DISABLED LED

LocalSwcFlags: 0x0

portType:  12.0

portState: 2    Offline

Protocol: FC

portPhys:  255 N/A     portScn:   2 Offline

port generation number:    0

state transition count:    2

portId:    cb1000

portIfId:    4302002a

portWwn:   20:10:00:27:f8:4a:7e:e4

portWwn of device(s) connected:

Distance:  normal

Port part of other ADs: No

lcpscfc03

lcrscfc03

# Disable port then bind IP and set Gateway interface

lcpscfc03:admin> portdisable ge0

lcpscfc03:admin> portcfg ipif ge0 create 10.101.42.1 255.255.255.0 1500

Operation Succeeded

lcpscfc03:admin> portcfg iproute ge0 create 10.101.42.0 255.255.255.0 10.101.42.230

Route is an interface subnet

# Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

lcpscfc03:admin> portcfg fciptunnel 16 create 10.101.42.3 10.101.42.1 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge0_lcrscfc03-ge0

# Re-enable interfaces

lcpscfc03:admin> portenable ge0

lcpscfc03:admin> portcfgpersistentenable ge0

lcpscfc03:admin> portenable 16

lcpscfc03:admin> portcfgpersistentenable 16

# Test via ping to remote site

lcrscfc03:admin> portcmd --ping ge0 -s 10.101.42.1 -d 10.101.42.3

# Display FCIP tunnel

lcpscfc03:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

16 -         Up      a------     1m5s    0.00 0.00    3      - -

-------------------------------------------------------------------------------

Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                 A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                 T=TPerf i=IPSec l=IPSec Legacy

# Disable port then bind IP and set Gateway interface

lcrscfc03:admin> portdisable ge0

lcrscfc03:admin> portcfg ipif ge0 create 10.101.42.3 255.255.255.0 1500

Operation Succeeded

lcrscfc03:admin> portcfg iproute ge0 create 10.101.42.0 255.255.255.0 10.101.42.230

Route is an interface subnet

# Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

lcrscfc03:admin> portcfg fciptunnel 16 create 10.101.42.1 10.101.42.3 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge0_lcrscfc03-ge0

# Re-enable interfaces

lcrscfc03:admin> portenable ge0

lcrscfc03:admin> portcfgpersistentenable ge0

# Test via ping to remote site

lcrscfc03:admin> portcmd --ping ge0 -s 10.101.42.3 -d 10.101.42.1

# Display FCIP tunnel

lcrscfc03:admin> portenable 16

lcrscfc03:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

16 -         InProg a------    1m59s 0.00    0.00    2 -      -

-------------------------------------------------------------------------------

Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                 A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                 T=TPerf i=IPSec l=IPSec Legacy

Research Frame support for site to support jumbo frames across WAN: http://wikibon.org/wiki/v/How_to_configure_FCIP_on_Brocade_7500%27s

LC-Router >>portcfg ipif ge0 create 10.255.2.3 255.255.255.0 2250

2250 is the MTU size (2148, is FC Frame size)

Repeat of above but this time to bind ge1 and ge1

lcpscfc03

lcrscfc03

# Disable port then bind IP and set Gateway interface

lcpscfc03:admin> portdisable ge1

lcpscfc03:admin> portcfg ipif ge1 create 10.101.42.2 255.255.255.0 1500

Operation Succeeded

lcpscfc03:admin> portcfg iproute ge1 create 10.101.42.0 255.255.255.0 10.101.42.230

Route is an interface subnet

# Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

lcpscfc03:admin> portcfg fciptunnel 17 create 10.101.42.4 10.101.42.2 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge1_lcrscfc03-ge1

# Re-enable interfaces

lcpscfc03:admin> portenable ge0

lcpscfc03:admin> portcfgpersistentenable ge0

lcpscfc03:admin> portenable 16

lcpscfc03:admin> portcfgpersistentenable 16

# Test via ping to remote site

lcrscfc03:admin> portcmd --ping ge1 -s 10.101.42.2 -d 10.101.42.4

# Display FCIP tunnel

lcpscfc03:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

16 -         Up      a------       3s 0.00    0.00    5 -      -

17 -         Up      a------    1m12s    0.00 0.00    1      - -

-------------------------------------------------------------------------------

Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                 A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                 T=TPerf i=IPSec l=IPSec Legacy

# Disable port then bind IP and set Gateway interface

lcrscfc03:admin> portdisable ge1

lcrscfc03:admin> portcfg ipif ge1 create 10.101.42.4 255.255.255.0 1500

Operation Succeeded

lcrscfc03:admin> portcfg iproute ge1 create 10.101.42.0 255.255.255.0 10.101.42.230

Route is an interface subnet

# Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

lcrscfc03:admin> portcfg fciptunnel 17 create 10.101.42.2 10.101.42.4 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge1_lcrscfc03-ge1

# Re-enable interfaces

lcrscfc03:admin> portenable ge1

lcrscfc03:admin> portcfgpersistentenable ge1

# Test via ping to remote site

lcrscfc03:admin> portcmd --ping ge1 -s 10.101.42.4 -d 10.101.42.2

# Display FCIP tunnel

lcrscfc03:admin> portenable 16

lcrscfc03:admin> portshow fciptunnel all

lcrscfc03:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

16 -         Up      a------       9s 0.00    0.00    5 -      -

17 -         Up      a------    1m18s    0.00 0.00    1      - -

-------------------------------------------------------------------------------

Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                 A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                 T=TPerf i=IPSec l=IPSec Legacy

Step 4: Now we create a zone for v7000s across sites

The router setup is to route between the WWP connections of the connected sites per table below.

Zones Routed

lcpscfc01_lcrscfc01

lcpscfc02_lcrscfc02

  (50:05:07:68:02:35:b7:83) IBM Corp

  (50:05:07:68:02:45:b7:83) IBM Corp

  (50:05:07:68:02:15:b7:84) IBM Corp

  (50:05:07:68:02:25:b7:84) IBM Corp

  (50:05:07:68:02:35:b7:ab) IBM Corp

  (50:05:07:68:02:45:b7:ab) IBM Corp

  (50:05:07:68:02:15:b7:ac) IBM Corp

  (50:05:07:68:02:25:b7:ac) IBM Corp

  (50:05:07:68:02:15:b7:83) IBM Corp

  (50:05:07:68:02:25:b7:83) IBM Corp

  (50:05:07:68:02:35:b7:84) IBM Corp

  (50:05:07:68:02:45:b7:84) IBM Corp

  (50:05:07:68:02:15:b7:ab) IBM Corp

  (50:05:07:68:02:25:b7:ab) IBM Corp

  (50:05:07:68:02:35:b7:ac) IBM Corp

  (50:05:07:68:02:45:b7:ac) IBM Corp

The above table will be created on each switch as a LSAN (routed) zone.  Below are the commands for each of the four switch sides.

lcpscfc01

lcpscfc02

lcrscfc01

lcrscfc02

Step 5: Now with the use of the Brocade Network Assistant GUI, we can quickly add switches and setup routed zones between the fabrics

Install 70day evaluation version of Brocade Network Advisor

na1202_brocade_windows

Username

Password

Administrator

password

Launch GUI login

Click to add new device

Input each of the six FC switches

Note that the three switches of each site are diagramed as being in a single connection

Now we can create a routed zone between the two sites for the v7000 controllers

Create LSAN zone

2)Typeinzone 4: 141 Ibi ÇIIP_New Zone New Member... OK Cancel Help Potential Members All Levels El lcrsctcO3 S 4 lcrsctcOl El lcrsctcO2 4 Type VMsN Name Product Type Class lcrsctcO3 lcrsctcOl lcrsctco2 site vl000s ¿3 Activate... :h la" height="374" border="0" width="576">

CLI LSAN zoning

lcpscfc03_lcpscfc01

lcpscfc03_lcpscfc2

lcrscfc03_lcrscfc01

lcrscfc03_lcrscfc2

External Moderator
Posts: 4,907
Registered: ‎02-23-2004

Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

have you created ONE or TWO LSAN config ?

I'm not sure, but i think such config you want/have created can not work with more than one LSAN

TechHelp24
Contributor
Posts: 41
Registered: ‎05-24-2010

Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

Yes there is / will be two VSAN segments.

The LSAN zone for the storage will be the interfaces in an LSAN that have "pscfc01" and "rscfc01" controller ports.  And a seperate LSAN containing controller ports of interfaces in "pscfc02" and "rscfc02".

Example attached of Aliase / WWPN from switch where I manually added the cross WWPN IDs to the LSAN_ zone

pscfc01rscfc01

zone:  lcpscv70001

                lcpscv70001_1_3; lcpscv70001_1_4; lcpscv70001_2_1;

                lcpscv70001_2_2

zone:  LSAN_lcpscv70001_lcrscv70001

                50:05:07:68:02:35:b7:83

                50:05:07:68:02:45:b7:83

                50:05:07:68:02:15:b7:84

                50:05:07:68:02:25:b7:84

                50:05:07:68:02:35:b7:ab

                50:05:07:68:02:45:b7:ab

                50:05:07:68:02:15:b7:ac

                50:05:07:68:02:25:b7:ac

zone:  lcrscv70001

                lcrscv70001_1_3; lcrscv70001_1_4; lcrscv70001_2_1;

                lcrscv70001_2_2

zone:  LSAN_lcrscv70001

                50:05:07:68:02:35:b7:83

                50:05:07:68:02:45:b7:83

                50:05:07:68:02:15:b7:84

                50:05:07:68:02:25:b7:84

                50:05:07:68:02:35:b7:ab

                50:05:07:68:02:45:b7:ab

                50:05:07:68:02:15:b7:ac

                50:05:07:68:02:25:b7:ac

alias: lcpscv70001_1_3

                50:05:07:68:02:35:b7:83

alias: lcpscv70001_1_4

                50:05:07:68:02:45:b7:83

alias: lcpscv70001_2_1

                50:05:07:68:02:15:b7:84

alias: lcpscv70001_2_2

                50:05:07:68:02:25:b7:84

alias: lcrscv70001_1_3

                50:05:07:68:02:35:b7:ab

alias: lcrscv70001_1_4

                50:05:07:68:02:45:b7:ab

alias: lcrscv70001_2_1

                50:05:07:68:02:15:b7:ac

alias: lcrscv70001_2_2

                50:05:07:68:02:25:b7:ac

alias: lcpscv70001_1_3

                50:05:07:68:02:35:b7:83

alias: lcpscv70001_1_4

                50:05:07:68:02:45:b7:83

alias: lcpscv70001_2_1

                50:05:07:68:02:15:b7:84

alias: lcpscv70001_2_2

                50:05:07:68:02:25:b7:84

alias: lcrscv70001_1_3

                50:05:07:68:02:35:b7:ab

alias: lcrscv70001_1_4

                50:05:07:68:02:45:b7:ab

alias: lcrscv70001_2_1

                50:05:07:68:02:15:b7:ac

alias: lcrscv70001_2_2

                50:05:07:68:02:25:b7:ac

What I don't follow is if their is any other step, besides creating the LSAN,  to join the two zones over FCR.

External Moderator
Posts: 4,907
Registered: ‎02-23-2004

Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

I don't think you can create a Routing with/between two LSAN, or better to say any such config is unknown to me.

TechHelp24
Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

Hi there,

I have taken a look at the diagram and it is conceptually wrong:

You cannot connect two VEX port together, a VEX port only can be connected to a VE port (as well as a EX port can only be connected to a E port). The only way to perform the config you want to implement is to configure a BB fabric made up of the two fcip tunnels.

Rgds

Contributor
Posts: 41
Registered: ‎05-24-2010

Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

Thanks,

I figured that out via some RTFM and poking around and got the environment working.

What I miss configured is that the brocade R06 is "routing"  what it is actually doing is "Bridging"

NPVID -> E -> EX ->VE ---VE <-EX <-E <- NPVID

That works.  Because what I needed to do is merge the FC03 switches (R06 router switch fabrics) together such that they were "one switch"  from which I could then "Bridge" the "red to blue" and "green to yellow".

Here is updated diagram

FCIP_Diagram.png

That works as expected.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.