Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

2 bladecenter FC SW and 2 External FC SW - Setup

Hi,

This is my first time setting up a solutiuon like this, as our normal customers only have small systems where a couple of switches are sufficient.

Here is what we have a IBM bladecenter with 2 brocade FC modules connecting back to 2 IBM branded Brocade 300 (24-B)

I have cabled them back in the following manner:

Bladecenter FCSW01 Port 0     --> FCSW01 Port 0

Bladecenter FCSW01 Port 15   --> FCSW02 Port 1

Bladecenter FCSW02 Port 0     --> FCSW01 Port 1

Bladecenter FCSW02 Port 15   --> FCSW02 Port 0

This will then give us full protection against any switch or blade port failure i think?

I have then set the two bladecenter switches into Access Gateway Mode, i hope this is correct?

When i go to look at the zone config on either of the two external switches (FCSW01/02) I dont see all the internal WWN's that the blades have.  For example.

We have 6 brand new blades each with 2 FC connections, once to each internal switch

FCSW01 can see 8 devices

FCSW02 can see 4 devices

And the two marry up exactly with all 12 WWN being visiable and in no particular order either.

So i suppose my questions are:

Should i be using gateway mode?

Have i missed something that would show the conencted devices on both switches?

do i need to connect the two external switches together?

Any advice would be most helpful.

Regards

Andy

Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Just to update this, since posting this I ahve read a couple of other threads, and noticed that the domain IDs should not be the same.

I ahve set the ID's to be different and also clicked the "automatic" port configuration on the two bladecenter switches (The switches when compared where different)

I now see on each Brocade (FCSW01/02) 6 devices each, one fo each blade on each switch.

Is this as it should be or should i expect to see both FC ports and WWNs?

Regular Contributor
Posts: 226
Registered: ‎01-08-2011

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Hi, isn't it a bit confusing that the switches have almost the same name? Is to me anyway

Bladecenter FCSW01 Port 0     --> FCSW01 Port 0

Bladecenter FCSW01 Port 15   --> FCSW02 Port 1

Bladecenter FCSW02 Port 0     --> FCSW01 Port 1

Bladecenter FCSW02 Port 15   --> FCSW02 Port 0

Both blade switches are connected to both external SAN-switches. This means you have one fabric.

The normal setup is to have two fabrics, as in: "Bladecenter FCSW01" connects to only FCSW01, for example.

If you run two fabrics you get fault isolation (one fabric is bad, you can turn it off) or if you want to do upgrades you can do one fabric at a time. If you also then run two cables from each Bladecenter switch to the external one in a trunk you can survive problem with one cable without any fabric initialization.

As for Access Gateway or not - usually AG is good for interopability in a heterogeneous environment or if you have a very large SAN (about to go over the Domain ID limit). I'd say it should be OK to run the Bladecenter Switch in native mode too, unless it's some kind of special version that only runs in AG mode.

If you run in AG you may not run normal E_Port ISL trunking so that requires a bit of extra work if you want to connect more than one cable between two switches.

As for your wwns, I'd say that you are seeing the WWnN (world-wide node names) and not the WWpN. Where are you looking?

Did you look in the output of CLI commands such as:

nscamshow

nsshow

switchshow

// Johan

Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Hi Johan,

Thanks for the reply.

The names I used above are not the actual names, it was just to try to explain the setup.

see below switchshow output:

BC FCSW01

Index Port Address Media Speed State Proto

==============================================

  0   0   010000   idN8   Online  FC  N-Port  10:00:00:05:33:c6:3c:d5 0x010000
  1   1   010100   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:48:64 0x010001
  2   2   010200   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:5c:5a 0x010003
  3   3   010300   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:3d:34 0x010002
  4   4   010400   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:46:8e 0x020403
  5   5   010500   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:3b:ba 0x020401
  6   6   010600   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:46:94 0x020402
  7   7   010700   cuAN   No_Sync FC 
  8   8   010800   cuAN   No_Sync FC 
  9   9   010900   cuAN   No_Sync FC 
10  10   010a00   cuAN   No_Sync FC 
11  11   010b00   cuAN   No_Sync FC 
12  12   010c00   cuAN   No_Sync FC 
13  13   010d00   cuAN   No_Sync FC 
14  14   010e00   cuAN   No_Sync FC 
15  15   010f00   idN8   Online  FC  N-Port  10:00:00:05:33:c6:52:f5 0x020400
16  16   011000   --N8   No_Module   FC 
17  17   011100   --N8   No_Module   FC 
18  18   011200   --N8   No_Module   FC 
19  19   011300   --N8   No_Module   FC

BC FCSW02

Index Port Address Media Speed State Proto

==============================================

  0   0   010000   idN8   Online  FC  N-Port  10:00:00:05:33:c6:52:f5 0x020000
  1   1   010100   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:48:65 0x020001
  2   2   010200   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:5c:5b 0x020003
  3   3   010300   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:3d:35 0x020002
  4   4   010400   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:46:8f 0x010401
  5   5   010500   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:3b:bb 0x010403
  6   6   010600   cuN8   Online  FC  F-Port  10:00:00:00:c9:cb:46:95 0x010402
  7   7   010700   cuAN   No_Sync FC 
  8   8   010800   cuAN   No_Sync FC 
  9   9   010900   cuAN   No_Sync FC 
10  10   010a00   cuAN   No_Sync FC 
11  11   010b00   cuAN   No_Sync FC 
12  12   010c00   cuAN   No_Sync FC 
13  13   010d00   cuAN   No_Sync FC 
14  14   010e00   cuAN   No_Sync FC 
15  15   010f00   idN8   Online  FC  N-Port  10:00:00:05:33:c6:3c:d5 0x010400
16  16   011000   --N8   No_Module   FC 
17  17   011100   --N8   No_Module   FC 
18  18   011200   --N8   No_Module   FC 
19  19   011300   --N8   No_Module   FC 

FCSW01

Index Port Address Media Speed State Proto

==============================================

   0   0   010000   id N8   Online  FC  F-Port  1 N Port + 3 NPIV public
   1   1   010100   id N8   Online  FC  F-Port  50:05:07:68:02:10:bd:84
   2   2   010200   id N8   No_LightFC 
   3   3   010300   -- N8   No_Module   FC 
   4   4   010400   id N8   Online  FC  F-Port  1 N Port + 3 NPIV public
   5   5   010500   id N8   Online  FC  F-Port  50:05:07:68:02:10:bd:85
   6   6   010600   id N8   No_LightFC 
   7   7   010700   id N8   No_LightFC 
   8   8   010800   id N8   No_LightFC 
   9   9   010900   id N8   No_LightFC 
  10  10   010a00   id N8   No_LightFC 
  11  11   010b00   id N8   No_LightFC 
  12  12   010c00   id N8   No_LightFC 
  13  13   010d00   id N8   No_LightFC 
  14  14   010e00   id N8   No_LightFC 
  15  15   010f00   id N8   No_LightFC 
  16  16   011000   id N8   No_LightFC 
  17  17   011100   id N8   No_LightFC 
  18  18   011200   id N8   No_LightFC 
  19  19   011300   id N8   No_LightFC 
  20  20   011400   id N8   No_LightFC 
  21  21   011500   id N8   No_LightFC 
  22  22   011600   id N8   No_LightFC 
  23  23   011700   id N8   No_LightFC 

FCSW02

Index Port Address Media Speed State Proto

==============================================

   0   0   020000   id N8   Online  FC  F-Port  1 N Port + 3 NPIV public
   1   1   020100   id N8   Online  FC  F-Port  50:05:07:68:02:20:bd:84
   2   2   020200   id N8   No_LightFC 
   3   3   020300   id N8   No_LightFC 
   4   4   020400   id N8   Online  FC  F-Port  1 N Port + 3 NPIV public
   5   5   020500   id N8   Online  FC  F-Port  50:05:07:68:02:20:bd:85
   6   6   020600   id N8   No_LightFC 
   7   7   020700   id N8   No_LightFC 
   8   8   020800   id N8   No_LightFC 
   9   9   020900   id N8   No_LightFC 
  10  10   020a00   id N8   No_LightFC 
  11  11   020b00   id N8   No_LightFC 
  12  12   020c00   id N8   No_LightFC 
  13  13   020d00   id N8   No_LightFC 
  14  14   020e00   id N8   No_LightFC 
  15  15   020f00   id N8   No_LightFC 
  16  16   021000   id N8   No_LightFC 
  17  17   021100   id N8   No_LightFC 
  18  18   021200   id N8   No_LightFC 
  19  19   021300   id N8   No_LightFC 
  20  20   021400   id N8   No_LightFC 
  21  21   021500   id N8   No_LightFC 
  22  22   021600   id N8   No_LightFC 
  23  23   021700   id N8   No_LightFC 

the reasonf or running them in AG, is because we will have some other bladecenters to conenect at a later date and these have QLogic FC modules, which will need to run in interop mode.

Andy

Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Also to add nscamshow returns "no entry found"

nsshow, reveals 10 entries, thise being

6 for the blades, 2 for the brocade BC switches and 2 for the SAN connected.

Regular Contributor
Posts: 226
Registered: ‎01-08-2011

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

OK, yes makes sense to keep going with AG if you're introducing QLogic devices too.

As far as I can see it looks good and all ports are accounted for in the switchshow outputs.

I like how on the switches in AG-mode you can see that the Fabric Address (the 0x10401 for example) comes from the DomainID and PortID of the switch in normal mode.

Oh yeah, nscamshow doesn't show anything because there's only one 'real' switch in each fabric.

Did I answer your questions?

Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Yes and no.

I am happy that I can now see 6 devices on each switch, but i would like it to show me 6 devices with both WWN's, were as i can only see 6 WWNs.

The reason i am thinking this is because i expect when i get my hosts up and running i should be able to see 4 paths to each LUN, but i assume i will only see 2, one from each switch?

I should be getting the hosts sorted by next week so hopefully i will be able to actually see how many path i can see. when i go into configure the zoning for instance, i can create an alias of host1 and add the specific WWN of 10:00:00:00:c9:cb:48:65 but shouldnt i also see :64 for it to have true multipathing of 4 paths?

Am i missing something or just not thinking about this right?

finally, do i need to connect the actual switches together (Not the bladecenters) for HA or just to get a full fabric.

Regular Contributor
Posts: 226
Registered: ‎01-08-2011

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Each FCSW sees 6 servers - one from each blade. 3 blades use one port to the AG-module and 3 uses the other port.

Then in each FCSW you have two ports to a disk array (supposedly one to each controller).

If both controllers allow access to the LUN then each server should have two paths per FCSW.

So 4 paths in total.

A switch in AG mode is not a "switch" anymore: it's not a member of the fabric, no name services, these are handled by the fabric, or in your case, the switch it's connected to.

And the switches are not connected. That's why you cannot see all the WWNs in both switches.

'fabricshow' should show only one switch.

If you connect the switches you ought to get 8 paths.

Occasional Contributor
Posts: 11
Registered: ‎05-30-2012

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

Thanks Johan,  last question i think.

The bladecenter switch 01 has 6 devices (blades) and passes traffic down port 0 and 15, 3 devices down each, is it possible to get it to have 6 on each if that makes sense?

As to connecting the main switches together, is there any point?

Regular Contributor
Posts: 226
Registered: ‎01-08-2011

Re: 2 bladecenter FC SW and 2 External FC SW - Setup

No problem Andrew,

you can customize the N_Port Mappings in many ways.

Command 'ag --mapshow' and 'ag --pgshow' gives an overview.

The "Access Gateway Administrator's Guide" has lots of information.

I went through the materials for the BCFP course and in the AG part it says that "user-defined port groups must be created to attach to more than one fabric". Looks like your module is also called a Brocade 5470.

I think you may run into problems by having 6 on each link with your setup where you have two fabrics connected to each AG module. The addressing may become a bit of a problem, then the same WWpN would have the two FCID?

If you have two links going to the same switch you could use a feature called N_Port Trunking - this one causes the address not to change in case one of the members of the trunk goes down. You need trunking license for this feature.

As for connecting switches together - some unix operating systems don't like when the path changes - so having more paths could be a problem. More paths doesn't always mean more bandwidth. But for your case, if you connect the switches I suppose this means both FC ports can still be used to connect in case one controller goes down in the san array. Personally I like to have two fabrics, makes upgrading feel a lot safer.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.