Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 13
Registered: ‎08-01-2008

Errors on long distance ISLs

Hello,

We have recently begun the process of replacing our old McData 6140 and DS-3232 switches with Brocade 5300s for our Tape SAN. The two switches in the environment are about 16km apart and pass through a DWDM.  There are a total of 10 x 2Gb ISLs between the two sites.

I have moved two of the 10 ISLs from the McData to the new Brocades (on each end, of course) using the same DWDM ports and cabling.  However, when I bring the Brocade ports online, I begin seeing many errors on the ISL switch ports.  Some ports show very few errors (a dozen per minute) while other ports show many errors (hundreds per second).  There are no errors on the DWDM units. All of the ISL ports are using longwave GBICs (1380nm). Both of the Brocades are running firmware 6.1.0h.

In this situation, the McData switch ports do not show any errors, but when I move those same cables to the Brocades, I see the errors.

Finally, I would just like to clarify that we are NOT connecting the Brocades and the McData switches into a single fabric.  I am disconnecting ISLs from the McDatas (on both ends) and then using those cables to connect the Brocades together.  I have already had the fibre cables at each end tested and some of them replaced and have had no further success.

Any help would be greatly appreciated!

External Moderator
Posts: 4,780
Registered: ‎02-23-2004

Re: Errors on long distance ISLs

Matt,

--->>>...with Brocade 5300s for ...

are in both site 5300 ?

--->>> I have moved two of the 10 ISLs from the McData to the new Brocades (on each end, of course) using the same DWDM ports and cabling.  However, when I bring the Brocade ports online, I begin seeing many errors on the ISL switch ports.

It see to me the error is caused from wrong set of BB-Credit beetwen beetwen the two site,  because.....

--->>>> In this situation, the McData switch ports do not show any errors, but when I move those same cables to the Brocades, I see the errors.

when you begin to move from 2GB switch to 8GB, a correct BB-Credit must be set.

Have you excluded this Issue ?

TechHelp24
Occasional Contributor
Posts: 13
Registered: ‎08-01-2008

Re: Errors on long distance ISLs

Thanks for the reply.

Yes, both sites have identical 5300s installed.

I never did actually move from 2Gb to 8Gb.  The DWDM is rather old and will not support speeds greater than 2Gb.  Therefore, I have hard-coded the ISLs on the Brocades to 2Gb.

As for the BB credit, I used the 'portcfglongdistance' command on all ISL ports and used the same parameters for each port (In this case, I used "LD" mode with a desired distance).

One thing of which I am unsure is the physical cabling.  The McData uses 9u longwave GBICs and the Brocades use 1310nm. As far as I know, both wavelengths are supported on the single-mode fibre we're using, but I was wondering if the tolerances are different for these wavelengths.  In other words, is it possible that a fibre with certain defects/sharp corners is within tolerance at one wavelength and not within tolerance at another?

External Moderator
Posts: 4,780
Registered: ‎02-23-2004

Re: Errors on long distance ISLs

--->>> I never did actually move from 2Gb to 8Gb.

I mean when you move from Old 2GB Switch to NEW Brocade.....

--->>>The McData uses 9u longwave GBICs....

i think your mean SFP ?

......and the Brocades use 1310nm.

9u longwave and 1310nm are the same

--->>> In other words, is it possible that a fibre with certain defects/sharp corners is within tolerance at one wavelength and not within tolerance at another?

sure, but i don't think this caused the problem, hovewer is probable both SFP are incompatible.

can you tell me please the exact SFP Part Number and Manufacrurung ?

About BB-Credit, see a table.

LongDistanceBB-Credit.bmp

TechHelp24
External Moderator
Posts: 4,780
Registered: ‎02-23-2004

Re: Errors on long distance ISLs

Matt,

BTW.

the table indicate BB value for Condor chipset, 5300 use GoldenEye2 Chipset.

as will check for the exact Value and coming back.

As mentioned, I'm 99% quite sure the error is caused by the BB-Credit.

TechHelp24
Occasional Contributor
Posts: 13
Registered: ‎08-01-2008

Re: Errors on long distance ISLs

The buffers were set automatically when I ran the portcfglongdistance command.  I admit to being more of a McData guy than Brocade, but I have not yet found a way to set the buffers arbitrarily on the Brocades as can be done on, say, the McData 6140.

Yes, I meant SFP.

The SFPs at both ends (i.e. on the Brocades) are identical and were received in the same shipment.  They are HP part numbers (these Brocades were re-sold from HP):

HP P/N:     AJ717A

Occasional Contributor
Posts: 13
Registered: ‎08-01-2008

Re: Errors on long distance ISLs

I appreciate your confidence that the issue it related to BB credits and I have not ruled that out.  However, I have done some testing and I am not able to confirm your belief that BB credits are to blame.

At the moment, I only have one ISL up and running since the other one gets so many errors that it causes the fabric to become unstable.  On the active ISL, the local switch receives very few errors (fewer than one error per minute).  On the remote switch, the same ISL receives about 600 errors per minute.

These error rates are almost exactly the same no matter what the BB setting is.  So, I use this command to set the reserved buffers to 39:

portcfglongdistance 0 ld 33

And that command is run on both switches (both switches are using port 0 for this ISL).  If I lower the buffers to 16 with this command (i.e. setting the distance to 10 km, which is as low as I can go in LD mode):

portcfglongdistance 0 ld 10

the error rates stay the same. 

Finally, if I increase the buffers to 106 using this command:

portcfglongdistance 0 ld 100

the error rates still stay the same.

If I am missing something, I would be happy to be corrected.  Thanks!

Frequent Contributor
Posts: 76
Registered: ‎02-13-2010

Re: Errors on long distance ISLs

Hi Matt,

Just read your post where you say that it is an old DWDM device. Which device exactly is it?

You can set up the specific number of buffers for a port with portcfgfportbuffers if you need a different value.

Cheers!

Occasional Contributor
Posts: 13
Registered: ‎08-01-2008

Re: Errors on long distance ISLs

The DWDM device is Nortel Optera Metro 5200.  Honestly, that's about all I can tell you about it.  All of the people who set it up originally are no longer with the company, though we do have one guy on the Telecom side who seems to know a bit about it.

It seems the command you suggested does not exist:

> portcfg

portcfg                   portcfgeport              portcfglongdistance       portcfgnport              portcfgshow

portcfgalpa               portcfgexport             portcfglosstov            portcfgpersistentdisable  portcfgspeed

portcfgcreditrecovery     portcfggport              portcfglport              portcfgpersistentenable   portcfgtrunkport

portcfgdefault            portcfgislmode            portcfgnpivport           portcfgqos                portcfgvexport

> help | grep buffers
portbuffershow             Print the buffer allocations for the ports of a


External Moderator
Posts: 4,780
Registered: ‎02-23-2004

Re: Errors on long distance ISLs

Matt,

I asked a Brocade SE about you problem, i think we found the problem. Step2

1)

Can you please create a simple Visio Sheet, Site A 5300 Switch - optera - Site B 5300 Switch, etc....

Question.

You say to have configured the port with the command "portcfglongdistance" , hovewer this command requiered a Extended Fabric License, so i assume both switch hare licensed. Is correct ? can you please confirm ?

"portcfgfportbuffers" as mentioned by neverdie is correct, but is supported in FOS 6.2 and above for F_Port, the command is not Support for

Long Distance Port.

2)

Can you please use "portcfglongdistance" "LS" mode with a desired distance ?

About the 5200 2 GB, this is continued supported by Brocade but not longer listed in the Compatibility Matrix as 2GB, can you please use in the Brocade 5300 other 1310 SFP ?

Example use here Finisar 2GB or 4GB or equivalent 1310 SFP ?



TechHelp24

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.