Fibre Channel (SAN)

Reply
New Contributor
Posts: 4
Registered: ‎01-18-2010

Port 11 negotiated R_RDY mode (mode = 2)

I'm reaching out to the community here as my team and I are stumped after several months looking into this.  If anyone has any ideas or thoughts please let me know.

Here is the config

Site A - TX

2 brocade M5000 switches (code level 6.1.0c)

1 Cienna CN2000 Fabric extender (has one fibre cabled connected to each switch)

Link between the sites is a DS3

Site B - CO

2 Brocade Silkworm 750 fabric routers (not sure the code level)

1 cienna CN2000 Fabric Extender (has one fibre cabled connected to each switch)

ISSUE:

Each month we are getting the a Port Offline critical SNMP alert on the TX switches for the ports connected tot he Cienna devices.  This happens on both switches at the same time. When we log into the switch we see the ports ONLINE and no link alerts are logged.  We do however see this message in the logs - Port 11 negotiated R_RDY mode (mode = 2).  We have verified teh Circuit is up and running.  The cienna's are not reporting ANY alarms and all ports are set to Auto Negotiate.  The timing ont hsi is what is confusing us.  Below is a list of the timestamps of when this error occurs.

Sat Feb 13 2010 01:46:47 CST
Sat Feb 13 2010 01:43:41 CST
Sat Feb 13 2010 01:31:21 CST
Sat Feb 13 2010 01:15:03 CST
Fri Feb 12 2010 22:58:13 CST
Fri Feb 12 2010 21:41:42 CST
Fri Feb 12 2010 19:14:39 CST
Fri Feb 12 2010 18:10:12 CST
Fri Feb 12 2010 17:43:48 CST
Fri Feb 12 2010 17:02:05 CST
Fri Feb 12 2010 16:33:48 CST
Fri Feb 12 2010 14:10:52 CST
Fri Feb 12 2010 13:59:11 CST
Fri Feb 12 2010 13:06:26 CST
Fri Feb 12 2010 12:54:53 CST
Fri Feb 12 2010 12:32:17 CST
Fri Feb 12 2010 10:21:06 CST
Fri Feb 12 2010 10:00:42 CST
Fri Feb 12 2010 09:20:05 CST
Fri Feb 12 2010 09:02:51 CST
Fri Feb 12 2010 08:37:47 CST
Fri Feb 12 2010 08:10:48 CST
Fri Feb 12 2010 07:39:00 CST
Fri Feb 12 2010 07:12:42 CST
Fri Feb 12 2010 06:57:48 CST
Fri Feb 12 2010 06:44:00 CST
Fri Feb 12 2010 06:27:52 CST
Fri Feb 12 2010 06:05:52 CST
Fri Feb 12 2010 03:50:44 CST
Fri Feb 12 2010 02:35:39 CST
Fri Feb 12 2010 00:42:53 CST
Thu Feb 11 2010 23:54:52 CST
Thu Feb 11 2010 21:52:27 CST
Thu Feb 11 2010 21:12:33 CST
Thu Feb 11 2010 07:54:56 CST
Thu Feb 11 2010 07:34:58 CST
Thu Feb 11 2010 03:57:14 CST
Thu Feb 11 2010 03:11:39 CST
Thu Feb 11 2010 02:04:33 CST
Thu Feb 11 2010 00:14:13 CST
Wed Feb 10 2010 23:53:23 CST
Wed Feb 10 2010 15:09:58 CST
Wed Feb 10 2010 15:02:33 CST
Wed Feb 10 2010 11:23:12 CST
Wed Feb 10 2010 10:43:23 CST
Tue Feb 09 2010 23:40:56 CST
Tue Feb 09 2010 23:00:07 CST
Tue Feb 09 2010 20:20:15 CST
Tue Feb 09 2010 12:28:06 CST
Tue Feb 09 2010 12:20:37 CST
Wed Jan 13 2010 01:48:20 CST
Wed Jan 13 2010 01:41:29 CST
Tue Jan 12 2010 03:47:55 CST
Tue Jan 12 2010 03:40:52 CST
Mon Jan 11 2010 23:03:04 CST
Mon Jan 11 2010 22:57:15 CST
Fri Jan 08 2010 20:02:19 CST
Fri Jan 08 2010 19:09:11 CST
Fri Jan 08 2010 18:06:26 CST
Thu Jan 07 2010 13:32:08 CST
Mon Dec 14 2009 13:11:27 CST
Sat Dec 12 2009 00:40:50 CST
Wed Dec 09 2009 06:54:44 CST
Mon Nov 30 2009 09:41:55 CST
Sun Nov 22 2009 20:17:28 CST
Sun Nov 22 2009 19:44:53 CST
Sun Nov 22 2009 19:22:08 CST
Fri Nov 20 2009 14:13:13 CST
Tue Nov 17 2009 04:37:18 CST
Mon Nov 09 2009 03:29:31 CST
Sat Nov 07 2009 08:46:00 CST
Sat Nov 07 2009 08:07:57 CST
Fri Nov 06 2009 19:03:36 CST
Fri Nov 06 2009 04:33:55 CST
Wed Oct 28 2009 23:30:50 CDT
Wed Oct 28 2009 04:08:02 CDT
Wed Oct 28 2009 01:17:21 CDT
Thu Oct 08 2009 20:51:33 CDT

Super Contributor
Posts: 260
Registered: ‎04-09-2008

Re: Port 11 negotiated R_RDY mode (mode = 2)

Hi Shawn,


It seems like a problem with the ciena or the inter site link, since you say that the problem occurs on both switches. If the link extender has some form of management console, you could look at logs or reports for service degradation.

On the switch - you could look at these logs for answers. If you could post them in a text file the experts here could also have a look.

fabriclog -s

portshow <port number>

portstats64show <port number> {for ports connected to the ciena)

New Contributor
Posts: 4
Registered: ‎01-18-2010

Re: Port 11 negotiated R_RDY mode (mode = 2)

Well from the Ciena side we are not seeing any alerts of anykind.  Not even resync notifications. Which is why I came here looking for help from the switch side.  We have contacted Cienna support and they have not been able to find any events that are causing our issues.

I have attached the information requested, please take a look and let me know if you see anything.

Super Contributor
Posts: 644
Registered: ‎03-01-2007

Re: Port 11 negotiated R_RDY mode (mode = 2)

99,99% I'm quite sure the descripted error is caused by wrong set of BB Credit.

Super Contributor
Posts: 260
Registered: ‎04-09-2008

Re: Port 11 negotiated R_RDY mode (mode = 2)

Antonio is usually 100% correct

There are a few things that I observe

The portstats64show displays a lack of buffer credit. This is quite high number, maybe you can take another o/p and compare to see if they are rising.


tim64_txcrd_z   0           top_int : Time BB_credit zero
                105422454   bottom_int : Time BB_credit zero

I also see that the connection to your router is lost intermittently

21:42:04.974 SCN Domain 201 reachable                    D2,NA  D2,NA  NA    NA   
21:42:05.041 SCN Domain 201 available                    D2,NA  D2,NA  NA    NA   
21:42:05.042 *Snd inquiry (201)                          D2,NA  D2,NA  NA    NA   
21:42:05.042 Dom 201 name fcr_xd_201_101                 D2,NA  D2,NA  NA    NA   
21:42:05.042 essQueryVendUniqCap returns -1002           D2,NA  D2,NA  NA    NA   
21:42:05.990 RDI Rcv Timer Expired                       D2,NA  D2,NA  NA    NA   
21:42:05.991 *EFP Send:p=0x1,w=0x1e906ad2,l=64,v=3       D2,P3  D2,P3  11    0xe46
21:42:06.018 EFP ACC Rcv                                 D2,P3  D2,P3  11    0xe46
21:42:06.019 EFP pld:p=0x1,w=0x1e906ad2,l=64             D2,P3  D2,P3  11    0xe46
22:58:14.229 SCN Port Offline;g=0x2f8                    D2,P3  D2,P0  11    NA   
22:58:14.229 *Snd port state: P0                         D2,P0  D2,P0  11    NA   
22:58:14.229 *Removing all nodes from port               D2,P0  D2,P0  11    NA   
22:58:14.229 No next principal pt                        D2,NA  D2,NA  NA    NA   
22:58:14.229 Reconfig due to Offline(port 11)            D2,NA  D2,NA  NA    NA   
22:58:14.229 *Canceling All Node Timers                  D2,NA  D2,NA  NA    NA   
22:58:14.229 *BF Flood                                   D2,NA  D2,NA  NA    NA   
22:58:14.230 *BF Flood Done                              D2,NA  F0,NA  NA    NA   
22:58:14.230 *Snd SW state: F2                           F0,NA  F0,NA  NA    NA   
22:58:14.230 *Cancel 2 * F_S_TOV Timer                   F0,NA  F0,NA  NA    NA   
22:58:14.230 *Cancel EFP Flood Timer                     F0,NA  F0,NA  NA    NA   
22:58:14.230 *Cancel RDI Receive Timer                   F0,NA  F0,NA  NA    NA   
22:58:14.230 *Cancel RDI Send  Timer                     F0,NA  F0,NA  NA    NA   
22:58:14.230 *Start F_S_TOV Timer                        F0,NA  F0,NA  NA    NA   
22:58:14.230 *Start NTP Timer                            F0,NA  F0,NA  NA    NA   
22:58:14.230 *Cancel FAB_SIZE Timer                      F0,NA  F0,NA  NA    NA   
22:58:14.331 SCN Domain 103 unreachable                  F0,NA  F0,NA  NA    NA   
22:58:14.331 SCN Domain 201 unreachable                  F0,NA  F0,NA  NA    NA   

22:58:14.554 SCN LR_PORT (0);g=0x2f8                     F0,P0  F0,P0  11    NA   
22:58:14.680 SCN Port Online;g=0x2f8                     F0,P0  F0,P1  11    NA   
22:58:14.681 *ELP Send:rev=2,flow=2,flen=20,size=104     F0,P1  F0,P1  11    0x1a8
22:58:15.027 ELP Rcv:rev=2,flow=2,flen=20,size=104       F0,P1  F0,P1  11    0x8aef
22:58:15.031 *ELP Snd ACC:rev=2,flow=2,flen=20,size=164  F0,P1  F0,P2  11    0x8aef
22:58:15.031 ELP RJT Rcv, lgcl bsy, in prgs              F0,P2  F0,P2  11    0x1a8
22:58:15.189 SCN AC_PORT                                 F0,P2  F0,P3  11    NA   
22:58:15.190 *BF Send                                    F0,P3  F0,P3  11    0x1b5
22:58:15.513 BF ACC Rcv                                  F0,P3  F0,P3  11    0x1b5

To analyse if there are lack of credits - portbuffershow slot/port would help in my opinion.

New Contributor
Posts: 4
Registered: ‎01-18-2010

Re: Port 11 negotiated R_RDY mode (mode = 2)

Thanks everyone this is definatly helping.  I just ran another portstat64show on the port today and here is the BB_credit data

tim64_txcrd_z   1           top_int : Time BB_credit zero
                32733901    bottom_int : Time BB_credit zero

And I have attached a copy of the "portbuffershow"

Super Contributor
Posts: 260
Registered: ‎04-09-2008

Re: Port 11 negotiated R_RDY mode (mode = 2)

Since most of the ports on this switch lying empty, you could afford to allocate some more BB credits to port 11. Also please analyse the o/p on both ends.

portcfglongdistance is the command. For links upto 10 km you can set the LE option for allocation of additional buffers, no license reqd.

But before you do this you need to check if the ISL R_RDY option is enabled on both switches. portcfgshow

You also seemed to have cleared the error counters since the prev reading is higher than this one. If I can think of any other suggestion I will keep posting it here.

Super Contributor
Posts: 644
Registered: ‎03-01-2007

Re: Port 11 negotiated R_RDY mode (mode = 2)

Is Port 11 the connected Port beetwen Ciena and Brocade ?

What is the exact ( km = kilometer ) Distance, and the Speed set of FC Connection ?

Port 11 show in o/p you post here a buffer of 8 by 5 KM.

This is a wrong value.

New Contributor
Posts: 4
Registered: ‎01-18-2010

Re: Port 11 negotiated R_RDY mode (mode = 2)

Correct port 11 is connected to the Cienna which is located in the same cabinet.  This Cienna is then connected to a DS3 and another Cienna on the other side of the link.  I am not seeing any of these errors on the other side of the link.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook