Fibre Channel (SAN)

Reply
New Contributor
Posts: 4
Registered: ‎10-09-2017

I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

I am getting an error message with the code of C2-1012.  I looked on the switch and there are no connections using the ICL ports on the CR16-4 card, does this card have an additional use as in controlling the other port cards on the switch?  How do I fix this issue 7.0.0b is the running code.

Brocade Moderator
Posts: 55
Registered: ‎03-29-2010

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Better if you can copy/paste the entire message. You have had a link timeout on an internal port. When we know which slot was affected, we will know better how to assist you. also, run the 'statsclear' and 'slotstatsclear' commands and then wait 12 hours and capture the output of porterrshow, and provide it here.

 

The core blades(CR16) process all slot to slot data traffic in addition to supporting the ICL connections if equipped, so they touch any and every exchange which does not enter and leave on the same slot and ASIC.

 

Gaather the info and report back with the output, and we'll go from there.

doc

Any and all information provided by me is for entertainment value and should not be relied upon as a guaranteed solution or warranty of mechantability. All systems and all networks are different and unique. If you have a concern about data loss, or network disconnection, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, Please mark it with the button at the bottom "Accept as solution".

New Contributor
Posts: 4
Registered: ‎10-09-2017

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Mon Oct 09 2017 02:25:18 GMT+00.00  Warning  S6,P-1(48): Link Timeout on internal port ftx=767897 tov=2000 (>1000) vc_no=26 crd(s)lost:1  Chassis  16156  1  C2-1012 DOE_DL_DCX_4S_2

 

The "number 16156 increments...)  the Count is always "1"

 

Here is the current porterrshow before running commands:

Brocade Moderator
Posts: 55
Registered: ‎03-29-2010

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Run the statsclear and slotstatsclear and wait 24 hours, then run porterrshow again. It's hard to say, but looks like you have some serious congestion and latency going on, but without current and accurate stats, I can't give you any specifics. Also would be useful to see the output of the switchshow command, to see what's logged in to this switch.

doc

Any and all information provided by me is for entertainment value and should not be relied upon as a guaranteed solution or warranty of mechantability. All systems and all networks are different and unique. If you have a concern about data loss, or network disconnection, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, Please mark it with the button at the bottom "Accept as solution".

Brocade Moderator
Posts: 383
Registered: ‎03-29-2011

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Hi,

 

in the excel sheet the following ports are logging physical errors, e.g. cable or SFPs:

 

133 & 221 with a large amout of CRC (good eof)

212,204,165 with a lower count.

 

For the error below

 

Mon Oct 09 2017 02:25:18 GMT+00.00  Warning  S6,P-1(48): Link Timeout on internal port ftx=767897 tov=2000 (>1000) vc_no=26 crd(s)lost:1  Chassis  16156  1  C2-1012 DOE_DL_DCX_4S_2

 

your switch have lost credit (buffers) between the core blade and a port blade, internally to the switch. 

Can you please check by below command if Backend (BE) error recovery is enabled or not:

 

switch:admin> bottleneckmon --showcredittools

 

Best practices is to have BE enabled. To enable backend port credit recovery with the link reset only option and to display the configuration:

 

switch:admin> bottleneckmon --cfgcredittools -intport -recover onLrOnly

 

Best practices is to have BE enabled. To enable backend port credit recovery with the link reset only option and to display the configuration: More details can be found at

 

https://www.brocade.com/content/html/en/administration-guide/fos-740-admin/GUID-EA02CA72-1A5D-4CAE-A220-A04906E39C8B.html

 

To determine what else is going around the C2-1012 message, can you provide the output from errorlog around the C2-1012, please? 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
New Contributor
Posts: 4
Registered: ‎10-09-2017

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Brocade Moderator
Posts: 383
Registered: ‎03-29-2011

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Hi,

 

The port 152 is logging C3TX, which indicate either a slow drain device (or possible lost credits) on this link (at some time). I am not sure when the stats were cleared the last time. BTW, you are on FOS7.0.0b which is really really low - recommending to upgrade to Target Path (7.4.1d or 7.4.1e or 7.4.2a) .

 

I see repeated link reset on the internal ports.  Please run tbe below command which will enable automatic Backend port credit recovery:

 

switch:admin> bottleneckmon --cfgcredittools -intport -recover onLrOnly

 

Then, check the error dump for further messages. We might have to do a link reset (forced) on the backplane if nothing triggers. And also can you please share the output from the following root command (sloterrshow -c2), please?

 

 

 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Brocade Moderator
Posts: 55
Registered: ‎03-29-2010

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

[ Edited ]

Port index 152 is timing out when attempting to deliver frames, and discarding Class 3 data frames.

 

 

  ......   frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout
   .....    tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig             ....    tx    rx
   =======================================================================================

152:  155.5m 178.2m   0      0      0      0      0      0      0    767      0      0      0      0      0    519      0 

 

You have to find out what is connected to index 152(without the switchshow, I can't tell you where it goes). and find out why it is not returning buffers so that the frames can be delivered.

 

Note that there are timeouts on the RX side of the following ports, with C3 discards coming into the ports:

 

 

156:   81.6m 256.9m   0      0      0      0      0      0      0      6      0      0      0      0      0      0      4   
157:   17.4m  30.3m   0      0      0      0      0      0      0      1      0      0      0      0      0      0      1   
158:   44.3m  23.6m   0      0      0      0      0      0      0      2      0      0      0      0      0      0      2   
159:   25.1k  32.7k   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0   
160:  136.7m 757.9m   0      0      0      0      0      0      0     14      0      0      0      0      0      0      8   
161:  842.0k 726.3k   0      0      0      0      0      0      0     15      0      0      0      0      0      0     12   
162:   44.3m  23.6m   0      0      0      0      0      0      0      5      0      0      0      0      0      0      4   
163:   66.4m  46.9m   0      0      0      0      0      0      0      2      0      0      0      0      0      0      2   
164:  140.1m 760.1m   0      0      0      0      0      0      0     22      0      0      0      0      0      0     15  

 

With the limited information about your fabric, it's hard to decide where to focus. However, you will need to investigate where these ports are loosing buffers. There is often, but not always a relationship between the ports which are experiencing RX timeouts, and the port which is experiencing TX timeouts. This is a classic example of congestion. Which has been causing some latency due to a slow draining device. If you have slow drain devices connected to this switch, it is best to upgrade, or segregate your legacy devices from the higher speed devices.

 

One of the ways to help you is to begin implementing MAPS and check for the FPI output showing Fabric Performance Impact. This will help guide you. There are good examples in the MAPS admin guide, found in the Documents library on my.brocade.com

 

EDIT >>> Martin was posting while I was composing, Turning on bottleneckmon will help recover VCs on the backend ports when you experience credit loss internally. Do that first, and still investigate the source of the problem. Tyically it is outside the fabric, and related to a low speed device cloging up the core switch.

doc

Any and all information provided by me is for entertainment value and should not be relied upon as a guaranteed solution or warranty of mechantability. All systems and all networks are different and unique. If you have a concern about data loss, or network disconnection, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, Please mark it with the button at the bottom "Accept as solution".

Highlighted
New Contributor
Posts: 4
Registered: ‎10-09-2017

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

here is the supportshow

Brocade Moderator
Posts: 383
Registered: ‎03-29-2011

Re: I have a DCX-4s and am getting a warning message C2-1012 and have NO ICL...what is going on?

Hi,

 

On port 152, there is NPIV device, storage

 

 N    029800;      3;50:00:d3:10:00:40:d0:1c;50:00:d3:10:00:40:d0:00; 0x00000003
    FC4s: FCP
    PortSymb: [110] "Compellent Port QLGC FC 8Gbps; Slot=05 Port=04 in Controller: SN 16592 of Storage Center: Storage Center 16592"
    NodeSymb: [47] "Compellent Storage Center: Storage Center 16592"
    Fabric Port Name: 20:98:00:05:33:c1:6e:00
    Permanent Port Name: 50:00:d3:10:00:40:d0:1c
    Port Index: 152
    Share Area: No
    Device Shared in Other AD: No
    Redirect: No
    Partial: No

 

to which the switch as sent at least 7 Link Reset

 

Suspended:         0          Lr_out:       7

 

and where we have logged bad os

 

Suspended:         0          Lr_out:       7

 

i would check with storage vendor if they support portcfgfillword 1,2, or 3 (instead of 0) since this a 8GB switch.

That might help reduce errors on this port. sfpshow looks good, though (so first fill word).

 

RX Power:    -2.9    dBm (507.7uW) 10.0   uW 1258.9 uW   15.8   uW  1000.0 uW
TX Power:    -3.2    dBm (476.1 uW)125.9  uW   631.0  uW  158.5  uW   562.3  uW

 

Further, for the

 

2017/10/11-15:08:56, [C2-1012], 16472, SLOT 4 | CHASSIS, WARNING, DL_DCX_4S,  S6,P-1(48): Link Timeout on internal port  ftx=143708672 tov=2000 (>1000) vc_no=26 crd(s)lost=1 complete_loss:1.

 

I can see possible 4 VC (only two VC above) which have lost potentially buffer credit on the backblane. 

Enable the BE credit recovery, and if verify if it correct the lost credit (s). If not, we need to force a check

on links in the backplane.

 

switch:admin> bottleneckmon --cfgcredittools -intport -recover onLrOnly

 

Last, for the links, I see that you use 4 x 1 ISL for link to the 5424, I would probably

went with 2 x 2, if you have trunking license.:

 

 grep DOE_M1000_1_C2_M5424 case955069943 | grep E-P
 135    7    7   028700   id     N8   Online      FC  E-Port  10:00:00:05:33:d2:61:ce "DOE_M1000_1_C2_M5424" (downstream)
 159    7   31   029f00   id     N8   Online      FC  E-Port  10:00:00:05:33:d2:61:ce "DOE_M1000_1_C2_M5424"
 199    8    7   02c700   id     N8   Online      FC  E-Port  10:00:00:05:33:d2:61:ce "DOE_M1000_1_C2_M5424"
 223    8   31   02df00   id     N8   Online      FC  E-Port  10:00:00:05:33:d2:61:ce "DOE_M1000_1_C2_M5424"

 

 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

Download FREE NVMe eBook