Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 17
Registered: ‎08-08-2012
Accepted Solution

The stuck ISL VC

Environment: DCX-4S(6.4.2a)、 5100(6.4.2a)
Problem: Stuck ISL VC


2015/08/20-01:05:10, [C2-1012], 2725, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2909
560 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-01:02:11, [SNMP-1008], 2724, SLOT 5 | FID 128, INFO, ZHSAN01, The last device change happened at : Thu Aug 20 01:02:04 2
015

2015/08/20-01:02:04, [C2-1012], 2723, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0219 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-01:01:32, [C2-1012], 2722, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4002946 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-01:01:31, [C2-1012], 2721, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862574 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-01:00:04, [C2-1012], 2720, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2909
201 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:57:03, [C2-1012], 2719, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0198 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:56:32, [C2-1012], 2718, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4002957 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:56:31, [C2-1012], 2717, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862585 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:55:03, [C2-1012], 2716, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2908
901 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:52:03, [C2-1012], 2715, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0173 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:51:32, [C2-1012], 2714, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4002975 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:51:31, [C2-1012], 2713, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862594 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:50:02, [C2-1012], 2712, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2908
576 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:47:02, [C2-1012], 2711, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0146 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:46:32, [C2-1012], 2710, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4002988 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:46:31, [C2-1012], 2709, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862599 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:44:55, [C2-1012], 2708, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2908
253 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:42:01, [C2-1012], 2707, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0119 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:41:32, [C2-1012], 2706, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4003002 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:41:31, [C2-1012], 2705, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862614 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:39:54, [C2-1012], 2704, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2907
930 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:37:01, [C2-1012], 2703, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0096 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:36:31, [C2-1012], 2702, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4003018 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:36:31, [C2-1012], 2701, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862618 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:34:53, [C2-1012], 2700, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2907
587 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:34:49, [SEC-1203], 2699, SLOT 5 | FID 128, INFO, ZHSAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 192.167.200.99

2015/08/20-00:32:00, [C2-1012], 2698, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0069 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:31:31, [C2-1012], 2697, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862627 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:31:30, [C2-1012], 2696, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4003028 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:29:48, [C2-1012], 2695, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2907
282 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:27:11, [SNMP-1008], 2694, SLOT 5 | FID 128, INFO, ZHSAN01, The last device change happened at : Thu Aug 20 00:27:08 2015

2015/08/20-00:26:59, [C2-1012], 2693, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7067
0045 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:26:56, [SNMP-1008], 2692, SLOT 5 | FID 128, INFO, ZHSAN01, The last device change happened at : Thu Aug 20 00:26:51 2015

2015/08/20-00:26:41, [SNMP-1008], 2691, SLOT 5 | FID 128, INFO, ZHSAN01, The last device change happened at : Thu Aug 20 00:26:31 2015

2015/08/20-00:26:31, [C2-1012], 2690, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S6,P-1(10): Link Timeout on internal port ftx=-213
1862641 tov=2000 (>1000) vc_no=1 crd(s)lost=3 complete_loss:1

2015/08/20-00:26:29, [C2-1012], 2689, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S8,P-1(50): Link Timeout on internal port ftx=-195
4003052 tov=2000 (>1000) vc_no=1 crd(s)lost=1 complete_loss:0

2015/08/20-00:24:48, [C2-1012], 2688, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(10): Link Timeout on internal port ftx=2906
961 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1

2015/08/20-00:21:59, [C2-1012], 2687, SLOT 5 | CHASSIS, WARNING, Brocade_DCX4S, S1,P-1(48): Link Timeout on internal port ftx=7066
9955 tov=2000 (>1000) vc_no=4 crd(s)lost=3 complete_loss:1


Questions:
1) Why such stuck ISL VC problem would happen ?
2) Why would not all the relvant ports within the same stuck ISL VC work failed, some of these ports could still work fine ?
3) How to resolve such stuck ISL VC problem on flight since it was invalid via 'bottleneckmon -cfgcredittools -intport -recover onLrOnly' ?

Brocadian
Posts: 3
Registered: ‎03-24-2011

Re: The stuck ISL VC

bottleneckmon --cfgcredittools -intport -recover onLrOnly should be a valid command for a DCX4S running 6.4.2a.

6.4.2a is where internal credit recovery was introduced. It is not valid on a 5100.

Valued Contributor
Posts: 554
Registered: ‎03-20-2011

Re: The stuck ISL VC

Later releases would even more improve most of the internal port issues. You'd really better go up to 7.3.x or even 7.4.x. Both platforms you mentioned support all the FOS releases available so far except 8.x.
Occasional Contributor
Posts: 17
Registered: ‎08-08-2012

Re: The stuck ISL VC

Thanks!!

Any other one have good idea for my above 3 questions !?

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: The stuck ISL VC

Hello,

 

As Far as I Know, without bottleneckmon --cfgcredittools -intport -recover onLrOnly, the only other method to solve a stuck vc on an internal port is a blade Remove/Reinsert (Slot 1 in your case).

 

There would be several reasons for the occurence of a stucked vc. Further investigation is required to answer this.

 

 

Occasional Contributor
Posts: 17
Registered: ‎08-08-2012

Re: The stuck ISL VC

Thanks again!!

What infomation & log would you like to analyze for !?   supportshow or supportsave ?!

 

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: The stuck ISL VC

A support show from all fabric switches and a SanHealth report would be a great starting point.

In time: San Health is a Brocade tool and it is free.

Occasional Contributor
Posts: 17
Registered: ‎08-08-2012

Re: The stuck ISL VC

Please see the attach files for supportshow !

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: The stuck ISL VC

Hello liuyl,


There is a great article from Sebastian Thaele about stuck vc's and i highly recommend that you take a look:
- https://www.ibm.com/developerworks/community/blogs/sanblog/entry/stuck_vcs_or_why_my_switch_began_to_nag3?lang=en

There you can find links to other articles that will complement the concept about vc's and how it works.

In a brief resume, since 6.4, brocade introduced a monitoring feature that act on internal backlink ports. Remember that a Brocade SAN director is built under several Condor/Condor2 asics arranged in a mesh topology. So, the interconnection between asics act as an ISL. It is just invisible to us, but the internal ports are - in a simple explanation - E-Ports that connect an asic to another (and them, a blade to another, or to a core edge blade in DCX).

In a normal E-Port, we are able to play with functions like credit recovery, but on internal ports that was not available to us. Then Brocade introduced the -cfgcredittools parameter to bottleneckmon. That enabled us to configure a method to recover the stucked vc by a link reset.

About your question 2, we need to do a deep dive in how data flows between the ISL's VC.

Again, looking at a good article from Seb's SAN Blog, we can find that there is a relationship between the port index and the VC used when the data flow trhough an ISL:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/sanblog/entry/how_to_not_connect_an_svc_in_a_core_edge_brocade_fabric16?lang=en_us

the message here is: the way you populate/plug cables in your switch/director will determine the ISL's VCs usage.

Hope that this provide you a heading to solve you problem.

Occasional Contributor
Posts: 17
Registered: ‎08-08-2012

Re: The stuck ISL VC

1) Yes, I had read the great article from Sebastian Thaele about stuck vc!  Could I thought that the problem of stuck VC would be triggered via  either FOS code bug or E-Ports' hardware issue(but except for link issue) ?
2) I had also read the good article from Seb's SAN Blog, and I had checked all the relevant F-Ports within the same VC ID, but it was so strange that some ports of them indeed still work fine !!

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook