Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 19
Registered: ‎12-16-2009
Accepted Solution

ports go offline/online every 4 seconds

[ Edited ]

Hi everybody.

BNA registers thouthands of events on 2 ports on the switch

 

Event: Monitoring and Alerting System notification - Rule defALL_OTHER_F_PORTSSTATE_CHG_20 violated. ObjSmiley TongueORT 6

Event: Monitoring and Alerting System notification - Rule defALL_OTHER_F_PORTSSTATE_CHG_20 violated. ObjSmiley TongueORT 7

 

fabriclog shows state change every 4 seconds

19:08:36.690997 SCN Port Offline;g=0x86068 A2,P0 A2,P0 6 NA
19:08:36.691015 *Removing all nodes from port A2,P0 A2,P0 6 NA
19:08:36.731416 SCN LR_PORT(0);g=0x86068 A2,P0 A2,P0 6 NA
19:08:36.731469 SCN Port Online; g=0x86068,isolated=0 A2,P0 A2,P1 6 NA
19:08:36.731631 Port Elp engaged A2,P1 A2,P0 6 NA
19:08:36.731708 *Removing all nodes from port A2,P0 A2,P0 6 NA
19:08:36.731849 SCN Port F_PORT A2,P1 A2,P0 6 NA
19:08:37.585108 SCN Port Offline;g=0x8606a A2,P0 A2,P0 7 NA
19:08:37.585127 *Removing all nodes from port A2,P0 A2,P0 7 NA
19:08:37.697383 SCN LR_PORT(0);g=0x8606a A2,P0 A2,P0 7 NA
19:08:37.697431 SCN Port Online; g=0x8606a,isolated=0 A2,P0 A2,P1 7 NA
19:08:37.698088 Port Elp engaged A2,P1 A2,P0 7 NA
19:08:37.698200 *Removing all nodes from port A2,P0 A2,P0 7 NA
19:08:37.698339 SCN Port F_PORT A2,P1 A2,P0 7 NA
19:08:40.742972 SCN Port Offline;g=0x8606c A2,P0 A2,P0 6 NA
19:08:40.742990 *Removing all nodes from port A2,P0 A2,P0 6 NA
19:08:40.784690 SCN LR_PORT(0);g=0x8606c A2,P0 A2,P0 6 NA
19:08:40.784745 SCN Port Online; g=0x8606c,isolated=0 A2,P0 A2,P1 6 NA
19:08:40.784905 Port Elp engaged A2,P1 A2,P0 6 NA
19:08:40.784983 *Removing all nodes from port A2,P0 A2,P0 6 NA
19:08:40.785125 SCN Port F_PORT A2,P1 A2,P0 6 NA
19:08:41.637125 SCN Port Offline;g=0x8606e A2,P0 A2,P0 7 NA
19:08:41.637143 *Removing all nodes from port A2,P0 A2,P0 7 NA
19:08:41.715646 SCN LR_PORT(0);g=0x8606e A2,P0 A2,P0 7 NA
19:08:41.715693 SCN Port Online; g=0x8606e,isolated=0 A2,P0 A2,P1 7 NA
19:08:41.715856 Port Elp engaged A2,P1 A2,P0 7 NA
19:08:41.715934 *Removing all nodes from port A2,P0 A2,P0 7 NA
19:08:41.716077 SCN Port F_PORT A2,P1 A2,P0 7 NA

 

portshow shows big amount of link reset

Lr_in: 124131
0 Lr_out: 124131
Ols_in: 124131
Ols_out: 124131

 

We've changed sfp and cable but the erorrs remained.

 

These two ports are used by NetBackup server.

 

Is it likely a software issue?

any ideas of further investigation?

Brocade Moderator
Posts: 225
Registered: ‎08-31-2009

Re: ports go offline/online every 4 seconds

Hello,

 

Can you share with us the complete output of portstatsshow for those ports and porterrshow?

 

 

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Brocade Moderator
Posts: 36
Registered: ‎03-29-2010

Re: ports go offline/online every 4 seconds

Lets do this first to get MAPS to stop squawking:

 

 

mapsconfig --config pause -type port -members "6,7"

 

This will kill the MAPS monitoring and stop filling up your BNA log with faults. Heck, it may even stop the ports from flapping depending on what actions you have in effect for that rule.

 

Next lets look at the details for the failing MAPS elements:

 

mapsdb --show details -day mm/dd/yyyy

 

This will list the errors for the day you choose.

 

And, we'll have a look at that specific rule that's being  violated:

 

mapsrule --show [rule_name]   (*input that rule name above that you want to show *)

 

Post this up and we can dig in deeper. If the ports don't stop flapping, you should disable them. It's using a fair amount of switch resources to keep up with all the RSCN and port management.

 

Best of luck

doc

Any and all information provided by me is for entertainment value and should not be relied upon as a guaranteed solution or warranty of mechantability. All systems and all networks are different and unique. If you have a concern about data loss, or network disconnection, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, Please mark it with the button at the bottom "Accept as solution".

Occasional Contributor
Posts: 19
Registered: ‎12-16-2009

Re: ports go offline/online every 4 seconds

 Thierry, here they are

 

frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err
0: 1.4g 3.1g 0 0 0 0 0 0 0 9 0 0 0 0 0 9 0 0
1: 2.4g 3.2g 0 0 0 0 0 0 0 9 0 0 0 0 0 9 0 0
2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4: 541.9k 541.6k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5: 195 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6: 2.9g 1.2g 0 0 0 0 0 0 6 0 4 0 4 0 0 0 0 0
7: 3.8g 1.4g 0 0 0 0 0 0 0 1 5 0 5 0 0 0 1 0

 

vz06bk14-e1-133:asfareny> portshow 6
portIndex: 6
portName: nbumed_vz_03_ft1
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24303 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGIN NOELP LED ACCEPT FLOGI PROBING
LocalSwcFlags: 0x0
portType: 24.0
POD Port: Port is licensed
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 568096
state transition count: 257512

portId: 850600
portIfId: 43020003
portWwn: 20:06:00:27:f8:22:72:9b
portWwn of device(s) connected:
21:00:00:24:ff:32:21:a5
Distance: normal
portSpeed: N8Gbps

FEC: Inactive
Credit Recovery: Inactive
Aoq: Inactive
FAA: Inactive
F_Trunk: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts: 0 Link_failure: 4 Frjt: 0
Unknown: 0 Loss_of_sync: 0 Fbsy: 0
Lli: 514914 Loss_of_sig: 4
Proc_rqrd: 128786 Protocol_err: 0
Timed_out: 0 Invalid_word: 6
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 128756
Suspended: 0 Lr_out: 128756
Parity_err: 0 Ols_in: 128756
2_parity_err: 0 Ols_out: 128756
CMI_bus_err: 0

Port part of other ADs: No

 

vz06bk14-e1-133:asfareny> portshow 7
portIndex: 7
portName: nbumed_vz_04_ft1
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24303 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGIN NOELP LED ACCEPT FLOGI PROBING
LocalSwcFlags: 0x0
portType: 24.0
POD Port: Port is licensed
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 568146
state transition count: 257774

portId: 850700
portIfId: 43020006
portWwn: 20:07:00:27:f8:22:72:9b
portWwn of device(s) connected:
21:00:00:24:ff:32:25:df
Distance: normal
portSpeed: N8Gbps

FEC: Inactive
Credit Recovery: Inactive
Aoq: Inactive
FAA: Inactive
F_Trunk: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts: 0 Link_failure: 5 Frjt: 0
Unknown: 0 Loss_of_sync: 0 Fbsy: 0
Lli: 515425 Loss_of_sig: 5
Proc_rqrd: 128918 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 128887
Suspended: 0 Lr_out: 128887
Parity_err: 0 Ols_in: 128887
2_parity_err: 0 Ols_out: 128887
CMI_bus_err: 0

Port part of other ADs: No

 

and one remark: its not one, they are 2 separate netbackup media servers

 

Interesting they have almost the same number of ling resets

Brocade Moderator
Posts: 225
Registered: ‎08-31-2009

Re: ports go offline/online every 4 seconds

Hello,

 

Can you add portstatsshow command also for those 2 ports?:

portstatsshow 6

portstatsshow 7

 

 

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Brocade Moderator
Posts: 225
Registered: ‎08-31-2009

Re: ports go offline/online every 4 seconds

[ Edited ]

Have you tryed to disable and enable ports and see if behaviour is back to normal? (disruptive).

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 19
Registered: ‎12-16-2009

Re: ports go offline/online every 4 seconds

Doc, thank you for response.

Currently no actions are enabled for MAPS, therefore port continues to flap.

the question is not concerned MAPS itself but the reason why this flapping occurs.

 

As I believe, MAPS settings itself can not put the port offline/online in such a manner, or am I wrong?

Anyway I post here output from what you've asked.

maps output

 

We can not turn this port off as backups are running via those ports. Backup admin says that they do not notice any troubles with backups.

 

Occasional Contributor
Posts: 19
Registered: ‎12-16-2009

Re: ports go offline/online every 4 seconds

[ Edited ]

portstatsshow 6
stat_wtx 980324153 4-byte words transmitted
stat_wrx 2788657065 4-byte words received
stat_ftx 2964121642 Frames transmitted
stat_frx 1206174147 Frames received
stat_c2_frx 0 Class 2 frames received
stat_c3_frx 1206174147 Class 3 frames received
stat_lc_rx 0 Link control frames received
stat_mc_rx 0 Multicast frames received
stat_mc_to 0 Multicast timeouts
stat_mc_tx 0 Multicast frames transmitted
tim_rdy_pri 0 Time R_RDY high priority
tim_txcrd_z 11 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0 0 0 0
tim_txcrd_z_vc 4- 7: 11 0 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0
tim_latency_vc 0- 3: 1 1 1 1
tim_latency_vc 4- 7: 1 1 1 1
tim_latency_vc 8-11: 1 1 1 1
tim_latency_vc 12-15: 1 1 1 1
fec_cor_detected 0 Count of blocks that were corrected by FEC
fec_uncor_detected 0 Count of blocks that were left uncorrected by FEC
er_enc_in 0 Encoding errors inside of frames
er_crc 0 Frames with CRC errors
er_trunc 0 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 0 Frames with bad end-of-frame
er_enc_out 6 Encoding error outside of frames
er_bad_os 3 Invalid ordered set
er_pcs_blk 0 PCS block errors
er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout
er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout
er_unroutable 0 Frames that are unroutable
er_unreachable 0 Frames with unreachable destination
er_other_discard 0 Other discards
er_type1_miss 0 frames with FTB type 1 miss
er_type2_miss 0 frames with FTB type 2 miss
er_type6_miss 0 frames with FTB type 6 miss
er_zone_miss 0 frames with hard zoning miss
er_lun_zone_miss 0 frames with LUN zoning miss
er_crc_good_eof 0 Crc error with good eof
er_inv_arb 0 Invalid ARB
er_single_credit_loss 0 Single vcrdy/frame loss on link
er_multi_credit_loss 0 Multiple vcrdy/frame loss on link
phy_stats_clear_ts 08-15-2016 Localtime Mon 23:09:16 Timestamp of phy_port stats clear
lgc_stats_clear_ts 08-15-2016 Localtime Mon 23:09:16 Timestamp of lgc_port stats clear


portstatsshow 7
stat_wtx 3163105509 4-byte words transmitted
stat_wrx 2522488255 4-byte words received
stat_ftx 3818762320 Frames transmitted
stat_frx 1421175407 Frames received
stat_c2_frx 0 Class 2 frames received
stat_c3_frx 1421175407 Class 3 frames received
stat_lc_rx 0 Link control frames received
stat_mc_rx 0 Multicast frames received
stat_mc_to 0 Multicast timeouts
stat_mc_tx 0 Multicast frames transmitted
tim_rdy_pri 0 Time R_RDY high priority
tim_txcrd_z 34 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0 0 0 0
tim_txcrd_z_vc 4- 7: 0 34 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0
tim_latency_vc 0- 3: 1 1 1 1
tim_latency_vc 4- 7: 1 1 1 1
tim_latency_vc 8-11: 1 1 1 1
tim_latency_vc 12-15: 1 1 1 1
fec_cor_detected 0 Count of blocks that were corrected by FEC
fec_uncor_detected 0 Count of blocks that were left uncorrected by FEC
er_enc_in 0 Encoding errors inside of frames
er_crc 0 Frames with CRC errors
er_trunc 0 Frames shorter than minimum
er_toolong 0 Frames longer than maximum
er_bad_eof 0 Frames with bad end-of-frame
er_enc_out 0 Encoding error outside of frames
er_bad_os 0 Invalid ordered set
er_pcs_blk 0 PCS block errors
er_rx_c3_timeout 1 Class 3 receive frames discarded due to timeout
er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout
er_unroutable 0 Frames that are unroutable
er_unreachable 0 Frames with unreachable destination
er_other_discard 0 Other discards
er_type1_miss 0 frames with FTB type 1 miss
er_type2_miss 0 frames with FTB type 2 miss
er_type6_miss 0 frames with FTB type 6 miss
er_zone_miss 0 frames with hard zoning miss
er_lun_zone_miss 0 frames with LUN zoning miss
er_crc_good_eof 0 Crc error with good eof
er_inv_arb 0 Invalid ARB
er_single_credit_loss 0 Single vcrdy/frame loss on link
er_multi_credit_loss 0 Multiple vcrdy/frame loss on link
phy_stats_clear_ts 08-15-2016 Localtime Mon 23:09:16 Timestamp of phy_port stats clear
lgc_stats_clear_ts 08-15-2016 Localtime Mon 23:09:16 Timestamp of lgc_port stats clear

 

I've just disabled and enabled back port 6. will see if it makes any difference

Contributor
Posts: 66
Registered: ‎12-24-2015

Re: ports go offline/online every 4 seconds

Hi!
Add some additional buffers to your flapping ports.
You have the latency on vc channels.

What is about link reset counts?
Brocade Moderator
Posts: 49
Registered: ‎04-27-2009

Re: ports go offline/online every 4 seconds

This is most likely caused by the attached devices itself. I am very surprised that your backup colleagues do not notice any trouble. I would go back to the timestamp when this started and see if there is a common thing on the server(s) side for these two ports.

Also if it started at the same time on both ports would be interessting.

 

 

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.