Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 5
Registered: ‎12-04-2017

Class 3 transmit frames discarded due to timeout

Dear all, Looking for help here! One of the ports where our SAN storage is connected is showing c3timeout tx errors. While having understood that during the c3timeout tx, the IO processing is not faster enough, hence the receiving count is more than the transfer, but I want to know whether the Storage connected is the not processing fast enough or the servers zoned to it? Below is the port stat output- --------------- FID128:admin> portstatsshow 18 stat_wtx 7151696221951 4-byte words transmitted stat_wrx 50195120110957 4-byte words received stat_ftx 3724140231 Frames transmitted stat_frx 1118044856 Frames received stat_c2_frx 0 Class 2 frames received stat_c3_frx 1118062508 Class 3 frames received stat_lc_rx 0 Link control frames received stat_mc_rx 0 Multicast frames received stat_mc_to 0 Multicast timeouts stat_mc_tx 0 Multicast frames transmitted tim_rdy_pri 0 Time R_RDY high priority tim_txcrd_z 69884976 Time TX Credit Zero (2.5Us ticks) tim_txcrd_z_vc 0- 3: 0 0 0 0 tim_txcrd_z_vc 4- 7: 69884976 0 0 0 tim_txcrd_z_vc 8-11: 0 0 0 0 tim_txcrd_z_vc 12-15: 0 0 0 0 tim_latency_vc 0- 3: 1 1 1 1 tim_latency_vc 4- 7: 1 1 1 1 tim_latency_vc 8-11: 1 1 1 1 tim_latency_vc 12-15: 1 1 1 1 fec_cor_detected 0 Count of blocks that were corrected by FEC fec_uncor_detected 0 Count of blocks that were left uncorrected by FEC er_enc_in 0 Encoding errors inside of frames er_crc 0 Frames with CRC errors er_trunc 0 Frames shorter than minimum er_toolong 0 Frames longer than maximum er_bad_eof 0 Frames with bad end-of-frame er_enc_out 0 Encoding error outside of frames er_bad_os 0 Invalid ordered set er_pcs_blk 0 PCS block errors er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout er_tx_c3_timeout 23211 Class 3 transmit frames discarded due to timeout er_unroutable 0 Frames that are unroutable er_unreachable 0 Frames with unreachable destination er_other_discard 1 Other discards er_type1_miss 0 frames with FTB type 1 miss er_type2_miss 0 frames with FTB type 2 miss er_type6_miss 0 frames with FTB type 6 miss er_zone_miss 1 frames with hard zoning miss er_lun_zone_miss 0 frames with LUN zoning miss er_crc_good_eof 0 Crc error with good eof er_inv_arb 0 Invalid ARB er_single_credit_loss 0 Single vcrdy/frame loss on link er_multi_credit_loss 0 Multiple vcrdy/frame loss on link phy_stats_clear_ts 11-18-2017 UTC Sat 11:13:55 Timestamp of phy_port stats clear lgc_stats_clear_ts 11-18-2017 UTC Sat 11:13:55 Timestamp of lgc_port stats clear ---------------
Broadcom Moderator
Posts: 415
Registered: ‎08-31-2009

Re: Class 3 transmit frames discarded due to timeout

[ Edited ]

Hello,

 

Yes looking like storage port is not able to handle frames from the servers.

What are the port speed of the servers and storage?

This might be a Q depth issue, an issue on storage itself or a wrong optimization on it.

 

 

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 5
Registered: ‎12-04-2017

Re: Class 3 transmit frames discarded due to timeout

Dear Thierry/ all, This storage port which is connected to switch port 18, it is zoned with multiple servers having speed of 8 Gbps as well as 16Gbps. The switch port speed is 16Gbps. Storage port speed is 8Gbps. In last 15 days, the "maximum" data transfer recorded on the storage's port is 750MB/s After running mapsdb --show, below health related output produced:- ------------ 3.2 Rules Affecting Health: =========================== Category(Rule Count)|RepeatCount|Rule Name |Execution Time |Object |Triggered Value(Units)| ------------------------------------------------------------------------------------------------------------------------ Fru Health(4) |2 |defALL_PORTSSFP_STATE_IN |12/20/17 15:23:26|U-Port 77 |IN | | | | |U-Port 77 |IN | |2 |defALL_PORTSSFP_STATE_OUT |12/20/17 15:23:04|U-Port 77 |OUT | | | | |U-Port 77 |OUT | Fabric Performance I|1 |defALL_PORTS_IO_LATENCY_CLE|12/20/17 10:55:45|F-Port 18 |IO_LATENCY_CLEAR | mpact(52) | |AR | | | | |1 |defALL_PORTS_IO_FRAME_LOSS |12/20/17 10:54:45|F-Port 18 |IO_FRAME_LOSS | |1 |defALL_PORTS_IO_LATENCY_CLE|12/19/17 16:04:46|F-Port 18 |IO_LATENCY_CLEAR | | |AR | | | | |1 |defALL_PORTS_IO_FRAME_LOSS |12/19/17 16:03:46|F-Port 18 |IO_FRAME_LOSS | |18 |defALL_PORTS_IO_LATENCY_CLE|12/18/17 11:02:48|F-Port 18 |IO_LATENCY_CLEAR | | |AR | | | | | | | |F-Port 18 |IO_LATENCY_CLEAR | | | | |F-Port 18 |IO_LATENCY_CLEAR | | | | |F-Port 18 |IO_LATENCY_CLEAR | | | | |F-Port 18 |IO_LATENCY_CLEAR | -------------------------- Please could you suggest any further troubleshooting? Thanks.
Broadcom Moderator
Posts: 87
Registered: ‎03-29-2010

Re: Class 3 transmit frames discarded due to timeout

Troubleshooting class 3 discards, and device latency issues is a large discussion and can include many aspects of a SAN. A slow draining device can cause this, a poor performing HBA, or a fan ratio issue can contribute, and ISL oversubscription can also affect latency. Please note that port 18 in the report is the port which is BEING affected, and is not likely the port which is the cause of the problem.

 

First things to do: run the commands: statsclear; slotstatsclear  on each switch in the fabric. After 24-48 hours run porterrshow on each switch in the fabric. Review those outputs for errors, and advise what you find. Also provide your output of firmwareshow, fabricshow, so that we know what kind of equipment we are looking at.

 

I would advise you to run the SAN Health report, and gather some info on your connections, and througput. Without a complete picture of the fabric, and all F and E port connections it will be impossible to diagnose. There are also records in the log file which may be useful in determining what ports are causing latency within the fabric. 

 

Start here:

https://my.brocade.com/wps/myportal/myb/tools/sanhealth/

 

You will get a report emailed to you showing the switch connections, and many attributes of the fabric. Once that report is checked, we can proceed with some options. As a first guess, try to find legacy devices which may be running at 4GB and are traversing the fabric via ISL(E_ports). This is a common issue causing class 3 discards, but it is only one possible issue. There are many other things which affect throughput and congestion.

doc

Any and all information provided by me is for entertainment value and should not be relied upon as a guaranteed solution or warranty of mechantability. All systems and all networks are different and unique. If you have a concern about data loss, or network disconnection, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, Please mark it with the button at the bottom "Accept as solution".

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.