This also media related issue, change the Fibre Cable, SFP or HBA...

Thanks hemant,

in our case it is not related to media, cabel or SFP.

It is related to 8 gbit compatiblity issues. But I don't understand what is going wrong between switch and device. I would like to get a more detailed description why idel nor abrff are not working correctly.

Andreas

.

Hi,

Pls do a portstatsclear and see thorugh portstatsshow  and porterrshow , if the same is increasing or not. You may have to upgrade  the driver and FW version of the HBAs

Thanks,

for your help to find a solution, but my question was to get an explenation for this behavior.  By the way a HBA firmware update on a storage array is not possible. The storage array runs the latest code.

I would like to know if other customers had the same issue with 8 Gbit storage ports on 8 Gbit Brocade switches.

This parameters is (platform/port specific). did you do a portstatsclear and check it increasing or not ?Check the compatibilty of HBA Driver and Firmware version with Storage microcode. You have to see other parameters also er_enc_out, if all these values are increasing after doing portstatsclear, then you have to change the cable and SFP.BTW which storage is this with 8Gbps CHA ports?There is no such compatibiltiy issue . SAN SW works on AN mode.

All other errcounter are at zero level. So no cabling issue. If I configure the storage port fixed to 4 Gbit everything is fine only at 8 Gbit speed I can see the increasing er_bad_os. Hitachi storage is affected. USP-V and AMS 2500 with 8 Gbit ports. As mentioned above FOS 6.3.1a is running on the switch and latest code on the array. And only the switch port where the array is connected is affected. we have EMULEX LPe12000 and Brocade 815 on the server side without any issues. Data traffic can pass the storage without problems. But I can see the counter increase very quickly...

xwdees02_a1_ds:FID128:A15710> portcfgshow 1/9
Area Number:              9
Speed Level:              AUTO(HW)
Fill Word:                0(Idle-Idle)

....

xwdees02_a1_ds:FID128:A15710> statsclear
xwdees02_a1_ds:FID128:A15710> portstatsshow 1/9
stat_wtx                0           4-byte words transmitted
stat_wrx                0           4-byte words received
stat_ftx                0           Frames transmitted
stat_frx                0           Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             0           Class 3 frames received
stat_lc_rx              0           Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             0           Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           0           0
tim_txcrd_z_vc  4- 7:  0           0           0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               409610422   Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

Wait some seconds...

xwdees02_a1_ds:FID128:A15710> portstatsshow 1/9
stat_wtx                0           4-byte words transmitted
stat_wrx                0           4-byte words received
stat_ftx                0           Frames transmitted
stat_frx                0           Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             0           Class 3 frames received
stat_lc_rx              0           Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             0           Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           0           0
tim_txcrd_z_vc  4- 7:  0           0           0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               716822618   Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy
xwdees02_a1_ds:FID128:A15710>

As you can see everything is fine.

If I create some load on the port there are also no increasing error count. These "er_bas_os" are not visible to the server and also not visible to the storage array. Hiatchi adviced us to configure the ports to the settings from above.

We have no transport erros currently.

I am looking for an explenation of this behavior.

Thanks,

Andreas

yes, definitely it is strange. Have you tried changing the port from this to other ports. you are saying that while putting load does not increase the value. is the server HBA idle .What else the HITACHI people said.we can ignore this value also. unless until you face congestion. this parameter increase only due to server reboot, bad cable, SFP, . Also do a portstatsclear not only statsclear. Is theer any error on porterrshow.what is showing on errdump? I do not hink this is a compatibiltiy issue at CHA port or switch port. you also say that while you set the port to 4 gbps, it is ok. Check portshow also and portloginshow.

Have you tried changing the port from this to other ports --> Yes, the same issue

you are saying that while putting load does not increase the value--> I am sorry my english is not very good. The er_bas_os increase in the same speed at the affected storage . The data  flow had no problems.

Is theer any error on porterrshow --> no

you also say that while you set the port to 4 gbps, it is ok --> Yes correct

Check portshow also and portloginshow --> No problems ervy thing is fine

What else the HITACHI people said --> Ignore the counter.

But this looks not as a well tested and compatible product combination. Looks like that Brocade is not talking to the rest of the world and make sure that new technology works as planned...

Can you ask Brocade the engineering what is going wrong? I think you are working for Brocade, correct?

I would like to understand what the problem is and who can fix it.

Thanks,

Andreas

Hi,

No, I am not working for Brocade, but I am BCFP, BCSD, BCFD, BCSM(4 & 8 Gbps) certified and working on Brocade Dir class products with a huge 4000 SAN SWs ports for 4. 6 yrs. I have seen these things. I can say just to ignore this.

let me describe :

in portstatsshow we see port hardware statistics counter.some counters are platform and port specific and display only with those platforms and ports.

This parameter  wants to say , that any config/parameter is not set correct. That has several reason....

I have seen my friends facing the same :If  we change the Speed from 8 to 4 G the counter stops or the other solution is to change the fillword to ARBFF in Link Init, ARBFF as fill word.

Since ordered sets do not contain data, it has nothing to do with the dataflow. So we can ignore these.

Remember, ordered sets are purely within the SAN; the OS will never see them.

Have you checked the compatibility matrices between your server/HBA and the switch/FOS level, and between the server/HBA and the storage?

about orderedset:The round trip delay is measured by transmitting a particular Primitive Signal. A Primitive Signal is an Ordered Set used to indicate an event. An Ordered Set is a 4-byte Transmission Word which has the Special Character as its first Transmission Character. An Ordered Set may be a Frame Delimited, a Primitive Signal, or a Primitive Sequence. Ordered Sets are used to distinguish Fibre Channel control information from data. A Transmission Word is a string of four consecutive Transmission Characters--a (valid or invalid) 10-bit character transmitted serially over the fibre. Valid Transmission Characters are determined by the 8B/10B encoding specification. The Special Character is a special 10-bit Transmission Character which does not have a corresponding 8-bit value, but is still considered valid. The Special Character is used to indicate that a particular Transmission Word is an Ordered Set. The Special Character is the only Transmission Character to have five 1's or 0's in a row. The Special Character is also referred to as K28.5 when using K/D format. For additional explanation of these various terms, one may refer to the Fibre Channel standards, particularly FC-PH, which is ANSI publication X3.230, and is hereby incorporated by reference.

Also we have seen when there is no data tranmission between HBA and storage port this value also increases.

If HITACHI people have said to ignore this then they must have queried Brocade.

Andreas, they already gave themselves the answer...

--->>>in our case it is not related to media, cabel or SFP.

"It is related to 8 gbit compatiblity issues."

Andreas, they already gave themselves the answer...

--->>>in our case it is not related to media, cabel or SFP.

"It is related to 8 gbit compatiblity issues."

simply tell them to change the CHA board, or wait till the Microcode upgrade

No need to replace hardware.  Simply upgrade to a FOS level with the portcfgfillword command.  (I think it was introduced somewhere in 6.2)  Brocade defaults to ARBff, some storage devices still expect six IDLEs between frames, and their state machines fail if those IDLEs aren't received.  portcfgfillword <port number>, 0 and you will be all fixed.

This problem CAN cause data flow issues due to excess interrupts in HBAs.  I've seen it on some QL models.

Note that the fill word was changed for good reasons (ARBff improves signal timing), so the latest FOS versions allow you send six IDLEs to satisfy the state machine, and then send ARBff's.  So, you get the IDLEs for devices that require them, and you get the improved signal characteristics of ARBff.

I do not have any idea about portcfgfillword because I have not used it. But at HITACHI level microcode upgrade may resolve the issue.