Fibre Channel (SAN)

Reply
Occasional Contributor
antonis.marcou
Posts: 7
Registered: ‎12-27-2010

AIX error log reports repeated FCP_ERR12 errors

Hi Guys,

About a week ago all aix servers that are connected to the SAN are giving repeated FCP_ERR12 errors.

example: error_log attached

LABEL:          FCP_ERR12

IDENTIFIER:     26623394

Date/Time:       Wed Sep  5 10:55:43 CYS 2012

Sequence Number: 395633

Machine Id:      00C3A7044C00

Node Id:         e_hercules

Class:           H

Type:            TEMP

WPAR:            Global

Resource Name:   fscsi1        

Resource Class:  driver

Resource Type:   efscsi

Location:        U789D.001.DQD29HP-P1-C2-T1

Description

COMMUNICATION PROTOCOL ERROR

    Recommended Actions

    PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data

SENSE DATA

0000 0010 0000 00AC 0000 0901 0204 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0002 1500 0000 0000 0001 0800 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0080 6000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0100 0000 FC02 0000 0112 0002 0000 0000 0001 0800 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

1000 0000 C97C E0C2 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

Our enviroment consists of two fabrics each with two Brocade 300 switches. I'm not sure if this can cause problems but the FOS on one switch in the fabric is v7.0.0a and the second is FOS v6.4.2

Also when we run cfgmgr on all aix servers we receive the following warning cfgmgr: 0514-621 WARNING: The following device packages are required for  device support but are not currently installed.

devices.fcp.tm

We update to the latest firmware all fiber adapters

** FC Adapter Information

Adapter Name:                com.ibm-df1000fe-1

Manufacturer:                IBM

SerialNumber:                1F8340C6B3

Model:                       df1000fe

Model Description:           FC Adapter

HBA WWN:                     20000000C97C2713

Node Symbolic Name:        

Hardware Version:          

Driver Version:              6.1.7.15

Option ROM Version:          02E8277F

Firmware Version:            271315

Vendor Specific ID:          0

Number Of Ports:             1

Driver Name:                 /usr/lib/drivers/pci/efcdd

Port Index:                  0

Node WWN:                    20000000C97C2713

Port WWN:                    10000000C97C2713

Port Fc Id:                  136960

Port Type:                   Fabric

Port State:                  Operational

Port Symbolic Name:        

OS Device Name:              fcs0

Port Supported Speed:        4 GBit/sec

Port Speed:                  4 GBit/sec

Port Max Frame Size:         2112

Fabric Name:                 100000051E8CB382

Number of Discovered Ports:  4

Seconds Since Last Reset:       3554027

Tx Frames:                      147215

Tx Words:                       11075

Rx Frames:                      521461

Rx Words:                       499283

LIP Count:                      0

NOS Count:                      0

Error Frames:                   0

Dumped Frames:                  0

Link Failure Count:             1

Loss of Sync Count:             1

Loss of Signal Count:           0

Primitive Seq Protocol Err Cnt: 0

Invalid Tx Word Count:          2

Invalid CRC Count:              0

** FC Adapter Information

Adapter Name:                com.ibm-df1000fe-2

Manufacturer:                IBM

SerialNumber:                1F8310C00E

Model:                       df1000fe

Model Description:           FC Adapter

HBA WWN:                     20000000C97CE0C2

Node Symbolic Name:        

Hardware Version:          

Driver Version:              6.1.7.15

Option ROM Version:          02E8277F

Firmware Version:            271315

Vendor Specific ID:          0

Number Of Ports:             1

Driver Name:                 /usr/lib/drivers/pci/efcdd

Port Index:                  0

Node WWN:                    20000000C97CE0C2

Port WWN:                    10000000C97CE0C2

Port Fc Id:                  136448

Port Type:                   Fabric

Port State:                  Operational

Port Symbolic Name:        

OS Device Name:              fcs1

Port Supported Speed:        4 GBit/sec

Port Speed:                  4 GBit/sec

Port Max Frame Size:         2112

Fabric Name:                 100000051E8C779C

Number of Discovered Ports:  4

Seconds Since Last Reset:       3554028

Tx Frames:                      156664

Tx Words:                       11367

Rx Frames:                      515399

Rx Words:                       450411

LIP Count:                      0

NOS Count:                      0

Error Frames:                   0

Dumped Frames:                  0

Link Failure Count:             1

Loss of Sync Count:             1

Loss of Signal Count:           0

Primitive Seq Protocol Err Cnt: 0

Invalid Tx Word Count:          2

Invalid CRC Count:              0

** FC Adapter Information

Adapter Name:                com.ibm-df1000fd-3

Manufacturer:                IBM

SerialNumber:                1F5440C0C1

Model:                       df1000fd

Model Description:           FC Adapter

HBA WWN:                     20000000C9509B54

Node Symbolic Name:        

Hardware Version:          

Driver Version:              6.1.7.15

Option ROM Version:          02C82774

Firmware Version:            271304

Vendor Specific ID:          0

Number Of Ports:             1

Driver Name:                 /usr/lib/drivers/pci/efcdd

Port Index:                  0

Node WWN:                    20000000C9509B54

Port WWN:                    10000000C9509B54

Port Fc Id:                  132352

Port Type:                   Fabric

Port State:                  Operational

Port Symbolic Name:        

OS Device Name:              fcs2

Port Supported Speed:        4 GBit/sec

Port Speed:                  4 GBit/sec

Port Max Frame Size:         2112

Fabric Name:                 100000051E8C779C

Number of Discovered Ports:  4

Seconds Since Last Reset:       15

Tx Frames:                      1239

Tx Words:                       77

Rx Frames:                      1164

Rx Words:                       100

LIP Count:                      0

NOS Count:                      0

Error Frames:                   0

Dumped Frames:                  0

Link Failure Count:             0

Loss of Sync Count:             1

Loss of Signal Count:           0

Primitive Seq Protocol Err Cnt: 0

Invalid Tx Word Count:          0

Invalid CRC Count:              0

Apart from the errors in the log we experince no problems with the disk array. But the backup application Calypso (Simpana) is showing that the Tape library is offline due to the controller is down (Aix fiber adapter fcs2). We reset the controller we bring the library online and after a few minitues the library will go offline again. Is this a low drain device problem?

We do not have much expertise with SAN so any help would be greatly appreciated

Thank you

Regards

Antonis Marcou

Valued Contributor
felipon
Posts: 546
Registered: ‎06-11-2010

Re: AIX error log reports repeated FCP_ERR12 errors

Hi,

You have to check if all the errors are reported on the same HBA in the 6 AIX servers, or if both HBAs report them.

Take a look at the errors in ports, specially the ISLs if storage array is connected to a different switch.

Usuallly tape libraries are more sensitive to SAN issues.

If you attach a diagram with the topology of the SAN, it could help to find the source of the error...

I have found the following post on IBM website stating that it was an AIX issue.

http://www-01.ibm.com/support/docview.wss?uid=isg1IZ98316

It is true, that it seems that your Fabric is not working in interopmode but maybe there has been a change in the SAN in the past few days that brought about these situation.

Kind regards

Rgds

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.