For more details, please see ourCookie Policy.


Fibre Channel (SAN)

Reply
Highlighted
jkm
Occasional Contributor
Posts: 8
Registered: ‎04-29-2018
Accepted Solution

In a Full Duplex World how do you know which direction a frame is travelling?

Topology

Site X Array Port <-> Site X Brocade Switch Port <-> Site X DWDM Port <-> (70 KMs Dark Fibre) <-> Site YDWDM Port <-> Site Y Brocade Switch Port <-> Site Y Array Port

Fabric at both sites are dedicated replication and replication occurs both directions.

Array Ports are hard set to 16Gb

The ISL Switch Ports are hard set to 4Gb

There are four ISLs trunked together. So Four configuration of: <-> Site X Brocade Switch Port <-> Site X DWDM Port <-> (70 KMs Dark Fibre) <-> Site YDWDM Port <-> Site Y Brocade Switch Port <->

 

My understanding is that a er_tx_c3_timeout or C3TXTO error will increase on the transmission port due to frames being sent by this port to a target port and no acknowledgment being received from the target port to this sending port before the EHT (220ms in out case) expires.

 

 

In a full duplex world how can I determine which direction I should investigate as both directions have the potential to be upstream or downstream? In my case Site Y on port 2 so one of the four available ISL E ports is consistently seeing C3TXTO errors.

 

I would like to understand which direction I troubleshoot in as this port could be sending frames to the Site Y Array Port and also could be sending frames to the Site X Array Port.

 

How would you troubleshoot this issue?

Contributor
Posts: 63
Registered: ‎01-19-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Tx means sending the frame from the switch port to somewhere outside.
Contributor
Posts: 63
Registered: ‎01-19-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

[ Edited ]

BTW, your topology is incomplete ;) See the difference:

Site X Array Port <-> Site X Brocade Switch Port ARRRAY FACING <-> Site X Brocade Switch Port DWDM FACING <-> Site X DWDM Port SWITCH FACING <-> Site X DWDM Port DARKFIBRE FACING <-> (70 KMs Dark Fibre) <-> Site Y DWDM Port DARKFIBRE FACING <-> Site Y DWDM Port SWITCH FACING <-> Site Y Brocade Switch Port DWDM FACING <-> Site Y Brocade Switch Port ARRAY FACING <-> Site Y Array Port

jkm
Occasional Contributor
Posts: 8
Registered: ‎04-29-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Thanks Alexy,

 

I understand that TX means that the frame will be sent from the switch port, but in what direction? 

Will a write always be a TX from a Init to Target?

Will a read always refer to a RX from Target to Init?

Contributor
Posts: 63
Registered: ‎01-19-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Tx means sending the frame _away_ from the switch!

So if "port A" says "I'm doing Tx" it means that the frame is going to whatever device is connected to that "port A". This device could be a host, a storage or another switch (maybe hidden behind your DWDM kit as well)

Regarding the reads and writes, the question is a bit more difficult.

Let's imagine we have a very simple single-switch setup: host is attached to port A and storage is attached to port B.

Imagine that host wants to read some data.

1) First of all, the host will need to tell storage what it wants. This will be a short frame that will be seen as Rx on the port A and then Tx on the port B.

2) Then there will be the main read data transfer from storage to host. It will be a series of one or more frames that will be seen as Rx on the port B and Tx on port A.

As you can see, even in a simple environment and during the simple Read command we can see both Tx and Rx on different ports during different phases.

Write command is even more complex...

But the idea is that the main data transfer phase will be seen as:
- Tx on the host-facing ports and Rx on the storage-facing ports for the Read commands;
- Rx on the host-facing ports and Tx on the storage-facing ports for the Write commands.
jkm
Occasional Contributor
Posts: 8
Registered: ‎04-29-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Thanks Alexey,

 

So if Tx means sending the frame away from the switch could this mean in the example provided above that a C3TXTO error on a port could be going in the direction of the storage array attached to the local switch or in the direction of the E-Port that connects to the DWDM circuit?

If so is there anyway to determin the direction of the traffic from which the C3TXTO are ocurring? Understanding the direction allows me to concentrate the troubleshooting time in a particular direction.

Contributor
Posts: 63
Registered: ‎01-19-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Sorry, I don't understand your example above because you didn't provide the full topology and it's not clear therefore what do you mean by "Site Y port 2".

You cannot connect more than one device to a particular switch port! (Well unless you use FL-port but it's not supported for years) This means that you don't have many Tx directions on a port - you only have one Tx direction on a port. It is a direction to something that is connected to that port.

So, you say you have a port (Site Y port 2 - whatever you mean by that) which reports C3TXTO. It means that there are problems sending the frames from that port (Site Y port 2) to a device attached to that same port (Site Y port 2). There could be many reasons for that, from bad SFP/cabling (you'd need to check other counters on both sides as well as SFPs Rx/Tx power etc...) and up to the logical issues on some of the receiving devices located somewhere upstream - what we used to call "slow drain" (again, you need to check other counters across your fabric etc..)
jkm
Occasional Contributor
Posts: 8
Registered: ‎04-29-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

[ Edited ]

Thanks Alexey and this is starting to become clearer. 

 

In regards to your previous comment:

 

1) First of all, the host will need to tell storage what it wants. This will be a short frame that will be seen as Rx on the port A and then Tx on the port B.

 

I alway thought this was the other way around and I believe this is why I am confused. I always thought when a Host Port A sent a frame it would increment the Tx and when it recieved a frame it would increment the Rx. Which appears to be the other way around given your example.

 

The fixes you applied to my topology above are correct. 

 

Site X Array Port <-> Site X Brocade Switch Port ARRRAY FACING <-> Site X Brocade Switch Port DWDM FACING <-> Site X DWDM Port SWITCH FACING <-> Site X DWDM Port DARKFIBRE FACING <-> (70 KMs Dark Fibre) <-> Site Y DWDM Port DARKFIBRE FACING <-> Site Y DWDM Port SWITCH FACING <-> Site Y Brocade Switch Port DWDM FACING <-> Site Y Brocade Switch Port ARRAY FACING <-> Site Y Array Port

 

To answer your question Side Y port 2 refers to an ISL port so:  Site Y Brocade Switch Port DWDM FACING

 

My understanding is that port Site Y Brocade Switch Port DWDM FACING could be sending frames (tx) to the Site Y DWDM Port SWITCH FACING or the Site Y Brocade Switch Port ARRAY FACING. 

Is there a way to confirm which direction is cause of the C3TXTO?

 

From the above reply it seems that either direction could be the culprit and I would need to check the errors on both ports in the above scenario. When relating to an E Port or ISL there could be many Array and Host ports that could be utilising this E-Port. Is there any way to narrow down this search besides checking all the ports that recieve traffic from this port?

jkm
Occasional Contributor
Posts: 8
Registered: ‎04-29-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

Thanks Alexey this is starting to become a little clearer, and the updated topology you provided is correct.


Site X Array Port <-> Site X Brocade Switch Port ARRRAY FACING <-> Site X Brocade Switch Port DWDM FACING <-> Site X DWDM Port SWITCH FACING <-> Site X DWDM Port DARKFIBRE FACING <-> (70 KMs Dark Fibre) <-> Site Y DWDM Port DARKFIBRE FACING <-> Site Y DWDM Port SWITCH FACING <-> Site Y Brocade Switch Port DWDM FACING <-> Site Y Brocade Switch Port ARRAY FACING <-> Site Y Array Port

 

From your example prviously;

Let's imagine we have a very simple single-switch setup: host is attached to port A and storage is attached to port B.

 

1) First of all, the host will need to tell storage what it wants. This will be a short frame that will be seen as Rx on the port A and then Tx on the port B.

 

I have always thought this was the opposite. As port A is sending a frame then it would appear as a tx on port A and an Rx on Port B as the storage port is recieving the frame. This is likely why I have been confused in the past.

 

In my example Port 2 Site Y refers to Site Y Brocade Switch Port DWDM FACING. As this port is an E Port this port also connects to multiple ports of type Site Y Brocade Switch Port ARRAY FACING and also the  Site Y DWDM Port SWITCH FACING.

 

My understanding is that the E Port (Site Y Brocade Switch Port DWDM FACING) could be sending frames to any of these ports. If this port is incrementing C3TXTO errors does this mean either dirction could be the issue?

Is there a simple way to determin which direction is having the problem? Is it one of the devices local devices connected to the switch such as Site Y Brocade Switch Port ARRAY FACING or is the issue in the direction of the Site Y DWDM Port SWITCH FACING.

 

Understanding which direction is generating the C3TXTO errors will  improve my ability to troubleshoot these issues. Thanks again

Contributor
Posts: 63
Registered: ‎01-19-2018

Re: In a Full Duplex World how do you know which direction a frame is travelling?

> I have always thought this was the opposite.

 

Well, here is what happens when host sends the single frame to the storage:

 

     1)Tx             2)Rx              3)Tx             4)Rx

Host HBA | --------> | Port A Switch Port B | --------> | Storage

 

(note that this is really a single frame, not the entire SCSI exchange that needs multiple frames to be sent in both directions)

 

1) The host HBA is naturally performing a Tx exactly like you expected. And I'm sure there is a host HBA utility that will let you know the HBA stats where you will see that this really was a Tx.

2) But when this frame arrives to the switch Port A, the switch port A receives (Rx) this frame, doesn't it?

3) Then the process continues with Tx of that frame from the switch Port B.

4) Finally the frame is Rx-ed by the storage port.

 

Then, when you say "I can see this and that stats on my Brocade switch, and I can see some Tx timeout stats on my Brocade switch" - you are talking about the switch side stats, right? You are not talking about the host HBA stats and about the storage array internal stats, right? That's why we are normally discussing only the red-colored part of the picture above.

 

In my example Port 2 Site Y refers to Site Y Brocade Switch Port DWDM FACING. 

 

Great, the picture becomes a little clearer to me as well.

 

As this port is an E Port this port also connects to multiple ports of type Site Y Brocade Switch Port ARRAY FACING and also the Site Y DWDM Port SWITCH FACING.

 

No, that's wrong! The only port where this port is connected is "Site Y DWDM Port SWITCH FACING". Think of it twice - a connection is literally the optical cable, how can you connect the DWDM port and also some storage ports to the same switch port? No way...

 

And therefore your C3TXTO problem is caused by the frames sent from "Site Y Brocade Switch Port DWDM FACING" to "Site Y DWDM Port SWITCH FACING". What is the cause of these timeouts is a different question. As already said, it could be caused by many different reasons starting from bad local optical components (SFP on the switch side, cable link to the DWDM including the patch pannel components if you have some of them, and up to the SFP in the DWDM box), then some problems inside or between your DWDM kits, and then some physical/logical problems on the receiving switches and/or devices on the opposite site. The further you go, the more complex the troubleshooting will be. I'm not sure I can help you in terms of this forum discussion, you'd better call for support from your SAN equipment vendor(s).

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.