Application Delivery (ADX)

Reply
New Contributor
Posts: 3
Registered: ‎09-15-2011

[ServerIron] Why SYN-ACK not sent to client (sometimes) ?

Hello all,

We have problems with old ServerIron 4G-SSL. Yes, I know, it is OLD. But maybe ADX's debug commands will be available on our v11.00.00 4G-SSL, for those who know nothing about 4G-SSLs.

Sometimes, the SLB does not forward SYN-ACKs from Real Server to Client...

What I would like to understand is... WHY ?

Could anyone help us with commands ? the problem may not happen for days, so oneshot-debug-on-the-screen-and-test won't be enough.

We mirrored the port of the switch to which the SLB is connected, so we don't want to know how to trace packets, that is not the question.

Below are the tcpdumps on the switch's mirrored port, and traffic seen by the Client. It is clear that SLB does not forward reply from Real at 06:57:33.620006 (otherwise we would see SYN-ACK from VIP to Client), that is why Client sends the SYN again (3 seconds after, TCP standard) at 06:57:36.619042.

Client and Real are on different subnets, SLB has an IP in both subnets, SLB is Real's default gateway.

Traffic seen by SLB (switch's mirrored port and tcpdumped on another server) :

06:57:33.619950 IP client.60953 > VIP-on-slb.80: Flags , seq 1265823460, win 5840, options , length 0
06:57:33.619957 IP client.60953 > Real-Srv.80: Flags , seq 1265823460, win 5840, options , length 0
06:57:33.620006 IP Real-Srv.80 > client.60953: Flags , seq 1584206080, ack 1265823461, win 5792, options , length 0
06:57:36.619018 IP client.60953 > VIP-on-slb.80: Flags , seq 1265823460, win 5840, options , length 0
06:57:36.619042 IP client.60953 > Real-Srv.80: Flags , seq 1265823460, win 5840, options , length 0
06:57:36.619098 IP Real-Srv.80 > client.60953: Flags , seq 1584206080, ack 1265823461, win 5792, options , length 0
06:57:36.619194 ARP, Reply client is-at 00:26:b9:fb:40:26, length 46
06:57:36.620213 IP Real-Srv.80 > client.60953: Flags , seq 1584206080, ack 1265823461, win 5792, options , length 0
06:57:38.674894 IP client.60954 > VIP-on-slb.80: Flags , seq 1268218316, win 5840, options , length 0
(...)
06:57:42.620100 IP Real-Srv.80 > client.60953: Flags , seq 1584206080, ack 1265823461, win 5792, options , length 0
06:57:42.620134 IP VIP-on-slb.80 > client.60953: Flags , seq 1584206080, ack 1265823461, win 5792, options , length 0
06:57:42.620289 IP client.60953 > VIP-on-slb.80: Flags , seq 1265823461, win 0, length 0
06:57:42.620296 IP client.60953 > Real-Srv.80: Flags , seq 1265823461, win 0, length 0

tcpdump on client :

06:57:33.619807 IP client.60953 > VIP-on-slb.80: S 1265823460:1265823460(0) win 5840 <mss 1460,sackOK,timestamp 3370005693 0,nop,wscale 7>
06:57:36.618945 IP client.60953 > VIP-on-slb.80: S 1265823460:1265823460(0) win 5840 <mss 1460,sackOK,timestamp 3370008693 0,nop,wscale 7>
06:57:42.620138 IP VIP-on-slb.80 > client.60953: S 1584206080:1584206080(0) ack 1265823461 win 5792 <mss 1460,sackOK,timestamp 392072228 3370005693,nop,wscale 7>
06:57:42.620147 IP client.60953 > VIP-on-slb.80: R 1265823461:1265823461(0) win 0

If anyone had an idea on how to know "why" (not "if") the SLB does not send this SYN-ACK, that would be great.

Regards,

Samuel

Contributor
Posts: 47
Registered: ‎07-14-2010

Re: [ServerIron] Why SYN-ACK not sent to client (sometimes) ?

Samuel,

SYN/ACK lost issue happens when we lose arp entry in ServerIron. In such case, I configure single line to prevent arp table from being aged-out.

Specific configuration is just following:

server real gw-1 ip-of-default-gw

By doing this, we health check defaut gateway so that arp age will be refreshed periodically. Arp table that I'm talking about is BP's arp table. You move to BP console by "rconsole 1 1" to get there and "rconsole-exit" to get out of the mode. On BP, you can issue "show arp" command.

Please get back to me if above command works on your environment.

In ADX, we will arp to the configured default gateway automatically before it completely age out. Hence we don't need to do above workaround.

Lastly, 11.0.00 is not recommended. please refer to following document. The latest patch is 10.2.01w as of today. I recommend you to use this patch.

http://www.brocade.com/downloads/documents/miscellaneous/supported-software-ma.pdf

Thanks.

//Kono

New Contributor
Posts: 3
Registered: ‎09-15-2011

Re: [ServerIron] Why SYN-ACK not sent to client (sometimes) ?

Kono,

Thank you for your answer but either I don't understand what you mean, or you didn't understand my problem (maybe I was not clear).

I don't understand what to use for "ip-of-default-gw" and which "default gateway" you're talking about.

Our network configuration is like this :

- Client is 10.1.1.131 / 24; it has a route to 10.2.2.0/24 via 10.1.1.143 (=IP of SLB)

- Real is 10.2.2.3/24, its default gateway is 10.2.2.1 (=IP of SLB)

- VIP (server virtual) is 10.1.1.160, it corresponds to Real 10.2.2.11.

What we seee is :

10.1.1.131 -> 10.1.1.160 SYN

10.1.1.131 -> 10.2.2.3   SYN (SLB does the job, forwards to Real)

10.2.2.3  -> 10.1.1.131  SYN-ACK  (Real replies)

But the SLB does not do the other part of the job, that is :

10.1.1.160 -> 10.1.1.131 SYN-ACK, right after the reply from the real.

What makes me think that it is not an ARP problem is the following case; within 6 seconds, a connection is fine while the other not :

Port 60953 hangs, port 60954 succeeds, within 6 seconds

06:57:36.619018 IP 10.1.1.131.60953 > 10.1.1.160.80: SYN
06:57:36.619042 IP 10.1.1.131.60953 > 10.2.2.11.80: SYN
06:57:36.619098 IP 10.2.2.11.80 > 10.1.1.131.60953: SYN-ACK
06:57:36.619194 ARP, Reply 10.1.1.131 is-at 00:26:b9:fb:40:26   <<=== SLB knows Client's MAC but does not send SYN-ACK
06:57:36.620213 IP 10.2.2.11.80 > 10.1.1.131.60953: SYN-ACK (reemission of 06:57:36.619098)
(...other traffic...)
(new connection)
06:57:38.674894 IP 10.1.1.131.60954 > 10.1.1.160.80: SYN
06:57:38.674921 IP 10.1.1.131.60954 > 10.2.2.3.80: SYN
06:57:38.675080 IP 10.2.2.3.80 > 10.1.1.131.60954: SYN-ACK
06:57:38.675087 IP 10.1.1.160.80 > 10.1.1.131.60954: SYN-ACK
06:57:38.675111 IP 10.1.1.131.60954 > 10.1.1.160.80: ACK
06:57:38.675156 IP 10.1.1.131.60954 > 10.1.1.160.80: ACK
06:57:38.675161 IP 10.1.1.131.60954 > 10.2.2.3.80: ACK
06:57:38.675172 IP 10.1.1.131.60954 > 10.2.2.3.80: PUSH
06:57:38.675336 IP 10.2.2.3.80 > 10.1.1.131.60954: ACK
06:57:38.675343 IP 10.1.1.160.80 > 10.1.1.131.60954:  ACK
(...traffic goes on...)
And then TCP session is closed (cleanly)
06:57:38.944515 IP 10.1.1.131.60954 > 10.1.1.160.80: FIN-ACK
06:57:38.944532 IP 10.1.1.131.60954 > 10.2.2.3.80: FIN-ACK
06:57:38.944585 IP 10.2.2.3.80 > 10.1.1.131.60954: FIN-ACK
06:57:38.944618 IP 10.1.1.160.80 > 10.1.1.131.60954: FIN-ACK
06:57:38.944657 IP 10.1.1.131.60954 > 10.1.1.160.80: ACK
06:57:38.944676 IP 10.1.1.131.60954 > 10.2.2.3.80: ACK

But then 6 secs later, Real asks for news, and gets rejected ('coz of 5 secs timeout on php script)

06:57:42.620100 IP 10.2.2.11.80 > 10.1.1.131.60953: SYN-ACK (reemission of 06:57:36.619098)
06:57:42.620134 IP 10.1.1.160.80 > 10.1.1.131.60953: SLB forwards, finaly
06:57:42.620289 IP 10.1.1.131.60953 > 10.1.1.160.80: but Client has already ended this session


I will set ARP static entries from the 4 Clients en the SLB, though, but I don't think it will help.I will trace ARP requests in dumps from SLB, we didn't in previous captures.

Regards, and thanks again for spending any time on this.

Samuel

Contributor
Posts: 47
Registered: ‎07-14-2010

Re: [ServerIron] Why SYN-ACK not sent to client (sometimes) ?

Samuel,

I see your client is not over the default gateway, then configuration should look like below.

server real gw-1 10.1.1.131

This configuration is quite harmless, so I recommend you to just configure it and see what will happen instead of configuring static arp.

When you have four client, just difine four real server definition.

In short, I imagine your SYN/ACK lost issue is caused by the absence of arp entry in BP CPU.

SI-4G has multi cpu architecture. MP CPU takes care of Health checking, etc while BP CPU takes care of L4-7 processing.

MP do arp and its entries will be copied to BP via IPC periodically, hence BP will not directly arp for the gateway nor for the client.

For some reasons, when arp on BP is lost or aged-out, then BP cannot send SYN/ACK back to client.

Hope this helps.

Thanks.

//Kono

New Contributor
Posts: 3
Registered: ‎09-15-2011

Re: [ServerIron] Why SYN-ACK not sent to client (sometimes) ?

Hi Kono,
Thank you sooooo much for this hint !!!
I did declare clients as reals, and we've had no problem for two weeks. Looking at previous alerts, I think the bug would have already reappeared.

If I were the boss, I would send you money, but I'm not.
Best Regards,
Samuel

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.