06-22-2016 03:11 AM
We have spent the last couple of weeks trying to resolve a problem where a HTTP POST to our server from another server was occasionally missing its body (causing an error in our application). The issue seems to be with our ADX 1000 load balancer - if we bypass the LB then the problem no longer occurs.
The LB is configured for SSL termination, firmware version 12.5.02eT403. We get many GB of traffic per day, this seems to be the only problem.
We have this WireShark trace from one of our real servers that shows the issue:
You can see that we get the HTTP headers but the final packet (with the body) is missing. It also seems a bit odd that the LB has split the data into so many tiny TCP segments (most of them have a payload of only 15 bytes).
We see this with the successful traffic as well - the headers are split into 13 small frames, with the body coming as a single frame at the end. Here is an example:
Does anyone have any idea of what the issue might be? RackSpace manage the LB for us, so far they have not been able to diagnose the problem. I'm hoping someone in this community might have some insight!
06-22-2016 03:51 AM - edited 06-22-2016 03:55 AM
here is a DOC about SSL termination. Did you already tried to follow this procedure ?
06-22-2016 04:41 AM
Yes, we have followed this (or rather Rackspace have followed it for us).
Our ADX SSL termination config has been working for several years and deals with millions of hits per month. It only seems to be traffic from this client that causes this problem, only in the last few months, and then only intermittently. We have 9 real servers, 107 SSL profiles and 75 virtual servers.
06-24-2016 07:40 AM
I'm very interested in this as well.
We're currently running 12.5.02bT403. We're running 10 real servers & get about 36m requests per day. About 29K/day are reported as "400" errors. I believe some of these are due to no request body - but I've not done wireshark captures yet.
Please post any further updates.
08-10-2016 02:07 AM
We are two months in now, and Brocade still don't seem to have any idea what the problem is. I will stay away from Brocade products in the future.
08-10-2016 02:28 AM
-->>We are two months in now, and Brocade still don't seem to have any idea what the problem is.
you wrote in preview post:
->It looks like a software issue on the ADX. Brocade are investigating...
did you opened a TAC with brocade ?
looking in Brocade KB, and found several Article related to ADX and SSL termination, however i'm not sure what exact you problem.
08-10-2016 03:13 AM - edited 08-10-2016 03:14 AM
Rackspace have the relationship with Brocade, they will have opened the TAC (I don't have a Brocade reference unfortunately).
It's Rackspace ticket 160607-03565, they say they opened the TAC with Brocade on Jun 28, 2016. The Rackspace employee who opened the case is Miroslaw Pabian. Maybe you can find the TAC from that?