Application Delivery (ADX)

Reply
Occasional Contributor
Posts: 5
Registered: ‎04-20-2013

adx - health checks not working

Hi.

I have an ADX 1000 switch, and I'm trying to configure SLB on it. So far I've configured two backends, and I've set one backend to constantly return 502 on a test site.

My problem is - seems like these health checks aren't working.

ADX config piece:

server real backend0 xxx.xxx.xxx.167

port http

port http keepalive

port http url "GET /"

port http status-code  200 201 300 302

!

server real pitchblack xxx.xxx.xxx.169

port http

port http keepalive

port http url "GET /"

port http status-code  200 201 300 302

!

!

server virtual test.server.tld xxx.xxx.xxx.172

port http

bind http backend0 http pitchblack http

!

Nginx log shows that nginx returns 502 on a configured health check:

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

xxx.xxx.xxx.170 - - "GET / HTTP/1.0" 502 172 "-" "-"

But the adx behaves weirdly. Below are some output for "sh server real" and "sh server". Weird thing is: while one of my nodes constantly returns 502, it's port state can be both ACT and FAL. From my point of view - it should be FAL all the time. Second part of questions:

- why this artificially failed node server state in "sh server" is "Active" ? shouldn't it be "Failed" ?

- furthermore, sometimes it is "Failed" indeed, but why it's becoming "Active" ? why the traffic is forwarded to this node even when it's in failed state  (I can still see 502 from nginx in my browser) ?

I also posted the software versions from this switch. May be it's the software ? Should I update it ?

Thanks.

telnet@adx1#sh server real

Real Servers Info

========================

State - ACT:active, ENB:enabled, FAL:failed, TST:test, SUS:suspect,

        GDN:grace-dn, DIS:disabled, UNK:unknown, UNB:unbind,

        AWU:await-unbind, AWD: await-shutdown

Name: backend0               State: Active              IP:xxx.xxx.xxx.167:   1

Mac: 984b.e169.18a8          Weight: 1/1              MaxConn: 2000000

SrcNAT: not-cfg, op          DstNAT: not-cfg, not-op    Serv-Rsts: 0

Port    St  Ms CurConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas

----    --  -- ------- -------    -------   -------   --------   --------   ----

default UNB 0  0       0          0         0         0          0          0

http    FAL 0  0       197        1074      1061      441729     146236     0

Server  Total  0       197        1074      1061      441729     146236     0

Name: pitchblack             State: Active              IP:xxx.xxx.xxx.169:   1

Mac: 984b.e169.4738          Weight: 1/1              MaxConn: 2000000

SrcNAT: not-cfg, op          DstNAT: not-cfg, not-op    Serv-Rsts: 0

Port    St  Ms CurConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas

----    --  -- ------- -------    -------   -------   --------   --------   ----

default UNB 0  0       0          0         0         0          0          0

http    ACT 0  0       195        1593      1320      1254145    163149     0

Server  Total  0       195        1593      1320      1254145    163149     0

telnet@adx1#sh server

Server Backup port not configured

Server Load Balancing - global parameters

Predictor =          least-conn

Force-deletion =     0

Reassign-threshold = 20

Reassign-limit =     3

TCP-age  =           30

UDP-age  =           5

Sticky-age  =        5

TCP-syn-limit =      65535

msl =                8

TCP-total conn =     392

Unsuccessful conn =  54

ICMP-message for UDP = Disabled

ICMP-message for TCP = Disabled

RESET-message = Enabled

NO-RESET-on-max-conn = Disabled

Port Holddown =      0

Port Holddown Timeout = 120

Ping-interval =      2

Ping-retries  =      4

Session ID age =    30

Bind info

Virtual server: test.server.tld           Status: enabled  IP: xxx.xxx.xxx.172

        http -------> backend0: xxx.xxx.xxx.167,  http (Active)

                      pitchblack: xxx.xxx.xxx.169,  http (Active)

Client->Server       =         85  Server->Client       =          0

Drops                =          0  Aged                 =         43

Fw_drops             =          0  Rev_drops            =          0

FIN_or_RST           =          0  old-conn             =          0

Disable_drop         =          0  Exceed_drop          =          0

Stale_drop           =          0  Unsuccessful         =         54

SYN def/proxy RST    =          0  Server Resets        =          0

Out of Memory        =          0  Out of Memory        =          0

last conn rate       =          0  max conn rate        =          6

last TCP attack rate =          0  max TCP attack rate  =          0

fast vport found     =          4  fast vport n found   =         42

Fwd to non-static FI =          0  Dup stale SYN        =          0

TCP forward FIN      =          0  TCP reverse FIN      =          0

Fast path FWD FIN    =          0  Fast path REV FIN    =          0

Fast path SLB SYN    =          0  Dup SYN after FIN    =          0

Duplicate SYN        =          0  Duplicate sessions   =          0

TCP ttl FIN recvd    =          0  TCP ttl reset recvd  =          0

Sessions in DEL_Q    =          0  Sess force deleted   =          0

Fwd sess not found   =          0  sess already in delQ =          0

Sess rmvd from delQ  =          0

New sess sync sent   =          0  New sess sync recvd  =          0

TCP SYN received     =          0  TCP SYN dropped      =          0

TCP SYN to MP        =          0  TCP SYN ACK to MP    =          0

TCP SYN ACK received =          0  TCP SYN ACK dropped  =          0

TCP pkt received     =          0  TCP pkt dropped      =          0

TCP pkt to MP        =          0

Dropped VIP pings    =          0

Avail. Sessions on MP       =     999892 Total Sessions on MP    =    1000000

bp-1 Avail.  Session =   1999996 Total Sessions =   2000000

bp-2 Avail.  Session =   1999996 Total Sessions =   2000000

bp-3 Avail.  Session =   1999996 Total Sessions =   2000000

bp-4 Avail.  Session =   1999996 Total Sessions =   2000000

Total C->S Conn      =        392  Total S->C Conn      =          0

Total Reassign       =          0  Unsuccessful Conn    =         54

Server State - 0: disabled, 1:enabled, 2:failed, 3:test, 4:suspect, 5:grace_dn, 6:active

Real Server        State   CurrConn    TotConn TotRevConn   CurrSess   PeakConn

backend0            6          0        197          0          0         16

pitchblack          6          0        195          0          0         15

last conn rate       =          0  max conn rate        =          6

last TCP attack rate =          0  max TCP attack rate  =          0

SYN def RST          =          0  SYN flood            =          0

telnet@adx1#sh ver

Copyright (c) 1996-2009 Brocade Communications Systems, Inc.

Boot Version 12.1.00aT405 Jul  9 2010 19:03:54 PDT label: dob12100a

Monitor Version 12.1.00aT405 Jul  9 2010 19:03:54 PDT label: dob12100a

System Version 12.2.01eT403 Apr 28 2011 20:51:30 PDT label: ASR12201e

AXP Version: 1.13 Dated: 2010/02/09 07:35:49

PAX Version: 8.4 Dated: 2010/10/06 14:31:02

MBRIDGE Version: 000b, Device ID # bebe

==========================================================================

Type:   ServerIron ADX 1216-4-PREM

      Backplane Serial #:   E20329H03M

      Chassis Serial #:   E32531H12D

      LID #: gIHKIGjGHf  License: SI-1216-4-SSL-PREM

      Part #:   35723-003

      SSL Card - Serial #: 1F0328H070       Part #: 40-1000373-02

      Version #: 13b626-05050505-111d8036-00

==========================================================================

Active management module:

       1499 MHz Power PC processor (SVR 2.2, PVR 8021/0030) 599 MHz bus

        512 KB Boot flash

      131072 KB Code flash

       2048 MB DRAM

The system uptime is 82 days 8 hours 43 minutes 41 seconds

The system started at 21:06:45, GMT+00, Mon Jan 28 2013

The system - boot source: primary, mode: cold start,   soft reset, total resets:0

Occasional Contributor
Posts: 5
Registered: ‎04-20-2013

Re: adx - health checks not working

This has become even more weird after I had a look in adx log:

Apr 21 06:21:45:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:43:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:40:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:38:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:35:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:33:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:30:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:28:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:25:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:23:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:20:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:18:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:15:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:13:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:10:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Apr 21 06:21:08:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is down due to healthcheck

Apr 21 06:21:05:N:L4 server xxx.xxx.xxx.167 backend0 port 80 is up

Why it's keeping to come up getting only 502 responses ?

Occasional Contributor
Posts: 5
Registered: ‎04-20-2013

Re: adx - health checks not working

"no-fast-bringup" did the trick.

Shame on Brocade, this is not covered in docs.

Plus, I find that logic "disable l4 checks and I will disable l7 checks" when using "no server l4-checks" is just wrong.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook