Application Delivery (ADX)

Reply
den
Occasional Contributor
Posts: 9
Registered: ‎06-15-2009

ServerIron / Cisco ASA compatibility issue

All,

I'm facing peculiar problem with ServerIrom GT-EGC16 server load balancing feature. I have 5 real servers, bound to one virtual server, HTTP and HTTPS (sticky) ports. All real servers are physically located in one rack, connected to core router by single physical ethernet link. I have 2 IP subnets, routed to the rack (one is for virtual server IP and real server IPs, another is for rack equipment management). One IP from every subnet is configured on core router interface (as main IP and alias IP), these IPs are used as gateway IPs by respective devices.

Everything was working just fine before I installed Cisco ASA firewall appliance in front of the ServerIron. ASA was configured in "routed" mode with permissive ACLs. In other words gateway IPs from both subnets are on the ASA, both IP subnets are statically routed to ASA from the core, ServerIron is connected to internal ASA port, and real servers are connected to ServerIron.

The problem is that Cisco ASA devices do not support IP aliases (traditional PIX/ASA limitation, as you probably know). Interface can be configured with one IP, if you want to set up additional IPs, they have to reside in separate vLANs. There's well known workaround for this limitation, if you don't want to set up vLANs for some reason (which is my case). It's possible to make Cisco interface respond to ARP queries for additional IPs, you just need to configure static ARP entries for "alias" IPs, enable proxy-arp feature and route "alias" subnet statically to main interface IP. Such "ghost" IP aliases work fine as gateway IPs, but obviously do not respond to any requests (like ICMP echo) as they do not really exist.

Virtual IP / real server IP subnet was configured via "ghost IP alias" in my setup. All infrastructure behind the ASA was accessible from Internet (real server IPs, virtual server IP, switch management IPs from 2nd subnet, everything). I was able to connect to HTTP port on any physical server just fine. However ServerIron itself was unable to ping or HTTP any of real servers (obviously they failed healthchecks, were marked as "failed", and virtual server IP was simply refusing HTTP connections). ARP entries for real servers were present on the ServerIron and associated with their respective ports, so, theoretically, real servers used to be accessible from the ServerIron regardless of ASA device presence. ARP cache was cleared on all devices (ServerIron, ASA, core router) with no result, new cache entries were identical to old ones. The problem just disappeared when I removed ASA firewall from the wire and connected ServerIron directly to the core, as it was before.

This definitely has something to do with how exactly ServerIron load balancing works on low level, so my questions are:

1. Can anybody give a big picture of server load balancing network level internals (or any other clue regarding that situation)?

2. Would it help with the above issue if "server subnet" gateway IP will be configured as real interface IP?

3. Would it help with the above issue if real servers will be configured as remote servers (my guess is that all ethernet level peculiarities will be bypassed this way)?

Thank you for your time and for your answers.

Super Contributor
Posts: 316
Registered: ‎05-01-2009

Re: ServerIron / Cisco ASA compatibility issue

Hi Den,

I had a look at your question. Pretty long post and I am not sure whether I got everything or not. You have mentioned that you have seen a health check issue and all servers are marked as failed - is this correct? The status of all reals is FAILED looking at the output of "show server bind"? The ServerIrons default way to do health checks for HTTP and for HTTPS is the following:

1st: try to ARP to get the real server MAC address

2nd: try to send an ICMP echo request to the real server IP (PING)

3rd: L4 health check (TCP 3-way handshake to port 80 or 443)

4th: L7 health check (HTTP GET talking about HTTP and SSL handshake talking about HTTPS)

The ServerIron is going to do this step by step. Step n is starting as soon as step (n-1) was successful.

You have mentioned as well that you are NOT able to ping the real server IPs from the ServerIron and this implies that step 2 is not going to be successful. Please try to ping the real servers another time from the ServerIron and see if that is possible with the ASA in place. If that is not the case I would suggest to disable health checking step 2 using the command "no-l3-check":

server real SERVER1

  no-l3-check


server real SERVER2

  no-l3-check



server real SERVER5

  no-l3-check

Doing this the ServerIron is moving to step 3 right after the first health checking step. I am pretty sure that it is going to work afterwards - the L4 check should be successful even with the ASA in between the ServerIron and the real servers. Let me know if it is working.

FYI: defining real servers as remote servers is going to skip step 1 of the health checking process but the ARP seems to be successful. I assume the ServerIron itself does have an IP address in the real server subnet as well (otherwise it would not be able to get the MAC addresses of the real servers).

den
Occasional Contributor
Posts: 9
Registered: ‎06-15-2009

Re: ServerIron / Cisco ASA compatibility issue

Adam,

Thanks for the insight, this is really bright answer. However I'm not sure this is just L3 healthcheck issue, not general connectivity issue. In other words, I'm afraid ICMP is not the only protocol that does not pass through, actual HTTP connections won't succeed as well, but I'm not sure about how to check this. If HTTP is ok, that would do.

Let me clarify my setup. ASA device is to be connected in front of the ServerIron, between core router and ServerIron. Real servers are (and will be) connected to ServerIron directly. Sorry for being unclean. I can quickly draw a diagram, if needed.

So, theoretically ASA presence shouldn't affect L2 and L3 traffic between servers and ServerIron at all, internal traffic just shouldn't be hitting the Cisco. But it actually does, and I don't understand what's going on. Possibly it has something to do with unresponsive subnet gateway IP or so. Perhaps you do have some idea about this.

Unfortunately the system is on production, and I can't test suggested L3 healthcheck solution right now, I need to schedule maintenance in advance. But I'll definitely test it and let you know.

Thank you for your time and effort.

Super Contributor
Posts: 316
Registered: ‎05-01-2009

Re: ServerIron / Cisco ASA compatibility issue

Hi Den,

it is around 03:00am here right now so please excuse my bad english. Looking at your last post you have mentioned that the ASA is in between the Core and the ServerIron and the real servers are directly connected to the ServerIron which would look like:

Core --- ASA --- ServerIron --- real Servers

I am not quite sure about the IP addresses but you have mentioned that there is a real server subnet and a VIP subnet and the ASA is acting as proxy ARP device for the real server IPs and the virtual servers. I am pretty sure that there is not any problem to reach the virtual server from the outside world but the problem seems to be the ServerIron to real server connectivity as you have mentioned. The real servers are directly connected to the ServerIron but you are seeing traffic from the ServerIron to the real server hitting the ASA.

Looking at the ARP entries at the SI - what do you see? Are they actually pointing to the real server facing port or to the ASA facing port?

One other problem might occur as soon as you are using L2 code at the ServerIron - ALL the ServerIron ARP requests will be send out via ALL ports of the ServerIron using L2 code by default. This could result in the fact that the ASA is going to answer first to an ARP request and the SIs ARP entry would point to the ASA instead of the real server. You should have seen this doing a "show arp".

The solution for this would be to use the ip-subnet command below the VLANs to restrict ARPs for a specific subnet to a given VLAN. Is that possibly you problem?

Do you know how to get traces at the ServerIron itself? If so: this might help as well.

Feel free to post your ServerIron config (with chaged IP addresses) to give me some more details.

den
Occasional Contributor
Posts: 9
Registered: ‎06-15-2009

Re: ServerIron / Cisco ASA compatibility issue

Hi, Adam,

Yes, physical diagram is correct. it has 2 variants indeed, production wihout ASA and experimental with ASA:

Core --- ServerIron --- Real servers and supplementary equipment

or:

Core --- ASA --- ServerIron --- Real servers and supplementary equipment

Your assumption about proxy-arp related effect sounds reasonable, I have a gut feeling we're close to the answer here, but this is not exactly what you say. ASA does not have real server static ARP entries and obviously it must not redistribute dynamic ARP. ARP table entries on the ServerIron were actually pointing to physical real server ports, not to the ASA port.

I actually didn't see traffic, hitting the ASA, but what confuses me is that ASA was involved in intra-rack traffic somehow. At least it broke intra-rack connectivity.

For our example let's assume we use 10.1.1.0/24 as VIP/RIP subnet (VIP is 10.1.1.2). Supplementary equipment (SeverIron itself, APC devices, IP-KVM device etc.) uses 192.168.50.0/24 as management IP subnet. One more subnet is used by ASA itself, for management IP on "external" interface, let's say 192.168.100.0/30, and 192.168.100.2 is ASA primary management IP.

#################################

On the ServerIron:

ver 09.4.00mTD2
!
module 1 bi-0-port-wsm6-management-module
module 2 bi-jc-16-port-gig-copper-module

!
!

healthck web1.url.com tcp
  dest-ip 10.1.1.3
  port http
  protocol http
  protocol http url "GET http://url.com"

healthck web2.url.com tcp
  dest-ip 10.1.1.4
  port http
  protocol http
  protocol http url "GET http://url.com"

healthck web3.url.com tcp
  dest-ip 10.1.1.5
  port http
  protocol http
  protocol http url "GET http://url.com"

healthck web4.url.com tcp
  dest-ip 10.1.1.6
  port http
  protocol http
  protocol http url "GET http://url.com"

healthck web5.url.com tcp
  dest-ip 10.1.1.7
  port http
  protocol http
  protocol http url "GET http://url.com"

!
server no-fast-bringup
server syn-def 12

!
server real web1.url.com 10.1.1.3

port ssl
port http
port http healthck web1.url.com
port http url "GET http://url.com"
!
server real web2.url.com 10.1.1.4

port ssl
port http
port http healthck web2.url.com
port http url "GET http://url.com"
!
server real web3.url.com 10.1.1.5

port ssl
  port http
  port http healthck web3.url.com
  port http url "GET http://url.com"
!
server real web4.url.com 10.1.1.6

port ssl
port http
port http healthck web4.url.com
port http url "GET http://url.com"
!
server real web5.url.com 10.1.1.7

port ssl
port http
port http healthck web5.url.com
port http url "GET http://url.com"
!
!
server virtual url.web 10.1.1.2
port ssl sticky
port http
bind ssl web1.url.com ssl web2.url.com ssl web3.url.com ssl web4.url.com ssl web5.url.com ssl
bind http web1.url.com http web2.url.com http web3.url.com http web4.url.com http web5.url.com http

!

..........................................

!
vlan 1 name DEFAULT-VLAN by port
!
!
enable telnet password .....
enable super-user-password .....
hostname SI-GT
ip tcp syn-proxy 12
ip address 192.168.50.15 255.255.255.0
ip default-gateway 192.168.50.1
ip dns server-address 4.2.2.1 4.2.2.2
username admin password .....

snmp-server community ..... ro
sntp server 66.96.98.9
sntp poll-interval 43200
no web-management front-panel
!

....................................

!
interface ethernet 2/16
port-name Uplink
syn-def

##############################

On the ASA (aaaa.bbbb.cccc is the MAC address of TO-FOUNDRY interface):

...............................

!
interface GigabitEthernet0/0
nameif TO-CORE
security-level 0
ip address 192.168.100.2 255.255.255.252
!
interface GigabitEthernet0/1
nameif TO-FOUNDRY
security-level 100
ip address 192.168.50.1 255.255.255.0

..................................

access-list TO-CORE_access_in extended permit ip any any
access-list TO-CORE_access_out extended permit ip any any
access-list TO-FOUNDRY_access_in extended permit ip any any
access-list TO-FOUNDRY_access_out extended permit ip any any

.................................

access-group TO-CORE_access_in in interface TO-CORE
access-group TO-CORE_access_out out interface TO-CORE
access-group TO-FOUNDRY_access_in in interface TO-FOUNDRY
access-group TO-FOUNDRY_access_out out interface TO-FOUNDRY

.................................

route TO-CORE 0.0.0.0 0.0.0.0 192.168.100.1 1

route TO-FOUNDRY 10.1.1.0 255.255.255.0 192.168.50.1 1

.................................

arp TO-FOUNDRY 10.1.1.1 aaaa.bbbb.cccc alias

.................................

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

###############################

On the Core:

............................

!
interface GigabitEthernet0/2
no switchport
ip address 192.168.100.1 255.255.255.252

.............................

ip route 10.1.1.0 255.255.255.0 192.168.100.2

ip route 192.168.50.0 255.255.255.0 192.168.100.2

Or, without ASA:

!
interface GigabitEthernet0/2
no switchport
ip address 192.168.50.1 255.255.255.0

ip address 10.1.1.1 255.255.255.0 secondary

###############################

Thank you for looking into it.

Super Contributor
Posts: 316
Registered: ‎05-01-2009

Re: ServerIron / Cisco ASA compatibility issue

Den,

the ServerIron configuration is pretty strange. The ServerIron itself does not have an IP address 10.1.1.x/24 subnet. There is a management IP defined via "ip address" and a default gateway and they are both in the 192.168.50.0/24 subnet. There is a virtual server and some real servers in the 10.1.1.x/24 subnet BUT no ServerIron IP address. The only way to reach the real servers from the ServerIrons point of view is to use the default gateway if you look at it from an IP point of view. It is recommended and necessary to define a so called source-ip at the ServerIron in case you are using real servers in a subnet other than the ServerIron management subnet.

Out of the documentation:

Adding a Source IP Address

You can define source IP addresses on a ServerIron system running switch code to place it in multi-netted environment. These source IP addresses could potentially be used as default gateways for real servers. You can also define source NAT IP addresses while using source NAT.

Have you ever tried it adding a source-ip to the ServerIron which is part of the 10.1.1.x/24 subnet? The config as shown below is not supported and it is unfortunately incorrect.

Command: server source-ip 10.1.1.? 255.255.255.0 0.0.0.0

den
Occasional Contributor
Posts: 9
Registered: ‎06-15-2009

Re: ServerIron / Cisco ASA compatibility issue

Adam,

Thanks, looks like this is the problem. I'll test it and let you know.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook