Application Delivery (ADX)

High Availability 101

by on ‎08-02-2009 09:29 AM (340 Views)

Redundancy

Most of the ADC setups today are redundant setups - it does not make a lot of sense to have a single point of failures in front of the real servers. The devices responsible for the traffic distribution needs to be redundant most of the time.


Brocade ServerIrons offer a range of redundancy features including sessions synchronization to ensure a failover is as hitless as possible from a clients point of view.


There are basically three redundancy mechanisms I would like to discuss here:


Hot-Standby

Symmetric Server Load Balancing (SSLB)

Sym-Active Server Load Balancing (Sym-Active SLB)

What is the difference?

Hot-Standby

Hot-Standby is a very simple HA mode. One of the ServerIrons is active and the other one is backup. Hot-Standby is only supported running L2 code (switch code). Hot-standby is not available running router / L3 code.

Symmetric Server Load Balancing

A lot of people call this active-standby or active-backup SLB. The master and backup status is not at the system level - the status is not at the level of the virtual server. Each of the two ServerIrons in an SSLB configuration is able to receive and process SLB traffic. Each ServerIron is responsible for a group of virtual server and it is backup for the virtual server which are active at the other SI. The responsibilities are clearly defined using SSLB - traffic for a given virtual Server is always ending up at the ServerIron being master for this virtual server.

Sym-Active Server Load Balancing

This is real active-active SLB. Both ServerIrons are able to process traffic for the same set of virtual servers. The ServerIron itself is not responsible for the distribution of the traffic. The "network" (upstream router) is able to send traffic for the same virtual server to both ServerIron and both are able to process the traffic.


Detailed explanation

Hot-Standby

Hot-Standby is only available running switch code as mentioned already. One of the ServerIrons is active and the other one is standby. The standby ServerIron is going to act as a dumb device in such a setup. All the backup is actually doing is to send its own health checks, to check whether the master ServerIron is still there and to receive session table updates from the master ServerIron. It is not forwarding any traffic which avoids loops in the network. Hot-Standby is the only HA mechanism which needs a dedicated heartbeat / synchronization port.

The failover itself is based on the availability of the master unit and the amount of active ports at both ServerIrons. The backup is going to take over as soon as the master is not available anymore or as soon as the port count of the master is lower than the port count of the backup switch. Both SIs are going to monitor their amount of upstream and downstream ports all the time.

Session table synchronization is getting done automatically talking about Hot-Standby - there is no need to enable this.

Hot-Standby is redundancy at the box level - each SI is doing all or nothing from an SLB point of view.

The configuration is pretty simple. The major steps are:

  • configure a dedicated sync vlan with at least a single port in
  • disable spanning tree- enable hot-standby
  • configure the upstream ports as so called router-ports

Example setup - network diagram (click to enlarge):

hot-standby.JPG

This would look like the following at ServerIron A:


vlan 99 name SYNC
  untagged ethe 16
  no spanning-tree
vlan 1
  no spanning-tree


server backup ethe 16 001b.ed3c.74c0 vlan-id 99
server router-ports ethernet 1


server real rs107 192.168.100.107

  port http

  port http keepalive

  port http url "GET /"


server real rs108 192.168.100.108

  port http

  port http keepalive

  port http url "GET /"


server real rs109 192.168.100.109

  port http

  port http keepalive

  port http url "GET /"


server virtual vs222 192.168.100.222

  port http

  bind http rs107 http rs108 http rs109 http


ip address 192.168.100.201 255.255.255.0

ip default-gateway 192.168.100.1

The MAC address 001b.ed3c.74c0 is the MAC address of one of the ServerIrons in the hot-standby setup - use show chassis to get the MAC address. The output of "show chassis" could look like the following at an ADX 1000:

ServerIronADX 1000#show chassis
Boot Prom MAC: 001b.ed3c.74c0
=================================
Fan and Power Supply Status
=================================
Fan 1 - STATUS: OK SPEED: MED-HI(RPM 7470)
Fan 2 - STATUS: OK SPEED: MED-HI(RPM 7700)
Fan 3 - STATUS: OK SPEED: MED-HI(RPM 7485)
Fan 4 - STATUS: OK SPEED: MED-HI(RPM 7723)
Fan 5 - STATUS: OK SPEED: MED-HI(RPM 7763)
Fan 6 - STATUS: OK SPEED: MED-HI(RPM 7440)
Power 1: not present
Power 2 (Part#:  3I50    Serial#: cccccccccccc) (AC 504W): Installed (OK)
Total power budget for system = 504 W
===============================
Temperatures per Module
===============================
Management CPU Temp: 43 deg C
BP 1 Temp: 50 deg C
BP 2 Temp: 44 deg C
AXP Temp: 34 deg C
PAX Temp: 29 deg C
Warning level : 85 C degrees, shutdown level : 100 C degrees

The configuration of ServerIron B would look exactly like the one at ServerIron A except the ServerIrons own IP address:

vlan 99 name SYNC
   untagged ethe 16
   no spanning-tree
vlan 1
   no spanning-tree


server backup ethe 16 001b.ed3c.74c0 vlan-id 99
server router-ports ethernet 1


server real rs107 192.168.100.107

  port http

  port http keepalive

  port http url "GET /"


server real rs108 192.168.100.108

  port http

  port http keepalive

  port http url "GET /"


server real rs109 192.168.100.109

  port http

  port http keepalive

  port http url "GET /"


server virtual vs222 192.168.100.222

  port http

  bind http rs107 http rs108 http rs109 http


ip address 192.168.100.202 255.255.255.0

ip default-gateway 192.168.100.1

Hot-standby is a pretty simple HA mechanism. Each ServerIron is going to count its amount of active server and router ports and the SI with the highest amount of ports is going to be the master. Router-ports (client facing ports) are ports specified in the configuration using "server router-ports ethernet x". The ServerIron detects server ports automatically - server port are ports with real servers behind. Health checks replies are getting used to identify server ports. Check the output of "show server backup" on both ServerIron after configuring hot-standby. One of the SIs needs to be the "Active" SI and the other one the "Standby" SI.

It is possible to have a preferred master using Hot-Standby. The preferred master is going to assume the active role after the configured amount of time:


ServerIron(config)#server backup-preference 5
Syntax: server backup-preference <wait-time>


It is possible to have multiple ServerIron Hot-Standby pairs in a single L2 broadcast domain. Each pair needs its own group ID in such a scenario. The group ID identifies the ServerIrons in a Hot-standby pair.


ServerIron(config)#server backup-group 1

Syntax: server backup-group <id>

There are other configuration settings related to hot-standby - please have a look at the documentation:

Hot-Standby

Symmetric SLB (aka SSLB/active-standby SLB)

SSLB is slightly different from hot-standby. SSLB is first of all available in switch and router code. Hot-Standby offers high availibility at the device level - SSLB is HA at the virtual servers level. Each ServerIron is master for a group of virtual servers - some virtual servers are active at ServerIron #1 and some others at ServerIron #2. Each ServerIron is backup for the virtual servers of the other ServerIron. It is therefore possible that both ServerIron are somehow active because both of them are responsible for part of the traffic.

There is no need for a dedicated synchronization port but it is possible to configure one. The recommendation is a untagged direct link in between both SIs. Using a dedicated link for the sync traffic ensures that there is no possibility for the sync traffic to influence production traffic and the another way around.

The failover itself is based on the availability of the master unit per default. It is nevertheless possible to check the amount of applications available per virtual server (amount of service ports with active reals behind) as failover criteria for a virtual server. It is as well possible to link the failover of a virtual server to the failover of a VRRP-E instance talking about router code. VRRP-E instances itself might track ports as failover criteria which allows a flexible HA design.

Session table synchronization is not getting done automatically talking about SSLB - it is necessary to enable it on a per port basis inside the port profiles.

SSLB is redundancy at the virtual server level.

The configuration is pretty simple. The major steps are:

  • configure a optional sync vlan with at least a single port in
  • configure upstream and downstream VLANs / subnets
  • configure SLB with reals, virtual servers and sym-priorities
  • configure VRRP-E (router code only)
  • link VIP to VRRP-E failover using vip-groups (router code only)

A vip-group is not necessary in all cases. Talk to your local SE to see what fits best to your situation.

Example setup - network diagram (click to enlarge):

SSLB.JPG

The setup above is still pretty simple because it is still using ServerIron A as master for ALL virtual server ( 192.168.8.222 (HTTP) and 192.168.8.223 (DNS) ). Both virtual servers are active at ServerIron A. ServerIron A is as well master for the VRRP-E addresses 192.168.8.1 and 192.168.100.1 - one for the upstream subnet and the other one for the real server subnet. It is of course possible to have some VRRP-E addresses being master at SI B as well and you might want to have some virtual server being master at SI B but this is more complex. Keep in mind that you need to stay below 50% utilization at both SIs anyway to be redundant because one system needs to be able to handle the load in case of problems with one of the devices.

This would look like the following at ServerIron A (check the stuff highlighted in red):

server symmetric-port ethernet 16 vlan-id 99

server port 80
  session-sync
  tcp
  tcp keepalive 3 2

server port 53
  udp keepalive 3 2


server real rs107 192.168.100.107
port http
  port http url "HEAD /"
  port dns

server real rs108 192.168.100.108
  port http
  port http url "HEAD /"
  port dns

server real rs109 192.168.100.109
  port http
  port http url "HEAD /"
  port dns


server virtual vs222 192.168.8.222
  sym-priority 109
  port http
  bind http rs107 http rs108 http rs109 http

server virtual vs223 192.168.8.223
  sym-priority 109
  port dns
  bind dns rs107 dns rs108 dns rs109 dns

vlan 1 name DEFAULT-VLAN by port
  router-interface ve 1

vlan 2 name SERVERS by port
  untagged ethe 2
  router-interface ve 2

vlan 99 name SYNC by port
  untagged ethe 16

router vrrp-extended


server vip-group 1
  vip 192.168.8.222
  vip 192.168.8.223

interface ve 1
  ip address 192.168.8.2 255.255.255.0
  ip vrrp-extended vrid 1
   backup priority 109 track-priority 10
   ip-address 192.168.8.1
   vip-group 1
   track-port e 1
   track-port e 2
   enable

interface ve 2
  ip address 192.168.100.2 255.255.255.0
  ip vrrp-extended vrid 2
   backup priority 109 track-priority 10
   ip-address 192.168.100.1
   track-port e 1
   track-port e 2
   enable

The configuration of ServerIron B looks slightly different (check the stuff highlighted in red):

server symmetric-port ethernet 16 vlan-id 99


server port 80
  session-sync
  tcp
  tcp keepalive 3 2


server port 53
  udp keepalive 3 2


server real rs107 192.168.100.107
  port http
  port http url "HEAD /"
  port dns


server real rs108 192.168.100.108
  port http
  port http url "HEAD /"
  port dns


server real rs109 192.168.100.109
  port http
  port http url "HEAD /"
  port dns


server virtual vs222 192.168.8.222
  sym-priority 100
  port http
  bind http rs107 http rs108 http rs109 http


server virtual vs223 192.168.8.223
  sym-priority 100
  port dns
  bind dns rs107 dns rs108 dns rs109 dns


vlan 1 name DEFAULT-VLAN by port
  router-interface ve 1


vlan 2 name SERVERS by port
  untagged ethe 2
  router-interface ve 2


vlan 99 name SYNC by port
  untagged ethe 16


router vrrp-extended


server vip-group 1
  vip 192.168.8.222
  vip 192.168.8.223


interface ve 1
  ip address 192.168.8.3 255.255.255.0
  ip vrrp-extended vrid 1
   backup priority 100 track-priority 10
   ip-address 192.168.8.1
   vip-group 1
   track-port e 1
   track-port e 2
   enable


interface ve 2
  ip address 192.168.100.3 255.255.255.0
  ip vrrp-extended vrid 2
   backup priority 100 track-priority 10
   ip-address 192.168.100.1
   track-port e 1
   track-port e 2
   enable

The priorities at the master ServerIron are 109 for the VRRP-E instances and as well for the virtual servers - the priorities at the backup ServerIron are 100. A vip-group is part of the configuration to bind the virtual servers 192.168.8.222 and 192.168.8.223 to the VRRP-E instance of interface "ve 1". The failover of the virtual servers is not linked to the failover of the VRRP-E instance. BOTH VRRP-E instances are configured to track the same ports (Upstream and Downstrean). Both instances will failover to the other ServerIron as soon as one of the links goes down (ethernet 1 or ethernet 2) - the virtual server failover is bound to the VRRP-E failover.

--> everything (VIPs and VRRP-E IPs) is going to failover to the other SI in case of a problem!

Session table synchronization is configure for port http/80 but not for port 53 - UDP DNS queries are pretty sure and a hitless failover is not that important here.

In short:


VRRP-E failover is based on the upstream and downstream link

VIP failover is linked to the VRRP failover

Session sync via dedicated port for HTTP port only

The symmetric-port / direct link between both SIs is optional. The ServerIron is going to use one of the data ports in case there is not any special sync link. A dedicated sync link is nevertheless a good idea because it is going to keep data traffic away from sync traffic and the other way around. Both types of traffic do not interfere each other doing it this way. It is possible to use a multi link trunk as sync link for redundancy reasons. The ServerIron is going to use a data port for sync traffic in case the sync link gets lost.

Use the commands "show server virtual" and "show ip vrrp-e brief" to check the ownership of virtual servers and VRRP-E ip addresses.

There are some more details related to SSLB - link to the documentation:

Symmetric SLB

Sym-active SLB (aka Sym-Active/active-active SLB)

Sym-active is very similar to SSLB - the configuration is nearly the same but it is true active-active load balancing. Sym-active offers HA at the virtual server level like SSLB but but ServerIrons are able to process traffic back and forth at the same time. There is a dedicated master per virtual server talking about SSLB. Active-active offers a way to have the same virtual server active at both ServerIrons at the same time. It is still a single ServerIron answering to ARP requests but both ServerIrons are able to process client requests and server answers at the same time.

Upstream router are able to feed both ServerIron with requests using ECMP or other way to distribute traffic via multiple routes. Some Real servers might use ServerIrons A as gateway and some others ServerIron B (using different VRRP-E instances). This ensures the load is getting processed by both ServerIrons at the same time. It is still recommended to keep the CPU utilisation below 50% at each ServerIron because a single ServerIron might need to process the traffic in case of a problem with the other ServerIron.

The sync link in between both ServerIron is recommended talking about active-active to ensure the session table synchronization is as fast as possible. Return traffic might use ServerIron B even if the incoming traffic is using ServerIron A for the same session. This requires a fast session synchronization. Active-active is best suitable for Layer 4 load balancing environments and not recommended for Layer 7 environments like cookie switching setups using cookie insertion at the ServerIron itself.

Active-actice is redundancy at the virtual server level like SSLB.

The configuration is pretty simple. The major steps are:

  • configure a recommended sync vlan with at least a single port in
  • configure upstream and downstream VLANs / subnets
  • configure SLB with reals, virtual servers and sym-priorities
  • enable sym-active for every virtual server
  • configure VRRP-E
  • link VIP to VRRP-E failover using vip-groups

A vip-group is not necessary in all cases. Talk to your local SE to see what fits best to your situation.

Example setup - network diagram (click to enlarge):

sym-active.JPG

The setup above is slightly more complex than the SSLB one - both virtual servers are active at both ServerIrons but there is still a "MASTER" ServerIron responsible for ICMP ECHO replies and ARP replies. Both ServerIrons are able to process traffic coming to the virtual server at Layer 4. ServerIron A is master for the VRRP-E address 192.168.8.1 and 192.168.100.253. ServerIron B is master for the VRRP-E address 192.168.8.254. Some of the real servers are configured to use 192.168.100.253 as their default gateway and some others to user 192.168.100.254.

Keep in mind that you still need to stay below 50% utilization at both SIs to be redundant because one system needs to be able to handle the load in case of problems with one of the devices.

This would look like the following at ServerIron A (check the stuff highlighted in red):

server symmetric-port ethernet 16 vlan-id 99

server port 80
  session-sync
  tcp
  tcp keepalive 3 2

server port 53

session-sync
  udp keepalive 3 2


server real rs107 192.168.100.107
  port http
  port http url "HEAD /"
  port dns

server real rs108 192.168.100.108
  port http
  port http url "HEAD /"
  port dns

server real rs109 192.168.100.109
  port http
  port http url "HEAD /"
  port dns


server virtual vs222 192.168.8.222
  sym-priority 109

  sym-active
  port http
  bind http rs107 http rs108 http rs109 http

server virtual vs223 192.168.8.223
  sym-priority 109

  sym-active
  port dns
  bind dns rs107 dns rs108 dns rs109 dns

vlan 1 name DEFAULT-VLAN by port
  router-interface ve 1

vlan 2 name SERVERS by port
  untagged ethe 2
  router-interface ve 2

vlan 99 name SYNC by port
  untagged ethe 16

router vrrp-extended


server vip-group 1
  vip 192.168.8.222
  vip 192.168.8.223

interface ve 1
  ip address 192.168.8.2 255.255.255.0
  ip vrrp-extended vrid 1
   backup priority 109 track-priority 10
   ip-address 192.168.8.1
   vip-group 1
   track-port e 1
   track-port e 2
   enable

interface ve 2
  ip address 192.168.100.2 255.255.255.0
  ip vrrp-extended vrid 2
   backup priority 109 track-priority 10
   ip-address 192.168.100.253
   track-port e 1
   track-port e 2
   enable

  ip vrrp-extended vrid 3
   backup priority 100 track-priority 10
   ip-address 192.168.100.254
   track-port e 1
   track-port e 2
   enable

The configuration of ServerIron B looks slightly different (check the stuff highlighted in red):

server symmetric-port ethernet 16 vlan-id 99


server port 80
  session-sync
  tcp
  tcp keepalive 3 2


server port 53

  session-sync
  udp keepalive 3 2


server real rs107 192.168.100.107
  port http
  port http url "HEAD /"
  port dns


server real rs108 192.168.100.108
  port http
  port http url "HEAD /"
  port dns


server real rs109 192.168.100.109
  port http
  port http url "HEAD /"
  port dns


server virtual vs222 192.168.8.222
  sym-priority 100

  sym-active
  port http
  bind http rs107 http rs108 http rs109 http


server virtual vs223 192.168.8.223
  sym-priority 100

  sym-active
  port dns
  bind dns rs107 dns rs108 dns rs109 dns


vlan 1 name DEFAULT-VLAN by port
  router-interface ve 1


vlan 2 name SERVERS by port
  untagged ethe 2
  router-interface ve 2


vlan 99 name SYNC by port
  untagged ethe 16


router vrrp-extended


server vip-group 1
  vip 192.168.8.222
  vip 192.168.8.223


interface ve 1
  ip address 192.168.8.3 255.255.255.0
  ip vrrp-extended vrid 1
   backup priority 100 track-priority 10
   ip-address 192.168.8.1
   vip-group 1
   track-port e 1
   track-port e 2
   enable


interface ve 2
  ip address 192.168.100.3 255.255.255.0
  ip vrrp-extended vrid 2
   backup priority 100 track-priority 10
   ip-address 192.168.100.253
   track-port e 1
   track-port e 2
   enable

  ip vrrp-extended vrid 3
   backup priority 109 track-priority 10
   ip-address 192.168.100.254
   track-port e 1
   track-port e 2
   enable

ServerIron A is responsible for ARP and ICMP traffic because it is using the higher sym-priority and as well the higher priority for the upstream VRRP-E instance (VRID 1). Session table synchronization is enabled for HTTP and DNS traffic due to the fact that return traffic might pass both devices. ServerIron A is default gateway for the real server .107 and ServerIron B for the real servers .108 and .109.

DNS is most probably not a good example for sym-active setups because session table synchronization for DNS does not make a lot of sense (DNS sessions are very short living) nevertheless it is OK as an example. The upstream router might send incoming traffic to ServerIron A or B - both SIs are able to process incoming and as well return traffic.

In short:


VRRP-E failover is based on the upstream and downstream link

VIP failover is linked to the VRRP failover BUT both ServerIron are able to process traffic at any time

Session sync via dedicated port for HTTP and DNS traffic

The symmetric-port / direct link between both SIs is recommended here - incoming traffic might hit ServerIron A and return traffic for the same session might go back via ServerIron B which requires a very fast sync process and a dedicated link in between both ServerIrons ensures that there is no other traffic influencing the sync traffic at the same wire.

Use the commands "show server virtual" and "show ip vrrp-e brief" to check the ownership of virtual servers and VRRP-E ip addresses.

There are some more details related to Sym-Active in the documentation - link to the documentation:

Sym-Active