Design & Build

Data Center Solution, Storage-Deployment Guide: Brocade GEN 5 Fibre Channel with Tegile Flash Array

by on ‎08-01-2014 10:11 AM - edited on ‎09-08-2014 04:36 PM by pillais (2,351 Views)
 

 

Preface

 

Overview

This document provides guidance for deploying Brocade Gen 5 Fibre Channel SAN products with Tegile HA2100EP flash storage array. The deployments shown have been validated and tested. The following document provides information about the Validation Test.

 

References

 

Two deployments are presented; a native Fibre Channel SAN deployment and an extension to this using FCoE in the Hosts to connect to the Tegile flash array.

 

Audience

Engineers responsible for design, deployment and operations who want to successfully add Tegile flash arrays to a Brocade Gen 5 Fibre Channel SAN fabric.

 

Objectives

This guide provides a base deployment using dual Fibre Channel SAN A/B fabrics and an alternative deployment with hosts using FCoE to connect to the SAN while the flash storage arrays continues to use native Fibre Channel. The content in this deployment guide assumes a Fibre Channel SAN has been designed and installed in accordance with the Brocade SAN Design Guide (see Related Documents). The procedures in this deployment guide only cover specific settings or configuration changes that ensure the best performance, reliability and management of a Gen 5 Fibre Channel SAN fabric when using Tegile flash HA2100EP series flash storage arrays.

 

Related Documents

The following publications are useful when deploying Brocade Gen 5 Fibre Channel fabrics, Brocade VCS Fabrics and Tegile InteiFlash flash arrays.

 

References

 

Document History

Date                  Version        Description

2014-08-01         1.0                Initial Release

2014-08-14         1.1                Updated "SAN A/B Core-Edge Deployment Template"

 

About Brocade

Brocade networking solutions help the world�s leading organizations transition smoothly to a world where applications and information reside anywhere. This vision is realized through the Brocade One® strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

 

Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and cost while enabling virtualization and cloud computing to increase business agility.

 

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, and professional services offerings.

 

To learn more, visit (www.brocade.com)

 

About Tegile Systems

Tegile SystemsTM is pioneering a new generation of flash-driven enterprise storage arrays that balance performance, capacity, features and price for virtualization, file services and database applications.

 

Our hybrid arrays are significantly faster than legacy arrays and significantly less expensive than all solid-state disk-based arrays. Featuring both NAS and SAN connectivity, these virtual data storage systems are easy-to-use, fully redundant, and highly scalable. Additionally, they come complete with built-in snapshot, replication, near-instant recovery, and virtualization management features.

 

Tegile’s patented IntelliFlash™ flash technology accelerates performance to solid state speeds without sacrificing the capacity or cost advantage of hard disk storage. Additionally it enables on-the-fly de-duplication and compression so usable capacity is far greater than its raw capacity.

 

Technical Architecture

Fibre Channel SAN storage is commonly found in data centers, particularly when low and consistent latency, high availability and simplicity are required. For this reason, existing Fibre Channel SAN fabrics are being upgraded with storage arrays that use flash storage arrays instead of rotating magnetic disk arrays. For some applications, including databases, virtual desktop infrastructure, CRM and ERP systems, the addition of flash storage arrays can boost the return on investment of the application software (often much more expensive than the server and storage infrastructure they run on) by increasing the transactions per second significantly. Application licenses often increase with server cores and size, but typically don’t change with the type of storage system. For this reason, adding high speed flash storage to existing Fibre Channel SANs vs. adding a new server or more servers to improve performance is economically attractive.

The high IOPS and low latency of flash arrays are much greater than disk storage and therefore, deployment requires elimination of IO bottlenecks in the Fibre Channel SAN or the economic benefit of flash storage in not realized. 

 

This deployment guide shows how to add Tegile flash arrays to existing Brocade Gen 5 Fibre Channel SAN fabrics and Brocade VDX switches supporting Fibre Channel and FCoE in a VCS Fabric.

 

Building Blocks

The Brocade Data Center Base Reference Architecture (see References below) includes building blocks for designing a Gen 5 Fibre Channel SAN and a VCS Fabric. An Edge block, Core block and VCS Fabric Edge block are included in this deployment example as shown below.

 

Fibre Channel Core Block

This block can include one, or as shown more than one, DCX Backbone. If multiple DCX are used, then Backbone ISL connections between them can be used to connect them without consuming ports on the port cards.

 

DataCenter_BlockSAN_CoreBackbone.JPG 

 

   Fibre Channel Core Building Block

 

Fibre Channel Edge Block

This block shows the hosts connected to two Edge switches. Each Edge switch is in a physically separate SAN fabric denoted as “Fabric A” and “Fabric B”. The Edge switch in Fabric A connects to the DCX(s) in Fabric A shown in the Core Block above.

 

DataCenter_BlockSAN_EdgeSwitch.JPG

 

  Fibre Channel SAN-Edge Building Block

 

VCS Fabric Leaf Block

This block includes the VDX 6730 Switch which includes 8 Gbps Fibre Channel ports and 10 GE converged enhanced Ethernet (CEE) ports. The Fibre Channel ports connect to the DCX Backbone switches in either Fabric A or Fabric B. The hosts connect to one or more leaf switches using NIC Teaming on the host and vLAG on the VDX 6730 switches for high availability and resiliency.

 

DataCenter_BlockVCSFabric_LeafConvergedDevices.JPG

 

   Brocade VCS Fabric-Converged Leaf Building Block

 

References

 

Design Template

These building blocks are combined into a design template as shown below.

 

Template_SAN+Tegile.jpg

 

   Deployment Template-SAN A/B With Tegile Flash Array

 

The topology is a dual core/edge SAN (A/B fabrics. The edge has both Fibre Channel attached hosts via Fibre Channel switches and Fibre Channel over Ethernet (FCoE) attached hosts via the Brocade VDX 6730 Switches. Two independent VCS Fabrics can be used at the edge if desired, each attaching to the Core block so that each VCS Fabric participates in either the Fabric A or B SAN. These switches have 10 GE ports with converged enhanced Ethernet (CEE) supporting FCoE and 8 Gbps Fibre Channel ports. The VDX 6730 Fibre Channel ports that connect to the core DCX Fibre Channel Backbone are configured as E_Ports while the corresponding DCX Backbone ports are configured as EX_Ports to provide Fibre Channel routing.

 

The Core block connects existing disk storage arrays and the Tegile flash array using 16 Gbps Fibre Channel ports. ISL trunks with 16 Gbps links connect the Edge Fibre Channel switches to the core DCX Backbone switches.

 

The template supports hosts with either Fibre Channel host bus adaptors (HBA) for native Fiber Channel or 10 GE converged network adaptors (CNA) for Fibre Channel over Ethernet (FCoE). The FCoE traffic terminates in the VDX 6730 switch and is forwarded to the core DCX Backbone switch(es) with native 8 Gbps Fibre Channel. The References below provide information about configuring Brocade Network OS (NOS) for VDX Switches and a VCS Fabric, and Brocade Fabric OS (FOS) for Fibre Channel switches and the DCS Backbone switches.

 

References

 

Base Configuration: Dual SAN Fabrics with Tegile Flash Array

 

Deployment Topology

The diagram below shows the deployment topology. There are two independent SAN fabrics (light blue and grey). The hosts/servers and Tegile flash array(s) connect to both fabrics for high availability and resiliency. This is a best practice for SAN design as IO failures from hosts to storage can create application failures that can require long recovery times for data synchronization that result in application outages.

 

Tegile_SANA-B.jpg

 

   Deployment Topology

 

This is a common topology used today although in larger data centers there can be many edge switches and multiple core switches hosting hundreds of servers and petabytes of storage. Regardless of the Gen 5 Fibre Channel fabric size, these deployment procedures can be used to add one or more Tegile flash arrays.

 

Pre-requisites

  1. An existing Brocade Gen 5 Fibre Channel SAN designed and deployed in accordance with the Brocade SAN Design Guide.
  2. Sufficient rack space, power and cooling for the Tegile IntelliFlash flash array(s).
  3. Correct firmware releases for Brocade switches and Tegile flash arrays.
  4. Supported servers/hosts with supported Fibre Channel host bus adaptors (HBA) and/or converged Ethernet adaptors (CNA).

 

Bill of Materials

The following table show the bill of materials used for this deployment. The references contain links to product data sheets.  The DCX 8510 Backbone and Brocade 6510 SAN switches are Gen 5 Fibre Channel capable products with up to 16 Gbps per port. Only a single core and single edge switch are shown in this procedure, but it is applicable when more DCX 8510 and/or Brocade 6510 switches are installed in the fabric.

 

Identifier

Vendor

Model / Release

Notes

Tegile Array

Tegile

HA2100EP Array

2.1.1.3(140308)-1473

Dual Controller array with support for FC, iSCSI, NFS, CIFS protocol support. Each controller has 2x8Gb FC ports and 2x10GbE ports in an active-passive configuration.

6510-1

Brocade

BR-6510

FOS 7.3.0

48 port Gen 5 16Gb FC switch

6510-2

Brocade

BF-6510

FOS 7.2.1

48 port Gen 5 16Gb FC switch

DCX-1

Brocade

DCX 8510-8

FOS 7.3.0

8 slot Gen 5 16Gb FC chassis

DCX-2

Brocade

DCX 8510-4

FOS 7.2.1

4 slot Gen 5 16Gb FC chassis

Hosts/Servers

Various

Various

Hosts and Servers supporting Brocade Gen 5 Fibre Channel switches and supported Fibre Channel host bus adapators (HBA) and converged Ethernet adaptors (CNA)

Host Bus Adaptors

QLogic       -- >

(Brocade)

 

 

 

QLogic       -- >

 

 

 

Emulex      -- >

 

 

 

 

Brocade     -- >

Brocade 1860

2-port 16Gb FC HB

Drvr: 3.2.4.0

Frmw: 3.2.4.0

 

QLogic QLE2672

2-port 16GB FC HBA

Drvr: 8.06.00.10.06.0-k, Frmw: 6.06.03

 

Emulex LPE 12002

2-port 8Gb Fc HBA

Drvr: 10.0.100.1

Frmw: 1.00A9

 

Brocade 1020

2-port CNA

Drvr: 3.2.4.0

Frmw: 3.2.4.0

Fibre Channel Host Bus Adaptors supporting Brocade Gen 5 Fibre Channel switches.

 

References

 

Task 1: Create Zones for Each HBA

 

Description

Fibre Channel zones are used to logically and securely isolate traffic between each HBA (initiator) in a host and all storage ports it connects to. New zones can be added for the existing HBA and the new Tegile array ports. An alternative is to update existing zones for each HBA that will connect to the Tegile array adding the World-wide Port Name (WWPN) of the Tegile array port(s)

 

Assumptions

  1. New host or HBA added to existing host connecting to Tegile flash array.

 

Step 1: Create / Modify zones

Use the zoneCreate command to create new zones or the zoneAdd command to add devices (Tegile array port targets) to an existing zone. The example below shows how to use the zoneCreate, cfgAdd and cfgEnable commands to create a new zone (hb067160_Tegile), add the zone to an existing zoneset (SSR) and enable the updated zone set.

 

<==========>

root> zonecreate hb067160_Tegile, "21:00:00:24:ff:48:b9:6a; 21:00:00:24:ff:48:b9:6b; 50:01:43:80:06:2d:08:28; 50:01:43:80:06:2d:08:2a; 10:00:8c:7c:ff:03:9b:00; 10:00:8c:7c:ff:03:9b:01"

root> cfgadd SSR, hb067160_Tegile

root> cfgenable SSR

<==========>

 

References

 

Confirm Zoning

Use the zoneShow command to display the zones connecting the Tegile array to the host.

 

<==========>

root> zoneshow hb067160_Tegile

 zone:  hb067160_Tegile

                21:00:00:24:ff:48:b9:6a; 21:00:00:24:ff:48:b9:6b;

                50:01:43:80:06:2d:08:28; 50:01:43:80:06:2d:08:2a;

                10:00:8c:7c:ff:03:bc:00; 10:00:8c:7c:ff:03:bc:01

<==========>

 

Task 2: Enable Bottleneck Detection on Fibre Channel Switches

 

Description

This step enables reporting of latency and congestion alerts on each switch in the SAN fabric. This aids performance monitoring and troubleshooting of SAN device performance problems.

 

Step 1: Turn on Bottleneck Monitoring

Enter the bottleneckmon command

 

<==========>

root>  bottleneckmon --enable -alert

root>  bottleneckmon --config -alert -time 150 -qtime 150 -cthresh 0.7 -lthresh 0.2

<==========>

 

Confirm Bottleneck Monitoring

Use the bottleneckmon status argument to verify the configuration settings for bottleneck monitoring.

 

<==========>

root> bottleneckmon --status

Bottleneck detection - Enabled

==============================

 

Switch-wide sub-second latency bottleneck criterion:

====================================================

Time threshold                 - 0.800

Severity threshold             - 50.000

 

Switch-wide alerting parameters:

================================

Alerts                         - Yes

Latency threshold for alert    - 0.200

Congestion threshold for alert - 0.700

Averaging time for alert       - 150 seconds

Quiet time for alert           - 150 seconds

<==========>

 

Task 3: Present Tegile Array LUNs

 

Description

The Tegile array needs to present LUNs (Logical Unit Numbers) to a specific host. The LUNs are show to both HBA on the host when it is configured with dual HBA and dual SAN A/B fabrics for high availability. The HBA multipath failover feature ensures traffic continues between the host and the Tegile array should any components fail in one of the SAN fabrics.

 

Assumptions

  1. Dual HBA installed in the host.

 

Step 1: Configure Tegile Array LUNs for Each Host

Configure LUNs using the Tegile management GUI.  

 

References

  • Tegile User Guide, V2.1.x (available from Tegile Support)

 

Confirm LUNs Configured

The Tegile management GUI shows LUNs and their configuration.

 

Tegile_LUNConfirmation.jpg 

 

   Tegile IntelliFlash LUN Configuration

 

Task 4: Configure Multipath Driver on Each Host

 

Description

Host HBA’s include multipath drivers that ensure IO can use more than one path between the ost and the storage array. As shown in the deployment topology, dual SAN A/B fabrics are used to ensure high availability and continued IO when any component in one SAN Fabric fails.

 

Assumptions

  1. Dual HBA installed in the host.
  2. HBA device driver supports multipath option.

 

Step 1: Configure Multipath Support on Linux Hosts.

This configuration allows all paths to be used in a round-robin fashion. This provides superior performance to the default Linux settings which would only use a single active path per LUN.

The recommended /etc/multipath.conf entry on a Linux system is shown below.

 

<==========>

devices {

   device {

          vendor                      "TEGILE"

          product                     "ZEBI-FC"

          hardware_handler      "1 alua"

          path_selector            "round-robin 0"

          path_grouping_policy "group_by_prio"

          no_path_retry             10

          dev_loss_tmo             50

          path_checker             tur

          prio                           alua

          failback                     30

          }

}

<==========>

 

Confirm Linux Multipath Support

Enter the multipath command to display the Linux host multipath configuration.

 

< =========== >

3600144f0ec770e000000536d06730008 dm-2 TEGILE  ,ZEBI-FC

size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 5:0:2:15 sdm  8:192  active ready running

| |- 5:0:0:15 sdr  65:16  active ready running

| |- 6:0:1:15 sdad 65:208 active ready running

| `- 6:0:2:15 sdai 66:32  active ready running

`-+- policy='round-robin 0' prio=1 status=enabled

  |- 5:0:3:15 sdd  8:48   active ready running

  |- 6:0:3:15 sde  8:64   active ready running

  |- 6:0:0:15 sdx  65:112 active ready running

  `- 5:0:1:15 sdw  65:96  active ready running

< =========== >

 

Step 2: Configure Multipath Support on VMware Hosts.

Run the following command from the VMware host CLI to setup round-robin multipathing rule for Tegile arrays:

 

< =========== >

esxcli storage nmp satp rule add -s VMW_SATP_ALUA –V "TEGILE" -M "ZEBI-FC" -c "tpgs_on” –P VMW_PSP_RR -e "Tegile Zebi FC"

< =========== >

 

Alternate #1: FCoE Hosts with Tegile Flash Array

 

An alternate configuration has hosts connecting wiith Fibre Channel over Ethernet (FCoE) to the Tegile flash array which is connected with native Fibre Channel. This option is often used to minimize cabling and cost of a rack of servers. Brocade recommends terminating the FCoE traffic at the top of rack switch at the edge and forwarding the traffic as native Fibre Channel to the core of the existing Fibre Channel SAN fabric.

 

Deployment Topology

The diagram below shows the deployment topology when adding hosts using FCoE. There are two independent SAN fabrics (light blue and grey). The host/servers and Tegile flash array are connected to both fabrics.

 

Tegile_SANA-B+FCOE.jpg

 

   Deployment Topology

 

At the edge, Brocade VDX 6730 switches are added in a VCS Fabric. The VCS Fabric can include other VDX switches and FCoE to connect a larger number of hosts to the Tegile flash array but the VDX 6730 switch is required to terminate the FCoE traffic within the VCS Fabric and forward it as native Fibre Channel traffic to the DCX Backbone at the core of the SAN Fabric. The VDX Fibre Channel ports connected to the DCX Backbone are configured as E_Ports while the ports on the DCX Backbone are configured as EX_Ports, or Fibre Channel routing ports.

 

This is a common topology used today although in larger data centers there can be many edge switches and multiple core switches hosting hundreds of servers and petabytes of storage. Regardless of the Gen 5 Fibre Channel fabric size, these deployment procedures can be used to add one or more Tegile flash arrays and hosts with FCoE for block storage access.

 

Pre-requisites

  1. An existing Brocade Gen 5 Fibre Channel SAN designed and deployed in accordance with the Brocade SAN Design Guide.
  2. Sufficient rack space, power and cooling for the Tegile flash array(s).
  3. Correct firmware releases for Brocade switches and Tegile flash arrays.
  4. Supported servers/hosts with supported converged network adaptors (CNA).
  5. An FCoE license for the VDX switches
  6. A Fibre Channel routing license for the DCX Backbone
  7. [Optional] A Virtual Fabric license for the DCX Backbone

 

Bill of Materials

The following table shows the bill of materials used for this deployment. The references contain links to product data sheets.  The DCX 8510 Backbone and Brocade 6510 SAN switches are Gen 5 Fibre Channel capable products with up to 16 Gbps per port. Only a single core and single edge switch per fabric are shown in this procedure, but it is applicable when more DCX 8510, Brocade 6510 and or VDX 67390 switches are installed in the fabric.

 

Identifier

Vendor

Model / Release

Notes

Tegile Array

Tegile

HA2100EP Array

2.1.1.3(140308)-1473

Dual Controller array with support for FC, iSCSI, NFS, CIFS protocol support. Each controller has 2x8Gb FC ports and 2x10GbE ports in an active-passive configuration.

6510-1

Brocade

BR-6510

FOS 7.3.0

48 port Gen 5 16Gb FC switch

6510-2

Brocade

BF-6510

FOS 7.2.1

48 port Gen 5 16Gb FC switch

DCX-1

Brocade

DCX 8510-8

FOS 7.3.0

8 slot Gen 5 16Gb FC chassis

DCX-2

Brocade

DCX 8510-4

FOS 7.2.1

4 slot Gen 5 16Gb FC chassis

VDX-1

Brocade

VDX 6730

NOS 4.1.1

60x10GbE ports and 16x8Gb FC port switch

VDX-2

Brocade

VDX 6730

NOS 4.1.1

60x10GbE ports and 16x8Gb FC port switch 

Hosts/Servers

Various

Various

Hosts and Servers supporting Brocade Gen 5 Fibre Channel switches and supported Fibre Channel host bus adapators (HBA) and converged Ethernet adaptors (CNA)

Host Bus Adaptors

QLogic     -- >
(Brocade)

 

 

 

QLogic     -- >

 

 

 

Emulex    -- >

 

 

 

 

Brocade  -- >

Brocade 1860

2-port 16Gbps FC HBA

Drvr: 3.2.4.0

Frmw: 3.2.4.0

 

QLogic QLE2672

2-port 16 Gbps FC HBA

Drvr: 8.06.00.10.06.0-k, Frmw: 6.06.03

 

Emulex LPE 12002

2-port 8 Gbps FC HBA

Drvr: 10.0.100.1

Frmw: 1.00A9

 

Brocade 1020

2-port CNA

Drvr: 3.2.4.0

Frmw: 3.2.4.0

Fibre Channel Host Bus Adaptors supporting Brocade Gen 5 Fibre Channel switches.

 

References

 

Task 1: Configure Zones for FCoE Initiators on VDX Switches

 

Description

Fibre Channel zones are added to the VCS Fabric on the VDX 6730 switches. These zones are exactly like the Fibre Channel zones used in the SAN A/B fabrics. But, the VCS Fabric acts as a separate Fibre Channel fabric from the perspective of the existing SAN A/B fabrics, so it has its own independent zones. To connect devices in the VCS Fabric to devices in the SAN A or SAN B fabrics requires Fibre Channel routing. Device connections where each device is in a separate fabric are defined using a special type of zone, an LSAN zone.

 

Assumptions

  1. FCoE license is installed in the VCS Fabric
  2. Supported CNA are installed in the hosts using FCoE to connect to the Tegile array.

 

Step 1: Configure Zones on a VDX Switch

Refer to the NOS Administrator’s Guide (see References below) for configuring zones on the VDX 6730 Switch. The following is an example of configuring Fibre Channel zones on the VDX 6730 switch.

 

<==========>

#

#Create Zoning Configuration

#

VDX6730_066_075# config t

VDX6730_066_075(config)# zoning defined-configuration cfg NOS_SSR

VDX6730_066_075(config-cfg-NOS_SSR)#

#

#Create New Zone

#

VDX6730_066_075(config-cfg-NOS_SSR)# member-zone lsan_hb067164_Tegile

#

#Add Device WWN to New Zone

#

VDX6730_066_075(config-cfg-NOS_SSR)# zoning defined-configuration zone lsan_hb067164_Tegile

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 10:00:8c:7c:ff:4f:d4:00

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 10:00:8c:7c:ff:4f:d4:01

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 21:00:00:24:ff:48:b9:6a

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 21:00:00:24:ff:48:b9:6b

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 50:01:43:80:06:2d:08:28

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# member-entry 50:01:43:80:06:2d:08:2a

 

#

#Save Zoning Changes

#

VDX6730_066_075(config-zone-lsan_hb067164_Tegile)# zoning enabled-configuration cfg-action cfg-save

#

#Enable Zoning Configuration

#

VDX6730_066_075(config)# zoning enabled-configuration cfg-name NOS_SSR

VDX6730_066_075(config)# exit

<==========>

 

References

 

Confirm FCoE Zones Configuration

Use the show zoning command to confirm the FCoE zoning configuration.

 

<==========>

# show zoning enabled-configuration

zoning enabled-configuration cfg-name NOS_SSR

zoning enabled-configuration enabled-zone lsan_hb067164_Tegile

 

 member-entry 10:00:8c:7c:ff:4f:d4:00

 member-entry 10:00:8c:7c:ff:4f:d4:01

 member-entry 21:00:00:24:ff:48:b9:6a

 member-entry 21:00:00:24:ff:48:b9:6b

 member-entry 50:01:43:80:06:2d:08:28

 member-entry 50:01:43:80:06:2d:08:2a

<==========>

 

Task 2: Configure Fibre Channel Routing

 

Description

Fibre Channel routing is configured on the DCX Backbone.

 

Assumptions

  1. VDX Switch connects to Brocade Gen 5 Fibre Channel switch that supports Fibre Channel routing
  2. Fibre Channel routing license deployed.

 

Step 1: Configure Fibre Channel Routing

There are various options and topologies that can be used to configure Fibre Channel routing on a DCX Backbone. Refer to the references below to understand how to configure Fibre Channel routing that conforms to your requirements.

 

References