Design & Build

Data Center Solution, Storage-Validation Test: Brocade VCS Fabric and Pure Storage FA-420 Flash Storage Array

by on ‎08-21-2014 07:38 AM - edited on ‎09-15-2014 08:57 AM by randylodes (2,793 Views)
 

 

Preface

 

Overview

The Solid State Ready (SSR) program is a comprehensive testing and configuration initiative to provide Fibre Channel SAN and IP interoperability with flash storage. This program provides testing of multiple fabrics, heterogeneous servers, NICs, and HBAs in large port-count Brocade environments.

 

The SSR qualification program will help verify seamless interoperability and optimum performance with solid state storage in Brocade SAN and Ethernet fabrics.

 

Purpose of This Document

This document provides the validation of Brocade VCS fabric technology with the Pure Storage FA-420 all-flash storage array, using multiple switch platforms, HBAs, and server operating systems. This validation shows that the FA-420 interoperates properly within a Brocade Ethernet fabric, while supporting   the performance and low latency associated with solid state storage.

 

Audience

The content in this document is written for a technical audience, including solution architects, system engineers, and technical development representatives.

 

Objectives

  1. Test the Pure FA-420 array with the Brocade Ethernet fabric, for different stress and error recovery scenarios to validate the interoperability and integration of the array with Brocade Ethernet fabrics.
  1. Validate the performance of IP fabric in a solid state storage environment for high throughput and low latency applications.

 

Test Conclusions

  1. Achieved 100% pass rate on all the test cases in the SSR qualification test plan. The network and the storage were able handle the various stress and error recovery scenarios without any issues.
  2. Different I/O workload scenarios were simulated using Medusa, Vdbench and VMware IOAnalyzer tools and sustained performance levels were achieved across all workload types. The results confirm that the Pure Storage FA-420 array interoperates seamlessly with the Brocade Ethernet fabric, and together demonstrate high availability, performance, and low latency.
  3. For optimal availability and performance, consideration should be given to multipath configuration on the host side. While Windows 2008 and 2012 will provide Round-Robin behavior by default, Linux systems will benefit from adding a custom entry to /etc/multipath.conf, and VMWare hosts systems should be changed from the default   ‘Most Recently Used (VMWare)’ setting to ‘Round-Robin (VMWare)’.  Actively using all available paths provides a significant improvement in performance throughput.
  4. The results confirm the assertion that the Pure Storage FA-420 array interoperates seamlessly with Brocade VCS fabric, and the configuration demonstrated high availability and sustained performance.
  5. The switches in the VCS fabric should have sufficient number of ISLss with multiple uplinks to provide sufficient bandwidth and redundancy.

 

Related Documents

 

References

 

Key Contributors

The content in this guide was provided by the following key contributors.

  • Test Architects: Mike Astry, Patrick Stander
  • Test Engineer: Randy Lodes

 

About Brocade

Brocade networking solutions help the world’s leading organizations transition smoothly to a world where applications and information reside anywhere. This vision is realized through the Brocade One™ strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

 

Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and cost while enabling virtualization and cloud computing to increase business agility.

 

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, and professional services offerings.

 

To learn more, visit (www.brocade.com)

 

About Pure Storage

Pure Storage has a simple mission: to enable the widespread adoption of flash in the enterprise data center. We're a team of some of the brightest minds in storage, flash memory and related technical industries. Founded on principles of teamwork and knowledge sharing, we focus on producing an exceptional result while we transform the landscape of the enterprise storage market (and have some fun along the way).

 

Document History

Date                  Version        Description

8-19-2014           1.0                Initial Version

 

Test Plan

The storage array is connected via a Brocade VCS Ethernet fabric to multiple server hosts to drive IO in a multipath configuration. Error injection is introduced, and failover & recovery behaviors are observed. IO performance is observed across different workload configurations.  

 

Scope

Testing is performed with a mix of GA and development versions of Brocade’s Network Operating System (NOS) running on Brocade VDX switches configured to form a Brocade VCS Ethernet fabric.

 

Testing is centered on interoperability and optimal configuration. Performance is observed within the context of best practice fabric configuration; however absolute maximum benchmark reporting of storage performance is beyond the scope of this test.

 

Details of the test steps are covered under “Test Case Descriptions” section. Standard test bed setup includes IBM/HP/Dell chassis server hosts with Brocade/Intel/Emulex/Broadcom CNAs and NICs with two uplinks from every host to a Brocade VCS fabric. IO generators included Medusa Labs Test Tools, vdbench, Iometer, and VMWare IOAnalyzer.

 

Test Configuration

The following diagram shows the test configuration and network topology.

 

 

Test Configuration.jpg

   Test Configuration

 

DUT Descriptions

The following tables provide details about the devices under test (DUT) and the test equipment.

 

Storage Array

DUT ID

Model

Vendor

Description

Pure Storage FA-420

FA-420

Pure Storage

Pure Storage FA-420

The Pure Storage FA-420 flash storage array is an all-flash array that supports 11-35 TB raw capacity. Each controller supports 4x 8Gb Fibre Channel connections, 2x 10Gb iSCSI connections, and 2x Infiniband connections.

 

Brocade Ethernet Fabric Switches

DUT ID

Model

Vendor

Description

VDX-1

VDX6730-32

Brocade

32 port 10Gb switch (24x10GbE/8GbFC)

VDX-2

VDX6730-76

Brocade

76 port 10Gb switch (60x10GbE/16x8GbFC)

VDX-3

VDX6720-24

Brocade

24 port 10Gb Ethernet Fabric switch

VDX-4

VDX6740

Brocade

48 port 10Gb switch (48x10Gb/4x40Gb)

VDX-5

VDX6740

Brocade

48 port 10Gb switch (48x10Gb/4x40Gb)

VDX-6

VDX6730-32

Brocade

32 port 10Gb switch (24x10GbE/8GbFC)

VDX-7

VDX6730-76

Brocade

76 port 10Gb switch (60x10GbE/16x8GbFC)

VDX-8

VDX6720-60

Brocade

60 port 10Gb Ethernet Fabric switch

 

DUT Specifications

 

Storage

Version

Pure Storage FA-420 solid state flash array

Purity version 3.4.0

 

Brocade switches

Version

VDX 6740 w/ VCS Fabric License

NOS 4.1.2

VDX 6730 w/ VCS Fabric License

NOS 4.1.2

VDX 6720 w/ VCS Fabric License

NOS 4.1.2

 

Adapters

Version

Brocade/QLogic 1020 2-port 10Gb CNA

3.2.4.0

Emulex OCe14102-UM 2-port 10Gb CNA

10.0.803.23

Intel X520-2 2-port 10Gb CNA

3.21.2

Broadcom BCM57810 10Gb NIC

1.78.17-0

 

DUT ID

Servers

RAM

Processor

OS

SRV-1

HP Proliant DL380P G8

160GB

Intel Xeon E5-2640

VMWare 5.5 [cluster]

SRV-2

HP Proliant DL380P G8

160GB

Intel Xeon E5-2640

VMWare 5.5 [cluster]

SRV-3

IBM System x3630 M4

24GB

Intel Xeon E5-2420

RHEL 6.5 x86_64

SRV-4

Dell Poweredge R720

64GB

Intel Xeon E5-2640

Windows Server 2012

SRV-5

ProLiant DL380p G8

32GB

Intel Xeon E5-2690

Windows Server 2012R2

SRV-6

HP Proliant DL385p G8

16GB

AMD Opteron 6212

Windows Server 2008R2

SRV-7

Dell Poweredge R720

16GB

Intel Xeon E5-2620

SLES 11.3 x86_64

SRV-8

Dell Poweredge R720

16GB

Intel Xeon E5-2620

RHEL 6.5 x86_64

 

Test Equipment

Version

Finisar 10Gb Analyzer/Jammer

Xgig-B2100C

Medusa Labs Test Tools

6.0

Vdbench

5.0401

Iometer

1.1.0-rc1

VMWare IOAnalyzer

1.6.0

 

Configure DUT and Test Equipment

The steps for initial configuration of the DUT and test equipment are shown below.

 

Step 1. Configure VCS Fabric in Logical Chassis Mode

The Brocade VDX switches are configured to form a Brocade VCS fabric in a Logical Chassis cluster mode. Refer to the Brocade Network OS Administrator’s Guide for details regarding VCS fabric configuration.

 

Step 2. Configure VLANs on VDX Switches

Create two VLANs as shown below.

 

< ========== >

VDX6730_066_075# conf t

VDX6730_066_075(config)# interface Vlan 7

VDX6730_066_075(config-Vlan-7)# exit

VDX6730_066_075(config)# interface Vlan 8

VDX6730_066_075(config-Vlan-8)# CTRL-Z

< ========== >

 

Step 3. Configure Network Interfaces on Hosts

Both interfaces are used from a dual-port 10GE NIC, configured in different subnets. Jumbo frames are configured on all host ports and switch ports. An example Linux host network configuration is shown below.

 

< ========== >

hb067170:~ # ifconfig p7p1

p7p1      Link encap:Ethernet  HWaddr 8C:7C:FF:21:A5:18

          inet addr:192.168.8.170  Bcast:192.168.8.255  Mask:255.255.255.0

          inet6 addr: fe80::8e7c:ffff:fe21:a518/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

hb067170:~ # ifconfig p7p2

p7p2      Link encap:Ethernet  HWaddr 8C:7C:FF:21:A5:19

          inet addr:192.168.7.170  Bcast:192.168.7.255  Mask:255.255.255.0

          inet6 addr: fe80::8e7c:ffff:fe21:a519/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

< ========== >

 

Example Network Configuration for a VMware Host:

 

VMware Host Configuration for Linux VM.jpg

 

   VMware Host Configuration for Linux VM

 

Step 4. Configure VDX Switch Ports

The host and storage network interfaces on the 192.168.7.0 subnet are placed in VLAN 7, and the  192.168.8.0 interfaces are placed in VLAN 8

 

Example syntax for configuring a VDX switch port is shown below:

 

< ========== >

VDX6730_066_075# conf t

VDX6730_066_075# int te 101/0/5

VDX6730_066_075(conf-if-te-101/0/5)# mtu 9216

VDX6730_066_075(conf-if-te-101/0/5)# no fabric isl enable

VDX6730_066_075(conf-if-te-101/0/5)# no fabric trunk enable

VDX6730_066_075(conf-if-te-101/0/5)# switchport

VDX6730_066_075(conf-if-te-101/0/5)# switchport mode access

VDX6730_066_075(conf-if-te-101/0/5)# switchport access vlan 7

VDX6730_066_075(conf-if-te-101/0/5)# spanning-tree shutdown

VDX6730_066_075(conf-if-te-101/0/5)# no shutdown

< ========== >

 

Step 5. Configure IP Addressing on Pure Storage iSCSI Interfaces

An example configuration using the Pure Storage web GUI is shown below.

 

Pure Storage iSCSI Port IP Addresses.jpg 

   Pure Storage iSCSI Port IP Addresses

 

Step 6. Configure Pure Storage iSCSI LUNs

a. Determine the host IQN. This example is for RedHat Linux.

 

< ========== >

hb067170:~ # cat /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.1994-05.com.redhat:8e532cb2edd

< ========== >

 

b.  Add the host IQN to the array host ports using the Pure Storage GUI.

 

Pure Storage Host IQN Configuration.jpg

   Pure Storage Host IQN Configuration

 

c. Eight 5GB LUNs are presented to host.  An example screen from the Pure Storage configuration tool for LUNs is shown below.

 

Pure Storage LUN Configuration.jpg 

   Pure Storage LUN Configuration

 

Step 7. Discover iSCSI Targets

On a Linux host, use the iscsiadm command set to discover and log into targets

 

a. Discover the targets.

< ========== >

hb067170:~ # iscsiadm -m discovery -t sendtargets -p 192.168.7.1

192.168.7.1:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

192.168.7.2:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

192.168.8.1:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

192.168.8.2:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

< ========== >

 

b. Log in to the targets.

 

< ========== >

hb067170:~ # iscsiadm -m node -L all

< ========== >

 

c. Verify the targets.

< ========== >

hb067170:~ # iscsiadm -m session

tcp: [18] 192.168.7.1:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

tcp: [19] 192.168.7.2:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

tcp: [20] 192.168.8.2:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

tcp: [21] 192.168.8.1:3260,1 iqn.2010-06.com.purestorage:flasharray.39cc4997ca5ea229

< ========== >

 

d. On a VMware host, target information is entered into ‘iSCSI Adapter Software -> Properties’ screen as shown below.

 

VMware iSCSI Target Configuration.jpg 

   VMware iSCSI Target Configuration

 

e. On a Windows Server, discovery is completed through the ‘iSCSI Initiator Properties’ dialog

 

Window Server iSCSI Properties.jpg

   Window Server iSCSI Properties

 

Step 8. Configure Multipathing on Each Host

This configuration allows all paths to be used in a round-robin fashion. This provides superior performance to the default Linux settings which would only use a single active path per LUN.

 

a. Recommended /etc/multipath.conf entry on Linux systems:

 

< ========== >

devices {

    device {

        vendor                "PURE"

        path_selector         "round-robin 0"

        path_grouping_policy  multibus

        rr_min_io             1

        path_checker          tur

        fast_io_fail_tmo      10

        dev_loss_tmo          30

    }

}

< ========== >

 

b. Example of multipath configuration.

 

< ========== >

# multipath -l

mpathe (3624a9370a15a66e949f7d1440001003d) dm-3 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:13 sdac 65:192 active undef running

  |- 1:0:1:13 sdm  8:192  active undef running

  |- 1:0:2:13 sdu  65:64  active undef running

  |- 1:0:3:13 sde  8:64   active undef running

  |- 2:0:0:13 sdas 66:192 active undef running

  |- 2:0:1:13 sdbi 67:192 active undef running

  |- 2:0:2:13 sdba 67:64  active undef running

  `- 2:0:3:13 sdak 66:64  active undef running

mpathd (3624a9370a15a66e949f7d1440001003c) dm-2 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:12 sdab 65:176 active undef running

  |- 1:0:1:12 sdl  8:176  active undef running

  |- 1:0:2:12 sdt  65:48  active undef running

  |- 1:0:3:12 sdd  8:48   active undef running

  |- 2:0:0:12 sdar 66:176 active undef running

  |- 2:0:1:12 sdbh 67:176 active undef running

  |- 2:0:2:12 sdaz 67:48  active undef running

  `- 2:0:3:12 sdaj 66:48  active undef running

mpathc (3624a9370a15a66e949f7d1440001003b) dm-1 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:11 sdaa 65:160 active undef running

  |- 1:0:1:11 sdk  8:160  active undef running

  |- 1:0:2:11 sds  65:32  active undef running

  |- 1:0:3:11 sdc  8:32   active undef running

  |- 2:0:0:11 sdaq 66:160 active undef running

  |- 2:0:1:11 sdbg 67:160 active undef running

  |- 2:0:2:11 sday 67:32  active undef running

  `- 2:0:3:11 sdai 66:32  active undef running

mpathb (3624a9370a15a66e949f7d1440001003a) dm-0 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:10 sdz  65:144 active undef running

  |- 1:0:1:10 sdj  8:144  active undef running

  |- 1:0:2:10 sdr  65:16  active undef running

  |- 1:0:3:10 sdb  8:16   active undef running

  |- 2:0:0:10 sdap 66:144 active undef running

  |- 2:0:1:10 sdbf 67:144 active undef running

  |- 2:0:2:10 sdax 67:16  active undef running

  `- 2:0:3:10 sdah 66:16  active undef running

mpathi (3624a9370a15a66e949f7d14400010070) dm-7 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:17 sdag 66:0   active undef running

  |- 1:0:1:17 sdq  65:0   active undef running

  |- 1:0:2:17 sdy  65:128 active undef running

  |- 1:0:3:17 sdi  8:128  active undef running

  |- 2:0:0:17 sdaw 67:0   active undef running

  |- 2:0:1:17 sdbm 68:0   active undef running

  |- 2:0:2:17 sdbe 67:128 active undef running

  `- 2:0:3:17 sdao 66:128 active undef running

mpathh (3624a9370a15a66e949f7d1440001006f) dm-6 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:16 sdaf 65:240 active undef running

  |- 1:0:1:16 sdp  8:240  active undef running

  |- 1:0:2:16 sdx  65:112 active undef running

  |- 1:0:3:16 sdh  8:112  active undef running

  |- 2:0:0:16 sdav 66:240 active undef running

  |- 2:0:1:16 sdbl 67:240 active undef running

  |- 2:0:2:16 sdbd 67:112 active undef running

  `- 2:0:3:16 sdan 66:112 active undef running

mpathg (3624a9370a15a66e949f7d1440001006e) dm-5 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:15 sdae 65:224 active undef running

  |- 1:0:1:15 sdo  8:224  active undef running

  |- 1:0:2:15 sdw  65:96  active undef running

  |- 1:0:3:15 sdg  8:96   active undef running

  |- 2:0:0:15 sdau 66:224 active undef running

  |- 2:0:1:15 sdbk 67:224 active undef running

  |- 2:0:2:15 sdbc 67:96  active undef running

  `- 2:0:3:15 sdam 66:96  active undef running

mpathf (3624a9370a15a66e949f7d1440001006d) dm-4 PURE    ,FlashArray

size=5.0G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=0 status=active

  |- 1:0:0:14 sdad 65:208 active undef running

  |- 1:0:1:14 sdn  8:208  active undef running

  |- 1:0:2:14 sdv  65:80  active undef running

  |- 1:0:3:14 sdf  8:80   active undef running

  |- 2:0:0:14 sdat 66:208 active undef running

  |- 2:0:1:14 sdbj 67:208 active undef running

  |- 2:0:2:14 sdbb 67:80  active undef running

  `- 2:0:3:14 sdal 66:80  active undef running

< ========== >

 

c. Recommended /etc/multipath.conf entry on VMWare systems:

This configuration allows all paths to be used in a round-robin fashion. This provides superior performance to the default VMWare ‘Most Recently Used’ settings which would only use a single active path per LUN.

 

Below is an example screenshot of a multipath configuration in VMware:

 

VMware Multipath Configuration.jpg

   VMware Multipath Configuration

 

Step 9. Apply Additional Host Tuning on Linux Systems

a. Set the 'noop' I/O scheduler, which has been shown to get better performance with lower CPU overhead than the default schedulers (usually 'deadline' or 'cfq'). 

 

b. Eliminate the collection of entropy for the kernel random number generator, which has high cpu overhead when enabled for devices supporting high IOPS.

 

c. Change CPU affinity. This third change reduces CPU load by redirecting completions to originating CPU

 

The following shows the rules applied at boot in /etc/udev/rules.d/99-pure-storage.rules file:

 

< ========== >

# Recommended settings for Pure Storage FlashArray.

# Use noop scheduler for high-performance solid-state storage

ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="noop"

#

# Reduce CPU overhead due to entropy collection

ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"

#

# Reduce CPU load by redirecting completions to originating CPU

ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"

< ========== >

 

d. Configuring these parameters on the host can ensure that the I/O pause time is reduced to an appropriate value. The configuration file can be found here: /etc/iscsi/iscsid.conf. The parameters that need to be changed are:

 

  • node.session.timeo.replacement_timeout = 10
  • node.session.initial_login_retry_max = 4

 

Step 10. Configure Workload Generators

Several different workload generators are used to get a variety of IO workloads. On Windows and Linux systems, Medusa Labs Test Tools and vdbench are used. On VMware systems, VMware’s IOAnalyzer is used.

 

Step 11. Configure QoS

For host adapters supporting Data Center Bridging protocol for iSCSI, DCB QoS can be configured on the VCS fabric.

 

a. Here is an example CEE map with iSCSI CoS value “4” configured in addition to the Auto-NAS configuration:

 

VDX6730_066_075# show running-config cee-map

cee-map default

precedence 1

priority-group-table 1 weight 30 pfc on

priority-group-table 15.0 pfc off

priority-group-table 15.1 pfc off

priority-group-table 15.2 pfc off

priority-group-table 15.3 pfc off

priority-group-table 15.4 pfc off

priority-group-table 15.5 pfc off

priority-group-table 15.6 pfc off

priority-group-table 15.7 pfc off

priority-group-table 2 weight 20 pfc off

priority-group-table 3 weight 20 pfc off

priority-group-table 4 weight 30 pfc on

priority-table 2 2 3 1 4 2 2 15.0

remap fabric-priority priority 2

remap lossless-priority priority 2

!

 

b. On switch ports which are attached to storage interconnects, default CoS is set to 4 for iSCSI traffic

< =========== >

VDX6730_066_075(conf-if-te-111/0/1)# qos cos 4

< =========== >

 

c. On switch ports which are attached to host adapters, default CEE map is applied

< =========== >

VDX6730_066_075(conf-if-te-75/0/10)# cee default

< =========== >

 

 

Test Cases

 

1.1

FABRIC INITIALIZATION – BASE FUNCTIONALITY

Confirm basic IP Fabric functionality of the storage array

1.1.1

Storage Device – Physical and Logical Login with Speed Negotiation

1.1.2

iSCSI LUN Mapping

1.1.3

Storage Device Multipath Configuration – iSCSI Path integrity

1.2

ETHERNET STORAGE – ADVANCED FUNCTIONALTY

Examine the storage behavior - bandwidth validation, traffic congestion, and protocol recovery

1.2.1

iSCSI Bandwidth Validation

1.2.2

Storage Device – w/ Congested Fabric

1.2.3

Storage Device – iSCSI Protocol Jammer Test Suite

1.3

STRESS & ERROR RECOVERY

Confirm device integrity and recovery during congestion and error injection conditions

1.3.1

Storage Device Fabric IO Integrity – Congested Fabric

1.3.2

Storage Device Nameserver Integrity – Device Recovery with Manual Cable Pull

1.3.3

Storage Device Nameserver Integrity – Device Recovery with Device Relocation

1.3.4

Storage Device Nameserver Stress – Device Recovery with Device Port Toggle

1.3.5

Storage Device Recovery – ISL Port Toggle

1.3.6

Storage Device Recovery – ISL Port Toggle (entire switch)

1.3.7

Storage Device Recovery – Switch Offline

1.3.8

Storage Device Recovery – Switch Firmware Download

1.3.9

Workload Simulation Test Suite

 

Test Case Descriptions

 

1.1 Fabric Initialization – Base Functionality

 

1.1.1 Storage Device – Physical and Logical Login with Speed Negotiation

 

Test Objective

  1. Verify device login to VDX switch with all supported speed settings.
  2. Configure VDX switch for AUTONAS
  3. Configure Storage Port for iSCSI connectivity. Validate Login & base connectivity.

Procedure

  1. Set switch port speed settings to 10Gb, and Auto_Negotiate.

<==========>

VDX6730_066_075(conf-if-te-101/0/5)# speed [1000 | auto]

<==========>

 

2. Confirm link is up

 

3. On VDX 6740 switch, configure autonas as shown below.

 

<==========>

VDX6730_066_075(config)# nas auto-qos

VDX6730_066_075(config)# nas server-ip 192.168.7.1 vlan 7

<==========>

 

4. Configure storage port for iSCSI is detailed above in ‘Configuration Steps’

 

Result

1. PASS. Storage negotiates with fabric successfully at 10Gb and auto speeds. Verify with IO.

 

1.1.2 iSCSI LUN Mapping

 

Test Objective

  1. Verify host to LUN access with each mapped OS-type.

Procedure

  1. Configure a LUN on each OS platform (see ‘Configuration Steps’ above for detail) and verify access.

Result

1. PASS. LUNS are presented and verified with IO.

 

1.1.3 Storage Device Multipath Configuration – iSCSI Path integrity

 

Test Objective

  1. Verify multi-path configures successfully.
  2. Each Adapter and Storage port is connected to in different switches.
  3. For all device paths, consecutively isolate individual paths and validate IO integrity and path recovery.

Procedure

  1. Configure multipath for each host (see ‘Configuration Steps above for detail)
  2. Isolate paths and verify failover.

 

Result

1. PASS. Multipath failover is successful, verify with IO.

 

1.2 Ethernet Storage - Advanced Functionality

 

1.2.1 iSCSI Bandwidth Validation

 

Test Objective

  1. Validate maximum sustained bandwidth to storage port via iSCSI.
  2. After 15 minutes
  3. Verify IO completes error free.

Procedure

  1. Run large-block IO from multiple hosts to storage to saturate storage ports.

Result

1. PASS. IO competes successfully with no errors.

 

1.2.2 Storage Device – w/Congested Fabric

 

Test Objective

  1. Create network bottleneck through a single Fabric ISL.
  2. Configure multiple ‘iSCSI to host’ data streams sufficient to saturate the ISL’s available bandwidth for 30 minutes. 
  3. Verify IO completes error free.

Procedure

  1. Isolate a single ISL and saturate with IO for 30 minutes

 

Result

1. PASS. IO competes successfully with no errors.

 

1.2.3 Storage Device – iSCSI Protocol Jammer Test Suite

 

Test Objective

 

1. Perform Protocol Jammer Tests including areas such as:

  • CRC corruption,
  • packet corruption,
  • missing frame,
  • host error recovery,
  • target error recovery

Procedure

  1. Insert Finisar Jammer/Analyzer between switch and storage port.
  2. Execute Jammer alterations in both directions (storage <-> host)
  3. Verify recovery

Result

1. PASS. Insert Finisar Jammer/Analyzer between storage port and switch. Introduce packet anomalies and verify proper recovery.

 

1.3 Stress and Error Recovery

 

1.3.1 Storage Device Fabric IO integtiry – Congested Fabric

 

Test Objective

  1. Configure fabric and devices for maximum link & device saturation.
  2. From all available initiators start a mixture of READ/WRITE/VERIFY traffic with random data patterns continuously to all their targets overnight. 
  3. Verify no host application failover or unexpected change in I/O throughput occurs.
  4. If needed, add L2 Ethernet traffic to fill available bandwidth.

Procedure

  1. Run large block read/write IO from multiple hosts and confirm saturation on links.

Result

1. PASS. Saturate host, storage, and switch links. IO completes error-free.

 

1.3.2 Storage Device Integrity – Device Recovery from Port Toggle – Manual Cable Pull

 

Test Objective

  1. With I/O running, perform a quick port toggle every Storage Device & Adapter port. 
  2. Verify host I/O will recover.

Procedure

  1. While running IO, perform a manual cable pull on host and storage ports

Result

1. PASS. Run a mix of IO and execute cable pulls (~ 10 seconds) on host and storage ports. IO fails over and recovers error-free.

 

1.3.3 Storage Device Integrity – Device Recovery from Device Relocation

 

Test Objective

  1. Sequentially performed for each Storage Device & Adapter port.
  2. With I/O running, manually disconnect and reconnect port to different switch in same fabric.
  3. Verify host I/O will failover to alternate path and toggled path will recover.
  4. Repeat test for all switch types.

Procedure

  1. Configure an alternate switch port with same settings as port under test.
  2. Move cable to alternate port while running IO and verify recovery.

Result

1. PASS. Move link to alternate port, verify recovery; move back to original port, verified recovery.

 

1.3.4 Storage Device Stress – Device Recovery from Device Port Toggle – Extended Run

 

Test Objective

  1. Sequentially toggle each Initiator and Target ports in fabric.  
  2. Verify host I/O will recover to alternate path and toggled path will recover.
  3. Run for 24 hours.

Procedure

  1. Run IO for 24 hours;
  2. Toggle all storage and host ports sequentially in a loop.
  3. Verify paths recover and IO completes error-free

Result

1. PASS. Paths fail over and IO continues error-free. 

 

1.3.5 Storage Device Recovery – ISL Port Toggle – Extended Run

 

Test Objective

  1. Verify fabric ISL path redundancy between hosts & storage devices.
  2. Sequentially toggle each ISL path on all switches. 
  3. Host I/O may pause, but should recover. 
  4. Verify host I/O throughout test.

Procedure

  1. Run IO and disable a single ISL port.
  2. Allow IO to recover; enable ISL port.
  3. Repeat sequentially for all ISL ports on all switches. Verify paths recover and IO completes error-free.

Result

1. PASS. IO recovers and continues error-free.

 

1.3.6 Storage Device Recovery – ISL Port Toggle (entire switch)

 

Test Objective

  1. Verify fabric switch path redundancy between hosts & storage devices.
  2. Sequentially, and for all switches, disable all ISLs on the switch under test.
    a. For Switches containing device under test (i.e. Switches A,B,E,F in diagram) IO will pause, and then resume after switch comes online.
    b. For intermediate Switches (i.e. Switches C & D in diagram), IO will pause, and then resume.

Procedure

  1. Run IO and disable all ISL ports on single switch.
  2. Allow IO to recover; enable all ISL ports.
  3. Repeat sequentially for all switches.
  4. Verify paths recover and IO completes error-free.

Result

1. PASS. IO recovers and continues error-free.

 

1.3.7 Storage Device Recovery – Switch Offline

 

Test Objective

  1. Toggle each switch in sequential order using a mix of switchDisable, reboot, commands and power-cycling the switch.
  2. For Switches containing device under test, IO will pause and then resume after switch comes online.
  3. For intermediate Switches IO will pause and then resume while switch is online.

Procedure

  1. Run IO.
  2. Disable each switch with ‘switchDisable’ command.
  3. Repeat sequentially for all switches.
  4. Verify paths recover and IO completes error-free.
  5. Reboot each switch with ‘reboot’ command.
  6. Repeat sequentially for all switches.
  7. Verify paths recover and IO completes error-free.
  8. Manually power-cycle each switch. Repeat sequentially for all switches.
  9. Verify paths recover and IO completes error-free.

Result

1. PASS. Execute switch disables, reboots, and power cycles on all switches. IO recovers and continues error-free. 

 

1.3.8 Storage Device Recovery – Switch Firmware Download HCL (where applicable)

 

Test Objective

  1. Sequentially perform firmware maintenance procedure on all device connected switches under test.
  2. Verify Host I/O will continue (with minimal disruption) through firmwaredownload and device pathing will remain consistent.

Procedure

  1. Download Brocade NOS firmware to all switches in fabric
  2. Activate each switch sequentially.
  3. Confirm that IO and multipathing continue error-free.

Result

1. PASS. Update NOS version from 4.1.2 to 4.1.2a on each switch.

2. PASS. Multipathing and IO recovered error-free.

 

1.3.9 Workload Simulation Test Suite (Optional)

 

Test Objective

  1. Validate Storage/Fabric behavior while running a workload simulation test suite.
  2. Areas of focus may include VM environments, de-duplication/compression data patterns, and database simulation.

Procedure

  1. Setup 4 standalone hosts for iSCSI.
  2. Use Medusa I/O tool to generate I/O and simulate workloads.
    - Run random and sequential I/O in a loop at block transfer sizes of 512, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, and 1m.
    - Include a nested loop of 100% read, 100% write, and 50% read/write.
    - Run File Server simulation workload
    - Run Microsoft Exchange Server simulation workload
  3. Setup an ESX cluster of 2 hosts with 4 worker VMs per host.
  4. Use VMware IOAnalyzer tool for generating I/O and simulating workloads as follows.
    a. Run random and sequential IO at large and small block transfer sizes.
    b. Run SQL Server simulation workload
    c. Run OLTP simulation workload
    d. Run Web Server simulation workload
    e. Run Video on Demand simulation workload
    f. Run Workstation simulation workload
    g. Run Exchange server simulation workload

Result

1. PASS. Workloads complete successfully, error-free, and with expected throughput.