Design & Build

Data Center Infrastructure-Deployment Guide: VCS Fabric In-Service Software Upgrades (ISSU) Feature

by ‎06-09-2014 02:30 PM - edited ‎08-06-2014 08:58 AM (3,458 Views)

Preface

 

 

Overview

This document provides guidance to performing the Brocade VCS In-Service Software Upgrades (ISSUs) and step-by-step instructions for performing ISSUs.

 

Audience

This document is intended for network design and operation staffs who are interested in deploying the Brocade VCS ISSUs feature for datacenter high-availability solution.

 

Objectives

This deployment provides guidance and recommendations for deployment of the Brocade VCS ISSUs for datacenter high-availability solution.

 

Related Documents

The following documents are valuable resources for the network designer. In addition, any Brocade release notes that have been published for Brocade VDX Switches should be reviewed.

 

References

 

About Brocade

Brocade®(NASDAQ: BRCD)networking solutions help the world’s leading organizationstransition smoothly to a world where applications and information reside anywhere.This vision is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

 

Innovative Ethernet and storage networking solutions for datacenter, campus, and service provider networks help reduce complexity and cost while enabling virtualization and cloud computing to increase business agility.

 

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, and professional services offerings. (www.brocade.com)

 

Key Contributors

The content in this guide was developed by the following key contributors.

  •          Lead Architect: Chris Yoon, Strategic Solutions Lab

           

Document History

Date                  Version        Description

2014-06-10          1.0               Initial Release

 

Introduction

Today’s datacenter network operators find that a significant percentage of downtime results from planned network maintenance. Software upgrades to implement new features or capabilities, or to apply maintenance are major causes for system downtime.

 

Ensuring a highly available network means removing all possible causes of downtime, and high availability has never been more important than today in datacenter network. The criticality of datacenter network services means maintenance windows for software upgrades have been reduced. Today’s datacenter networks need to function 24 hours a day, but enhancements and software upgrades to yield new capabilities and services , and maintenance must go on.

 

Hence, in order to improve datacenter network high-availability dramatically, Brocade implemented ISSUs feature on Brocade NOS 4.0.0 and onward allowing packet forwarding to continue while the NOS software is being upgraded. 

 

ISSU Overview

In-service software upgrades (ISSUs) are supported in Brocade NOS 4.0.0 and higher. An ISSU allows a dual management module system to be upgraded non-disruptively and is invoked by entering the firmware downloadcommand from the active management module. For more information about Brocade NOS ISSUs, refer to NOS v4.1.1 Configuration Guide.

 

ISSU Supported Platform

ISSUs is supported in Brocade VDX 8770 series switches with dual management modules, and is supported in both fabric cluster mode and logical chassis cluster mode. Thus, in order to perform an ISSU upgrade, you must have:

 

  •          A VDX 8770 series switch with dual (redundant) management modules and running NOS 4.0.0 or onward.

 

ISSU Compatibility

ISSUs are supported for the following Brocade NOS upgrade paths:

  •          4.0.0 to 4.0.1
  •          4.0.0 to 4.1.0
  •          4.0.1 to 4.1.0
  •          4.1.1 to 4.1.2

 

ISSUs are supported for the following Brocade NOS downgrade path:

  •          4.1.0 to 4.0.1
  •          4.1.2 to 4.1.1

 

==> Note: Prior to performing an ISSU, network designers and operators must check the NOS software version release notes to determine whether an upgrade can be performed using the Brocade NOS ISSU process.

 

ISSU Behavior

While performing ISSU, no Layer 2 traffic is disrupted. The following summarizes the ISSU behavior in NOS 4.1.1 and onward:

 

  • No data path disruption results for Layer 2 and FCoE traffic.
  • All Layer 2 control protocol states are retained.
  • The topology state and interface state are retained.
  • All running configuration is retained (including the last accepted user configuration just before HA failover).
  • May result in data path and control path disruption for the Layer 3 protocol.
  • Layer 3 configuration is replayed during ISSUs.
  • During ISSUs, the principal switch in a logical chassis cluster remains the principal switch. After ISSUs, the principal switch reestablishes cluster management layer connection with other switches and reforms the cluster. A secondary switch in a logical chassis cluster reestablishes cluster management layer connection with the principal switch and rejoins the cluster after warm recovery.

ISSU Procedure

An in-service Brocade NOS software upgrade or downgrade is accomplished by entering a series of four commands. The command sequence is as follows:

 

  • firmware download
  • show firmwaredownloadstatus rbridge-id x
  • show firmwaredownloadstatus summary rbridge-id x
  • firmware activate rbridge-id x

Test Bed Diagram

For testing NOS ISSU, two VDX 8770 switches are used and are in a VCS Fabric configured in Logical-Chassis mode as shown in the diagram below. Two 10GE STC (Spirent Test Center) ports are used to generate Layer 2 traffic, and to verify if there is any Layer 2 traffic loss while performing the ISSU for RB1 from NOS 4.1.1 to 4.1.2. Thus, RB1 is the DUT (Device under Test).

 

VCSFabricISSUTestBed.jpg

   ISSU Test Bed Diagram

 

Procedure

 

Step 1: Verify the VCS Fabric

Verify that the VCS Fabric has formed and is in Logical-Chassis.

 

<=============>

sw0# sh fabric all

 

VCS Id: 10

Config Mode: Distributed

 

Rbridge-id     WWN                               IP Address       Name

----------------------------------------------------------------------------------------------

   1              10:00:00:05:33:EB:00:00   10.254.6.89        >"sw0"*

   2              10:00:00:05:33:EA:76:54   10.254.6.86         "sw0"

 

The Fabric has 2 Rbridge(s)

 

sw0# show fabric isl all

 

No. of nodes in cluster: 2

 

Rbridge-id: 2   #ISLs: 3

 

 Src       Src           Nbr       Nbr  

Index   Interface       Index   Interface      Nbr-WWN                        BW   Trunk  Nbr-Name

----------------------------------------------------------------------------------------------------------------------------------

60      Te 2/2/1         60     Te 1/4/1        10:00:00:05:33:EB:00:00   20G   Yes   "sw0"

112     Fo 2/4/5         4      Fo 1/1/5        10:00:00:05:33:EB:00:00   40G           "sw0"

113     Fo 2/4/6         5      Fo 1/1/6        10:00:00:05:33:EB:00:00   40G            "sw0"

 

 

Rbridge-id: 1   #ISLs: 3

 

 Src       Src           Nbr       Nbr  

Index   Interface      Index   Interface       Nbr-WWN                         BW       Trunk  Nbr-Name

----------------------------------------------------------------------------------------------------------------------------------------

4         Fo 1/1/5      112     Fo 2/4/5        10:00:00:05:33:EA:76:54   40G                 "sw0"

5         Fo 1/1/6      113     Fo 2/4/6        10:00:00:05:33:EA:76:54   40G                 "sw0"

60       Te 1/4/1       60      Te 2/2/1        10:00:00:05:33:EA:76:54   20G        Yes    "sw0"

<=============>

 

Step 2: Configure VLAN 4001

Configure VLAN 4401 on the VCS Fabric interfaces connected to the STC ports. Connect using  switch port trunk allowing all VLANs on the trunks.

 

<=============>

interface Vlan 4001

 

sw0# show running-config interface TenGigabitEthernet 1/4/3

interface TenGigabitEthernet 1/4/3

 fabric isl enable

 fabric trunk enable

 switchport

 switchport mode trunk

 switchport trunk allowed vlan all

 switchport trunk tag native-vlan

 spanning-tree shutdown

 no shutdown

 

sw0# show running-config interface TenGigabitEthernet 2/2/3

interface TenGigabitEthernet 2/2/3

 fabric isl enable

 fabric trunk enable

 switchport

 switchport mode trunk

 switchport trunk allowed vlan all

 switchport trunk tag native-vlan

 spanning-tree shutdown

 no shutdown

<=============>

 

Step 3: Start Layer 2 traffic

Start Layer 2 traffic using the STC ports, and then verify if the L2 traffic is working with no traffic loss.

 

Step 4: Verify NOS Version

Verify that the DUT, which is RB1, is running NOS 4.1.1. 

 

<=============>

sw0# show version rbridge-id 1

rbridge-id 1

Network Operating System Software

Network Operating System Version: 4.1.1

Copyright (c) 1995-2014 Brocade Communications Systems, Inc.

Firmware name:      4.1.1

Build Time:         17:05:19 Apr  1, 2014

Install Time:       15:33:02 Apr 22, 2014

Kernel:             2.6.34.6

 

BootProm:           1.0.1

Control Processor:  e500mc with 7168 MB of memory

 

Slot    Name    Primary/Secondary Versions           Status

-----------------------------------------------------------------------------------------------

M1      NOS     4.1.1                                             STANDBY

                      4.1.1                                              

M2      NOS     4.1.1                                              ACTIVE*

                      4.1.1                                              

L1/0    NOS     4.1.1                                              STANDBY

                      4.1.1                                              

L1/1    NOS     4.1.1                                               ACTIVE

                      4.1.1                                               

L2/0    NOS     4.1.1                                              STANDBY

                      4.1.1                                              

L2/1    NOS     4.1.1                                              ACTIVE

                      4.1.1                                              

L3/0    NOS     4.1.1                                              STANDBY

                      4.1.1                                              

L3/1    NOS     4.1.1                                              ACTIVE

                      4.1.1                                              

L4/0    NOS     4.1.1                                              STANDBY

                      4.1.1                                               

L4/1    NOS     4.1.1                                              ACTIVE

                      4.1.1                                              

<=============>

 

Step 5: Verify Active and Standby Management Cards.

Verify the DUT is running with Active and Standby Management cards. 

 

<=============>

sw0# show redundancy rbridge-id 1

Rbridge-id: 1

=== MM Redundancy Statistics ===

Current Active Session:

Active Slot = M2 (Local), Failover Cause: Failed Over, Warm Recovered

Standby Slot = M1 (Remote)

Start Time: 15:49:59 PDT Tue Apr 22 2014

 

Previous Active Session:

Active Slot = M1

Standby Slot = M2

Start Time: 15:44:25 PDT Tue Apr 22 2014

End Time: 15:45:09 PDT Tue Apr 22 2014

 

System Uptime: 14:55:11 PDT Tue Apr 22 2014

 

!!!--- The output shows that currently, M2 management module is Active and M1 management module is Standby.

<=============>

 

Step 6: Perform Firmware Upgrade

Use the following firmware command on the principal switch to upgrade firmware on the DUT (i.e., RB1).

 

firmware download logical-chassis ftp host  <host IP>  directory   <NOS 4.1.2 directory in FTP server>  user <FTP username>  password  <FTP password>  rbridge-id  1

 

==> Note: Below is the output when performing the firmware upgrade command from NOS 4.1.1 to NOS 4.1.2. It shows the result of the sanity check on the RB1 is “Non-disruptive (ISSU)” which means the ISSU will be performed when the firmware is activated.  

 

<=============>

sw0# firmware download logical-chassis ftp host 10.254.8.30 directory /nos4.1.2 user eyoon password 1234 rbridge-id 1

 

Following is the result of the sanity check on the specified nodes.

 

Rbridge-id    Sanity Result               Current Version              

----------------------------------------------------------------------------

1                 Non-disruptive(ISSU)    4.1.1

 

This command will download firmware to the nodes. Please run "firmware activate" after the completion of installation.

 

Do you want to continue? [y/n]:y

<=============>

 

After typing “y” in the Step 6, the telnet session to the principal switch will be frozen. Disconnect the telnet session to the principal switch, and re-connect it.

 

Step 7: Check Status of Firmware Upgrade

Perform “show firmwaredownloadstatus rbridge-id 1” and “show firmwaredownloadstatus summary rbridge-id 1” commands on the principal switch to check the firmware installation status. 

 

<=============>

sw0# show firmwaredownloadstatus rbridge-id 1

rbridge-id 1

[1]: Fri Apr 25 16:47:23 2014

Slot M1: Firmware install begins.

 

 

sw0# show firmwaredownloadstatus summary rbridge-id 1

rbridge-id 1

Firmware Download session is in progress.

 

!!!! ---- For now, RB1 firmware installation is work in progress.

 

sw0# show firmwaredownloadstatus rbridge-id 1       

rbridge-id 1

[1]: Fri Apr 25 16:47:23 2014

Slot M1: Firmware install begins.

 

[2]: Fri Apr 25 16:52:12 2014

Slot M1: Firmware install ends.

 

[3]: Fri Apr 25 16:52:12 2014

Slot M2: Firmware install begins.

 

[4]: Fri Apr 25 16:52:12 2014

Slot L1/0: Firmware install begins.

 

………..........<snip>…..…………..

 

 

[19]: Fri Apr 25 16:54:18 2014

Slot L4/1: Firmware install ends.

 

[20]: Fri Apr 25 16:55:20 2014

Slot M2: Firmware install ends.

 

 

sw0# show firmwaredownloadstatus summary rbridge-id 1

rbridge-id 1

Firmware Download completed. Execute Firmware Activate for Activation.

 

!!!! ---- For now, RB1 firmware installation is finished, and RB1 is ready to perform “Firmware Activation”.

<=============>

 

Wait until you see the following output from “show firmwaredownloadstatus summary rbridge-id 1” command:

<=============>

Firmware Download completed. Execute Firmware Activate for Activation.

<=============>

 

Step 8: Activate New Firmware on DUT

Perform “firmware activate rbridge-id 1” command on the principal switch to activate the installed NOS 4.1.2 firmware.

 

<=============>

sw0# firmware activate rbridge-id 1

 

This command will activate the firmware on the following nodes.

rbridge-id 1 :  uses ISSU protocol, non-disruptive.

 

Do you want to continue? [y/n]:y

<=============>

 

Step 9: Verify Status of New Firmware on DUT

Perform “show firmwaredownloadstatus rbridge-id 1” and “show firmwaredownloadstatus summary rbridge-id 1” commands on the principal switch to check the firmware activation status. 

 

<=============>

sw0# show firmwaredownloadstatus rbridge-id 1       

rbridge-id 1

[1]: Fri Apr 25 16:47:23 2014

Slot M1: Firmware install begins.

 

[2]: Fri Apr 25 16:52:12 2014

Slot M1: Firmware install ends.

 

[3]: Fri Apr 25 16:52:12 2014

Slot M2: Firmware install begins.

 

[4]: Fri Apr 25 16:52:12 2014

Slot L1/0: Firmware install begins.

 

………..........<snip>…..…………..

 

 [40]: Fri Apr 25 17:02:36 2014

Slot M2: Firmware is swapped.

 

[41]: Fri Apr 25 17:02:36 2014

Slot M1: The blade begins to reboot.

 

 

sw0# show firmwaredownloadstatus summary rbridge-id 1

rbridge-id 1

Firmware Download session is in progress.

!!!! ---- For now, RB1 firmware activation is work in progress.

 

<=============>

sw0# show firmwaredownloadstatus summary rbridge-id 1

rbridge-id 1

Firmware Commit is in progress.

!!!! ---- For now, RB1 firmware commit is work in progress.

 

sw0# show firmwaredownloadstatus summary rbridge-id 1

rbridge-id 1

No Firmware Download session in progress.

!!!! ---- For now, RB1 firmware activation is finished.

 

 

sw0# show redundancy

=== MM Redundancy Statistics ===

Current Active Session:

Active Slot = M1 (Local), Failover Cause: Failed Over, Warm Recovered

Standby Slot = M2 (Remote)

 

Previous Active Session:

Active Slot = M2, Failover Cause: Failed Over, Warm Recovered

Standby Slot = M1

Start Time: 17:07:04 PDT Fri Apr 25 2014

End Time: 17:07:49 PDT Fri Apr 25 2014

 

System Uptime: 14:55:11 PDT Tue Apr 22 2014

 

!!!--- The following “show redundancy” output show for firmware activation for ISSU, management module switchover took placed compared with the output of “Step 5”.  

<=============>

 

Step 10: Verify Firmware Release on DUT

Perform “show version rbridge-id 1” command on the principal switch to verify the firmware upgrade to NOS 4.1.2. 

 

<=============>

sw0# show version rbridge-id 1

rbridge-id 1

Network Operating System Software

Network Operating System Version: 4.1.2

Copyright (c) 1995-2014 Brocade Communications Systems, Inc.

Firmware name:      4.1.2

Build Time:         17:37:55 Mar 19, 2014

Install Time:       16:51:23 Apr 25, 2014

Kernel:             2.6.34.6

 

BootProm:           1.0.1

Control Processor:  e500mc with 7168 MB of memory

 

Slot    Name    Primary/Secondary Versions             Status

---------------------------------------------------------------------------------------------

M1      NOS     4.1.2                                              ACTIVE*

                       4.1.2                                              

M2      NOS     4.1.2                                              STANDBY

                      4.1.2                                              

L1/0    NOS     4.1.2                                              ACTIVE

                      4.1.2                                              

L1/1    NOS     4.1.2                                              STANDBY

                      4.1.2                                              

L2/0    NOS     4.1.2                                              ACTIVE

                      4.1.2                                               

L2/1    NOS     4.1.2                                              STANDBY

                      4.1.2                                              

L3/0    NOS     4.1.2                                              ACTIVE

                      4.1.2                                              

L3/1    NOS     4.1.2                                              STANDBY

                      4.1.2                                              

L4/0    NOS     4.1.2                                              ACTIVE

                      4.1.2                                              

L4/1    NOS     4.1.2                                              STANDBY

                      4.1.2                                               

<=============>

 

Step 11: Confirm no Layer 2 Traffic Loss During ISSU

Verify there was no Layer 2 traffic loss during the ISSU procedure. The STC Detailed Stream Results shows there was no Layer 2 traffic loss while performing ISSU from NOS 4.1.1 to 4.1.2:

 

Summary

This document has provided information to allow you to take advantage of Brocade ISSU for future upgrades. It discussed step-by-step instructions for performing in-service software version changes in Brocade VDX 8770 switches.