Mainframe Solutions

New Brocade IP Extension technology enhances virtual tape IP replication and enhances virtual tape solutions.


PREVIEW: SHARE Orlando 2015 and Brocade

by Dr.Steve.Guendert on ‎08-06-2015 12:00 PM (2,737 Views)

SHARE Inc. is a volunteer-run user group for IBM mainframe computers that was founded in 1955 by Los Angeles-area IBM 701 users. It evolved into a forum for exchanging technical information about programming languages, operating systems, database systems, and user experiences for enterprise users of IBM computers.


At next week’s 125th SHARE Conference in Orlando there promises to be a lot of great educational content delivered and here are a few of the topics I am specifically looking forward to discussing:


Day 2 – A mainframe guy at EMC World 2015

by Dave Lytle on ‎05-06-2015 09:00 AM (801 Views)

Day 2 of EMC World 2015 kept pace with the energy and activity that began on day 1. Every session seems to be well attended and in some cases overflowing. Lots of knowledge transfer occurring both in the formal sessions and in all of the many vendor booths at the Expo – especially the Brocade booth.


Day 1 - A Mainframe Guy at EMC World 2015

by Dave Lytle ‎05-05-2015 10:08 AM - edited ‎05-05-2015 10:15 AM (769 Views)

Brocade is a Diamond sponsor for EMC's largest customer event of the year. As a mainframe subject matter expert I was chosen to attend this industry gathering to gain greater insight to EMC solutions as well as to participate one-on-one with other attendees at the Brocade pavilion. This is a great event and I will be chronicaling each of my days of attendance.


EMC World 2015: Brocade, EMC and Mainframe

by Dr.Steve.Guendert on ‎05-01-2015 08:16 AM (1,580 Views)

Brocade and EMC have a great history of working together to provide world class FICON attached storage solutions to IBM z Systems end users all over the globe. While that history is important, the future looks even brighter! I will be delivering a presentation as part of the mainframe track at EMC World 2015 and wanted to take this opportunity to share some of the things I will be discussing in the presentation.


Brocade GEN5 Forward Error Correction (FEC) technology is now available for FICON Express 16S channel connected F_ports, in addition to E_ports.


FICON Express16S Arrives with IBM z Systems z13

by Dr.Steve.Guendert ‎02-09-2015 01:23 PM - edited ‎02-17-2015 12:47 PM (2,735 Views)

16 Gbps FICON channels have arrived!


The Holy Grail of Extension for IBM System z: Introducing the IBM System Storage SAN42B-R

by Dr.Steve.Guendert ‎12-01-2014 02:34 PM - edited ‎12-02-2014 06:50 AM (1,763 Views)

The IBM System Storage SAN42B-R (aka Brocade 7840 Extension Switch) is a true game changer for business continuity in IBM System z environments. It is a quantum leap forward over all existing extension products in the market today in terms of performance, advanced features, management, scalability, and availability.


Business Continuity and Disaster Recovery (BC/DR) continues to be a primary challenge and top of mind concern for most IT organizations running their businesses on IBM System z. The Brocade 7840 Extension Switch offers an ideal, high availability, high performance extension solution platform for IBM System z.


Cascaded FICON, available since the introduction of 2Gbps switch technology, allows mainframe enterprises tremendous flexibility and the potential for fabric cost savings in their FICON architectures. It is extremely important for business continuity and disaster recovery implementations. But if a customer is to take full advantage of the capabilities of a cascaded FICON fabric they must pay attention to the performance aspects of those deployed fabrics. This article will explore a few of the features of FICON switching devices and fabrics that affect the performance of FICON I/O.


The Need For an End to End Connectivity Architecture Team

by Dr.Steve.Guendert ‎09-05-2014 02:42 PM - edited ‎09-05-2014 03:01 PM (2,426 Views)

System z enterprises span multiple data centers, often times over multiple countries.  How do you handle the technical and political issues of end to end network architecture?


To the mainframe, I/O is arguably just as important as the computing done by the host on behalf of its applications. While a mainframe processor complex may be capable of running thousands of applications simultaneously across as many as 60 logical partitions (LPARs), each System z (or zEnterprise) has only 256 channels (paths) that it can supply to any given LPAR… so channel addresses are a precious commodity. Making the best use of all of a system’s I/O resources is imperative if the full value of your host and storage assets is to be achieved. Suggested in this article are five tips for making better use, and driving more value, out of your I/O subsystem.


Proactive vs. Reactive Management of your Storage Networks

by Dr.Steve.Guendert ‎07-15-2014 01:04 PM - edited ‎07-15-2014 01:06 PM (2,670 Views)

I work for a networking company, and have worked with storage networks for nearly 15 years.  I will be the first to admit that at least in the realm of storage network management, management software has historically been far more reactive in nature.  Recently there have been some significant efforts made to change this at Brocade, based in large part on feedback received from end users.


Cut-through the complexity of Storage Networking Management!


Complexity in the data center not only complicates your day-to-day tasks it also exacerbates the problem of maintaining high availability, impairs your ability to provide acceptable performance and efficiency, often increases your overall data center costs and too often inhibits your ability to scale and grow to meet internal demands.


Expensive, home grown expertise is often utilized in data centers to coax as much benefit and value as possible, using a maze of tools and techniques, from your growing FC fabrics and I/O interconnects. However the value actually achieved from these efforts is always variable depending upon each SAN administrator’s capabilities. And troubleshooting actual problems that occur within a SAN infrastructure can be very frustrating and time consuming.


Brocade has been delivering a Gen5 switch portfolio solution that has evolved and now offers you a smarter and more consistent answer for managing all of your storage networking fabrics. Brocade Network Advisor contains several new Fabric Vision technologies that will cut through SAN management complexity while improving your level of confidence and satisfaction in your Brocade I/O infrastructure. Monitoring and Alerting Policy Suite (MAPS) along with Flow Vision, two of those Fabric Vision technologies, deploys breakthrough capabilities that dramatically simplify open systems and mainframe SAN administration, improves the availability and performance of fibre channel SANs, reduces operational costs and can be deployed in a matter of just a few minutes.


So, the truth is, the old mainframe way of doing things that I grew up with back in the 1960’s and 1970’s is gone. The mainframe back then was a monolithic approach to computing. Within a few decades it became a dinosaur – a large and efficient dinosaur – but a dinosaur nonetheless. It became slow moving in a faster and faster paced world. Users wanted, and needed, new applications and application updates at a rapid pace while the mainframe programming staffs seemed to move at glacial speed. As a direct result of this disconnect between expanding user’s needs and the inability of the centralized computer system to respond appropriately, the users looked for new ways to fulfill their needs. And off they went to decentralized processing and direct control of their computing resources. Over time, the terminal result was that computer platform competition was reducing the mainframe’s feeding ground to the point that by the latter part of the twentieth century it was headed for extinction. But then, rather than succumbing to obsolescence it reinvented itself and evolved into the magnificent machine that it has become today. In the end the twentieth century “computer wars” made the mainframe supremely powerful, more agile and incredibly more capable. Let's see why.


The mainframe blogs that I have written in the past have typically been about technology and how our customers can better use their FICON capabilities. But on this occasion I am going to provide a more personal blog about my own 45 year journey through this historical era of the Mainframe and what being a Mainframe Practitioner has meant to me. The reason that you might want to read about this journey is that you will discover that I had to reinvent myself a number of times during all of these years. I made changes to my career in order to continue to challenge myself professionally and to continue to provide value to my employers. Change is not bad. It is usually uncomfortable being taken out of one’s comfort zone but the end result is a better, more talented and valuable you. If you can embrace change, as I have, then your experiences in computer technology and, in particular, the mainframe world, will be absolutely amazing. Also in this reading, for those of you with some decades of experience, you’ll relate to the way things were at various times in our industry – it is kind of fun to remember it all.


Happy 50th Anniversary IBM System Z!

by Dr.Steve.Guendert ‎04-05-2014 07:49 PM - edited ‎04-07-2014 08:38 AM (5,134 Views)

April 7 is a momentous occasion in the history of computing.  It marks the 50th anniversary of IBM's announcement of the System/360 and 50 years of IBM mainframe computing.  Brocade has been right there in partnership with IBM for over 25 of those fifty years, introducing innovations in I/O and storage networking that have helped change the world.




Top 5 Reasons Why Brocade Should Be Your FICON Partner of Choice.

by Dr.Steve.Guendert ‎03-17-2014 07:46 AM - edited ‎03-18-2014 12:55 PM (5,240 Views)

The 30 largest banks in the world run their mission critical mainframe business workloads on a Brocade FICON SAN infrastructure. Why does the world's financial system run on Brocade FICON SANs?  


SHARE Anaheim 2014 and Brocade

by Dr.Steve.Guendert ‎03-07-2014 07:01 AM - edited ‎03-07-2014 07:09 AM (3,914 Views)

Brocade continues its long relationship and history of participation in SHARE



Fabric OS Firmware: The Intelligence Behind Switched FC Fabrics

by Dave Lytle ‎12-06-2013 02:27 PM - edited ‎12-06-2013 02:40 PM (6,424 Views)

When working in any industry, there are many technical names, terms and acronyms that are used constantly.  For those of us in IT one of those familiar terms is “firmware”.  Like most professional organizations, I have found that when firmware is being discussed it is about “what it does” rather than “what it is”.  We often take for granted that, since firmware just seems to be part of today’s intelligent devices, everyone knows exactly what it is.  However, I suspect that is not always the case so read on to find out more about “what it is” in regards to switched Fibre Channel protocols.


FICON Dynamic Channel Path Management (DCM) 2013

by Dr.Steve.Guendert on ‎10-22-2013 03:03 AM (2,051 Views)

My last blog posting discussed the 23 July 2013 set of IBM System z announcements, and how those announcements encourage channel consolidation by IBM System z customers.   One of the items I briefly covered was Dynamic Channel  Path Management (DCM) for FICON.  This blog post will discuss DCM in more detail, and make the case why you should seriously consider implementing DCM.


DCM support for native FICON channels was originally introduced in z/OS V1.11 with support for a single, intermediate FICON switching device between the FICON channel and storage control units.  The recent IBM z/OS V 2.1 announcement significantly enhanced DCM for FICON.  In z/OS V2.1, DCM was enhanced to support FICON channel path connections through two intermediate FICON switching devices, i.e. with z/OS V 2.1, DCM for FICON now supports cascaded FICON configurations.


This really is an important step in the evolution of System z I/O architectures.  With DCM for FICON now supporting cascaded configurations, it will be much easier to use a smaller number of channels (channel consolidation) and optical fiber connections for FICON I/O, particularly for multi-site installations that rely on cascaded FICON.  Remember, z/OS V2.1 also introduced support for up to 24K subchannels per FICON channel...


On 23 July IBM made a series of hardwareoperating system and software announcements for the zEnterprise platforms.  IBM also announced the eagerly awaited new business class zEnterprise with the zBC12 announcement.  There were several parts of each announcement that pertained to I/O and channel technology. One common denominator of each of these parts was that they 1) encourage a movement towards consolidation of channels on the host, and 2) encourage avoiding direct attached FICON channels and adopting switched FICON architectures.


The first channel subsystem enhancement IBM announced was very big news: IBM announced increased addressing with up to 24k subchannels per channel (port) for the FICON Express features.  To help facilitate growth as well as continuing to enable server consolidation, IBM now supports up to 24k subchannels per FICON Express channel (channel path identifier - CHPID). End users will now be able to define more devices per FICON channel, which includes primary, secondary, and alias devices. The maximum number of subchannels across all device types addressable within an LPAR remains at 63.75k for subchannel set 0 and 64k-1 for subchannel sets 1 and higher. This support is exclusive to the zEC12 and the zBC12 and applies to the FICON Express8S, FICON Express8, and FICON Express4 features when defined as CHPID type FC. This is supported by z/OS , z/VM , and Linux on System z...


Out of (South) Africa

by Dr.Steve.Guendert on ‎07-12-2013 06:01 AM - last edited on ‎10-28-2013 09:05 PM by bcm1 (1,203 Views)

I recently returned from a week long business trip to South Africa.  I typically go to South Africa every 12-15 months to meet with our OEM partners, and our FICON/mainframe customers.  Brocade has a great two man team in South Africa with Nick Pateman and Carlos Isidro.  On this most recent trip I met with several FICON customers as well as IBM, HDS, and EMC.  Our meetings consisted of a general Brocade FICON update: how our new technology enhancements with FOS 7.0 and 7.1 work, a product roadmap discussion, and usually some more technical Q&A.  I will be going back in late November to teach our FICON architect certification course.


One of the things I have noticed since my first trip to South Africa in early 2008 is the growth in the IT industry.  The three OEM partners I work with most often (IBM, HDS, EMC) are all experiencing tremendous growth in their business.  The same is true for Brocade.  This growth from our OEMs is not just in South Africa-it is in the entire African continent.  They already have, or are opening offices and facilities in Kenya, Morocco, Namibia, Nigeria, Tanzania, Senegal, and Cameroon.  IBM is even apparently opening a mainframe skills center in Johannesburg to cover the continent and train new mainframers.  Africa is the next great growth market for the IT industry, and with that it is the next great growth market for mainframes, mainframe attached storage, and SAN.  It could even be thought of as the last frontier of IT.  And the growth is being driven from South Africa.


Here are some interesting facts:


According to the Harvard Business Review:

1) Over the past decade, Africa’s real GDP grew by 4.7% a year, on average—twice the pace of its growth in the 1980s and 1990s.

2) By 2009, Africa’s collective GDP of $1.6 trillion was roughly equal to Brazil’s or Russia’s.

3) Telecom companies in Africa have added 316 million subscribers—more than the entire U.S. population—since 2000.

4) According to UN data, Africa offers a higher return on investment than any other emerging market.


How important is Africa to IBM?  According to an article in the February 16, 2013 issue of The Economist

1) Since mid-2011 it has set up shop in Angola, Mauritius and Tanzania, as well as Senegal.

2) IBM now boasts a presence in more than 20 of Africa’s 54 countries.

3) Last August IBM opened a research lab in Nairobi, one of only 12 in the world.

4) And between February 5th and 7th 2013, Ginni Rometty, IBM chief executive, and all who report directly to her met dozens of African customers, actual and prospective, in Johannesburg and the Kenyan capital. It was, Mrs Rometty said, the first time the whole top brass had assembled outside New York since she became the boss just over a year ago.


According to a February 21, 2013 Bloomberg Business Week article titled For IBM, Africa Is Risky and Rife With Opportunity":


"IBM’s global revenue dipped 2.3 percent, to $104.5 billion, in 2012, about the same level it was in 2008. Of that, sales out of Africa kicked in about $400 million and are forecast to more than double and surpass $1 billion in 2015...... That’s faster growth than IBM saw in India, where it started a push in 1992 and surpassed $1 billion in revenue in 2007, the person said."


Here is a slide from a recent IBM Investor Briefing:





Over the past three years, IBM has new mainframe customers in Senegal, Cameroon, Ethiopia, Nigeria, and Namibia.  Financial institutions in Africa are rapidly realizing what the rest of the world has known since April 1964: the mainframe is the best platform to run mission critical business applications. 


Africa truly is the next great frontier for IBM mainframes, as well as mainframe storage and its connectivity. And, it's being driven out of South Africa.  Brocade is right there.


I look forward to future trips to South Africa, and meeting many of the new mainframers in other African nations.


Dr. Steve

Brocade's Ongoing Commitment to IBM System z and Mutual Customers

by Dr.Steve.Guendert on ‎06-11-2013 12:48 AM - last edited on ‎10-28-2013 09:06 PM by bcm1 (1,545 Views)

Greetings from Munich, Germany.  I am here this week at the EMEA IBM System z Technical University.  This event is IBM's primary System z technical conference in Europe each year.  Brocade is proud to be the Gold Sponsor at this year's event.  We have a prominent booth, are giving three presentations, and have several people at the event including yours truly.


Why is this important?  It's just the latest example of the deep commitment Brocade has to IBM System z, and our mutual mainframe customers.


Brocade's CEO, Lloyd Carney recently had a Q&A session/interview with Enterprise Executive Magazine.  The interview appears as the cover story in the May/June issue of Enterprise Executive.   The article had three really important points that I felt needed to be shared and highlighted.  They sum up our experience, and commitment to IBM System z and our mutual customers very well.



  1. Historical perspective:  According to Lloyd in this Q&A, “Brocade has a long, unique history with mainframe technology going back more than 25 years to 1987 and bus and tag channel extension. Our corporate “family tree” includes companies such as Data Switch, Computerm, Inrange, CNT, McDATA and Foundry Networks. We developed and manufactured many generations of products supporting both ESCON and FICON technologies, including the DCX 8510, which is our ninth-generation FICON director.”
  2. On why the mainframe is still a relevant technology today:  According to Lloyd "The IBM mainframe is the Mark Twain of IT; reports of its death have been greatly exaggerated. In fact, mainframes are alive and well, and are continuing to deliver business value to the companies that depend on them. The IBM mainframe is still extremely relevant technology in 2013. All 50 of the world’s top global banks depend on the IBM System z to run the most sophisticated, I/O-intensive transactions on the planet. More than 20 of the 25 largest U.S. retailers continue to run their businesses on IBM mainframes. Brocade is proud that our FICON directors account for more than 80 percent of the installed storage area networking infrastructure in these global enterprises. They’re our most important and valued customers, and it’s an honor that we’re trusted for their mission-critical infrastructure."
  3. On how much longer will Brocade continue to invest in the mainframe market: According to Lloyd "Brocade will continue to invest in the mainframe market for as long as there are mainframes, which likely means for a long time to come."


Which brings me back to the Munich IBM System z Technical University.  Brocade recognized the importance of this event. Its the primary IBM System z Conference in Europe in 2013   We're supporting IBM and the System z platform as a Gold Sponsor.  We're meeting with customers, giving presentations, advice, etc.  We view this as very important.  When we talk the talk and say System z is important to Brocade, we walk the walk so to speak.  Actions speak more loudly than words.  For another European example, Brocade is a member of Guide Share Europe.


How important is IBM System z and its customers to Cisco?  I expected a huge Cisco presence here given their recent new product announcement with their MDS 9710 directors.


How important is IBM System z and its customers to Cisco?  I wanted to ask someone here from Cisco, but unfortunately that was not possible.  You see, IBM System z and its customers are so important to Cisco, that Cisco did not show up here.  Yes, that is right, Cisco did not show up at the most important IBM System z Conference in Europe this year.  No booth, no presentations, not even any attendees here to learn.  I almost felt like I was involved in a Where's Waldo episode entitled "Where's Cisco"?




Let me close this post by contrasting that absence with one of Lloyd's quotes:


"Brocade is proud that our FICON directors account for more than 80 percent of the installed storage area networking infrastructure in these global enterprises. They’re our most important and valued customers, and it’s an honor that we’re trusted for their mission-critical infrastructure."


Auf Wiedersehen from Munich!


Dr. Steve

So here I am at the end of my last day at this year’s EMC World conference. Yes, EMC World 2013 goes on for another three quarters of a day tomorrow but I fly back home to Atlanta tomorrow morning. My brain is full, my legs are wobbly from walking between meeting rooms and the Solutions Pavilion, and my nether regions ache from sitting in conference chairs – you know what I mean, you have been there yourselves!


Yesterday, as my blog pointed out, began on a sad note for me. In contrast, I had a really good day and wonderful experience with an attendee at the conference today. I met this gentleman on the first day of the conference and, as I related in my first blog, we set up a schedule at that point for a meeting to be held today. That meeting took place and, once again, my very good friend and exceptional professional AJ Casamento lead the discussion. This business professional, Blair, is a key officer in a company that is a premier systems integrator, located in California, that provides quality business automation, computer network and security solutions, remote and on-site support, and cost-effective IT outsourcing for small to mid-sized businesses. They do business on several continents...



EMC World 2013 Day 2

by Dave-Lytle on ‎05-07-2013 10:01 AM (937 Views)

Hello from day 2 at EMC World!


I had a sad experience right away this morning. On the elevator on the way down to the conference area I ran into a guy from a very well-known company in California. We were the only people on the elevator and both of us were wearing EMC World badges. I introduced myself and then we shook hands. I mentioned that I was one of millions of people who had connected to their website. He said thanks and then said “but it isn’t hosted on Brocade switching.” I asked him why and he said that Brocade’s presence was few and far between, from his perspective, at his company location and that our competitor was there all the time. Then they were offered a deal that was hard to refuse and, as far as he knew, Brocade was never in the hunt for the business. Out of sight, out of mind I suspect. Sad, sad, sad.


By nature I am a hunter. Most of you probably know what that means. Marketing and sales are often broken up into personality groups known as hunters and farmers. The breakdown is that there are people good at finding new opportunities (hunters) and people good at nurturing current relationships (farmers). Of that group, I am a hunter. And when a hunter hears that an opportunity has gotten away from them, we are always sad. It is one thing to lose an opportunity in a fair competition – but to lose because we never showed up to the tournament – that is hard. No person and no company can be everywhere all of the time. And when there are conflicts it is critical to use resources to target the more important and valuable of those opportunities even at the expense of the others.  But that does not make it any easier to lose an opportunity. This was not a mainframe opportunity, so I was never involved and would not have become involved, except for my accidental meeting with this gentleman on an elevator. I have already spoken to one of my upper management people about this and I will speak with others today. Lost is lost but we need to reflect on the reason why we lost and see if there are things we can do going forward to avoid as many situations like this as possible. In my mind all Brocadians must take responsibility for finding and nurturing prospects and customers and making sure that, ultimately, they feel like a part of the Brocade family. That is just one of the many ways in which we excel as a company and become trusted advisers to our customers and partners. I guess I am just on my soap box because I am competitive by nature and hate to lose so much. Enough said.


EMC World 2013 Day 1

by Dave-Lytle on ‎05-06-2013 01:30 PM (784 Views)

Hi All,


Just thought I'd post a little about my experiences at EMC World 2013 being held in Las Vegas, Nevada this week at the Venetian Hotel. Today was day 1 of the conference and the halls were crowded with technology leaders and users from all over the world. I do not know the actual numbers but I heard that as many as 15,000 participants are here to learn and network and visit the Expo to see and touch the latest computer technology in the world.


I am here as a member of the Brocade contingent attending and supporting this conference. I was invited to attend as a subject matter expert (SME) in mainframes and FICON and, in particular, to meet and work with that specialty area of our customer set. Of course, I am available to any customer that I can assist, but I really do like working with our mainframe customers.


As a value-add to this conference, Brocade is hosting meetings between customers and Brocade management and leading Brocade technologists. It is a very well-orchestrated effort and all of the main participants are running around with a report that tells them what customer they will be meeting with and what the purpose of the meeting is all about. It is all about getting the right people in front of a customer to discuss their needs and requirements and begin a process that will lead to a mutually beneficial outcome...


Using the FICON Director Activity Report (RMF 74-7)

by Dr.Steve.Guendert on ‎03-21-2013 07:53 AM - last edited on ‎10-28-2013 09:08 PM by bcm1 (3,108 Views)

This blog post is based on an article on the same subject I recently wrote for Enterprise Tech Journal Magazine.  The article appeared in slightly different format in the March/April 2013 issue.


The introduction of the FICON I/O protocol to the mainframe I/O subsystem provided the ability to process data rapidly and efficiently. As a result of two main changes that FICON made to the mainframe channel I/O infrastructure, the requirements for a new Resource Measurement Facility (RMF) record came into being. The first change was that unlike ESCON, FICON uses buffer credits to account for packet delivery and provide a data flow control mechanism. The second change was the introduction of FICON cascading, and the long distance capabilities it introduced, which was not as practical with ESCON.


Similar to the ESCON directors that preceded them, FICON directors and switches have a feature called Control Unit Port (CUP). Among the many functions of the CUP feature is an ability to provide host control functions such as blocking and unblocking ports, safe switching, and in-band host communication functions such as port monitoring and error reporting. Enabling CUP on FICON directors while also enabling RMF 74 subtype 7 (RMF 74-7) records for your z/OS system, yields a new RMF report called the “FICON Director Activity Report”. Data is collected for each RMF interval if FCD is specified in yourERBRMFnn parmlib member AND… in SYS1.Parmlib the IECIOSnn says FICON STATS=YES. RMF will format one of these reports per interval per each FICON director that has CUP enabled and the parmlib specified.  If you are using FICON Virtual Fabrics and have your FICON directors partitioned into multiple, smaller logical switches you will have one RMF 74-7 report per switch address (per logical switch). The FICON Director Activity Report captures information based on an interval which is set for RMF and tells it when to create this report along with others. In essence, the report captures a snapshot of data and the counters based on a time interval, such as 20 minutes. Often, you need to run these reports more than once and change the interval periods for troubleshooting to determine if there is a trend.


This RMF report is often overlooked but contains very meaningful data concerning FICON I/O performance—in particular, frame pacing delay. Frame pacing delay has been around since fibre channel SAN was first implemented in the late 1990s by our open systems friends. But until the increased use of cascaded FICON, its relevance in the mainframe space has been completely overlooked


The article “Performance Troubleshooting Using the RMF Device Activity Report” (Oct/Nov 2012 Enterprise Tech Journal)continued a series of articles I have been writing on System z I/O performance.  As a quick review, that article assumed there was an application Service Level Agreement (SLA)/Service Level Objective (SLO) for transaction response time that wasn't being met.  It then went through one of the key RMF reports used in mainframe I/O performance management/troubleshooting: SMF 74-1, the RMF Direct Access Device Activity report. This report contains response time information and information on the various components of response time. It can be used to further narrow down what may be the root cause of the problem and provide a good idea of what other RMF reports we should check.  This article continues where that discussion left off, and will examine the RMF 74-7 record, the RMF FICON Director Activity Report.


Figure 1 illustrates our environment and where the various I/O related RMF reports fit.  Let us assume that our review and analysis of the RMF Device Activity Report showed that the specific components of response time that are outside of our normal parameters are PEND and CONN. We wish to determine if our FICON SAN could be the component of our infrastructure causing PEND and/or CONN to be abnormally high.3.jpg


Figure 2 shows an example of a FICON Director Activity Report. It should be noted that this report is vendor agnostic, meaning that it will contain the same fields and information regardless of which FICON director vendor manufactured the director.  It also is the same for any model director/switch from a given vendor.




The fields of critical interest contained in the report are:


  • AVG FRAME PACING: the number of times during the RMF reporting interval that there was a 2.5 microsecond period in which data (frame) transmission was stopped by this port. 
  • AVG FRAME SIZE READ:  the average frame size (in bytes) of data received by a given port during the RMF reporting interval.
  • AVG FRAME SIZE WRITE: the average frame size (in bytes) of data transmitted by a given port during the RMF reporting interval.
  • PORT BANDWIDTH READ: the average rate (in MB/sec) of data received by a given port during the RMF reporting interval.
  • PORT BANDWIDTH WRITE: the average rate (in MB/sec) of data transmitted by a given port during the RMF reporting interval.
  • ERROR COUNT:  the number of errors which were encountered by a given port during the RMF reporting interval. This should be cross referenced with hardware errors reported on the operations console(s).


Other data in the report include:


  • SWITCH DEVICE:  the hexadecimal number of the switch device of the FICON Director for which measurements are being reported.
  • SWITCH ID: the hexadecimal switch ID of the FICON Director which is associated with this switching device. In case of cascaded switches, a double asterisk may be shown.
  • TYPE MODEL MAN PLANT SERIAL: a hardware description of the switch device.
  • PORT ADDR: the hexadecimal address of the port.
  • CONNECTION: provides information about what is attached to a given port (CU, CHPID, or SWITCH). CHP-H indicates that this port is attached to a CHPID being used to collect the FICON CUP data from the FICON director.


For additional detailed information, please review Resource Measurement Facility Report Analysis, SC33-7991-12.


What is frame pacing and what is the difference between frame pacing and frame latency?


Frame Pacing is an FC-4 application data exchange level measurement and/or throttling mechanism. It uses buffer credits to provide a flow control mechanism for FICON to assure delivery of data across the FICON fabric. When all buffer credits for a port are exhausted a frame pacing delay can occur. Frame Latency, on the other hand, is a frame delivery level measurement. It is somewhat akin to measuring frame friction. Each element that handles the frame contributes to this latency measurement (CHPID port, switch/Director, storage port adapter, link distance, etc.). Frame latency is the average amount of time it takes to deliver a frame from the source port to the destination port.


If frame pacing delay is occurring then the buffer credits have reached zero on a port for 1 or more intervals of 2.5 microseconds. Data transmission by the port reporting frame pacing delay ceases until a credit has been added back to the buffer credit counter kept by that port. Frame pacing delay causes unpredictable performance delays. These delays generally result in elongated FICON CONNect time and/or elongated PEND times that show up on the volumes attached to these links. Therefore, when you see abnormally high PEND and CONN metrics, particularly in a multi-site cascaded FICON architecture, one of the first places to look at should be the RMF 74-7 records. 


Figure 2 shows an example of a “clean” FICON Director Activity Report, meaning that none of the ports are reporting frame pacing issues. All ports show an AVG FRAME PACING of “0”.    This is the ideal.  Note that this is a report from a non-cascaded FICON director (there are no ports connected to another SWITCH).  Next, let’s look at a potential problem with frame pacing.




Figure 3 is an example of another RMF 74-7 report from a different FICON director.  As you can see, Ports (PORT ADDR) 27, 29, 2E, 5E, and 5F are all reporting some degree of frame pacing issues of varying magnitudes.  In other words, these ports all stopped transmitting for “x” intervals of 2.5 microsecond duration. This happened because each of these ports, for some reason, had an indication that the port it was connected to had run out of buffer credits. Recall that for two ports connected together, the transmit half of each port is in constant communication with the receive half of its partner port.  The buffer credits based fibre channel flow control mechanism used by FICON storage networks has the transmit half keep track of how many buffer credits its partner port’s receive half has remaining.  When this counter reaches “0”, frame transmission temporarily stops. This in a nutshell is what is reflected in the AVG FRAME PACING field. 


The question then becomes “is everything that is a non-zero value of AVG FRAME PACING” bad?  The answer is the usual “it depends”. Remember, you are likely closely looking at the RMF 74-7 reports because you are doing some performance troubleshooting to determine what is causing an abnormally high response time.  You noticed higher than normal PEND or CONN time.  Most of us do not look at the RMF 74-7 report as a first step.  If the ports associated with the devices exhibiting abnormally high PEND and/or CONN times are showing non-zero values for AVG FRAME PACING, you likely have narrowed down the problem.  As a next step, you should do some trend analysis by looking at a series of the RMF 74-7 reports for this specific FICON director to determine if the frame pacing issue occurs consistently, and are the values getting worse?


As a personal rule of thumb, the author looks for AVG FRAME PACING values that are >100.  Rarely have I seen AVG FRAME PACING values <100 as cause for concern.  It deserves scrutiny and looking for a trend to see if it becomes worse.  Values greater than 100 such as shown by port 29, 2E, and especially 5F in this example deserve further analysis.  Port 5F is attached to a control unit (CU) so this is likely a slow drain device issue on the storage host/fibre adapter.  This is when we would turn to the ESS Link Statistics report for further analysis (which will be the subject of the next article in this series). 


The one exception to the aforementioned rule of thumb is if the port exhibiting AVG FRAME PACING >0 is for interswitch links (ISLs) connecting cascaded FICON directors.  An example is illustrated in Figure 4.  Port (address) 25 shows an AVG FRAME PACING value of 570.  Since we know that this port is attached to a port on another FICON director, there are more troubleshooting options available outside of RMF such as using the FICON director management software and/or Command Line Interface (CLI).  It may be something as simple as not having enough buffer credits configured on the port attached to port 25 in this example.  In which case, the port buffer credit configuration can be altered. Since ISLs are typically used for remote data replication such as PPRC or XRC, any frame pacing delay is cause for concern.




This article has explained how you can use the FICON Director Activity Report (RMF 74-7) to drill down further into a I/O performance problem. It provides a valuable way to narrow down the potential root cause(s), but is not a one stop place to completely solve the problem you are troubleshooting. Problems such as slow drain devices need further examination which can be done using the ESS Link Statistics Report. Fabric contention issues such as frame pacing delay being exhibited on ISLs should also be examined further with the IBM z/OS I/O health check mechanism.  That and the ESS Link Statistics report will be discussed in greater detail in future articles. 


I look forward to hearing your questions, comments and concerns.  Thanks for reading!


Dr. Steve

Buffer-to-buffer credit management affects performance over distances; therefore, allocating a sufficient number of buffer credits for long-distance traffic is essential to performance. With FOS 7.1 Brocade introduced an enhancement to our FOS Extended Fabrics Feature:   The portCfgLongDistance CLI command now includes the option to configure the number of buffers by using the -frameSize option command along with the -distance option. Prior to FOS 7.1, the only option was the –distance option.


Buffer Credit Background and Review


To prevent a target device (either host or storage) from being overwhelmed with frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frames. If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire Fibre Channel sequences needing to be retransmitted across the link.


Because the number of buffer credits available for use within each port group is limited, configuring buffer credits for extended links may affect the performance of the other ports in the group used for core-to-edge connections. You must balance the number of long-distance ISL connections and core-to-edge ISL connections within a switch.Buffer-to-buffer (BB) credit flow control is implemented to limit the amount of data that a port may send, and is based on the number and size of the frames sent from that port. Buffer credits represent finite physical-port memory. Within a fabric, each port may have a different number of buffer credits. Within a connection, each side may have a different number of buffer credits...


Brocade Vyatta Controller 2.0 has been released!