I’ve spent the last few months working closely with the OpenDaylight and OpenStack developer teams here at Brocade and I’ve gained a heightened appreciation for how hard it is to turn a giant pile of source code from an open source project into something that customers can deploy and rely on.
Not to criticize open source in any way – it’s a great thing. These new open source projects in the networking industry, such as OpenDaylight, OpenNFV and OpenStack are going to do great things to advance networking technology.
No, it’s just the day to day grind of delivering a real product that challenges our team every day.
On any given day, when we are trying to build the code, we’ll get new random errors and in many cases it’s not immediately obvious where the problem is. In another test we’ll get unexpected compatibility problems between different controller elements. Again, somebody made a change and you can’t trace the problem. On some days, certain features will stop working for no known reason. Because of the above, we need to continuously update and revise test automation and test plans - that is also done daily.
When it comes to debugging a problem, unless you’re working with the source code and regularly navigating to find problems, diagnosis is difficult. Some of the controller internals are extremely complex, for example the MD-SAL. Digging into that to make either enhancements or fixes is not for the faint of heart.
The OpenDaylight controller is actually several projects that must be built separately and then loaded via Karaf. This can be non-intuitive.
Another area of complexity is around managing your own development train. If you’re going to have a non-forked controller that stays very close to the open source, you cannot risk being totally dependent upon the project (for the above reasons and others), and so you basically have to manage a parallel development thread. At the same time, you find problems or want to make minor enhancements that you need in service, but cannot contribute immediately back to the project (that takes some review and time). So you’re left with this problem of branching and re-converging all the time. Balancing the pace of this with the projects pace is a challenge every day.
Then there’s all the maintenance associated with managing our own development thread, supporting internal teams, maintaining and fixing the documentation etc. Contributing or committing code back to the project, when needed, is not a slam dunk either. There is a commit and review process for that. It takes some time and effort.
I think we’ll find the quality of the new Helium release to be significantly better than Hydrogen. Lithium will no doubt be an improvement over Helium and so on. The number of features and capabilities of the controller will also increase rapidly.
But after going through this product development effort the last few months I have a real appreciation for the value that a commercial distribution can bring. And that’s just for the controller itself – what about support, training and so on? Well, I leave those things for another blog.