The Open Networking Foundation announced an OpenFlow Conformance Testing Program that validates vendor implementations of the SDN protocol in switches, routers and network software. However, conformance won’t necessarily mean interoperability between OpenFlow equipment.
"We have scores of test cases that are embedded in the OpenFlow 1.1.1 test harness and we will have hundreds in 1.3 as the specification becomes more complex and the behaviors that OpenFlow manages and specifies become more complex," said Rick Bauer, the ONF's technical program manager. Bauer did not say when the ONF would offer conformance testing for OpenFlow 1.3.
The conformance program validates that vendors have implemented OpenFlow properly, but it does not guarantee that all those implementations are equal.
"People bandy about certification and other terms, and I think what happens is the end user thinks they're going to get a memory stick-like experience from any switch they buy from any vendor," Bauer said. "We need to start somewhere and help customers understand that OpenFlow is a great networking protocol and customers need to start asking for OpenFlow-conformant devices."
"I do think this is a good thing, because it sets some sort of expectation for vendors," said Brent Salisbury, network architect at the University of Kentucky. "If customers start asking for this, that's a good thing. The question is, will they?"
Salisbury said many engineers don't trust what vendors are saying about their Openflow implementations right now.
Network engineers are struggling to navigate the nascent OpenFlow market, according to Andre Kindness, senior analyst at Forrester Research. Today, buyers don't know how to compare the OpenFlow capabilities of different vendors, and no one seems ready to help.
"A lot of [vendors] using marketing terms like OpenFlow-capable and OpenFlow-supported, yet it's so hard to know what the vendors are actually doing. A lot of the features might be in software while some are in hardware," Kindness said. "[Conformance testing] doesn't solve everything, but it's a start. This will eliminate some of the confusion out there.
Will OpenFlow testing mean interoperability? Eventually
The OpenFlow testing program is just the ONF's latest effort to push forward the vendor ecosystem for OpenFlow-based SDN, said Brad Casemore, research director for IDC.
"With OpenFlow there has been inconsistent support from the vendor community in terms of the range of product they offer and within the switch products themselves," he said. "They've implemented [OpenFlow] in different ways. So the ONF is trying to make sure there is conformance testing and standards, and they hope vendors will see it is in their self-interest to participate."
Yet OpenFlow conformance does not guarantee interoperability, Bauer said.
"End users have to be careful with user experience. Deploying one piece of OpenFlow-conformant gear from every vendor is not the right way to go. Maybe there is a table-tying aspect to one hard switch that is different from another vendor's hard switch. They may have different ways of table routing."
The ONF does want to see interoperability testing and validation emerge in the market as well, Bauer said. Some of that interoperability testing is already happening in an ad hoc manner at the ONF's OpenFlow plugfests.
"When you have twenty-plus companies there for a week, you have a fair amount of casual 'Let's see how this switch works with that switch and that controller,'" Bauer said. He did not say whether the ONF would take the lead on an interoperability program.
The Indiana Center for Network Translational Research and Education (InCNTRE) is the first independent lab approved by the ONF to perform OpenFlow Conformance testing. Bauer said his organization is working with several other labs to get them into the program.
"We don’t' want to own the testing process," he said. "We see labs being able to provide and compete for that business. Hopefully those same labs will step up when interoperability testing becomes more important, more viable and more doable."
Kindness said the ONF needs to adopt an approach like that of the Wi-Fi Alliance, a non-profit industry group that certifies Wi-FI technology implementations by various vendors. The Alliance was able to foster a market for wireless LAN infrastructure by certifying pre-standard implementations of Wi-FI. This strategy helped boost demand for next-generation Wi-Fi standards before the IEEE had ratified them.
"I said [to the ONF] produce a little OpenFlow logo to go on certified switches. It would help adoption but also set a baseline for a way to look at the different products out there. If you look at market today, it's impossible to do an apples-to-apples comparison of OpenFlow switches and controllers," he said.