When some longtime LinkedIn network engineers look back on the company's first automated networking initiative, officially known as "zero-touch provisioning," they jokingly refer to it as "zillion-touch provisioning."
"It was basically a Frankenstein of different codes and scripts packaged together," principal network architect Shawn Zandi said. "It was painful."
But that early pain led to later automation gains for the social media site, based in Mountain View, Calif. -- gains that Zandi said have allowed LinkedIn to dramatically scale its network infrastructure. The company now has a dedicated automation team and requires that every new network engineering hire demonstrate proficiency in automation scripts.
Network automation -- often mentioned in the same breath as software-defined networking but arguably a distinct, if related, concept -- uses code and scripts to trigger and carry out various network tasks in a standardized fashion. Advocates say it increases efficiency, minimizes inconsistencies and reduces the risk of human error. A variety of automation tools have already edged out the traditional command line interface in some networks, such as LinkedIn's.
Since those early days of zillion-touch provisioning, Zandi said his team has continually improved its approach to automation by simplifying infrastructure, standardizing processes and minimizing unique "snowflake" elements in the network. Today, virtually every networking task is completed using scripts.
"People don't go to the individual router or switch and punch commands, because we want to make sure the configurations are intact and that there is integrity and data consistency," Zandi said. "Otherwise, you could have switches with different setups and configurations."
Automation for all
Automated networking isn't limited to web-scale companies like LinkedIn. According to Enterprise Strategy Group analyst Dan Conde, 44% of midmarket organizations -- those with 100 to 999 employees -- said they use network automation. Among large enterprises of 1,000 employees, 56% said they do also.
Dave Chandler, practice director of enterprise network solutions for World Wide Technology (WWT), a third-party reseller, said he sees a rapidly growing interest in tools to automate basic network tasks, such as those from Puppet, Chef and Ansible.
"Those tools don't really require you to change a business process, so they can be implemented very quickly," he said. "We see both small and large companies adopting [them]."
LinkedIn's Zandi stressed that network automation is achievable in any environment, with organizations of all sizes benefitting from consistent, efficient task management.
"Enterprises say, 'We don't have enough resources'; however, once you move to automation, you free up your resources to do more with the people you have," Zandi said.
While conversations about network automation often focus on the configuration of routers and switches, Conde said the big picture is more complex, and automated networking itself enjoys a more pervasive presence than many realize.
"Automation in different guises is sort of sneaking up on us," Conde said. "It's already here; you just may not realize it."
While some vendors -- such as Brocade and Hewlett Packard Enterprise -- have clearly labeled network automation products, others have simply folded automation capabilities into their security, telemetry, analytics or software-defined WAN (SD-WAN) tools.
Shawn Zandiprinicipal network architect, LinkedIn
Cato Networks, for example, uses automated networking elements to ensure plug-and-play provisioning for its SD-WAN customers. Leslie Cothren, IT director of Universal MH/DD/SAS -- a distributed healthcare provider based in Lenoir, N.C. -- said he used to spend much of his time running between 11 branch sites to configure, maintain and troubleshoot connections. Since adopting Cato's technology, however, he can now spin up a new site in a matter of minutes.
"It is a very slick and simple process," he said. "Plug the device in, they set it up ... and connectivity just happens between our branches."
At LinkedIn, Zandi's team also uses automation to collect performance data, delivering detailed insights into network conditions at any given time. Still other in-house automation tools scan for network anomalies, stress test network services and auto-remediate performance problems before they become critical. If light levels in an optic fiber have degraded beyond a pre-established point, for example, LinkedIn's system automatically files a ticket with a data center engineer or technician requesting a repair.
While automation helps reduce the likelihood of human error and can make network managers' lives easier, it isn't foolproof. Experts agree enterprises should proceed with caution.
"[If the automation] goes wrong, it's going to be catastrophic," Zandi said, adding that a major cloud vendor he declined to name recently brought down an entire data center with an automation error, affecting millions of users.
"It is certainly possible to use automation to automate a disaster," WWT's Chandler agreed. Both he and Zandi said organizations should try to rigorously test their code. LinkedIn applies all scripts in a staging environment, for example, before moving them into production.
Chandler added that it's important to build reliable checks into scripts -- so if something does go wrong, an intelligent feedback loop will recognize the error and automatically prompt a course correction.
"Automation has to be used wisely," Zandi said. "It's just yet another tool -- you have to know how to use it."
Those who don't know how to use it might need to learn sooner rather than later. Based on feedback from WWT enterprise customers, Chandler believes the command line interface's days are numbered.
"If a network manager is not willing to learn those new tool sets and those new processes, ... then I think he's a little bit at risk," he said. "If he moves into understanding [APIs and automation scripts], then I think he's going to be successful. There is going to be plenty of work for people to do that."
Network, heal thyself
LinkedIn's automation efforts have come a long way since that early, Frankenstein-like array of code and scripts. Zandi said he now views network infrastructure -- including switches, power, cooling elements and optic fiber -- as "just another set of data points" that can be controlled through code. While some would consider this a cutting-edge automated networking paradigm, LinkedIn has other ideas.
Zandi said the company ultimately envisions a self-healing or "self-defined" network, capable of configuring and maintaining itself automatically without any human intervention. In service of that goal, his team constantly adds new automated services and features while removing legacy equipment and processes.
"Perfection is not when there is nothing left to add, but when there is nothing left to take away," he said, quoting one of the Internet Engineering Task Force's Twelve Networking Truths. "Make it as simple as possible."
Network automation can fix processes, but it can also break them
With network automation, 'set it and forget it'
What NASA and software-defined networking have in common