British Airways: A Cautionary Tale in Data Center Management

BA Data Center failure grounds flightsAn IT engineer on contract to an enterprise data center disconnects and then reconnects the power supply. He isn’t authorized to do so. Whoops. No big deal, right? Wrong.

Last month such a scenario played out at a British Airways data center near Heathrow Airport. A power surge then damaged servers providing major applications for online check-in, baggage handling and customer contact systems.

The resulting outage caused the cancellation of 479 flights and impacted 75,000 customers. Estimated cost to British Airways: 100 to 150 million Euros. Estimated damage to British Airway’s reputation: sky high.

Today, many people in data centers have physical access to all sorts of equipment. Locking it down and enabling remote, out-of-band (OOB) management by trusted personnel is a great way to secure it from intentional or accidental misuse.

Human Error is No Longer a Viable Excuse

Following the British Airways debacle, industry experts weighed in to criticize the excuse that the outage was the result of human error. The Uptime Institute’s president, Lee Kirby, said human error is “an overarching label that describes the outcomes of poor management choices.” He pointed to the redundancy requirements in Tier 3 data centers as an antidote to this kind of mishap.

But increasingly, especially with the growth of edge networks serving the Internet of Things, out-of-band management is gaining in popularity as a way to securely and cost-efficiently control infrastructure wherever configurable network devices exist.

Remote Management in Action

Providing out-of-band access to critical infrastructure for reconfiguration or repair isn’t new. It was introduced in the 1980s. Terminal servers and routers with serial ports were used to access infrastructure remotely. Later, reverse telnet and reverse Secure Shell (SSH) enabled administrators to access gear via serial over Ethernet redirection with CLI and terminal access to device consoles.

Secura data centre uses opengear console servers How can out-of-band access provide secure and cost-effective management in the data center? Cloud hosting provider Secura provides a great case study in the Data Center Journal. With infrastructure in three different Tier 3 data centers in the U.K., the company needed access to it 24 x 7. Sending engineers out for every update, patch or configuration change would have been inefficient and costly. So the company chose console servers with Smart OOB technology.

The out-of-band gear provides monitoring, diagnostics and remediation features to maximize network uptime through efficient recovery from any infrastructure fault condition. Secura has since reduced onsite visits by several days each quarter, more than paying for the solution. If problems arise, the out-of-band management solution is already deployed to quickly help troubleshoot devices.

What OOB Management in the Data Center Can Provide

  • Simplicity and reliability: If there is a problem, the solution should be able to quickly and simply help you determine what is wrong and how to resolve the issue. The latest solutions are very intelligent, containing condition-based features that can automate a recovery process even before an admin gets involved.
  • Automated provisioning: Deploying devices with a zero-touch philosophy will reduce time to deployment and minimize errors.
  • Backup access: If the network connection to your console server goes down, an OOB solution can provide access through Plain Old Telephone Service (POTS), fiber or cellular, to ensure continued connectivity. The state-of-the-art is cellular failover.
  • Logging: Insight into the failure is a key step in resolving the issue.  Without this information it becomes difficult and time consuming leading to longer downtime.

With more and more workers meeting, working and accessing company resources remotely, it’s no wonder that out-of-band management is becoming ever more popular. Add to that security concerns within the physical data center and in edge networks and the benefits of secure OOB management are clear.

For more information on Out-of-band management in the data center, check out Opengear’s eBook “How Resilient is Your Data Center.”