Out-of band (OOB) access to critical infrastructure for reconfiguration or repair was pioneered more than 30 years ago. It began as a DIY solution where engineers used terminal servers, repurposed server computers or routers with serial ports to access their infrastructure. Reverse telnet (later reverse SSH) functionality allowed serial over Ethernet redirection and command line/terminal access to the device console.
Fifteen years ago, OOB experienced a massive transformation resulting from the growth of crammed data closets, machine rooms and sophisticated data centers. Due to the density and wide array of critical IT, networking and power infrastructure, tens, hundreds and thousands of serial consoles needed to be accessed and monitored to keep the corporate IT engine running. To cope with this, software tools like PuTTY, OpenSSH, and Conserver allowed sysadmins to create “console management” solutions running under BSD and Linux on dedicated servers.
The unresolved issue was that these systems had few serial ports for connections to the consoles on the ever increasing number of critical systems being managed. The industry responded with a better “terminal server” that had all this functionality in a secure networking box with 4~48 serial ports — known as a “console server.” Within a few years it became the “industry-standard” OOB solution to reduce the reliance on a warm body logging-in to check system health.
These “industry-standard” devices provided access to a remote console from the desktop, they added port menus to easily locate devices and consoles, and they provided keystroke/character/session logging. This feature could be used to trigger SNMP traps to their NMS and send emails according to severity of console message and signals. New web-CLI terminal emulators were introduced so that browser based access was all that was needed. Inbuilt POTS dial-up modems were introduced for integrated OOB connectivity when the main network was down.
Although there have been changes, the expectations of OOB access have remained stuck in a time warp. Many new routers and M2M devices have added decades-old, simplistic functionality and mistakenly branded it as “OOB management (OOBM)” but in reality, it is “dumb OOB”.
Distributed edge networks
Fast forward to the 21st century where distributed enterprise edge networks are tied to key data and services via the cloud and where the cost of downtime is phenomenal. Sustained outages can bring an enterprise to its knees. These distributed edge networks need to be more resilient even if the data centers that make up the clouds have 5 nines reliability.
How should these distributed sites be managed? Well the holistic approach to managing the entire IT infrastructure at each site is one approach. With sites now being tightly tied to the cloud they rely heavily on Internet/WAN access devices, firewalls, routers and switches. These devices are under severe load both from data throughput, cyber-attacks, firmware exploits, table overflows and adverse environmental conditions, to name a few. Eventually they can lock up or their configuration or entire firmware becomes corrupted and they’re out of service. Now a critical site is offline and expert IT/networking resources located at the distributed site are not available.
Enter Smart OOB™
Smart OOB™ is Opengear’s solution to raise secure remote access and infrastructure management to a new level. Smart OOB™ extends traditional out-of-band management to monitor and log systems health and environmental conditions, proactively detecting faults before they become failures.
Opengear first introduced on-board storage facilities for local logging and back up of router/firewall/switch configuration state allowing an administrator to repair/upgrade configurations or firmware remotely. If a router is offline on the Internet/WAN side you still need to get to it. Today most OOB solutions provide a POTS dial-up modem link. Doing anything serious can be a long exercise over such a slow, high-latency medium. For five years Opengear has provided integrated cellular connectivity (4G, 3G, 2G on AT&T, Verizon and Sprint in the US) providing an over-the-air alternate link to the equipment that needs attention. Not only is this connection fast with low latency, by comparison try getting a new copper line for OOB provisioned — it can cause a lot of pain and cost. Cellular is ubiquitous supporting Internet and private network connectivity.
What if the router is completely unresponsive on the local network or even the local console port? Then flicking the AC switch is the only alternative. For decades, network or serial accessible AC power distribution/reboot switches have been available, so an administrator can send commands to recycle power to all sorts of equipment. Opengear again raised the bar by incorporating support for the widest range of power distribution switches making it virtually vendor agnostic.
Other outages? Take AC mains power for example; you don’t want active complex systems to lose power randomly through critical computations, file transfers and so on. Opengear incorporated support for the widest range of power backup/UPS systems from scores of vendors. Therefore graceful power-downs can be facilitated.
We are now all too familiar with natural disasters and their effect on business. What about less monumental issues like air-con failure, smoke, water leakage? Smart OOB™ also monitors temperature, humidity, smoke, water and more.
Business continuity at the hardware layer
Smart OOB™ solutions can reduce business disruptions by monitoring and managing infrastructure during outages. But how much effort does it take to recover from known outages? Do you need a team of experts to figure out what to do? To address this, Opengear developed an auto-response system in the OOB device. It can self-heal using diagnosis and remediation utilities for common problems. Problems can be proactively solved before they become critical outages. This level of automation effectively installs a virtual network administrator at each distributed site, not just to optimize MTTR with automatic recovery scripts, but to mitigate human error and cyber sabotage. Simple tasks can be performed automatically, like detecting a router is down then automatically sending an alert (SMS or e-mail) and finally power cycling it to get it back online. A popular remediation action is to configure the OOB device to shutdown critical equipment gracefully when the UPS detects main AC failure or the temperature in the rack is too high and possibly load shed battery power amongst the most critical equipment — all without human intervention.
Resilient distributed networks
Opengear’s Smart OOB™ solutions make networks in distributed sites resilient to typical and complex outages providing affordable business continuity at the hardware-layer, normally the realm of data centers. Opengear solutions can fail-over the main wired/fiber connection to high-speed 4G-LTE for sustained periods while the failed equipment or main carrier service is being repaired. Other 4G link fail-over solutions can’t re-establish the primary link automatically, repair its configuration or download new router firmware at the remote site to recover it.
To orchestrate Smart OOB™ from a central location with visibility into distributed enterprise sites, branch offices, kiosks, replicated site and pop-up offices, Opengear developed the Lighthouse central management appliance. It consolidates access to critical infrastructure (consoles and other managed devices) scattered across tens, hundreds or thousands of locations. Opengear OOB appliances call-home to Lighthouse and make their attached devices available for management over secure primary management LAN, 3G or 4G LTE or dial-out. Unlike other solutions, alerts and smart responses are performed locally on the OOB appliance. This eliminates the sogginess of solutions that hammer the management cloud with continual small updates exacerbating latency and eliminating any possibility of near real-time responses.
When you’re next considering an OOB solution for critical sites, what price do you put on the quality of the solution compared to the damage the various types of outages can cause to the businesses? Don’t settle for “dumb” or “standard” OOB solutions when Smart OOB™ from Opengear provides complete site resiliency (network, server, power, environmental, storage) and reduced downtime.
