Last week, HP announced that running a Heartbleed vulnerability scan against its iLO and iLO2 lights-out management interfaces found in older blade and rack mount servers, can cause them to completely lock up. Since the iLO itself controls power to the server, the only way to restore management access is to physically remove power from the chassis so that the iLO gets a cold restart. Ironically, the iLOs are not vulnerable to Heartbleed.
Amidst table-flipping frustration, the ensuing discussion over at Reddit’s r/sysadmin notes that many of the lock ups were caused by concerned users taking a “carpet bombing” approach and scanning their entire internal network. This raises a few good points about best practice out-of-band management.
1. Don’t connect management interfaces such as lights-out server and PDU SNMP cards to your main corporate LAN.
Worst case is they’re still set to use default credentials, leaving the back door wide
open and banging in the breeze. Secondly, they’re often powered by basic microcontrollers that can be easily DOSed by corporate LAN traffic. The third whammy – these interfaces are embedded devices running their own operating systems and as such don’t tend to be as rigorously updated as other systems, so security holes may go unpatched.
To address this, set up a separate management VLAN and as best practice, use a purpose-built out-of-band management appliance as a bastion to authenticate, encrypt and log access to this network.
2. Service processors and lights-out cards like iLO are an invaluable part of out-of-band management, but they’re not the only part.
For comprehensive remote power control use a remotely-switchable PDU, so you can cold restart that server. As best practice, ensure the PDU itself can be controlled out-of-band, e.g. by serial console, for when you need to power cycle the switch between you and the PDU.
Also ensure you have out-of-band remote access to the management network itself, e.g. via redundant WAN and/or PSTN dial-in and/or 3G/4G LTE cellular.
3. Monitor your management network.
Unfortunately, the first that some iLO users will learn of this lock up issue in a few months or a few years from now while they’re responding to an incident that requires emergency management access, only to find they’ve been locked out of the back door.
An out-of-band management appliance can help here, by monitoring that consoles are plugged in and lights-out cards are up, and alerting you directly or via your corporate NMS should your contingency plan need attention.
