Tue, 20 Feb 2007
IPMI
The main problem with servers is the remote access - you should be able to reboot the server (or even change its NVRAM/BIOS settings) remotely, without actually having to walk to the server room. We use various servers from multiple vendors here, so naturally the means of remote access varies. SGI servers have the L1 controller, HP has iLO. For custom-made servers we use a combination of PC with a multiport serial card for a remote access (yes, good mainboards allow the access to the BIOS setup also over the serial console), and a master switch for power cycling locked-up systems. Now with bigger systems with redundant power supplies there is a problem that the server would use three or four sockets in the master switch alone. This is what IMPI is for.
IPMI (intelligent platform management interface) is a standard designed by Intel. It adds a small computer/microcontroller to the main server, and this computer handles things like power cycling the chassis, reading the sensor values, and even a remote serial console. We have ordered two IPMI boards for two of our Tyan-based servers, so I've got a chance to play with it.
Probably the most interesting thing is how the IPMI-over-IP works on some Tyan servers: their IPMI board does not have its own NIC, so it uses one of the NICs on the mainboard. With Broadcom-based NICs they patch the firmware in the network card, so that the card relays the frames with the IPMI board's MAC address to the IPMI board, completely transparently to the running OS. The OS can use the NIC for the original purpose, the firmware just "steals" some frames. I wonder what security nightmares would be possible should the attacker have a way of reprogramming the NIC's firmware.
The good thing is, that IPMI can also be used from the computer itself as well as remotely. Under Linux, IPMItool is the key utility, together with OpenIPMI at the kernel level. So you have instantly available all sensors (without searching for the driver of the sensor chip), hardware event log, etc.
Probably the ugliest thing is the remote serial console access. Serial (i.e. stream by nature) access over UDP? WTF? Also, IPMI probably maintains some
notion of "connection" over UDP, so when one IPMI-over-LAN client crashes
(yes, ipmitool has also its
bugs),
the new client has to deactivate the original session first.
Apart from that, IPMI seems to be quite usable. You cannot ssh to it like iLO, and it does not have the nice features of SGI L1, but it is still worth its price (SMDC 3921 board is priced around 100 Euro here, IIRC).