Yenya's World

Wed, 28 Feb 2007

Security Advice

SlashDot has been running a story referring to the Eric Allman's article on handling security bugs. I thought: the author of sendmail teaching us about secure software, WTF?

Apparently I was not alone, but the following comment is definitely amongst the funniest yet still to-the-point comments I have seen on /. at all:

Getting advice on how to handle security bugs in your software from someone who works on Sendmail

It could be worse; it could be advice on how to write readable code from the person who wrote qmail.

What a shame I don't have moderation points today.

Section: /computers (RSS feed) | Permanent link | 2 writebacks

Mon, 26 Feb 2007

Superlinear Algorithms

A while ago I wrote about the Email::Address Perl module for parsing addresses in the e-mail headers (such as To:, Cc:, etc.). We use it in production system, but on Thursday we have got a problem, which I narrowed to an inoptimality in this module. The mail delivery daemon (which - amongst other things - parses the mail message) seemed to be stuck in an infinite loop.

Later I found out that the message in question contained an overly long To: header - it had more than 300 mail addresses there. I have investigated this further, and it seems that the Email::Address module has complexity far above O(n):

# of addresses	parsing time
60	0.13 s
65	0.13 s
70	0.29 s
75	0.45 s
80	0.79 s
85	0.58 s
90	0.86 s
95	1.63 s
100	2.38 s
105	3.17 s
110	4.02 s
115	5.97 s
120	11.37 s
125	230.89 s
130	1045.07 s

So it was definitely impossible for it to parse a header with >300 addresses in a reasonable amount of time. The alternative module, Mail::Address, is unusable on this particular message as well - it crashes perl with SIGSEGV :-(.

As a workaround, I wanted to truncate the header somewhere, but in order to truncate the header exactly between e-mail addresses, I would have to parse the header. Which is what I has been doing, and it took too much time. A nice Catch 22, isn't it? So I have invented a heuristics where I truncate the header after a comma followed by a newline. When this is not enough, I use just a sole comma as a separator, and when even this is not enough, I truncate the header to a fixed number of characters. So far it seems to be sufficient.

Section: /computers (RSS feed) | Permanent link | 3 writebacks

Fri, 23 Feb 2007

Gentoo Linux

I use J-Pilot as a desktop PIM application for synchronizing with my Palm. Few days ago I found that it apparently leaks memory (J-Pilot bug entry, Fedora bug entry). The response to the J-Pilot bug entry was that the bug cannot be reproduced on the author's Gentoo, so it has to be a Fedora bug. So I have decided to install Gentoo in a virtual machine to test J-Pilot under it.

So far the installation was pretty straightforward, I created a virtual disk image, created a filesystem on it, mounted, unpacked the stage3 tarball and Portage snapshot to this filesystem, and in the rest I have followed the Gentoo Handbook pretty closely (omitting steps which are not necessary for chroot or virtual machine environment, such as building my own kernel). In fact, since the Gentoo install runs in the chroot environment, the virtual machine is not necessary when everything you want to do is to compile and test a single GUI application. So I am compiling the packages in chroot instead of the virtual machine (it is faster that way).

We have some Gentoo servers here, so I am already somewhat familiar with the emerge command and USE flags. So far I did not ran into bigger problems, just few annoyances (some of which may definitely be an user error, as I am new to Gentoo):

The stage3 does not contain a text editor! Every UNIX has to include vi!
The network setup mentions that either ifconfig(8) or ip(8) can be used in the network scripts. I wanted to use the later, but emerge iproute2 has failed because of missing (or different) kernel headers. I had 2.6.20 unpacked in <chroot>/usr/src/linux, as this is what I currently run. I think the dependencies for iproute2 should be fixed.
Gentoo Linux is apparently not FHS-compliant: it has its Portage directory under /usr even though this directory is often written to.
When I tried to run "USE=gtk emerge -p -v jpilot" (or something like that), the dependencies tried to install not only the X libraries (which is expected from the X app), but also the X server package together with ~20 graphics hardware driver packages. I don't need an X server there, it is a headless virtual machine!
The stage3 is huge. The whole system before my first emerge command had 1.6 GB. Of course, 600 MB of it was /usr/portage, but a gigabyte is still a lot for a minimal (essentially empty) system.
The /etc/conf.d sucks. I had to symlink it as /etc/sysconfig, because I tend to use the later name anyway.

Now I am waiting for the compile to finish. When I am at it, I want to explore the inner workings of their startup scripts (rc-update and friends) and other Gentoo specialities as well. I also wonder how they do SELinux and how their default security policy look like.

Section: /computers (RSS feed) | Permanent link | 6 writebacks

Wed, 21 Feb 2007

Device Event Handler

An useless but nice hack of the day: I have explored the udev rules a bit further - with udev, it is possible even to run a script when the device is created. I wrote two simple scripts - one at home, where it loads images from my camera after I plug the camera in, and stores them into my image repository. The other at work - around 6:30pm, I download the main daily news from the Czech radio station Radiožurnál, encode it to OGG/Vorbis, and when my Palm is plugged in, the script started by udev copies the audio file to the Palm. The rule itself is pretty simple:

$ cat /etc/udev/rules.d/60-palm-news.rules
KERNEL=="sd*1", SYSFS{serial}=="50xxxxxxxxxxxxxxxxxxxx39", \
    SYSFS{product}=="palmOne Handheld", \
    SYSFS{manufacturer}=="palmOne, Inc.", \
    RUN+="/usr/local/sbin/news-to-palm"

The tricky part was to not interchange double "==" with the single "=" by accident, and using the "KERNEL" parameter (otherwise, the script would be run for every virtual device along the path (USB device, virtual SCSI controller of the mass storage device, the whole USB disk, and finally every partition on that disk).

Another tricky part is to use device nodes from /dev/disk/by-uuid in the mount(8) command, so that the device path remains the same no matter which USB port I plug my PDA into, or what other mass storage devices are currently plugged in.

As an user-friendly bonus, the "news-to-palm" script uses notify-send to send a completion message over D-bus to inform me that I can unplug the PDA.

Update - Thu, 22 Feb 2007: More details

I forgot to mention some important tips:

Sending a notification is done with the following command:
```
su kas -c 'DISPLAY=:0 notify-send "News" "News copied to Palm."'
```
This of course requires that the notification is being sent to user "kas", logged in at the ":0" display. In theory it should be possible to send the notification over the system d-bus (dbus-0), but I did not found a suitable parameter for notify-send(1).
The parameters for the udev rule are easy to find. Just plug the device in, find (looking at the dmesg(8) output) the name of the special file the device currently uses (for example, /dev/sdb), and run the following command:
```
$ udevinfo -a -p `udevinfo -q path -n /dev/sdb`
```
You can use the SYSFS{} parameters not only from the leaf (top-most) node, but from all nodes along the path.

Section: /computers/desktops (RSS feed) | Permanent link | 2 writebacks

Tue, 20 Feb 2007

IPMI

The main problem with servers is the remote access - you should be able to reboot the server (or even change its NVRAM/BIOS settings) remotely, without actually having to walk to the server room. We use various servers from multiple vendors here, so naturally the means of remote access varies. SGI servers have the L1 controller, HP has iLO. For custom-made servers we use a combination of PC with a multiport serial card for a remote access (yes, good mainboards allow the access to the BIOS setup also over the serial console), and a master switch for power cycling locked-up systems. Now with bigger systems with redundant power supplies there is a problem that the server would use three or four sockets in the master switch alone. This is what IMPI is for.

IPMI (intelligent platform management interface) is a standard designed by Intel. It adds a small computer/microcontroller to the main server, and this computer handles things like power cycling the chassis, reading the sensor values, and even a remote serial console. We have ordered two IPMI boards for two of our Tyan-based servers, so I've got a chance to play with it.

Probably the most interesting thing is how the IPMI-over-IP works on some Tyan servers: their IPMI board does not have its own NIC, so it uses one of the NICs on the mainboard. With Broadcom-based NICs they patch the firmware in the network card, so that the card relays the frames with the IPMI board's MAC address to the IPMI board, completely transparently to the running OS. The OS can use the NIC for the original purpose, the firmware just "steals" some frames. I wonder what security nightmares would be possible should the attacker have a way of reprogramming the NIC's firmware.

The good thing is, that IPMI can also be used from the computer itself as well as remotely. Under Linux, IPMItool is the key utility, together with OpenIPMI at the kernel level. So you have instantly available all sensors (without searching for the driver of the sensor chip), hardware event log, etc.

Probably the ugliest thing is the remote serial console access. Serial (i.e. stream by nature) access over UDP? WTF? Also, IPMI probably maintains some notion of "connection" over UDP, so when one IPMI-over-LAN client crashes (yes, ipmitool has also its bugs), the new client has to deactivate the original session first.

Apart from that, IPMI seems to be quite usable. You cannot ssh to it like iLO, and it does not have the nice features of SGI L1, but it is still worth its price (SMDC 3921 board is priced around 100 Euro here, IIRC).

Section: /computers (RSS feed) | Permanent link | 0 writebacks

Mon, 19 Feb 2007

Virtualization Overhead

In a followup to my previous article I want to sum up the speed penalty introduced by the two virtualization systems, KVM and lguest. The measurements are by no means meaningful for everyone, just exact data I have measured. And yes, it sometimes compares apples-to-oranges - read on at your own risk :-)

The test systems were the following:

Pyrrha: Dual AMD Opteron 2220 (4 cores total, 2.8GHz), 6GB RAM, 4-way RAID-10 disk, gigabit LAN, Gentoo Linux/AMD64.
Terminus: KVM-based virtual AMD64 machine running at Pyrrha, 10 GB disk image, 512 MB RAM, bridged LAN to the main Pyrrha's LAN, Fedora 6/AMD64.
Scylla: Pentium 4 3.0GHz HT, 1 GB RAM, single SATA drive, 100baseTx LAN, Fedora Rawhide/IA32.
Glauke: lguest-based virtual IA32 machine running at Scylla, 10 GB disk image, 512 MB RAM, routed/masqueraded LAN to the main Scylla's LAN, Fedora 6/IA32.

Test	Pyrrha	Terminus	KVM overhead	Scylla	Glauke	`lguest` overhead
bc	6.286	5.876	-6.52 %	8.130	8.240	1.35 %
wget	0.441	10.885	2368.25 %	3.732	3.770	1.02 %
tar unpack	15.118	20.322	34.42 %	27.566	40.701	47.65 %
rm -rf	0.538	0.634	17.84 %	0.477	0.640	34.17 %
compile	6.410	21.929	242.11 %	126.005	184.833	46.69 %

The numbers in the above table are in seconds (so lower is better). I ran each test five times and used the lowest time from these five runs. I did not bother to reboot between the tests or switch the system daemons off.

Description of the tests

bc: time sh -c 'echo 2^1000000|bc >/dev/null'
A simple test of a CPU-intensive job. Why is Terminus faster than Pyrrha? Maybe the clock skew inside the guest? Or Gentoo-compiled bc being slower than Fedora-prebuilt one?
wget: time wget -q -O /dev/null ftp://ftp.linux.cz/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2
Network performance. KVM (having to emulate the real-world NIC) is waaay slower. However, Pyrrha has a gigabit NIC, so the baseline is 10 times off. But still, raw bandwidth used for KVM was ~22 Mbit/s, while lguest has filled the Scylla's 100 Mbit pipe without trouble. lguest could be even faster in the future, if they use a bounce buffer bigger than a single page (which is what they use now).
tar unpack and rm: time tar xjf /usr/src/linux-2.6.20.tar.bz2 ; time rm -rf linux-2.6.20.tar.bz2
A simple filesystem-related test. Nothing to see here (KVM is a bit faster).
compile: make clean; time make modules > /dev/null
A simple part of kernel compile. Both the architecture and kernel config was different between Pyrrha+Terminus and Scylla+Glauke, so do not try to compare the absolute times between those two groups. Interestingly enough, KVM was much slower than lguest.

From a subjective point of view, lguest feels faster, which was totally unexpected. I am looking forward to the further development (especially lguest with AMD64 support). Anyway, James Morris did a more in-depth measurements of the lguest network performance.

Section: /computers (RSS feed) | Permanent link | 3 writebacks

Fri, 16 Feb 2007

Virtualization

In the last few days I finally managed to take a look at virtualization systems under Linux. In particular, I have tested two of them: KVM, and Rustyvisor (aka lhype aka lguest).

Firstly the architectural differences: KVM is a full virtualization system - it requires pretty new CPU with virtualization support (Intel VT or AMD-V). With this support, it allows the computer to be fully virtualized (yes, I even tried to install The Other OS Which Should Not Be Named under KVM, and it worked). On the other hand, lguest is pretty minimalistic paravirtualization system, which requires slight modification of both the host and quest kernel - so it is Linux-under-Linux only.

KVM virtualizes the CPU, but to run a complete guest OS, the whole computer needs to be virtualized. KVM uses a patched version of Qemu, which emulates PC with a NE2k compatible NIC, Cirrus Logic VGA card, IDE drive (with a plain file in the host OS as a backing store), etc. Qemu itself can work as a full PC (and PowerPC and Sparc too) emulator, but with KVM, it just provides the necessary virtual components to the already-virtualized CPU. Qemu supports many file formats for the disk image (including Xen, User Mode Linux, VMDK and others). It also has the nice feature that when started with -nographic option, it connects its terminal to the virtual serial port of the quest system, providing a seamless serial console! As for architecture, KVM+Qemu runs on IA32 and AMD64, while the guest system can be 32-bit or 64-bit. The guest system is single-CPU only. Networking can be done using a shared file (so even an ordinary user can run his own set of virtual machines, and connect them together), or with TUN/TAP interface it can communicate with the host OS as well.

lguest is still a proof-of-concept code, but it has some nice features: it works on older CPUs as well (AMD64 support is being developed as well, but now it is 32bit only). It can use memory-to-memory communication between host and quest (so it does not have to emulate the NE2k card, for example). Subjectively lguest was faster and more responsive than KVM. Here are the drawbacks of lguest: its virtual disk is a raw file only, and it even does not support partitioning, so existing Qemu images did not work. It does not "boot" the block device per se, instead it starts an external kernel (in form of a vmlinux file), which then mounts its root.

I will measure the performance penalty of both virtualization systems, and post it to my blog later.

Section: /computers (RSS feed) | Permanent link | 4 writebacks

Thu, 01 Feb 2007

Girls Everywhere

It seems that these days, every university or faculty[*] needs to have an attractive girl in their home page, presumably to attract applicants :-)

We had to follow the suit, of course :-) I brought my tripod, flashlight, and camera, and we spent some time taking photos. Tomáš then did an excellent work when creating the banner from my photos. While the title page and the page for applicants are nice, it still feels kind of weird to see the people I know in person in this banner. Oh, and BTW - this is my laptop they are looking at. To make them smile, I even displayed an episode of Azumanga Daioh there :-)

[*] the ESF page needs to be reloaded few times to actually get an image of a girl.

Section: /world (RSS feed) | Permanent link | 4 writebacks