Wed, 15 Nov 2006
Hunting Ghosts
Today I worked on synchronizing filesystems on some of our high-availability
systems. We use custom-made
rsync-based
setup for checking for differences between filesystems in a cluster.
One of the hosts in a H-A pair has been down for a while because of a faulty
hardware, so I had to manually check whether the changes on the active system
can be propagated to the backup as well. I have synchronized the filesystems,
and switched the load to the newly plugged-in host (because it is faster
than the other one). Just to be sure, I re-ran the checks again,
and was surprised: some files have been different on the new host now.
What was worse, the set of files which were different was a bit suspicious:
bash, login, tcpdump, some other utils
and libraries, including those which are run every time system boots
(such as heartbeat and its libraries). I ran "rpm -V", just to be sure the files
are different
than in the RPM database, but it displayed that all files are OK and well
matching the database. I took the clean RPMs from the FTP file repository,
and the files in question were shorter in the package than on my filesystem.
I thought: are current rootkits so smart that they modify the RPM database,
and so stupid that "ls -l" still can tell the difference?
"rpm -qlv bash|grep /bin/bash" displayed that there was
a different size in the RPM database than in the file itself, yet
"rpm -V bash" said the package was perfectly OK.
Strange. So I suspected the rpm program has been modified
as well (even though it did not show up in the list of modified files).
To prove this, I used strace. On a clean system its output
was shorter, and the difference was that on a modified system
rpm spawned some more threads/processes.
"strace -f" then showed the quilty party - the rpm
command executed prelink on each modified binary.
So I have been hunting ghosts all the time: the files in question have only
not been prelinked yet, or the prelinking info has been overwritten (or not
overwritten, I don't know) by my synchronization scripts.
After running "/etc/cron.daily/prelink" on
a "modified" system both filesystems look the same. Problem solved.
For a long time I wondered
how prelinking can be done without modifying the binary (and thus breaking
the packaging system). The answer for rpm appears to be:
the package manager needs to know about prelinking as well. I have to find
some time to read Jakub's
prelink paper (PDF).
Back to a serious work now.