Why people hate Microsoft…

I found this nice article by Jeremy Allison, developer of Samba, the widely used *nix implemtation of Microsoft’s CIFS protocol. I think it illustrates very well why lots of people started hating Microsoft. Apparently this company is all for standards if it’s in its own interests in order to defeat the competition, but once it is in a leading position itself, everything will be done to kill any standardisation and openness…

http://tuxdeluxe.org/node/255

New graphical template

I updated this blog to Serendipity 1.2 (svn branch snapshot) and set up a new graphical theme: bulletproof. There is still some tweaking left (reinstallation of some plug-ins, probably just like before I will still need to patch Serendipity a bit to play well with multilingual posts, etc…), but for now, I am very happy with the result.

Now I should try to blog a little bit more than the last few months :-)

Clamav is great

Like a lot of people, I use the free anti-virus program Clamav on my mail server. Last week, I was seriously impressed with its performance.

It started last wednesday, 25 July. At about noon, I received a mail by amavisd-new that it had blocked an e-mail containing a virus, Trojan.Downloader-11827. What was strange, is that I received this message on an e-mail account which is protected by my ISPs proprietary anti-virus solution. So it had not caught this virus, while Clamav did. Then I submitted the file to virustotal.com, and apparently only a few (about five) anti-virus programs detected the virus. Amongst others, Kaspersky, F-Secure, NOD32, Bitdefender, Symantec and of course Clamav. In the clamav-virusdb mailing list archives, I found that Clamav had detection for this virus since 7h21 CEST, so it was really amongs the first to detect this virus.

Then friday evening, I was looking at the blocked spam messages (I use spamassassin too on this server), and noticed that it had blocked an e-mail message containing an exe file. A spam message with an exe file, that sounded suspicious, but Clamav could not detect a virus. Again I submitted the file to virustotal, and there was one positive result: Ikarus detected it as a trojan horse. I submitted the file via clamav’s website at around 19h CEST. About half an hour later, I received a message that detection for this virus had been added. I updated Clamav, and indeed, it was recognized. I checked the file again on virustotal, around 20h, and then there were 4 anti-virus programs recognizing it: Clamav, F-Prot Ikarus and Virusbuster. 1.5h later, Antivir, AVG and Kaspersky had also added detection. Other well-known anti-virus vendors, still did not detect it at that moment such as Bitdefender, F-Secure, NOD32, Panda, Sophos and Symantec.

In the meantime, F-Secure blogged about these two virus outbreaks: funny.zip and fungame.zip

Two conclusions:

  • Clamav has an excellent response time, which is comparable to the best proprietary anti-virus solutions. If you have an e-mail server, you definitely want to integrate Clamav in it, even if you already have a proprietary solution (Clamav is particulary good in detecting phishing mails too!)
  • No anti-virus program is perfect. If you receive an e-mail message at the start of a virus outbreak, it’s quite possible that your anti-virus solution will not detect it yet, no matter which anti-virus you have.

The end of the CK kernel patch set

Today kernel developer Con Kolivas announced that he will stop developing his Linux patch which improves desktop performance. For people who have followed recent discussions about his SD CPU scheduler and about the inclusion of his swap prefetching patches in the Linux kernel this will not come as a surprise.

The CK patch set was popular especially amongst desktop users who want to get maximum performance out of their machine. The CK kernel came with a different CPU scheduler (first Staircase, later SD), which improves the smoothness of desktop applications (for example no more sound stuttering), the mapped watermark patches, which makes the OS use less swap, and the swap prefetching patches, which makes the system more responsive after a memory hungry application caused others to be temporarily swapped out. The CK patch set was also used in several distro kernels, such as the Mandriva’s tmb kernel and kernels in Gentoo and Arch Linux.

The decision to completely stop kernel development, came after the critical reactions by other kernel developers about the SD scheduler and swap prefetching. After the first releases of the SD kernel, some developers preferred trolling instead of helping out to fix the problems which existed at that time. While the SD scheduler slowly became more and more stable, only thanks to Con Kolivas efforts, a competing scheduler (CFS) which was based on the same concepts, was started. Now that both schedulers are mature and stable, a lot of CK kernel users and Con Kolivas himself are left wondering why it was even necessary to start competing with SD, instead of uniting all powers to make one great scheduler.

Swap prefetching was already proposed for inclusion in the Linux kernel a long time ago. But several developers remained critical, while a lots of users reported improvements by these patches. The patches were included in the mm kernel, but developers did not really review it and proposed it for the mainline kernel. Until Ingo Molnar finally stepped up recently, and gave some positive comments after a code review. Again some developers started criticizing the patch, and the future of this patch became again unclear.

With all this in mind, it’s normal that Con Kolivas got fed up with Linux kernel development. It seems some Linux developers really need to do something to improve their communication, and need to be a bit more reasonable and constructive, instead of immediately criticizing one’s efforts. This is at least the second kernel developer who got fed up with the way the Linux kernel development goes in a short time.

Developers come and go, that’s a normal process. Still I think Con Kolivas’ departure could have been avoided. In the end, we can only thank him for his great work, which certainly was not useless. In the end, the CFS scheduler which will be included in Linux owes a lot to Con Kolivas’ ideas, and I hope the other patches will find their way to inclusion in other patch sets in way or another.

Resistance is futile, you will be packaged!

Today, I had again the honour to work with an operating system which is not based around a package manager. The victim: Mac OS X Server. It’s a brand new Mac Pro machine being used as a mail and web server.

Mac OS X Server already comes with most software for configuring a web and mail server included, and has graphical configuration tools. Postfix, Cyrus, Amavisd-new, Spamassassin, Clamav, Apache HTTPD, etc, are all there by default, and easy to configure. Sounds great? Wait a minute…

The problem is that the versions included, are really old, even completely outdated. Let’s take Apache. The version included is some 1.3 version. If you need Subversion running on Apache with Webdav support, then you need at least Apache 2.0… Clamav? the included version is some 0.88 version, which cannot use today’s virusdb updates anymore. That makes Clamav completely useless… Spamassassin? You have the outdated version 3.0.1, hardly impressive if you need to filter’s todays spammer’s creations.

So to make your system really useful, you have to compile a lot of programs by hand. On the system there was Macports installed, and Perl version 5.8 (not sure if it came like this by default, or someone else installed these on this machine before I touched it). So I installed Spamassassin with Perl 5.8 CPAN. All went fine. Let’s restart amavisd, and it will be using the new spamassassin, right? Wrong! Amavisd-new itself is a Perl program, and does not use the spamassassin or spamc binaries, but directly accesses the Spamassassin Perl module. amavisd-new was still using Perl 5.6 as installed by default in Mac OS X, while Spamassassin used Perl 5.8 from Macports, so amavisd-new only found the old Spamassassin in @INC. That should not be too difficult to fix: let’s just change the shebang in amavisd-new, so it uses Perl 5.8 in /opt/local/bin. I restart amavisd, and got a lot of errors of missing Perl modules. By trial and error (read: installing dependency, trying to start amavisd, getting new error, installing dependency,….), I succeed in the end in installing all its dependencies, and amavisd starts fine now. A bit later, new mail starts arriving, and this causes weird errors in the amavisd log (something about wrong file handles). Huh? Well, the amavisd-new included in Mac OS X is very old (from 2004 or 2005 if I remember correctly). Maybe it simply does not work with Perl 5.8?

So now I had to upgrade amavisd-new too… Fortunately some great documentation on the web helped me a lot. Again I had to install some Perl dependencies with cpan, I had to patch amavisd-new for Mac OS X as instructed in the guide, and I had to recreate a new amavisd.conf file. But in the end, I finally had a working amavisd-new installation.

But we don’t have finished yet! Now it seems mails are not scanned anymore with Razor2, although it is installed by default in Mac OS X and I have activated it in my Spamassassin 3.2 configuration… Well it’s the same problem again: Razor2 is installed in Perl 5.6 @INC, but not in Perl 5.8. So again I had to grab the sources and install it by hand, to make it work. While at it, I also compiled Pyzor and dcc-client. And I created a little cronjob wich uses sa-update to grab new rules from SARE.

So, after several hours of work, I think I finally have an adequate working spam filtering system on Mac OS X Server. On an operating system with a good package manager and enough available packages, such as Debian or Mandriva, this would have cost me about an hour at most. Operating systems like Mac OS X, Slackware and others which lack a complete and well integrated packaging system and ditto repositories, really make this a terrible experience. Avoid them if you can!

Virtualbox 1.4

Only one week after I had no success with running Virtualbox on my Athlon 64 system, a new version was announced. One of the important changes in Virtualbox 1.4 is support for AMD64 hosts, so this seemed exactly what I was looking for! To test new distributions and software, I have already been using VMWare Server for some time, which is free (read: it costs nothing), but a real Free (as in free speech) virtualisation solution always sounds interesting, especially as Fedora 7 always crashed VMWare Server and my host.

Installation of Virtualbox was very easy. It has been packaged and integrated in Mandriva, so a simple “urpmi virtualbox” sufficed to install it. Already a lot easier than VMWare Server, which comes in different RPM and ZIP files you have to download and extract. There was no hassle with licences, as Virtualbox is released under the GPL unlike VMWare Server for which you need to register on the site to request a licence key.

The kernel modules for Virtualbox were automatically built with dkms. This time, there were no problems with my x86_64 2.6.21-tmb kernel! Again this was easier than in VMWare, which often needs the installation of an extra patch if you are running a recent kernel.

Configuration is a bit different than VMWare, but actually very easy. The only thing which seems more complex than VMWare, is configuration of bridged networking, i.e. if you want to integrate your virtual machine directly in your real network like a real physical machine. According to the documentation it requires some manual bridge configuration on the host, but I did not try this. For simple NAT networking, I had not to do anything, this worked out of the box and was sufficient for me.

Virtualbox supports everything you would expect from a modern virtualisation system: ACPI, networking, cd/dvd drives (you can access a physical drive or use an ISO file, like VMWare) and sound. The sound implementation in Virtualbox is even better than VMWare, as it can use both OSS and Alsa. With VMWare I never succeeded in having working sound, because I’m using Alsa, and VMWare always complained that /dev/snd was in use. With Virtualbox and Alsa, everything is working great now. Virtualbox also supports creation of snapshots. In VMWare Server you can only create one snapshot, if you need to create more, you have to pay for another edition. Did I say that Virtualbox has everything you would expect? Well, maybe that’s not true. There’s one important thing missing: unfortunately it does not have USB support. This is an important omission which I hope will be added soon, as this works great in VMWare.

Unlike VMWare, Virtualbox does not have any problem with the fact that I am using frequency scaling on my processor (AMD’s Cool’n’Quiet with the powernowd daemon in Linux). In VMWare I had to disable frequency scaling, otherwise the clock of the virtual machine went too fast or too slow most of the time. But not with Virtualbox!

Virtualbox uses a nice QT interface, which integrates very well in a KDE environment. I don’t like QT’s open and save dialogs too much, but as this is a virtualisation product, and not a document editor, you won’t need these too much, so I can live with that. Virtualbox can use VMWare images, but unfortunately it is still not so easy to import your VMWare virtual machines as the virtual hardware is different. My Mandriva 2007 Spring installation in VMWare did not succeed to mount the root partition in Virtualbox, because of the different hard drive controller. With a rescue CD and some manual regeneration of the initrd, it should be possible to overcome this problem though.

Performance of Virtualbox is good. It feels at least as fast as VMWare, so there are no bad surprises here. Virtualbox is more of a workstation virtualisation product though. Unlike VMWare Server, you cannot run virtual machines in the background, and connect to the virtualisation server from the network. At least, I did not see this functionality.

So, in the end I have to say I like Virtualbox a lot! It has a lot of advantages to VMWare Server: it has better sound support, better time keeping, creation of snapshots and generally is a bit easier to install and configure. And it installs Fedora 7 without crashing my machine! If you need USB support or a client-server virtualisation solution, you still have to take a look at VMWare Server though.

Good things ahead!

Today I got an account on Mandriva’s build cluster! This means it will be easier for me to submit RPM packages for inclusion in the distribution. I’ve still got a lot to learn, but with some reading on the wiki and the greatly appreciated help from Dvalin, this will work out fine in the end. Currently working on a package for DrScheme, which is a Scheme IDE also used at university here.

Virtualbox released version 1.4.0 of their virtualisation software today. Especially interesting is that they added AMD64 support according to the changelog. This will probably fix the problems I was experiencing a few days ago when trying Virtualbox on my Athlon 64 machine.

At work, I’m currently installing a nice new server consisting of four dual core Opteron CPUs with 16 GB of memory :-) It will be used for running virtual machines (not with Virtualbox, but OpenVZ). Also a new version of the Linux clustering software Kerrighed was released, which I should definitely try out on one of the clusters at work, because the previous version was not much of a success (it just crashed when activating the cluster).

Other good news, I finally fixed my summer holidays. Now I really should start planning what I will do then. Gentse Feesten will of course be high on the list :-)

Liberation fonts

I quickly redid some of the font settings in the CSS file of this blog. This blog is now using the Liberation fonts!

Packages for these new True Type Fonts are available for all kind of OSes. Those using Mandriva Cooker, can install the fonts-ttf-liberation package with their favourite package manager (urpmi, rpmdrake, smart). Mandriva 2007.1 Spring users, can download this backported RPM package (SRPM available too).

I also changed the font size used in the blog a bit. The template was often using small, x-small and even xx-small fonts, which was a bit too small for my taste. Now it should all be a bit more readable I hope. Let me know what you think of it!

Virtualisation mess

I have downloaded the Fedora 7 installation DVD ISO and wanted to give it a try in a virtual machine. I am using VMWare Server already for some time as it was the first free (as in free beer) available feature-complete and fast virtualisation software. I Created a virtual disk, configured the ISO as source for the CD device, and started up the virtual machine. But then while booting the Fedora 7 kernel, VMWare just crashed, also making my host OS unstable, so I had to do a hard reset. I was still using VMWare Server 1.0.1, so I tried an upgrade to 1.0.2 with latest vmware-any-any patch, but all to no avail: VMWare just keeps on crashing.

Now there’s also Virtualbox, which is freely (as in free speech!) available, so this seemed like an excellent time to give it a try. Virtualbox is packaged for Mandriva, so urpmi virtualbox should suffice to install it. It automatically installs some dkms-virtualbox package, probably containing drivers for virtual network cards and such, like VMWare does too. But while compiling these modules, it bombed out with some compilation errors, and a warning that Virtualbox is not tested with kernels > 2.6.17. As I’m using 2.6.21 x86_64 tmb kernel, and I did not immediately find a reference to this error on Google, I’m stuck here I’m afraid. Let’s hope new versions of VMWare Server or Virtualbox fix these issues soon. In the meantime, I’ll continue to use VMWare Server.

Linux kernel development thoughts

One week ago, kernel hacker Ingo Molnar reviewed Con Kolivas’ swap prefetch patches and approved them to be added to the official Linux kernel. Swap prefetching is a technique, which will load swapped out memory pages back in memory if the system is idle and memory has become available. This is useful for people starting temporary jobs which use a lot of memory, which makes other processes move to swap. Once the memory hungry process is finished, swap prefetching will kick in and slowly reload the swapped out pages back to memory, so that the system potentially does not need to do this anymore when the user again uses one of the swapped out processes. Tjos functionality can be enabled and disabled at will during compilation of the kernel.

Swap prefetching has been available for a long time in Con Kolivas patch set, and was also added to the mm development kernels some time ago. In that period, no bugs have been reported, and it seems people are happy with this feature. So it was hoped that the push by Ingo Molnar, would finally make swap prefetching available for all Linux users in version 2.6.22.

Developer Nick Piggin seemed rather critical of the swap prefetching feature. I I have understood the thread correctly, he discovered a problem (actually swap prefetching did not seem to work anymore because of some unrelated changes in Linux), and there was some serious disagreement how it should be dealt with. Con Kolivas’ got fed up with the unreasonable objections to the patch and proposed to dump the patch completely. So this is yet another performance improving feature we won’t be seeing in the Linux kernel for some time…

Update 12 May 2007: Con Kolivas posted an updated version of swap prefetch, addressing some problems, and the patch has not been dropped from the mm kernel. Maybe things do not look so bad after all?

This whole story makes me remind of the recent RSDL/SD scheduler. Con Kolivas implements a feature which is clearly working very good for most people, and then someone comes up who vetoes it for whatever reason, and in the case of SD other implementations are started (which still do not have the same maturity as SD). In the end a lot of work seems to duplicated and interesting features are delayed or even cancelled completely. Is it me, or is the behaviour of some kernel developers really hurting Linux development?

Also in related news, kernel hacker Adrian Bunk decided to not track regressions anymore. He was tired of Linux 2.6.21 being released with lots of known regressions. It seems Linux kernel development has really some issues these days…