Leap second causing ksoftirqd and java to use lots of cpu time

Today there was a leap second at 23:59:60 UTC. On one of my systems, this caused a high CPU load starting from around 02h00 GMT+2 (which corresponds with the time of the leap second). ksoftirqd and some java (glassfish) process where using lots of CPU time. This system was running Debian Squeeze with kernel 2.6.32-45. The problem is very easy to fix: just run

# date -s "`date`"

and everything will be fine again. I found this solution on the Linux Kernel Mailing List: http://marc.info/?l=linux-kernel&m=134113389621450&w=2. Apparently a similar problem can happen with Firefox, Thunderbird, Chrome/Chromium, Java, Mysql, Virtualbox and probably other processes.

I was a bit suprised that this problem only happened on this particular machine, because I have several other servers running similar kernel versions.

Linux performance improvements

Two years ago I wrote an article presenting some Linux performance improvements. These performance improvements are still valid, but it is time to talk about some new improvements available. As I am using Debian now, I will focus on that distribution, but you should be able to easily implement these things on other distributions too. Some of these improvements are best suited for desktop systems, other for server systems and some are useful for both. Continue reading “Linux performance improvements”

Linux kernel: The battle of the CPU schedulers

Since some time already, different patches are being written for the Linux kernel, which improve the CPU scheduler. The CPU scheduler, is that part of the kernel, that’s responsible for assigning CPU time to the different task running on your system. If you sometimes experience problems with sound stuttering or your mouse becoming jerky while running other CPU intensive tasks, then this is definitely a problem caused by the task scheduler.

Con Kolivas has been maintaining an alternative scheduler for some time. His Staircase scheduler was designed with interactivity in mind, especially for desktop systems, where people want their system to be quickly responsive under all kinds of workloads. This scheduler has been optimized a lot through the years, and as such is very stable. Still there are some rare cases where “starvation” is possible.

At the start of March, Con Kolivas published a new scheduler, which was called RSDL (Rotating Staircase DeadLine scheduler) at first, and has been renamed to SD (Staircase Deadline) afterwards. Based on the experience Con Kolivas gathered with his Staircase scheduler, SD is a more general purpose scheduler, trying to give absolute fairness to the different running tasks, without favouring any process (for example lots of other schedulers favour X). This way, no starvation issues should be possible with this scheduler. A lot of discussion followed after his announcement, and it became quickly clear that a lot of people were not happy with the current scheduler in Linux. Important kernel developers like Mike Galbraith, Nick Piggin, Ingo Molnar, Willy Tarreau and Andrew Morton joined the discussion and also posted other scheduler patches, sometimes not without some trolling and flaming as sometimes happen on such mailing lists. Con Kolivas’ scheduler was added to Andrew Morton’s mm kernel tree to get some more testing. The development of RDSL/SD went up and down sometimes, because of Con Kolivas’ health problems.

Ingo Molnar, which was rather critical of some of the ideas in the new scheduler at first, also recently began the development of CFS (Completely Fair Scheduler, which actually is based on the same basic concept of fairness. Con Kolivas announced that he would stop development of the RD scheduler, because of his health problems, and because his ideas would now continue to be used in the CFS scheduler. But things came out differently, and Con Kolivas continued development in the end. The result is that the SD scheduler is now at its 46’th version (v. 0.46), and it seems most problems have been fixed. Based on all the testing done on the kernel mailing list, it seems SD 0.46 is more mature than CFS 6. Even Willy Tarreau, maintainer of the 2.4 Linux kernel tree, said that thanks to SD, he did make Linux 2.6 the default kernel on his laptop, as he found the scheduler in mainline 2.6 too bad compared to 2.4. It’s unclear however which of these schedulers will be integrated in linux finally, and when this will happen.

Personally I think SD 0.46 should be integrated now in Linux 2.6.22 pre-releases. There has been a lot of testing and bug fixing, and it seems there are no serious bugs open anymore now. I also hope that Mandriva 2008 will come out with one of these new schedulers. The tmb kernel in Mandriva Cooker, already uses the SD scheduler now. People interested in this discussion, can subscribe to the ck mailing list where a lot of the discussion is happening. Sites like LWN.net and Kerneltrap also often post about the progress of this subject.

Struggling with Linux’ OOM killer when building RPMs

Last two weeks, I have created a lot of updated packages for Mandriva 2007.1. I packaged Gnome 2.18.1, and also updated subversion snaphots of kdepim and kdegraphics. Kdepim, because it has received a lot of bug fixing love the last two months, and kdegraphics, because it contains kpdf using new xpdf code, which should be compatible with PDF 1.6 and 1.7 specifications. In kdepim, they also removed the kitchensynk tool, which offered synchronization options with external devices. Apparently it was too buggy to be really useful. I have the impression that Kmail is indeed also more stable than in 3.5.6. I could not reproduce yet the hangs I sometimes experienced with 3.5.6.

Compiling kdepim on an AMD64 system, seems to require a huge amount of memory (much more than on x86 32 bit). 1 GB of RAM and about 250 MB of swap did not prevent g++ eating up all of my memory (even when no other services were active!). Unfortunately, this made Linux become completely unstable: the hard drive started thrashing the whole time, and the system was completely unresponsive. I could only stop it by doing a hard reset. I am clearly not the only one hating this stupid Linux behaviour.

In the Mandriva Cooker channel (irc.freenode.org, #mandriva-cooker), couriousous suggested to execute

# echo 2 > /proc/sys/vm/overcommit_memory

The default value is 0: when an application asks more memory than is available, Linux will still try to allocate it, even if chances exist that it won’t be available. This seems to be done because some applications ask more memory than what they will really use. If in the end, no more memory or swap is availed, Linux’ OOM killer will kill some (random) processes to free up memory. By setting overcommit_memory to 2, the allocation of too much memory will fail immediately. The application can then react itself to the fact that not enough memory is available. The result was that instead of bringing my system to death, the g++ compiler just exited with the message that I was out of memory. Much nicer! I found a complete technical explanation about memory allocation in Linux on the web. To make this setting default, I put vm.overcommit_memory = 2 in my /etc/sysctl.conf. I do not understand why it is not the default value, making a system unresponsive for several (tens of) minutes does not seem very friendly…

Update 16 april 2007: It seems like this setting has severe problems as well. On my system with 384 MB RAM, which I use as server and desktop, several applications randomly crashed because they did not get the memory they wanted (Evolution, Tilda,…). Changing the setting back to 0, made these applications work correctly again. I suppose playing with the overcommit_ratio value as explained in the article which I mentioned above, can improve this behaviour. But anyway, it sucks that such things are so difficult to get right. This should really be working nicely out of the box

Anyway, in the end I got kdepim compiled on AMD64 by adding some more swap. From now on, I’ll be creating bigger swap partitions is Linux, it can realy be useful, even when you think you have enough memory…

I also built a freetype 2.3.4 RPM for Mandriva. Font rendering on my flat panel is now much nicer comparing with freetype 2.3.1 which is included in Mandriva 2007.1! Not that it was ugly before, but it’s a nice surprise to still see such big improvements, especially from a minor update.