Virtualisation mess

I have downloaded the Fedora 7 installation DVD ISO and wanted to give it a try in a virtual machine. I am using VMWare Server already for some time as it was the first free (as in free beer) available feature-complete and fast virtualisation software. I Created a virtual disk, configured the ISO as source for the CD device, and started up the virtual machine. But then while booting the Fedora 7 kernel, VMWare just crashed, also making my host OS unstable, so I had to do a hard reset. I was still using VMWare Server 1.0.1, so I tried an upgrade to 1.0.2 with latest vmware-any-any patch, but all to no avail: VMWare just keeps on crashing.

Now there’s also Virtualbox, which is freely (as in free speech!) available, so this seemed like an excellent time to give it a try. Virtualbox is packaged for Mandriva, so urpmi virtualbox should suffice to install it. It automatically installs some dkms-virtualbox package, probably containing drivers for virtual network cards and such, like VMWare does too. But while compiling these modules, it bombed out with some compilation errors, and a warning that Virtualbox is not tested with kernels > 2.6.17. As I’m using 2.6.21 x86_64 tmb kernel, and I did not immediately find a reference to this error on Google, I’m stuck here I’m afraid. Let’s hope new versions of VMWare Server or Virtualbox fix these issues soon. In the meantime, I’ll continue to use VMWare Server.

Bug fixing progress

Since a few days, the bug triaging process is in full force. The purpose is to review all old, unconfirmed bugs, and verify if they are still valid. And of course, making sure those valid bugs are fixed. I have the impression that it’s really a great success. Lots of Bugzilla e-mails are arriving in my mailbox, and looking at changelog mailing list, several old bugs are finally getting fixed.

I started to concentrate on bugs in Kaffeine and KMplayer. I think Kaffeine is now as stable as is possible now. That means: far from perfect, but I think all remaining problems should now be handled by its upstream authors. Already since some time, Kaffeine has XCB support, which should make it a lot more stable when viewing embedded videos in the Konqueror web browser. And since this weekend, Kaffeine correctly disables the screen saver when you are viewing a full screen video, and won’t copy the whole media file you open from media:/ URIs to your home directory. I proposed an update for Mandriva 2007.0 and 2007.1 Spring to fix the last two issues. Fixing the first one, is unfortunately a lot more difficult for older Mandriva versions, because it requires a more recent version of xine, and depends on libxcb, which is in contribs…

KMplayer was actually in a much better state. The only real serious issue, is that KMplayer does not add itself to KDEs service menus which appear when you insert a DVD disk. This is rather serious as KMplayer is Mandriva’s default video player, and thus user’s don’t get an option to actually play the movie when they insert a DVD disk. This should not be too difficult to fix though, so let’s hope we’ll have a fix soon.

I also reviewed Kopete bugs. About all of them are fixed in current Cooker, possibly we’ll also have udpated packages for 2007.0 and 2007.1 Spring implementing a higher framerate for MSN webcam support, fixing decryption of gpg encrypted messages (2007.1 only) and fixing errors when chatting on conference.jabber.org (2007.0 only).

For my own use (and pleasure), I also recompile a lot of Cooker packages for 2007.1. While recompiling a more recent version of libxml, I had problems with python crashing. After a lot of trying to find out what causes it, I found out that python-reportlab as shipped in both 2007.1 and Cooker makes python crash. Python-reportlab is a dependency of the hplip printer drivers, so if you don’t have these installed, you’ll probably never notice this problem. Ia also makes certain hplip utilities crash in 2007.1 and Cooker. Compiling a more recent version of python-reportlab, fixed all problems, so I hope we will see this as an update for 2007.1 soon.

Next on the bug review list, will be cd burning applications. Thanks to the article I wrote on this blog about audio cd burning applications, I already tested some of them, which always helps in reviewing bugs.

If you always wanted to get involved in free software development, this is actually a great opportunity to join the Mandriva bug reviewing effort! You don’t really need to have development skills, just some willingness to compare different bugs, to try to reproduce them, and to look them up in other bug trackers. Often those bugs are really already fixed, it’s just that someone has to confirm they actually are, or provide the good patch or solution which has been created by the upstream developers. If you are interested, do not hesitate to contact us on the bugteam mailing list!

Disk encryption in Mandriva

Last weekend, I bough an external USB hard drive to finally start regularly making back-ups of my computers at home. For security reasons, I wanted to store back-ups on an encrypted partition, because one never knows what may happen. Using an encrypted partition was not too hard, but still some bugs prevented it from being newbie proof.

For the encrypted partition, I chose to use LUKS. It seems to be some kind of standard, widely supported (by Hal for example), and it has some graphical utilities available which should make it a no-brainer.

First problem, I’m using x86_64, and apparently luks-tools only exists for i586 in Mandriva. I filed a bug, and a new luks-tools package for x86_64 should already appear very shortly on a Cooker mirror near you. If you have luks-tools installed, it should be as simply as running gnome-luks-format to set up an existing partition.

So I did it at the console. Of course, make sure you have the package cryptsetup installed, otherwise you won’t have the necessary utilities. First we’ll add a header to the partition, indicating that this is a LUKS encrypted partition, and which encryption type we are using:

cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sdb1

(as I already mentioned with gnome-luks-format, you should create the partition first, for example with diskdrake or cfdisk).
Once the header is in place, we’ll open the encrypted partition:

cryptsetup luksOpen /dev/sdb1 encbackup

Encback is a name you can freely replace by your own choice of course. Once you have done this, your encrypted partition will be mapped to /dev/mapper/encback. You can use this like any normal partition for example to create an ext3 file system on it:

mkfs.ext3 /dev/mapper/encback

Then we can mount it:

mount /dev/mapper/encback /media/encrypted-backup

And we’re ready to go. To close the encrypted device after I have umount it, I run

cryptsetup luksClose encback

When you plug in the USB disk, hall will automatically detect you have a LUKS encrypted partition. A dialog will appear, where you can enter your passphrase, and after that it is mounted automatically. At least, that’s the theory. Unfortunately, this was not working, probably because of bug #30015. Let’s hope this gets fixed soon! Also note that KDE does not seem to support this at all, so even then this is Gnome only. For now I have created two small scripts, which will run cryptsetup and mount or umount, so I don’t have to retype these commands by hand each time.

I opened another bug to request LUKS support in diskdrake. Disk encryption is becoming more and more common these days with all those portable storage media, so Mandriva’s default partition tool should have this support built in. I guess it should not be too hard to implement this.

Some nice documentation which helped me a lot during this exploration, can be found in Red Hat Magazine.

Update 22 May 2007: I updated the howto to use aes-cbc-essiv:sha256 instead of aes-cbc-plain. Reader David Crick pointed me to the fact that aes-cbc-plain is vulnerable to a cryptography weakness, which is called the “watermarking attack”. Thanks!

More community = less employees…

Last Thursday, at least four Mandriva employees received the news that their contract will be stopped next month. All of them are people who are very active in the Mandriva community, and sometimes maintainer of important packages (such as Apache). There has not been much more explanation about this decision. This gives the news last week that Mandriva wants more involvement of the community (read: people who are not payed) another dimension to me… :-(

Oh well, we’ll have to wait to see if this will impact development much. But it’s never a good sign that people have to leave a company.

Update 21 May 2007: It seems that the list referred to in the above posting in not completely correct. Currently a lot of rumours are going around, even on Mandriva’s official IRC channels, and even between current Mandriva employees. Probably more than four people are losing their job, but it seems not even Mandriva employees themselves know exactly who is concerned. The communication clearly has been catastrophic, and this is severely damaging Mandriva’s image :-(

Update 22 May 2007:To get a more complete picture of the problems, please also read Dvalin’s blog post.

Mandriva – The Next Generation

Mandriva 2007 Spring was already a very nice distribution with a lots of improvements and changes in style from the previous versions. severely improved graphical themes, the public availability of non-free packages, getting update notifications on the desktop without Mandriva subscription, and a generally more polished system.

For the next version of Mandriva, the bar is again set a lot higher than before. Last week, two interesting announcements were made:

  • David Barth announced that some internal re-organization was done at Mandriva, and that the community will be more actively involved in the development of the next Mandriva version. What this means in practice, and how this will work out, remains to be seen, but I have the feeling that this is a confirmation of something we have seen already in 2007.1: Mandriva really cares about the community now, and makes use of the community’s remarks to improve the distribution.
  • Anne Nicolas, who will follow-up development of next Mandriva’s versions from now on, announced that an effort will be done to get Mandriva’s bugzilla cleaned up and to improve the workflow of bug reports, which should result in a more bug free distribution. All new bug reports will first be checked by a team, which will verify the validity of the report, and which will collect all necessary information for the developers to be able to analyze and fix the bug. If developers receive more high quality reports, this will hopefully result in quicker and more bug fixing. Statistics about the bug fixing progress will be collected, in order to pinpoint problematic parts of the distribution, so this can be acted upon.

This is all great news. Mandriva shows that it is not wanting to be yet another ordinary nice Linux distribution, but that it has the ambition to be a real leader. Interesting times are coming!

Linux kernel development thoughts

One week ago, kernel hacker Ingo Molnar reviewed Con Kolivas’ swap prefetch patches and approved them to be added to the official Linux kernel. Swap prefetching is a technique, which will load swapped out memory pages back in memory if the system is idle and memory has become available. This is useful for people starting temporary jobs which use a lot of memory, which makes other processes move to swap. Once the memory hungry process is finished, swap prefetching will kick in and slowly reload the swapped out pages back to memory, so that the system potentially does not need to do this anymore when the user again uses one of the swapped out processes. Tjos functionality can be enabled and disabled at will during compilation of the kernel.

Swap prefetching has been available for a long time in Con Kolivas patch set, and was also added to the mm development kernels some time ago. In that period, no bugs have been reported, and it seems people are happy with this feature. So it was hoped that the push by Ingo Molnar, would finally make swap prefetching available for all Linux users in version 2.6.22.

Developer Nick Piggin seemed rather critical of the swap prefetching feature. I I have understood the thread correctly, he discovered a problem (actually swap prefetching did not seem to work anymore because of some unrelated changes in Linux), and there was some serious disagreement how it should be dealt with. Con Kolivas’ got fed up with the unreasonable objections to the patch and proposed to dump the patch completely. So this is yet another performance improving feature we won’t be seeing in the Linux kernel for some time…

Update 12 May 2007: Con Kolivas posted an updated version of swap prefetch, addressing some problems, and the patch has not been dropped from the mm kernel. Maybe things do not look so bad after all?

This whole story makes me remind of the recent RSDL/SD scheduler. Con Kolivas implements a feature which is clearly working very good for most people, and then someone comes up who vetoes it for whatever reason, and in the case of SD other implementations are started (which still do not have the same maturity as SD). In the end a lot of work seems to duplicated and interesting features are delayed or even cancelled completely. Is it me, or is the behaviour of some kernel developers really hurting Linux development?

Also in related news, kernel hacker Adrian Bunk decided to not track regressions anymore. He was tired of Linux 2.6.21 being released with lots of known regressions. It seems Linux kernel development has really some issues these days…

Mandriva work this week

This week I was contacting the mailing lists for various upstream projects to find fixes for some problems.

First there was the problem that F-Spot does not automatically handle the upgrade from sqlite2 to sqlite3. In the past, F-spot used sqlite2 as its database, today it uses sqlite3 by default, but you can still continue to use your sqlite2 database. That is, if sqlite2 libraries are installed on your system. And one can imagine that in the future, distributions will stop shipping sqlite2, because it is not maintained anymore and because sqlite3 is far superior. It is possible to upgrade from sqlite2 to sqlite3 by upgrading the db at the command line, but we really cannot expect normal users to do this, do we?

In Pan, I had the problem that new headers were not appearing in a newsgroup, until I killed the “get new headers”-task by hand. I found out it was caused by a news server I had configured in Pan, which was not reachable. I filed a bug.

I experienced a crash in ogginfo when some wrongly encoded characters are found in the Vorbis tag, for which I found a patch with Google. I filed a bug but I’ll need to investigate if the tag is really invalid: Easytag shows it correctly without any problem.

Another problem I experienced already some time ago, but which I never really investigated, was that Amarok did not add files on a read-only Samba share to its library. Thanks to Debian changelog mailing list, I discovered today that there was a patch for libtag fixing this. This is a real serendipity!

I did again some testing with Kaffeine. I contact its mailing list for the problem that it copied everything to ~/tmp when using system:/ URI in Konqueror. I found a fix now, but I’m still not impressed by Kaffeine’s stability. For now, you’re better of continuing to use kmplayer.

I contacted Mandriva’s web discuss mailing list (which does not seem to be archived unfortunately) for a new structure of the End user documentation on the Mandriva wiki and posted a proposed for some documentation about the tmb kernel. I’m a bit disappointed by the reactions: like this we won’t succeed in creating a very active wiki community with lots of useful user documentation. I’ll try to implement a proposal in my sandbox this weekend: practical work is often better than theoretical discussion

Linux kernel: The battle of the CPU schedulers

Since some time already, different patches are being written for the Linux kernel, which improve the CPU scheduler. The CPU scheduler, is that part of the kernel, that’s responsible for assigning CPU time to the different task running on your system. If you sometimes experience problems with sound stuttering or your mouse becoming jerky while running other CPU intensive tasks, then this is definitely a problem caused by the task scheduler.

Con Kolivas has been maintaining an alternative scheduler for some time. His Staircase scheduler was designed with interactivity in mind, especially for desktop systems, where people want their system to be quickly responsive under all kinds of workloads. This scheduler has been optimized a lot through the years, and as such is very stable. Still there are some rare cases where “starvation” is possible.

At the start of March, Con Kolivas published a new scheduler, which was called RSDL (Rotating Staircase DeadLine scheduler) at first, and has been renamed to SD (Staircase Deadline) afterwards. Based on the experience Con Kolivas gathered with his Staircase scheduler, SD is a more general purpose scheduler, trying to give absolute fairness to the different running tasks, without favouring any process (for example lots of other schedulers favour X). This way, no starvation issues should be possible with this scheduler. A lot of discussion followed after his announcement, and it became quickly clear that a lot of people were not happy with the current scheduler in Linux. Important kernel developers like Mike Galbraith, Nick Piggin, Ingo Molnar, Willy Tarreau and Andrew Morton joined the discussion and also posted other scheduler patches, sometimes not without some trolling and flaming as sometimes happen on such mailing lists. Con Kolivas’ scheduler was added to Andrew Morton’s mm kernel tree to get some more testing. The development of RDSL/SD went up and down sometimes, because of Con Kolivas’ health problems.

Ingo Molnar, which was rather critical of some of the ideas in the new scheduler at first, also recently began the development of CFS (Completely Fair Scheduler, which actually is based on the same basic concept of fairness. Con Kolivas announced that he would stop development of the RD scheduler, because of his health problems, and because his ideas would now continue to be used in the CFS scheduler. But things came out differently, and Con Kolivas continued development in the end. The result is that the SD scheduler is now at its 46’th version (v. 0.46), and it seems most problems have been fixed. Based on all the testing done on the kernel mailing list, it seems SD 0.46 is more mature than CFS 6. Even Willy Tarreau, maintainer of the 2.4 Linux kernel tree, said that thanks to SD, he did make Linux 2.6 the default kernel on his laptop, as he found the scheduler in mainline 2.6 too bad compared to 2.4. It’s unclear however which of these schedulers will be integrated in linux finally, and when this will happen.

Personally I think SD 0.46 should be integrated now in Linux 2.6.22 pre-releases. There has been a lot of testing and bug fixing, and it seems there are no serious bugs open anymore now. I also hope that Mandriva 2008 will come out with one of these new schedulers. The tmb kernel in Mandriva Cooker, already uses the SD scheduler now. People interested in this discussion, can subscribe to the ck mailing list where a lot of the discussion is happening. Sites like LWN.net and Kerneltrap also often post about the progress of this subject.

Testing KDE video players, setting up backups and a cluster, cleaning up poppler and sqlite stuff

Today I tried Kaffeine 0.8.4 with XCB support. It is supposed to fix the stability issues when playing embedded videos in Konqueror, so I wanted to verify if it is a real contender to KMPlayer. Kaffeine indeed does not seem to cause instabilities in Konqueror anymore. That’s good. I tried viewing videos on vrtnieuws.net and this worked fine. Unfortunately, trying to view an Apple movie trailer failed, while KMPlayer (with mplayer back-end) worked fine. When double clicking on local video files while browsing system:/home in Konqueror, Kaffeine still copies the whole file to a temporary directory. Actually the default use of system:/home instead of /home/login by Konqueror is completely braindead, but I do not understand why Kaffeine needs to copy the whole file, while KMplayer does not need to do this. So for now, I still recommend KMPlayer.

At work, I started to install a cluster system, consisting of about 6 (IIRC) computers. I will try Kerrighed on Debian Etch. It seems an interesting project: more actively developed than OpenMosix with a clear roadmap for the future. And it does not seem too difficult to install. I’ll post my experiences soon.

Also at work, I recently discovered the rdiff-backup and backupninja projects. These programs really rule for making backups on free hard drive space. For tape back-ups I like Bacula, but it is not that easy to configure. For simple backups on hard drives, nothing beats the simplicity of rdiff-backup and backupninja. I should definitely check whether these good backuptools are also included in the Mandriva repositories.

Some things I have on my personal TODO list: I want to find out which programs still depend on xpdf, or have their proper xpdf code included in the source. We should maximise the use of poppler instead, because it is a shared library. If all PDF programs use poppler, all programs will easily benefit from improvements to poppler. And it will make security updates for all these programs much easier. And we will gain some space, which is always interesting for the One live CDs. Because now, a lot of packages have to be separately updated which is a PITA. I would like to do the same for sqlite. Seems like a lot of packages still depend on the old sqlite 2 instead of the more modern version 3, and some programs still use their own private copy. Let’s hunt these packages down, and see if we can compile them with the sqlite 3 shared library. If not, bug reports should be opened upstream so this gets fixed.

Mandriva 2007 Spring released, looking forward to 2008

Now that Mandriva 2007 Spring has been released and the corrupted Mandriva Cooker subversion repository has been fixed, development for Mandriva 2008 has started now.

Some things which I have on my wishlist for Mandriva 2008:

  • New kernel, Linux 2.6.22 or later which I hope will give us these things: CFQ IO scheduler as default (since Linux 2.6.18), better task scheduler, probably CFS, better wireless drivers (bcm43xx is stable in 2.6.20, but it’s not in Mandriva’s 2.6.17, finally good Ralink drivers: complete documentation exists, but drivers are still not production quality)
  • Mandriva still uses OSS drivers for some sound cards which are perfectly supported with Alsa (for example for some Intel ICH2 810 AC97 cards, which was not fixed in 2007.1 as far as I know). This is annoying as OSS does not have sound mixing, and can cause applications complain about occupied /dev/dsp and it also causes missing sound in Flash. Fedora even does not ship OSS anymore for some time already. A maximum of cards should be switched to Alsa now early in Cooker development , so that things get tested.
  • slocate is still being installed by default by rpmsrate instead of the much performance friendlier mlocate (already in 2007.1 Main). We should make the switch now in rpmsrate so it’s the default in 2008.
  • Almost a year ago, I proposed a new program menu structure for Mandriva. According to Gnome and menu maintainer Frédéric Crozat, it would be implemented for 2008. Let’s hope this becomes true, as it will make the menu at the same time friendly for new users, and more practical for power users. Unfortunately, I lost the complete original document (what a shame!). If somebody still has it somewhere, I would be grateful if you could send it to me!
  • Drop esound, and use Pulseaudio instead. Pulseaudio is a more modern and actively maintained sound server. There was some discussion about it some time ago, and I still hope Pulseaudio will not be forced on all users (personally I prefer not to use a sound server, and let all the mixing be done by Alsa. For this reason, I still hope more and more applications will use GStreamer, as GStreamer makes it easy to choose your preferred output method, such as Alsa or Pulseaudio for all applications. But Pulseaudio should certainly replace esound in main.
  • Test new Kaffeine: In Mandriva 2007.0, Kaffeine was replaced by kmplayer as the default video player in KDE, because Kaffeine was unstable paying embedded videos in Konqueror. Now that both Xine and Kaffeine support XCB, this problem should be fixed. We should test if it is indeed stable now, and if Kaffeine plays supports all kinds of embedded videos in web pages like kmplayer. If this is the case, we should consider moving back to Kaffeine, as its interface is a bit better than KMplayer’s.
  • Mandriva’s net_applet and mdkonline system tray utilities still poll too much, which causes unnecessary CPU usage, and which also eats battery life on laptops. These applications should be scrutinized a bit more, so they are completely idle most of the time.
  • Up to date printing support: Unfortunately printing support did not really advance in Mandriva 2007 Spring. The drivers are mostly still the same as in 2007.0, and some newer printer models supported by newer versions of hpijs and other models which appeared on linuxprinting.org, are not support in 2007 Spring. Mandriva 2008 should really have up to date printing support, like was always the case in previous versions.
  • Cleaning up the repositories: the Main repository now contains a lot of programs which are not used by a lot of people (things like flphoto, uucp, uw-imapd, xpdf,…). On the other hand, some very interesting packages are in Contribs (Transmission, dovecot,…). The contents of both repositories should be looked at, in order to clean up, and put packages in the appropriate repository.
  • We need a simple application installation program: rpmdrake has improved a lot in 2007.0 and now in 2007.1 Spring. Still it is too complex for new users. Suppose a user wanting to search a DTP application. He should easily find Scribus. Well, Scribus does not even appear in the Publishing category in rpmdrake. Instead, these category contains mostly technnical console programs not suitable for beginners. We should have something like Ubuntu’s gnome-app-install which only shows userfriendly graphical applications.
  • More and easier to find documentation: Mandriva already comes with a lot of documentation, but the included handbooks are difficult to find (being hidden in More Applications). They should be in a more obvious place. On the other hand, we should make sure more documentation becomes available in different languages on the wiki, and the wiki should be easy to find (include a link on the desktop?). This way, users will know where to look for help.
  • Related to the menu structure and the reorganization of the repositories, we should constantly be evaluating which applications should be in the default installation and which not. We had such discussion shortly before 2007.1 Spring went out, but this was a bit too little too late. We should make sure we start with this process before the first beta is released, so we have more time to evaluate and stress test programs.

Currently I already started by taking a quick look at the contents of the main repository. Maybe other people are interested in participating in this effort, so that it can happen a bit more organized?