Going back to my roots: testing Mageia 4 beta

Many years ago I used to be a Mandriva user and contributor, mostly active in packaging software. I stopped my contributions because I had the feeling the distribution was having more and more trouble keeping up with all new evolutions in the GNU Linux free software world and was loosing ground to other, more innovative distributions. Finally I settled for Debian myself. Even though it is not always the most innovative distribution itself, I liked its open, independent community-based nature.

Now after all this time, I was curious to see how my former favourite distribution had evolved. Mandriva was forked by former Mandriva employees and contributors, and so Mageia was born. Mageia is currently developing version 4 of its distribution and released beta 2 two weeks ago, on 13 December 2013. I decided to try out this version.
Continue reading “Going back to my roots: testing Mageia 4 beta”

Leap second causing ksoftirqd and java to use lots of cpu time

Today there was a leap second at 23:59:60 UTC. On one of my systems, this caused a high CPU load starting from around 02h00 GMT+2 (which corresponds with the time of the leap second). ksoftirqd and some java (glassfish) process where using lots of CPU time. This system was running Debian Squeeze with kernel 2.6.32-45. The problem is very easy to fix: just run

# date -s "`date`"

and everything will be fine again. I found this solution on the Linux Kernel Mailing List: http://marc.info/?l=linux-kernel&m=134113389621450&w=2. Apparently a similar problem can happen with Firefox, Thunderbird, Chrome/Chromium, Java, Mysql, Virtualbox and probably other processes.

I was a bit suprised that this problem only happened on this particular machine, because I have several other servers running similar kernel versions.

Creating your own GNOME session based on cairo-dock and Compiz

Personally I absolutely do not like the gnome-shell in GNOME 3. I actually even hate it: it is slow, messy and cumbersome to use and I have the feeling that developers are not listening to criticism. Obvious and trivial design bugs which are well known, are totally ignored (bug 662738 is an example).

For that reason, I went looking for an alternative desktop. KDE is way too bloated for a netbook with 1 GB of RAM, while XFCE is not as polished as a traditional GNOME 2.32 desktop. The best alternative I could find out right now, was to just replace the GNOME Shell by a custom panel or dock implementation. In the end I chose cairo-dock: it is written in C, so it is probably not as memory hungry as AWN (which uses Python) and Docky (which uses Mono, which I also consider as a possible patent minefield). Cairo-dock is also actively maintained. I paired cairo-dock with the compiz window manager to get some nicely looking desktop.
Continue reading “Creating your own GNOME session based on cairo-dock and Compiz”

MegaCLI: useful commands

Recently I installed a server with a Supermicro SMC2108 RAID adapter, which is actually a LSI MegaRAID SAS 9260. LSI created a command line utility called MegaCLI for Linux to manage this adapter. You can download it from their support pages. The downloaded archive contains an RPM file. I installed mc and rpm on Debian with apt-get, and then extracted the MegaCli64 binary (for x86_64) to /usr/local/sbin, and the libsysfs.so.2.0.2 from the Lib_utils RPM to /opt/lsi/3rdpartylibs/x86_64/ (that’s the location where MegaCli64 looks for this library).

Here are some useful commands:

View information about the RAID adapter

For checking the firmware version, battery back-up unit presence, installed cache memory and the capabilities of the adapter:

# MegaCli64 -AdpAllInfo -aAll

View information about the battery backup-up unit state

# MegaCli64 -AdpBbuCmd -aAll

View information about virtual disks

Useful for checking RAID level, stripe size, cache policy and RAID state:

# MegaCli64 -LDInfo -Lall -aALL

View information about physical drives

# MegaCli64 -PDList -aALL

Patrol read

Patrol read is a feature which tries to discover disk error before it is too late and data is lost. By default it is done automatically (with a delay of 168 hours between different patrol reads) and will take up to 30% of IO resources.

To see information about the patrol read state and the delay between patrol read runs:
# MegaCli64 -AdpPR -Info -aALL

To find out the current patrol read rate, execute
# MegaCli64 -AdpGetProp PatrolReadRate -aALL

To reduce patrol read resource usage to 2% in order to minimize the performance impact:
# MegaCli64 -AdpSetProp PatrolReadRate 2 -aALL

To disable automatic patrol read:
# MegaCli64 -AdpPR -Dsbl -aALL

To start a manual patrol read scan:
# MegaCli64 -AdpPR -Start -aALL

To stop a patrol read scan:
# MegaCli64 -AdpPR -Stop -aALL

You could use the above commands to run patrol read in off-peak times.

Migrate from one RAID level to another

In this example, I migrate the virtual disk 0 from RAID level 6 to RAID 5, so that the disk space of one additional disk becomes available. The second command is used to make Linux detect the new size of the RAID disk.

# /usr/local/sbin/MegaCli64 -LDRecon -Start -r5 -L0 -a0
# echo 1 > /sys/block/sda/device/rescan

Extending an existing RAID array with a new disk

./MegaCli64 -LDRecon -Start -r5 -Add -PhysDrv[32:3] -L0 -a0

Create a new RAID 5 virtual disk from a set of new hard drives

First we need to now the enclosure and slot number of the hard drives we want to use for the new RAID disk. You can find them out by the first command. Then I add a virtual disk using RAID level 5, followed by the list of drives I want to use, specified by enclosure:slot syntax.

# MegaCli64 -PDList -aALL | egrep 'Adapter|Enclosure|Slot|Inquiry'
# MegaCli64 -CfgLdAdd -r5'[252:5,252:6,252:7]' -a0

Extending an existing RAID array with a new disk

First check the enclosure device ID and the slot number of the newly added disk with the command above. Then we reconstruct the logical drive, adding the new drive. For a RAID 5 array this command is used:

# MegaCli64 -LDRecon -Start -r5 -Add -PhysDrv[32:3] -L0 -a0

View reconstruction progress

When reconstructing a RAID array, you can check its progress with this command.
# MegaCli64 -LDRecon ShowProg L0 -a0

(replace L0 by L1 for the second virtual disk, and so on)

Configure write-cache to be disabled when battery is broken

# MegaCli64 -LDSetProp NoCachedBadBBU -LALL -aALL

Change physical disk cache policy

If your system is not connected to a UPS, you should disable the physical disk cache in order to prevent data loss.

# MegaCli -LDGetProp -DskCache -LAll -aALL

To enable it (only do this if you have a UPS and redundant power supplies):

# MegaCli -LDGetProp -DskCache -LAll -aALL

More information

http://ftzdomino.blogspot.com/2009/03/some-useful-megacli-commands.html
https://twiki.cern.ch/twiki/bin/view/FIOgroup/DiskRefPerc
http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
http://kb.lsi.com/KnowledgebaseArticle16516.aspx

Fixing grub-probe error: Couldn’t find PV, check your device.map.

Today I was getting this error when installing a new kernel on a server running Debian:

/usr/sbin/grub-probe: error: Couldn't find PV pv2. Check your device.map.

The error can be reproduce by running the update-grub command.

The day before, a new RAID disk was added to this server, so I suspected this could be the cause. The file /boot/grub/device.map contained a reference to the first RAID disk as (hd0) but did not contain a reference to the new RAID disk. I ran

# ls -l /dev/disk/by-id/

to find out which SCSI ID referred to sdb (the new RAID disk), and then added the following line to device.map:


(hd1) /dev/disk/by-id/scsi-3600304800087c4f015fb4f2e4cc7a8e5

Now installing the new kernel works fine!

Linux performance improvements

Two years ago I wrote an article presenting some Linux performance improvements. These performance improvements are still valid, but it is time to talk about some new improvements available. As I am using Debian now, I will focus on that distribution, but you should be able to easily implement these things on other distributions too. Some of these improvements are best suited for desktop systems, other for server systems and some are useful for both. Continue reading “Linux performance improvements”

GNOME Shell moving forward

Some news about GNOME 3 and GNOME Shell:

  • The minimize and maximize window decoration buttons are now removed. It is estimated that these buttons are not useful actually, and users should be using Alt-Tab, the dock or different workspaces to switch between different applications, and maximize windows by double clicking on the title bar. As this will also make the desktop more difficult to access, I guess this also means that there are no plans to re-implement desktop icons.
  • The problem with the ellipsis of long application names has been fixed by enlarging the icons in the application browser.
  • On the #gnome-shell IRC channel there was a discussion earlier today about the implementation of shutdown in GNOME Shell. Several developers were in favour of just suspending to RAM by default and not showing a real shut down button by default. After 30 minutes, the system would wake up again and suspend to disk. Several developers did not seem to care about the risks of waking up a laptop while it’s being transported in a bag. Or about the fact suspend is not working properly on all systems.

I am extremely disappointed by these three things. When writing my previous GNOME Shell article, I still had some hopes that things would improve for the better, but I am giving up all hope: the GNOME Shell in GNOME 3.0 will definitely not be something I will like to use. I think it is also unacceptable that such important, drastic changes are made just before or even after the UI freeze. I have the feeling that GNOME Shell is purely the work of a few developers and designers who made some radical changes without any feedback or testing by real end users. The user community seems to be completely forgotten in the GNOME 3.0 development process. As only a few distributions are shipping live CDs, which are often rather unstable and rarely have a completely up to date GNOME Shell, only a very small amount of users is actually able to test and give feedback.

What will I do now? Skip GNOME 3.0 and hope that GNOME 3.2 will be better, once developers have taken into account users reactions? But that means that I will not benefit for more than another 6 months of any improvements to many of my preferred applications. Or use GNOME 3.0 with the old GNOME Panels (but will that give back my desktop icons)? Or shall I finally switch to KDE? Time will tell.

Update: the changes I described here can be seen in screenshots on Webupd8.

Improving Mediawiki performance

Now that I am on the subject of improving performance, I configured some performance improvements for a Mediawiki installation here:

  • Make sure you run the latest Mediawiki version. Mediawiki 1.16 introduced a new localisation caching system which is supposed to improve performance, so you definitely want this to get the best performance.
  • Create a directory where Mediawiki can store the localisation cache (make sure it is writable by your web server). By preference store it on a tmpfs (at least if you are sure it will be big enough to store the cache), and configure it in LocalSettings.php:
    $wgCacheDirectory = "/tmp/mediawiki";
    Iif /tmp is on a tmpfs, you might add creation of this directory with the right permissions to /etc/rc.local, so that it still exists after a reboot.
  • Enable file caching in Mediawiki’s LocalSettings.php:
    $wgFileCacheDirectory = "{$wgCacheDirectory}/html";
    $wgUseFileCache = true;
    $wgShowIPinHeader = false;
    $wgUseGzip = true;
  • Make sure you have installed some PHP accelerator for caching. I have APC installed and configured it in Mediawiki’s LocalSettings.php:
    $wgMainCacheType = CACHE_ACCEL;

Here is a benchmark before implementing the above configuration (with CACHE_NONE, but APC still installed):

$ ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 255 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12750 bytes

Concurrency Level: 1
Time taken for tests: 30.084 seconds
Complete requests: 255
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 3344070 bytes
HTML transferred: 3251250 bytes
Requests per second: 8.48 [#/sec] (mean)
Time per request: 117.978 [ms] (mean)
Time per request: 117.978 [ms] (mean, across all concurrent requests)
Transfer rate: 108.55 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 6 2.8 7 21
Processing: 88 112 11.1 112 163
Waiting: 66 90 9.1 89 125
Total: 95 118 11.9 118 170

Percentage of the requests served within a certain time (ms)
50% 118
66% 122
75% 125
80% 127
90% 132
95% 138
98% 145
99% 156
100% 170 (longest request)

And here a benchmark after implementing the changes:

ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 649 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12792 bytes

Concurrency Level: 1
Time taken for tests: 30.015 seconds
Complete requests: 649
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 8538244 bytes
HTML transferred: 8302008 bytes
Requests per second: 21.62 [#/sec] (mean)
Time per request: 46.248 [ms] (mean)
Time per request: 46.248 [ms] (mean, across all concurrent requests)
Transfer rate: 277.80 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 9 3.7 8 29
Processing: 23 37 6.0 37 62
Waiting: 13 23 4.9 24 41
Total: 28 46 7.8 45 82

Percentage of the requests served within a certain time (ms)
50% 45
66% 47
75% 49
80% 50
90% 56
95% 62
98% 68
99% 73
100% 82 (longest request)

So Mediawiki can deal with more than 2,5 times as much requests now.

Some people use Apache’s mod_disk_cache to cache Mediawiki pages, but I prefer Mediawiki’s own caching system because it is more standard and does not require patching Mediawiki, even if it might not get as much benefit as a real proxy or mod_disk_cache.

Improving performance by using tmpfs

Today I analyzed disk reads and writes on a server with iotop and strace and found some interesting possible optimizations.

With iotop you can check which processes are reading and writing from the disks. I always press the o, p and a keys in iotop so that it only shows me processes doing I/O and so that it will show accumulated I/O instead of the bandwidth. With the left and right arrows I select on which columns to sort the list.

Once you have identified the processes wich are doing much I/O, you can check what they are reading or writing with strace, for example
# strace  -f -p $PID  -e trace=open,read,write

(you can leave out read and/or write if this gives too much noise)

This way I identified some locations where processes do lots of read and write operations on temporary files.

For nagios I placed /var/lib/nagios3/spool and /var/cache/nagios3 on a tmpfs, for Amavis /var/lib/amavis/tmp and for PostgreSQL /var/lib/postgresql/8.4/main/pg_stat_tm.

Other candidates you might want to consider: /tmp, /var/tmp and /var/lib/php5. There are probably many others, depending on which services you use.

Is Unity a better alternative to the GNOME Shell?

After my disappointment with the current GNOME 3.0 development version with GNOME Shell, I thought it would be interesting to compare it with Ubuntu’s Unity. Ubuntu has just published a new alpha version of what will become Ubuntu 11.04, so I used that for a quick test.

On the positive side:

  • On the dock on the left side there is a button which opens the workspace switcher which gives a nice overview of your virtual desktops and their contents. The workspace switcher is easy to find and it looks awesome: this might be exactly what is needed to make more end users finally get to use virtual desktops.
  • The list of Favourite folders is easily accessible by one of the buttons on the dock, as are all mounted volumes and the Thrash, unlike in GNOME Shell. Ubuntu’s desktop also supports desktop icons.
  • Integration of Banshee in the volume mixer applet is nice: the pop-up in the volume mixer will show the playing song and has some buttons to control playback in Banshee. I do not know whether this integration also works for other audio players though.

The negative:

  • Unity uses uses a development version of Compiz which is very unstable. The first time I booted the Ubuntu live CD, Compiz crashed within one minute. In my next test sessions Compiz crashed again different times. Currently GNOME Shell and Mutter are definitely much more stable than Unity and Compiz.
  • Just like GNOME Shell there is no way to show the date in panel, only the time is displayed.
  • When clicking on the Ubuntu icon in the panel, some kind of empty window pops up. Maybe this ought to be the application launcher, but it is clearly not working.
  • The application launcher can be opened from a button on the dock at the left side of the screen. However that button is rather near the bottom of the dock, above the mounted volumes icons. The Application button should be much more easy to find without having to scan all icons on the dock. Maybe this will get fixed when/if the Ubuntu icon launches the application browser.
  • Applications are not organized in categories. Instead I got a huge table of all applications and preferences tools laid out horizontally and vertically. The Scrollbar in the application browser does not seem to be working so I could not access applications which were out of the view.
  • In the application browser, there is something which looks like a text entry field which permits you to search for an application, but I could not type in it.
  • After using the application browser for a few times, it just shows as an empty window, just like the Ubuntu icon. When this happens, you have no possibility to start applications anymore.
  • When moving the mouse over an icon in the application launcher, a white border is drawn around the icon. The border is always a fixed size: if the application name is too long and wrapped over two lines, the border will cover part of the text.
  • Just like in GNOME Shell, it looks like I cannot add custom applets and application launchers in the panel.
  • The panel is used as a global menu bar for applications but not all applications support it: for example Firefox and LibreOffice do not use it. The menu is only shown when moving the mouse over the panel. If my mouse cursor is in an application itself, there is no trace of the menu, so people might be wondering where it is. I do not know whether this is by design or whether it is simply a bug. Personally I am also not convinced that a global menu is nice: when applications are not maximized, you  need to move your mouse back and forward between the application window and the top of the screen, which is cumbersome.
  • Mounted drives are shown in the dock and on the desktop. This looks a bit superfluous at first sight and especially when having lots of partitions on an external disk and lots of applications opened, the dock might become too small to show all icons.
  • It is still GNOME 2.32. You do not have the nice windowless pop-up dialogs from GTK+3, nor the nice date and time applet from GNOME Shell or the chat integration in the notifications. Users will not benefit from the improvements included in GNOME 3 applications.

While GNOME Shell looked like an unpolished and cumbersome to use product, Unity feels like a completely broken proof of concept. In its current state it is even impossible to do anything useful with it because even launching applications is almost impossible.

It is also questionable how Unity will remain usable in the future after Ubuntu 11.04 Natty is out: will they port it to GTK+ 3? And what will they do about the desktop icons, a feature which is currently still provided by Nautilus 2.32, but not present anymore in 3.0?

Canonical has decided to choose Unity as default for its next Ubuntu version because they thought GNOME Shell was not going into the right direction. However, Unity is currently even a much bigger failure than GNOME Shell. I have the feeling that Canonical’s decision was bad for both GNOME and Ubuntu: now we have two different unfinished, unpolished and in the case of Unity even totally broken desktop shells. I am wondering what would be the current state of GNOME Shell if Canonical had decided to dedicate its resources to GNOME Shell instead of Unity… I am also wondering how users will react to a desktop with Unity by default. Will Ubuntu derivatives with a different default desktop, like Mint, take over Ubuntu as the most popular distribution for desktops? Or will GNOME get into a similar crisis like KDE when 4.0 was out and will many users start moving to other desktops, either temporarily or permanently? Or will they just continue using standard GNOME 2.x until the dust settles? I do not have any answer to these questions, but for sure we are arriving at an important crossroads in the history of GNOME.

For screenshots and more information about Ubuntu 11.04 Natty and Unity, I refer to this Tech Drive-in article.