Improving Mediawiki performance

Now that I am on the subject of improving performance, I configured some performance improvements for a Mediawiki installation here:

  • Make sure you run the latest Mediawiki version. Mediawiki 1.16 introduced a new localisation caching system which is supposed to improve performance, so you definitely want this to get the best performance.
  • Create a directory where Mediawiki can store the localisation cache (make sure it is writable by your web server). By preference store it on a tmpfs (at least if you are sure it will be big enough to store the cache), and configure it in LocalSettings.php:
    $wgCacheDirectory = "/tmp/mediawiki";
    Iif /tmp is on a tmpfs, you might add creation of this directory with the right permissions to /etc/rc.local, so that it still exists after a reboot.
  • Enable file caching in Mediawiki’s LocalSettings.php:
    $wgFileCacheDirectory = "{$wgCacheDirectory}/html";
    $wgUseFileCache = true;
    $wgShowIPinHeader = false;
    $wgUseGzip = true;
  • Make sure you have installed some PHP accelerator for caching. I have APC installed and configured it in Mediawiki’s LocalSettings.php:
    $wgMainCacheType = CACHE_ACCEL;

Here is a benchmark before implementing the above configuration (with CACHE_NONE, but APC still installed):

$ ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 255 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12750 bytes

Concurrency Level: 1
Time taken for tests: 30.084 seconds
Complete requests: 255
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 3344070 bytes
HTML transferred: 3251250 bytes
Requests per second: 8.48 [#/sec] (mean)
Time per request: 117.978 [ms] (mean)
Time per request: 117.978 [ms] (mean, across all concurrent requests)
Transfer rate: 108.55 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 6 2.8 7 21
Processing: 88 112 11.1 112 163
Waiting: 66 90 9.1 89 125
Total: 95 118 11.9 118 170

Percentage of the requests served within a certain time (ms)
50% 118
66% 122
75% 125
80% 127
90% 132
95% 138
98% 145
99% 156
100% 170 (longest request)

And here a benchmark after implementing the changes:

ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 649 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12792 bytes

Concurrency Level: 1
Time taken for tests: 30.015 seconds
Complete requests: 649
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 8538244 bytes
HTML transferred: 8302008 bytes
Requests per second: 21.62 [#/sec] (mean)
Time per request: 46.248 [ms] (mean)
Time per request: 46.248 [ms] (mean, across all concurrent requests)
Transfer rate: 277.80 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 9 3.7 8 29
Processing: 23 37 6.0 37 62
Waiting: 13 23 4.9 24 41
Total: 28 46 7.8 45 82

Percentage of the requests served within a certain time (ms)
50% 45
66% 47
75% 49
80% 50
90% 56
95% 62
98% 68
99% 73
100% 82 (longest request)

So Mediawiki can deal with more than 2,5 times as much requests now.

Some people use Apache’s mod_disk_cache to cache Mediawiki pages, but I prefer Mediawiki’s own caching system because it is more standard and does not require patching Mediawiki, even if it might not get as much benefit as a real proxy or mod_disk_cache.

Improving performance by using tmpfs

Today I analyzed disk reads and writes on a server with iotop and strace and found some interesting possible optimizations.

With iotop you can check which processes are reading and writing from the disks. I always press the o, p and a keys in iotop so that it only shows me processes doing I/O and so that it will show accumulated I/O instead of the bandwidth. With the left and right arrows I select on which columns to sort the list.

Once you have identified the processes wich are doing much I/O, you can check what they are reading or writing with strace, for example
# strace  -f -p $PID  -e trace=open,read,write

(you can leave out read and/or write if this gives too much noise)

This way I identified some locations where processes do lots of read and write operations on temporary files.

For nagios I placed /var/lib/nagios3/spool and /var/cache/nagios3 on a tmpfs, for Amavis /var/lib/amavis/tmp and for PostgreSQL /var/lib/postgresql/8.4/main/pg_stat_tm.

Other candidates you might want to consider: /tmp, /var/tmp and /var/lib/php5. There are probably many others, depending on which services you use.

Is Unity a better alternative to the GNOME Shell?

After my disappointment with the current GNOME 3.0 development version with GNOME Shell, I thought it would be interesting to compare it with Ubuntu’s Unity. Ubuntu has just published a new alpha version of what will become Ubuntu 11.04, so I used that for a quick test.

On the positive side:

  • On the dock on the left side there is a button which opens the workspace switcher which gives a nice overview of your virtual desktops and their contents. The workspace switcher is easy to find and it looks awesome: this might be exactly what is needed to make more end users finally get to use virtual desktops.
  • The list of Favourite folders is easily accessible by one of the buttons on the dock, as are all mounted volumes and the Thrash, unlike in GNOME Shell. Ubuntu’s desktop also supports desktop icons.
  • Integration of Banshee in the volume mixer applet is nice: the pop-up in the volume mixer will show the playing song and has some buttons to control playback in Banshee. I do not know whether this integration also works for other audio players though.

The negative:

  • Unity uses uses a development version of Compiz which is very unstable. The first time I booted the Ubuntu live CD, Compiz crashed within one minute. In my next test sessions Compiz crashed again different times. Currently GNOME Shell and Mutter are definitely much more stable than Unity and Compiz.
  • Just like GNOME Shell there is no way to show the date in panel, only the time is displayed.
  • When clicking on the Ubuntu icon in the panel, some kind of empty window pops up. Maybe this ought to be the application launcher, but it is clearly not working.
  • The application launcher can be opened from a button on the dock at the left side of the screen. However that button is rather near the bottom of the dock, above the mounted volumes icons. The Application button should be much more easy to find without having to scan all icons on the dock. Maybe this will get fixed when/if the Ubuntu icon launches the application browser.
  • Applications are not organized in categories. Instead I got a huge table of all applications and preferences tools laid out horizontally and vertically. The Scrollbar in the application browser does not seem to be working so I could not access applications which were out of the view.
  • In the application browser, there is something which looks like a text entry field which permits you to search for an application, but I could not type in it.
  • After using the application browser for a few times, it just shows as an empty window, just like the Ubuntu icon. When this happens, you have no possibility to start applications anymore.
  • When moving the mouse over an icon in the application launcher, a white border is drawn around the icon. The border is always a fixed size: if the application name is too long and wrapped over two lines, the border will cover part of the text.
  • Just like in GNOME Shell, it looks like I cannot add custom applets and application launchers in the panel.
  • The panel is used as a global menu bar for applications but not all applications support it: for example Firefox and LibreOffice do not use it. The menu is only shown when moving the mouse over the panel. If my mouse cursor is in an application itself, there is no trace of the menu, so people might be wondering where it is. I do not know whether this is by design or whether it is simply a bug. Personally I am also not convinced that a global menu is nice: when applications are not maximized, you  need to move your mouse back and forward between the application window and the top of the screen, which is cumbersome.
  • Mounted drives are shown in the dock and on the desktop. This looks a bit superfluous at first sight and especially when having lots of partitions on an external disk and lots of applications opened, the dock might become too small to show all icons.
  • It is still GNOME 2.32. You do not have the nice windowless pop-up dialogs from GTK+3, nor the nice date and time applet from GNOME Shell or the chat integration in the notifications. Users will not benefit from the improvements included in GNOME 3 applications.

While GNOME Shell looked like an unpolished and cumbersome to use product, Unity feels like a completely broken proof of concept. In its current state it is even impossible to do anything useful with it because even launching applications is almost impossible.

It is also questionable how Unity will remain usable in the future after Ubuntu 11.04 Natty is out: will they port it to GTK+ 3? And what will they do about the desktop icons, a feature which is currently still provided by Nautilus 2.32, but not present anymore in 3.0?

Canonical has decided to choose Unity as default for its next Ubuntu version because they thought GNOME Shell was not going into the right direction. However, Unity is currently even a much bigger failure than GNOME Shell. I have the feeling that Canonical’s decision was bad for both GNOME and Ubuntu: now we have two different unfinished, unpolished and in the case of Unity even totally broken desktop shells. I am wondering what would be the current state of GNOME Shell if Canonical had decided to dedicate its resources to GNOME Shell instead of Unity… I am also wondering how users will react to a desktop with Unity by default. Will Ubuntu derivatives with a different default desktop, like Mint, take over Ubuntu as the most popular distribution for desktops? Or will GNOME get into a similar crisis like KDE when 4.0 was out and will many users start moving to other desktops, either temporarily or permanently? Or will they just continue using standard GNOME 2.x until the dust settles? I do not have any answer to these questions, but for sure we are arriving at an important crossroads in the history of GNOME.

For screenshots and more information about Ubuntu 11.04 Natty and Unity, I refer to this Tech Drive-in article.

GNOME 3.0: Making the same mistakes as KDE 4.0?

Yesterday Fedora held a GNOME 3 test day. In order to facilitate testing, they published a Rawhide live CD containing the latest builds of GNOME 3. This is a great opportunity to test the latest GNOME-shell and other new things without having to upgrade your system to unstable alpha stuff. I tried it out on my laptop with Intel graphics chipset.

After a quick test, I am quite disappointed. Just like KDE 4, GNOME 3 tries to explore some new innovative desktop stuff, but just like KDE 4 in its first versions, it feels extremely unpolished and very unfinished, resulting in a rather cumbersome experience.

On the positive side:

  • The desktop feels pretty fast. Starting up applications, using the search function to find applications and the graphical desktop effects are all pretty smooth.
  • The black GNOME shell interface looks nice with subtle but pleasing effects like drop shadows and transparency.
  • There is a nice Expose effect when clicking on activities, nicely showing you all windows running on your system.
  • I have not tried this myself, but it looks like instant messaging is nicely integrated into the desktop. The user menu in the top panel makes it easy to set your status, and you can chat directly from the notifications on the desktop.
  • Just like chatting is nicely integrated,  the user’s agenda is also nicely integrated on the desktop. Just click on the time and see a list of all your appointments at a glance. It looks much nicer than GNOME 2’s calendar view.
  • The fact that some application dialogs now appear to be popping out of the main window and do not have their own window decoration looks nice and avoids clutter. You can see this in the About dialogs in GTK+ applications.

On the negative side:

  • I have the feeling that a lot of space is wasted in the top bar. By default it contains the Activities menu (which is not really a menu and not really needed, because you can open it by just moving your mouse to the top left of your screen), the time, and a few applets (NetworkManager, Accessibility, Volume control, Battery monitor and a user menu allowing to set some personal preferences and instant messaging status. There does not seem to be a way to add application launchers, or any other custom applets.
  • When an application is launched, the name of the application appears in the panel, together with its icon. However, it is a partial version of the large icon, it looks a bit like it is zoomed in. Because the icon is cropped, it is very hard to recognize and looks rather ugly. I fail to understand why they do not simply show the scaled icon.
  • The application launcher is not handy to use. First you have to move your mouse to the top left of the screen to open Activities, then you have to click on Applications, and then you have to move your mouse to the right of your screen to select an application category and move again back to the center of the screen to launch the requested application. This way, launching an application requires much more mouse movements and clicks than simply navigating in the Applications menu in GNOME 2. Moving the search box and application category list to the left, would already be a huge improvement.
  • The list of applications is shown as a table of application icons, laid out both horizontally and vertically. Having to scan the list in two different directions is cumbersome compared to scanning a vertical application menu like in GNOME 2.
  • When opening the application list, by default it shows all applications in alphabetical order. Even if you want to keep this system, I think it would be much nicer to show them like in SUSE’s GNOME appliation launcher, where all icons are still organized by category and separated by the category headers.
  • Under the icons in the application launcher, the name of the application is shown. Or at least a part of the name: most names are shortened by an ellipsis (…) and this is very ugly. For example Transmission becomes “Transmiss…”, Remote Desktop Viewer becomes “Remote…”. There are also no tooltips showing the full name when hovering your mouse over them. LibreOffice is not included on this live CD, but if they are installed will both LibreOffice Writer and LibreOffice Calc become “Libreoffice…” then? It is clear that not seeing the full names at a glance is terrible for finding applications.
  • There does not seem to be support for desktop icons. The ~/Desktop folder contains some .desktop files, but they are not shown on the desktop. Left or right clicking on the desktop does not do anything. This reminds me of KDE 4.0, which also had broken support for desktop icons which was only fixed completely in later KDE versions when the Folder View widget came out.
  • There is no easy way to open your home directory, a bookmarked directory or a removable drive in a file browser except by opening the application launcher and finding the file manager. No more handy Places menu like in GNOME 2, or desktop icons for your home folder and removable disks.
  • There is no way to move the “dock” to another screen border. Personally I would prefer having it at the bottom side of my screen inside of at the left side, but I could not find a way to change that.
  • I do not like the default GTK+ theme too much. The big white scrollbars and high white tab pages look pretty hard to my eyes, and the grey backgrounded toolbars look a bit weird, especially the sunken View combobox  in Nautilus’ toolbar (it has a white line underneath). The default icon theme also needs an update, because it looks old-fashioned to me. I could not find a way to change the themes. There was no Appearance configuration tool in System Settings.
  • The day of the month and month are not shown by default in the date in the top panel. In the Date and Time Settings I could not find a way to show the complete date.
  • Similar to the above problem, it looks like we will not be able to easily configure what will happen when the laptop lid is closed. I have always preferred GNOME’s nice default settings over KDE’s settings clutter , but now I have the feeling that they are going much too far in GNOME 3.

A lot of my criticism seems pretty fundamental and as th final version of GNOME 3.0 is currently scheduled in 2 months (4th of april), it is sure that many of these issues will not be resolved. Just like KDE 4.0, there is definitely potential here. However in its current state, I consider GNOME 3.0 still as a playground for developers and unsuitable for use except by early adopters. Especially the fact that basic features have been removed (desktop icons) and basic tasks have become harder (launching applications) is really unforgivable.

Another interesting read is this quick screenshot based review by a QT developer. Also check out GNOME 3’s official website.

Update: GNOME 3 developer hadess responded to some frequently reported problems during the test day. Some of the problems I have written about here, are currently being worked on.

Flash and Java: the end of obsolete technologies

Already for a long time, I am a total opponent of Flash and things are clearly not improving. Quite on the contrary: Flash is still unstable on my systems, often not working correctly or causing browser hangs and it has many serious security flaws, which are fixed rather slowly by Adobe.

Another piece of technology which is annoying is Java. Java is a serious memory hog, does not integrate very well in Linux distributions (no centralized package management system for the gazillion of Java libraries and frameworks), and Java’s new owner, Oracle, is trying to make Java a patent minefield by suing Google.

Recently, Oracle also decided to publish a paying version of the Java Virtual Machine, and it has also doubled prices for MySQL support.

In the meantime, OpenOffice.org, also in hands of Oracle now, has been forked to LibreOffice. Third-party contributors were dissatisfied with Sun/Oracle’s bureaucracy which was a serious limitation for external contributions. It seems that Sun/Oracle’s behaviour to keep total control of OpenOffice.org is now having the total opposite effect: LibreOffice is gaining wide support by many Linux distributors, which will quickly make OpenOffice.org totally irrelevant.

ironically Apple, another company which I dislike about their anti-competitive behaviour, has become an ally by refusing to install Flash on the iPhone and iPad and by deprecating Java support, which makes it likely that Java will not be included in future OS X versions.

As GNOME developer Colin Walters recently stated on his blog: enough is enough! I am regularly installing computers for other users. From now on:

  • I will install Flasblock browser extensions by default, to help protecting against malicious Flash animations and to send out a signal to web developers that they should stop using Flash.
  • I will not install any Java Runtime Environment or Java Development Kit by default, except if explicitly requested by the user.
  • Instead of OpenOffice.org I will install LibreOffice on all Windows and Mac systems I install. For Linux I will keep the default office suite installed by the distribution, but this will also be LibreOffice in about 6 months when all major Linux distributors have published a new release.
  • I will try to use PostgreSQL instead of MySQL for web applications. Currently I already installed a LimeSurvey instance and a MediaWiki instance with PostgreSQL instead of the more common MySQL and moved a Roundcube instance from MySQL to Postgresql. I expect to install a Drupal 7 instance on PostgreSQL in the future.

As users, we have a lot of power to show companies that we do not accept their behaviour. Let’s use that power now!

Server migration to Debian

Since this afternoon, this server is now running Debian GNU/Linux Squeeze. Just like the previous system, this is a KVM virtual machine running on a HP Proliant DL185G5 host. The host server has always been running Debian. This was my last production system still running Mandriva. I might have forgotten to move over a few things or there might be some breakage somewhere, so let me know if you encounter a problem.

Mandriva’s future? Mageia.

In spite of the fact that Mandriva has decided on its shareholder’s meeting to publish new release 2011 at the end of the first half of 2011 and to improve its communication towards the community, one can safely say that both the distribution and the company do not have any realistic future anymore. A bunch of ex-employees (most of them fired when Edge-IT was liquidated) and a big part of the community, have decided to fork the distribution under the new name Mageia. Now with only a handful of developers left and with most of the community contributors moved to Mageia, I do not see how new Mandriva releases will happen anymore in the future. Let’s hope that the Mageia community manages to take over what is left from Mandriva and that it can become a nice distribution bringing Linux to the masses.

Mandriva’s future (2)

Now a few weeks after my first article about Mandriva’s future things have even more moved in a negative direction. The new management has decided to close down Edge-IT, a company which was bought by Mandriva several years ago. Apparently lots of the employees working on the distribution, were in fact Edge-IT employees and so they now have to leave the sinking ship. [1] [2] [3] [4] [5] [6]

As is summarized by a mailing list post, Mandriva as being a distribution developed by a company and paid employees, is now an empty box. There is no more head in charge of the future direction of the distribution, who can take the necessary decisions for new releases. It looks like no more paid employees are working on GNOME and KDE packaging, and most people working on the installer and Mandriva’s configuration tools are gone. Looking at the SVN commit history it seems like the members of the security team are almost the only paid employees still working on the distribution. One can safely say that Mandriva is now de facto already a community developed distribution.

In the meantime, the CEO has reacted on a bug report. He says that Mandriva will concentrate on the server market in Europe and on the desktop market in countries like Brazil. Because European developers would have refused to be part of that future direction, they have been fired, according to the CEO. Of course this does not make any sense. There no reason why developers in Europe could not be working on the desktop if the target market for the desktop is mostly Brazil. The real reason why people are being fired, is of course that there is simply no more money to pay them anymore. Furthermore, I am pretty sure that Mandriva can never be a real competitor for distributions like Red Hat Enterprise Linux and Novell’s Suse Linux Enterprise Server, which are much more popular in large companies. Already even 5 years ago Mandriva was often not taken seriously anymore in corporate environments, according to my experience when working for an IT company specializing in free software. Even free distributions like CentOS and Debian have a much higher reputation in the server market than Mandriva.

Tomorrow there will be a shareholder’s meeting. After that, we might have a clear view on the future of Mandriva as a company. But there should not be much doubt: Mandriva as a distribution developed by the company, has already died now.

And Mandriva as being a communtiy developed distribution? It might work if some solution is found for things like the build system and other infrastructure, now being provided by Mandriva. Maybe there will be a merger with one of these Mandriva forks like PCLinuxOS and Unity Linux. But I am convinced that these distributions currently only exist thanks to Mandriva and will definitely suffer a lot if there is no more upstream where they can copy packages from. So a merger with such a distribution will not magically fix all problems caused by Mandriva’s death. People who are looking for a distribution to install on their system, should rather remove Mandriva from their shortlist, because there are too much doubts about its future…

Migrating from Microsoft SQL Server to PostgreSQL

One of the servers I manage at work is still running Microsoft Windows 2000. This system is hosting a few old forgotten web sites and it runs Microsoft SQL Server containing a few databases still in use. This server was already there when I started working at the university. Fortunately I never had to do much work on it and there never was a serious crash: it would have been a serious headache for me restoring this system because I do not have much knowledge about Windows and SQL Server.

During some spare time at work I decided taking a look at migrating the MS SQL Server databases to PostgreSQL. And all in all it was not very complicated. Fortunately, the databases itself are not very large and have a rather simple structure. The first thing I did, was recreating the structure of the database in PostgreSQL. With a tool like pgAdmin3 this is pretty straightforward. An alternative is to export the database structure in SQL format and adapt the resulting file so that it contains valid PostgreSQL table creation statements. The latter method is explained on the PostgreSQL wiki.

To migrate the data itself, I wrote a little Perl script. Actually I do not have much knowledge about Perl but it is a fairly straightforward language, so this did not prove to be too difficult. I connect to the MS SQL Server database using DBI, the Perl Database Interface, with the ODBC and FreeTDS drivers. The script gets all records from the tables you define, and inserts them in the PostgreSQL database using the PostgreSQL DBI driver.

These are the required packages on a Debian Squeeze system:

libdbd-odbc-perl
libdbd-pg-perl
tdsodbc
unixodbc
tdsodbc

Define the FreeTDS ODBC driver in /etc/odbcinst.ini:

[FreeTDS]
Description	= TDS driver (Sybase/MS SQL)
Driver		= /usr/lib/odbc/libtdsodbc.so
Setup		= /usr/lib/odbc/libtdsS.so
UsageCount	= 2

Then define the database in odbc.ini:

[database]
Driver = FreeTDS
Trace = No
Server = hostname
Port = 1433
Database = databasename

The script assumes that all field names in the PostgreSQL database are using lowercase names and that the MS SQL database is using ISO8859-15 encoding and the PostgreSQL database is using UTF-8 encoding. For every table, I make a migrateTable call with 3 arguments: the table name, an array containing all field names and the name of the primary key field for which you created a sequence (named fieldname_seq).

Once the database itself was migrated, the applications using the database had to be modified to use PostgreSQL. This was not too difficult, because the applications were PHP scripts using ODBC. These additional packages were needed:

odbc-postgresql
php5-odbc

To configure the PostgreSQL ODBC driver, add this in /etc/odbcinst.ini:

[PostgreSQL Unicode]
Description	= PostgreSQL ODBC driver (Unicode version)
Driver		= /usr/lib/odbc/psqlodbcw.so
Setup		= /usr/lib/odbc/libodbcpsqlS.so
Debug		= 0
CommLog		= 0
UsageCount	= 0

Then define the databse in odbc.ini:

[database]
Driver              = PostgreSQL Unicode
ServerName          = localhost
Database            = dbname
Username            = username
sslmode             = require

Then be sure not to call odbc_connect with the SQL_CUR_USE_ODBC option (which was actually needed with FreeTDS to fix some weird errors), because it causes segfaults.

The result? WISE’s publications pages is now using PostgreSQL as a back-end!