Noteworthy Debian Trixie/Sid changes week 1 (June 11 – June 17 2023)

A long time ago, I used to regularly post an overview of noteworthy changes in the Mandriva development version. For years now I am using Debian testing though. With the release of Debian 12 Bookworm, I though it could be interesting to keep track of noteworthy changes in the upcoming Debian version, Trixie.

I will be tracking sid trough the debian-devel-changes maling list. Usually about 10 days after a package entered sid, it should move to testing, at least if there are not important bugs in the package. The selection of which packages I mention here, is very personal. I will try to cover important changes for both desktop and server packages, but this list will never be complete. If you noticed an interesting change not mentioned, feel free to add a comment to this article.

I’m not sure whether I will make this kind of post regularly without interruption, but let’s see where this goes.

The first week of development, saw a huge amount of packages updated to the latest upstream versions. Some of these were already available in Experimental for some time. Let’s dive in.

Kernel, hardware support, low-level libraries

The Linux kernel was updated to the 6.3 series, coming from 6.1 in Bookworm. I refer to kernelnewbies.org for a complete overview of what’s new in Linux 6.2 and Linux 6.3, but I can mention BTRFS performance improvements in both versions (including discard=async being default on SSDs with TRIM support, performance improvements if you are using an Intel Skylake CPU and add retbleed=stuff to the kernel options and the usual driver improvements which improve hardware support, for example for the current Intel Arc GPUs. If you have an AMD processor with at least the Zen2 microarchitecture, you can enable the new amd_pstate_epp frequency scaling driver by adding the kernel option amd_pstate=active.

btrfs-progs has been updated to version 6.3.1. The major change in the 6.3 series is that block-group-tree is out of experimental mode. This will reduce the mount time of BTRFS file systems. You can enable this on an existing file system with the command

btrfstune --convert-to-block-group-tree <device>

Developers warn to be careful, because there might still be bugs.

The Mesa 3D drivers were upgraded from version 22.3.6 to 23.1.2. You will want to upgrade to this version if you are using an Intel Arc GPU because there have been many bug fixes. Also new in Mesa 23.1 is OpenCL support for AMD GPUs using rusticl.

power-profies-daemon version 0.13 entered Debian Sid. It adds support for the amd_pstate_epp driver which can be activated in Linux 6.3.

Tthe LLVM based Fortran compiler Flang is now available in Debian as the package flang-15. LLVM 16 is available in sid. Clang 16 and libc++ 16 are only available in experimental at this time and version 15 is still the default version in sid.

Server and virtualization

Samba 4.18.3 brings some performance improvements relative to version 4.17.8 in Bookworm.

Qemu was updated to 8.0.2. The 8 series brings various improvements, but maybe the most important thing to mention is that virtiofsd, a daemon which allows you to share directories on the host with guests, is not included in the qemu package any more. If you use this, you will need to install the new virtiofsd package which contains a new implementation in Rust.

Desktop

LibreOffice is now at version 7.5.4. The 7.5 series bings improved dark mode support, new application icons, nicer default table styles in Draw and Impress and other various improvements. See the release notes and the New Features in Libreoffice 7.5 video for more information.

This week we saw the first GNOME 44 packages enter sid. gnome-backgrounds 44 brings you new desktop wallpapers and evolution 3.48 brings lay-out improvements. If you don’t like the headerbar layout, you can disable it and switch back to the traditional toolbar.

New in Debian is the gdm-settings package. It lets you configure the GDM login manager and change its appearance through a user friendly user interface.

NetworkManager-openconnect 1.2.10 finally adds support for Single Sign-on implementations using SAML on the Cisco AnnyConnect and Palo Alto GlobalProtect VPNs. Unfortunately it does not seem to work for Pulse/Juniper. I have opened an issue for that, and in the meantime I use openconnect-pulse-gui.

The Transmission Bittorrent client 4 has moved from C to C++, uses less CPU and memory and has support for the Bittorrent v2 protocol, amongst other improvements.

The first KDE Gear 23.04 applications are now being uploaded to sid. Now in the repositories is the KDE Mastodon client Tokodon. Gwenview (supports pinch gestures to zoom in Wayland mode), Ghostwriter (automatic language detection for the spellchecker) and the Falkon web browser (dark colour scheme support) were updated to 23.04.

Tellico, the application which helps you to keep track of your music, movie and other collections has support for new data sources and improved support for existing data sources and has reports with an image grid.

Thhe dav1d AV1 video decoder was update d from 1.0 to version 1.2.1. This version brings many performance improvements thanks to new SIMD code. Also the svt-av1 encoder and decoder was updated to version 1.5.0, also adding some optimizations.

There are plans to remove the old SDL 1.2 packages, replacing them by SDL12-compat (Debian package libsdl1.2-compat-shim) which implements SDL 1.2 using the SDL 2. Version 1.64, which entered sid, added compatibility for some old games, such as Sid Meier’s Alpha Centauri and others.

Science, education and technical tools

GNU Octave 8.2 brings improved Matlab compatibility, the GUI has dark mode support and various other improvements.

The R statistical computing programming (package r-base) 4.3.1 (was 4.2.2 in Bookworm) brings many new features.

Labplot 2.10 comes with many new features, improvements and performance optimizations in different areas, as well as with support for new data formats and visualization types.

People who are into geographic information systems will by happy with the QGIS update. Version 3.28 introduces many improvements. Look at the changelog for more details.

The Electronics Design Automation Suite kicad has been updated to version 7. I don’t know anything of this kind of software, but the release announcement lists a large number major of improvements.

Others improvements

There are too many changes to list here in detail. I mention updated Pipewire 0.3.71, Wireplumer 0.4.14, GStreamer 1.22.3, Opus 1.4, Gajim 1.8.0, OpenJDK 20, Phosh 0.28.0 and much more.

Conclusion

The first week of Trixie development saw a huge amount of software enter Debian sid. This is of course due to the backlog of all new upstream versions which could not be submitted during the Bookworm freeze, are now all trickling in now. Some of these packages were already in Experimental.

My personal favourites are Linux 6.3 which now allows me to use the amd_state_epp driver and Evolution 3.48, which has an some improvements to the UI which I like a lot.

Now that the first flood of new packages have arrived, things will probably calm down a bit, also because of the upcoming summer and holidays in the northern hemisphere. But I guess we will see more of GNOME 44 and KDE Gear 23.04 entering sid soon.

Upgrading from Debian 11 Bullseye to Debian 12 Bookworm

Debian 12 Bookworm will be released very soon, on June 10 2023. The Debian Testing tree is now very close to the final release, so now is a good moment to start testing Bookworm if you did not do so. I already upgraded some of my server systems to Bookworm and I’m also running on all my desktop systems, so here are some notes of the upgrade process. Keep in mind that upgrading to Bookworm is only supported if you are running Bullseye. If you are running an older version of Debian (Buster), you will need to upgrade to Bullseye first and after that upgrade to Bookworm.

First of all, start with reading the release notes, it contains a very detailed howto guide describing all steps to upgrade your system to Bookworm. It also lists all major changes and important things to know before you upgrade.

First check which packages you have installed which do not come from the official Debian repositories with this command:

# apt list '?narrow(?installed, ?not(?origin(Debian)))'

Because these are not official Debian packages, Debian developers cannot guarantee that they will work correctly and will not conflict or cause compatibility problems when upgrading your system. For that reason, you should seriously consider uninstalling them during the upgrade process.

On one system I had a locally built snuffleupagus package installed. This package was built against a particular PHP version and because a newer Debian release will also include a newer PHP version, this could break things. So i removed this package:

# apt remove snuffleupagus

Then you need to verify whether you have put any packages on hold. Packages on hold will never be upgraded, so this can prevent a correct upgrade. Check all held packages with this command:

# apt-mark showhold

You can unhold them with this command:

# apt-mark unhold packagename

Then we need to adapt our apt sources.list and preferences.

You should have this in /etc/apt/sources.list (or in a .list file /etc/apt/sources.list.d):

deb [arch=amd64,i386] http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb-src http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://deb.debian.org/debian/ bookworm-proposed-updates main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm-proposed-updates main contrib non-free non-free-firmware

Note the new non-free-firmware repository: non-free firmware used to be included in the non-free repository, but now they are in a new separate repository, so you will need to add that.

Then we need to set up the priorities of the different repositories in /etc/apt/preferences (or a ;pref file in /etc/apt/preferences.d):

Package: *
Pin: release n=bookworm
Pin-Priority: 810

Package: *
Pin: release n=bookworm-security
Pin-Priority: 810

Package: *
Pin: release n=bookworm-proposed-updates
Pin-Priority: 809

Package: *
Pin: release n=bookworm-backports
Pin-Priority: 808


Package: *
Pin: release n=bullseye
Pin-Priority: 710

Package: *
Pin: release n=bullseye-security
Pin-Priority: 710

Package: *
Pin: release n=bullseye-proposed-updates
Pin-Priority: 709

Package: *
Pin: release n=bullseye-backports
Pin-Priority: 708


Package: *
Pin: release n=trixie
Pin-Priority: 310


Package: *
Pin: release a=unstable
Pin-Priority: 200


Package: *
Pin: release a=experimental
Pin-Priority: 160

This gives the highest priority to all packages in Bookworm and the security updates, with a lower priority to the Bookworm proposed updates and then Bookworm backports. I added Bullseye in case you still need the Bullseye repositories for some reason. I also add Trixie (code name for what will become testing when Bullseye gets released) and sid (unstable) and experimental on the lowest priorities. Of course you can remove them from your preferences file if you don’t have set up these repositories.

I strongly recommend installing apt-listchanges, because it will give you information about important changes which might affect you before packages are upgraded:

# apt install apt-listchanges

I upgrade dpkg and apt first. I personally prefer to take advantages of eventual new improvements and bug fixes during the upgrade process.

# apt install -t bookworm apt dpkg

In Bookworm the systemd-resolved service now is in a seperate package. If you are currently using systemd-resolved, this can cause failures in DNS resolution. Before upgrading, make sure you know the addresses of your DNS servers, so that you can set them up manually if required; You can run the resolvectl command to find them. If DNS resolution breaks during the upgrade process later on, you can add them to /etc/resolv.conf manually to fix the problem. But I prefer to immediately install the new systemd-resolved package before upgrading everything else to take care of this problem:

# apt install -t bookworm systemd-resolved

Then we can upgrade all packages which can be upgraded without installing new packages:

# apt upgrade -t bookworm --without-new-pkgs

Once that’s done we proceed with the upgrade of all remaining packages, which will also install new dependencies:

# apt full-upgrade

During these two steps pay attention to which packages are going to be removed. It’s expected that old unused libraries and other packages (old PHP and Perl versions for example) will be removed, but you might to check this.

When the upgrade is done, I remove all unneeded packages with this command:

# apt autoremove --purge

Then run this command to remove all library packages which have no other packages depending on them any more:

# deborphan | xargs dpkg --purge

A package which often stays behind is libssl1.1. Normally you don’t need it any more so you can remove it safely:

# apt remove libssl1.1

Finally I also prefer to remove rsyslog. It is not installed any more by default on Debian Bookworm and everything is already logged to the systemd journal and I don’t want any double logging.

# apt remove rsyslog

Then personally I also install dbus-broker on Debian Bookworm. It replaces the traditional dbus implementation and is supposed to be more performant.

# apt install dbus-broker

I always recommend to verify that the metapackage linux-image-amd64 is installed, so that you are really running the latest kernel version.

# apt install linux-image-amd64

After upgrading all packages, reboot your system.

Debian 12 Bookworm and OpenLDAP

One major change in Debian 12 Bookworm is that it ships with OpenLDAP 2.5 which has removed the BDB and HDB back-ends. If your LDAP directory is still using this backend, you will have to convert it to the new MDB backend. There are some instructions in /usr/share/doc/slapd/README.Debian.gz and I might write some post here in the future about this. In any case, make sure you have recent backups of your LDAP directory in the form of LDIF gerenated with slapcat.

Is Debian Bookworm stable?

I’m permanently running Debian Testing on my laptop, and now I have also installed Bookworm on some servers. I strongly recommend using Testing for all desktop usage (even after Bookworm has been released), and to start using Bookworm on any new server installations. For upgrades of critical and more complicated server systems, I generally recommend to wait until at least the first point release.

At the moment I can think of two problems I am encountering on my systems. On my HP Elitebook 845 G8 when suspending the system (s2idle), Linux fails to read the current time from the RTC, resulting in the clock jumping years into the future. At the next resume you can fix the clock by restarting systemd-timesyncd, but the uptime command will continue to give a wrong output. I’m currently testing version 6.3 of the Linux kernel from experimental to see whether this bug also happens with this version.

Another problem is that in Debian 12 Bookworm the Spice plugin of Remmina is disabled. There is a work-around using packages from sid and experimental: upgrade to libspice-client-glib-2.0-8 version 0.42-2, which is currently in sid and then install remmina and remmina-plugin-spice from experimental. Maybe this problem will be fixed in a point release for Debian Bookworm, but that remains to be seen.

Alternatives for the net-tools utilities

Many modern distributions, like for example the upcoming Debian 12 Bookworm, do not install the package net-tools by default. This package contains popular utilities like ifconfig, route, netstat, arp and mii-tool. In this post I give alternatives for these utilities. You can of course just install the net-tools package if you prefer to keep using these commands.

ifconfig

To see the current network configuration:

$ ip addr

To see the currrent configuration for one specific interface, for example enp25s0:

$ ip addr show enp25s0

To add a static IP address to a network interface

$ ip addr add 192.168.10.2/24 dev enp25s0

Replace add by del to remove an IP address.

route

To see the current route table:

$ ip route

To set the default gateway:

$ ip route add default via 192.168.10.1 dev enp25s0

netstat

The ss command lists all open sockets. Some interesting options:

-ashow both open and listening sockets
-lonly show listening sockets
-pshows the process using the socket
-tshow only TCP sockets
-ushow only UDP sockets
-rresolve all IP addresses

To see all open and listening sockets on the system:

$ ss -a

To see all listening TPC and UDP ports:

$ ss -plut

arp

Display the contents of the ARP table:

$ ip neigh

mii-tool

Show the status of an Ethernet interface:

$ ethtool enp25s0

Increasing PHP security with Snuffleupagus

In a previous article, I discussed how to set up ModSecurity with the Core Rule Set on Debian. This can be considered as a first line of defense against malicious HTTP traffic. In a defense in depth strategy of course we want to add additional layers of protection to your web servers. One such layer is Snuffleupagus. Snuffleupagus is a PHP module which protects your web applications against various attacks. Some of the hardening features it offers are encryption of cookies, disabling XML External Entity (XXE) processing, a white or blacklist for the functions which can be used in the eval() function and the possibility to selectively disable PHP functions with specific arguments (virtual-patching).

Installing Snuffleupagus on Debian

Unfortunately there is no package for Snuffleupagus included in Debian, but it is not too difficult to build one yourself:

$ apt install php-dev
$ mkdir snuffleupagus
$ cd snuffleupagus
$ git clone https://github.com/jvoisin/snuffleupagus
$ cd snuffleupagus
$ make debian

This will build the latest development code from the master branch. If you want to build the latest stable release, before running make debian, use these commands to view all tags and to checkout the latest table tag, which in this case was v0.8.2:

$ git tag
$ git checkout v0.8.2

If all went well, you should now have a file snuffleupagus_0.8.2_amd64.deb in the above directory, which you can install:

$ cd ..
$ apt install ./snuffleupagus_0.8.2_amd64.deb

Configuring Snuffleupagus

First we take the example configuration file and put it in PHP’s configuration directory. For example for PHP 7.4:

# zcat /usr/share/doc/snuffleupagus/examples/default.rules.gz > /etc/php/7.4/snuffleupagus.rules

Also take a look at the config subdirectory in the source tree for more example rules.

Edit the file /etc/php/7.4/fpm/conf.d/20-snuffleupagus.ini so that it looks like this:

extension=snuffleupagus.so
sp.configuration_file=/etc/php/7.4/snuffleupagus.rules

Now we will edit the file /etc/php/7.4/snuffleupagus.rules.

We need to set a secret key, which will be used for various cryptographic features:

sp.global.secret_key("YOU _DO_ NEED TO CHANGE THIS WITH SOME RANDOM CHARACTERS.");

You can generate a random key with this shell command:

$ echo $(head -c 512 /dev/urandom | tr -dc 'a-zA-Z0-9')

Simulation mode

Snuffleupagus can run rules in simulation mode. In this mode, the rule will not block further execution of the PHP file, but will just output a warning message in your log. Unfortunately there is no global simulation mode, but it has to be set per rule. You can run a rule in simulation mode by appending .simulation() to it. For example to run INI protection in simulation mode:

sp.ini_protection.simulation();

INI protection

To prevent PHP applications from modifying php.ini settings, you can set this in snuffleupagus.rules:

sp.ini_protection.enable();
sp.ini_protection.policy_readonly();

Cookie protection

The following configuration options sets the SameSite attribute to Lax on session cookies, which offers protection against CSFR on this cookie. We enforce setting the secure option on cookies, which instructs the web browser to only send them over an encrypted HTTPS connection and also enable encryption of the content of the session on the server. The encryption key being used is derived of the value of the global secret key you have set, the client’s user agent and the environment variable SSL_SESSION_ID.

sp.cookie.name("PHPSESSID").samesite("lax");

sp.auto_cookie_secure.enable();
sp.global.cookie_env_var("SSL_SESSION_ID");
sp.session.encrypt();

Note that the definition of cookie_env_var needs to happen before sp.session.encrypt(); which enables the encryption.

You have to make sure the variable SSL_SESSION_ID is passed to PHP. In Apache you can do so by having this in your virtualhost:

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

eval white- or blacklist

eval() is used to evaluate PHP content, for example in a variable. This is very dangerous if the PHP code to be evaluated can contain user provided data. Therefore it is strongly recommended that you create a whitelist of functions which can be called by code evaluated by eval().

Start by putting this in snuffleupagus.rules and restart PHP:

sp.eval_whitelist.list().simulation();

Then test your websites and see which errors you get in the logs, and add them separated by commas to the eval_whitelist.list(). After that you need to remove .simulation() and restart PHP in order to activate this protection. For example

sp.eval_whitelist.list("array_pop,array_push");

You can also use a blacklist, which only blocks certain functions. For example:

sp.eval_blacklist.list("system,exec,shell_exec,proc_open");

Limit execution to read-only PHP files

The read_only_exec() feature of Snuffleupagus will prevent PHP from execution of PHP files on which the PHP process has write permissions. This will block any attacks where an attacker manages to upload a malicious PHP file via a bug in your website, and then attempts to execute this malicious PHP script.

It is a good practice to let your PHP scripts be owned by a different user than the PHP user, and give PHP only read-only permissions on your PHP files.

To test this feature, add this to snuffleupagus.rules:

sp.readonly_exec.simulation();

If you are sure all goes well, enable it:

sp.readonly_exec.enable();

Virtual patching

One of the main features of Snuffleupagus is virtual patching. Thjs feature will disable functions, depending on the parameters or and values they are given. The example rules file contains a good set of generic rules which blocks all kinds of dangerous behaviour. You might need to fine-tune the rules if your PHP applications hits certain rules.

Some examples of virtual-patching rules:

sp.disable_function.function("chmod").param("mode").value("438").drop();
sp.disable_function.function("chmod").param("mode").value("511").drop();

These rules will drop calls to the chmod function with octal values 438 and 511, which correspond to the dangerous 0666 and 0777 decimal permissions.

sp.disable_function.function("include_once").value_r(".(inc|phtml|php)$").allow();
sp.disable_function.function("include_once").drop();

These two rules will only allow the include_once function to include files which file name are ending with inc, phtml or php. All other include_once calls will be dropped.

Using generate-rules.php to automatically site-specific hardening rules

In the scripts subdirectoy of the Snuffleupagus source tree, there is a file named generate_rules.php. You can run this script from the command line, giving it a path to a directory with PHP files, and it will automatically generate rules which specifically allow all needed dangerous function calls, and then disable them globally. For example to generate rules for the /usr/share/tt-rss/www and /var/www directories:

# php generate_rules.php /usr/share/tt-rss/www/ /var/www/

This will generate rules:

sp.disable_function.function("function_exists").filename("/usr/share/tt-rss/www/api/index.php").hash("fa02a93e2724d7e818c5c13f4ba8b110c47bbe7fb65b74c0aad9cff2ed39cf7d").allow();
sp.disable_function.function("function_exists").filename("/usr/share/tt-rss/www/classes/pref/prefs.php").hash("43926a95303bc4e7adefe9d2f290dd8b66c9292be836908081e3f2bd8a198642").allow();
sp.disable_function.function("function_exists").drop();

The first two rules allow these two files to call function_exists and the last rule drops all requests to function_exists from any other files. Note that the first two rules limit the rules not only to the specified file name, but also define the SHA256 of the file. This way, if the file is changed, the function call will be dropped. This is the safest way, but it can be annoying if the files are often or automatically updated because it will break the site. In this case, you can call generate_rules.php with the --without-hash option:

# php generate_rules.php --without-hash /usr/share/tt-rss/www/ /var/www/

After you have generated the rules, you will have to add them to your snuffleupagus.rules file and restart PHP-FPM.

File Upload protection

The default Snuffleupagus rule file contains 2 rule which will block any attempts uploading a html or PHP file. However, I noticed that they were not working with PHP 7.4 and these rules would cause this error message:

PHP Warning: [snuffleupagus][0.0.0.0][config][log] It seems that you are filtering on a parameter 'destination' of the function 'move_uploaded_file', but the parameter does not exists. in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 0 parameter's name: 'path' in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 1 parameter's name: 'new_path' in /var/www/html/foobar.php on line 15'

The snuffleupagus rules use the parameter destination for the move_uploaded_file instead of the parameter new_path. You will have to change the rules like this:

sp.disable_function.function("move_uploaded_file").param("new_path").value_r("\.ph").drop();<br />sp.disable_function.function("move_uploaded_file").param("new_path").value_r("\.ht").drop();

Note that on PHP 8, the parameter name is to instead of new_path.

Enabling Snuffleupagus

To enable Snuffleupagus in PHP 7.4, link the configuration file to /etc/php/7.4/fpm/conf.d:

# cd /etc/php/7.4/fpm/conf.d
# ln -s ../../mods-available/snuffleupagus.ini 20-snuffleupagus.ini
# systemctl restart php7.4-fpm

After restarting PHP-FPM, always check the logs to see whether snuffleupagus does not give any warning or messages for example because of a syntax error in your configuration:

# journalctl -u php7.4-fpm -n 50

Snuffleupagus logs

By default Snuffleupagus logs via PHP. Then if you are using Apache with PHP-FPM, you will find Snuffleupagus logs, just like any PHP warnings and errors in the Apache error_log, for example /var/log/apache/error.log. If you encounter any problems with your website, go check this log to see what is wrong.

Snuffleupagus can also be configured to log via syslog, and actually even recommends this, because PHP’s logging system can be manipulated at runtime by malicious scripts. To log via syslog, add this to snuffleupagus.rules:

sp.log_media("syslog");

I give a few examples of errors you can encounter in the logs and how to fix them:

[snuffleupagus][0.0.0.0][xxe][log] A call to libxml_disable_entity_loader was tried and nopped in /usr/share/tt-rss/www/include/functions.php on line 22

tt-rss calls the function libxml_disable_entity_loader but this is blocked by the XXE protection. Commenting this in snuffleupagus.rules should fix this:

sp.xxe_protection.enable();

Another example:

[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'ini_set', because its argument '$varname' content (display_errors) matched a rule in /usr/share/tt-rss/www/include/functions.php on line 37'

Modifying the PHP INI option display_errors is not allowed by this rule:

sp.disable_function.function("ini_set").param("varname").value_r("display_errors").drop();

You can completely remove (or comment) this rule in order to disable it. But a better way is to add a rule before this rule which allows it for specially that PHP file. So add this rule before:

sp.disable_function.function("ini_set").filename("/usr/share/tt-rss/www/include/functions.php").param("varname").value_r("display_errors").allow();

If you get something like this:

[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'function_exists', because its argument '$function_name' content (exec) matched a rule in /var/www/wordpress/wp-content/plugins/webp-express/vendor/rosell-dk/exec-with-fallback/src/ExecWithFallback.php on line 35', referer: wp-admin/media-new.php

It’s caused by this rule:

sp.disable_function.function("function_exists").param("function_name").value("exec").drop();

You can add this rule before to allow this:

sp.disable_function.function("function_exists").filename("/var/www/wordpress/wp-admin/media-new.php").param("function_name").value("exec").allow();

More information

Snuffleupagus documentation

Snuffleupagus on Github

Julien Voisin blog archives

Setting up Wireguard VPN with IPv6

I wanted to set up Wireguard on a VPS, not only to tunnel IPv4 traffic, but also allowing me to tunnel IPv6 traffic. As this is IPv6 of course I preferred not to use NAT, but to assign a public IP address to the client. I read some documentation and blog posts, but I struggled getting it to work. Most tutorials I found on the Internet, create a separate IPv6 subnet for the VPN but I could not get this to work. For some reason, IPv6 traffic successfully went through the VPN tunnel and then exited the VPN gateway, but then any response never reached my VPN gateway and hence also not the client.

I decided to try another way: using an NDP proxy. NDP or the Neighbour Discovery Protocol, is similar to ARP which is used in IPv4. Using this protocol, network devices can discover where on the network a certain IP is located. By letting the VPN gateway answer NDP requests for the VPN client, the gateway would correctly send back all responses to the VPN gateway, which then forwards it to the VPN clients.

Configuring the network on the VPN gateway

I use systemd-networkd to set up the network. It’s the most modern way of network configuration and works the same on all distributions using systemd, but of course you can make the same settings in /etc/network/interfaces or whatever your distribution uses. Of course when making changes to a remote server, make sure you can access a console without needing a working network connection on the server, in case things go wrong and the network connection breaks.

On my VPN server, the public network interface is named ens192 (use the command $ ip addr to find it on your system). My public IPv4 address is www.xxx.yyy.zzz with subnet 255.255.255.0 and gateway ww.xx.yy.1. I have the 64 bit IPV6 prefix aaaa:bbbb:cccc:dddd and the IPv6 gateway is fe80::1.

I set this in /etc/systemd/network/internet.net:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=1.1.1.2
DNS=1.0.0.2
Address= www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
DNS=2606:4700:4700::1112
DNS=2606:4700:4700::1002

In this example I’m using the Cloudflare malware blocking DNS filters, but you can of course just use your ISP’s DNS servers here.

Setting up Wireguard

Run these commands on the Wireguard VPN gateway, and on all clients:

# apt install wireguard-tools
# cd /etc/wireguard
# umask 077
# wg genkey | tee privatekey | wg pubkey > publickey

Then create /etc/wireguard/wg0.conf on the VPN gateway with these contents:

[Interface]
Address = 192.168.7.1,fd42:42:42::1/64
PrivateKey = contents_of_file_privatekey
ListenPort = 51820

#client1
[Peer]
PublicKey = contents_of_publickey_of_client
AllowedIPs = 192.168.7.2/32,aaaa:bbbb:cccc:dddd:ffff::2/128

Add a [Peer] section for every client, and change the both the IPv4 and IPv6 address in AllowedIPs so that they are unique (replace 2 by 3 and so on) .

On the clients, create /etc/wireguard/wg0.conf with these contents:

[Interface]
Address = 192.168.7.2/32,aaaa:bbbb:cccc:dddd:ffff::2/128
PrivateKey = contents_of_privatekey_of_client
DNS = 2606:4700:4700::1112, 2606:4700:4700::1002, 1.1.1.2, 1.0.0.2

[Peer]
PublicKey = contents_of_publickey_of_vpn_gateway
Endpoint = vpngateway.example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0

In the [Interface] section make sure to use the same IP addresses as the ones you have set in the corresponding [Peer] section on the VPN gateway. Set the DNS name (or IP address) of the VPN gateway as Endpoint in the [Peer] section. The hostname’s DNS entry can have both an A and AAAA record. You can replace your DNS servers by your preferred ones. You can also consider running your own DNS server on the VPN gateway.

Make sure that all wg*.conf files on client and server are only readable by root, because they contain private keys.

Configuring the firewall (Shorewall)

I use the Shoreline firewall, Shorewall, as firewall.

Make sure you have shorewall and shorewall6 installed:

# apt install shorewall shorewall6

Shorewall6

First we create a separate zone for our VPN in /etc/shorewall6/zones:

fw firewall
net ipv6
vpn ipv6

Then we configure the network interfaces and assign it to the right zone in /etc/shorewall6/interfaces:

net     NET_IF          tcpflags,routeback,proxyndp,physical=ens192
vpn     wg0		tcpflags,routeback,optional

Then we allow connections from the VPN to the firewall and to the Internet in /etc/shorewall6/policy:

$FW	net		ACCEPT
vpn     net		ACCEPT
vpn     $FW		ACCEPT
net	all		DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all		REJECT		$LOG_LEVEL

Keep in mind that your VPN client will have a public IPv6 address, which is accessible from the Internet. The rule net all DROP protects your VPN clients against access from the Internet.

Then we create some rules which allows access to the SSH server and the Wireguard VPN server from the Internet in /etc/shorewall6/rules:

Invalid(DROP)      net    	$FW		tcp
Ping(DROP)	   net		$FW
ACCEPT		   $FW		net		ipv6-icmp
AllowICMPs(ACCEPT) all		all
ACCEPT		   all		all		ipv6-icmp	echo-request


SSH(ACCEPT)	   net		$FW
ACCEPT		   net		$FW		udp	51820 # Wireguard

We allow some required ICMPv6 message types defined in /usr/share/shorewall/action.AllowICMPs, as well as the echo-request type, which should not be dropped on IPv6.

For security reasons you could even choose to not open the SSH port for the net zone. SSH will only be accessible via the VPN then.

Finally we need to enable IP forwarding in /etc/shorewall6/shorewall6.conf:

IP_FORWARDING=Yes

Then we check whether everything compiles fine and enable and start the service:

# shorewall6 compile
# systemctl restart shorewall6
# systemctl enable shorewall6

Shorewall

For IPv4 we configure Shorewall to use NAT to provide Internet access to the VPN clients.

/etc/shorewall/zones:

fw	firewall
net	ipv4
vpn	ipv4

/etc/shorewall/interfaces:

net     NET_IF          dhcp,tcpflags,logmartians,nosmurfs,sourceroute=0,routefilter,routeback,physical=ens192
vpn	wg0		tcpflags,logmartians,nosmurfs,sourceroute=0,optional,routefilter,routeback

/etc/shorewall/policy:

$FW	net	ACCEPT
vpn	net	ACCEPT
vpn	$FW	ACCEPT
net	all	DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all	REJECT		$LOG_LEVEL

/etc/shorewall/rules:

# Drop packets in the INVALID state

Invalid(DROP)  net    	        $FW		tcp

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP)	net		$FW

SSH(ACCEPT)	net		$FW
ACCEPT		net		$FW		udp	51820

/etc/shorewall/snat:

MASQUERADE	192.168.7.0/24	NET_IF

/etc/shorewall/shorewall.conf:

IP_FORWARDING=Yes

Compile and load the rules and enable Shorewall permanently:

# shorewall compile
# systemctl restart shorewall
# systemctl enable shorewall

Setting up NDP proxying

Then in order to make sure that the gateway knows that the VPN client aaa:bbb:cccc:dddd::2 is reachable via the VPN gateway, we need to set up NDP proxying. The Neighbor Discovery Protocol is similar to ARP in IPv6.

In a previous version of this guide, I configured NDP proxying in Shorewall6. However, we can directly set this up with systemd-networkd, so this will also work if you don’t use Shorewall6 but another firewall like Firewalld. Furthermore I also experienced problems with NDP proxy settings being lost after some time, requiring a restart of Shorewall6 to make the IPv6 connection over Wireguard work again. I hope this will be fixed by settings this up in systemd-networkd.

Edit again the file /etc/systemd/network/internet.net and in the [NETWORK] section add this

IPv6ProxyNDP=1
IPv6ProxyNDPAddress=aaaa:bbbb:cccc:dddd:ffff::2

If you have more than 1 Wireguard client, you can add multiple IPv6ProxyNDPAddress lines to the file, one for each IPv6 address you want to proxy.

Then restart the systemd-networkd service:

# systemctl restart systemd-networkd

With this command you can check whether they have been set up correctly:

# ip -6 neighbour show proxy

Enabling and testing the VPN

On the server run this to enable the Wireguard server:

# systemctl enable --now wg-quick@wg0

To connect to the VPN, run this on the client:

# systemctl start wg-quick@wg0

Check if you can browse the world wide web. Use these websites to check your IP address and whether you have a working IPv6 connection:

https://test-ipv6.com/

https://ipv6-test.com/

You can also use traceroute and traceroute6 to test whether traffic is correctly going through the VPN tunnel:

# traceroute www.google.com
# traceroute6 www.google.com

Debugging Wireguard

If things don’t work as expected, you can enable debug logging in the Wireguard module with this command:

# echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

Replace +p by -p in order to disable debug logging. You can find the logs in your kernel messages, for example by running

# journalctl -f -k

Also firewall log messages will appear here.

You can use tcpdump to check the traffic on the wire (or in the VPN tunnel). For example to see all ipv6 traffic in the tunnel on the gateway:

# tcpdump -nettti wg0 "ip6"

Sources

Setup WireGuard with global IPv6

Setting up WireGuard IPv6

Reddit: Wireguard doesn’t seem to work with IPv6

Wireguard: enable debug logging to fix network issues

Shorewall6: Proxy NDP

Debian GNU/Linux on a HP Elitebook 845 G8

Some time ago, I received a new laptop, the HP Elitebook 845 G8. This is a 14″ laptop with an AMD CPU of the Renoir family, in my case an AMD Ryzen 7 PRO 5850U. As always, I run Debian GNU/Linux testing (currently Bookworm) on it. In this post, I will explain how to get all hardware working. This guide probably also works for other G8 Elitebooks, such as the Elitebook 835 G8 and Elitebook 855 G8, because they are all quite similar.

You can find detailed logs and reports of people running Linux on the Elitebook 845 G8 in the Hardware for Linux database.

Installation

I used a USB disk to boot the installer and a USB-C dock with an Ethernet interface to do a network installation. If you use the Debian installer with non-free firmware, you can also do the installation over wifi. I have not tried the current stable release Debian 11 Bullseye on this system. For best compatibility I strongly recommend testing because it has a more recent kernel and drivers.

Download the installation cd for Debian testing with non-free firmware: you need the file firmware-testing-amd64-netinst.iso.

Now we need to write it to a USB disk. Make sure there is no data on the drive you want to keep, because this process will completely wipe the disk.

To write the ISO image to a USB disk, Windows users can use the application Rufus, MacOS users can use Balena Etcher. If you are using Linux, you can dd the ISO image on your USB disk, or use a GUI like Fedora Media Writer.

Reboot the system and press F10 when the HP logo appears to load the BIOS/UEFI setup. Go to the Advanced page and select Boot Options. There make sure that USB Storage Boot is enabled. If you want to work with custom kernels, it can be handy to disable Secure Boot in SecuritySecure Boot Configuration, but it’s not needed to install and use Debian.

Save the changes you made (if any) and reboot the system and press F9 at the HP logo to get the boot menu. In the boot menu, select your USB drive to start the Debian installer.

Enabling non-free repositories

We will need to configure the non-free repositories for apt because we need several firmware packages from non-free. Edit /etc/apt/sources.list and check whether any deb line has main contrib and non-free at the end. If not add it, and then run

# apt update

Updating the BIOS/UEFI firmware

If you have a Windows installation, you can update the firmware from there, even before you install Linux. But you can also update the firmware without Windows. Follow the instructions in that blog post. It’s important to do this, not only because this gives you essential security fixes, but also bug fixes, some of which specifically for Linux compatibility.

Updating CPU firmware

Install the package amd64-microcode to ensure your AMD CPU is always running the latest microcode, which includes security fixes:

# apt-get install amd64-microcode

Flashing other firmware

The fwupd utility can download and install firmware from the LVFS. The firmware of the fingerprint reader of the Elitebook 845 G8, can be updated like this, and may be necessary to get the fingerprint reader working in Linux. First make sure fwupd is installed:

# apt install fwupd fwupd-amd64-signed

Now update all available firemware:

# fwupdtool update

If you have a HP USB-C Dock G5, then new firmware is also available in the LVFS, but it’s in the testing repository. To enable this repository, run this command:

# fwupdmgr enable-remote lvfs-testing 

Radeon Vega GPU

lspci identifies this GPU as

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev d1)

First make sure the firmware is installed for the GPU. This is needed to get any hardware acceleration.

# apt install firmware-amd-graphics

Then make sure these packages are installed in order to get Vulkan, VA-API, VDPAU and OpenCL support:

 # apt install mesa-va-drivers mesa-vulkan-drivers mesa-vdpau-drivers mesa-opencl-icd

Now add this to /etc/environment so that it uses the correct VDPAU driver:

VDPAU_DRIVER=radeonsi

In order to get GStreamer based players to use VA-API, you need to install this package:

# apt install gstreamer1.0-vaapi

After installing the firmware and editing /etc/environment you will need to reboot your system.

Unfortunately most video players and web browsers still don’t use VA-API hardware acceleration by default, but this needs to be configured manually. I will write a separate article about that later.

Realtek wifi adapter

The wifi adapter is a Realtek RTL8822CE according to lspci:

01:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter [10ec:c822]

Install the firmware to get it working:

# apt install firmware-realtek

This laptop can also be delivered with an Intel AX200 Wi-Fi 6 adapter (which is actually a better option than this one from Realtek). If you have this one, you will need to install the firmware-iwlwifi package instead.

Smartcard reader

lsusb identifies this smartcard reader as an Alcor AU9540:

Bus 005 Device 004: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader

Note that it only sees the smartcard reader when a card has been inserted.

You will need pscd with the CCID driver to use this smartcard reader:

# apt install pscsd

Fingerprint reader

The fingerprint reader can be seen like this in lsusb:

Bus 003 Device 003: ID 06cb:00df Synaptics, Inc. 

Make sure you have installed all firmware updates with fwupd and then you need to install these packages:

# apt install fprintd libpam-fprintd

In GNOME, under SettingsUsers you can enable login on fingerprint and add your fingerprints.

Sound

lspci:

04:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 01)<br />04:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller

Sound is working out of the box. I recommend switching from Pulseaudio to Pipewire.

Webcam

lsusb:

Bus 001 Device 002: ID 0408:5348 Quanta Computer, Inc. HP HD Camera

The webcam works out of the box. Many applications will see the IR camera as a second camera.

Bluetooth

lsusb:

Bus 003 Device 002: ID 0bda:b00c Realtek Semiconductor Corp. 802.11ac WLAN Adapter

If you use lsusb -v you will see that this is actually the Bluetooth Radio adapter. It is combined with the wifi adapter, hence the confusion.

Suspend/resume

HP does not support S3 (traditional suspend-to-ram/standby) in its recent Elitebooks any more, but instead uses s0ix (s2idle/suspend-to-idle/modern standby). S2idle support for AMD CPU’s was only added in Linux 5.11 with the amd_pmc driver. I recommend a very recent kernel, because later kernel versions had bug fixes in this regard too. However suspend regressed in stable update 5.17.3 (and others), a bug which was fixed in 5.17.5. I’m using a custom-built 5.17.5 kernel, but a fixed kernel will appear soon in Debian.

If you have HP Drivelock enabled, then your system will fail to resume. Drivelock is a security feature which can be set up in the BIOS and requires you to enter a password when starting up the system in order to access the contents of the disk. When trying to resume the system, fans start running, the keyboard backlight reacts to key presses, but the screen remains blank, nothing is written to logs and also network does not come up. Apparently this is a bug in HP’s BIOS/UEFI firmware which can be worked around by adding iommu=pt to the kernel command line. To do so, edit /etc/default/grub and add this to the variable GRUB_CMDLINE_LINUX_DEFAULT. For example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

Then update the GRUB configuration:

# update-grub

Install isenkram to help install drivers when plugging in hardware

Isenkram is a utitliy which will show a message when you connect hardware to your system and extra software or firmware is available for that hardware.

# apt install isenkram

Enabling trimming of the NVME SSD

Enable the fstrim timer to make sure the SSD is trimmed on regular intervals:

# systemctl enable --now fstrim.timer

Increasing battery time

Switch to AMD P-state driver

AMD developed the amd_pstate driver which was introduced in Linux 5.17. amd_pstate offers better performance per Watt than acpi-cpufreq

/etc/modprobe.d/acpi-cpufreq-blacklist.conf

blacklist acpi-cpufreq

/etc/modules-load.d/amd-pstate.conf

amd_pstate

If you are using Linux 6.3 you don’t need to do this, but you will have to add to the the GRUB_CMDLINE_LINUX_DEFAULT options in /etc/default/grub.conf:

amd_pstate=active

and run update-grub.

Set up TLP

TLP is a tool which optimizes power consumption of your system in order to increase battery time. TLP also has an options Radio Device Wizard, which I will use here to automatically disable wifi when the system is connected via an Ethernet cable.

# apt install tlp tlp-rdw

Configure the Radio Device Wizard by creating the file /etc/tlp.d/10-tlp-rdw.conf:

# tlp-rdw - Parameters for the radio device wizard

# Possible devices: bluetooth, wifi, wwan.
# Separate multiple radio devices with spaces.
# Default: <none> (for all parameters below)

DEVICES_TO_DISABLE_ON_LAN_CONNECT="wifi wwan"
DEVICES_TO_DISABLE_ON_WIFI_CONNECT="wwan"
DEVICES_TO_DISABLE_ON_WWAN_CONNECT="wifi"

# Radio devices to enable on disconnect.

DEVICES_TO_ENABLE_ON_LAN_DISCONNECT="wifi wwan"
DEVICES_TO_ENABLE_ON_WIFI_DISCONNECT=""
DEVICES_TO_ENABLE_ON_WWAN_DISCONNECT=""

# Radio devices to enable/disable when docked.

DEVICES_TO_ENABLE_ON_DOCK=""
DEVICES_TO_DISABLE_ON_DOCK=""

# Radio devices to enable/disable when undocked.

DEVICES_TO_ENABLE_ON_UNDOCK="wifi"
DEVICES_TO_DISABLE_ON_UNDOCK=""

To enable best ASPM power saving features when on battery, create /etc/tlp.d/20-aspm.conf:

PCIE_ASPM_ON_AC=default
PCIE_ASPM_ON_BAT=powersupersave

You will also need to edit /etc/default/grub file to add pcie_aspm=force to the Linux command line, for example

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_aspm=force"

And run update-grub to update the GRUB configuration.

Disable Bluetooth and wifi when on battery and they are not in use by creating /etc/tlp.d/30-disable-devices-on-battery.conf:

DEVICES_TO_DISABLE_ON_BAT_NOT_IN_USE="bluetooth nfc wifi wwan"

Submitting your system to the Linux Hardware database

The Linux Hardware database is a useful tool where users searching for hardware, can check the compatibility of systems with Linux. I recommend running this tool on all your Linux systems. After submission, you will get a link where you can view the data and indicate whether all hardware works and which work-arounds you had to apply. Click on the Review button on the page to do so.

# apt install hw-probe
# hwprobe --all --upload

Conclusion

Actually Linux compatibility of the HP Elitebook 845 is actually in good shape. It’s not perfect, but all hardware can be made to work. On distros like Ubuntu, which install non-free firmware by default, it should even be easier to make everything work. Still HP lags behind Dell and Lenovo in Linux support, because they don’t make it possible to flash the BIOS/UEFI firmware using fwupd, while all recent Dell and Lenovo business laptops have their firmware available in the LVFS. Also the problem that iommu=pt needs to be used to successfully resume the laptop when Drivelock is enabled, is a problem that HP should address in a BIOS update.

Web application firewall: Modsecurity and Core Rule Set

A web application firewall (WAF) filters HTTP traffic. By integrating this in your web server, you can make sure potentially dangerous requests are blocked before they arrive to your web application or sensitive data leaks out of your web server. This way you add an extra defensive layer potentially offering extra protection against zero-day vulnerabilities in your web server or web applications. In this blog post, I give a tutorial how to install and configure ModSecurity web application firewall and the Core Rule Set on Debian. With some minor adaptions you can also use this guide for setting up ModSecurity on Ubuntu or other distributions.

ModSecurity is the most well-known open source web application firewall. The future of ModSecurity does not look too bright but fortunately with Coraza WAF an alternative which is completely compatible with ModSecurity is in development. At this moment Coraza only integrates with the Caddy web server, and does not have a connector for Apache or NGinx so for that reason it is currently not yet usable as a replacement for ModSecurity.

While ModSecurity provides the framework for filtering HTTP traffic, you also need rules which define what to bloc and that’s where the Core Rule Set (CRS) comes in. CRS is a set of generic rules winch offer protection to a various range of common attacks via HTTP, such as SQL injection, code injection and cross-site scripting (XSS) attacks.

Install ModSecurity and the Core Rule Set on Debian

I install the Apache module for ModSecurity, the geoip-database, which can be used for blocking all requests from certain countries, and modsecurity-crs, which contains the Core Rule Set. I take this package from testing, because it has a newer version (version 3.3.2 at the time of writing). There is no risk in taking this package from testing, because it only contains the rules and does not depend on any other packages from testing/unstable. If you prefer faster updates, you can also use unstable.

# apt install libapache2-mod-security2 geoip-database
# apt install -t testing modsecurity-crs

Configuring ModSecurity

In order to load the ModSecurity module in Apache, run this command:

# a2enmod security2

Then copy the example ModSecurity configuration file to /etc/modsecurity/modsecurity.conf:

cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Now edit /etc/modsecurity/modsecurity.conf. I highlight some of the options:

SecRuleEngine on
SecRequestBodyLimit 536870912
SecRequestBodyNoFilesLimit 131072
SecAuditLog /var/log/apache2/modsec_audit.log
#SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
#"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"
SecPcreMatchLimit 500000
SecPcreMatchLimitRecursion 500000
SecStatusEngine Off

The SecRuleEngine option controls whether rules should be processed. If set to Off, you completely disable all rules, with On you enable them and it will block malicious actions. If set to DetectionOnly, ModSecurity will only log potential malicious activity flagged by your rules, but will not block them. DetectionOnly can be useful for temporary trying out the rules in order to find false positives before you really start blocking potential malicious activity.

The SecAuditLog option defines a file which contains audit logs. This file will contain detailed logs about every request triggering a ModSecurity rule.

The SecPcreMatchLimit and SecPcreMatchLimitRecursion set the match limit and match limit recursion for the regular expression library PCRE. Setting this high enough will prevent errors that the PCRE limits were exceeded while analyzing data, but setting it too high can make ModSecurity vulnerable to a Denial of Service (DoS) attack. A Core Rule Set developer recommends a value of 50000 so that’s what I use here.

I change SecRequestBodyLimit to a higher value to allow large file uploads.

I disable the rule 200004 because it is known to cause false positives.

Set SecStatusEngine to Off to prevent ModSecurity sending version information back its developers.

After changing any configuration related to ModSecurity or the Core Rule Set, reload your Apache web server:

# systemctl reload apache2

Configuring the Core Rule Set

The Core Rule Set can be configured via the file /etc/modsecurity/crs/crs-setup.conf.

Anomaly Scoring

By default the Core Rule Set is using anomaly scoring mode. This means that individual rules add to a so called anomaly score, which at the end is evaluated. If the anomaly score exceeds a certain threshold, then the traffic is blocked. You can read more about this configuration in crs-setup.conf but the default configuration should be fine for most people.

Setting the paranoia level

The paranoia level is a number from 1 to 4 which determines which rules are active and contribute to the anomaly scoring. The higher the paranoia level, the more rules are activated and hence the more aggressive the Core Rule Set is, offering more protection but potentially also causing more false positives. By default the paranoia level is set to 1. If you work with sensitive data, it is recommended to increase the paranoia level.

The executing paranoia level defines the rules which will be executed but their score will not be added to the anomaly scoring. When HTTP traffic hits rules of the executing paranoia level, this traffic will only be logged but not be blocked. It is a especially useful to prepare for increasing the paranoia level and finding false positives on this higher level, without causing any disruption for your users.

To set the paranoia level to 1 and the executing paranoia level to 2, make sure you have these rules set in crs-setup.conf:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.paranoia_level=1"
SecAction \
  "id:900001,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.executing_paranoia_level=2"

Once you have fixed all false positives, you can raise the paranoia level to 2 to increase security.

Defining the allowed HTTP methods

By default the Core Rule Set only allows the GET, HEAD, POST and OPTIONS HTTP methods. For many standard sites this will be enough but if your web applications also use restful APIs or WebDAV, then you will need to add the required methods. Change rule 900200, and add the HTTP methods mentioned in the comments in crs-setup.conf.

SecAction \
 "id:900200,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'"

Disallowing old HTTP versions

There is a rule which determines which HTTP versions you allow in HTTP requests. I uncomment it and modify it to only allow HTTP versions 1.1 and 2.0. Legitimate browsers and bots always use one of these modern HTTP versions and older versions usually are a sign of malicious activity.

SecAction \
 "id:900230,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.allowed_http_versions=HTTP/1.1 HTTP/2 HTTP/2.0'"

Blocking specific countries

Personally I’m not a fan of completely blocking all traffic from a whole country, because you will also block legitimate visitors to your site, but in case you want to this, you can configure this in crs-setup.conf:

SecGeoLookupDB /usr/share/GeoIP/GeoIP.dat
SecAction \
 "id:900600,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.high_risk_country_codes='"

Add the two-letter country codes you want to block to the last line (before the two quotes), multiple country codes separated by a space.

Make sure you have the package geoip-database installed.

Core Rule Set Exclusion rules for well-known web applications

The Core Rule Set contains some rule exclusions for some well-known web applications like WordPress, Drupal and NextCloud which reduces the number of false positives. I add the following section to crs-setup.conf which will allow me to enable the exclusions in the Apache configuration by setting the WEBAPPID variable in the Apache configuration whenever I need them.

SecRule WEBAPPID '@beginsWith wordpress' 'id:20000,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_wordpress=1'
SecRule WEBAPPID '@beginsWith drupal' 'id:20001,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_drupal=1'
SecRule WEBAPPID '@beginsWith dokuwiki' 'id:20002,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_dokuwiki=1'
SecRule WEBAPPID '@beginsWith nextcloud' 'id:20003,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_nextcloud=1'
SecRule WEBAPPID '@beginsWith cpanel' 'id:20004,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_cpanel=1'
SecRule WEBAPPID '@beginsWith xenforo' 'id:20005,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_xenforo=1'

Adding rules for Log4Shell and Spring4Shell detection

At the end of 2021 a critical vulnerability CVE-2021-44228, named Log4Shell, was detected in Log4j, which allows remote attackers to run code on a server with the vulnerable Log4j version. While the Core Rule Set offered some mitigation of this vulnerability out of the box, this protection was not complete. New improved detection rules against Log4Shell were developed. Because of the severity of this bug and the fact that it’s being exploited in the wild, I strongly recommend adding this protection manually when using ModSecurity version 3.3.2 (or older). Newer, not yet released versions, should have complete protection out of the box.

First modify /etc/apache2/mods-enabled/security2.conf so that it looks like this:

<IfModule security2_module>
        # Default Debian dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        # Include all the *.conf files in /etc/modsecurity.
        # Keeping your local configuration in that directory
        # will allow for an easy upgrade of THIS file and
        # make your life easier
        IncludeOptional /etc/modsecurity/*.conf

        # Include OWASP ModSecurity CRS rules if installed
        IncludeOptional /usr/share/modsecurity-crs/*.load
        SecRuleUpdateTargetById 932130 "REQUEST_HEADERS"
</IfModule>

Then create the file /etc/modsecurity/99-CVE-2021-44228.conf with this content:

# Generic rule against CVE-2021-44228 (Log4j / Log4Shell)
# See https://coreruleset.org/20211213/crs-and-log4j-log4shell-cve-2021-44228/
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx (?:\${[^}]{0,4}\${|\${(?:jndi|ctx))" \
    "id:1005,\
    phase:2,\
    block,\
    t:none,t:urlDecodeUni,t:cmdline,\
    log,\
    msg:'Potential Remote Command Execution: Log4j CVE-2021-44228', \
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

In March 2022 CVE-2022-22963, another remote code execution (RCE) vulnerability was published in the Spring framework was published. The Core Rule Set developed a new rule to protect against this vulnerability which will be included in the next version, but the rule can be added manually if you are running the Core Rule Set version 3.3.2 or older.

To do so, create the file /etc/modsecurity/99-CVE-2022-22963.conf with this content:

# This rule is also triggered by the following exploit(s):
# - https://www.rapid7.com/blog/post/2022/03/30/spring4shell-zero-day-vulnerability-in-spring-framework/
# - https://www.ironcastle.net/possible-new-java-spring-framework-vulnerability-wed-mar-30th/
#
SecRule ARGS|ARGS_NAMES|REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@* \
    "@rx (?:class\.module\.classLoader\.resources\.context\.parent\.pipeline|springframework\.context\.support\.FileSystemXmlApplicationContext)" \
    "id:1006,\
    phase:2,\
    block,\
    t:urlDecodeUni,\
    msg:'Remote Command Execution: Malicious class-loading payload',\
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',\
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/248',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/2',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl2=+%{tx.critical_anomaly_score}'"

Don’t forget to reload your Apache configuration after adding these rules.

Testing ModSecurity and checking the logs

We can now easily test ModSecurity by doing a request which tries to abuse a cross-site scripting (XSS) vulnerability:

$ curl -I "https://example.org/?search=<script>alert('CRS+Sandbox+Release')</script>"

This should return HTTP response 403 (Forbidden).

Whenever something hits your ModSecurity rules, this will be logged in your Apache error log. The above request has created these messages in the error log:

[Sat Apr 09 22:22:02.716558 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. detected XSS using libinjection. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "55"] [id "941100"] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.716969 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Pattern match "(?i)<script[^>]*>[\\\\s\\\\S]*?" at ARGS:search. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "82"] [id "941110"] [msg "XSS Filter - Category 1: Script Tag Vector"] [data "Matched Data: <script> found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.717249 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Pattern match "(?i:(?:<\\\\w[\\\\s\\\\S]*[\\\\s\\\\/]|['\\"](?:[\\\\s\\\\S]*[\\\\s\\\\/])?)(?:on(?:d(?:e(?:vice(?:(?:orienta|mo)tion|proximity|found|light)|livery(?:success|error)|activate)|r(?:ag(?:e(?:n(?:ter|d)|xit)|(?:gestur|leav)e|start|drop|over)|op)|i(?:s(?:c(?:hargingtimechange ..." at ARGS:search. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "199"] [id "941160"] [msg "NoScript XSS InjectionChecker: HTML Injection"] [data "Matched Data: <script found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.718018 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "93"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 15)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.718596 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/usr/share/modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "91"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 15 - SQLI=0,XSS=15,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 15, 0, 0, 0"] [ver "OWASP_CRS/3.3.2"] [tag "event-correlation"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]

In the first 3 lines we see that we hit different filters which check for XSS vulnerabilities, more specifically rules 941100, 941110 and 941160 all of them having the tag paranoia-level/1.

Then the fourth line shows that we hit rule 949110 which caused the web server to return the HTTP 403 Forbidden response because the inbound anomaly score, 15, is higher than 5. Then rule 980130 gives us some more information about the scoring: we hit a score of 15 at the paranoia level 1, while rules at the other paranoia levels rules contributed 0 to the total score. We also see the scores for individual types of attack: in this case all 15 points where scored by rules detecting XSS attacks. This is the meaning of the different abbreviations used:

SQLISQL injection
XSScross-site scripting
RFIremote file inclusion
LFIlocal file inclusion
RCEremote code execution
PHPIPHP injection
HTTPHTTP violation
SESSsession fixation

More detailed logs about the traffic hitting the rules can be found in the file /var/log/apache2/modsec_audit.log.

Fixing false positives

First of all, in order to minimize the amount of false positives, you should set the WEBAPPID variable if you are using one of the known web applications for which the Core Rule Set has a default exclusion set. These web applications are currently WordPress, Drupal, Dokuwiki, Nextcloud, Xenforo and cPanel. You can do so by using the <a href="https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecWebAppId">SecWebAppId</a> option in a VirtualHost of Location definition in the Apache configuration. For example if you have a VirtualHost which is used by Nextcloud, set this within the VirtualHost definition:

<Virtualhost nextcloud.example.org>
    ...OTHER OPTIONS HERE...
    <IfModule security2_module>
        SecWebAppId "nextcloud"
    </IfModule>
</VirtualHost>

If you have a WordPress installation in a subdirectory, then add SecWebAppId within Location tags.

<Location /wordpress>
    <IfModule security2_module>
        SecWebAppId "wordpress-mysite"
    </IfModule>
</Location>

If you have multiple WordPress sites, give each of them a unique WEBAPPID which name starts with wordpress. Add a different suffix for every instance so that each one run its in own application namespace in ModSecurity.

If you still encounter false positives, you can completely disable rules by using the configuration directive SecRuleRemoveById. I strongly recommend not disabling rules globally, but limiting its removal to the specific location from which you want them to be removed, for example by putting them with <Location> or <LocationMatch> tags in the Apache configuration. For example:

<LocationMatch ^/wp-admin/(admin-ajax|post)\.php>
    <IfModule security2_module>
        SecRuleRemoveById 941160 941100 941130 932105 932130 932100
    </IfModule>
</LocationMatch>

Pay attention not to disable any of the 949*, 959*, and 980* rules: disabling the 949* and 959* rules would disable all the blocking rules, while disabling the 980* rules would give you less information about what is happening in the logs.

Conclusion

ModSecurity and the Core Rule Set offer an additional security layer for web servers in your defence in depth strategy. I strongly recommend implementing this on your servers because it makes it harder to abuse security vulnerabilities.

Keep an eye on the Core Rule Set blog and Twitter account: sometimes they post new rules for specific new critical vulnerabilities, which can be worthwhile to add to your configuration.

Fixing crackling/popping while playing music in Debian GNU/Linux

I was experiencing crackling/popping sounds while playing music with Rhythmbox in my Debian GNU/Linux Testing (Bookworm) system. The noises start when starting music playback and stop as soon as I stop the playback.

I came around this bug report for Pipewire but I’m currently still using PulseAudio 15 instead. However it contained a comment which led to the solution: speech-dispatcher is known for causing problems of crackling sounds.

Speech-dispatcher is used for speech synthesis often used by blind or visually impaired people. If you don’t use this functionality, you can either disable speech-dispatcher (speechd) or completely uninstall it:

Disable speech-dispatcher by editing /etc/speech-dispatcher/speechd.conf, removing the comment sign # before this line near the end of the file:

DisableAutoSpawn

Then you will have to log out and log in your desktop or you can kill all speech-dispatcher processes manually.

If you want, you can remove speech-dispatcher completely by running this command:

# apt remove speech-dispatcher

Note that this will remove the gnome meta-package, which in itself is harmless, but might lead to new GNOME components not being installed automatically in the future.

This fixed the constant popping crackling noises while playing music completely for me.

Missing video thumbnails in Nautilus in Debian Bullseye

I am using Debian Bullseye and already for a long time I noticed that Nautilus failed to generate thumbnails for certain video files, leading to only a generic video icon instead of a thumbnail.

In the kernel log (dmesg), you will see this error:

qtdemux0:sink[70839]: segfault at 0 ip 0000000000000000 sp 00007f724fe61d18 error 14 in totem-video-thumbnailer[5638abe9b000+3000]
Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.

This is Debian bug #967941: gnome-video-thumbnailer crashes with a segmentation fault when thumbnailing H.264 encoded video files when you have the package libopenblas0-pthread installed.

Available Work-arounds

Remove libopenblas0-pthread

# apt-get remove libopenblas0-pthread

This might not be possible if you need this package for other reasons.

For performance reasons you might also want to install the BLIS BLAS implementation and set it as default. Install the packages libblis3-pthread and libblis64-3-pthread and set them default using the update-alternatives commands from the next work-around.

Switch the default BLAS implementation from OpenBlas to Atlas

If you cannot uninstall libopenblas0-pthread, you can change the default BLAS and LAPACK implementation on your system to a different implementation than OpenBLAS

# update-alternatives --config libblas64.so.3-x86_64-linux-gnu
# update-alternatives --config libblas.so.3-x86_64-linux-gnu
# update-alternatives --config liblapack.so.3-x86_64-linux-gnu

Choose a different implementation than openblas in all 3 cases. Usually Atlas will be installed and available by default. You can also install the packages libblis3-pthread and libblis64-3-pthread and choose the BLIS implementation for the former two and Atlas for the latter.

Disable the sandbox in totem-video-thumbnailer

Totem-video-thumbnailer uses some kind of sandbox limiting how much CPU and memory resources the process can use. This enhances security and prevents the process from using all resources in case something would go wrong. OpenBLAS causes problems in combination with these restrictions. You can disable them in totem-video-thumbnailer by calling it with the -l option. To automatically let GNOME call it with that option, edit the file /usr/share/thumbnailers/totem.thumbnailer and edit the Exec= line so that it looks like this:

Exec=/usr/bin/totem-video-thumbnailer -l -s %s %u %o

If you can, rather use one of two previous work-arounds, because disabling this restrictions could have security implications.

Regenerating the failed thumbnails

Finally after implementing one of the 3 work-arounds, you will need to regenerate the failed thumbnails. GNOME Thumbnail Factory keeps a list of all failed thumbnails, so that it does not retry to generate them over and over again. They are stored in ~/.cache/thumbnails/fail/gnome-thumbnail-factory so you can just delete this directory:

rm -rf ~/.cache/thumbnails/fail/gnome-thumbnail-factory

Now when you browse a directory which contains video files with missing thumbnails with Nautilus, it should generate them automatically.

Setting up Linux on a desktop or laptop system, part 1: choosing a Linux distribution and a desktop environment

Which Linux distribution?

The most widely used distribution is Ubuntu. Other popular desktop distributions are Linux Mint, Fedora, OpenSUSE Tumbleweed and Manjaro. Personally I prefer to use Debian, not only on servers but also on desktop systems, but this distribution does require more manual work to set up on a desktop system, so it is not the easiest choice if you are new to Linux. In that case I would recommend some of the Ubuntu variants or Linux Mint.

Choose a recent version of your Linux distribution of choice. If you use Ubuntu, install the latest LTS version (20.04 Focal Fossa at the time of writing) and if you encounter problems related to hardware support, consider using the latest non-LTS version (20.10 Groovy Gorilla will be available end of October 2020. You can download daily builds if you want to test this version in development). I recommend Debian users to install the testing version on recent hardware.

Which desktop environment?

You will have to choose which desktop environment you want to use on your Linux system. Depending on your distribution, you make this choice before downloading the ISO, or during the installation.

The most widely used desktop environments are GNOME, KDE Plasma, Cinnamon and XFCE. The first three desktop environments get updates and improvements more often than XFCE and are more complete: for this reason I would recommend one of these. XFCE on the other hand is interesting as a lightweight desktop for older hardware.

KDE is very customizable, but the many configuration options can be overwhelming at times. GNOME is less customizable and tries to deliver a user-friendly modern desktop out of the box. If you want a simple clean desktop with a more traditional, Windows- or macOS-like desktop, then you can consider Cinnamon.

In the next table you will find links to the different editions of popular distributions. Some distributions have a default or preferred desktop, one in which they invest most work. I have set these in bold.

GNOMEKDE PlasmaCinnamonXFCE
UbuntuUbuntu DesktopKubuntuUbuntu Cinnamon RemixXubuntu
Linux Mintno installer availableno installer availableLinux Mint CinnamonLinux Mint XFCE
FedoraFedora WorkstationFedora KDE PlasmaFedora CinnamonFedora XFCE
OpenSUSE TumbleweedYou choose your desktop during the installation: OpenSUSE Tumbleweed installation ISOs. Cinnamon can only be added after the installation.
ManjaroManjaro GNOMEManjaro KDE PlasmaManjaro CinnamonManjaro XFCE
DebianYou choose your desktop during the installation: Debian Testing installer

If you like KDE Plasma, you can also consider installing the KDE Neon distribution. It is an Ubuntu LTS with the latest KDE packages installed. This way it provides a more up to date and more standard KDE experience than Kubuntu. However it misses a lot of standard non-KDE application, which you will have to install yourself afterwards.

If you want to get a feeling of these distributions and desktops, you can test them out in a web browser on the website distrotest.net. All in all, I would recommend not spending too much time on choosing your distribution and desktop. Pick one of the beginner-friendly distributions and try it. If you really don’t like it, it is easy enough to install another distribution.