Protecting your server from known bad IPs with Foomuuri iplists

On the Internet we can find (usually crowdsourced) lists of malicious IP addresses responsible for attacks. We can easily integrate them in Foomuuri in order to block connections from these bad hosts. Not only does this improve security, it is also a performance win, because our daemons don’t don’t have to waste any more time dealing with these malicious connections.

The blocklists

  • blocklist.de: a crowdsourced list of IP addresses involved in all kinds of attacks
  • techmdw blacklist: this list is compiled by the Swedish company TechMDW AB and is based on Crowdsec with their own additions
  • Greensow
  • Spamhaus DROP: The Don’t Route or Peer List by Spamhaus contains netblocks which you should never interact with because they are leased or stolen by criminal organisations
  • Emerging Threats: compilation of Spamhaus DROP list, the top attackers list by DShield and lists by abuse.ch
  • Interserver: list consisting of IP addresses attacking servers of the web hosting company Interserver
  • Stopforumspam: list of toxic IP addresses which are believed to be used only for spamming websites

Blocking incoming connections from malicious IPs

Create file/etc/foomuuri/iplist.conf with these contents:

iplist {
	@blocklist_de https://lists.blocklist.de/lists/all.txt refresh=15m
        @techmdw https://blacklist.techmdw.com/ refresh=30m
	@greensnow https://blocklist.greensnow.co/greensnow.txt refresh=30m
	@et https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt refresh=24h
	@interserver https://rbldata.interserver.net/ip.txt refresh=15m
	@stopforumspam https://www.stopforumspam.com/downloads/toxic_ip_cidr.txt refresh=2h
        @spamhausdrop https://www.spamhaus.org/drop/drop.txt refresh=24h
        @spamhausdropv6 https://www.spamhaus.org/drop/dropv6.txt refresh=24h
}

Then in the public-localhost section (which I usually put in /etc/foomuuri/public-localhost.conf), add this at the top:

        saddr @blocklist_de drop counter "blocklist.de"
        saddr @techmdw drop counter "techmdw" 
        saddr @greensnow drop counter "greensnow"
        saddr @spamhausdrop drop counter "spamhausdrop"
        saddr @spamhausdropv6 drop counter "spamhausdropv6"
        saddr @et drop counter "et"
        saddr @interserver drop counter "interserver"
        saddr @stopforumspam drop counter "stopforumspam"

In order to prevent getting locked out of your system yourself because of a false positive or an error in one of the lists, I recommend adding one rule which allows you to always access your system before these rules. For example to make sure you can always connect from IP xxx.xxx.xxx.xxx put this as first rule in public-localhost section:

        ssh saddr xxx.xxx.xxx.xxx accept

In the rules above I’m not logging all the details of the dropped connections, but I’m keeping a counter so that I can see how many times such a rule has been hit. You can use the command

# foomuuri list counter

to see how many packets and bytes have been dropped by these rules.

If you want to log all individual dropped connections , you can add this at the end of every line:

log "name_of_the_blocklsit"

They will then be logged with the prefix name_of_the_blocklist.

Rejecting outgoing connections to malicious IPs

We can also block outgoing connections. You should especially do this for the Spamhaus DROP lists. Add this to the localhost-public section:

        daddr @blocklist_de reject counter "out_blocklist.de" log "out_blocklist.de"
        daddr @techmdw reject counter "out_techmdw" log "out_techmdw"
        daddr @greensnow reject counter "out_greensnow" log "out_greensnow"
        daddr @spamhausdrop reject counter "out_spamhausdrop" log "out_spamhausdrop"
        daddr @spamhausdropv6 reject counter "out_spamhausdropv6" log "out_spamhausdropv6"
        daddr @et reject counter "out_et" log "out_et"
        daddr @interserver reject counter "out_interserver" log "out_interserver"
        daddr @stopforumspam reject counter "out_stopforumspam" log "out_stopforumspam"

Notice that I’m rejecting instead of dropping these connections so that applications don’t keep on waiting until the connection attempt times out and I’m logging these. Normally these rules should only very rarely get triggered, but if they do you want detailed logs so you easily investigate what’s going on.

Dropping or allowing incoming connections by country of origin

Another very effective method to prevent abuse is to limit connections to services like SSH and your mail server to certain countries of origin. You can find lists of IP (both IPv4 and IPv6) addresses per country on https://github.com/ipverse/rir-ip/tree/master/country . You can add them to an iplist in Foomuuri and then use these in the public-localhost section. Note that these lists are not perfect and sometimes connections can come from another country than from the one the IP address is registered to in this database. Especially public VPN services sometimes suffer from these problems, so be careful if you are using these.

You can also use this aggregated list of all European IP addresses, but unfortunately that list only exists for IPv4 addresses.

To use this aggregated list, add this to the iplist section:

        @europe https://ipv4.fetus.jp/krfilter.4.txt refresh=24h 

Then with these rules in the the public-localhost section, I only allow IPv4 connections to port 143 (IMAP), 993 (IMAPs), port 587 (Submission), port 465 (Submissions) from European IPs. Note that I allow IPv6 from the whole world because this aggregated list only contains IPv4 addresses.

        ssh ipv4 saddr @europe
        ssh ipv6
        ssh drop counter "non-europe" log "non-europe"

        imap ipv4 saddr @europe
        imap ipv6
        imap drop counter "non-europe" log "non-europe"
        imaps ipv4 saddr @europe
        imaps ipv6
        imaps drop counter "non-europe" log "non-europe"

        submission ipv4 saddr @europe
        submission ipv6
        submission drop counter "non-europe" log "non-europe"
        submissions ipv4 saddr @europe
        submissions ipv6
        submissions drop counter "non-europe" log "non-europe"

More information

https://github.com/FoobarOy/foomuuri/wiki/Configuration#iplist

Some various performance improvements for Debian 12 Bookworm

Here some various improvements I implemented on some of my Debian 12 Bookworm servers in order to improve performance.

zswap: use zsmalloc allocator with newer kernel

If your system has little memory, you might be using zswap already. When memory is getting full, the system will try to swap out less used data from memory to a compressed swap in memory instead of writing it immediately to a swap partition or swap file on slower storage. In Linux kernel version 6.7 the zsmalloc allocator, which is superior to other allocators (zbud and z3fold), became the default.

So first upgrade to a more recent kernel. You can get a recent kernel from bookworm-backports or Debian testing or unstable.

To enable zswap at boot you can create a file named /etc/default/grub.d/zswap.cfg which contains:

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=30"

If you want higher compression at the cost of more CPU time, you can replace lz4 by zstd.

You will need to add the compression module of your preference to your initramfs. So in the case of lz4, just add

lz4

to /etc/initramfs-tools/modules and then run

# update-initramfs -u

Finally execute

# update-grub

to update your Grub configuration so that these settings will become automatically active at the next boot.

I upgraded on to Linux 6.11 on a VPS with 2GB of RAM with these settings, and the system feels much snappier now.

To check effectiveness of zswap, you can use the zswap-stats script.

Update to systemd 254 or higher to improve behaviour under memory pressure

systemd 254 includes a change which makes journald and resolved flush their caches when the system is under memory pressure. This will free memory, reducing swapping. When this happens, this can be found in the logs:

systemd-journald[587721]: Under memory pressure, flushing caches.

You can find systemd 254 for Debian 12 in bookworm-backports.

Exclude cron from audit logging

On one of my systems, my audit logs where rapidly filling. You can check whether this is happening for you by looking at the dates of the files in /var/log/audit/:

# ls -l /var/log/audit/

By default auditd will write files up to 8 MB, after which it will rotate the file. If these different files have their modification date very close to each others, then you might consider reducing the logging.

Possible causes for audit logs filling up are Apparmor logs. Improve your Apparmor profiles to reduce warnings and errors. On one of my system, the cause was the logging caused by the execution of cron jobs. Especially because mailman3-web contains a cron job which is executed every single minute.

To prevent logging everything related to cron, create a file /etc/audit/rules.d/excludecron.rules:

-a exclude,always -F exe=/usr/sbin/cron

Then run

# augenrules --load

to load the new rules.

Securing PHP-FPM with AppArmor

PHP-FPM is an ideal candidate to secure with AppArmor. Not only can the security of a web server be endangered by security bugs in PHP itself, it can also be affected by security holes in PHP applications. By confining PHP-FPM with AppArmor, we can limit abuse when a security hole is exploited, by preventing PHP-FPM for example from reading arbitrary files on your system or executing random binaries, which may contain a Linux backdoor or crypto-miner malware.

Preparation

First it is important that you run different PHP web applications as different users by running them in a different pool. The different pools can then be confined with their own separate AppArmor subprofile or hat, so that we can protect the different PHP applications from each other.

Then on Debian 12 Bookworm, I recommend that you upgrade to the AppArmor packages from my own repository, because they contain some important bug fixes, notably aa-logprof and aa-genprof supporting exec events in hats. Set up the bookworm-frehi repository in apt and upgrade AppArmor to the version available in this repository:

# apt install -t bookworm-frehi apparmor

Now we are ready to create our AppArmor profile for PHP-FPM. Debian already ships a basic profile /etc/apparmor.d/php-fpm as part of the package apparmor-profiles. However aa-logprof and aa-genprof will want to write everything in one single file, while the /etc/apparmor.d/php-fpm heavily relies on including other files and also the comments in that file will be lost. For that reason I choose to create my own profile from scratch with aa-genprof. So first I disable the php-fpm profile and remove it:

# aa-disable php-fpm
# rm /etc/apparmor.d/disable/php-fpm
# rmdir /etc/apparmor.d/php-fpm.d/

Generating a profile for /usr/sbin/php-fpm8.2

I start aa-genprof:

# aa-genprof /usr/sbin/php-fpm8.2

In another shell, I stop PHP-FPM:

# systemctl stop php8.2-fpm

and now I modify all pools in /etc/php/8.2/fpm/pool.d/*.conf and I add the apparmor_hat value to the name of the pool. For example:

# apparmor_hat = myblog

This instructs PHP-FPM to switch to this subprofile or hat for this pool. If every pool uses a different hat, we can isolate the different pools running different web applications from each other.

Now verify with aa-status whether /usr/sbin/php.php-fpm8.2 is in complain mode and if not, put it in complain mode manually with the aa-complain command:

# aa-complain /etc/apparmor.d/usr.sbin.php-fpm8.2

Now we start PHP-FPM:

# systemctl start php8.2-fpm

It’s a good idea to immediately press S to scan for the first events. Process them like I showed in my previous AppArmor tutorial. As usual, in case of doubt choose the more fine-grained option instead of broad abstractions.

I will discuss some specific things you will encounter.

You will see that PHP-FPM requests read access to /etc/passwd and /etc/group. It needs this in order to look up the UID and GID of the user and group value you have set in your FPM pools. As we are running our different pools in their own subprofiles, which don’t have access to these files, this is not a security problem: your PHP applications won’t have access to these files if you have properly set apparmor_hat in every pool.

At some point our PHP-FPM process switches to the profile /usr/sbin/php-fpm8.2//null-/usr/sbin/php-fpm8.2//myblog. This is because we have set apparmor_hat = myblog in the myblog pool. The null part in the profile name indicates that this profile does not exist yet and so AppArmor uses this as a temporary name. We will have to fix this manually later. Now accept this rule.

Profile:        /usr/sbin/php-fpm8.2
Exec Condition: ALL
Target Profile: /usr/sbin/php-fpm8.2//null-/usr/sbin/php-fpm8.2//myblog

 [1 - change_profile -> /usr/sbin/php-fpm8.2//null-/usr/sbin/php-fpm8.2//myblog,]
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish

Continue processing all events, and when you are done save the profile and quit aa-genprof.

Check with aa-status in which mode the /usr/sbin/php-fpm8.2 profile is and switch it back to complain mode if necessary. Otherwise if it’s in enforce mode, your PHP applications will probably not work correctly any more.

We will now edit the file /etc/apparmor.d/usr.sbin.php-fpm8.2 in a text editor. Find the change_profile rules in that file, and replace them by this line:

change_profile -> /usr/sbin/php-fpm8.2//*,

This allows PHP-FPM to switch to all PHP-FPM hats.

In the file, you will also find some signal rules which refer to the temporary null profile. We need to fix these too like this:

  signal send set=kill peer=/usr/sbin/php-fpm8.2//*,<br />  signal send set=quit peer=/usr/sbin/php-fpm8.2//*,<br />  signal send set=term peer=/usr/sbin/php-fpm8.2//*,

Then within the profile /usr/sbin/php-fpm8.2 flags=(complain) {

create an empty myblog hat in complain mode:

  ^myblog flags=(complain) {
  }

Now reload the AppArmor profile:

# apparmor_parser -r /etc/apparmor.d/usr.sbin.php-fpm8.2

Restart aa-genprof:

# aa-genprof /usr/sbin/php-fpm8.2

and restart PHP-FPM in another shell:

# systemctl restart php8.2-fpm

Now exercise your web applications and process all generated events. Particularly test posting new items on your website and uploading pictures and other files. You will notice that the events triggered by the www pool, are now in the hat myblog, which is shown by aa-genprof as /usr/sbin/php-fpm8.2^myblog.

In WordPress I am using the WebP Express plugin, which automatically creates a webp file for every uploaded image by calling the command cwebp.

aa-genprof will show this event:

Profile:  /usr/sbin/php-fpm8.2^myblog
Execute:  /usr/bin/dash
Severity: unknown

(I)nherit / (N)amed / (U)nconfined / (X) ix On / (D)eny / Abo(r)t / (F)inish

So the myblog pool tries to launch the dash shell as part of this process. By pressing I you can let it inherit the www profile, so it will have exactly the same permissions and cannot read or modify any files from other web applications running in a different PHP-FPM pool, or other files on the system. I also had to do this for the cwebp executable and a few others which are used by WebP Express. Important is that you never choose unconfined, because then you allow these executables to do anything on your system without any AppArmor restrictions.

When WordPress sends e-mail, it will call sendmail:

[(S)can system log for AppArmor events] / (F)inish
Reading log entries from /var/log/audit/audit.log.
Target profile exists: /etc/apparmor.d/usr.sbin.sendmail

Profile:  /usr/sbin/php-fpm8.2^myblog
Execute:  /usr/sbin/sendmail
Severity: unknown

(I)nherit / (P)rofile / (N)amed / (U)nconfined / (X) ix On / (D)eny / Abo(r)t / (F)inish

Sendmail should run in its own profile, so you can choose P here.

Creating a child profile for dash

In our example our PHP-FPM worker calls dash and cwebp to process images. We have chosen to let dash and cwebp inherit all permissions of the worker. While this already is a great security improvement because it will prevent dash from executing random binaries and reading files unrelated to this web application it is still too broad. There is no reason why dash and cwebp should be able to read our PHP files for example and any other files than webp files. So we can improve our security even more by running these commands in a separate child profile.

To do so, I edit /etc/apparmor.d/usr.sbin.php-fpm8.2 in a text editor, and I remove all these lines from the www hat:

    /usr/bin/cwebp mrix,
    /usr/bin/dash mrix,
    /usr/bin/nice mrix,
    /usr/bin/cwebp mrix,
    /usr/bin/which.debianutils mrix,

and I replace it by this rule which will force dash to run in a child profile named myblogdash:

/usr/bin/dash Px -> /usr/sbin/php-fpm8.2//myblogdash,

Then in the usr/sbin/php-fpm8.2 profile, but outside the myblog hat, I create this empty child profile:

  profile myblogdash flags=(complain) {
  }

reload the profile, restart PHP-FPM and start aa-genprof again:

# /etc/apparmor.d/usr.sbin.php-fpm8.2
# systemctl restart php8.2-fpm
# aa-genprof /usr/sbin/php8.2-fpm

Now I upload again an image in order to trigger the execution of dash and and I process the generated events with aa-genprof. Finally I end up with this child profile:

  profile myblogdash {
    include <abstractions/base>
    include <abstractions/bash>

    deny /var/www/example.com/wordpress/wp-admin/admin-ajax.php r,
    deny /var/www/example.com/wordpress/wp-admin/async-upload.php r,
    deny /var/www/example.com/wordpress/wp-admin/options-general.php r,
    deny /var/www/blog.frehi.be/wordpress/xmlrpc.php r,

    /usr/bin/cwebp mrix,
    /usr/bin/dash mr,
    /usr/bin/nice mrix,
    /usr/bin/which.debianutils mrix,
    owner /var/www/example.com/wordpress/**.{jpg,jpeg,png} r,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossless.webp w,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossy.webp w,
    owner /var/www/example.com/wordpress/wp-content/webp-express/webp-images/doc-root/**.webp w,

  }

As you can see our dash process and all other processes which inherit this profile, including cwepb, now have much more limited permissions and can only read JPEG and PNG files from our website and write WEBP images. I had some events of the sh process reading files like wp-admin/options-general.php, but I have no idea why this process would do that. I chose to deny this, and even when this child profile is in enforce mode, everything appeared to be working OK. Anyway, these files don’t contain anything sensitive, so it would not be a real problem if you accept this.

Write permissions on PHP files

A topic which is worth thinking about: which permissions are you giving your PHP worker process on the PHP files? Obviously PHP-FPM will need read access to these files in order to execute them, but do you also want to give them write access? There is a strong argument not too: it will prevent hackers who manage to exploit your PHP application to write a new PHP file with their own code (such as a PHP backdoor) and then execute it. Or modify your existing PHP files and insert malicious code in them. So confining your PHP process to prevent it from writing PHP files is a huge advantage for security.

However applications like WordPress will try to update themselves and in order to do so, it needs write access to its own files. Also installing new plug-ins via the web interface requires the permissions to write PHP files. Automatic updates is also a big advantage for security. As an alternative to WordPress updating itself automatically, you could easily write a shell script which uses wp-cli to update WordPress and all modules and themes and regularly call this from a cron job or systemd timer.

On my system, I noticed that the wp-supercache module was writing some PHP files. If I understand this correctly, it does this in order to deal with pages which contain dynamic content. So in order not to break this, I need to allow writing PHP files in wp-content/cache/supercache anyway.

You will have to decide for each web application whether you want it to allow writing PHP files. At least run your different PHP applications in different PHP-FPM workers with different AppArmor hats, so that it’s impossible for one PHP applications to affect other PHP applications, but also strongly consider not giving write access to its own PHP files because that would be a huge win security wise.

If you decide to give a web application write access to its own PHP files, I recommend even more setting up Modsecurity with the CoreRuleSet, blocking outgoing network connections by default for that user account with your firewall (such as Foomuuri) and installing Snuffleupagus to further limit the things PHP applications can do on a PHP level.

The usr.sbin.php-fpm8.2 profile

Finally I ended up with the following profile.

abi <abi/3.0>,

include <tunables/global>

/usr/sbin/php-fpm8.2 {
  include <abstractions/base>

  capability chown,
  capability dac_override,
  capability kill,
  capability net_admin,
  capability setgid,
  capability setuid,

  signal send set=kill peer=/usr/sbin/php-fpm8.2//*,
  signal send set=quit peer=/usr/sbin/php-fpm8.2//*,
  signal send set=term peer=/usr/sbin/php-fpm8.2//*,

  /run/php/*.sock w,
  /usr/sbin/php-fpm8.2 mr,
  @{PROC}/@{pid}/attr/{apparmor/,}current rw,
  owner /etc/group r,
  owner /etc/host.conf r,
  owner /etc/hosts r,
  owner /etc/nsswitch.conf r,
  owner /etc/passwd r,
  owner /etc/php/8.2/fpm/conf.d/ r,
  owner /etc/php/8.2/fpm/php-fpm.conf r,
  owner /etc/php/8.2/fpm/php.ini r,
  owner /etc/php/8.2/fpm/pool.d/ r,
  owner /etc/php/8.2/fpm/pool.d/*.conf r,
  owner /etc/php/8.2/mods-available/*.ini r,
  owner /etc/resolv.conf r,
  owner /etc/ssl/openssl.cnf r,
  owner /proc/sys/kernel/random/boot_id r,
  owner /run/php/php8.2-fpm.pid w,
  owner /run/systemd/userdb/ r,
  owner /sys/devices/system/node/ r,
  owner /sys/devices/system/node/node0/meminfo r,
  owner /tmp/.ZendSem.* rwk,
  owner /var/log/php8.2-fpm.log w,

  change_profile -> /usr/sbin/php-fpm8.2//*,


  ^myblog {
    include <abstractions/base>
    include <abstractions/ssl_certs>

    network inet dgram,
    network inet stream,
    network inet6 dgram,
    network inet6 stream,
    network netlink raw,
    network unix stream,

    signal receive set=kill peer=/usr/sbin/php-fpm8.2,
    signal receive set=quit peer=/usr/sbin/php-fpm8.2,
    signal receive set=term peer=/usr/sbin/php-fpm8.2,

    /etc/gai.conf r,
    /etc/hosts r,
    /tmp/.ZendSem.* k,
    /usr/bin/dash Px -> /usr/sbin/php-fpm8.2//blogdash,
    /usr/sbin/postdrop Px,
    /usr/sbin/sendmail Px,
    /var/www/example.com/wordpress/**.php r,
    /var/www/example.com/wordpress/**.{css,js,svg,po} r,
    /var/www/example.com/wordpress/**/ r,
    /var/www/example.com/wordpress/.htaccess r,
    /var/www/example.com/wordpress/wp-content/uploads/wpcf7_uploads/.htaccess r,
    /var/www/example.com/wordpress/wp-includes/*.json r,
    /var/www/example.com/wordpress/wp-includes/certificates/ca-bundle.crt r,
    owner /var/www/example.com/tmp/* rw,
    owner /var/www/example.com/wordpress/.maintenance rw,
    owner /var/www/example.com/wordpress/wp-content/autoptimize_404_handler.php rw,
    owner /var/www/example.com/wordpress/wp-content/cache/**/ rw,
    owner /var/www/example.com/wordpress/wp-content/cache/**/index.html w,
    owner /var/www/example.com/wordpress/wp-content/cache/*.tmp rw,
    owner /var/www/example.com/wordpress/wp-content/cache/autoptimize/.htaccess w,
    owner /var/www/example.com/wordpress/wp-content/cache/autoptimize/css/*.css rw,
    owner /var/www/example.com/wordpress/wp-content/cache/autoptimize/js/*.js rw,
    owner /var/www/example.com/wordpress/wp-content/cache/example.com_wp_cache_gc.txt w,
    owner /var/www/example.com/wordpress/wp-content/cache/lyteCache/*.jpg rw,
    owner /var/www/example.com/wordpress/wp-content/cache/preload_permalink.txt rw,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/**.html w,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/**.html.gz rw,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/**.php rw,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/**.tmp rw,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/**.tmp.gz rw,
    owner /var/www/example.com/wordpress/wp-content/cache/supercache/*/feed/*.php rw,
    owner /var/www/example.com/wordpress/wp-content/cache/taxonomy_category.txt rw,
    owner /var/www/example.com/wordpress/wp-content/cache/taxonomy_post_tag.txt rw,
    owner /var/www/example.com/wordpress/wp-content/languages/**.{po,mo} rw,
    owner /var/www/example.com/wordpress/wp-content/languages/plugins/*.json w,
    owner /var/www/example.com/wordpress/wp-content/plugins/**.json r,
    owner /var/www/example.com/wordpress/wp-content/plugins/*/ rw,
    owner /var/www/example.com/wordpress/wp-content/plugins/webp-express/lib/options/options/**.inc r,
    owner /var/www/example.com/wordpress/wp-content/plugins/webp-express/lib/options/options/conversion-options/*.inc r,
    owner /var/www/example.com/wordpress/wp-content/plugins/webp-express/test/*.jpg r,
    owner /var/www/example.com/wordpress/wp-content/plugins/webp-express/test/*.png r,
    owner /var/www/example.com/wordpress/wp-content/themes/.htaccess rwk,
    owner /var/www/example.com/wordpress/wp-content/themes/webp-express-test-images/*.JPEG rw,
    owner /var/www/example.com/wordpress/wp-content/themes/webp-express-test-images/*.PNG rw,
    owner /var/www/example.com/wordpress/wp-content/upgrade-temp-backup/**/ rw,
    owner /var/www/example.com/wordpress/wp-content/upgrade-temp-backup/plugins/** rw,
    owner /var/www/example.com/wordpress/wp-content/upgrade/** rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/**.{jpg,jpeg,png,webp} rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/**/ rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/.htaccess rwk,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp w,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossless.webp w,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossy.webp rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-images/*.JPEG rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-images/*.PNG rw,
    owner /var/www/example.com/wordpress/wp-content/uploads/wp-statistics/GeoLite2-Country.mmdb r,
    owner /var/www/example.com/wordpress/wp-content/webp-express/config/*.json rw,
    owner /var/www/example.com/wordpress/wp-content/webp-express/webp-images/.htaccess rwk,
    owner /var/www/example.com/wordpress/wp-content/webp-express/webp-images/doc-root/**.webp rw,

  }

  profile myblogdash {
    include <abstractions/base>
    include <abstractions/bash>

    deny /var/www/example.com/wordpress/wp-admin/admin-ajax.php r,
    deny /var/www/example.com/wordpress/wp-admin/async-upload.php r,
    deny /var/www/example.com/wordpress/wp-admin/options-general.php r,
    deny /var/www/example.com/wordpress/xmlrpc.php r,

    /usr/bin/cwebp mrix,
    /usr/bin/dash mr,
    /usr/bin/nice mrix,
    /usr/bin/which.debianutils mrix,
    owner /var/www/example.com/wordpress/**.{jpg,jpeg,png} r,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossless.webp w,
    owner /var/www/example.com/wordpress/wp-content/uploads/webp-express-test-conversion.webp.lossy.webp w,
    owner /var/www/example.com/wordpress/wp-content/webp-express/webp-images/doc-root/**.webp w,

  }
}

Some advice

Creating an AppAmor profile for more complicated applications like this can be challenge. You will need to retest different things a lot and be prepared to do some hand editing of the profile. Some useful advice that I learnt along the way:

  • Often check /var/log/audit.log yourself to see what is happening. At a certain moment I ended up in a loop where my audit log was constantly being filled because the process tried to switch to a not yet existing profile, so you want to intervene immediately if this happens to prevent the logs from filling up your disk.
  • After making modifications to an AppArmor profile, it’s not a bad idea to restart your process. I am not sure, but I think some problems I encountered were fixed by restarting the process.
  • Stop aa-genprof/aa-logprof when you manually edit a profile and start hem again after the editing and reloading it with apparmor_parser, otherwise these processes won’t be aware of the modification you have made to the profile file.
  • When adding hats or child profiles, revise the permissions of your main profile: maybe some things can be moved to the new hat or child profile and can thus be removed completely from the main profile.
  • Keep your profile long enough in complain mode and only switch to enforce mode after enough time without events (I would say: a couple of days). If you encounter problems with your PHP applications later on, don’t forget to check your audit.log.

Protecting your Linux server against security exploits with AppArmor

AppArmor is a security feature available in the Linux kernel which adds Mandatory Access Control (MAC) to your system. Mandatory Access Control defines a security policy which the administrator of the system defines and cannot be overridden by the user. You use AppArmor to harden your system against all kinds of attacks, protecting your system against known or unknown zero-day exploits. It does so by restricting the applications on your system. You define an AppArmor profile for security sensitive applications which confines them to access only specific files, network access and other capabilities. This way it will become much harder or even impossible for an attacker to abuse a security hole in a service for which you have defined an AppArmor profile. SuSE sometimes calls this immunization.

AppArmor is similar to SELinux (Security-Enhanced Linux), the other well-known Linux security module which is used by default on Fedora and Red Hat Enterprise Linux. Debian and Ubuntu have opted for AppArmor by default. AppArmor profiles are a bit easier to manage than SELinux.

Setting up Apparmor on Debian

First we install all the AppArmor utilities and all default profiles:

# apt install apparmor apparmor-profiles apparmor-utils apparmor-profiles-extra libpam-apparmor auditd

With the aa-status command you can check the AppArmor status:

# aa-status
43 profiles are loaded.
20 profiles are in enforce mode.
   /usr/bin/freshclam
   /usr/bin/man
   /usr/bin/pidgin
   /usr/bin/pidgin//sanitized_helper
   /usr/bin/totem
   /usr/bin/totem-audio-preview
   /usr/bin/totem-video-thumbnailer
   /usr/bin/totem//sanitized_helper
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/clamd
   /{,usr/}sbin/dhclient
   apt-cacher-ng
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   tcpdump
23 profiles are in complain mode.
   /usr/bin/irssi
   /usr/sbin/sssd
   avahi-daemon
   dnsmasq
   dnsmasq//libvirt_leaseshelper
   identd
   klogd
   mdnsd
   nmbd
   nscd
   php-fpm
   ping
   samba-bgqd
   samba-dcerpcd
   samba-rpcd
   samba-rpcd-classic
   samba-rpcd-spoolss
   smbd
   smbldap-useradd
   smbldap-useradd///etc/init.d/nscd
   syslog-ng
   syslogd
   traceroute
0 profiles are in kill mode.
0 profiles are in unconfined mode.
17 processes have profiles defined.
0 processes are in enforce mode.
14 processes are in complain mode.
   /usr/sbin/php-fpm8.2 (834763) php-fpm
   /usr/sbin/php-fpm8.2 (834764) php-fpm
   /usr/sbin/php-fpm8.2 (834765) php-fpm
   /usr/sbin/php-fpm8.2 (834766) php-fpm
   /usr/sbin/php-fpm8.2 (834767) php-fpm
   /usr/sbin/php-fpm8.2 (834768) php-fpm
   /usr/sbin/php-fpm8.2 (834769) php-fpm
   /usr/sbin/php-fpm8.2 (834770) php-fpm
   /usr/sbin/php-fpm8.2 (834771) php-fpm
   /usr/sbin/php-fpm8.2 (834772) php-fpm
   /usr/sbin/php-fpm8.2 (834773) php-fpm
   /usr/sbin/php-fpm8.2 (834774) php-fpm
   /usr/sbin/php-fpm8.2 (834775) php-fpm
   /usr/sbin/php-fpm8.2 (834826) php-fpm
3 processes are unconfined but have a profile defined.
   /usr/bin/freshclam (797288)
   /usr/sbin/clamd (662585)
   /usr/sbin/sssd (647439)
0 processes are in mixed mode.
0 processes are in kill mode.

We see that AppArmor is enabled and that it has 32 profiles of which 16 are being enforced and another 16 are in complain mode, which means that a warning will be logged when they violate the policy, but they will not be blocked. Currently 1 running process is confined in enforce mode.

All the profiles are installed in /etc/apparmor.d. The text files, usually named after the full path to the binary with the slashes replaced by dots.

To completely disable an Apparmor profile you use the aa-disable command. For example:

# aa-disable /usr/sbin/haveged

To enable a profile in complain mode, use the aa-complain command:

# aa-complain /usr/sbin/haveged

Use the aa-enforce command to set enforce an Apparmor profile:

# aa-enforce /usr/sbin/haveged

If you make a modification to one of the profiles in /etc/apparmor.d, you will need to make the kernel reload the profile with the apparmor_parser command:

# apparmor_parser -r /etc/apparmor.d/usr.sbin.haveged

If you want to remove a profile currently loaded in the kernel, you can use this command:

# apparmor_parser -R /etc/apparmor.d/usr.sbin.haveged

When you have done this, you can remove the file if you don’t need it any more.

Creating Apparmor profiles with aa-genprof

I recommend creating an AppArmor profile for every service which is accessible via the network end/or deals with data from the outside world.

You can run the aa-unconfined command to find applications which listen on a TCP or UDP port and do not have an AppArmor profile loaded. These are excellent candidates to create a profile for.

Example 1: Creating an AppArmor profile for Knot Resolver

One of the unconfined processes indicated by aa-unconfined, is /usr/sbin/kresd which is Knot Resolver, the caching DNS resolver. I will generate a profile with aa-genprof. I need two shells for that: one where I run aa-genprof, and another one where I exercise the functionality of Knot Resolver. aa-genprof creates an empty profile and puts it in complain mode, so that all events generated by the process will be logged. aa-genprof will then propose rules for any of them.

So in the first shell I run:

# aa-genprof /usr/sbin/kresd
Updating AppArmor profiles in /etc/apparmor.d.

Before you begin, you may wish to check if a
profile already exists for the application you
wish to confine. See the following wiki page for
more information:
https://gitlab.com/apparmor/apparmor/wikis/Profiles

Profiling: /usr/sbin/kresd

Please start the application to be profiled in
another window and exercise its functionality now.

Once completed, select the "Scan" option below in
order to scan the system logs for AppArmor events.

For each AppArmor event, you will be given the
opportunity to choose whether the access should be
allowed or denied.

[(S)can system log for AppArmor events] / (F)inish

Now in the second shell we need to exercise the functionality of kresd. Let’s start by stopping, starting and restarting kresd:

# systemctl stop system-kresd.slice && systemctl start kresd.target && systemctl restart system-kresd.slice

I also recommend reloading the service, but the kresd service does not support this, so this is of no use here. Now we exercise kresd by doing some DNS lookups, for example with the host and the kdig commands:

$ host example.com
$ kdig -t mx debian.org

Wait for some time, and then you go back to the shell where aa-genprof is running and press S to scan the audit log for any generated events.

Profile:    /usr/sbin/kresd
Capability: net_bind_service
Severity:   8

 [1 - include <abstractions/nis>]
  2 - capability net_bind_service,
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish

Here kresd requests the net_bind_service capability. This is needed to listen to a privileged port (port number < 1024). aa-genprof proposes 2 possible rules. The first option is to include abstractions/nis which contains capability net_bind_service. Files in /etc/apparmor.d/abstractions contain common rules which may be included in multiple different profiles. Often these abstractions are too broad and in case of doubt I recommend to choose the more specific option, which is option 2 here. I press 2 select this option, then press A to allow this.

Then Knot Resolver checks whether we have transparent hugepages enabled by reading /sys/kernel/mm/transparent_hugepage/enabled. This is because Knot Resolve uses jemalloc which reads this file.

Profile:  /usr/sbin/kresd
Path:     /sys/kernel/mm/transparent_hugepage/enabled
New Mode: r
Severity: 4

 [1 - /sys/kernel/mm/transparent_hugepage/enabled r,]
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / Abo(r)t / (F)inish

I press A to allow this.

Knot Resolver uses lua for plug-ins, and so wants to read lua code:

Profile:  /usr/sbin/kresd
Path:     /usr/share/lua/5.1/cqueues/socket.lua
New Mode: r
Severity: unknown

 [1 - include <abstractions/totem>]
  2 - /usr/share/lua/5.1/cqueues/socket.lua r,
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / Abo(r)t / (F)inish

Now I want to allow reading of all lua files in /usr/share/lua, independent of the lua version, so that when Knot Resolver starts using a newer lua version in the future, the rule is still valid. Press 2 select that rule and then press E for glob with extension multiple times, until you get option 5 - /usr/share/lua/**.lua r,. This matches all files which name end with .lua anywhere within /usr/share/lua. Make sure this options is selected and then press A to allow this.

kresd creates a control file:

Profile:  /usr/sbin/kresd
Path:     /run/knot-resolver/control/1
New Mode: owner w
Severity: unknown

 [1 - owner /run/knot-resolver/control/1 w,]
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / (O)wner permissions off / Abo(r)t / (F)inish

The mode owner w means the permission to write the file only if the user as which the writing process is running is also the owner of the file.

If you are running multiple kresd processes, each process will create its own control file, so we want to use a wildcard here. Press G for glob so that you get the option 2 – owner /run/knot-resolver/control/* w, and press A to accept this.

kresd tries to take an exclusive lock on /var/cache/knot-resolver/lock.mdb:

Profile:  /usr/sbin/kresd
Path:     /var/cache/knot-resolver/lock.mdb
New Mode: owner rwk
Severity: unknown

 [1 - owner /var/cache/knot-resolver/lock.mdb rwk,]
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / (O)wner permissions off / Abo(r)t / (F)inish

Notice the mode rwk: k is the permission, and in combination with w this is an exclusive lock. You can find the explanation of all file permissions in the AppArmor Core Policy Reference. Press A to allow this.

I’m using rpz-downloader to download RPZ files containing malicious domains I want to block. Knot Resolver wants to read these RPZ files:

Profile:  /usr/sbin/kresd
Path:     /var/lib/rpz-downloader/urlhaus.abuse.ch.rpz
New Mode: r
Severity: unknown

 [1 - /var/lib/rpz-downloader/urlhaus.abuse.ch.rpz r,]
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / Abo(r)t / (F)inish

I want to allow access to all RPZ files in /var/lib/rpz-downloader, so I press N and I change the path to /var/lib/rpz-downloader/*.rpz and press A to accept this.

Knot Resolver tries to open an IPv4 TCP socket:

Profile:        /usr/sbin/kresd
Network Family: inet
Socket Type:    stream

 [1 - include <abstractions/apache2-common>]
  2 - include <abstractions/nameservice>
  3 - network inet stream,
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish

The proposed abstractions are too broad, so I select 3. You will also need to allow network inet dgram (IPv4 UDP), network inet6 stream (IPv6 TCP) and network inet6 dgram (IPv6 UDP).

When you have processed all events, you will get this:

= Changed Local Profiles =

The following local profiles were changed. Would you like to save them?

 [1 - /usr/sbin/kresd]
(S)ave Changes / Save Selec(t)ed Profile / [(V)iew Changes] / View Changes b/w (C)lean profiles / Abo(r)t

Only one profile is modified and you can view the changes by pressing V. You can save the changes by pressing S. After doing so, aa-genprof will continue running and you can press S to scan the logs again for any newly generated events. This way you can gradually improve your profile. If you are finished, you can quit aa-genprof by pressing F. You can run aa-genprof for your process again at any time, it will then modify the existing profile if it finds any new events.

When you have finished; check with aa-status whether your profile is still in complain mode, and if not change it back to complain mode with aa-complain.

This is how my final profile /etc/apparmor.d/usr.sbin.kresd looks:

abi <abi/3.0>,

include <tunables/global>

/usr/sbin/kresd flags=(complain) {
  include <abstractions/base>

  capability net_bind_service,

  network inet dgram,
  network inet stream,
  network inet6 dgram,
  network inet6 stream,

  /etc/group r,
  /etc/knot-resolver/kresd.conf r,
  /etc/nsswitch.conf r,
  /etc/passwd r,
  /etc/ssl/certs/ca-certificates.crt r,
  /etc/ssl/openssl.cnf r,
  /sys/kernel/mm/transparent_hugepage/enabled r,
  /usr/sbin/kresd mr,
  /usr/share/dns/root.hints r,
  /usr/share/dns/root.key r,
  /usr/share/lua/**.lua r,
  /var/lib/rpz-downloader/*.rpz r,
  owner /run/knot-resolver/control/ w,
  owner /run/knot-resolver/control/* w,
  owner /var/cache/knot-resolver/data.mdb rw,
  owner /var/cache/knot-resolver/lock.mdb rwk,

}

The flags=(complain) indicate that this profile will be loaded in complain mode. Keep it like that for some time until you are sure no new events are generated any more. You can check this by running aa-logprof. aa-logprof is similar to aa-genprof, except that it also processes past events logged in /var/log/audit.log and processes events for all processes for which you have defined an AppArmmor profile, and not a single one like aa-genprof. If you are sure the profile is complete, enforce it by running aa-enforce /usr/sbin/kresd.

Example 2: creating an AppArmor profile for Postfix

Another process marked by aa-unconfined on my system is /usr/lib/postfix/sbin/master, Postfix’ master process. I create a profile with aa-genprof:

# aa-genprof /usr/lib/postfix/sbin/master

In another shell, I stop, start, restart and reload Postfix:

# systemctl stop postfix@- && systemctl start postfix@- && systemctl restart postfix@- && systemctl reload postfix@-

To further exercise Postfix, send some mails, both outgoing mails to other mail servers as incoming mails originating from an external mail server. Check the mail queue with

# mailq

I’m not going through all events generated by Postfix, but I want to point out one particular type of event which we did not see with Knot Resolver:

Profile:  /usr/lib/postfix/sbin/master
Execute:  /usr/lib/postfix/sbin/showq
Severity: unknown

(I)nherit / (C)hild / (P)rofile / (N)amed / (U)nconfined / (X) ix On / (D)eny / Abo(r)t / (F)inish

The master process tries to start another executable, in this case showq. You have several options:

  • (I)nherit: the new process inherits the profile from the parent process. This means that it will run with the same permissions as the parent process.
  • (C)hild: the new process will use a subprofile defined within this profile. This is useful if you want to run the process with a different profile, depending on how it was started.
  • (P)rofile: the new process will run in its own generic profile, in this case /etc/apparmor.d/usr.lib.postfix.sbin.showq . Use this if you want to run it in the same profile as when you would have started /usr/lib/postfix/sbin/showq directly.
  • (N)amed: the new process will run with the profile with a name of your choice.
  • (U)confined: the new process will run unconfined, without any AppArmor restrictions.

Here I choose P because I want showq to run always with the same profile, no matter how it is started. Not only will aa-genprof create a rule which allows master to launch showq, it will also create a profile for showq, and start logging future events.

Then you will get the question whether you want to sanitize the environment:

Should AppArmor sanitise the environment when
switching profiles?

Sanitising environment is more secure,
but some applications depend on the presence
of LD_PRELOAD or LD_LIBRARY_PATH.

[(Y)es] / (N)o

My Postfix installation does not need any enviroment variables like LD_LIBRARY_PATH to be set, so it can safely sanitize the environment. I press Y.

When you have finished, make sure that all profiles are running in complain mode, otherwise you risk problems with mail delivery:

# cd /etc/apparmor.d
# for i in usr.lib.postfix.sbin.*; do aa-complain $i; done

My /etc/apparmor.d/usr.lib.postfix.sbin.master profile looks like this at the moment after some manual tweaking:

abi <abi/3.0>,

include <tunables/global>

/usr/lib/postfix/sbin/master flags=(complain) {
  include <abstractions/base>
  include <abstractions/postfix-common>

  capability dac_read_search,
  capability kill,
  capability net_bind_service,

  network inet stream,
  network inet6 dgram,
  network inet6 stream,
  network netlink raw,

  signal send peer=/usr/lib/postfix/sbin/*,

  /usr/lib/postfix/sbin/anvil Px,
  /usr/lib/postfix/sbin/cleanup Px,
  /usr/lib/postfix/sbin/lmtp Px,
  /usr/lib/postfix/sbin/local Px,
  /usr/lib/postfix/sbin/master mr,
  /usr/lib/postfix/sbin/pickup Px,
  /usr/lib/postfix/sbin/postscreen Px,
  /usr/lib/postfix/sbin/proxymap Px,
  /usr/lib/postfix/sbin/qmgr Px,
  /usr/lib/postfix/sbin/scache Px,
  /usr/lib/postfix/sbin/showq Px,
  /usr/lib/postfix/sbin/smtp Px,
  /usr/lib/postfix/sbin/smtpd Px,
  /usr/lib/postfix/sbin/tlsmgr Px,
  /usr/lib/postfix/sbin/trivial-rewrite Px,
  owner /etc/gai.conf r,
  owner /etc/group r,
  owner /etc/nsswitch.conf r,
  owner /etc/passwd r,
  owner /var/lib/postfix/master.lock rwk,
  owner /var/spool/postfix/pid/master.pid rwk,
  owner /var/spool/postfix/private/anvil w,
  owner /var/spool/postfix/private/bounce w,
  owner /var/spool/postfix/private/bsmtp w,
  owner /var/spool/postfix/private/defer w,
  owner /var/spool/postfix/private/discard w,
  owner /var/spool/postfix/private/dnsblog w,
  owner /var/spool/postfix/private/error w,
  owner /var/spool/postfix/private/ifmail w,
  owner /var/spool/postfix/private/lmtp w,
  owner /var/spool/postfix/private/local w,
  owner /var/spool/postfix/private/maildrop w,
  owner /var/spool/postfix/private/mailman w,
  owner /var/spool/postfix/private/proxymap w,
  owner /var/spool/postfix/private/proxywrite w,
  owner /var/spool/postfix/private/relay w,
  owner /var/spool/postfix/private/retry w,
  owner /var/spool/postfix/private/rewrite w,
  owner /var/spool/postfix/private/scache w,
  owner /var/spool/postfix/private/scalemail-backend w,
  owner /var/spool/postfix/private/smtp w,
  owner /var/spool/postfix/private/smtp-amavis w,
  owner /var/spool/postfix/private/smtpd w,
  owner /var/spool/postfix/private/tlsmgr w,
  owner /var/spool/postfix/private/tlsproxy w,
  owner /var/spool/postfix/private/trace w,
  owner /var/spool/postfix/private/uucp w,
  owner /var/spool/postfix/private/verify w,
  owner /var/spool/postfix/private/virtual w,
  owner /var/spool/postfix/public/cleanup w,
  owner /var/spool/postfix/public/flush w,
  owner /var/spool/postfix/public/pickup rw,
  owner /var/spool/postfix/public/qmgr rw,
  owner /var/spool/postfix/public/showq w,

}

I’m not going to add all profiles for other Postfix processes here, but you get the idea. More rules might still be needed, so that’s why I keep them running in complain mode and I regularly run aa-logprof.

Debugging problems caused by AppArmor

AppAmor logs events in /var/log/audit/audit.log if you have auditd running. You can grep for apparmor to see all events.

To easily process all new events logged in /var/log/audit/audit.log you can use aa-logprof:

# aa-logprof

Sources and more information

Inter and IBM Plex fonts for your Linux desktop

Recently I came around a post on the Fediverse mentioning the Inter fonts. There is even a GNOME issue open discussing making the Inter fonts the default in a future version of the GNOME desktop. This prompted me to try this font, and I have to say I am liking it so far.

The Inter font, does not have a monospace version available, but the Inter developer recommended some nice monospace fonts which match Inter. I decided to go for the IBM Plex Mono font.

Both Inter and IBM Plex are packaged in Debian, so you can easily install them with apt:

# apt install fonts-inter fonts-ibm-plex

To change the fonts in the GNOME desktop, you need to launch gnome-tweaks (install the package with apt if it’s not present on your system, and go to Fonts. I set Interface Text and Document Text fonts to Inter Light 10, the Monospace Text font to IBM Plex Mono Regular 10 and the Legacy Windows Titles fonts to Inter Bold.

Then in Firefox in the menu click on Settings and if you scroll down you will find the Fonts section with a button Advanced… next to it. Click on that button and set Proportional font to Sans Serif, the Serif font to IBM Plex Serif, the Sans Serif font to Inter and the Monospace font to IBM Plex Mono.

Of course you can use these fonts in other desktops and browsers.

Enjoy your fresh desktop fonts!

Setting up Foomuuri, an nftables based firewall

Up to now I have always been using the Shorewall firewall on all my Linux systems. I find it very easy to configure while at the same time it’s very powerful and flexible so that you can also use it with more complicated set-ups, such as routers with multiple network interfaces, VPN’s and bridges. Unfortunately Shorewall is still based on the old xtables (iptables, ip6tables, ebtables, etc…) infrastructure. While it still works and in reality the iptables commands are actually now front-ends to the more modern nftables back-end, Shorewall development has stalled and it looks very unlikely it will ever be ported to nftables.

I started using Firewalld, a firewall which is used by default on Red Hat and Fedora based systems. However I did not like it. Configuration of Firewalld happens through the command line with firewall-cmd, which I find much more complicated than just editing a configuration file which usually contains examples and gives you an easy overview of the configuration. Firewalld saves its configuration in XML files. You could edit these files instead of using firewall-cmd, but that is obviously much more complicated than editing configuration files which were designed for human editing. Furthermore I found Firewalld to be very inflexible. Firewalld does not have support of filtering traffic on a bridge (layer 2 filtering), unlike Shorewall.

Recently I discovered the nftables based firewall foomuuri. It’s still a very young project but it’s actively developed, already has extensive features, is packaged in Debian and is configured through human-readable configuration files. I decided to try it on a server where I wanted to filter incoming and outgoing network traffic.

Installing Foomuuri on Debian

Foomuuri is availabe in Debian testing and unstable, but it has also been backported to Debian 12 Bookworm. To use that package, you have to enable the bookworm-backports repository first. Then install the foomuuri package

# apt install foomuuri

If you are using NetworkManager also install foomuuri-firewalld, because it will allow NetworkManager to set the zone the network interface belongs to.

Configuring Foomuuri

Foomuuri can be configured through files in the /etc/foomuuri directory. Foomuuri will read all files which name ends with .conf, so you can split up the configuration in as many files as you want or just put everything in a single file, as you prefer. I like the split configuration files of Shorewall, so I will do something similar here.

Before activating the configuration, always run

# foomuuri check

to validate your configuration. You can start and stop the firewall by starting and stopping the systemd service, you can reload the configuration by running

 # foomuuri reload

You can find the documentation of Foomuuri on the Foomuuri wiki.

Defining zones

The first ting we have to do is define the zones and set which interfaces belongs to which zone. I create /etc/foomuuri/zones.conf:

zone {
  localhost
  public enp1s0
}

I create the zone localhost and the zone public and add the network interface enp1s0 to it. You can add multiple interfaces to a zone by separating them by spaces. If you are using NetworkManager, you don’t have to add the interfaces here and can leave the zone empty. You can configure the firewall zone in NetworkManager and it will set it through foomuuri-firewalld.

Using macros to alias configuration options

Macros can be used to define certain configuration options you want to use multiple times without having to write them completely every time. In practice a lot of macros are already configured which define the configuration for common services. You can see all defined macros by running

# foomuuri list macro

For example the macro imap defines the configuration tcp 143, so that you can just write imap instead of tcp 143 in the configuration. I added a few which were not defined by default in /etc/foomuuri/services.conf:

macro {
	nrpe	tcp 5666
	nmb	udp 137 138 139; tcp 139
}

Macros can be used to configure common subnets. For example I have a file named /etc/foomuuri/subnets.conf:

macro {
	mysubnet		192.168.0.1/24
	othersubnet		192.168.1.1/24
}

I also use macros to create lists of individual hosts, such as all NFS clients which need to access this NFS server in /etc/foomuuri/nfs_clients.conf

macro {
	nfs_clients   192.168.0.1 # web server
	nfs_clients + 192.168.0.2 # gitlab
	nfs_clients + 192.168.0.3 # nextcloud
}

For easy readability, I put every host in a single line, and I add a comment for my own reference. With the + sign I add all next hosts to the macro.

Firewall for incoming connections

To configure Foomuuri to filter incoming connections to my servers, I create a section public-localhost which contains the firewall rules for traffic coming from the public zone to localhost. I put this in the file /etc/foomuuri/public-localhost.conf:

public-localhost {
  dhcp-server
  ssh
  ping  saddr mysubnet
  nmb   saddr mysubnet
  smb   saddr mysubnet
  nfs   saddr nfs_clients
  nrpe  saddr 192.168.0.5
  drop log
}

My server is acting as a DCHP-server, so I use the dhcp-server macro to allow all this traffic, just as I allow all incoming ssh traffic. I allow ping, nmb and smb traffic from mysubnet. Notice that in these rules I use my custom macros nmb and mysubnet. Then I allow nfs from all addresses listed in my macro nfs_clients, and I allow nrpe from a specific IP address. Finally I end with a rule which drops and logs all traffic which has not matched any of the rules before.

Firewall for outgoing connections

I think that filtering outgoing connections is a very effective security hardening measure. In case people with bad intentions get access to your server through a non-root user account, this will severely limit their abilities to move laterally through your network and attack other systems, to run a crypto-miner, or download malware from the Internet.

localhost-public {
  dhcp-client
  nmb uid root
  ntp uid systemd-timesync
  ping uid root daddr mysubnet # dhcpd sometimes pings
  smtp daddr 192.168.0.1 uid postfix
  domain daddr 192.168.0.255 192.168.0.254
  uid root tcp daddr 192.168.0.5 dport 8140 # puppet agent
  uid _apt tcp dport 3142 daddr 192.168.0.6
  uid root ssh daddr 192.168.0.250 # backups
  drop daddr 169.254.169.254 tcp dport 80 # don't fill logs with Puppetlabs facter trying to collect facts from Amazon EC2/Azure
  reject log
}

I allow outgoing connections for different services, and for most services I set the user which can create that connection, and to which host I allow the connection. I explicitly drop without logging connections to 169.254.169.254 port 80, because facter tries to connect to this address every time it runs in order to get some metadata from your cloud service provider. If your system is running on Amazon or Microsoft Azure cloud services, you will probably want to allow this connection instead, so you can then just remove the drop word.

In order to log the UID of the process which tried to establish a rejected connection, in future Foomuuri versions (starting from Foomuuri version 0.22) you can replace the last rule by

reject log log_level "level warn flags skuid"

In current version 0.21, it is possible by setting this globally for all connections. I created /etc/foormuuri/loglevel.conf:

foomuuri {
  log_level "level info flags skuid"
}

Integrating Fail2ban with Foomuuri

I found inspiration for integrating Fail2ban with Foomuuri in issue 9 on the Foomuuri issue tracker.

Create /etc/fail2ban/action.d/foomuuri with these contents:

[Definition]
actionstart =
actionstop  =
actioncheck =
actionban   = /usr/sbin/foomuuri iplist add fail2ban 999d <ip>
actionunban = /usr/sbin/foomuuri iplist del fail2ban <ip>
actionflush = /usr/sbin/foomuuri iplist flush fail2ban

Then set foomuri as the default banaction by creating /etc/fail2ban/jail.d/foomuri.conf:

[DEFAULT]
banaction = foomuuri

Then foomuuri should create the fail2ban iplist. We can configure it to so by creating /etc/foomuuri/fail2ban.conf:

iplist {
	@fail2ban
}

Then I add this rule as first rule to the public-localhost section:

  saddr @fail2ban drop log fail2ban drop

This will drop all connections coming from an address in the iplist fail2ban, and will also log them with prefix fail2ban. If you don’t want this to be logged, just remove log fail2ban.

To ensure that Foomuuri is started before Fail2ban, so that the fail2ban iplist exists before Fail2ban starts to use it, create

/etc/systemd/system/fail2ban.service.d/override.conf:

[Unit]
After=foomuuri.service

After making these changes, first restart Foomuuri and then Fail2ban.

Conclusion

I found Foomuuri easy to use for a system with one network interface. Configuration through the configuration files is easy, also when implementing filtering for outgoing packets. Even though Foomuuri is still a young project, it already has many features and its author is very reactive to discussions and issues on Github. I also found the documentation on the wiki very helpful

I will try to implement Foomuuri on more complex setups in the future, such as on a host for virtual machines of which the network interface is bridged to the main network interface of the host, VPN servers, routers, etc…

Finally I want to thank the Foomuri developer Kim B. Heino and the maintainer of the Debian package Romain Francoise for their work and making this available to the community.

Noteworthy Debian Trixie/Sid changes July 8 2023

Here is a late quick overview of important changes in Debian Sid during the last two weeks.

glibc was updated from version 2.36 to version 2.37. This version mostly contains bug fixes and mintor improvements. An important regression by this update was fixed in the Debian package 2.37-4, which is not yet in testing/trixie, so you might want to update immediately to the sid version of glibc if you are on testing.

Linux 6.3.11 fixes the so-called StackRot security vulnerability (CVE-2023-3269). An exploit wil be made public soon and will allow any local user to get root access rights, so make sure you are running this kernel on all your systems. This kernel also re-enables CONFIG_VIRTIO_MEM which got disabled by mistake in Debian 12 Bookworm. Both the StackRot security vulnerability as the CONFIG_VIRTIO_MEM regression are now also fixed in the latest kernel release in bookworm-security.

Server software

PowerDNS Authorative Server 4.8.0 adds Lightning Stream support, while PowerDNS Recursor 4.9.0 has some performance improvements, amongst others.

Cyrus IMAP 3.8.0 adds support for some IMAP RFCs and implements new JMAP features.

NGinx 1.24.0 enables TLSv1.3 by default.

Debian’s Slurm package was updated to version 23.02.3.

Desktop software

Firefox 115 is now available in sid. Keep in mind that the firefox package never moves to testing and stable, only the firefox-esr package does. Firefox 115 will become the future Firefox ESR release though. Most important change in Firefox 115 is that it enables hardware video decoding for Intel GPUs with VA-API on Linux by default.

Remmina in sid was updated from 1.4.29 to 1.4.31. This version brings back the remmina-plugin-spice package which was also disabled in Debian 12.

Digikam 8.0.0 has improved file format support, new OCR tool and other improvements.

The Kdenlive video editor version 23.04 adds nested timelines, new effects and transitions, improvements to subtile handling and integrates the Whisper speech recognition system.

Shotwell 0.32 also brings improved file format support and various other improvements.

Phosh, the GNOME Phone Shell was updated to version 0.29. Improvements include call notification on the lock screen and audio device selection to the settings.

Noteworthy Debian Trixie/Sid changes June 24 2023

As expected, the second week of Trixie development was a lot more quiet than the first week.

KDE Frameworks was updated from version 5.103 to 5.107. While this has little visible changes, these are libraries which lay the foundations for improvements and bug fixes in KDE applications.

KDE Gear apps Neochat (a Matrix client), Elisa (a music player), Dragon Player (a movie player), Filelight (a disk space visualizer) and Spectacle (a screenshot application) were updated to the 23.04 release. Spectacle has gotten a complete redesign of the user interface and supports screen recording on Wayland.

The QT based display manager sddm version 0.20 now has experimental Wayland support and has enabled HiDPI scaling by default.

The AV1 encoder and decoder svt-av1 was updated to version 1.6.0. This brings once again performance and quality improvements.

Other upgraded packages in sid include Homebank 5.6.5, Deluge 2.1, GNOME Music 44 and many others.

Noteworthy Debian Trixie/Sid changes week 1 (June 11 – June 17 2023)

A long time ago, I used to regularly post an overview of noteworthy changes in the Mandriva development version. For years now I am using Debian testing though. With the release of Debian 12 Bookworm, I though it could be interesting to keep track of noteworthy changes in the upcoming Debian version, Trixie.

I will be tracking sid trough the debian-devel-changes maling list. Usually about 10 days after a package entered sid, it should move to testing, at least if there are not important bugs in the package. The selection of which packages I mention here, is very personal. I will try to cover important changes for both desktop and server packages, but this list will never be complete. If you noticed an interesting change not mentioned, feel free to add a comment to this article.

I’m not sure whether I will make this kind of post regularly without interruption, but let’s see where this goes.

The first week of development, saw a huge amount of packages updated to the latest upstream versions. Some of these were already available in Experimental for some time. Let’s dive in.

Kernel, hardware support, low-level libraries

The Linux kernel was updated to the 6.3 series, coming from 6.1 in Bookworm. I refer to kernelnewbies.org for a complete overview of what’s new in Linux 6.2 and Linux 6.3, but I can mention BTRFS performance improvements in both versions (including discard=async being default on SSDs with TRIM support, performance improvements if you are using an Intel Skylake CPU and add retbleed=stuff to the kernel options and the usual driver improvements which improve hardware support, for example for the current Intel Arc GPUs. If you have an AMD processor with at least the Zen2 microarchitecture, you can enable the new amd_pstate_epp frequency scaling driver by adding the kernel option amd_pstate=active.

btrfs-progs has been updated to version 6.3.1. The major change in the 6.3 series is that block-group-tree is out of experimental mode. This will reduce the mount time of BTRFS file systems. You can enable this on an existing file system with the command

btrfstune --convert-to-block-group-tree <device>

Developers warn to be careful, because there might still be bugs.

The Mesa 3D drivers were upgraded from version 22.3.6 to 23.1.2. You will want to upgrade to this version if you are using an Intel Arc GPU because there have been many bug fixes. Also new in Mesa 23.1 is OpenCL support for AMD GPUs using rusticl.

power-profies-daemon version 0.13 entered Debian Sid. It adds support for the amd_pstate_epp driver which can be activated in Linux 6.3.

Tthe LLVM based Fortran compiler Flang is now available in Debian as the package flang-15. LLVM 16 is available in sid. Clang 16 and libc++ 16 are only available in experimental at this time and version 15 is still the default version in sid.

Server and virtualization

Samba 4.18.3 brings some performance improvements relative to version 4.17.8 in Bookworm.

Qemu was updated to 8.0.2. The 8 series brings various improvements, but maybe the most important thing to mention is that virtiofsd, a daemon which allows you to share directories on the host with guests, is not included in the qemu package any more. If you use this, you will need to install the new virtiofsd package which contains a new implementation in Rust.

Desktop

LibreOffice is now at version 7.5.4. The 7.5 series bings improved dark mode support, new application icons, nicer default table styles in Draw and Impress and other various improvements. See the release notes and the New Features in Libreoffice 7.5 video for more information.

This week we saw the first GNOME 44 packages enter sid. gnome-backgrounds 44 brings you new desktop wallpapers and evolution 3.48 brings lay-out improvements. If you don’t like the headerbar layout, you can disable it and switch back to the traditional toolbar.

New in Debian is the gdm-settings package. It lets you configure the GDM login manager and change its appearance through a user friendly user interface.

NetworkManager-openconnect 1.2.10 finally adds support for Single Sign-on implementations using SAML on the Cisco AnnyConnect and Palo Alto GlobalProtect VPNs. Unfortunately it does not seem to work for Pulse/Juniper. I have opened an issue for that, and in the meantime I use openconnect-pulse-gui.

The Transmission Bittorrent client 4 has moved from C to C++, uses less CPU and memory and has support for the Bittorrent v2 protocol, amongst other improvements.

The first KDE Gear 23.04 applications are now being uploaded to sid. Now in the repositories is the KDE Mastodon client Tokodon. Gwenview (supports pinch gestures to zoom in Wayland mode), Ghostwriter (automatic language detection for the spellchecker) and the Falkon web browser (dark colour scheme support) were updated to 23.04.

Tellico, the application which helps you to keep track of your music, movie and other collections has support for new data sources and improved support for existing data sources and has reports with an image grid.

Thhe dav1d AV1 video decoder was update d from 1.0 to version 1.2.1. This version brings many performance improvements thanks to new SIMD code. Also the svt-av1 encoder and decoder was updated to version 1.5.0, also adding some optimizations.

There are plans to remove the old SDL 1.2 packages, replacing them by SDL12-compat (Debian package libsdl1.2-compat-shim) which implements SDL 1.2 using the SDL 2. Version 1.64, which entered sid, added compatibility for some old games, such as Sid Meier’s Alpha Centauri and others.

Science, education and technical tools

GNU Octave 8.2 brings improved Matlab compatibility, the GUI has dark mode support and various other improvements.

The R statistical computing programming (package r-base) 4.3.1 (was 4.2.2 in Bookworm) brings many new features.

Labplot 2.10 comes with many new features, improvements and performance optimizations in different areas, as well as with support for new data formats and visualization types.

People who are into geographic information systems will by happy with the QGIS update. Version 3.28 introduces many improvements. Look at the changelog for more details.

The Electronics Design Automation Suite kicad has been updated to version 7. I don’t know anything of this kind of software, but the release announcement lists a large number major of improvements.

Others improvements

There are too many changes to list here in detail. I mention updated Pipewire 0.3.71, Wireplumer 0.4.14, GStreamer 1.22.3, Opus 1.4, Gajim 1.8.0, OpenJDK 20, Phosh 0.28.0 and much more.

Conclusion

The first week of Trixie development saw a huge amount of software enter Debian sid. This is of course due to the backlog of all new upstream versions which could not be submitted during the Bookworm freeze, are now all trickling in now. Some of these packages were already in Experimental.

My personal favourites are Linux 6.3 which now allows me to use the amd_state_epp driver and Evolution 3.48, which has an some improvements to the UI which I like a lot.

Now that the first flood of new packages have arrived, things will probably calm down a bit, also because of the upcoming summer and holidays in the northern hemisphere. But I guess we will see more of GNOME 44 and KDE Gear 23.04 entering sid soon.

Upgrading from Debian 11 Bullseye to Debian 12 Bookworm

Debian 12 Bookworm will be released very soon, on June 10 2023. The Debian Testing tree is now very close to the final release, so now is a good moment to start testing Bookworm if you did not do so. I already upgraded some of my server systems to Bookworm and I’m also running on all my desktop systems, so here are some notes of the upgrade process. Keep in mind that upgrading to Bookworm is only supported if you are running Bullseye. If you are running an older version of Debian (Buster), you will need to upgrade to Bullseye first and after that upgrade to Bookworm.

First of all, start with reading the release notes, it contains a very detailed howto guide describing all steps to upgrade your system to Bookworm. It also lists all major changes and important things to know before you upgrade.

First check which packages you have installed which do not come from the official Debian repositories with this command:

# apt list '?narrow(?installed, ?not(?origin(Debian)))'

Because these are not official Debian packages, Debian developers cannot guarantee that they will work correctly and will not conflict or cause compatibility problems when upgrading your system. For that reason, you should seriously consider uninstalling them during the upgrade process.

On one system I had a locally built snuffleupagus package installed. This package was built against a particular PHP version and because a newer Debian release will also include a newer PHP version, this could break things. So i removed this package:

# apt remove snuffleupagus

Then you need to verify whether you have put any packages on hold. Packages on hold will never be upgraded, so this can prevent a correct upgrade. Check all held packages with this command:

# apt-mark showhold

You can unhold them with this command:

# apt-mark unhold packagename

Then we need to adapt our apt sources.list and preferences.

You should have this in /etc/apt/sources.list (or in a .list file /etc/apt/sources.list.d):

deb [arch=amd64,i386] http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb-src http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware

deb [arch=amd64,i386] http://deb.debian.org/debian/ bookworm-proposed-updates main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm-proposed-updates main contrib non-free non-free-firmware

Note the new non-free-firmware repository: non-free firmware used to be included in the non-free repository, but now they are in a new separate repository, so you will need to add that.

Then we need to set up the priorities of the different repositories in /etc/apt/preferences (or a ;pref file in /etc/apt/preferences.d):

Package: *
Pin: release n=bookworm
Pin-Priority: 810

Package: *
Pin: release n=bookworm-security
Pin-Priority: 810

Package: *
Pin: release n=bookworm-proposed-updates
Pin-Priority: 809

Package: *
Pin: release n=bookworm-backports
Pin-Priority: 808


Package: *
Pin: release n=bullseye
Pin-Priority: 710

Package: *
Pin: release n=bullseye-security
Pin-Priority: 710

Package: *
Pin: release n=bullseye-proposed-updates
Pin-Priority: 709

Package: *
Pin: release n=bullseye-backports
Pin-Priority: 708


Package: *
Pin: release n=trixie
Pin-Priority: 310


Package: *
Pin: release a=unstable
Pin-Priority: 200


Package: *
Pin: release a=experimental
Pin-Priority: 160

This gives the highest priority to all packages in Bookworm and the security updates, with a lower priority to the Bookworm proposed updates and then Bookworm backports. I added Bullseye in case you still need the Bullseye repositories for some reason. I also add Trixie (code name for what will become testing when Bullseye gets released) and sid (unstable) and experimental on the lowest priorities. Of course you can remove them from your preferences file if you don’t have set up these repositories.

I strongly recommend installing apt-listchanges, because it will give you information about important changes which might affect you before packages are upgraded:

# apt install apt-listchanges

I upgrade dpkg and apt first. I personally prefer to take advantages of eventual new improvements and bug fixes during the upgrade process.

# apt install -t bookworm apt dpkg

In Bookworm the systemd-resolved service now is in a seperate package. If you are currently using systemd-resolved, this can cause failures in DNS resolution. Before upgrading, make sure you know the addresses of your DNS servers, so that you can set them up manually if required; You can run the resolvectl command to find them. If DNS resolution breaks during the upgrade process later on, you can add them to /etc/resolv.conf manually to fix the problem. But I prefer to immediately install the new systemd-resolved package before upgrading everything else to take care of this problem:

# apt install -t bookworm systemd-resolved

Then we can upgrade all packages which can be upgraded without installing new packages:

# apt upgrade -t bookworm --without-new-pkgs

Once that’s done we proceed with the upgrade of all remaining packages, which will also install new dependencies:

# apt full-upgrade

During these two steps pay attention to which packages are going to be removed. It’s expected that old unused libraries and other packages (old PHP and Perl versions for example) will be removed, but you might to check this.

When the upgrade is done, I remove all unneeded packages with this command:

# apt autoremove --purge

Then run this command to remove all library packages which have no other packages depending on them any more:

# deborphan | xargs dpkg --purge

A package which often stays behind is libssl1.1. Normally you don’t need it any more so you can remove it safely:

# apt remove libssl1.1

Finally I also prefer to remove rsyslog. It is not installed any more by default on Debian Bookworm and everything is already logged to the systemd journal and I don’t want any double logging.

# apt remove rsyslog

Then personally I also install dbus-broker on Debian Bookworm. It replaces the traditional dbus implementation and is supposed to be more performant.

# apt install dbus-broker

I always recommend to verify that the metapackage linux-image-amd64 is installed, so that you are really running the latest kernel version.

# apt install linux-image-amd64

After upgrading all packages, reboot your system.

Debian 12 Bookworm and OpenLDAP

One major change in Debian 12 Bookworm is that it ships with OpenLDAP 2.5 which has removed the BDB and HDB back-ends. If your LDAP directory is still using this backend, you will have to convert it to the new MDB backend. There are some instructions in /usr/share/doc/slapd/README.Debian.gz and I might write some post here in the future about this. In any case, make sure you have recent backups of your LDAP directory in the form of LDIF gerenated with slapcat.

Is Debian Bookworm stable?

I’m permanently running Debian Testing on my laptop, and now I have also installed Bookworm on some servers. I strongly recommend using Testing for all desktop usage (even after Bookworm has been released), and to start using Bookworm on any new server installations. For upgrades of critical and more complicated server systems, I generally recommend to wait until at least the first point release.

At the moment I can think of two problems I am encountering on my systems. On my HP Elitebook 845 G8 when suspending the system (s2idle), Linux fails to read the current time from the RTC, resulting in the clock jumping years into the future. At the next resume you can fix the clock by restarting systemd-timesyncd, but the uptime command will continue to give a wrong output. I’m currently testing version 6.3 of the Linux kernel from experimental to see whether this bug also happens with this version.

Another problem is that in Debian 12 Bookworm the Spice plugin of Remmina is disabled. There is a work-around using packages from sid and experimental: upgrade to libspice-client-glib-2.0-8 version 0.42-2, which is currently in sid and then install remmina and remmina-plugin-spice from experimental. Maybe this problem will be fixed in a point release for Debian Bookworm, but that remains to be seen.