Tag: systemd

Some various performance improvements for Debian 12 Bookworm

Posted on by Frederik

Here some various improvements I implemented on some of my Debian 12 Bookworm servers in order to improve performance.

zswap: use zsmalloc allocator with newer kernel

If your system has little memory, you might be using zswap already. When memory is getting full, the system will try to swap out less used data from memory to a compressed swap in memory instead of writing it immediately to a swap partition or swap file on slower storage. In Linux kernel version 6.7 the zsmalloc allocator, which is superior to other allocators (zbud and z3fold), became the default.

So first upgrade to a more recent kernel. You can get a recent kernel from bookworm-backports or Debian testing or unstable.

To enable zswap at boot you can create a file named /etc/default/grub.d/zswap.cfg which contains:

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=30"

If you want higher compression at the cost of more CPU time, you can replace lz4 by zstd.

You will need to add the compression module of your preference to your initramfs. So in the case of lz4, just add

lz4

to /etc/initramfs-tools/modules and then run

# update-initramfs -u

Finally execute

# update-grub

to update your Grub configuration so that these settings will become automatically active at the next boot.

I upgraded on to Linux 6.11 on a VPS with 2GB of RAM with these settings, and the system feels much snappier now.

To check effectiveness of zswap, you can use the zswap-stats script.

Update to systemd 254 or higher to improve behaviour under memory pressure

systemd 254 includes a change which makes journald and resolved flush their caches when the system is under memory pressure. This will free memory, reducing swapping. When this happens, this can be found in the logs:

systemd-journald[587721]: Under memory pressure, flushing caches.

You can find systemd 254 for Debian 12 in bookworm-backports.

Exclude cron from audit logging

On one of my systems, my audit logs where rapidly filling. You can check whether this is happening for you by looking at the dates of the files in /var/log/audit/:

# ls -l /var/log/audit/

By default auditd will write files up to 8 MB, after which it will rotate the file. If these different files have their modification date very close to each others, then you might consider reducing the logging.

Possible causes for audit logs filling up are Apparmor logs. Improve your Apparmor profiles to reduce warnings and errors. On one of my system, the cause was the logging caused by the execution of cron jobs. Especially because mailman3-web contains a cron job which is executed every single minute.

To prevent logging everything related to cron, create a file /etc/audit/rules.d/excludecron.rules:

-a exclude,always -F exe=/usr/sbin/cron

Then run

# augenrules --load

to load the new rules.

Protecting systemd services with AppArmor

Posted on by Frederik

In a previous blog post I explained how to create an AppArmor profile for a specific binary. However it is also possible to make a profile which is not linked to a specific executable, but you can load the profile in any systemd service. This is for example useful for services which share the same binary and only differ in the arguments you give on the command line. This can be useful for applications you start with uvicorn, or interpreted languages like Lisp or Rscript.

I assume you already have a systemd service file called myservice.service. In order to protect this service, create /etc/apparmor.d/myservice:

abi <abi/3.0>,
include <tunables/global>

profile myservice flags=(complain) {
  include <abstractions/base>
}

You see that I don’t use the path for a binary, but chose an arbitrary name.

Load the profile:

# apparmor_parser <em>/etc/apparmor.d/myservice</em>

Modify /etc/systemd/system/myservice.service or run systemctl edit myservice and in the [Service] section add:

AppArmorProfile=myservice

Reload the service files:

# systemctl daemon-reload

Start your service:

# systemctl start myservice

Exercise your service and process all events with aa-logprof:

# aa-logprof

Don’t forget to put it in enforce mode with aa-enforce if you are sure the profile is complete.

Wireguard VPN with systemd-networkd and Foomuri

Posted on by Frederik

After my first successful implementation of Foomuuri on a server with an IPv4 connection, I wanted to try Foomuuri in a different environment. This time I choose to implement it on my IPv4/IPv6 dual stack Wireguard VPN server. I originally set up this system with Shorewall, so let’s see how we should configure this with Foomuuri.

While at it, I also moved the configuration of Wireguard to systemd-networkd, where the main network interface was already configured. This was also useful because some things which were configured in Shorewall before and which Foomuuri does not do by itself, can now be configured in systemd-networkd.

systemd-networkd configuration

I create /etc/systemd/network/wg0.netdev with these contents:

[NetDev]
Name = wg0
Kind = wireguard
Description = wg0 - Wireguard VPN server

[WireGuard]
PrivateKeyFile = /etc/systemd/network/wg0.privkey
ListenPort = 51820

# client 1
[WireGuardPeer]
PublicKey = publickey_of_client
AllowedIPs = 192.168.7.2/32
AllowedIPs = aaaa:bbbb:cccc:dddd:ffff::2/128

I moved the /etc/wireguard/privatekey file to /etc/systemd/network/wg0.privkey, and then give it appropriate permissions so that user systemd-network can read it:

# chown root:systemd-network /etc/systemd/network/wg0.privkey
# chmod 640 /etc/systemd/network/wg0.privkey

Then I create /etc/systemd/network/wg0.network:

[Match]
Name = wg0

[Network]
Address = 192.168.7.1/24
Address = fd42:42:42::1/64

[Route]
Destination = aaaa:bbbb:cccc:dddd:ffff::2/128

For IPv4, we set the address to 192.168.7.1/24 and systemd-networkd will automatically take care of adding this subnet to the routing table. As we are using public IPv6 addresses for the VPN clients, I add a [ROUTE] section which takes care of adding these IP address to the routing table.

The configuration of the public network interface is stored in /etc/systemd/network/public.network:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=2a0f:fc80::
DNS=2a0f:fc81::
DNS=193.110.81.0
DNS=185.253.5.0
Address=www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
IPForward=yes
IPv6ProxyNDP=1
IPv6ProxyNDPAddress=aaaa:bbbb:cccc:dddd:ffff::2

Important here is that we enable IP forwarding and IPv6 NDP proxy here. Both were things we could configure in Shorewall before, but Foomuuri does not support setting these. This is not a problem, because this can be set up directly in systemd-networkd.

To reload the configuration for all network interface, I run:

networkctl reload

To bring up the Wireguard connection:

networkctl up wg0

Because of systemd issue #25547, networkctl reload is not enough if you make changes to the peer configuration in wg0.netdev. You will first have to delete the network device with the command

networkctl delete wg0

after which you can run networkctl reload and bring up the network connection. In case of doubt all network interfaces are configured correctly, you can also completely restart the systemd-networkd service:

# systemctl restart systemd-networkd

While working on the network configuration, of course make sure you have access to a real console of the system, so that in case your system becomes inaccessible, you can still fix things through the console.

Foomuuri configuration

Now we define the zones in /etc/foomuuri/zones.conf:

zone {
  localhost
  public ens192
  vpn wg0
}

Foomuuri by default does not define a macro for the Wireguard UDP port, so I create one in /etc/foomuuri/services.conf:

macro {
	wireguard udp dport 51820
}

I adjust some logging settings in /etc/foomuuri/log.conf. In case I want to filter outgoing connections from the machine in the future, I want to log the UID of the process and I also increase the log rate, as I had the impression that I sometimes was missing valuable log messages while debugging. Adjust the values if you wan to reduce log spam.

foomuuri {
  log_rate "2/second burst 20"
  log_level "level info flags skuid"
}

I set up masquerading (SNAT) in /etc/foomuuri.conf/snat.conf :

snat {
  saddr 192.168.7.0/24 oifname ens192 masquerade
}

Then I set up these rules for traffic going through our firewall:

public-localhost {
  ssh
  wireguard
  icmpv6 1 2 3 4 128
  drop log
}

localhost-public {
  accept
}

vpn-public {
  accept
}

public-vpn {
  icmpv6 1 2 3 4 128
  drop log
}

vpn-localhost {
  accept
}

localhost-vpn {
  icmpv6 1 2 3 4 128
  reject log
}

Notice that I allow ICMPv6 traffic that should not be dropped.

As usually check your configuration before reloading it:

# foomuuri check
# foomuuri reload

Testing and debugging

If things don’t work as expected, enable debugging in the wireguard kernel module and check the kernel logs. I refer to the previous article about this for more details.

Conclusion

Setting up Foomuuri was pretty easy again. The most difficult thing was getting the systemd-networkd configuration completely right. Especially with IPv6 it can take quite some time debugging before everything works as expected.