Using the Solo V2 FIDO2 security key

Last year I supported the Solo V2 Kickstarter camaign. Solo is a completely open source FIDO2 security key. You can use it for Two-Factor Authentication (2FA) on web sites, for protecting your private SSH keys and other things. The Solo2 is similar to keys such as the Yubikey from Yubico, the Google Titan Security Key, the Kensington Verimark or Nitrokey. Because all these keys implement the standards of the FIDO2 project, many of the examples here work with these keys too.

The Kickstarter campagin has ended, however now you can buy Solo V2 security keys via their Indiegogo campaign. If you decide to buy a security key, then I strongly recommend buying at least two of them so that you can use the second key as a back-up key in case the first key breaks or gets lost.

It appears that the firmware of the Solo V2 currently has some problems, preventing it to work correctly on some sites and there are some complaints about the lack of progress in this matter and a lack of communication. There is hope that these problems will be fixed in the near future though. There is also little documentation, which can make it a bit difficult to get started if you are new to FIDO2 security keys. That’s why I decided to create this guide, to serve as a tutorial explaining how to use the Solo2.

Installing software

For basic usage of the key, you actually do not need to install any software. However there are some utilities available which allow you to update the firmware, set a PIN, view all credentials stored on the key, etc. First of all, we will install the solo2 CLI. It’s not yet packaged in Debian, so we need to download it from Github. I check the sha256sum to ensure I get the right files. This utility written in Rust does not support all functionality yet and for that reason I also install the Python based Solo1 CLI, which is packaged in Debian. The fido2-tools package finally contains some utilities which work on all FIDO2 keys.

# apt install solo-python fido2-tools
$ curl -L -O https://github.com/solokeys/solo2-cli/releases/download/v0.2.0/70-solo2.rules
$ curl -L -O https://github.com/solokeys/solo2-cli/releases/download/v0.2.0/solo2-v0.2.0-x86_64-unknown-linux-gnu
$ curl -L -O https://github.com/solokeys/solo2-cli/releases/download/v0.2.0/solo2.completions.bash
$ sha256sum 70-solo2.rules solo2-v0.2.0-x86_64-unknown-linux-gnu solo2.completions.bash
4133644b12a4e938f04e19e3059f9aec08f1c36b1b33b2f729b5815c88099fe3  70-solo2.rules
d03b20e2ba3be5f9d67f7a7fc1361104960243ebbe44289224f92b513479ed9b  solo2-v0.2.0-x86_64-unknown-linux-gnu
a892afc3c71eb09c1d8e57745dabbbe415f6cfd3f8b49ee6084518a07b73d9a8  solo2.completions.bash
# mv 70-solo2.rules /etc/udev/rules.d/
# mv solo2-v0.2.0-x86_64-unknown-linux-gnu /usr/local/bin/solo2
# chmod 755 /usr/local/bin/solo2
# mv solo2.completions.bash /etc/bash_completion.d/

Touching the Solo2

When authenticating to websites or doing other operations, you will be asked to tap or touch the security key. The Solo2, unlike the Solo1 or Yubikey, does not have a physical button which needs to be depressed, but has 3 touch areas. These are the 3 gold coloured areas at both sides and at the back of the key. You do not need to press them, gently touching one of them is enough. In practice, I had most success touching the two touch zones at both sides of the key simultaneously with 2 fingers.

Updating the Solo V2 firmware version

You can check which version of the firmware is currently installed on your key with this command:

$ solo2 app admin version

At the moment of writing, the most recent version is 1:20200101.9.

To update the firmware version, run this command:

$ solo2 update

Setting a PIN on the Solo V2 key

I strongly recommend setting up a PIN on your FIDO2 key. It will be required to do any administrative tasks on your key, such as adding or removing credentials such as SSH keys,

You cannot set a pin with the solo2 CLI, but you can simply use the solo1 CLI:

$ solo key set-pin

If a PIN has already been set and you want to modify it, run:

$ solo key change-pin

You can also use any Chromium based browser (such as Google Chrome), and go the the URI: chrome://settings/securityKeys . There click on Create a PIN.

Yet another alternative is to use the fido2-token utility, part of fido2-tools. First you need to get the device path of the key:

$ fido2-token -L
/dev/hidraw4: vendor=0x1209, product=0xbeee (SoloKeys Solo 2 Security Key)

So in my case it’s /dev/hidraw4. Then change the PIN like this:

$ fido2-token -C /dev/hidraw4

Do not forget your PIN, otherwise you cannot use your key any more to authenticate to registered sites!

In case you forgot your FIDO2 PIN, you will need to completely reset your key. This will erase all keys and generate new ones, so you will need to have an alternative way to authenticate to websites where you registered this key.

$ solo key reset

FIDO2 Two-Factor Authentication

Usually you go the security settings on the website and there you can enable 2FA. For some sites, you will be required to set up TOTP first before you can register a security key. So make sure you have a TOTP application such as FreeOTP+ for Android or Raivo OTP on iOS. TOTP is then a back-up method for 2FA in case you loose access to your key. If you have multiple FIDO2 keys, don’t forget to register them all.

A side note: don’t use SMS as a second factor for authentication. SMS 2FA is insecure because these messages are transferred in clear text and there are various ways they can be intercepted.

2FA with the Solo2 on Android

You can connect the Solo2 to your Android device by USB, or you can use NFC. When a web application tries to authenticate your key, you will get a pop-up message where you can choose whether you want to connect it via USB or use NFC. In the case of USB, connect your key to the USB port and tap it, just like you would do on your PC. If you chose NFC, just bring your Solo2 key to the back of your phone and it should authenticate.

This all works fine in Chromium based browsers, however I was not able to successfully authenticate with the Solo2 in Firefox. I managed to get it working with Firefox Nightly though. You will need to go to about:config and set security.webauthn.ctap2 and security.webauth.webauthn_enable_usb_token both to true in order to get it working.

Enabling 2FA on well-know websites

Google

Go to https://myaccount.google.com/ and in the left menu click on Security. Under Signing in to Google click on 2-Step Verification. There click on Enable two-factor authentication.In the wizard that appears, you will have to click on Security Key and follow to instructions to add your key.

Github

In the right top corner, click on your avatar and choose Settings. Then in the left menu click on Password & Authentication where you can enable Two-Factor Authentication. You will have to set up TOTP first, and after that, you can register your security key.

Gitlab

In the right top corner, click on your avatar and choose Preferences. Then in the left menu click on Account where you can enable Two-Factor Authentication. You will have to register a Two-Factor Authenticator (TOTP) first, and after that, you can register your WebAuthn devices.

Masstodon

Click on the Preferences icon then choose AccountTwo-Factor Auth. You will need to set up TOTP first, and after that you can add a security key.

Nextcloud

The app Two-Factor WebAuthn needs to be installed on your Nextcloud instance.

Click on your avator in the top right corner and choose Settings. Then choose Security in the left menu and there you can add Webauthn devices.

Microsoft

This is currently not working with Solo2 and a firmware update is needed to fix this.

Normally it should be possible to register the key by going with a Chromium based browser to https://account.microsoft.com/. There click on SecuritySecurity DashboardAdvanced security optionsAdd a new way to sign in or verifyUse a security key.

Twitter

On the website in the left menu, click on MoreSettings and privacy and then on Security and account access – SecurityTwo-factor authentication. There choose Security key.

https://help.twitter.com/en/managing-your-account/two-factor-authentication

Facebook

Really? You should not be using Facebook.

If you really must use Facebook: https://www.facebook.com/help/148233965247823/

LinkedIn

It appears that at the moment of writing LinkedIn does not support 2FA with FIDO2. You can set up TOTP though, which I recommend doing. Click on Me in the top menu and choose Settings & Privacy. Then in the left menu choose Sign in & security and click on Two-Step verification.

WordPress

To enable FIDO2 two-factor authentication in WordPress, install the plugins two-factor and two-factor-provider-webauthn. Enable both modules and then in the WordPress administration menu go to SettingsTwoFactor WebAuthn. Use the option: Disable old U2F provider. the two-factor plugin includes U2F by default, but this is not supported any more by Chromium based browsers, so you want to use the more modern webauthn instead. Then you can set up 2FA in the menu UsersProfile: enable WebAuthn . Then under Security Keys (WebAuthn) click on Register New Key, tap your key and give it a unique name. Do this for both your security keys.

If you have set up Modsecurity with the Core Rule Set, you will end up with a HTTP 403 Forbidden error when trying to register your key or try to authenticate with it. Create /etc/modsecurity/99-wordpress-webauthn.conf with this content:

SecRule REQUEST_FILENAME "@endsWith /wp-admin/profile.php" \
    "id:1100,\
    phase:2,\
    pass,\
    t:none,\
    nolog,\
    chain"
    SecRule ARGS:action "@streq update" \
        "t:none,\
        chain"
        SecRule &ARGS:action "@eq 1" \
            "t:none,\
            ctl:ruleRemoveTargetByTag=OWASP_CRS;ARGS:u2f_response,\
            ctl:ruleRemoveTargetByTag=OWASP_CRS;ARGS:webauthn_response"

SecRule REQUEST_FILENAME "@endsWith /wp-login.php" \
    "id:1101,\
    phase:2,\
    pass,\
    t:none,\
    nolog,\
    ctl:ruleRemoveTargetByTag=OWASP_CRS;ARGS:u2f_response,\
    ctl:ruleRemoveTargetByTag=OWASP_CRS;ARGS:webauthn_response

and reload your Apache configuration. It should now work.

What if it does not work?

If registering your key or authenticating with your key fails on a website, try with a Chromium based browser. Firefox does not support CTAP2 yet, and this can cause trouble on sites which require verification of a PIN.

OpenSSH

To use your Solo2 key for OpenSSH authentication, you will at least version 8.2p1 on both server and client. OpenSSH 8.3p1 adds support for discoverable credentials or resident keys: with discoverable credentials, the FIDO2 security key itself is enough to do SSH public key authentication. This has a slight security risk though if people get access to your Solo2 key because now the only protection is the PIN you have set on the key. Non-discoverable keys don’t have this security risk, because you also need the private key stored on your computer to authenticate.

SSHD configuration for FIDO2 keys

As written before, you need at least version 8.2p1 or 8.3p1 of OpenSSH. The default settings as provided by Debian should be OK, but I strongly recommend to add this option to sshd_config if you only use FIDO2 keys for interactive login:

PubkeyAuthOptions verify-required

I prefer doing this by creating a file /etc/ssh/sshd_config.d/fido2.conf with this line.

This options ensures that only keys which require a PIN can be used, at least adding some protection against theft of a FIDO2 key which contains discoverable credentials.

You can also add the option touch-required to PubkeyAuthOptions in order to require touching the key when authenticating. This will make it impossible to authenticate with keys which were created with the no-touch-required option.

Setting up FIDO2 credentials for SSH

To generate credentials for SSH with your FIDO2, you basically use this command

$ ssh-keygen -t ed25519-sk

There are diffferent options available which you can add:

  • -O resident: You want to create discoverable credentials.
  • -O no-touch-required: You want to disable the requirement of touching the key for authenticating.
  • -O verify-required: You require that the PIN is entered when authenticating. I strongly recommend this option.
  • -O application=ssh:SomeUniqueName: In case you want to store different SSH keys on your Solo2, you will have to give each of them a different application name starting with ssh:
  • -f ~/.ssh/id_ed25519_sk_solo2_blue: If you use multiple FIDO2 keys, you may want to store the key in a unique file for every FIDO2 key. Replace the file name of this example by the name of your choice.

You can verify that the credentials are correctly stored on your Solo2 using this command:

$ solo key credential ls

In case you would want to remove the credentials stored on your key, you can do so by using this command:

$ solo key credential rm CREDENTIALID

Replace CREDENTIALID by the value you found with the previous command.

After creating the key, you need to copy the public key to the authorized_keys file on your server. You can use ssh-copy-id for that:

$ ssh-copy-id -i ~/.ssh/id_ed25519_sk_solo2_blue.pub username@server.example.org

Of course use the correct file name for the public key.

If you used the option no-touch-required when generating the key, you will have to edit the ~/.ssh/authorized_keys file on your server so that this options precedes the key. For example if authorized_keys contains this:

sk-ssh-ed25519@openssh.com AAAA....= username@host

Change it to this:

no-touch-required sk-ssh-ed25519@openssh.com AAAA....= username@host

Now it should be possible to log in to the server using this command:

$ ssh -i ~/.ssh/id_ed25519_sk_solo2_blue -o IdentitiesOnly=yes username@server.example.org

You will be asked to enter the PIN of your key and to touch it, depending on the options you used when creating the key. I add -o IdentitiesOnly=yes because otherwise ssh will first try to authenticate using the keys loaded in your SSH agent. With this option we enforce it to use only the private key we have specified with the -i parameter.

You can make this default by editing ~/.ssh/config, so that you don’t need to repeat the -i and -o parameters every time when connecting:

Host server.example.org
    Username myusername
    IdentityFile ~/.ssh/id_ed25519_sk_solo2_blue
    IdentitiesOnly yes

Importing discoverable credentials on another system

When you use discoverable credentials, all information needed for authentication is stored on the key itself, in contrast to non-discoverable credentials, where part of that information is also stored in the private key file on the computer. For this reason, with discoverable credentials, it is easy to import them on any computer.

$ cd ~/.ssh/ 
$ ssh-keygen -K

The public and private key will be written in the .ssh directory, and then you can authenticate again using the ssh -o IdentiesOnly=yes -i command just like on the system where you generated the key.

Troubleshooting FIDO2 SSH authentication

On the server check the sshd logs, which can be found in /var/log/auth.log or in the ssh journal:

# journalctl -u ssh

Successful authentication with your FIDO2 key, should be logged like this:

Accepted publickey for username from xxx.xxx.xxx.xxx port zzzzzz ssh2: ED25519-SK SHA256:....

Notice the ED25519-SK part which indicates that the credentials on your FIDO2 key were used.

If you see this:

error: public key ED25519-SK SHA256:... signature for username from xxx.xxx.xxx.xxx port zzzzz rejected: user presence (authenticator touch) requirement not met

This means that you have created a key with the no-touch-required options not set. Try adding no-touch-required to the authorized_keys on the server, as noted above, at least if your server does not have PubkeyAuthOptions touch-required set.

On the client-side, you can add the -v parameter to debug what happens:

$ ssh -v -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_sk_solo2_blue username@server.example.org

If you are using GNOME with gnome-keyring as ssh-agent, you will encounter this problem:

debug1: Offering public key: /home/username/.ssh/id_ed25519_sk_solo2_blue ED25519-SK SHA256:... explicit authenticator agent
debug1: Server accepts key: /home/username/.ssh/id_ed25519_sk_solo2_blue ED25519-SK SHA256:... explicit authenticator agent
sign_and_send_pubkey: signing failed for ED25519-SK "/home/username/.ssh/id_ed25519_sk_solo2_blue" from agent: agent refused operation

This is because of the lack of support of verify-required credentials in ssh-agent/gnome-keyring.

A work-around is to rename the public key, so that gnome-keyring will ignore it:

$ mv ~/.ssh/id_ed25519_sk_solo2_blue.pub ~/.ssh/id_ed25519_sk_solo2_blue.public

you will need to log out and login again after making this change.

Sources and more information

Securing SSH with FIDO2

Solo2 discussions

Online resizing block devices and file systems

Here is a short reference sheet for resizing block devices and file systems.

Resizing block devices

Logical volumes

Add 100 GiB to the logical volume with name logicalvolume in volume group volumegroup.

# lvextend -L +100G volumegroup/logicalvolume 

QEMU block device

If you resized a block device which is used to store a virtual disk for a QEMU VM, you will need to expand the virtual disk itself. First we need to know the name of the virtual disk. If you are managing your QEMU VMs via libvirt, you can use this command to see all virtual disks:

# virsh qemu-monitor-command VMname – hmp "info block"
drive-virtio-disk0 (#block108): /dev/vm/web-www (raw)
    Attached to:      /machine/peripheral/virtio-disk0/virtio-backend
    Cache mode:       writeback, direct

drive-virtio-disk1 (#block302): /dev/vm/web-logs (raw)
    Attached to:      /machine/peripheral/virtio-disk1/virtio-backend
    Cache mode:       writeback, direct

Then if you resized the logical volume /dev/vm/web-www from 100 to 200 GiB using the command mentioned before, you can resize the corresponding QEMU virtual disk drive-virtio-disk0 using this command:

# virsh qemu-monitor-command VMname – hmp "block_resize drive-virtio-disk0 200G"

Resizing file systems

When you have resized a block device, you will need to resize the file system on the block device too in order to use the disk space.

EXT4

# resize2fs /dev/vda

XFS

# xfs_growfs /var/www/

BTRFS

# btrfs filesystem resize max /var/www

Setting up Wireguard VPN with IPv6

I wanted to set up Wireguard on a VPS, not only to tunnel IPv4 traffic, but also allowing me to tunnel IPv6 traffic. As this is IPv6 of course I preferred not to use NAT, but to assign a public IP address to the client. I read some documentation and blog posts, but I struggled getting it to work. Most tutorials I found on the Internet, create a separate IPv6 subnet for the VPN but I could not get this to work. For some reason, IPv6 traffic successfully went through the VPN tunnel and then exited the VPN gateway, but then any response never reached my VPN gateway and hence also not the client.

I decided to try another way: using an NDP proxy. NDP or the Neighbour Discovery Protocol, is similar to ARP which is used in IPv4. Using this protocol, network devices can discover where on the network a certain IP is located. By letting the VPN gateway answer NDP requests for the VPN client, the gateway would correctly send back all responses to the VPN gateway, which then forwards it to the VPN clients.

Configuring the network on the VPN gateway

I use systemd-networkd to set up the network. It’s the most modern way of network configuration and works the same on all distributions using systemd, but of course you can make the same settings in /etc/network/interfaces or whatever your distribution uses. Of course when making changes to a remote server, make sure you can access a console without needing a working network connection on the server, in case things go wrong and the network connection breaks.

On my VPN server, the public network interface is named ens192 (use the command $ ip addr to find it on your system). My public IPv4 address is www.xxx.yyy.zzz with subnet 255.255.255.0 and gateway ww.xx.yy.1. I have the 64 bit IPV6 prefix aaaa:bbbb:cccc:dddd and the IPv6 gateway is fe80::1.

I set this in /etc/systemd/network/internet.net:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=1.1.1.2
DNS=1.0.0.2
Address= www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
DNS=2606:4700:4700::1112
DNS=2606:4700:4700::1002

In this example I’m using the Cloudflare malware blocking DNS filters, but you can of course just use your ISP’s DNS servers here.

Setting up Wireguard

Run these commands on the Wireguard VPN gateway, and on all clients:

# apt install wireguard-tools
# cd /etc/wireguard
# umask 077
# wg genkey | tee privatekey | wg pubkey > publickey

Then create /etc/wireguard/wg0.conf on the VPN gateway with these contents:

[Interface]
Address = 192.168.7.1,fd42:42:42::1/64
PrivateKey = contents_of_file_privatekey
ListenPort = 51820

#client1
[Peer]
PublicKey = contents_of_publickey_of_client
AllowedIPs = 192.168.7.2/32,aaaa:bbbb:cccc:dddd:ffff::2/128

Add a [Peer] section for every client, and change the both the IPv4 and IPv6 address in AllowedIPs so that they are unique (replace 2 by 3 and so on) .

On the clients, create /etc/wireguard/wg0.conf with these contents:

[Interface]
Address = 192.168.7.2/32,2a00:6d40:40:45ef:ffff::2/128
PrivateKey = contents_of_privatekey_of_client
DNS = 2606:4700:4700::1112, 2606:4700:4700::1002, 1.1.1.2, 1.0.0.2

[Peer]
PublicKey = contents_of_publickey_of_vpn_gateway
Endpoint = vpngateway.example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0

In the [Interface] section make sure to use the same IP addresses as the ones you have set in the corresponding [Peer] section on the VPN gateway. Set the DNS name (or IP address) of the VPN gateway as Endpoint in the [Peer] section. The hostname’s DNS entry can have both an A and AAAA record. You can replace your DNS servers by your preferred ones. You can also consider running your own DNS server on the VPN gateway.

Make sure that all wg*.conf files on client and server are only readable by root, because they contain private keys.

Configuring the firewall (Shorewall)

I use the Shoreline firewall, Shorewall, as firewall.

Make sure you have shorewall and shorewall6 installed:

# apt install shorewall shorewall6

Shorewall6

First we create a separate zone for our VPN in /etc/shorewall6/zones:

fw firewall
net ipv6
vpn ipv6

Then we configure the network interfaces and assign it to the right zone in /etc/shorewall6/interfaces:

net     NET_IF          tcpflags,routeback,proxyndp,physical=ens192
vpn     wg0		tcpflags,routeback,optional

Then we allow connections from the VPN to the firewall and to the Internet in /etc/shorewall6/policy:

$FW	net		ACCEPT
vpn     net		ACCEPT
vpn     $FW		ACCEPT
net	all		DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all		REJECT		$LOG_LEVEL

Keep in mind that your VPN client will have a public IPv6 address, which is accessible from the Internet. The rule net all DROP protects your VPN clients against access from the Internet.

Then we create some rules which allows access to the SSH server and the Wireguard VPN server from the Internet in /etc/shorewall6/rules:

Invalid(DROP)      net    	$FW		tcp
Ping(DROP)	   net		$FW
ACCEPT		   $FW		net		ipv6-icmp
AllowICMPs(ACCEPT) all		all
ACCEPT		   all		all		ipv6-icmp	echo-request


SSH(ACCEPT)	   net		$FW
ACCEPT		   net		$FW		udp	51820 # Wireguard

We allow some required ICMPv6 message types defined in /usr/share/shorewall/action.AllowICMPs, as well as the echo-request type, which should not be dropped on IPv6.

For security reasons you could even choose to not open the SSH port for the net zone. SSH will only be accessible via the VPN then.

Then in order to make sure that the gateway knows that the VPN client aaa:bbb:cccc:dddd::2 is reachable via the VPN gateway, we need to set up NDP proxying. The Neighbor Discovery Protocol is similar to ARP in IPv6. We set this in /etc/shorewall6/proxyndp:

aaa:bbb:cccc:dddd::2 wg0 ens192

Finally we need to enable IP forwarding in /etc/shorewall6/shorewall6.conf:

IP_FORWARDING=Yes

Then we check whether everything compiles fine and enable and start the service:

# shorewall6 compile
# systemctl restart shorewall6
# systemctl enable shorewall6

Shorewall

For IPv4 we configure Shorewall to use NAT to provide Internet access to the VPN clients.

/etc/shorewall/zones:

fw	firewall
net	ipv4
vpn	ipv4

/etc/shorewall/interfaces:

net     NET_IF          dhcp,tcpflags,logmartians,nosmurfs,sourceroute=0,routefilter,routeback,physical=ens192
vpn	wg0		tcpflags,logmartians,nosmurfs,sourceroute=0,optional,routefilter,routeback

/etc/shorewall/policy:

$FW	net	ACCEPT
vpn	net	ACCEPT
vpn	$FW	ACCEPT
net	all	DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all	REJECT		$LOG_LEVEL

/etc/shorewall/rules:

# Drop packets in the INVALID state

Invalid(DROP)  net    	        $FW		tcp

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP)	net		$FW

SSH(ACCEPT)	net		$FW
ACCEPT		net		$FW		udp	51820

/etc/shorewall/snat:

MASQUERADE	192.168.7.0/24	NET_IF

/etc/shorewall/shorewall.conf:

IP_FORWARDING=Yes

Compile and load the rules and enable Shorewall permanently:

# shorewall compile
# systemctl restart shorewall
# systemctl enable shorewall

Enabling and testing the VPN

On the server run this to enable the Wireguard server:

# systemctl enable – now wg-quick@wg0

To connect to the VPN, run this on the client:

# systemctl start wg-quick@wg0

Check if you can browse the world wide web. Use these websites to check your IP address and whether you have a working IPv6 connection:

https://test-ipv6.com/

https://ipv6-test.com/

You can also use traceroute and traceroute6 to test whether traffic is correctly going through the VPN tunnel:

# traceroute www.google.com
# traceroute6 www.google.com

Debugging Wireguard

If things don’t work as expected, you can enable debug logging in the Wireguard module with this command:

# echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

Replace +p by -p in order to disable debug logging. You can find the logs in your kernel messages, for example by running

# journalctl -f -k

Also firewall log messages will appear here.

You can use tcpdump to check the traffic on the wire (or in the VPN tunnel). For example to see all ipv6 traffic in the tunnel on the gateway:

# tcpdump -nettti wg0 "ip6"

Sources

Setup WireGuard with global IPv6

Setting up WireGuard IPv6

Reddit: Wireguard doesn’t seem to work with IPv6

Wireguard: enable debug logging to fix network issues

Shorewall6: Proxy NDP

Debian GNU/Linux on a HP Elitebook 845 G8

Some time ago, I received a new laptop, the HP Elitebook 845 G8. This is a 14″ laptop with an AMD CPU of the Renoir family, in my case an AMD Ryzen 7 PRO 5850U. As always, I run Debian GNU/Linux testing (currently Bookworm) on it. In this post, I will explain how to get all hardware working. This guide probably also works for other G8 Elitebooks, such as the Elitebook 835 G8 and Elitebook 855 G8, because they are all quite similar.

You can find detailed logs and reports of people running Linux on the Elitebook 845 G8 in the Hardware for Linux database.

Installation

I used a USB disk to boot the installer and a USB-C dock with an Ethernet interface to do a network installation. If you use the Debian installer with non-free firmware, you can also do the installation over wifi. I have not tried the current stable release Debian 11 Bullseye on this system. For best compatibility I strongly recommend testing because it has a more recent kernel and drivers.

Download the installation cd for Debian testing with non-free firmware: you need the file firmware-testing-amd64-netinst.iso.

Now we need to write it to a USB disk. Make sure there is no data on the drive you want to keep, because this process will completely wipe the disk.

To write the ISO image to a USB disk, Windows users can use the application Rufus, MacOS users can use Balena Etcher. If you are using Linux, you can dd the ISO image on your USB disk, or use a GUI like Fedora Media Writer.

Reboot the system and press F10 when the HP logo appears to load the BIOS/UEFI setup. Go to the Advanced page and select Boot Options. There make sure that USB Storage Boot is enabled. If you want to work with custom kernels, it can be handy to disable Secure Boot in SecuritySecure Boot Configuration, but it’s not needed to install and use Debian.

Save the changes you made (if any) and reboot the system and press F9 at the HP logo to get the boot menu. In the boot menu, select your USB drive to start the Debian installer.

Enabling non-free repositories

We will need to configure the non-free repositories for apt because we need several firmware packages from non-free. Edit /etc/apt/sources.list and check whether any deb line has main contrib and non-free at the end. If not add it, and then run

# apt update

Updating the BIOS/UEFI firmware

If you have a Windows installation, you can update the firmware from there, even before you install Linux. But you can also update the firmware without Windows. Follow the instructions in that blog post. It’s important to do this, not only because this gives you essential security fixes, but also bug fixes, some of which specifically for Linux compatibility.

Updating CPU firmware

Install the package amd64-microcode to ensure your AMD CPU is always running the latest microcode, which includes security fixes:

# apt-get install amd64-microcode

Flashing other firmware

The fwupd utility can download and install firmware from the LVFS. The firmware of the fingerprint reader of the Elitebook 845 G8, can be updated like this, and may be necessary to get the fingerprint reader working in Linux. First make sure fwupd is installed:

# apt install fwupd fwupd-amd64-signed

Now update all available firemware:

# fwupdtool update

If you have a HP USB-C Dock G5, then new firmware is also available in the LVFS, but it’s in the testing repository. To enable this repository, run this command:

# fwupdmgr enable-remote lvfs-testing 

Radeon Vega GPU

lspci identifies this GPU as

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev d1)

First make sure the firmware is installed for the GPU. This is needed to get any hardware acceleration.

# apt install firmware-amd-graphics

Then make sure these packages are installed in order to get Vulkan, VA-API, VDPAU and OpenCL support:

 # apt install mesa-va-drivers mesa-vulkan-drivers mesa-vdpau-drivers mesa-opencl-icd

Now add this to /etc/environment so that it uses the correct VDPAU driver:

VDPAU_DRIVER=radeonsi

In order to get GStreamer based players to use VA-API, you need to intsall this package:

# apt install gstreamer1.0-vaapi

After installing the firmware and editing /etc/environment you will need to reboot your system.

Unfortunately most video players and web browsers still don’t use VA-API hardware acceleration by default, but this needs to be configured manually. I will write a separate article about that later.

Realtek wifi adapter

The wifi adapter is a Realtek RTL8822CE according to lspci:

01:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter [10ec:c822]

Install the firmware to get it working:

# apt install firmware-realtek

This laptop can also be delivered with an Intel AX200 Wi-Fi 6 adapter (which is actually a better option than this one from Realtek). If you have this one, you will need to install the firmware-iwlwifi package instead.

Smartcard reader

lsusb identifies this smartcard reader as an Alcor AU9540:

Bus 005 Device 004: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader

Note that it only sees the smartcard reader when a card has been inserted.

You will need pscd with the CCID driver to use this smartcard reader:

# apt install pscsd

Fingerprint reader

The fingerprint reader can be seen like this in lsusb:

Bus 003 Device 003: ID 06cb:00df Synaptics, Inc. 

Make sure you have installed all firmware updates with fwupd and then you need to install these packages:

# apt install fprintd libpam-fprintd

In GNOME, under SettingsUsers you can enable login on fingerprint and add your fingerprints.

Sound

lspci:

04:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 01)<br />04:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller

Sound is working out of the box. I recommend switching from Pulseaudio to Pipewire.

Webcam

lsusb:

Bus 001 Device 002: ID 0408:5348 Quanta Computer, Inc. HP HD Camera

The webcam works out of the box. Many applications will see the IR camera as a second camera.

Bluetooth

lsusb:

Bus 003 Device 002: ID 0bda:b00c Realtek Semiconductor Corp. 802.11ac WLAN Adapter

If you use lsusb -v you will see that this is actually the Bluetooth Radio adapter. It is combined with the wifi adapter, hence the confusion.

Suspend/resume

HP does not support S3 (traditional suspend-to-ram/standby) in its recent Elitebooks any more, but instead uses s0ix (s2idle/suspend-to-idle/modern standby). S2idle support for AMD CPU’s was only added in Linux 5.11 with the amd_pmc driver. I recommend a very recent kernel, because later kernel versions had bug fixes in this regard too. However suspend regressed in stable update 5.17.3 (and others), a bug which was fixed in 5.17.5. I’m using a custom-built 5.17.5 kernel, but a fixed kernel will appear soon in Debian.

If you have HP Drivelock enabled, then your system will fail to resume. Drivelock is a security feature which can be set up in the BIOS and requires you to enter a password when starting up the system in order to access the contents of the disk. When trying to resume the system, fans start running, the keyboard backlight reacts to key presses, but the screen remains blank, nothing is written to logs and also network does not come up. Apparently this is a bug in HP’s BIOS/UEFI firmware which can be worked around by adding iommu=pt to the kernel command line. To do so, edit /etc/default/grub and add this to the variable GRUB_CMDLINE_LINUX_DEFAULT. For example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

Then update the GRUB configuration:

# update-grub

Install isenkram to help install drivers when plugging in hardware

Isenkram is a utitliy which will show a message when you connect hardware to your system and extra software or firmware is available for that hardware.

# apt install isenkram

Enabling trimming of the NVME SSD

Enable the fstrim timer to make sure the SSD is trimmed on regular intervals:

# systemctl enable – now fstrim.timer

Increasing battery time

Switch to AMD P-state driver

AMD developed the amd_pstate driver which was introduced in Linux 5.17. amd_pstate offers better performance per Watt than acpi-cpufreq

/etc/modprobe.d/acpi-cpufreq-blacklist.conf

blacklist acpi-cpufreq

/etc/modules-load.d/amd-pstate.conf

amd_pstate

Set up TLP

TLP is a tool which optimizes power consumption of your system in order to increase battery time. TLP also has an options Radio Device Wizard, which I will use here to automatically disable wifi when the system is connected via an Ethernet cable.

# apt install tlp tlp-rdw

Configure the Radio Device Wizard by creating the file /etc/tlp.d/10-tlp-rdw.conf:

# tlp-rdw - Parameters for the radio device wizard

# Possible devices: bluetooth, wifi, wwan.
# Separate multiple radio devices with spaces.
# Default: <none> (for all parameters below)

DEVICES_TO_DISABLE_ON_LAN_CONNECT="wifi wwan"
DEVICES_TO_DISABLE_ON_WIFI_CONNECT="wwan"
DEVICES_TO_DISABLE_ON_WWAN_CONNECT="wifi"

# Radio devices to enable on disconnect.

DEVICES_TO_ENABLE_ON_LAN_DISCONNECT="wifi wwan"
DEVICES_TO_ENABLE_ON_WIFI_DISCONNECT=""
DEVICES_TO_ENABLE_ON_WWAN_DISCONNECT=""

# Radio devices to enable/disable when docked.

DEVICES_TO_ENABLE_ON_DOCK=""
DEVICES_TO_DISABLE_ON_DOCK=""

# Radio devices to enable/disable when undocked.

DEVICES_TO_ENABLE_ON_UNDOCK="wifi"
DEVICES_TO_DISABLE_ON_UNDOCK=""

To enable best ASPM power saving features when on battery, create /etc/tlp.d/20-aspm.conf:

PCIE_ASPM_ON_AC=default<br />PCIE_ASPM_ON_BAT=powersupersave

You will also need to edit /etc/default/grub file to add pcie_aspm=force to the Linux command line, for example

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_aspm=force"

And run update-grub to update the GRUB configuration.

Disable Bluetooth and wifi when on battery and they are not in use by creating /etc/tlp.d/30-disable-devices-on-battery.conf:

DEVICES_TO_DISABLE_ON_BAT_NOT_IN_USE="bluetooth nfc wifi wwan"

Submitting your system to the Linux Hardware database

The Linux Hardware database is a useful tool where users searching for hardware, can check the compatibility of systems with Linux. I recommend running this tool on all your Linux systems. After submission, you will get a link where you can view the data and indicate whether all hardware works and which work-arounds you had to apply. Click on the Review button on the page to do so.

# apt install hw-probe
# hwprobe – all – upload

Conclusion

Actually Linux compatibility of the HP Elitebook 845 is actually in good shape. It’s not perfect, but all hardware can be made to work. On distros like Ubuntu, which install non-free firmware by default, it should even be easier to make everything work. Still HP lags behind Dell and Lenovo in Linux support, because they don’t make it possible to flash the BIOS/UEFI firmware using fwupd, while all recent Dell and Lenovo business laptops have their firmware available in the LVFS. Also the problem that iommu=pt needs to be used to successfully resume the laptop when Drivelock is enabled, is a problem that HP should address in a BIOS update.

Updating HP BIOS firmware from Linux

I have an HP Elitebook 845 G8 laptop. I wanted to update the BIOS/UEFI firmware to the latest version. This is important because firmware updates include essential security fixes.

On Linux you can use the LVFS (Linux Vendor Firmware Service) to easily install firmware updates with the fwupd utility or any of its graphical front-ends (such as GNOME Software or KDE Discover). Dell and Lenovo offer firmware updates for many of their systems via LVFS, however HP only has a very small number of firmware updates available on LVFS.

Fortunately it is possible to install firmware updates without having Windows installed, but it’s a bit more manual work. Here I will explain how I managed to update my Elitebook 845 G8, but this should work for most recent HP laptops and desktops.

First you need to download the firmware update for your system. Use your favourite web search engine and search for the model name and add the word downloads to it, for example: HP Elitebook 845 G8 downloads. The first hit will probably bring you to the right page, in my case: https://support.hp.com/us-en/drivers/selfservice/hp-elitebook-845-g8-notebook-pc/38492638

If you don’t know the exact model of your system, use this command (as root):

# /usr/sbin/dmidecode | grep "Product Name"

On the HP downloads page, you will probably get a mssage that it was unable to find drivers for your product. Click on the link Choose a different OS and select any Windows 10 version. Now you will find the latest firmware for your system under BIOS-System Firmware. Click on Download. It will try to let you install a Windows Download and Install Assistant: click on No thanks, I will download and install manually to directly get the firmware file, which should have a name similar to sp138978.exe.

Once you have downloaded this file, we need to extract it. You can use 7-Zip for that. Make sure it is installed on your system:

# apt-install 7zip

Then we will make a directory and extract the firmware package in it:

$ mkdir /tmp/hpfirmware
$ cd /tmp/hpfirmware
$ 7zz x  ~/Downloads/sp138978.exe

You will see many files in the /tmp/hpfirmware directory, such as the History.txt file which you can read if you want to know which changes and bug fixes this update brings. The firmware itself is stored in a file with the extension .bin, in my case it’s named T82_01082000.bin.

To install this firmware, we have to copy it to a directory HP/DEVFW/firmware.bin in your EFI directory (this assumes that you are booting your system in UEFI mode and not in legacy BIOS mode). So as root run these commands from the /tmp/hpfirmware directory:

# mkdir -p /boot/efi/EFI/HP/DEVFW
# cp *.bin /boot/efi/EFI/HP/DEVFW/firmware.bin

Now reboot your system. In GRUB’s menu, choose UEFI Firmware Settings. You will get HP’s Startup Menu of which the last menu item is Update System and Supported Device Firmware. This should now automatically install the firmware update. This takes several minutes , and you screen will go blank during some time: don’t panic and let it run.

Web application firewall: Modsecurity and Core Rule Set

A web application firewall (WAF) filters HTTP traffic. By integrating this in your web server, you can make sure potentially dangerous requests are blocked before they arrive to your web application or sensitive data leaks out of your web server. This way you add an extra defensive layer potentially offering extra protection against zero-day vulnerabilities in your web server or web applications. In this blog post, I give a tutorial how to install and configure ModSecurity web application firewall and the Core Rule Set on Debian. With some minor adaptions you can also use this guide for setting up ModSecurity on Ubuntu or other distributions.

ModSecurity is the most well-known open source web application firewall. The future of ModSecurity does not look too bright but fortunately with Coraza WAF an alternative which is completely compatible with ModSecurity is in development. At this moment Coraza only integrates with the Caddy web server, and does not have a connector for Apache or NGinx so for that reason it is currently not yet usable as a replacement for ModSecurity.

While ModSecurity provides the framework for filtering HTTP traffic, you also need rules which define what to bloc and that’s where the Core Rule Set (CRS) comes in. CRS is a set of generic rules winch offer protection to a various range of common attacks via HTTP, such as SQL injection, code injection and cross-site scripting (XSS) attacks.

Install ModSecurity and the Core Rule Set on Debian

I install the Apache module for ModSecurity, the geoip-database, which can be used for blocking all requests from certain countries, and modsecurity-crs, which contains the Core Rule Set. I take this package from testing, because it has a newer version (version 3.3.2 at the time of writing). There is no risk in taking this package from testing, because it only contains the rules and does not depend on any other packages from testing/unstable. If you prefer faster updates, you can also use unstable.

# apt install libapache2-mod-security2 geoip-database
# apt install -t testing modsecurity-crs

Configuring ModSecurity

In order to load the ModSecurity module in Apache, run this command:

# a2enmod security2

Then copy the example ModSecurity configuration file to /etc/modsecurity/modsecurity.conf:

cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Now edit /etc/modsecurity/modsecurity.conf. I highlight some of the options:

SecRuleEngine on
SecRequestBodyLimit 536870912
SecRequestBodyNoFilesLimit 131072
SecAuditLog /var/log/apache2/modsec_audit.log
#SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
#"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"
SecPcreMatchLimit 500000
SecPcreMatchLimitRecursion 500000
SecStatusEngine Off

The SecRuleEngine option controls whether rules should be processed. If set to Off, you completely disable all rules, with On you enable them and it will block malicious actions. If set to DetectionOnly, ModSecurity will only log potential malicious activity flagged by your rules, but will not block them. DetectionOnly can be useful for temporary trying out the rules in order to find false positives before you really start blocking potential malicious activity.

The SecAuditLog option defines a file which contains audit logs. This file will contain detailed logs about every request triggering a ModSecurity rule.

The SecPcreMatchLimit and SecPcreMatchLimitRecursion set the match limit and match limit recursion for the regular expression library PCRE. Setting this high enough will prevent errors that the PCRE limits were exceeded while analyzing data, but setting it too high can make ModSecurity vulnerable to a Denial of Service (DoS) attack. A Core Rule Set developer recommends a value of 50000 so that’s what I use here.

I change SecRequestBodyLimit to a higher value to allow large file uploads.

I disable the rule 200004 because it is known to cause false positives.

Set SecStatusEngine to Off to prevent ModSecurity sending version information back its developers.

After changing any configuration related to ModSecurity or the Core Rule Set, reload your Apache web server:

# systemctl reload apache2

Configuring the Core Rule Set

The Core Rule Set can be configured via the file /etc/modsecurity/crs/crs-setup.conf.

Anomaly Scoring

By default the Core Rule Set is using anomaly scoring mode. This means that individual rules add to a so called anomaly score, which at the end is evaluated. If the anomaly score exceeds a certain threshold, then the traffic is blocked. You can read more about this configuration in crs-setup.conf but the default configuration should be fine for most people.

Setting the paranoia level

The paranoia level is a number from 1 to 4 which determines which rules are active and contribute to the anomaly scoring. The higher the paranoia level, the more rules are activated and hence the more aggressive the Core Rule Set is, offering more protection but potentially also causing more false positives. By default the paranoia level is set to 1. If you work with sensitive data, it is recommended to increase the paranoia level.

The executing paranoia level defines the rules which will be executed but their score will not be added to the anomaly scoring. When HTTP traffic hits rules of the executing paranoia level, this traffic will only be logged but not be blocked. It is a especially useful to prepare for increasing the paranoia level and finding false positives on this higher level, without causing any disruption for your users.

To set the paranoia level to 1 and the executing paranoia level to 2, make sure you have these rules set in crs-setup.conf:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.paranoia_level=1"
SecAction \
  "id:900001,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.executing_paranoia_level=2"

Once you have fixed all false positives, you can raise the paranoia level to 2 to increase security.

Defining the allowed HTTP methods

By default the Core Rule Set only allows the GET, HEAD, POST and OPTIONS HTTP methods. For many standard sites this will be enough but if your web applications also use restful APIs or WebDAV, then you will need to add the required methods. Change rule 900200, and add the HTTP methods mentioned in the comments in crs-setup.conf.

SecAction \
 "id:900200,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'"

Disallowing old HTTP versions

There is a rule which determines which HTTP versions you allow in HTTP requests. I uncomment it and modify it to only allow HTTP versions 1.1 and 2.0. Legitimate browsers and bots always use one of these modern HTTP versions and older versions usually are a sign of malicious activity.

SecAction \
 "id:900230,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.allowed_http_versions=HTTP/1.1 HTTP/2 HTTP/2.0'"

Blocking specific countries

Personally I’m not a fan of completely blocking all traffic from a whole country, because you will also block legitimate visitors to your site, but in case you want to this, you can configure this in crs-setup.conf:

SecGeoLookupDB /usr/share/GeoIP/GeoIP.dat
SecAction \
 "id:900600,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.high_risk_country_codes='"

Add the two-letter country codes you want to block to the last line (before the two quotes), multiple country codes separated by a space.

Make sure you have the package goeip-database installed.

Core Rule Set Exclusion rules for well-known web applications

The Core Rule Set contains some rule exclusions for some well-known web applications like WordPress, Drupal and NextCloud which reduces the number of false positives. I add the following section to crs-setup.conf which will allow me to enable the exclusions in the Apache configuration by setting the WEBAPPID variable in the Apache configuration whenever I need them.

SecRule WEBAPPID '@beginsWith wordpress' 'id:20000,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_wordpress=1'
SecRule WEBAPPID '@beginsWith drupal' 'id:20001,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_drupal=1'
SecRule WEBAPPID '@beginsWith dokuwiki' 'id:20002,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_dokuwiki=1'
SecRule WEBAPPID '@beginsWith nextcloud' 'id:20003,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_nextcloud=1'
SecRule WEBAPPID '@beginsWith cpanel' 'id:20004,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_cpanel=1'
SecRule WEBAPPID '@beginsWith xenforo' 'id:20005,phase:1,nolog,pass,t:none setvar:tx.crs_exclusions_xenforo=1'

Adding rules for Log4Shell and Spring4Shell detection

At the end of 2021 a critical vulnerability CVE-2021-44228, named Log4Shell, was detected in Log4j, which allows remote attackers to run code on a server with the vulnerable Log4j version. While the Core Rule Set offered some mitigation of this vulnerability out of the box, this protection was not complete. New improved detection rules against Log4Shell were developed. Because of the severity of this bug and the fact that it’s being exploited in the wild, I strongly recommend adding this protection manually when using ModSecurity version 3.3.2 (or older). Newer, not yet released versions, should have complete protection out of the box.

First modify /etc/apache2/mods-enabled/security2.conf so that it looks like this:

<IfModule security2_module>
        # Default Debian dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        # Include all the *.conf files in /etc/modsecurity.
        # Keeping your local configuration in that directory
        # will allow for an easy upgrade of THIS file and
        # make your life easier
        IncludeOptional /etc/modsecurity/*.conf

        # Include OWASP ModSecurity CRS rules if installed
        IncludeOptional /usr/share/modsecurity-crs/*.load
        SecRuleUpdateTargetById 932130 "REQUEST_HEADERS"
</IfModule>

Then create the file /etc/modsecurity/99-CVE-2021-44228.conf with this content:

# Generic rule against CVE-2021-44228 (Log4j / Log4Shell)
# See https://coreruleset.org/20211213/crs-and-log4j-log4shell-cve-2021-44228/
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx (?:\${[^}]{0,4}\${|\${(?:jndi|ctx))" \
    "id:1005,\
    phase:2,\
    block,\
    t:none,t:urlDecodeUni,t:cmdline,\
    log,\
    msg:'Potential Remote Command Execution: Log4j CVE-2021-44228', \
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

In March 2022 CVE-2022-22963, another remote code execution (RCE) vulnerability was published in the Spring framework was published. The Core Rule Set developed a new rule to protect against this vulnerability which will be included in the next version, but the rule can be added manually if you are running the Core Rule Set version 3.3.2 or older.

To do so, create the file /etc/modsecurity/99-CVE-2022-22963.conf with this content:

# This rule is also triggered by the following exploit(s):
# - https://www.rapid7.com/blog/post/2022/03/30/spring4shell-zero-day-vulnerability-in-spring-framework/
# - https://www.ironcastle.net/possible-new-java-spring-framework-vulnerability-wed-mar-30th/
#
SecRule ARGS|ARGS_NAMES|REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@* \
    "@rx (?:class\.module\.classLoader\.resources\.context\.parent\.pipeline|springframework\.context\.support\.FileSystemXmlApplicationContext)" \
    "id:1006,\
    phase:2,\
    block,\
    t:urlDecodeUni,\
    msg:'Remote Command Execution: Malicious class-loading payload',\
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',\
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/248',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/2',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl2=+%{tx.critical_anomaly_score}'"

Don’t forget to reload your Apache configuration after adding these rules.

Testing ModSecurity and checking the logs

We can now easily test ModSecurity by doing a request which tries to abuse a cross-site scripting (XSS) vulnerability:

$ curl -I "https://example.org/?search=<script>alert('CRS+Sandbox+Release')</script>"

This should return HTTP response 403 (Forbidden).

Whenever something hits your ModSecurity rules, this will be logged in your Apache error log. The above request has created these messages in the error log:

[Sat Apr 09 22:22:02.716558 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. detected XSS using libinjection. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "55"] [id "941100"] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.716969 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Pattern match "(?i)<script[^>]*>[\\\\s\\\\S]*?" at ARGS:search. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "82"] [id "941110"] [msg "XSS Filter - Category 1: Script Tag Vector"] [data "Matched Data: <script> found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.717249 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Pattern match "(?i:(?:<\\\\w[\\\\s\\\\S]*[\\\\s\\\\/]|['\\"](?:[\\\\s\\\\S]*[\\\\s\\\\/])?)(?:on(?:d(?:e(?:vice(?:(?:orienta|mo)tion|proximity|found|light)|livery(?:success|error)|activate)|r(?:ag(?:e(?:n(?:ter|d)|xit)|(?:gestur|leav)e|start|drop|over)|op)|i(?:s(?:c(?:hargingtimechange ..." at ARGS:search. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "199"] [id "941160"] [msg "NoScript XSS InjectionChecker: HTML Injection"] [data "Matched Data: <script found within ARGS:search: <script>alert('CRS Sandbox Release')</script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.718018 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "93"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 15)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]
[Sat Apr 09 22:22:02.718596 2022] [:error] [pid 847584:tid 140613499016960] [client client-ip:49688] [client client-ip] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/usr/share/modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "91"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 15 - SQLI=0,XSS=15,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 15, 0, 0, 0"] [ver "OWASP_CRS/3.3.2"] [tag "event-correlation"] [hostname "example.org"] [uri "/"] [unique_id "YlHq6gKxO9SgyEd0xH9N5gADLgA"]

In the first 3 lines we see that we hit different filters which check for XSS vulnerabilities, more specifically rules 941100, 941110 and 941160 all of them having the tag paranoia-level/1.

Then the fourth line shows that we hit rule 949110 which caused the web server to return the HTTP 403 Forbidden response because the inbound anomaly score, 15, is higher than 5. Then rule 980130 gives us some more information about the scoring: we hit a score of 15 at the paranoia level 1, while rules at the other paranoia levels rules contributed 0 to the total score. We also see the scores for individual types of attack: in this case all 15 points where scored by rules detecting XSS attacks. This is the meaning of the different abbreviations used:

SQLISQL injection
XSScross-site scripting
RFIremote file inclusion
LFIlocal file inclusion
RCEremote code execution
PHPIPHP injection
HTTPHTTP violation
SESSsession fixation

More detailed logs about the traffic hitting the rules can be found in the file /var/log/apache2/modsec_audit.log.

Fixing false positives

First of all, in order to minimize the amount of false positives, you should set the WEBAPPID variable if you are using one of the known web applications for which the Core Rule Set has a default exclusion set. These web applications are currently WordPress, Drupal, Dokuwiki, Nextcloud, Xenforo and cPanel. You can do so by using the <a href="https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecWebAppId">SecWebAppId</a> option in a VirtualHost of Location definition in the Apache configuration. For example if you have a VirtualHost which is used by Nextcloud, set this within the VirtualHost definition:

<Virtualhost nextcloud.example.org>
    ...OTHER OPTIONS HERE...
    <IfModule security2_module>
        SecWebAppId "nextcloud"
    </IfModule>
</VirtualHost>

If you have a WordPress installation in a subdirectory, then add SecWebAppId within Location tags.

<Location /wordpress>
    <IfModule security2_module>
        SecWebAppId "wordpress-mysite"
    </IfModule>
</Location>

If you have multiple WordPress sites, give each of them a unique WEBAPPID which name starts with wordpress. Add a different suffix for every instance so that each one run its in own application namespace in ModSecurity.

If you still encounter false positives, you can completely disable rules by using the configuration directive SecRuleRemoveById. I strongly recommend not disabling rules globally, but limiting its removal to the specific location from which you want them to be removed, for example by putting them with <Location> or <LocationMatch> tags in the Apache configuration. For example:

<LocationMatch ^/wp-admin/(admin-ajax|post)\.php>
    <IfModule security2_module>
        SecRuleRemoveById 941160 941100 941130 932105 932130 932100
    </IfModule>
</LocationMatch>

Pay attention not to disable any of the 949*, 959*, and 980* rules: disabling the 949* and 959* rules would disable all the blocking rules, while disabling the 980* rules would give you less information about what is happening in the logs.

Conclusion

ModSecurity and the Core Rule Set offer an additional security layer for web servers in your defence in depth strategy. I strongly recommend implementing this on your servers because it makes it harder to abuse security vulnerabilities.

Keep an eye on the Core Rule Set blog and Twitter account: sometimes they post new rules for specific new critical vulnerabilities, which can be worthwhile to add to your configuration.

Fixing crackling/popping while playing music in Debian GNU/Linux

I was experiencing crackling/popping sounds while playing music with Rhythmbox in my Debian GNU/Linux Testing (Bookworm) system. The noises start when starting music playback and stop as soon as I stop the playback.

I came around this bug report for Pipewire but I’m currently still using PulseAudio 15 instead. However it contained a comment which led to the solution: speech-dispatcher is known for causing problems of crackling sounds.

Speech-dispatcher is used for speech synthesis often used by blind or visually impaired people. If you don’t use this functionality, you can either disable speech-dispatcher (speechd) or completely uninstall it:

Disable speech-dispatcher by editing /etc/speech-dispatcher/speechd.conf, removing the comment sign # before this line near the end of the file:

DisableAutoSpawn

Then you will have to log out and log in your desktop or you can kill all speech-dispatcher processes manually.

If you want, you can remove speech-dispatcher completely by running this command:

# apt remove speech-dispatcher

Note that this will remove the gnome meta-package, which in itself is harmless, but might lead to new GNOME components not being installed automatically in the future.

This fixed the constant popping crackling noises while playing music completely for me.

Missing video thumbnails in Nautilus in Debian Bullseye

I am using Debian Bullseye and already for a long time I noticed that Nautilus failed to generate thumbnails for certain video files, leading to only a generic video icon instead of a thumbnail.

In the kernel log (dmesg), you will see this error:

qtdemux0:sink[70839]: segfault at 0 ip 0000000000000000 sp 00007f724fe61d18 error 14 in totem-video-thumbnailer[5638abe9b000+3000]
Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.

This is Debian bug #967941: gnome-video-thumbnailer crashes with a segmentation fault when thumbnailing H.264 encoded video files when you have the package libopenblas0-pthread installed.

Available Work-arounds

Remove libopenblas0-pthread

# apt-get remove libopenblas0-pthread

This might not be possible if you need this package for other reasons.

For performance reasons you might also want to install the BLIS BLAS implementation and set it as default. Install the packages libblis3-pthread and libblis64-3-pthread and set them default using the update-alternatives commands from the next work-around.

Switch the default BLAS implementation from OpenBlas to Atlas

If you cannot uninstall libopenblas0-pthread, you can change the default BLAS and LAPACK implementation on your system to a different implementation than OpenBLAS

# update-alternatives – config libblas64.so.3-x86_64-linux-gnu
# update-alternatives – config libblas.so.3-x86_64-linux-gnu
# update-alternatives – config liblapack.so.3-x86_64-linux-gnu

Choose a different implementation than openblas in all 3 cases. Usually Atlas will be installed and available by default. You can also install the packages libblis3-pthread and libblis64-3-pthread and choose the BLIS implementation for the former two and Atlas for the latter.

Disable the sandbox in totem-video-thumbnailer

Totem-video-thumbnailer uses some kind of sandbox limiting how much CPU and memory resources the process can use. This enhances security and prevents the process from using all resources in case something would go wrong. OpenBLAS causes problems in combination with these restrictions. You can disable them in totem-video-thumbnailer by calling it with the -l option. To automatically let GNOME call it with that option, edit the file /usr/share/thumbnailers/totem.thumbnailer and edit the Exec= line so that it looks like this:

Exec=/usr/bin/totem-video-thumbnailer -l -s %s %u %o

If you can, rather use one of two previous work-arounds, because disabling this restrictions could have security implications.

Regenerating the failed thumbnails

Finally after implementing one of the 3 work-arounds, you will need to regenerate the failed thumbnails. GNOME Thumbnail Factory keeps a list of all failed thumbnails, so that it does not retry to generate them over and over again. They are stored in ~/.cache/thumbnails/fail/gnome-thumbnail-factory so you can just delete this directory:

rm -rf ~/.cache/thumbnails/fail/gnome-thumbnail-factory

Now when you browse a directory which contains video files with missing thumbnails with Nautilus, it should generate them automatically.

Using HTTP headers to protect your visitor’s security and privacy

Recently there has been a lot of controversy over Google starting to use Federated Learning of Cohorts (FLoC) in its Chrome browser. This new technique is used to track users without using third party cookies, but has severe privacy implications because it actually makes fingerprinting users easier and can reveal your interests to websites.

To prevent tracking by FLoC and other tracking techniques, there is only one good solution: stop using Google Chrome. The best privacy friendly browser is Firefox, especially if you set it to strict enhanced tracking protection. If you really need to use Chrome, then at least install one of the open source extensions which disable FLoC and Privacy Badger for other tracking protection.

As a website owner, you can also do something to protect your users. You can opt out your website to be included in cohort computation by sending the header Permissions-Policy: interest-cohort=()

This can be easily done for all your websites by modifying your Apache configuration. While at it, you should set some other security and privacy related headers, notably:

  • X-Frame-Options "SAMEORIGIN": this makes sure that browsers will not allow your website to be included in an iframe on another domain.
  • X-Content-Type-Options "nosniff": This will prevent the browser from trying to automatically detect the file type of a downloaded file instead of using the MIME type sent by the server. This can mitigate attacks where a hacker manages to upload a malicious file by giving it a filename which makes it look like a harmless file type which is then served to your visitors.
  • Referrer-Policy "no-referrer-when-downgrade": when a visitor clicks on a link, the browser will only send the referrer when it’s not going from a HTTPS to a HTTP connection. This is fine if your URLs don’t contain any private information. If they do, then consider using strict-origin-when-cross-origin, so that only your domain name instead of the complete URL is sent as referrer if people click on a link leading to an external website, or even same-origin, which prevents any referrer being sent to external sites. You should probably do this for an internal website, web application or wiki, webmail, etc. More information about Referrer-Policy

To set these in Apache in Debian, create a file /etc/apache2/conf-available/security-headers.conf with these contents:

<IfModule mod_headers.c>
   Header always set X-Frame-Options "SAMEORIGIN"
   Header always set X-Content-Type-Options "nosniff"
   Header always set Referrer-Policy "no-referrer-when-downgrade"
   Header always set Permissions-Policy: interest-cohort=()
</IfModule>

Then make sure the mod_headers module is loaded and this file is enabled by running these commands:

# a2enmod headers
# a2enconf security-headers
# systemctl reload apache2

Another important header to set in your SSL virtualhosts is the HSTS header: it ensures that the browser will automatically use HTTPS every time when connecting to the website in the future. Place this in your SSL enabled virtualhost:

<IfModule mod_headers.c>
   Header always set Strict-Transport-Security "max-age=63072000"
</IfModule>

Then you should also add this to your non-SSL virtualhost to redirect all visitors using HTTP to HTTPS:

<IfModule mod_rewrite.c>
   RewriteEngine on
   RewriteCond %{HTTPS} !=on
   RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
</IfModule>

Of course make sure mod_rewrite is enabled if that’s not yet the case:

# a2enmod rewrite
# systemctl reload apache2

You can check your server configuration on securityheaders.com. There you can also find more information about the headers Cross-Origin-Embedder-Policy, Cross-Origin-Opener-Policy and Cross-Origin-Resource-Policy, some other security related headers. Because they require more changes to your website to implement correctly, I’m not discussing them here.

More information

LineageOS 18.1 (Android 11) on OnePlus 3/3T status

Table of contents

Now that LineageOS 18.1 is officially out, it’s time to take a look at how it runs on Oneplus 3/3T phones, just like I did for LineageOS 17.1 on Oneplus 3/3T. At this time I am not yet running LineageOS 18.1 myself so the information here is based on what I read in the LineageOS 18.1 on Oneplus 3/3T thread on XDA-developers forum. Because of this, information here can be incomplete and inaccurate. If you encounter any problems, search in the thread on XDA Developers and post a message there if necessary. This page will be updated as more additional information becomes known.

Questions, problems and solutions

Is it possible to upgrade from LineageOS 17 to 18.1?

Even though this is usually discouraged on XDA and people will often refuse to give you any support if you did not do a clean installation, it is actually possible. The procedure can lead to problems, so make sure you have proper back-ups and know how to use TWRP.

First make a complete back-up of your device using TWRP. I strongly recommend to back up everything, both the system partition files as the system image. While this is both the same, sometimes a restored system only boots successfully with one of them. Definitely also back up data and boot.

After a successful complete backup, connect your phone to your computer, and via the android notification that it’s charging via USB, select that you want to transfer files. Now open your device’s internal storage disk on your computer, and copy the complete backup in TWRP/BACKUPS/ to your computer. I recommend also making a complete back-up of all apps, with an app like OAndBackupX and then copying this back-up to your PC. While connected to your PC, also make a back-up of your photos in the DCIM directory, because these are not included in backups made with TWRP and OAndBackupX.

Download the following files, either using a browser in Android, either using your computer and in that case copy them to your phone’s internal storage.

Now go to TWRP and install Magisk Unnstaller zip. Then choose Wipe – Advanced wipe, and wipe system, boot, cache and dalvik (don’t wipe data or storage: you will loose all data on your phone if you do so). Then go to Install – Install image, and choose the twrp image. Choose recovery and let it flash TWRP. Then reboot your phone back into the new recovery. Then flash LineageOS 18.1 via Install, then wipe cache and dalvik cache, and reboot. If all goes fine, you can boot back in TWRP and flash Magisk.

If all this fails, restore the backups you made.

Which gapps should I use?

You can use MindTheGapps or OpenGapps for LineageOS 18.1. If you choose OpenGapps, use OpenGapps nano. Never use any of the more complete packages, they will cause problems. Install the missing Google applications afterwards from the Play Store.

I cannot find Netflix in the Google Play Store

When you go the Settings in the Play Store app, under Play Protect certification you will see that your device is not certified.

To solve this, flash the latest Magisk zip in TWRP. Then back in Android go to Settings – Apps & Notifications and there go to Play Store App-Info. Go to Storage and there clear the cache and the storage. Now restart your phone. If you go to the Play Store settings, you will see that the message that it is not certified is gone, and you will be able to install Netflix. After clearing the storage, verify that the Play Store settings are OK for you, because these will have been reset.

Android Gadget Hacks: Fix Play Store Uncertified Errors When You Forget to Flash Magisk

My bank app refuses to start because the phone is rooted

Flash Magisk in TWRP. Then Start Magisk Manager and tap on SafetyNet Check. It should return Success and a green check mark for both CtsProfile and basicIntegrity. Then in Magisk’s menu go to Settings and make sure Magisk Hide is enabled. Then in the menu go to Magisk Hide and enable it for your bank app and for Google Play Services. If it does not work yet, try to remove data and cache of your bank app and reboot. Then try to reconfigure your account.

Known bugs in LineageOS 18.1 on OnePlus 3/3T

The phone crashes from time to time

Some users are reporting stability problems: they experience random reboots in LineageOS because of a kernel panic, at a frequency of about once a week. Logs posted on XDA point to a kernel oops in ffs_data_clear, which is a function in the USB gadget code.

https://forum.xda-developers.com/t/rom-official-nightly-lineageos-18-1-for-oneplus-3-3t.4230665/post-85294671

No way to disable a SIM card when using multiple SIMs

https://forum.xda-developers.com/oneplus-3/oneplus-3–3t-cross-device-development/rom-lineageos-17-0-oneplus-3-3t-t3990061/post82770687#post82770687
https://forum.xda-developers.com/oneplus-3/oneplus-3–3t-cross-device-development/rom-lineageos-17-0-oneplus-3-3t-t3990061/post82771459#post82771459

In LineageOS 16.0 it’s possible to disable one of the SIM cards if you have two SIMs installed. In LineageOS 17.1 and 18.1, the only way to disable a SIM is to physically remove the card from the slot.

This problem is not OnePlus 3/3T specific: it is like this in all LineageOS builds for all devices, and is a deliberate choice by the developers, because it was too hard to implement this for Qualcomm devices (“it’s not a bug, it’s a feature”).