All the general media have written articles about Mastodon and the increasing popularity of it (CNN, BBC, Le Monde, Tagesschau, VRTNws,…) fueling the stream of people creating a Mastodon account even more.
For many people this is probably the first time that they realize what the risks are of one huge single commercial platform and now discover the advantages of an independent open network based on free and open-source software.
Mastodon dealing with the growth
In the meantime Mastodon server administrators are scrambling to increase the capacity of their servers. Fortunately also new administrators have stepped in and set up new servers to deal with the growth of the network. Even some organizations, like MIT, have already set up their own Mastodon server. There have been temporary slowdowns and outages of Mastodon servers, but all in all no fundamental problems. It looks like the network can deal with the huge influx and scales well on a technical level.
The question remains whether the social network will be able to prevent the problems that plagued Twitter and which at times made it an unfriendly and hostile environment where fake news and attacks were common. First of all, Mastodon has a few features which should help in preventing it in becoming a toxic environment. First of all, the timeline in Mastodon is purely chronological. Unlike Twitter, messages from people you are not following but which are liked by others, will not be pushed in your timeline by some vague algorithm. Only the people you follow, will determine what you get to see in your timeline, so you have much more control over this than on Twitter. Mastodon does not have a quote feature like Twitter. The Mastodon creator deliberately decided not to implement a quote feature, because it stimulates toxic behaviour. And Mastodon does not prominently show in the timeline how many times a toot has been boosted, marked as favourite and how many answers it got, which reduces the drive to maximize boosts, favourites and answers. By default search engines will not index your messages on Mastodon, making it much harder for people with bad intentions to search your old messages. The usage of Content Warnings is pretty common on Mastodon and helps hiding controversial content from people not interested in them.
Will this be enough to prevent trolls from dominating the platform? I dare to doubt that. Up to now, very little news media and politicians are present on Mastodon. Often messages from these accounts, trigger negative reactions and toxic behaviour. I can imagine things will become more difficult once more politicians and their fan base and news media start posting on Mastodon.
Much will depend of the instance moderators and also of the choices all people on Mastodon make themselves. First of all, every instance has its own set of rules and its up to the moderators to enforce them. For example see the rules for some Mastodon servers: Fosstodon, mastodon.social, mastodon.art,mastodon-belgium.be. Some instances have more strict rules, while others have more vague rules. I think in the future we will see some instances adapting their rules in order to deal with certain problematic situations. People joining Mastodon should properly take into account the rules when choosing an instance: if you want to minimize the risk of trolling, choose an instance which has an explicit zero-tolerance policy against such behaviour. If you are not happy with how your instance deals with annoying behaviour, you can always move to another Mastodon instance.
Then of course the question remains whether moderators will be able to enforce the policy in practice. Will there be enough moderators to do all the work and will they dare to intervene and block people? And what will they do with people who stay in the grey zone, who strictly speaking don’t break the rules but create a negative atmosphere?
Server moderators are only responsible for their own server, so trolling can still happen from other instances. However, the administrator of an instance has the option to completely block other instances. On the web pages with the server rules mentioned above, you can also see which servers are blocked by this instance and for which reason. So there’s again a big responsibility here for the instance admins: will they dare to completely block instances which have a lax policy and are too tolerant against abuse?
But the administrators and moderators will not be able to solve all problems. A huge responsibility falls also on the shoulders of the users themselves. How will people react when politicians start launching controversial ideas on Mastodon? If people start fueling controversial discussions, for example by posting screenshots of controversial ideas, together with indignant remarks, this will only spread the message to others, including to people who don’t want these messages and will only annoying them. People on Mastodon do have the tools to protect them against such annoyances: just like on Twitter, you can mute and block people, and there are also filter options which allow you to hide all posts containing certain words.
It will be up to all the people on Mastodon to use all these options wisely to prevent heated discussions, attacks and annoyances leading to a negative atmosphere like on Twitter.
It’s hard to say how this will end. Twitter has huge problems, but in spite of this, I think that it is too big to fail. At least for now. What’s for sure is that many people now have a real choice: a proprietary platform which is ruled by someone whose impulsiveness causes havoc and which is at times unfriendly and hostile, or an open platform in hands of the community and with stricter moderation and a more friendly atmosphere. Mastodon is growing and it looks like the growth is sustainable, even though currently it is still much smaller than Twitter.
I would not be surprised that somewhere in the future Twitter would implement the ActivityPub protocol and so it would become part of the Fediverse. That way they could give companies access to people who fled to Mastodon. Many companies however have invested a lot in Twitter and use it to promote content of their own websites to the public or for customer support. For this they often use specialized software, often integrating in their own website CMS. Unless these software tools are adapted for Mastodon, it will be hard for such companies to move to Mastodon. So for big brands, Twitter definitely still has an advantage. However the question is whether companies will still have faith in Twitter.
Now that Mastodon and the Fediverse are getting a big influx of new users, I thought it would be worthwhile mentioning that you can follow this blog on the Fediverse. Just search for @frederik on your favourite Fediverse server and you should be able to follow this blog.
If you are not a fan of social media, you can subscribe to the RSS feed of this site: https://blog.frehi.be/feed/. You will need an RSS aggregator on your system, such as Thunderbird, NewsFlash (GNOME), Akregator (KDE), Feeder (Android), NetNewsWire (macOS, iOS). There are also web-based solutions available. Web-based feed aggregators which you can install on your own server are TT-RSS, Nextcloud News. Feedly is a popular proprietary web-based solution, but as usual the problem is that this limits your privacy, and makes you depend on a commercial party.
In a previous article, I discussed how to set up ModSecurity with the Core Rule Set on Debian. This can be considered as a first line of defense against malicious HTTP traffic. In a defense in depth strategy of course we want to add additional layers of protection to your web servers. One such layer is Snuffleupagus. Snuffleupagus is a PHP module which protects your web applications against various attacks. Some of the hardening features it offers are encryption of cookies, disabling XML External Entity (XXE) processing, a white or blacklist for the functions which can be used in the eval() function and the possibility to selectively disable PHP functions with specific arguments (virtual-patching).
Installing Snuffleupagus on Debian
Unfortunately there is no package for Snuffleupagus included in Debian, but it is not too difficult to build one yourself:
$ apt install php-dev
$ mkdir snuffleupagus
$ cd snuffleupagus
$ git clone https://github.com/jvoisin/snuffleupagus
$ cd snuffleupagus
$ make debian
This will build the latest development code from the master branch. If you want to build the latest stable release, before running make debian, use these commands to view all tags and to checkout the latest table tag, which in this case was v0.8.2:
$ git tag
$ git checkout v0.8.2
If all went well, you should now have a file snuffleupagus_0.8.2_amd64.deb in the above directory, which you can install:
$ cd ..
$ apt install ./snuffleupagus_0.8.2_amd64.deb
First we take the example configuration file and put it in PHP’s configuration directory. For example for PHP 7.4:
Snuffleupagus can run rules in simulation mode. In this mode, the rule will not block further execution of the PHP file, but will just output a warning message in your log. Unfortunately there is no global simulation mode, but it has to be set per rule. You can run a rule in simulation mode by appending .simulation() to it. For example to run INI protection in simulation mode:
To prevent PHP applications from modifying php.ini settings, you can set this in snuffleupagus.rules:
The following configuration options sets the SameSite attribute to Lax on session cookies, which offers protection against CSFR on this cookie. We enforce setting the secure option on cookies, which instructs the web browser to only send them over an encrypted HTTPS connection and also enable encryption of the content of the session on the server. The encryption key being used is derived of the value of the global secret key you have set, the client’s user agent and the environment variable SSL_SESSION_ID.
eval() is used to evaluate PHP content, for example in a variable. This is very dangerous if the PHP code to be evaluated can contain user provided data. Therefore it is strongly recommended that you create a whitelist of functions which can be called by code evaluated by eval().
Start by putting this in snuffleupagus.rules and restart PHP:
Then test your websites and see which errors you get in the logs, and add them separated by commas to the eval_whitelist.list(). After that you need to remove .simulation() and restart PHP in order to activate this protection. For example
You can also use a blacklist, which only blocks certain functions. For example:
The read_only_exec() feature of Snuffleupagus will prevent PHP from execution of PHP files on which the PHP process has write permissions. This will block any attacks where an attacker manages to upload a malicious PHP file via a bug in your website, and then attempts to execute this malicious PHP script.
It is a good practice to let your PHP scripts be owned by a different user than the PHP user, and give PHP only read-only permissions on your PHP files.
To test this feature, add this to snuffleupagus.rules:
If you are sure all goes well, enable it:
One of the main features of Snuffleupagus is virtual patching. Thjs feature will disable functions, depending on the parameters or and values they are given. The example rules file contains a good set of generic rules which blocks all kinds of dangerous behaviour. You might need to fine-tune the rules if your PHP applications hits certain rules.
These two rules will only allow the include_once function to include files which file name are ending with inc, phtml or php. All other include_once calls will be dropped.
Using generate-rules.php to automatically site-specific hardening rules
In the scripts subdirectoy of the Snuffleupagus source tree, there is a file named <a href="/home/frederik/software/snuffleupagus/scripts/generate_rules.php">generate_rules.</a><a href="https://raw.githubusercontent.com/jvoisin/snuffleupagus/master/scripts/generate_rules.php">php</a>. You can run this script from the command line, giving it a path to a directory with PHP files, and it will automatically generate rules which specifically allow all needed dangerous function calls, and then disable them globally. For example to generate rules for the /usr/share/tt-rss/www and /var/www directories:
The first two rules allow these two files to call function_exists and the last rule drops all requests to function_exists from any other files. Note that the first two rules limit the rules not only to the specified file name, but also define the SHA256 of the file. This way, if the file is changed, the function call will be dropped. This is the safest way, but it can be annoying if the files are often or automatically updated because it will break the site. In this case, you can call generate_rules.php with the --without-hash option:
After you have generated the rules, you will have to add them to your snuffleupagus.rules file and restart PHP-FPM.
File Upload protection
The default Snuffleupagus rule file contains 2 rule which will block any attempts uploading a html or PHP file. However, I noticed that they were not working with PHP 7.4 and these rules would cause this error message:
PHP Warning: [snuffleupagus][0.0.0.0][config][log] It seems that you are filtering on a parameter 'destination' of the function 'move_uploaded_file', but the parameter does not exists. in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 0 parameter's name: 'path' in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 1 parameter's name: 'new_path' in /var/www/html/foobar.php on line 15'
The snuffleupagus rules use the parameter destination for the move_uploaded_file instead of the parameter new_path. You will have to change the rules like this:
After restarting PHP-FPM, always check the logs to see whether snuffleupagus does not give any warning or messages for example because of a syntax error in your configuration:
# journalctl -u php7.4-fpm -n 50
By default Snuffleupagus logs via PHP. Then if you are using Apache with PHP-FPM, you will find Snuffleupagus logs, just like any PHP warnings and errors in the Apache error_log, for example /var/log/apache/error.log. If you encounter any problems with your website, go check this log to see what is wrong.
Snuffleupagus can also be configured to log via syslog, and actually even recommends this, because PHP’s logging system can be manipulated at runtime by malicious scripts. To log via syslog, add this to snuffleupagus.rules:
I give a few examples of errors you can encounter in the logs and how to fix them:
[snuffleupagus][0.0.0.0][xxe][log] A call to libxml_disable_entity_loader was tried and nopped in /usr/share/tt-rss/www/include/functions.php on line 22
tt-rss calls the function libxml_disable_entity_loader but this is blocked by the XXE protection. Commenting this in snuffleupagus.rules should fix this:
[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'ini_set', because its argument '$varname' content (display_errors) matched a rule in /usr/share/tt-rss/www/include/functions.php on line 37'
Modifying the PHP INI option display_errors is not allowed by this rule:
[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'function_exists', because its argument '$function_name' content (exec) matched a rule in /var/www/wordpress/wp-content/plugins/webp-express/vendor/rosell-dk/exec-with-fallback/src/ExecWithFallback.php on line 35', referer: wp-admin/media-new.php
Last year I supported the Solo V2 Kickstarter camaign. Solo is a completely open source FIDO2 security key. You can use it for Two-Factor Authentication (2FA) on web sites, for protecting your private SSH keys and other things. The Solo2 is similar to keys such as the Yubikey from Yubico, the Google Titan Security Key, the Kensington Verimark or Nitrokey. Because all these keys implement the standards of the FIDO2 project, many of the examples here work with these keys too.
The Kickstarter campagin has ended, however now you can buy Solo V2 security keys via their Indiegogo campaign. If you decide to buy a security key, then I strongly recommend buying at least two of them so that you can use the second key as a back-up key in case the first key breaks or gets lost.
There is also little documentation, which can make it a bit difficult to get started if you are new to FIDO2 security keys. That’s why I decided to create this guide, to serve as a tutorial explaining how to use the Solo2.
For basic usage of the key, you actually do not need to install any software. However there are some utilities available which allow you to update the firmware, set a PIN, view all credentials stored on the key, etc. First of all, we will install the solo2 CLI. It’s not yet packaged in Debian, so we need to download it from Github. I check the sha256sum to ensure I get the right files. This utility written in Rust does not support all functionality yet and for that reason I also install the Python based Solo1 CLI, which is packaged in Debian. The fido2-tools package finally contains some utilities which work on all FIDO2 keys.
When authenticating to websites or doing other operations, you will be asked to tap or touch the security key. The Solo2, unlike the Solo1 or Yubikey, does not have a physical button which needs to be depressed, but has 3 touch areas. These are the 3 gold coloured areas at both sides and at the back of the key. You do not need to press them, gently touching one of them is enough. In practice, I had most success touching the two touch zones at both sides of the key simultaneously with 2 fingers.
Updating the Solo V2 firmware version
You can check which version of the firmware is currently installed on your key with this command:
$ solo2 app admin version
At the moment of writing, the most recent version is 1:20200101.9 2:20220822.0.
To update the firmware version, run this command:
$ solo2 update
Setting a PIN on the Solo V2 key
I strongly recommend setting up a PIN on your FIDO2 key. It will be required to do any administrative tasks on your key, such as adding or removing credentials such as SSH keys,
So in my case it’s /dev/hidraw4. Then change the PIN like this:
$ fido2-token -C /dev/hidraw4
Do not forget your PIN, otherwise you cannot use your key any more to authenticate to registered sites!
In case you forgot your FIDO2 PIN, you will need to completely reset your key. This will erase all keys and generate new ones, so you will need to have an alternative way to authenticate to websites where you registered this key.
$ solo key reset
FIDO2 Two-Factor Authentication
Usually you go the security settings on the website and there you can enable 2FA. For some sites, you will be required to set up TOTP first before you can register a security key. So make sure you have a TOTP application such as FreeOTP+ for Android or Raivo OTP on iOS. TOTP is then a back-up method for 2FA in case you loose access to your key. If you have multiple FIDO2 keys, don’t forget to register them all.
A side note: don’t use SMS as a second factor for authentication. SMS 2FA is insecure because these messages are transferred in clear text and there are various ways they can be intercepted.
2FA with the Solo2 on Android
You can connect the Solo2 to your Android device by USB, or you can use NFC. When a web application tries to authenticate your key, you will get a pop-up message where you can choose whether you want to connect it via USB or use NFC. In the case of USB, connect your key to the USB port and tap it, just like you would do on your PC. If you chose NFC, just bring your Solo2 key to the back of your phone and it should authenticate.
This all works fine in Chromium based browsers, however I was not able to successfully authenticate with the Solo2 in Firefox. I managed to get it working with Firefox Nightly though. You will need to go to about:config and set security.webauthn.ctap2 and security.webauth.webauthn_enable_usb_token both to true in order to get it working.
Enabling 2FA on well-know websites
Go to https://myaccount.google.com/ and in the left menu click on Security. Under Signing in to Google click on 2-Step Verification. There click on Enable two-factor authentication.In the wizard that appears, you will have to click on Security Key and follow to instructions to add your key.
In the right top corner, click on your avatar and choose Settings. Then in the left menu click on Password & Authentication where you can enable Two-Factor Authentication. You will have to set up TOTP first, and after that, you can register your security key.
In the right top corner, click on your avatar and choose Preferences. Then in the left menu click on Account where you can enable Two-Factor Authentication. You will have to register a Two-Factor Authenticator (TOTP) first, and after that, you can register your WebAuthn devices.
Click on the Preferences icon then choose Account – Two-Factor Auth. You will need to set up TOTP first, and after that you can add a security key.
Click on your avator in the top right corner and choose Settings. Then choose Security in the left menu and there you can add Webauthn devices.
Microsoft personal account
Make sure you are using the newest firmware because this is not working with older firmware versions.
You need to go with a Chromium based browser to https://account.microsoft.com/ (Firefox does not work at the moment). There click on Security – Security Dashboard – Advanced security options – Add a new way to sign in or verify – Use a security key.
Microsoft Azure Directory account
If you have a Microsoft Directory Azure account, for example if you are using Office 365 in your organization, then it might be possible to use your Solo2, however this depends on the settings made by your administrator. if Enforce key restrictions is enabled, certain keys can be blocked, or only specific keys are allowed. Also the option Enforce attestation needs to be disabled, because otherwise only keys which have been tested by Microsoft are allowed. Unfortunately Solo keys have not been validated by Microsoft (also Google’s Titan security keys are in this case). Note that this attestation does not include an evaluation of the security of the key.
It appears that at the moment of writing LinkedIn does not support 2FA with FIDO2. You can set up TOTP though, which I recommend doing. Click on Me in the top menu and choose Settings & Privacy. Then in the left menu choose Sign in & security and click on Two-Step verification.
To enable FIDO2 two-factor authentication in WordPress, install the plugins two-factor and two-factor-provider-webauthn. Enable both modules and then in the WordPress administration menu go to Settings – TwoFactor WebAuthn. Use the option: Disable old U2F provider. the two-factor plugin includes U2F by default, but this is not supported any more by Chromium based browsers, so you want to use the more modern webauthn instead. Then you can set up 2FA in the menu Users – Profile: enable WebAuthn . Then under Security Keys (WebAuthn) click on Register New Key, tap your key and give it a unique name. Do this for both your security keys.
If you have set up Modsecurity with the Core Rule Set, you will end up with a HTTP 403 Forbidden error when trying to register your key or try to authenticate with it. Create /etc/modsecurity/99-wordpress-webauthn.conf with this content:
and reload your Apache configuration. It should now work.
What if it does not work?
If registering your key or authenticating with your key fails on a website, try with a Chromium based browser. Firefox does not support CTAP2 yet, and this can cause trouble on sites which require verification of a PIN. Firefox has CTAP2 support now, but it’s disabled by default. Make sure you use the latest version of Firefox (109 at the time of writing) and activate CTAP2 support by going to about:config and setting security.webauthn.ctap2 to true.
To use your Solo2 key for OpenSSH authentication, you will at least version 8.2p1 on both server and client. OpenSSH 8.3p1 adds support for discoverable credentials or resident keys: with discoverable credentials, the FIDO2 security key itself is enough to do SSH public key authentication. This has a slight security risk though if people get access to your Solo2 key because now the only protection is the PIN you have set on the key. Non-discoverable keys don’t have this security risk, because you also need the private key stored on your computer to authenticate.
SSHD configuration for FIDO2 keys
As written before, you need at least version 8.2p1 or 8.3p1 of OpenSSH. The default settings as provided by Debian should be OK, but I strongly recommend to add this option to sshd_config if you only use FIDO2 keys for interactive login:
I prefer doing this by creating a file /etc/ssh/sshd_config.d/fido2.conf with this line.
This options ensures that only keys which require a PIN can be used, at least adding some protection against theft of a FIDO2 key which contains discoverable credentials.
You can also add the option touch-required to PubkeyAuthOptions in order to require touching the key when authenticating. This will make it impossible to authenticate with keys which were created with the no-touch-required option.
Setting up FIDO2 credentials for SSH
To generate credentials for SSH with your FIDO2, you basically use this command
$ ssh-keygen -t ed25519-sk
There are diffferent options available which you can add:
-O resident: You want to create discoverable credentials.
-O no-touch-required: You want to disable the requirement of touching the key for authenticating.
-O verify-required: You require that the PIN is entered when authenticating. I strongly recommend this option.
-O application=ssh:SomeUniqueName: In case you want to store different SSH keys on your Solo2, you will have to give each of them a different application name starting with ssh:
-f ~/.ssh/id_ed25519_sk_solo2_blue: If you use multiple FIDO2 keys, you may want to store the key in a unique file for every FIDO2 key. Replace the file name of this example by the name of your choice.
You can verify that the credentials are correctly stored on your Solo2 using this command:
$ solo key credential ls
In case you would want to remove the credentials stored on your key, you can do so by using this command:
$ solo key credential rm CREDENTIALID
Replace CREDENTIALID by the value you found with the previous command.
After creating the key, you need to copy the public key to the authorized_keys file on your server. You can use ssh-copy-id for that:
Of course use the correct file name for the public key.
If you used the option no-touch-required when generating the key, you will have to edit the ~/.ssh/authorized_keys file on your server so that this options precedes the key. For example if authorized_keys contains this:
You will be asked to enter the PIN of your key and to touch it, depending on the options you used when creating the key. I add -o IdentitiesOnly=yes because otherwise ssh will first try to authenticate using the keys loaded in your SSH agent. With this option we enforce it to use only the private key we have specified with the -i parameter.
You can make this default by editing ~/.ssh/config, so that you don’t need to repeat the -i and -o parameters every time when connecting:
Importing discoverable credentials on another system
When you use discoverable credentials, all information needed for authentication is stored on the key itself, in contrast to non-discoverable credentials, where part of that information is also stored in the private key file on the computer. For this reason, with discoverable credentials, it is easy to import them on any computer.
$ cd ~/.ssh/
$ ssh-keygen -K
The public and private key will be written in the .ssh directory, and then you can authenticate again using the ssh -o IdentiesOnly=yes -i command just like on the system where you generated the key.
Troubleshooting FIDO2 SSH authentication
On the server check the sshd logs, which can be found in /var/log/auth.log or in the ssh journal:
# journalctl -u ssh
Successful authentication with your FIDO2 key, should be logged like this:
Accepted publickey for username from xxx.xxx.xxx.xxx port zzzzzz ssh2: ED25519-SK SHA256:....
Notice the ED25519-SK part which indicates that the credentials on your FIDO2 key were used.
If you see this:
error: public key ED25519-SK SHA256:... signature for username from xxx.xxx.xxx.xxx port zzzzz rejected: user presence (authenticator touch) requirement not met
This means that you have created a key with the no-touch-required options not set. Try adding no-touch-required to the authorized_keys on the server, as noted above, at least if your server does not have PubkeyAuthOptions touch-required set.
On the client-side, you can add the -v parameter to debug what happens:
Here is a short reference sheet for resizing block devices and file systems.
Resizing block devices
Add 100 GiB to the logical volume with name logicalvolume in volume group volumegroup.
# lvextend -L +100G volumegroup/logicalvolume
QEMU block device
If you resized a block device which is used to store a virtual disk for a QEMU VM, you will need to expand the virtual disk itself. First we need to know the name of the virtual disk. If you are managing your QEMU VMs via libvirt, you can use this command to see all virtual disks:
# virsh qemu-monitor-command VMname --hmp "info block"
drive-virtio-disk0 (#block108): /dev/vm/web-www (raw)
Attached to: /machine/peripheral/virtio-disk0/virtio-backend
Cache mode: writeback, direct
drive-virtio-disk1 (#block302): /dev/vm/web-logs (raw)
Attached to: /machine/peripheral/virtio-disk1/virtio-backend
Cache mode: writeback, direct
Then if you resized the logical volume /dev/vm/web-www from 100 to 200 GiB using the command mentioned before, you can resize the corresponding QEMU virtual disk drive-virtio-disk0 using this command:
I wanted to set up Wireguard on a VPS, not only to tunnel IPv4 traffic, but also allowing me to tunnel IPv6 traffic. As this is IPv6 of course I preferred not to use NAT, but to assign a public IP address to the client. I read some documentation and blog posts, but I struggled getting it to work. Most tutorials I found on the Internet, create a separate IPv6 subnet for the VPN but I could not get this to work. For some reason, IPv6 traffic successfully went through the VPN tunnel and then exited the VPN gateway, but then any response never reached my VPN gateway and hence also not the client.
I decided to try another way: using an NDP proxy. NDP or the Neighbour Discovery Protocol, is similar to ARP which is used in IPv4. Using this protocol, network devices can discover where on the network a certain IP is located. By letting the VPN gateway answer NDP requests for the VPN client, the gateway would correctly send back all responses to the VPN gateway, which then forwards it to the VPN clients.
Configuring the network on the VPN gateway
I use systemd-networkd to set up the network. It’s the most modern way of network configuration and works the same on all distributions using systemd, but of course you can make the same settings in /etc/network/interfaces or whatever your distribution uses. Of course when making changes to a remote server, make sure you can access a console without needing a working network connection on the server, in case things go wrong and the network connection breaks.
On my VPN server, the public network interface is named ens192 (use the command $ ip addr to find it on your system). My public IPv4 address is www.xxx.yyy.zzz with subnet 255.255.255.0 and gateway ww.xx.yy.1. I have the 64 bit IPV6 prefix aaaa:bbbb:cccc:dddd and the IPv6 gateway is fe80::1.
In the [Interface] section make sure to use the same IP addresses as the ones you have set in the corresponding [Peer] section on the VPN gateway. Set the DNS name (or IP address) of the VPN gateway as Endpoint in the [Peer] section. The hostname’s DNS entry can have both an A and AAAA record. You can replace your DNS servers by your preferred ones. You can also consider running your own DNS server on the VPN gateway.
Make sure that all wg*.conf files on client and server are only readable by root, because they contain private keys.
For IPv4 we configure Shorewall to use NAT to provide Internet access to the VPN clients.
net NET_IF dhcp,tcpflags,logmartians,nosmurfs,sourceroute=0,routefilter,routeback,physical=ens192
vpn wg0 tcpflags,logmartians,nosmurfs,sourceroute=0,optional,routefilter,routeback
$FW net ACCEPT
vpn net ACCEPT
vpn $FW ACCEPT
net all DROP $LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all all REJECT $LOG_LEVEL
# Drop packets in the INVALID state
Invalid(DROP) net $FW tcp
# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..
Ping(DROP) net $FW
SSH(ACCEPT) net $FW
ACCEPT net $FW udp 51820
MASQUERADE 192.168.7.0/24 NET_IF
Compile and load the rules and enable Shorewall permanently:
Then in order to make sure that the gateway knows that the VPN client aaa:bbb:cccc:dddd::2 is reachable via the VPN gateway, we need to set up NDP proxying. The Neighbor Discovery Protocol is similar to ARP in IPv6.
In a previous version of this guide, I configured NDP proxying in Shorewall6. However, we can directly set this up with systemd-networkd, so this will also work if you don’t use Shorewall6 but another firewall like Firewalld. Furthermore I also experienced problems with NDP proxy settings being lost after some time, requiring a restart of Shorewall6 to make the IPv6 connection over Wireguard work again. I hope this will be fixed by settings this up in systemd-networkd.
Edit again the file /etc/systemd/network/internet.net and in the [NETWORK] section add this
Some time ago, I received a new laptop, the HP Elitebook 845 G8. This is a 14″ laptop with an AMD CPU of the Renoir family, in my case an AMD Ryzen 7 PRO 5850U. As always, I run Debian GNU/Linux testing (currently Bookworm) on it. In this post, I will explain how to get all hardware working. This guide probably also works for other G8 Elitebooks, such as the Elitebook 835 G8 and Elitebook 855 G8, because they are all quite similar.
I used a USB disk to boot the installer and a USB-C dock with an Ethernet interface to do a network installation. If you use the Debian installer with non-free firmware, you can also do the installation over wifi. I have not tried the current stable release Debian 11 Bullseye on this system. For best compatibility I strongly recommend testing because it has a more recent kernel and drivers.
Now we need to write it to a USB disk. Make sure there is no data on the drive you want to keep, because this process will completely wipe the disk.
To write the ISO image to a USB disk, Windows users can use the application Rufus, MacOS users can use Balena Etcher. If you are using Linux, you can dd the ISO image on your USB disk, or use a GUI like Fedora Media Writer.
Reboot the system and press F10 when the HP logo appears to load the BIOS/UEFI setup. Go to the Advanced page and select Boot Options. There make sure that USB Storage Boot is enabled. If you want to work with custom kernels, it can be handy to disable Secure Boot in Security – Secure Boot Configuration, but it’s not needed to install and use Debian.
Save the changes you made (if any) and reboot the system and press F9 at the HP logo to get the boot menu. In the boot menu, select your USB drive to start the Debian installer.
Enabling non-free repositories
We will need to configure the non-free repositories for apt because we need several firmware packages from non-free. Edit /etc/apt/sources.list and check whether any deb line has maincontrib and non-free at the end. If not add it, and then run
# apt update
Updating the BIOS/UEFI firmware
If you have a Windows installation, you can update the firmware from there, even before you install Linux. But you can also update the firmware without Windows. Follow the instructions in that blog post. It’s important to do this, not only because this gives you essential security fixes, but also bug fixes, some of which specifically for Linux compatibility.
Updating CPU firmware
Install the package amd64-microcode to ensure your AMD CPU is always running the latest microcode, which includes security fixes:
# apt-get install amd64-microcode
Flashing other firmware
The fwupd utility can download and install firmware from the LVFS. The firmware of the fingerprint reader of the Elitebook 845 G8, can be updated like this, and may be necessary to get the fingerprint reader working in Linux. First make sure fwupd is installed:
# apt install fwupd fwupd-amd64-signed
Now update all available firemware:
# fwupdtool update
If you have a HP USB-C Dock G5, then new firmware is also available in the LVFS, but it’s in the testing repository. To enable this repository, run this command:
This laptop can also be delivered with an Intel AX200 Wi-Fi 6 adapter (which is actually a better option than this one from Realtek). If you have this one, you will need to install the firmware-iwlwifi package instead.
lsusb identifies this smartcard reader as an Alcor AU9540:
Bus 005 Device 004: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader
Note that it only sees the smartcard reader when a card has been inserted.
You will need pscd with the CCID driver to use this smartcard reader:
# apt install pscsd
The fingerprint reader can be seen like this in lsusb:
Bus 003 Device 003: ID 06cb:00df Synaptics, Inc.
Make sure you have installed all firmware updates with fwupd and then you need to install these packages:
# apt install fprintd libpam-fprintd
In GNOME, under Settings – Users you can enable login on fingerprint and add your fingerprints.
04:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 01)<br />04:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
Bus 001 Device 002: ID 0408:5348 Quanta Computer, Inc. HP HD Camera
The webcam works out of the box. Many applications will see the IR camera as a second camera.
Bus 003 Device 002: ID 0bda:b00c Realtek Semiconductor Corp. 802.11ac WLAN Adapter
If you use lsusb -v you will see that this is actually the Bluetooth Radio adapter. It is combined with the wifi adapter, hence the confusion.
HP does not support S3 (traditional suspend-to-ram/standby) in its recent Elitebooks any more, but instead uses s0ix (s2idle/suspend-to-idle/modern standby). S2idle support for AMD CPU’s was only added in Linux 5.11 with the amd_pmc driver. I recommend a very recent kernel, because later kernel versions had bug fixes in this regard too. However suspend regressed in stable update 5.17.3 (and others), a bug which was fixed in 5.17.5. I’m using a custom-built 5.17.5 kernel, but a fixed kernel will appear soon in Debian.
If you have HP Drivelock enabled, then your system will fail to resume. Drivelock is a security feature which can be set up in the BIOS and requires you to enter a password when starting up the system in order to access the contents of the disk. When trying to resume the system, fans start running, the keyboard backlight reacts to key presses, but the screen remains blank, nothing is written to logs and also network does not come up. Apparently this is a bug in HP’s BIOS/UEFI firmware which can be worked around by adding iommu=pt to the kernel command line. To do so, edit /etc/default/grub and add this to the variable GRUB_CMDLINE_LINUX_DEFAULT. For example:
Then update the GRUB configuration:
Install isenkram to help install drivers when plugging in hardware
Isenkram is a utitliy which will show a message when you connect hardware to your system and extra software or firmware is available for that hardware.
# apt install isenkram
Enabling trimming of the NVME SSD
Enable the fstrim timer to make sure the SSD is trimmed on regular intervals:
If you are using Linux 6.3 you don’t need to do this, but you will have to add to the the GRUB_CMDLINE_LINUX_DEFAULT options in /etc/default/grub.conf:
and run update-grub.
Set up TLP
TLP is a tool which optimizes power consumption of your system in order to increase battery time. TLP also has an options Radio Device Wizard, which I will use here to automatically disable wifi when the system is connected via an Ethernet cable.
# apt install tlp tlp-rdw
Configure the Radio Device Wizard by creating the file /etc/tlp.d/10-tlp-rdw.conf:
# tlp-rdw - Parameters for the radio device wizard
# Possible devices: bluetooth, wifi, wwan.
# Separate multiple radio devices with spaces.
# Default: <none> (for all parameters below)
# Radio devices to enable on disconnect.
# Radio devices to enable/disable when docked.
# Radio devices to enable/disable when undocked.
To enable best ASPM power saving features when on battery, create /etc/tlp.d/20-aspm.conf:
Submitting your system to the Linux Hardware database
The Linux Hardware database is a useful tool where users searching for hardware, can check the compatibility of systems with Linux. I recommend running this tool on all your Linux systems. After submission, you will get a link where you can view the data and indicate whether all hardware works and which work-arounds you had to apply. Click on the Review button on the page to do so.
# apt install hw-probe
# hwprobe --all --upload
Actually Linux compatibility of the HP Elitebook 845 is actually in good shape. It’s not perfect, but all hardware can be made to work. On distros like Ubuntu, which install non-free firmware by default, it should even be easier to make everything work. Still HP lags behind Dell and Lenovo in Linux support, because they don’t make it possible to flash the BIOS/UEFI firmware using fwupd, while all recent Dell and Lenovo business laptops have their firmware available in the LVFS. Also the problem that iommu=pt needs to be used to successfully resume the laptop when Drivelock is enabled, is a problem that HP should address in a BIOS update.
Fortunately it is possible to install firmware updates without having Windows installed, but it’s a bit more manual work. Here I will explain how I managed to update my Elitebook 845 G8, but this should work for most recent HP laptops and desktops.
If you don’t know the exact model of your system, use this command (as root):
# /usr/sbin/dmidecode | grep "Product Name"
On the HP downloads page, you will probably get a mssage that it was unable to find drivers for your product. Click on the link Choose a different OS and select any Windows 10 version. Now you will find the latest firmware for your system under BIOS-System Firmware. Click on Download. It will try to let you install a Windows Download and Install Assistant: click on No thanks, I will download and install manually to directly get the firmware file, which should have a name similar to sp138978.exe.
Once you have downloaded this file, we need to extract it. You can use 7-Zip for that. Make sure it is installed on your system:
# apt-install 7zip
Then we will make a directory and extract the firmware package in it:
$ mkdir /tmp/hpfirmware
$ cd /tmp/hpfirmware
$ 7zz x ~/Downloads/sp138978.exe
You will see many files in the /tmp/hpfirmware directory, such as the History.txt file which you can read if you want to know which changes and bug fixes this update brings. The firmware itself is stored in a file with the extension .bin, in my case it’s named T82_01082000.bin.
To install this firmware, we have to copy it to a directory HP/DEVFW/firmware.bin in your EFI directory (this assumes that you are booting your system in UEFI mode and not in legacy BIOS mode). So as root run these commands from the /tmp/hpfirmware directory:
Now reboot your system. In GRUB’s menu, choose UEFI Firmware Settings. You will get HP’s Startup Menu of which the last menu item is Update System and Supported Device Firmware. This should now automatically install the firmware update. This takes several minutes , and you screen will go blank during some time: don’t panic and let it run.
A web application firewall (WAF) filters HTTP traffic. By integrating this in your web server, you can make sure potentially dangerous requests are blocked before they arrive to your web application or sensitive data leaks out of your web server. This way you add an extra defensive layer potentially offering extra protection against zero-day vulnerabilities in your web server or web applications. In this blog post, I give a tutorial how to install and configure ModSecurity web application firewall and the Core Rule Set on Debian. With some minor adaptions you can also use this guide for setting up ModSecurity on Ubuntu or other distributions.
ModSecurity is the most well-known open source web application firewall. The future of ModSecurity does not look too bright but fortunately with Coraza WAF an alternative which is completely compatible with ModSecurity is in development. At this moment Coraza only integrates with the Caddy web server, and does not have a connector for Apache or NGinx so for that reason it is currently not yet usable as a replacement for ModSecurity.
While ModSecurity provides the framework for filtering HTTP traffic, you also need rules which define what to bloc and that’s where the Core Rule Set (CRS) comes in. CRS is a set of generic rules winch offer protection to a various range of common attacks via HTTP, such as SQL injection, code injection and cross-site scripting (XSS) attacks.
Install ModSecurity and the Core Rule Set on Debian
I install the Apache module for ModSecurity, the geoip-database, which can be used for blocking all requests from certain countries, and modsecurity-crs, which contains the Core Rule Set. I take this package from testing, because it has a newer version (version 3.3.2 at the time of writing). There is no risk in taking this package from testing, because it only contains the rules and does not depend on any other packages from testing/unstable. If you prefer faster updates, you can also use unstable.
Now edit /etc/modsecurity/modsecurity.conf. I highlight some of the options:
#SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
#"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"
The SecRuleEngine option controls whether rules should be processed. If set to Off, you completely disable all rules, with On you enable them and it will block malicious actions. If set to DetectionOnly, ModSecurity will only log potential malicious activity flagged by your rules, but will not block them. DetectionOnly can be useful for temporary trying out the rules in order to find false positives before you really start blocking potential malicious activity.
The SecAuditLog option defines a file which contains audit logs. This file will contain detailed logs about every request triggering a ModSecurity rule.
Set SecStatusEngine to Off to prevent ModSecurity sending version information back its developers.
After changing any configuration related to ModSecurity or the Core Rule Set, reload your Apache web server:
# systemctl reload apache2
Configuring the Core Rule Set
The Core Rule Set can be configured via the file /etc/modsecurity/crs/crs-setup.conf.
By default the Core Rule Set is using anomaly scoring mode. This means that individual rules add to a so called anomaly score, which at the end is evaluated. If the anomaly score exceeds a certain threshold, then the traffic is blocked. You can read more about this configuration in crs-setup.conf but the default configuration should be fine for most people.
Setting the paranoia level
The paranoia level is a number from 1 to 4 which determines which rules are active and contribute to the anomaly scoring. The higher the paranoia level, the more rules are activated and hence the more aggressive the Core Rule Set is, offering more protection but potentially also causing more false positives. By default the paranoia level is set to 1. If you work with sensitive data, it is recommended to increase the paranoia level.
The executing paranoia level defines the rules which will be executed but their score will not be added to the anomaly scoring. When HTTP traffic hits rules of the executing paranoia level, this traffic will only be logged but not be blocked. It is a especially useful to prepare for increasing the paranoia level and finding false positives on this higher level, without causing any disruption for your users.
To set the paranoia level to 1 and the executing paranoia level to 2, make sure you have these rules set in crs-setup.conf:
Once you have fixed all false positives, you can raise the paranoia level to 2 to increase security.
Defining the allowed HTTP methods
By default the Core Rule Set only allows the GET, HEAD, POST and OPTIONS HTTP methods. For many standard sites this will be enough but if your web applications also use restful APIs or WebDAV, then you will need to add the required methods. Change rule 900200, and add the HTTP methods mentioned in the comments in crs-setup.conf.
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'"
Disallowing old HTTP versions
There is a rule which determines which HTTP versions you allow in HTTP requests. I uncomment it and modify it to only allow HTTP versions 1.1 and 2.0. Legitimate browsers and bots always use one of these modern HTTP versions and older versions usually are a sign of malicious activity.
Personally I’m not a fan of completely blocking all traffic from a whole country, because you will also block legitimate visitors to your site, but in case you want to this, you can configure this in crs-setup.conf:
Add the two-letter country codes you want to block to the last line (before the two quotes), multiple country codes separated by a space.
Make sure you have the package geoip-database installed.
Core Rule Set Exclusion rules for well-known web applications
The Core Rule Set contains some rule exclusions for some well-known web applications like WordPress, Drupal and NextCloud which reduces the number of false positives. I add the following section to crs-setup.conf which will allow me to enable the exclusions in the Apache configuration by setting the WEBAPPID variable in the Apache configuration whenever I need them.
Adding rules for Log4Shell and Spring4Shell detection
At the end of 2021 a critical vulnerability CVE-2021-44228, named Log4Shell, was detected in Log4j, which allows remote attackers to run code on a server with the vulnerable Log4j version. While the Core Rule Set offered some mitigation of this vulnerability out of the box, this protection was not complete. New improved detection rules against Log4Shell were developed. Because of the severity of this bug and the fact that it’s being exploited in the wild, I strongly recommend adding this protection manually when using ModSecurity version 3.3.2 (or older). Newer, not yet released versions, should have complete protection out of the box.
First modify /etc/apache2/mods-enabled/security2.conf so that it looks like this:
# Default Debian dir for modsecurity's persistent data
# Include all the *.conf files in /etc/modsecurity.
# Keeping your local configuration in that directory
# will allow for an easy upgrade of THIS file and
# make your life easier
# Include OWASP ModSecurity CRS rules if installed
SecRuleUpdateTargetById 932130 "REQUEST_HEADERS"
Then create the file /etc/modsecurity/99-CVE-2021-44228.conf with this content:
In the first 3 lines we see that we hit different filters which check for XSS vulnerabilities, more specifically rules 941100, 941110 and 941160 all of them having the tag paranoia-level/1.
Then the fourth line shows that we hit rule 949110 which caused the web server to return the HTTP 403 Forbidden response because the inbound anomaly score, 15, is higher than 5. Then rule 980130 gives us some more information about the scoring: we hit a score of 15 at the paranoia level 1, while rules at the other paranoia levels rules contributed 0 to the total score. We also see the scores for individual types of attack: in this case all 15 points where scored by rules detecting XSS attacks. This is the meaning of the different abbreviations used:
remote file inclusion
local file inclusion
remote code execution
More detailed logs about the traffic hitting the rules can be found in the file /var/log/apache2/modsec_audit.log.
Fixing false positives
First of all, in order to minimize the amount of false positives, you should set the WEBAPPID variable if you are using one of the known web applications for which the Core Rule Set has a default exclusion set. These web applications are currently WordPress, Drupal, Dokuwiki, Nextcloud, Xenforo and cPanel. You can do so by using the <a href="https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecWebAppId">SecWebAppId</a> option in a VirtualHost of Location definition in the Apache configuration. For example if you have a VirtualHost which is used by Nextcloud, set this within the VirtualHost definition:
If you have multiple WordPress sites, give each of them a unique WEBAPPID which name starts with wordpress. Add a different suffix for every instance so that each one run its in own application namespace in ModSecurity.
If you still encounter false positives, you can completely disable rules by using the configuration directive SecRuleRemoveById. I strongly recommend not disabling rules globally, but limiting its removal to the specific location from which you want them to be removed, for example by putting them with <Location> or <LocationMatch> tags in the Apache configuration. For example:
Pay attention not to disable any of the 949*, 959*, and 980* rules: disabling the 949* and 959* rules would disable all the blocking rules, while disabling the 980* rules would give you less information about what is happening in the logs.
ModSecurity and the Core Rule Set offer an additional security layer for web servers in your defence in depth strategy. I strongly recommend implementing this on your servers because it makes it harder to abuse security vulnerabilities.
Keep an eye on the Core Rule Set blog and Twitter account: sometimes they post new rules for specific new critical vulnerabilities, which can be worthwhile to add to your configuration.
I was experiencing crackling/popping sounds while playing music with Rhythmbox in my Debian GNU/Linux Testing (Bookworm) system. The noises start when starting music playback and stop as soon as I stop the playback.
Speech-dispatcher is used for speech synthesis often used by blind or visually impaired people. If you don’t use this functionality, you can either disable speech-dispatcher (speechd) or completely uninstall it:
Disable speech-dispatcher by editing /etc/speech-dispatcher/speechd.conf, removing the comment sign # before this line near the end of the file:
Then you will have to log out and log in your desktop or you can kill all speech-dispatcher processes manually.
If you want, you can remove speech-dispatcher completely by running this command:
# apt remove speech-dispatcher
Note that this will remove the gnome meta-package, which in itself is harmless, but might lead to new GNOME components not being installed automatically in the future.
This fixed the constant popping crackling noises while playing music completely for me.