So I decided to build Apparmor 3.0.12 from sid with the additional patches mentioned above for Debian Bookworm, as well as the new libapache2-mod-qos which fixes the crash at Apache startup. I have created a public repository you can use if you are interested in these fixes. The packages work for me, but I cannot guarantee that they won’t cause any problem for you, so use them at your own risk. I only build for AMD64, so other architectures are not available.
Setting up the bookworm-frehi repository on Debian
In order to use these packages, create a file /etc/apt/sources.list.d/bookworm-frehi.list with this content:
deb http://debian.frehi.be/debian bookworm-frehi main contrib non-free
deb-src http://debian.frehi.be/debian bookworm-frehi main contrib non-free
You can also use https in case you prefer that, but I try to use http because then I can cache packages with apt-cacher-ng.
Then create a file /etc/apt/preferences.d/bookworm-frehi:
This makes sure that by default you will still be using packages from the Debian repository, and it will only use packages from this repository when you explicitly request to do so.
Then you will have to request he public GPG key from pgp.surf.nl and add this to your trusted apt keys:
In a previous article, I discussed how to set up ModSecurity with the Core Rule Set on Debian. This can be considered as a first line of defense against malicious HTTP traffic. In a defense in depth strategy of course we want to add additional layers of protection to your web servers. One such layer is Snuffleupagus. Snuffleupagus is a PHP module which protects your web applications against various attacks. Some of the hardening features it offers are encryption of cookies, disabling XML External Entity (XXE) processing, a white or blacklist for the functions which can be used in the eval() function and the possibility to selectively disable PHP functions with specific arguments (virtual-patching).
Installing Snuffleupagus on Debian
Unfortunately there is no package for Snuffleupagus included in Debian, but it is not too difficult to build one yourself:
$ apt install php-dev
$ mkdir snuffleupagus
$ cd snuffleupagus
$ git clone https://github.com/jvoisin/snuffleupagus
$ cd snuffleupagus
$ make debian
This will build the latest development code from the master branch. If you want to build the latest stable release, before running make debian, use these commands to view all tags and to checkout the latest table tag, which in this case was v0.8.2:
$ git tag
$ git checkout v0.8.2
If all went well, you should now have a file snuffleupagus_0.8.2_amd64.deb in the above directory, which you can install:
$ cd ..
$ apt install ./snuffleupagus_0.8.2_amd64.deb
Configuring Snuffleupagus
First we take the example configuration file and put it in PHP’s configuration directory. For example for PHP 7.4:
Snuffleupagus can run rules in simulation mode. In this mode, the rule will not block further execution of the PHP file, but will just output a warning message in your log. Unfortunately there is no global simulation mode, but it has to be set per rule. You can run a rule in simulation mode by appending .simulation() to it. For example to run INI protection in simulation mode:
sp.ini_protection.simulation();
INI protection
To prevent PHP applications from modifying php.ini settings, you can set this in snuffleupagus.rules:
The following configuration options sets the SameSite attribute to Lax on session cookies, which offers protection against CSFR on this cookie. We enforce setting the secure option on cookies, which instructs the web browser to only send them over an encrypted HTTPS connection and also enable encryption of the content of the session on the server. The encryption key being used is derived of the value of the global secret key you have set, the client’s user agent and the environment variable SSL_SESSION_ID.
eval() is used to evaluate PHP content, for example in a variable. This is very dangerous if the PHP code to be evaluated can contain user provided data. Therefore it is strongly recommended that you create a whitelist of functions which can be called by code evaluated by eval().
Start by putting this in snuffleupagus.rules and restart PHP:
sp.eval_whitelist.list().simulation();
Then test your websites and see which errors you get in the logs, and add them separated by commas to the eval_whitelist.list(). After that you need to remove .simulation() and restart PHP in order to activate this protection. For example
sp.eval_whitelist.list("array_pop,array_push");
You can also use a blacklist, which only blocks certain functions. For example:
The read_only_exec() feature of Snuffleupagus will prevent PHP from execution of PHP files on which the PHP process has write permissions. This will block any attacks where an attacker manages to upload a malicious PHP file via a bug in your website, and then attempts to execute this malicious PHP script.
It is a good practice to let your PHP scripts be owned by a different user than the PHP user, and give PHP only read-only permissions on your PHP files.
To test this feature, add this to snuffleupagus.rules:
sp.readonly_exec.simulation();
If you are sure all goes well, enable it:
sp.readonly_exec.enable();
Virtual patching
One of the main features of Snuffleupagus is virtual patching. Thjs feature will disable functions, depending on the parameters or and values they are given. The example rules file contains a good set of generic rules which blocks all kinds of dangerous behaviour. You might need to fine-tune the rules if your PHP applications hits certain rules.
These two rules will only allow the include_once function to include files which file name are ending with inc, phtml or php. All other include_once calls will be dropped.
Using generate-rules.php to automatically site-specific hardening rules
In the scripts subdirectoy of the Snuffleupagus source tree, there is a file named generate_rules.php. You can run this script from the command line, giving it a path to a directory with PHP files, and it will automatically generate rules which specifically allow all needed dangerous function calls, and then disable them globally. For example to generate rules for the /usr/share/tt-rss/www and /var/www directories:
The first two rules allow these two files to call function_exists and the last rule drops all requests to function_exists from any other files. Note that the first two rules limit the rules not only to the specified file name, but also define the SHA256 of the file. This way, if the file is changed, the function call will be dropped. This is the safest way, but it can be annoying if the files are often or automatically updated because it will break the site. In this case, you can call generate_rules.php with the --without-hash option:
After you have generated the rules, you will have to add them to your snuffleupagus.rules file and restart PHP-FPM.
File Upload protection
The default Snuffleupagus rule file contains 2 rule which will block any attempts uploading a html or PHP file. However, I noticed that they were not working with PHP 7.4 and these rules would cause this error message:
PHP Warning: [snuffleupagus][0.0.0.0][config][log] It seems that you are filtering on a parameter 'destination' of the function 'move_uploaded_file', but the parameter does not exists. in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 0 parameter's name: 'path' in /var/www/html/foobar.php on line 15PHP message: PHP Warning: [snuffleupagus][0.0.0.0][config][log] - 1 parameter's name: 'new_path' in /var/www/html/foobar.php on line 15'
The snuffleupagus rules use the parameter destination for the move_uploaded_file instead of the parameter new_path. You will have to change the rules like this:
After restarting PHP-FPM, always check the logs to see whether snuffleupagus does not give any warning or messages for example because of a syntax error in your configuration:
# journalctl -u php7.4-fpm -n 50
Snuffleupagus logs
By default Snuffleupagus logs via PHP. Then if you are using Apache with PHP-FPM, you will find Snuffleupagus logs, just like any PHP warnings and errors in the Apache error_log, for example /var/log/apache/error.log. If you encounter any problems with your website, go check this log to see what is wrong.
Snuffleupagus can also be configured to log via syslog, and actually even recommends this, because PHP’s logging system can be manipulated at runtime by malicious scripts. To log via syslog, add this to snuffleupagus.rules:
sp.log_media("syslog");
I give a few examples of errors you can encounter in the logs and how to fix them:
[snuffleupagus][0.0.0.0][xxe][log] A call to libxml_disable_entity_loader was tried and nopped in /usr/share/tt-rss/www/include/functions.php on line 22
tt-rss calls the function libxml_disable_entity_loader but this is blocked by the XXE protection. Commenting this in snuffleupagus.rules should fix this:
sp.xxe_protection.enable();
Another example:
[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'ini_set', because its argument '$varname' content (display_errors) matched a rule in /usr/share/tt-rss/www/include/functions.php on line 37'
Modifying the PHP INI option display_errors is not allowed by this rule:
You can completely remove (or comment) this rule in order to disable it. But a better way is to add a rule before this rule which allows it for specially that PHP file. So add this rule before:
[snuffleupagus][0.0.0.0][disabled_function][drop] Aborted execution on call of the function 'function_exists', because its argument '$function_name' content (exec) matched a rule in /var/www/wordpress/wp-content/plugins/webp-express/vendor/rosell-dk/exec-with-fallback/src/ExecWithFallback.php on line 35', referer: wp-admin/media-new.php
A web application firewall (WAF) filters HTTP traffic. By integrating this in your web server, you can make sure potentially dangerous requests are blocked before they arrive to your web application or sensitive data leaks out of your web server. This way you add an extra defensive layer potentially offering extra protection against zero-day vulnerabilities in your web server or web applications. In this blog post, I give a tutorial how to install and configure ModSecurity web application firewall and the Core Rule Set on Debian. With some minor adaptions you can also use this guide for setting up ModSecurity on Ubuntu or other distributions.
ModSecurity is the most well-known open source web application firewall. The future of ModSecurity does not look too bright but fortunately with Coraza WAF an alternative which is completely compatible with ModSecurity is in development. At this moment Coraza only integrates with the Caddy web server, and does not have a connector for Apache or NGinx so for that reason it is currently not yet usable as a replacement for ModSecurity.
While ModSecurity provides the framework for filtering HTTP traffic, you also need rules which define what to bloc and that’s where the Core Rule Set (CRS) comes in. CRS is a set of generic rules winch offer protection to a various range of common attacks via HTTP, such as SQL injection, code injection and cross-site scripting (XSS) attacks.
Install ModSecurity and the Core Rule Set on Debian
I install the Apache module for ModSecurity, the geoip-database, which can be used for blocking all requests from certain countries, and modsecurity-crs, which contains the Core Rule Set. I take this package from testing, because it has a newer version (version 3.3.2 at the time of writing). There is no risk in taking this package from testing, because it only contains the rules and does not depend on any other packages from testing/unstable. If you prefer faster updates, you can also use unstable.
Now edit /etc/modsecurity/modsecurity.conf. I highlight some of the options:
SecRuleEngine on
SecRequestBodyLimit 536870912
SecRequestBodyNoFilesLimit 131072
SecAuditLog /var/log/apache2/modsec_audit.log
#SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
#"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"
SecPcreMatchLimit 500000
SecPcreMatchLimitRecursion 500000
SecStatusEngine Off
The SecRuleEngine option controls whether rules should be processed. If set to Off, you completely disable all rules, with On you enable them and it will block malicious actions. If set to DetectionOnly, ModSecurity will only log potential malicious activity flagged by your rules, but will not block them. DetectionOnly can be useful for temporary trying out the rules in order to find false positives before you really start blocking potential malicious activity.
The SecAuditLog option defines a file which contains audit logs. This file will contain detailed logs about every request triggering a ModSecurity rule.
The SecPcreMatchLimit and SecPcreMatchLimitRecursion set the match limit and match limit recursion for the regular expression library PCRE. Setting this high enough will prevent errors that the PCRE limits were exceeded while analyzing data, but setting it too high can make ModSecurity vulnerable to a Denial of Service (DoS) attack. A Core Rule Set developer recommends a value of 50000 so that’s what I use here.
Set SecStatusEngine to Off to prevent ModSecurity sending version information back its developers.
After changing any configuration related to ModSecurity or the Core Rule Set, reload your Apache web server:
# systemctl reload apache2
Configuring the Core Rule Set
The Core Rule Set can be configured via the file /etc/modsecurity/crs/crs-setup.conf.
Anomaly Scoring
By default the Core Rule Set is using anomaly scoring mode. This means that individual rules add to a so called anomaly score, which at the end is evaluated. If the anomaly score exceeds a certain threshold, then the traffic is blocked. You can read more about this configuration in crs-setup.conf but the default configuration should be fine for most people.
Setting the paranoia level
The paranoia level is a number from 1 to 4 which determines which rules are active and contribute to the anomaly scoring. The higher the paranoia level, the more rules are activated and hence the more aggressive the Core Rule Set is, offering more protection but potentially also causing more false positives. By default the paranoia level is set to 1. If you work with sensitive data, it is recommended to increase the paranoia level.
The executing paranoia level defines the rules which will be executed but their score will not be added to the anomaly scoring. When HTTP traffic hits rules of the executing paranoia level, this traffic will only be logged but not be blocked. It is a especially useful to prepare for increasing the paranoia level and finding false positives on this higher level, without causing any disruption for your users.
To set the paranoia level to 1 and the executing paranoia level to 2, make sure you have these rules set in crs-setup.conf:
Once you have fixed all false positives, you can raise the paranoia level to 2 to increase security.
Defining the allowed HTTP methods
By default the Core Rule Set only allows the GET, HEAD, POST and OPTIONS HTTP methods. For many standard sites this will be enough but if your web applications also use restful APIs or WebDAV, then you will need to add the required methods. Change rule 900200, and add the HTTP methods mentioned in the comments in crs-setup.conf.
SecAction \
"id:900200,\
phase:1,\
nolog,\
pass,\
t:none,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'"
Disallowing old HTTP versions
There is a rule which determines which HTTP versions you allow in HTTP requests. I uncomment it and modify it to only allow HTTP versions 1.1 and 2.0. Legitimate browsers and bots always use one of these modern HTTP versions and older versions usually are a sign of malicious activity.
Personally I’m not a fan of completely blocking all traffic from a whole country, because you will also block legitimate visitors to your site, but in case you want to this, you can configure this in crs-setup.conf:
Add the two-letter country codes you want to block to the last line (before the two quotes), multiple country codes separated by a space.
Make sure you have the package geoip-database installed.
Core Rule Set Exclusion rules for well-known web applications
The Core Rule Set contains some rule exclusions for some well-known web applications like WordPress, Drupal and NextCloud which reduces the number of false positives. I add the following section to crs-setup.conf which will allow me to enable the exclusions in the Apache configuration by setting the WEBAPPID variable in the Apache configuration whenever I need them.
Adding rules for Log4Shell and Spring4Shell detection
At the end of 2021 a critical vulnerability CVE-2021-44228, named Log4Shell, was detected in Log4j, which allows remote attackers to run code on a server with the vulnerable Log4j version. While the Core Rule Set offered some mitigation of this vulnerability out of the box, this protection was not complete. New improved detection rules against Log4Shell were developed. Because of the severity of this bug and the fact that it’s being exploited in the wild, I strongly recommend adding this protection manually when using ModSecurity version 3.3.2 (or older). Newer, not yet released versions, should have complete protection out of the box.
First modify /etc/apache2/mods-enabled/security2.conf so that it looks like this:
<IfModule security2_module>
# Default Debian dir for modsecurity's persistent data
SecDataDir /var/cache/modsecurity
# Include all the *.conf files in /etc/modsecurity.
# Keeping your local configuration in that directory
# will allow for an easy upgrade of THIS file and
# make your life easier
IncludeOptional /etc/modsecurity/*.conf
# Include OWASP ModSecurity CRS rules if installed
IncludeOptional /usr/share/modsecurity-crs/*.load
SecRuleUpdateTargetById 932130 "REQUEST_HEADERS"
</IfModule>
Then create the file /etc/modsecurity/99-CVE-2021-44228.conf with this content:
Whenever something hits your ModSecurity rules, this will be logged in your Apache error log. The above request has created these messages in the error log:
In the first 3 lines we see that we hit different filters which check for XSS vulnerabilities, more specifically rules 941100, 941110 and 941160 all of them having the tag paranoia-level/1.
Then the fourth line shows that we hit rule 949110 which caused the web server to return the HTTP 403 Forbidden response because the inbound anomaly score, 15, is higher than 5. Then rule 980130 gives us some more information about the scoring: we hit a score of 15 at the paranoia level 1, while rules at the other paranoia levels rules contributed 0 to the total score. We also see the scores for individual types of attack: in this case all 15 points where scored by rules detecting XSS attacks. This is the meaning of the different abbreviations used:
SQLI
SQL injection
XSS
cross-site scripting
RFI
remote file inclusion
LFI
local file inclusion
RCE
remote code execution
PHPI
PHP injection
HTTP
HTTP violation
SESS
session fixation
More detailed logs about the traffic hitting the rules can be found in the file /var/log/apache2/modsec_audit.log.
Fixing false positives
First of all, in order to minimize the amount of false positives, you should set the WEBAPPID variable if you are using one of the known web applications for which the Core Rule Set has a default exclusion set. These web applications are currently WordPress, Drupal, Dokuwiki, Nextcloud, Xenforo and cPanel. You can do so by using the <a href="https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecWebAppId">SecWebAppId</a> option in a VirtualHost of Location definition in the Apache configuration. For example if you have a VirtualHost which is used by Nextcloud, set this within the VirtualHost definition:
If you have multiple WordPress sites, give each of them a unique WEBAPPID which name starts with wordpress. Add a different suffix for every instance so that each one run its in own application namespace in ModSecurity.
If you still encounter false positives, you can completely disable rules by using the configuration directive SecRuleRemoveById. I strongly recommend not disabling rules globally, but limiting its removal to the specific location from which you want them to be removed, for example by putting them with <Location> or <LocationMatch> tags in the Apache configuration. For example:
Pay attention not to disable any of the 949*, 959*, and 980* rules: disabling the 949* and 959* rules would disable all the blocking rules, while disabling the 980* rules would give you less information about what is happening in the logs.
Conclusion
ModSecurity and the Core Rule Set offer an additional security layer for web servers in your defence in depth strategy. I strongly recommend implementing this on your servers because it makes it harder to abuse security vulnerabilities.
Keep an eye on the Core Rule Set blog and Twitter account: sometimes they post new rules for specific new critical vulnerabilities, which can be worthwhile to add to your configuration.
To prevent tracking by FLoC and other tracking techniques, there is only one good solution: stop using Google Chrome. The best privacy friendly browser is Firefox, especially if you set it to strict enhanced tracking protection. If you really need to use Chrome, then at least install one of the open source extensions which disable FLoC and Privacy Badger for other tracking protection.
This can be easily done for all your websites by modifying your Apache configuration. While at it, you should set some other security and privacy related headers, notably:
X-Frame-Options "SAMEORIGIN": this makes sure that browsers will not allow your website to be included in an iframe on another domain.
X-Content-Type-Options "nosniff": This will prevent the browser from trying to automatically detect the file type of a downloaded file instead of using the MIME type sent by the server. This can mitigate attacks where a hacker manages to upload a malicious file by giving it a filename which makes it look like a harmless file type which is then served to your visitors.
Referrer-Policy "no-referrer-when-downgrade": when a visitor clicks on a link, the browser will only send the referrer when it’s not going from a HTTPS to a HTTP connection. This is fine if your URLs don’t contain any private information. If they do, then consider using strict-origin-when-cross-origin, so that only your domain name instead of the complete URL is sent as referrer if people click on a link leading to an external website, or even same-origin, which prevents any referrer being sent to external sites. You should probably do this for an internal website, web application or wiki, webmail, etc. More information about Referrer-Policy
To set these in Apache in Debian, create a file /etc/apache2/conf-available/security-headers.conf with these contents:
<IfModule mod_headers.c>
Header always set X-Frame-Options "SAMEORIGIN"
Header always set X-Content-Type-Options "nosniff"
Header always set Referrer-Policy "no-referrer-when-downgrade"
Header always set Permissions-Policy: interest-cohort=()
</IfModule>
Then make sure the mod_headers module is loaded and this file is enabled by running these commands:
Another important header to set in your SSL virtualhosts is the HSTS header: it ensures that the browser will automatically use HTTPS every time when connecting to the website in the future. Place this in your SSL enabled virtualhost:
<IfModule mod_headers.c>
Header always set Strict-Transport-Security "max-age=63072000"
</IfModule>
Then you should also add this to your non-SSL virtualhost to redirect all visitors using HTTP to HTTPS:
Of course make sure mod_rewrite is enabled if that’s not yet the case:
# a2enmod rewrite
# systemctl reload apache2
You can check your server configuration on securityheaders.com. There you can also find more information about the headers Cross-Origin-Embedder-Policy, Cross-Origin-Opener-Policy and Cross-Origin-Resource-Policy, some other security related headers. Because they require more changes to your website to implement correctly, I’m not discussing them here.
Often you run different web applications on the same web servers. For security reasons, it is strongly recommended to run them in separate PHP-FPM processes under different user accounts. This way permissions can be set so that the user account of one PHP application, cannot access the files from another PHP application. Also open_basedir can be set so that accessing any files outside the base directory becomes impossible.
To create a separate PHP-FPM process for a PHP application on Debian Stretch with PHP 7.0, create a file /etc/php/7.0/fpm/pool.d/webapp.conf with these contents:
In the above example the webapp is located in /home/webapp, but you can of course also use a directory somewhere in /var/www.
I strongly recommend against making all your PHP files in /home/webapp owned by webapp_php. This is a dangerous situation, because PHP can overwrite the code itself. This makes it possible for malware to overwrite your PHP files with malicious code. Only make the directories where PHP really needs to be able to write into (for example a directory where files uploaded in your web applications are stored), writable for the webapp_php user. Your code itself should be owned by a different user than webapp_php. It can be a dedicated user account, or just root.
Finally we need to configure Apache to contact the right php-fpm instance for the web application. Create a file /etc/apache2/conf-available/php7.0-webapp-fpm.conf:
<Directory /home/webapp/public_html>
# Redirect to local php-fpm if mod_php is not available
<IfModule proxy_fcgi_module>
# Enable http authorization headers
<IfModule setenvif_module>
SetEnvIfNoCase ^Authorization$ "(. )" HTTP_AUTHORIZATION=$1
</IfModule>
<FilesMatch ". \.ph(p[3457]?|t|tml)$">
SetHandler "proxy:unix:/run/php/php7.0-webapp-fpm.sock|fcgi://localhost-webapp"
</FilesMatch>
<FilesMatch ". \.phps$">
# Deny access to raw php sources by default
# To re-enable it's recommended to enable access to the files
# only in specific virtual host or directory
Require all denied
</FilesMatch>
# Deny access to files without filename (e.g. '.php')
<FilesMatch "^\.ph(p[3457]?|t|tml|ps)$">
Require all denied
</FilesMatch>
</IfModule>
</Directory>
This file is based on the default php7.0-fpm.conf. You will need to create a symlink to make sure this gets activated:
# cd /etc/apache2/conf-enabled
# ln -s ../conf-available/php7.0-webapp-fpm.conf .
Now restart your Apache and PHP-FPM services and you should be ready. You can see the user your code in /home/webapp/public_html is being run as in the output of the phpinfo() function.