Protecting your server from known bad IPs with Foomuuri iplists

On the Internet we can find (usually crowdsourced) lists of malicious IP addresses responsible for attacks. We can easily integrate them in Foomuuri in order to block connections from these bad hosts. Not only does this improve security, it is also a performance win, because our daemons don’t don’t have to waste any more time dealing with these malicious connections.

The blocklists

  • blocklist.de: a crowdsourced list of IP addresses involved in all kinds of attacks
  • techmdw blacklist: this list is compiled by the Swedish company TechMDW AB and is based on Crowdsec with their own additions
  • Greensow
  • Spamhaus DROP: The Don’t Route or Peer List by Spamhaus contains netblocks which you should never interact with because they are leased or stolen by criminal organisations
  • Emerging Threats: compilation of Spamhaus DROP list, the top attackers list by DShield and lists by abuse.ch
  • Interserver: list consisting of IP addresses attacking servers of the web hosting company Interserver
  • Stopforumspam: list of toxic IP addresses which are believed to be used only for spamming websites

Blocking incoming connections from malicious IPs

Create file/etc/foomuuri/iplist.conf with these contents:

iplist {
	@blocklist_de https://lists.blocklist.de/lists/all.txt refresh=15m
        @techmdw https://blacklist.techmdw.com/ refresh=30m
	@greensnow https://blocklist.greensnow.co/greensnow.txt refresh=30m
	@et https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt refresh=24h
	@interserver https://rbldata.interserver.net/ip.txt refresh=15m
	@stopforumspam https://www.stopforumspam.com/downloads/toxic_ip_cidr.txt refresh=2h
        @spamhausdrop https://www.spamhaus.org/drop/drop.txt refresh=24h
        @spamhausdropv6 https://www.spamhaus.org/drop/dropv6.txt refresh=24h
}

Then in the public-localhost section (which I usually put in /etc/foomuuri/public-localhost.conf), add this at the top:

        saddr @blocklist_de drop counter "blocklist.de"
        saddr @techmdw drop counter "techmdw" 
        saddr @greensnow drop counter "greensnow"
        saddr @spamhausdrop drop counter "spamhausdrop"
        saddr @spamhausdropv6 drop counter "spamhausdropv6"
        saddr @et drop counter "et"
        saddr @interserver drop counter "interserver"
        saddr @stopforumspam drop counter "stopforumspam"

In order to prevent getting locked out of your system yourself because of a false positive or an error in one of the lists, I recommend adding one rule which allows you to always access your system before these rules. For example to make sure you can always connect from IP xxx.xxx.xxx.xxx put this as first rule in public-localhost section:

        ssh saddr xxx.xxx.xxx.xxx accept

In the rules above I’m not logging all the details of the dropped connections, but I’m keeping a counter so that I can see how many times such a rule has been hit. You can use the command

# foomuuri list counter

to see how many packets and bytes have been dropped by these rules.

If you want to log all individual dropped connections , you can add this at the end of every line:

log "name_of_the_blocklsit"

They will then be logged with the prefix name_of_the_blocklist.

Rejecting outgoing connections to malicious IPs

We can also block outgoing connections. You should especially do this for the Spamhaus DROP lists. Add this to the localhost-public section:

        daddr @blocklist_de reject counter "out_blocklist.de" log "out_blocklist.de"
        daddr @techmdw reject counter "out_techmdw" log "out_techmdw"
        daddr @greensnow reject counter "out_greensnow" log "out_greensnow"
        daddr @spamhausdrop reject counter "out_spamhausdrop" log "out_spamhausdrop"
        daddr @spamhausdropv6 reject counter "out_spamhausdropv6" log "out_spamhausdropv6"
        daddr @et reject counter "out_et" log "out_et"
        daddr @interserver reject counter "out_interserver" log "out_interserver"
        daddr @stopforumspam reject counter "out_stopforumspam" log "out_stopforumspam"

Notice that I’m rejecting instead of dropping these connections so that applications don’t keep on waiting until the connection attempt times out and I’m logging these. Normally these rules should only very rarely get triggered, but if they do you want detailed logs so you easily investigate what’s going on.

Dropping or allowing incoming connections by country of origin

Another very effective method to prevent abuse is to limit connections to services like SSH and your mail server to certain countries of origin. You can find lists of IP (both IPv4 and IPv6) addresses per country on https://github.com/ipverse/rir-ip/tree/master/country . You can add them to an iplist in Foomuuri and then use these in the public-localhost section. Note that these lists are not perfect and sometimes connections can come from another country than from the one the IP address is registered to in this database. Especially public VPN services sometimes suffer from these problems, so be careful if you are using these.

You can also use this aggregated list of all European IP addresses, but unfortunately that list only exists for IPv4 addresses.

To use this aggregated list, add this to the iplist section:

        @europe https://ipv4.fetus.jp/krfilter.4.txt refresh=24h 

Then with these rules in the the public-localhost section, I only allow IPv4 connections to port 143 (IMAP), 993 (IMAPs), port 587 (Submission), port 465 (Submissions) from European IPs. Note that I allow IPv6 from the whole world because this aggregated list only contains IPv4 addresses.

        ssh ipv4 saddr @europe
        ssh ipv6
        ssh drop counter "non-europe" log "non-europe"

        imap ipv4 saddr @europe
        imap ipv6
        imap drop counter "non-europe" log "non-europe"
        imaps ipv4 saddr @europe
        imaps ipv6
        imaps drop counter "non-europe" log "non-europe"

        submission ipv4 saddr @europe
        submission ipv6
        submission drop counter "non-europe" log "non-europe"
        submissions ipv4 saddr @europe
        submissions ipv6
        submissions drop counter "non-europe" log "non-europe"

More information

https://github.com/FoobarOy/foomuuri/wiki/Configuration#iplist

Wireguard VPN with systemd-networkd and Foomuri

After my first successful implementation of Foomuuri on a server with an IPv4 connection, I wanted to try Foomuuri in a different environment. This time I choose to implement it on my IPv4/IPv6 dual stack Wireguard VPN server. I originally set up this system with Shorewall, so let’s see how we should configure this with Foomuuri.

While at it, I also moved the configuration of Wireguard to systemd-networkd, where the main network interface was already configured. This was also useful because some things which were configured in Shorewall before and which Foomuuri does not do by itself, can now be configured in systemd-networkd.

systemd-networkd configuration

I create /etc/systemd/network/wg0.netdev with these contents:

[NetDev]
Name = wg0
Kind = wireguard
Description = wg0 - Wireguard VPN server

[WireGuard]
PrivateKeyFile = /etc/systemd/network/wg0.privkey
ListenPort = 51820

# client 1
[WireGuardPeer]
PublicKey = publickey_of_client
AllowedIPs = 192.168.7.2/32
AllowedIPs = aaaa:bbbb:cccc:dddd:ffff::2/128

I moved the /etc/wireguard/privatekey file to /etc/systemd/network/wg0.privkey, and then give it appropriate permissions so that user systemd-network can read it:

# chown root:systemd-network /etc/systemd/network/wg0.privkey
# chmod 640 /etc/systemd/network/wg0.privkey

Then I create /etc/systemd/network/wg0.network:

[Match]
Name = wg0

[Network]
Address = 192.168.7.1/24
Address = fd42:42:42::1/64

[Route]
Destination = aaaa:bbbb:cccc:dddd:ffff::2/128

For IPv4, we set the address to 192.168.7.1/24 and systemd-networkd will automatically take care of adding this subnet to the routing table. As we are using public IPv6 addresses for the VPN clients, I add a [ROUTE] section which takes care of adding these IP address to the routing table.

The configuration of the public network interface is stored in /etc/systemd/network/public.network:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=2a0f:fc80::
DNS=2a0f:fc81::
DNS=193.110.81.0
DNS=185.253.5.0
Address=www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
IPForward=yes
IPv6ProxyNDP=1
IPv6ProxyNDPAddress=aaaa:bbbb:cccc:dddd:ffff::2

Important here is that we enable IP forwarding and IPv6 NDP proxy here. Both were things we could configure in Shorewall before, but Foomuuri does not support setting these. This is not a problem, because this can be set up directly in systemd-networkd.

To reload the configuration for all network interface, I run:

networkctl reload

To bring up the Wireguard connection:

networkctl up wg0

Because of systemd issue #25547, networkctl reload is not enough if you make changes to the peer configuration in wg0.netdev. You will first have to delete the network device with the command

networkctl delete wg0

after which you can run networkctl reload and bring up the network connection. In case of doubt all network interfaces are configured correctly, you can also completely restart the systemd-networkd service:

# systemctl restart systemd-networkd

While working on the network configuration, of course make sure you have access to a real console of the system, so that in case your system becomes inaccessible, you can still fix things through the console.

Foomuuri configuration

Now we define the zones in /etc/foomuuri/zones.conf:

zone {
  localhost
  public ens192
  vpn wg0
}

Foomuuri by default does not define a macro for the Wireguard UDP port, so I create one in /etc/foomuuri/services.conf:

macro {
	wireguard udp dport 51820
}

I adjust some logging settings in /etc/foomuuri/log.conf. In case I want to filter outgoing connections from the machine in the future, I want to log the UID of the process and I also increase the log rate, as I had the impression that I sometimes was missing valuable log messages while debugging. Adjust the values if you wan to reduce log spam.

foomuuri {
  log_rate "2/second burst 20"
  log_level "level info flags skuid"
}

I set up masquerading (SNAT) in /etc/foomuuri.conf/snat.conf :

snat {
  saddr 192.168.7.0/24 oifname ens192 masquerade
}

Then I set up these rules for traffic going through our firewall:

public-localhost {
  ssh
  wireguard
  icmpv6 1 2 3 4 128
  drop log
}

localhost-public {
  accept
}

vpn-public {
  accept
}

public-vpn {
  icmpv6 1 2 3 4 128
  drop log
}

vpn-localhost {
  accept
}

localhost-vpn {
  icmpv6 1 2 3 4 128
  reject log
}

Notice that I allow ICMPv6 traffic that should not be dropped.

As usually check your configuration before reloading it:

# foomuuri check
# foomuuri reload

Testing and debugging

If things don’t work as expected, enable debugging in the wireguard kernel module and check the kernel logs. I refer to the previous article about this for more details.

Conclusion

Setting up Foomuuri was pretty easy again. The most difficult thing was getting the systemd-networkd configuration completely right. Especially with IPv6 it can take quite some time debugging before everything works as expected.

Setting up Foomuuri, an nftables based firewall

Up to now I have always been using the Shorewall firewall on all my Linux systems. I find it very easy to configure while at the same time it’s very powerful and flexible so that you can also use it with more complicated set-ups, such as routers with multiple network interfaces, VPN’s and bridges. Unfortunately Shorewall is still based on the old xtables (iptables, ip6tables, ebtables, etc…) infrastructure. While it still works and in reality the iptables commands are actually now front-ends to the more modern nftables back-end, Shorewall development has stalled and it looks very unlikely it will ever be ported to nftables.

I started using Firewalld, a firewall which is used by default on Red Hat and Fedora based systems. However I did not like it. Configuration of Firewalld happens through the command line with firewall-cmd, which I find much more complicated than just editing a configuration file which usually contains examples and gives you an easy overview of the configuration. Firewalld saves its configuration in XML files. You could edit these files instead of using firewall-cmd, but that is obviously much more complicated than editing configuration files which were designed for human editing. Furthermore I found Firewalld to be very inflexible. Firewalld does not have support of filtering traffic on a bridge (layer 2 filtering), unlike Shorewall.

Recently I discovered the nftables based firewall foomuuri. It’s still a very young project but it’s actively developed, already has extensive features, is packaged in Debian and is configured through human-readable configuration files. I decided to try it on a server where I wanted to filter incoming and outgoing network traffic.

Installing Foomuuri on Debian

Foomuuri is availabe in Debian testing and unstable, but it has also been backported to Debian 12 Bookworm. To use that package, you have to enable the bookworm-backports repository first. Then install the foomuuri package

# apt install foomuuri

If you are using NetworkManager also install foomuuri-firewalld, because it will allow NetworkManager to set the zone the network interface belongs to.

Configuring Foomuuri

Foomuuri can be configured through files in the /etc/foomuuri directory. Foomuuri will read all files which name ends with .conf, so you can split up the configuration in as many files as you want or just put everything in a single file, as you prefer. I like the split configuration files of Shorewall, so I will do something similar here.

Before activating the configuration, always run

# foomuuri check

to validate your configuration. You can start and stop the firewall by starting and stopping the systemd service, you can reload the configuration by running

 # foomuuri reload

You can find the documentation of Foomuuri on the Foomuuri wiki.

Defining zones

The first ting we have to do is define the zones and set which interfaces belongs to which zone. I create /etc/foomuuri/zones.conf:

zone {
  localhost
  public enp1s0
}

I create the zone localhost and the zone public and add the network interface enp1s0 to it. You can add multiple interfaces to a zone by separating them by spaces. If you are using NetworkManager, you don’t have to add the interfaces here and can leave the zone empty. You can configure the firewall zone in NetworkManager and it will set it through foomuuri-firewalld.

Using macros to alias configuration options

Macros can be used to define certain configuration options you want to use multiple times without having to write them completely every time. In practice a lot of macros are already configured which define the configuration for common services. You can see all defined macros by running

# foomuuri list macro

For example the macro imap defines the configuration tcp 143, so that you can just write imap instead of tcp 143 in the configuration. I added a few which were not defined by default in /etc/foomuuri/services.conf:

macro {
	nrpe	tcp 5666
	nmb	udp 137 138 139; tcp 139
}

Macros can be used to configure common subnets. For example I have a file named /etc/foomuuri/subnets.conf:

macro {
	mysubnet		192.168.0.1/24
	othersubnet		192.168.1.1/24
}

I also use macros to create lists of individual hosts, such as all NFS clients which need to access this NFS server in /etc/foomuuri/nfs_clients.conf

macro {
	nfs_clients   192.168.0.1 # web server
	nfs_clients + 192.168.0.2 # gitlab
	nfs_clients + 192.168.0.3 # nextcloud
}

For easy readability, I put every host in a single line, and I add a comment for my own reference. With the + sign I add all next hosts to the macro.

Firewall for incoming connections

To configure Foomuuri to filter incoming connections to my servers, I create a section public-localhost which contains the firewall rules for traffic coming from the public zone to localhost. I put this in the file /etc/foomuuri/public-localhost.conf:

public-localhost {
  dhcp-server
  ssh
  ping  saddr mysubnet
  nmb   saddr mysubnet
  smb   saddr mysubnet
  nfs   saddr nfs_clients
  nrpe  saddr 192.168.0.5
  drop log
}

My server is acting as a DCHP-server, so I use the dhcp-server macro to allow all this traffic, just as I allow all incoming ssh traffic. I allow ping, nmb and smb traffic from mysubnet. Notice that in these rules I use my custom macros nmb and mysubnet. Then I allow nfs from all addresses listed in my macro nfs_clients, and I allow nrpe from a specific IP address. Finally I end with a rule which drops and logs all traffic which has not matched any of the rules before.

Firewall for outgoing connections

I think that filtering outgoing connections is a very effective security hardening measure. In case people with bad intentions get access to your server through a non-root user account, this will severely limit their abilities to move laterally through your network and attack other systems, to run a crypto-miner, or download malware from the Internet.

localhost-public {
  dhcp-client
  nmb uid root
  ntp uid systemd-timesync
  ping uid root daddr mysubnet # dhcpd sometimes pings
  smtp daddr 192.168.0.1 uid postfix
  domain daddr 192.168.0.255 192.168.0.254
  uid root tcp daddr 192.168.0.5 dport 8140 # puppet agent
  uid _apt tcp dport 3142 daddr 192.168.0.6
  uid root ssh daddr 192.168.0.250 # backups
  drop daddr 169.254.169.254 tcp dport 80 # don't fill logs with Puppetlabs facter trying to collect facts from Amazon EC2/Azure
  reject log
}

I allow outgoing connections for different services, and for most services I set the user which can create that connection, and to which host I allow the connection. I explicitly drop without logging connections to 169.254.169.254 port 80, because facter tries to connect to this address every time it runs in order to get some metadata from your cloud service provider. If your system is running on Amazon or Microsoft Azure cloud services, you will probably want to allow this connection instead, so you can then just remove the drop word.

In order to log the UID of the process which tried to establish a rejected connection, in future Foomuuri versions (starting from Foomuuri version 0.22) you can replace the last rule by

reject log log_level "level warn flags skuid"

In current version 0.21, it is possible by setting this globally for all connections. I created /etc/foormuuri/loglevel.conf:

foomuuri {
  log_level "level info flags skuid"
}

Integrating Fail2ban with Foomuuri

I found inspiration for integrating Fail2ban with Foomuuri in issue 9 on the Foomuuri issue tracker.

Create /etc/fail2ban/action.d/foomuuri with these contents:

[Definition]
actionstart =
actionstop  =
actioncheck =
actionban   = /usr/sbin/foomuuri iplist add fail2ban 999d <ip>
actionunban = /usr/sbin/foomuuri iplist del fail2ban <ip>
actionflush = /usr/sbin/foomuuri iplist flush fail2ban

Then set foomuri as the default banaction by creating /etc/fail2ban/jail.d/foomuri.conf:

[DEFAULT]
banaction = foomuuri

Then foomuuri should create the fail2ban iplist. We can configure it to so by creating /etc/foomuuri/fail2ban.conf:

iplist {
	@fail2ban
}

Then I add this rule as first rule to the public-localhost section:

  saddr @fail2ban drop log fail2ban drop

This will drop all connections coming from an address in the iplist fail2ban, and will also log them with prefix fail2ban. If you don’t want this to be logged, just remove log fail2ban.

To ensure that Foomuuri is started before Fail2ban, so that the fail2ban iplist exists before Fail2ban starts to use it, create

/etc/systemd/system/fail2ban.service.d/override.conf:

[Unit]
After=foomuuri.service

After making these changes, first restart Foomuuri and then Fail2ban.

Conclusion

I found Foomuuri easy to use for a system with one network interface. Configuration through the configuration files is easy, also when implementing filtering for outgoing packets. Even though Foomuuri is still a young project, it already has many features and its author is very reactive to discussions and issues on Github. I also found the documentation on the wiki very helpful

I will try to implement Foomuuri on more complex setups in the future, such as on a host for virtual machines of which the network interface is bridged to the main network interface of the host, VPN servers, routers, etc…

Finally I want to thank the Foomuri developer Kim B. Heino and the maintainer of the Debian package Romain Francoise for their work and making this available to the community.