Protecting your server from known bad IPs with Foomuuri iplists

On the Internet we can find (usually crowdsourced) lists of malicious IP addresses responsible for attacks. We can easily integrate them in Foomuuri in order to block connections from these bad hosts. Not only does this improve security, it is also a performance win, because our daemons don’t don’t have to waste any more time dealing with these malicious connections.

The blocklists

  • blocklist.de: a crowdsourced list of IP addresses involved in all kinds of attacks
  • techmdw blacklist: this list is compiled by the Swedish company TechMDW AB and is based on Crowdsec with their own additions
  • Greensow
  • Spamhaus DROP: The Don’t Route or Peer List by Spamhaus contains netblocks which you should never interact with because they are leased or stolen by criminal organisations
  • Emerging Threats: compilation of Spamhaus DROP list, the top attackers list by DShield and lists by abuse.ch
  • Interserver: list consisting of IP addresses attacking servers of the web hosting company Interserver
  • Stopforumspam: list of toxic IP addresses which are believed to be used only for spamming websites

Blocking incoming connections from malicious IPs

Create file/etc/foomuuri/iplist.conf with these contents:

iplist {
	@blocklist_de https://lists.blocklist.de/lists/all.txt refresh=15m
        @techmdw https://blacklist.techmdw.com/ refresh=30m
	@greensnow https://blocklist.greensnow.co/greensnow.txt refresh=30m
	@et https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt refresh=24h
	@interserver https://rbldata.interserver.net/ip.txt refresh=15m
	@stopforumspam https://www.stopforumspam.com/downloads/toxic_ip_cidr.txt refresh=2h
        @spamhausdrop https://www.spamhaus.org/drop/drop.txt refresh=24h
        @spamhausdropv6 https://www.spamhaus.org/drop/dropv6.txt refresh=24h
}

Then in the public-localhost section (which I usually put in /etc/foomuuri/public-localhost.conf), add this at the top:

        saddr @blocklist_de drop counter "blocklist.de"
        saddr @techmdw drop counter "techmdw" 
        saddr @greensnow drop counter "greensnow"
        saddr @spamhausdrop drop counter "spamhausdrop"
        saddr @spamhausdropv6 drop counter "spamhausdropv6"
        saddr @et drop counter "et"
        saddr @interserver drop counter "interserver"
        saddr @stopforumspam drop counter "stopforumspam"

In order to prevent getting locked out of your system yourself because of a false positive or an error in one of the lists, I recommend adding one rule which allows you to always access your system before these rules. For example to make sure you can always connect from IP xxx.xxx.xxx.xxx put this as first rule in public-localhost section:

        ssh saddr xxx.xxx.xxx.xxx accept

In the rules above I’m not logging all the details of the dropped connections, but I’m keeping a counter so that I can see how many times such a rule has been hit. You can use the command

# foomuuri list counter

to see how many packets and bytes have been dropped by these rules.

If you want to log all individual dropped connections , you can add this at the end of every line:

log "name_of_the_blocklsit"

They will then be logged with the prefix name_of_the_blocklist.

Rejecting outgoing connections to malicious IPs

We can also block outgoing connections. You should especially do this for the Spamhaus DROP lists. Add this to the localhost-public section:

        daddr @blocklist_de reject counter "out_blocklist.de" log "out_blocklist.de"
        daddr @techmdw reject counter "out_techmdw" log "out_techmdw"
        daddr @greensnow reject counter "out_greensnow" log "out_greensnow"
        daddr @spamhausdrop reject counter "out_spamhausdrop" log "out_spamhausdrop"
        daddr @spamhausdropv6 reject counter "out_spamhausdropv6" log "out_spamhausdropv6"
        daddr @et reject counter "out_et" log "out_et"
        daddr @interserver reject counter "out_interserver" log "out_interserver"
        daddr @stopforumspam reject counter "out_stopforumspam" log "out_stopforumspam"

Notice that I’m rejecting instead of dropping these connections so that applications don’t keep on waiting until the connection attempt times out and I’m logging these. Normally these rules should only very rarely get triggered, but if they do you want detailed logs so you easily investigate what’s going on.

Dropping or allowing incoming connections by country of origin

Another very effective method to prevent abuse is to limit connections to services like SSH and your mail server to certain countries of origin. You can find lists of IP (both IPv4 and IPv6) addresses per country on https://github.com/ipverse/rir-ip/tree/master/country . You can add them to an iplist in Foomuuri and then use these in the public-localhost section. Note that these lists are not perfect and sometimes connections can come from another country than from the one the IP address is registered to in this database. Especially public VPN services sometimes suffer from these problems, so be careful if you are using these.

You can also use this aggregated list of all European IP addresses, but unfortunately that list only exists for IPv4 addresses.

To use this aggregated list, add this to the iplist section:

        @europe https://ipv4.fetus.jp/krfilter.4.txt refresh=24h 

Then with these rules in the the public-localhost section, I only allow IPv4 connections to port 143 (IMAP), 993 (IMAPs), port 587 (Submission), port 465 (Submissions) from European IPs. Note that I allow IPv6 from the whole world because this aggregated list only contains IPv4 addresses.

        ssh ipv4 saddr @europe
        ssh ipv6
        ssh drop counter "non-europe" log "non-europe"

        imap ipv4 saddr @europe
        imap ipv6
        imap drop counter "non-europe" log "non-europe"
        imaps ipv4 saddr @europe
        imaps ipv6
        imaps drop counter "non-europe" log "non-europe"

        submission ipv4 saddr @europe
        submission ipv6
        submission drop counter "non-europe" log "non-europe"
        submissions ipv4 saddr @europe
        submissions ipv6
        submissions drop counter "non-europe" log "non-europe"

More information

https://github.com/FoobarOy/foomuuri/wiki/Configuration#iplist

Wireguard VPN with systemd-networkd and Foomuri

After my first successful implementation of Foomuuri on a server with an IPv4 connection, I wanted to try Foomuuri in a different environment. This time I choose to implement it on my IPv4/IPv6 dual stack Wireguard VPN server. I originally set up this system with Shorewall, so let’s see how we should configure this with Foomuuri.

While at it, I also moved the configuration of Wireguard to systemd-networkd, where the main network interface was already configured. This was also useful because some things which were configured in Shorewall before and which Foomuuri does not do by itself, can now be configured in systemd-networkd.

systemd-networkd configuration

I create /etc/systemd/network/wg0.netdev with these contents:

[NetDev]
Name = wg0
Kind = wireguard
Description = wg0 - Wireguard VPN server

[WireGuard]
PrivateKeyFile = /etc/systemd/network/wg0.privkey
ListenPort = 51820

# client 1
[WireGuardPeer]
PublicKey = publickey_of_client
AllowedIPs = 192.168.7.2/32
AllowedIPs = aaaa:bbbb:cccc:dddd:ffff::2/128

I moved the /etc/wireguard/privatekey file to /etc/systemd/network/wg0.privkey, and then give it appropriate permissions so that user systemd-network can read it:

# chown root:systemd-network /etc/systemd/network/wg0.privkey
# chmod 640 /etc/systemd/network/wg0.privkey

Then I create /etc/systemd/network/wg0.network:

[Match]
Name = wg0

[Network]
Address = 192.168.7.1/24
Address = fd42:42:42::1/64

[Route]
Destination = aaaa:bbbb:cccc:dddd:ffff::2/128

For IPv4, we set the address to 192.168.7.1/24 and systemd-networkd will automatically take care of adding this subnet to the routing table. As we are using public IPv6 addresses for the VPN clients, I add a [ROUTE] section which takes care of adding these IP address to the routing table.

The configuration of the public network interface is stored in /etc/systemd/network/public.network:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=2a0f:fc80::
DNS=2a0f:fc81::
DNS=193.110.81.0
DNS=185.253.5.0
Address=www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
IPForward=yes
IPv6ProxyNDP=1
IPv6ProxyNDPAddress=aaaa:bbbb:cccc:dddd:ffff::2

Important here is that we enable IP forwarding and IPv6 NDP proxy here. Both were things we could configure in Shorewall before, but Foomuuri does not support setting these. This is not a problem, because this can be set up directly in systemd-networkd.

To reload the configuration for all network interface, I run:

networkctl reload

To bring up the Wireguard connection:

networkctl up wg0

Because of systemd issue #25547, networkctl reload is not enough if you make changes to the peer configuration in wg0.netdev. You will first have to delete the network device with the command

networkctl delete wg0

after which you can run networkctl reload and bring up the network connection. In case of doubt all network interfaces are configured correctly, you can also completely restart the systemd-networkd service:

# systemctl restart systemd-networkd

While working on the network configuration, of course make sure you have access to a real console of the system, so that in case your system becomes inaccessible, you can still fix things through the console.

Foomuuri configuration

Now we define the zones in /etc/foomuuri/zones.conf:

zone {
  localhost
  public ens192
  vpn wg0
}

Foomuuri by default does not define a macro for the Wireguard UDP port, so I create one in /etc/foomuuri/services.conf:

macro {
	wireguard udp dport 51820
}

I adjust some logging settings in /etc/foomuuri/log.conf. In case I want to filter outgoing connections from the machine in the future, I want to log the UID of the process and I also increase the log rate, as I had the impression that I sometimes was missing valuable log messages while debugging. Adjust the values if you wan to reduce log spam.

foomuuri {
  log_rate "2/second burst 20"
  log_level "level info flags skuid"
}

I set up masquerading (SNAT) in /etc/foomuuri.conf/snat.conf :

snat {
  saddr 192.168.7.0/24 oifname ens192 masquerade
}

Then I set up these rules for traffic going through our firewall:

public-localhost {
  ssh
  wireguard
  icmpv6 1 2 3 4 128
  drop log
}

localhost-public {
  accept
}

vpn-public {
  accept
}

public-vpn {
  icmpv6 1 2 3 4 128
  drop log
}

vpn-localhost {
  accept
}

localhost-vpn {
  icmpv6 1 2 3 4 128
  reject log
}

Notice that I allow ICMPv6 traffic that should not be dropped.

As usually check your configuration before reloading it:

# foomuuri check
# foomuuri reload

Testing and debugging

If things don’t work as expected, enable debugging in the wireguard kernel module and check the kernel logs. I refer to the previous article about this for more details.

Conclusion

Setting up Foomuuri was pretty easy again. The most difficult thing was getting the systemd-networkd configuration completely right. Especially with IPv6 it can take quite some time debugging before everything works as expected.

Setting up Foomuuri, an nftables based firewall

Up to now I have always been using the Shorewall firewall on all my Linux systems. I find it very easy to configure while at the same time it’s very powerful and flexible so that you can also use it with more complicated set-ups, such as routers with multiple network interfaces, VPN’s and bridges. Unfortunately Shorewall is still based on the old xtables (iptables, ip6tables, ebtables, etc…) infrastructure. While it still works and in reality the iptables commands are actually now front-ends to the more modern nftables back-end, Shorewall development has stalled and it looks very unlikely it will ever be ported to nftables.

I started using Firewalld, a firewall which is used by default on Red Hat and Fedora based systems. However I did not like it. Configuration of Firewalld happens through the command line with firewall-cmd, which I find much more complicated than just editing a configuration file which usually contains examples and gives you an easy overview of the configuration. Firewalld saves its configuration in XML files. You could edit these files instead of using firewall-cmd, but that is obviously much more complicated than editing configuration files which were designed for human editing. Furthermore I found Firewalld to be very inflexible. Firewalld does not have support of filtering traffic on a bridge (layer 2 filtering), unlike Shorewall.

Recently I discovered the nftables based firewall foomuuri. It’s still a very young project but it’s actively developed, already has extensive features, is packaged in Debian and is configured through human-readable configuration files. I decided to try it on a server where I wanted to filter incoming and outgoing network traffic.

Installing Foomuuri on Debian

Foomuuri is availabe in Debian testing and unstable, but it has also been backported to Debian 12 Bookworm. To use that package, you have to enable the bookworm-backports repository first. Then install the foomuuri package

# apt install foomuuri

If you are using NetworkManager also install foomuuri-firewalld, because it will allow NetworkManager to set the zone the network interface belongs to.

Configuring Foomuuri

Foomuuri can be configured through files in the /etc/foomuuri directory. Foomuuri will read all files which name ends with .conf, so you can split up the configuration in as many files as you want or just put everything in a single file, as you prefer. I like the split configuration files of Shorewall, so I will do something similar here.

Before activating the configuration, always run

# foomuuri check

to validate your configuration. You can start and stop the firewall by starting and stopping the systemd service, you can reload the configuration by running

 # foomuuri reload

You can find the documentation of Foomuuri on the Foomuuri wiki.

Defining zones

The first ting we have to do is define the zones and set which interfaces belongs to which zone. I create /etc/foomuuri/zones.conf:

zone {
  localhost
  public enp1s0
}

I create the zone localhost and the zone public and add the network interface enp1s0 to it. You can add multiple interfaces to a zone by separating them by spaces. If you are using NetworkManager, you don’t have to add the interfaces here and can leave the zone empty. You can configure the firewall zone in NetworkManager and it will set it through foomuuri-firewalld.

Using macros to alias configuration options

Macros can be used to define certain configuration options you want to use multiple times without having to write them completely every time. In practice a lot of macros are already configured which define the configuration for common services. You can see all defined macros by running

# foomuuri list macro

For example the macro imap defines the configuration tcp 143, so that you can just write imap instead of tcp 143 in the configuration. I added a few which were not defined by default in /etc/foomuuri/services.conf:

macro {
	nrpe	tcp 5666
	nmb	udp 137 138 139; tcp 139
}

Macros can be used to configure common subnets. For example I have a file named /etc/foomuuri/subnets.conf:

macro {
	mysubnet		192.168.0.1/24
	othersubnet		192.168.1.1/24
}

I also use macros to create lists of individual hosts, such as all NFS clients which need to access this NFS server in /etc/foomuuri/nfs_clients.conf

macro {
	nfs_clients   192.168.0.1 # web server
	nfs_clients + 192.168.0.2 # gitlab
	nfs_clients + 192.168.0.3 # nextcloud
}

For easy readability, I put every host in a single line, and I add a comment for my own reference. With the + sign I add all next hosts to the macro.

Firewall for incoming connections

To configure Foomuuri to filter incoming connections to my servers, I create a section public-localhost which contains the firewall rules for traffic coming from the public zone to localhost. I put this in the file /etc/foomuuri/public-localhost.conf:

public-localhost {
  dhcp-server
  ssh
  ping  saddr mysubnet
  nmb   saddr mysubnet
  smb   saddr mysubnet
  nfs   saddr nfs_clients
  nrpe  saddr 192.168.0.5
  drop log
}

My server is acting as a DCHP-server, so I use the dhcp-server macro to allow all this traffic, just as I allow all incoming ssh traffic. I allow ping, nmb and smb traffic from mysubnet. Notice that in these rules I use my custom macros nmb and mysubnet. Then I allow nfs from all addresses listed in my macro nfs_clients, and I allow nrpe from a specific IP address. Finally I end with a rule which drops and logs all traffic which has not matched any of the rules before.

Firewall for outgoing connections

I think that filtering outgoing connections is a very effective security hardening measure. In case people with bad intentions get access to your server through a non-root user account, this will severely limit their abilities to move laterally through your network and attack other systems, to run a crypto-miner, or download malware from the Internet.

localhost-public {
  dhcp-client
  nmb uid root
  ntp uid systemd-timesync
  ping uid root daddr mysubnet # dhcpd sometimes pings
  smtp daddr 192.168.0.1 uid postfix
  domain daddr 192.168.0.255 192.168.0.254
  uid root tcp daddr 192.168.0.5 dport 8140 # puppet agent
  uid _apt tcp dport 3142 daddr 192.168.0.6
  uid root ssh daddr 192.168.0.250 # backups
  drop daddr 169.254.169.254 tcp dport 80 # don't fill logs with Puppetlabs facter trying to collect facts from Amazon EC2/Azure
  reject log
}

I allow outgoing connections for different services, and for most services I set the user which can create that connection, and to which host I allow the connection. I explicitly drop without logging connections to 169.254.169.254 port 80, because facter tries to connect to this address every time it runs in order to get some metadata from your cloud service provider. If your system is running on Amazon or Microsoft Azure cloud services, you will probably want to allow this connection instead, so you can then just remove the drop word.

In order to log the UID of the process which tried to establish a rejected connection, in future Foomuuri versions (starting from Foomuuri version 0.22) you can replace the last rule by

reject log log_level "level warn flags skuid"

In current version 0.21, it is possible by setting this globally for all connections. I created /etc/foormuuri/loglevel.conf:

foomuuri {
  log_level "level info flags skuid"
}

Integrating Fail2ban with Foomuuri

I found inspiration for integrating Fail2ban with Foomuuri in issue 9 on the Foomuuri issue tracker.

Create /etc/fail2ban/action.d/foomuuri with these contents:

[Definition]
actionstart =
actionstop  =
actioncheck =
actionban   = /usr/sbin/foomuuri iplist add fail2ban 999d <ip>
actionunban = /usr/sbin/foomuuri iplist del fail2ban <ip>
actionflush = /usr/sbin/foomuuri iplist flush fail2ban

Then set foomuri as the default banaction by creating /etc/fail2ban/jail.d/foomuri.conf:

[DEFAULT]
banaction = foomuuri

Then foomuuri should create the fail2ban iplist. We can configure it to so by creating /etc/foomuuri/fail2ban.conf:

iplist {
	@fail2ban
}

Then I add this rule as first rule to the public-localhost section:

  saddr @fail2ban drop log fail2ban drop

This will drop all connections coming from an address in the iplist fail2ban, and will also log them with prefix fail2ban. If you don’t want this to be logged, just remove log fail2ban.

To ensure that Foomuuri is started before Fail2ban, so that the fail2ban iplist exists before Fail2ban starts to use it, create

/etc/systemd/system/fail2ban.service.d/override.conf:

[Unit]
After=foomuuri.service

After making these changes, first restart Foomuuri and then Fail2ban.

Conclusion

I found Foomuuri easy to use for a system with one network interface. Configuration through the configuration files is easy, also when implementing filtering for outgoing packets. Even though Foomuuri is still a young project, it already has many features and its author is very reactive to discussions and issues on Github. I also found the documentation on the wiki very helpful

I will try to implement Foomuuri on more complex setups in the future, such as on a host for virtual machines of which the network interface is bridged to the main network interface of the host, VPN servers, routers, etc…

Finally I want to thank the Foomuri developer Kim B. Heino and the maintainer of the Debian package Romain Francoise for their work and making this available to the community.

Setting up Wireguard VPN with IPv6

I wanted to set up Wireguard on a VPS, not only to tunnel IPv4 traffic, but also allowing me to tunnel IPv6 traffic. As this is IPv6 of course I preferred not to use NAT, but to assign a public IP address to the client. I read some documentation and blog posts, but I struggled getting it to work. Most tutorials I found on the Internet, create a separate IPv6 subnet for the VPN but I could not get this to work. For some reason, IPv6 traffic successfully went through the VPN tunnel and then exited the VPN gateway, but then any response never reached my VPN gateway and hence also not the client.

I decided to try another way: using an NDP proxy. NDP or the Neighbour Discovery Protocol, is similar to ARP which is used in IPv4. Using this protocol, network devices can discover where on the network a certain IP is located. By letting the VPN gateway answer NDP requests for the VPN client, the gateway would correctly send back all responses to the VPN gateway, which then forwards it to the VPN clients.

Configuring the network on the VPN gateway

I use systemd-networkd to set up the network. It’s the most modern way of network configuration and works the same on all distributions using systemd, but of course you can make the same settings in /etc/network/interfaces or whatever your distribution uses. Of course when making changes to a remote server, make sure you can access a console without needing a working network connection on the server, in case things go wrong and the network connection breaks.

On my VPN server, the public network interface is named ens192 (use the command $ ip addr to find it on your system). My public IPv4 address is www.xxx.yyy.zzz with subnet 255.255.255.0 and gateway ww.xx.yy.1. I have the 64 bit IPV6 prefix aaaa:bbbb:cccc:dddd and the IPv6 gateway is fe80::1.

I set this in /etc/systemd/network/internet.net:

[Match]
Name=ens192

[Network]
Address=aaaa:bbbb:cccc:dddd:0000:0000:0000:0001/64
Gateway=fe80::1
DNS=1.1.1.2
DNS=1.0.0.2
Address= www.xxx.yyy.zzz/24
Gateway=www.xxx.yyy.1
DNS=2606:4700:4700::1112
DNS=2606:4700:4700::1002

In this example I’m using the Cloudflare malware blocking DNS filters, but you can of course just use your ISP’s DNS servers here.

Setting up Wireguard

Run these commands on the Wireguard VPN gateway, and on all clients:

# apt install wireguard-tools
# cd /etc/wireguard
# umask 077
# wg genkey | tee privatekey | wg pubkey > publickey

Then create /etc/wireguard/wg0.conf on the VPN gateway with these contents:

[Interface]
Address = 192.168.7.1,fd42:42:42::1/64
PrivateKey = contents_of_file_privatekey
ListenPort = 51820

#client1
[Peer]
PublicKey = contents_of_publickey_of_client
AllowedIPs = 192.168.7.2/32,aaaa:bbbb:cccc:dddd:ffff::2/128

Add a [Peer] section for every client, and change the both the IPv4 and IPv6 address in AllowedIPs so that they are unique (replace 2 by 3 and so on) .

On the clients, create /etc/wireguard/wg0.conf with these contents:

[Interface]
Address = 192.168.7.2/32,aaaa:bbbb:cccc:dddd:ffff::2/128
PrivateKey = contents_of_privatekey_of_client
DNS = 2606:4700:4700::1112, 2606:4700:4700::1002, 1.1.1.2, 1.0.0.2

[Peer]
PublicKey = contents_of_publickey_of_vpn_gateway
Endpoint = vpngateway.example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0

In the [Interface] section make sure to use the same IP addresses as the ones you have set in the corresponding [Peer] section on the VPN gateway. Set the DNS name (or IP address) of the VPN gateway as Endpoint in the [Peer] section. The hostname’s DNS entry can have both an A and AAAA record. You can replace your DNS servers by your preferred ones. You can also consider running your own DNS server on the VPN gateway.

Make sure that all wg*.conf files on client and server are only readable by root, because they contain private keys.

Configuring the firewall (Shorewall)

I use the Shoreline firewall, Shorewall, as firewall.

Make sure you have shorewall and shorewall6 installed:

# apt install shorewall shorewall6

Shorewall6

First we create a separate zone for our VPN in /etc/shorewall6/zones:

fw firewall
net ipv6
vpn ipv6

Then we configure the network interfaces and assign it to the right zone in /etc/shorewall6/interfaces:

net     NET_IF          tcpflags,routeback,proxyndp,physical=ens192
vpn     wg0		tcpflags,routeback,optional

Then we allow connections from the VPN to the firewall and to the Internet in /etc/shorewall6/policy:

$FW	net		ACCEPT
vpn     net		ACCEPT
vpn     $FW		ACCEPT
net	all		DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all		REJECT		$LOG_LEVEL

Keep in mind that your VPN client will have a public IPv6 address, which is accessible from the Internet. The rule net all DROP protects your VPN clients against access from the Internet.

Then we create some rules which allows access to the SSH server and the Wireguard VPN server from the Internet in /etc/shorewall6/rules:

Invalid(DROP)      net    	$FW		tcp
Ping(DROP)	   net		$FW
ACCEPT		   $FW		net		ipv6-icmp
AllowICMPs(ACCEPT) all		all
ACCEPT		   all		all		ipv6-icmp	echo-request


SSH(ACCEPT)	   net		$FW
ACCEPT		   net		$FW		udp	51820 # Wireguard

We allow some required ICMPv6 message types defined in /usr/share/shorewall/action.AllowICMPs, as well as the echo-request type, which should not be dropped on IPv6.

For security reasons you could even choose to not open the SSH port for the net zone. SSH will only be accessible via the VPN then.

Finally we need to enable IP forwarding in /etc/shorewall6/shorewall6.conf:

IP_FORWARDING=Yes

Then we check whether everything compiles fine and enable and start the service:

# shorewall6 compile
# systemctl restart shorewall6
# systemctl enable shorewall6

Shorewall

For IPv4 we configure Shorewall to use NAT to provide Internet access to the VPN clients.

/etc/shorewall/zones:

fw	firewall
net	ipv4
vpn	ipv4

/etc/shorewall/interfaces:

net     NET_IF          dhcp,tcpflags,logmartians,nosmurfs,sourceroute=0,routefilter,routeback,physical=ens192
vpn	wg0		tcpflags,logmartians,nosmurfs,sourceroute=0,optional,routefilter,routeback

/etc/shorewall/policy:

$FW	net	ACCEPT
vpn	net	ACCEPT
vpn	$FW	ACCEPT
net	all	DROP		$LOG_LEVEL
# The FOLLOWING POLICY MUST BE LAST
all	all	REJECT		$LOG_LEVEL

/etc/shorewall/rules:

# Drop packets in the INVALID state

Invalid(DROP)  net    	        $FW		tcp

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP)	net		$FW

SSH(ACCEPT)	net		$FW
ACCEPT		net		$FW		udp	51820

/etc/shorewall/snat:

MASQUERADE	192.168.7.0/24	NET_IF

/etc/shorewall/shorewall.conf:

IP_FORWARDING=Yes

Compile and load the rules and enable Shorewall permanently:

# shorewall compile
# systemctl restart shorewall
# systemctl enable shorewall

Setting up NDP proxying

Then in order to make sure that the gateway knows that the VPN client aaa:bbb:cccc:dddd::2 is reachable via the VPN gateway, we need to set up NDP proxying. The Neighbor Discovery Protocol is similar to ARP in IPv6.

In a previous version of this guide, I configured NDP proxying in Shorewall6. However, we can directly set this up with systemd-networkd, so this will also work if you don’t use Shorewall6 but another firewall like Firewalld. Furthermore I also experienced problems with NDP proxy settings being lost after some time, requiring a restart of Shorewall6 to make the IPv6 connection over Wireguard work again. I hope this will be fixed by settings this up in systemd-networkd.

Edit again the file /etc/systemd/network/internet.net and in the [NETWORK] section add this

IPv6ProxyNDP=1
IPv6ProxyNDPAddress=aaaa:bbbb:cccc:dddd:ffff::2

If you have more than 1 Wireguard client, you can add multiple IPv6ProxyNDPAddress lines to the file, one for each IPv6 address you want to proxy.

Then restart the systemd-networkd service:

# systemctl restart systemd-networkd

With this command you can check whether they have been set up correctly:

# ip -6 neighbour show proxy

Enabling and testing the VPN

On the server run this to enable the Wireguard server:

# systemctl enable --now wg-quick@wg0

To connect to the VPN, run this on the client:

# systemctl start wg-quick@wg0

Check if you can browse the world wide web. Use these websites to check your IP address and whether you have a working IPv6 connection:

https://test-ipv6.com/

https://ipv6-test.com/

You can also use traceroute and traceroute6 to test whether traffic is correctly going through the VPN tunnel:

# traceroute www.google.com
# traceroute6 www.google.com

Debugging Wireguard

If things don’t work as expected, you can enable debug logging in the Wireguard module with this command:

# echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

Replace +p by -p in order to disable debug logging. You can find the logs in your kernel messages, for example by running

# journalctl -f -k

Also firewall log messages will appear here.

You can use tcpdump to check the traffic on the wire (or in the VPN tunnel). For example to see all ipv6 traffic in the tunnel on the gateway:

# tcpdump -nettti wg0 "ip6"

Sources

Setup WireGuard with global IPv6

Setting up WireGuard IPv6

Reddit: Wireguard doesn’t seem to work with IPv6

Wireguard: enable debug logging to fix network issues

Shorewall6: Proxy NDP

Linux security hardening recommendations

In a previous blog post, I wrote how to secure OpenSSH against brute force attacks. However, what if someone manages to get a shell on your system, despite all your efforts? You want to protect your system from your users doing nasty things? It is important to harden your system further according to the principle of defense in depth in order.

Software updates

Make sure you are running a supported distribution, and by preference the most recent version one. For example, Debian Jessie is still supported, however upgrading to Debian Stretch is strongly recommended, because it offers various security improvements (more recent kernel with new security hardening, PHP 7 with new security related features, etc…)

Install amd64-microcode (for AMD CPU’s) or intel-microcode (for Intel CPU’s) which are needed to protect against hardware vulnerabilities such as Spectre, Meltdown and L1TF. I recommend installing it from stretch-backports in order to have the latest firmware.

Automatic updates and needrestart

I recommend installing unattened-upgrades . You can configure it to just download updates or to download and install them automatically. By default, unattended-upgrades will only install updates from the official security repositories. This way it is relatively safe to let it do this automatically. If you have already installed it, you can run this command to reconfigure it:

# dpkg-reconfigure unattended-upgrades

When you update system libraries, you should also restart all daemons which are using these libraries to make them use the newly installed version. This is exactly what needrestart does. After you have run apt-get, it will check whether there are any daemons running with older libraries, and will propose you to restart them. If you use it with unattended-upgrades, you should set this option in /etc/needrestart/needrestart.conf to make sure that all services which require a restart are indeed restarted:

$nrconf{restart} = 'a';

Up-to-date kernel

Running an up-to-date kernel is very important, because also the kernel can be vulnerable. In the worst case, an outdated kernel can be exploited to gain root permissions. Do not forget to reboot after updating the kernel.

Every new kernel version also contains various extra security hardening measures. Kernel developer Kees Cook has an overview of security related changes in the kernel.

In case you build your own kernel, you can use kconfig-hardened-check to get recommendation for a hardened kernel configuration.null

Firewall: filtering outgoing traffic

It is very obvious to install a firewall which filters incoming traffic. However, have you considered also filtering outgoing traffic? This is a bit more difficult to set up because you need to whitelist all outgoing hosts to which connections are needed (I think of your distribution’s repositories, NTP servers, DNS servers,…), but it is a very effective measure which will help limiting damage in case a user account gets compromised, despite all your other protective efforts.

Ensuring strong passwords

Prevent your users from setting bad passwords by installing libpam-pwquality, together with some word lists for your language and a few common languages. These will be used for verifying that the user is not using a common word as his password. libpam-quality will be enabled automatically after installation with some default settings.

# apt-get install libpam-pwquality wbritish wamerican wfrench wngerman wdutch

Please note that by default, libpam-pwquality will only enforce strong passwords when a non-root user changes its password. If root is setting a password, it will give a warning if a weak password is set, but will still allow it. If you want to enforce it for root too (which I recommend), then add enforce_for_root in the pam_pwquality line in /etc/pam.d/common-password:

password	requisite			pam_pwquality.so retry=3 enforce_for_root

Automatically log out inactive users

In order to log out inactive users, set a timeout of 600 seconds on the Bash shell. Create /etc/profile.d/tmout.sh:

export TMOUT=600
readonly TMOUT

Prevent creating cron jobs

Make sure users cannot set cron jobs. In case an attacker gets a shell on your system, often cron will be used to ensure the malware continues running after a reboot. In order to prevent normal users to set up cron jobs, create an empty /etc/cron.allow.

Protect against fork bombs and excessive logins and CPU usage

Create a new file in /etc/security/limits.d to impose some limits to user sessions. I strongly recommend setting a value for nproc, in order to prevent fork bombs. maxlogins is the maximum number of logins per user, and cpu is used to set a limit on the CPU time a user can use (in minutes):

*	hard	nproc		1024
*	hard	maxlogins 	4
1000:	hard	cpu		180

Hiding processes from other users

By mounting the /proc filesystem with the hidepid=2 option, users cannot see the PIDs of processes by other users in /proc, and hence these processes also become invisible when using tools like top and ps. Put this in /etc/fstab to mount /proc by default with this option:

none	/proc	proc	defaults,hidepid=2	0	0

Restricting /proc/kallsyms

It is possible to restrict access to /proc/kallsyms at boot time by setting 004 permissions. Put this in /etc/rc.local:

chmod 400 /proc/kallsyms

/proc/kallsyms contains information about how the kernel’s memory is laid out. With this information it becomes easier to attack the kernel itself, so hiding this information is always a good idea. It should be noted though that attackers can get this information from other sources too, such as from the System.map files in /boot.

Harden kernel configuration with sysctl

Several kernel settings can be set at run time using sysctl. To make these settinsg permanent, put these settings in files with the .conf extension in /etc/sysctl.d.

It is possible to hide the kernel messages (which can be read with the dmesg command) from other users than root by setting the sysctl kernel.dmesg_restrict to 1. On Debian Stretch and later this should already be the default value:

kernel.dmesg_restrict = 1

From Linux kernel version 4.19 on it’s possible to disallow opening FIFOs or regular files not owned by the user in world writable sticky directories. This setting would have prevented vulnerabilities found in different user space programs the last couple of years. This protection is activated automatically if you use systemd version 241 or higher with Linux 4.19 or higher. If your kernel supports this feature but you are not using systemd 241, you can activate it yourself by setting the right sysctl settings:

fs.protected_regular = 1
fs.protected_fifos = 1

Also check whether the following sysctl’s have the right value in order to enable protection hard links and symlinks. These work with Linux 3.6 and higher, and likely will already be enabled by default on your system:

fs.protected_hardlinks = 1
fs.protected_symlinks = 1

Also by default on Debian Stretch only root users can access perf events:

kernel.perf_event_paranoid = 3

Show kernel pointers in /proc as null for non-root users:

kernel.kptr_restrict = 1

Disable the kexec system call, which allows you to boot a different kernel without going through the BIOS and boot loader:

kernel.kexec_load_disabled = 1

Allow ptrace access (used by gdb and strace) for non-root users only to child processes. For example strace ls will still work, but strace -p 8659 will not work as non-root user:

kernel.yama.ptrace_scope = 1

The Linux kernel includes eBPF, the extended Berkeley Packet Filter, which is a VM in which unprivileged users can load and run certain code in the kernel. If you are sure no users need to call bpf(), it can be disabled for non-root users:

kernel.unprivileged_bpf_disabled = 1

In case the BPF Just-In-Time compiler is enabled (it is disabled by default, see sysctl net/core/bpf_jit_enable), it is possible to enable some extra hardening against certain vulnerabilities:

net.core.bpf_jit_harden = 2

Take a look at the Kernel Self Protection Project Recommended settings page to find an up to date list of recommended settings.

Lynis

Finally I want to mention Lynis, a security auditing tool. It will check the configuration of your system, and make recommendations for further security hardening.

Further ideas

“The Great Firewall of Belgium” active

Since today, Belgium has got it’s own version of “The Great Firewall of China”. The biggest Belgian ISPs are blocking access to several web sites, often related to child porn.

The idea already existed for several months, but the implementation was probably accelerated after a Dutch guy recently created a website where he posted detailed personal information about child abusers in Belgium. While publishing such detailed private information is forbidden in Belgium, it was very difficult to take real action against the website, because it operated from abroad.

So now this website is not accessible anymore from most Belgian ISPs. People who try to access this website, get redirected to a web page which explains that the web site is not accessible because it is considered illegal in Belgium.

Technically, it’s not really a firewall. The redirection happens on the DNS level. Instead of returning the real IP of the server, the DNS servers now return the address of a server in Belgium containing the warning page.

While I agree (like every sane person) that things like child’s pornography are completely sick and should be severely acted against, I think that creating a blacklist of websites which people cannot visit any more is a very dangerous precedent. It’s not clear at all how it is decided to put a website on the blacklist. Currently this is not based on a judge’s decision after an official juridical procedure. Also how long will it take until someone makes a mistake in the list and blocks half of the Internet by error (which is not unrealistic, it happened to Google recently!), or worse, until sites of political dissidents are blocked? For this reason, I have decided to stop my internal Bind DNS server at home from forwarding its requests to my ISPs DNS and instead I let it do iterative recursion now.  I read that many others are starting to use OpenDNS now, but this seems to have privacy issues by itself too.