Author: Frederik

Security hardening the OpenSSH server is one of the first things that should be done on any newly installed system. Brute force attacks on the SSH daemon are very common and unfortunately I see it going wrong all too often. That’s why I think it’s useful to give a recapitulation here with some best practices, even though this should be basic knowledge for any system administrator.

Firewall

The first thing to think about: should the be SSH server be accessible from the whole world, or can we limit it to certain IP addresses or subnets. This is the most simple and effective form of protection: if your SSH daemon is is only accessible from specific IP addresses, then there is no risk any more from attacks coming from elsewhere.

I prefer to use Shorewall as a firewall, as it’s easy to configure. Be sure to also configure shorewall6 if you have an IPv6 address.

However as defense in depth is an essential security practice, you should not stop here even if you protected your SSH daemon with a firewall. Maybe your firewall one day fails to come up at boot automatically, leaving your system unprotected. Or maybe one day the danger comes from within your own network. That’s why in any case you need to carefully review the next recommendations too.

SSHd configuration

Essential security settings

The SSH server configuration can be found in the file /etc/ssh/sshd_config. We review some essential settings:

• PermitRootLogin: I strongly recommend setting this to No. This way, you always log in to your system with a normal user account. If you need root access, use su or sudo. The root account is then protected from brute force attacks. And you can always easily find out who used the root account.
• PasswordAuthentication: This setting really should be No. You will first need to add your SSH public key to your ~/.ssh/authorized_keys . Disabling password authentication is the most effective protection against brute force attacks.
• X11Forwarding: set this to No, except if your users need to be able to run X11 (graphical) applications remotely.
• AllowTcpForwarding: I strongly recommend setting this to No. If this is allowed, any user who can ssh into your system, can establish connections from the client to any other system using your host as a proxy. This is even the case even if your users can only use SFTP to transfer files. I have seen this being abused in the past to connect to the local MTA and send spam via the host this way.
• PermitOpen: this allows you to set the hosts to which TCP forwarding is allowed. Use this if you set AllowTcpForwarding to indicate to which hosts TCP forwarding is limited.
• ClientAliveInterval, ClientAliveCountMax: These values will determine when a connection will be interrupted when it’s unresponsive (for example in case of network problems). I set ClientAliveInterval to 600 and ClientAliveCountMax to 0. Note that this does not drop the connection when the user is simply inactive. If you want to set a timeout for that, you can set the TMOUT environment variable in Bash.
• MaxAuthTries: the maximum number of authentication attempts permitted per connection. Set this to 3.
• AllowUsers: only the users in this space separated list are allowed to log in. I strongly recommend using this (or AllowGroups) to whitelist users that can log in by SSH. It protects against possible disasters when a test user or a system users with a weak password is created.
• AllowGroups: only the users from the groups in this space separated list are allowed to log in.
• DenyUsers: users in this space separated list are not allowed to log in
• DenyGroups: users from the groups in this space separated list are not allowed to log in.
• These values should already be fine by default, but I recommend verifying them: PermitEmptyPasswords (no), UsePrivilegeSeparation (sandbox), UsePAM (yes), PermitUserEnvironment (no), StrictModes (yes), IgnoreRhosts (yes)

So definitely disable PasswordAuthentication and TCP and X11 forwarding by default and use the AllowUsers or AllowGroups to whitelist who is allowed to log in by SSH.

Match conditional blocks

With Match conditional blocks you can modify some of the default settings for certain users, groups or IP addresses. I give a few examples to illustrate the usage of Match blocks.

To allow TCP forwarding to a specific host for one user:

Match User username
        AllowTcpForwarding yes
        PermitOpen 192.168.0.120:8080

To allow PasswordAuthentication for a trusted IP address (make sure the user has a strong password, even if you trust the host!) :

Match Address 192.168.0.20
        PasswordAuthentication yes

The Address can also be a subnet in CIDR notation, such as 192.168.0.0/24.

To only allow SFTP access for a group of users, disabling TCP, X11 and streamlocal forwarding:

Match group sftponly
        ForceCommand internal-sftp
        AllowTcpForwarding no
        X11Forwarding no
        AllowStreamLocalForwarding no

chroot

You can chroot users to a certain directory, so that they cannot see and access what’s on the file system outside that directory. This is a a great way to protect your system for users who only need SFTP access to a certain location. For this to work, you need to make the user’s home directory being owned by root:root. This means they cannot write directly in their home directory. You can create a subdirectory within the user’s home directory with the appropriate ownership and permissions where the user can write into. Then you can use a Match block to apply this configuration to certain users or groups:

Match Group chrootsftp
        ChrootDirectory %h
        ForceCommand internal-sftp
        AllowTcpForwarding no
        X11Forwarding no
        AllowStreamLocalForwarding no

If you use authentication with keys, you will have to set a custom location for the authorized_keys file:

AuthorizedKeysFile /etc/ssh/authorized_keys/%u .ssh/authorized_keys

Then the keys for every user have to be installed in a file /etc/ssh/authorized_keys/username

Fail2ban

Fail2ban is a utility which monitors your log files for failed logins, and will block IPs if too many failed log in attempts are made within a specified time. It cannot only watch for failed login attempts on the SSH daemon, but also watch other services, like mail (IMAP, SMTP, etc.) services, Apache and others. It is a useful protection against brute force attacks. However, versions of Fail2ban before 0.10.0, only support IPv4, and so don’t offer any protection against attacks from IPv6 addresses. Furthermore, attackers often slow down their brute force attacks so that they don’t trigger the Fail2ban threshold. And then there are distributed attacks: by using many different source IPs, Fail2ban will never be triggered. For this reasons, you should not rely on Fail2ban alone to protect against brute force attacks.

If you want to use Fail2ban on Debian Stretch, I strongly recommend using the one from Debian-backports, because this version has IPv6 support.

# apt-get install -t stretch-backports fail2ban python3-pyinotify python3-systemd

I install python3-systemd in order read the log messages from Systemd’s Journal, while python3-pyinotify is needed to efficiently watch log files.

First we will increase the value for dbpurgeage which is set to 1 day in /etc/fail2ban/fail2ban.conf. We can do this by creating the file /etc/fail2ban/fail2ban.d/local.conf:

[Definition]
dbpurgeage = 10d

This lets us ban an IP for a much longer time than 1 day.

Then the services to protect, the thresholds and the action to take when these are exceeded are defined in /etc/fail2ban/jail.conf. By default all jails, except the sshd jail, are disabled and you have to enable the ones you want to use. This can be done by creating a file /etc/fail2ban/jail.d/local.conf:

[DEFAULT]
banaction = iptables-multiport
banaction_allports = iptables-allports
destemail = email@example.com
sender = root@example.com

[sshd]
mode = aggressive
enabled = true
backend = systemd

[sshd-slow]
filter   = sshd[mode=aggressive]
maxretry = 10
findtime = 3h
bantime  = 8h
backend = systemd
enabled = true

[recidive]
enabled=true
maxretry = 3
action = %(action_mwl)s

First we override some default settings valid for all jails. We configure it to use iptables to block banned users. If you use Shorewall as firewall, then set banaction and banaction_allports to shorewall in order to use the blacklist functionality of Shorewall. In that case, read the instructions in /etc/fail2ban/action.d/shorewall.conf to configure Shorewall to also block established blacklisted connections. Other commonly used values for banactions and banactions_allports are ufw and firewallcmd-ipset, if you use UFW respectively Firewalld. We also define the sender address and destination address where emails should be sent when a host is banned.

Then we set up 3 jails. The sshd and recidive jail are jails which are already defined in /etc/fail2ban/jails.conf and we enable them here. The sshd jail will give a 10 minute ban to IPs which do 5 unsuccessful login attempts on the SSH server in a time span of 10 minutes. The recidive jail gives a one week ban to IPs getting banned 3 times by another Fail2ban jail in a time span of 1 day. Furthermore I define another jail sshd-slow, which gives a 8 hour ban to IPs doing 10 failed attempts on the SSH server in a time span of 3 hours. This catches many attempts which try to evade the default Fail2ban settings by slowing down their brute force attack. In both the sshd and sshd-slow jails I use the aggressive mode which catches more errors, such as probes without trying to log in, and attempts with wrong (outdated) ciphers. See /etc/fail2ban/filter.d/sshd.conf for the complete lists of log message it will search for. The recidive jail will sent a mail to the defined address in case a host gets banned. I enable this only for recidive in order not to receive too much e-mails.

Two-factor authentication

It is possible to enable two-factor authentication (2FA) using the libpam-google-authenticator package. Then you can use an application like FreeOTP+ (Android), AndOTP (Android), Authenticator (iOS), KeepassXC (Linux) to generate the time based token you need to log in.

First install the required PAM module on your SSH server:

# apt-get install libpam-google-authenticator

Then edit the file /etc/ssh/sshd_config:

ChallengeResponseAuthentication yes
AuthenticationMethods publickey keyboard-interactive:pam

You can also put this in a Match block to only enable this for certain users or groups.

This will allow you to log in with either key based authentication, either by password and your time-based token.

Now you need to set up the new secret for the user account you want to use OTP authentication using the google-authenticator command. Run it as the user. Choose time-based authentication tokes, disallow multiple uses of the same authentication token, and don’t choose to increase the time window to 4 minute and enable rate-limiting.

$ google-authenticator
Do you want authentication tokens to be time-based (y/n) y
                                                          
Do you want me to update your "/home/username/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds. In order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with
poor time synchronization, you can increase the window from its default
size of +-1min (window size of 3) to about +-4min (window size of
17 acceptable tokens).
Do you want to do so? (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

Now enter the code given by this command in your OTP client or scan the QR code.

Edit the file /etc/pam.d/sshd and add this line:

auth required pam_google_authenticator.so noskewadj

You need to make sure to add this line before the line

@include common-auth

Otherwise an attacker can still brute force the password, and then abuse it on other services. That is because of the auth requisite pam_deny.so line in common-auth: this will immediately return a failure message when the password is wrong. The time-based token would only be asked when the password is correct.

The noskewadj option increases security by disabling the option to automatically detect and adjust time skew between client and server.

Now restart the sshd service, and in another shell, try the OTP authentication. Don’t close your existing SSH connection yet, because otherwise you might lock yourself out if something went wrong.

The biggest disadvantage of pam_googleauthenticator is that it allows every individual user to set values for the window size, rate limiting, whether to use HOTP or TOTP, etc. By modifying some of these, the user can reduce the security of the one-time-password. For this reason, I recommend only enabling this for users you trust.