in Technology

Using Fail2ban with Nginx and UFW

I was recently hit with a denial of service attack on this very blog, and it hold up surprisingly well. The only reason I found out about it was when, after the attack was well under way, the WordPress Jetpack plugin alerted me about my site being down.

The reason it went down had in fact nothing to do with Nginx itself, but a crash in the upstream PHP handler I use, being a WordPress blog and all.

The first thing I had to do was to block the attack in some way. Tailing the Nginx access log revealed that the attack originated from a single German IP address, which made things a lot easier. All I had to do was to block access on the firewall level.

Using the built-in ufw command lets one easily modify firewall rules without having to deal with iptables directly, so blocking the IP address was just a matter of:

$ sudo ufw insert 1 deny from [IP]

The attack stopped immediately and I could very well be done here, but I wanted to automate this so I wouldn’t have to deal with the same thing ever again. This is where Fail2ban comes in.

Fail2ban monitors log files for specific patterns matched using regular expressions, and can perform specific actions on the matched lines. I just needed Nginx to tell me when it noticed an unusual amount of traffic from one specific host, and that feature just happens to be available as a plugin.

The limit-req plugin is well suited for this type of automation, and all that is required for Nginx to warn when a client is crossing the threshold, are the following lines in your nginx.conf config:

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
    limit_req zone=one burst=50 nodelay;
}

This will keep a 10 Mb state cache with up to 10 normal requests per second and temporary bursts with up to 50 requests. You may need to tweak these numbers depending on your site content.

The next step is to make Fail2ban aware of this log file and to trigger a firewall rule when encountering the predetermined log. Some defaults are set in the /etc/fail2ban/jail.local file:

[DEFAULT]
ignoreip = 127.0.0.1/8
banaction = ufw
maxRetry = 5
findtime = 600
bantime = 7200

The next step is to define the ufw ban action referenced above. Open up /etc/fail2ban/action.d/ufw.conf and paste the following configuration stanza:

[Definition]
actionstart =
actionstop =
actioncheck =
actionban = ufw insert 1 deny from <ip> to any
actionunban = ufw delete deny from <ip> to any

This will insert the deny rule on the top of the ufw ruleset. There is also an unban action which will trigger after a defined timeout occurs.

The next step is to define the filter which will enable fail2ban to see when Nginx finds and offending client. Open up /etc/fail2ban/filter.d/nginx-req-limit.conf and paste the following stanza:

[Definition]
failregex = ^.*limiting requests, excess:.* by zone.*client: <HOST>, server.*$
ignoreregex =

Finally, we add the jail which ties everything together. Open up /etc/fail2ban/jail.d/nginx-req-limit.confand paste the following:

[nginx-req-limit]
enabled = true
filter = nginx-req-limit
action = ufw
logpath = /var/log/nginx/*error.log

To activate the new configuration, just do a sudo service fail2ban reload and the same for Nginx using sudo service nginx reload and you should be all set.

Testing this could be problematic if you are unable to do so from a third-party IP address, since you will be blocked if the test passes. If you do have a secondary Linux server or equivalent, using the standard Apache Benchmark (ab) command will suffice. Run the following command to test the configuration:

ab -c 100 -n 100 http://[your site]/

If you do a sudo ufw status you will see the banned IP at the top. To remove it, just run sudo ufw delete 1.

Further improvements can be made by for example letting you know by email when an IP address has been banned. Tweaking this for other firewall wrappers than ufw should be trivial as well as long as there is a command-line for it.

Fail2ban can be used in a wide variety of ways, not just for banning IP addresses, and I’m sure that I will return to this topic in the future when something needs to be automated.