Layer 3 Controls vs Layer 7 Exposures: The fail2ban Case

IT Security is not a simple area. You need more than knowing how to configure a firewall rule. Indeed, in my opinion, you must have a depth knowledge of the network stack (TCP and ISO), details about the involved protocols (IP headers, HTTP headers, LDAP authentication just to mention some) and configuration details of the operative system, the involved software (knowing MariaDB options if you are doing database security) and any other theoretical concept about your environment. And this is only for the technical part.

If you want to go further, you should understand security concepts such as risk, vulnerability, exposure, control and others.

At this point, you can have a clear vision about what is the best doing in your strategy. With all this said, I will talk about a classic error when taking calls about implementing security controls.

A Firewall is not always the Solution of Everything

First, I must say for me a proxy is not considered a firewall. Some authors state that a proxy can be considered a layer 7 firewall. In this article, a firewall could be a Netscreen device, a Checkpoint or an IPTables in Linux. A proxy is something like Squid or even better a mod_security for Apache. The big difference is the layer where they operate. Firewalls work on layer 2, 3 and 4; the maximum control you can have by controlling the port. Some firewalls, like IPTables, try to work around this with the string module, where you can put some criteria based on the payload, but again, they don't understand the protocol, it is just a dummy machine hitting a blind condition. Proxies, on the other hand, they operate on layer 7; they fully understand the protocol. Squid is an excellent example. Squid can understand the HTTP protocol and you can create rules based on HTTP elements such as authenticated user, the path of the files, POST payload, cookies and many other things.

I have always stated that to fully cover a vulnerability exposition, you must use a control at the same level. If the vulnerability is in layer 7, then your control should operate in the same layer.

When I was studying for my CISM certification, I remember one of the questions was about due diligence. If you, as a security manager, don't have the full picture of your IT environment, there is a minimal set of task to implement. Firewall is one of them. Which I believe it is okay, but the problem goes beyond this scenario.

One example of the issue I am trying to explain happens when IT Security Managers install an RBL to reduce the exposition of having the mailboxes spammed. In my opinion, this is the perfect example where a layer 3 tries to mitigate the exposition of a layer 7 (and some might say 8) vulnerability. The problem is the way RBL's are implemented. Usually, you configure you MTA (Postfix, Sendmail, Exim) to compare the source IP (layer 3) against it. This approach automatically discards possible valid email (this is called a false positive) and you still can have incoming spam (this is called a false negative). The right solution would it be the installation of an SMTP proxy that fully understands the protocol and literally it can read the email. The good news is that tools like SpamAssassin and Amavisd already exist; this kind of tools have a better understanding. Indeed, they can still use RBL as a parameter, but it is not a binary decision on the RBL, more factors including the wording of the text will determinate if this is a SPAM or HAM.

Other IT Security Manager are more aggressive, and they just start blocking the port 25/tcp based on the source IP they detect. I don't need to explain the issue here.

Blocking rules to stop access to some services based on the source IP is good, but not everybody has a static IP. Even worst, almost every one is sharing a dynamic public IP somehow.

No Proxy? Fail2ban, the Tool that tries to fill this Gap

Fail2ban is a tool that operates with firewalls (layer 3 again) but the way it is though makes things less dramatic. Fail2ban it is a quarantine server with log parser that triggers commands (usually a firewall rule) based on regular expressions. This means that if regular expression matches, the software is smart enough to detect the source IP and executes an action for a determined period of time (period).

Fail2ban will allow you to close the gap between layer 3 controls and layer 7 exposures. If you don't have a proxy, this is a viable solution.

Fail2ban is not all happiness, it is very good for some technologies like stopping SSH brute force attacks. Others technologies like the VoIP, specifically speaking of the SIP protocol, have a very different story. Because the nature of SIP, signals will not be always authenticated and when they are, users share a single public dynamic IP. This is very common in business offices. When the call is not authenticated (aka IP authenticated), the security measures that can be placed are in terms of the SIP payload (for example invalid INVITES or special characteristics in the payload), when authenticated if a user lost the password, the IP phone will keep retrying more than once before giving up. This retrying usually is the most common reason why a whole company could get banned because of the NAT.

Fail2ban covers this issue of not having a static IP. If someone is attacking the server (even unintentionally), the logs will report some activity and fail2ban will put in quarantine the attacker. Because we are protecting based on the fact that it is a dynamic IP, the offending address will be blocked for a period of time; new users who get that IP in the future will not be affected as the quarantine will end.

The remaining issue is the popularity of the NAT. NAT is a nasty solution because IP (version 4) are not enough; we have IPv6 now, but the full transition to it will take some time (not all residential ISP are offering IPv6 yet). So because of this, if someone does the most common issue like trying more than three times a bad password, fail2ban will automatically ban the public IP and all the office will suffer.

Should I do Whitelists?

Whitelists are a capability that fail2ban has to ignore the quarantine for specific source IP's. But again, we are working in the case where you don't have a static IP. So, it is pretty useless. Fortunately for everyone, there is another way.

Fail2ban has something I will call dynamic white lists, it is a mechanism that checks an offending IP in real time to know if it should be whitelisted or not. To do this, you should have a way to build a temporary database of authorized IP's. Some packages, like Squid, FreeSWITCH or Apache have a way to extract a tuple of username-password or any other criteria where an IP is evaluated. I don't doubt you can pass more variables, but I haven't tried it.

According to the documentation I have read, you just need to configure the jail with the following line:

ignorecommand = /path/to/script <ip>

The script will get the offending IP as a parameter. The script must return an exit level of 0 if the IP should be whitelisted.

For example, I have done a script that checks FreeSWITCH registered users and if a jail tries to ban an IP that is has a valid registration, it will be whitelisted. Please note that this will fill the gap, but it is more a workaround rather than a true solution. For me, the solution is done by implementing a control that operates at the same level.

Good luck!