smtpIptablesBlocker

This script is the result of my search for a way to make my mail server more resilient against spam waves. Take a look at following links(1,2,3) if you want to know more about the history/process to this solution. I use courier as MTA and IMAP backend for it.

Introduction

To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff. The ratio of spam which gets through is quite low but thats not the problem at this point. The problem is the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically (and courier creates a process for each). For example look at the values which show the amount of open SMTP connections at the given moment.


Mon Jul 27 19:12:17 CEST 2009 4
....
Mon Jul 27 19:12:32 CEST 2009 7
Mon Jul 27 19:12:37 CEST 2009 13
Mon Jul 27 19:12:42 CEST 2009 9
Mon Jul 27 19:12:47 CEST 2009 8
Mon Jul 27 19:12:52 CEST 2009 50
Mon Jul 27 19:12:57 CEST 2009 88
Mon Jul 27 19:13:02 CEST 2009 101
Mon Jul 27 19:13:07 CEST 2009 120
.....
Mon Jul 27 19:13:33 CEST 2009 163
Mon Jul 27 19:13:38 CEST 2009 167
.....
Mon Jul 27 19:17:24 CEST 2009 301
Mon Jul 27 19:17:30 CEST 2009 301
Mon Jul 27 19:17:35 CEST 2009 301
Mon Jul 27 19:17:40 CEST 2009 301


300 is the limit of active connections. At this point the mail server adds following to the log and does not accept any new connections.


Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.


This script now solves my problem. It looks through the maillog and adds every DNS RBL listed IP to the firewall for 10 minutes. After that it just terminates the process which handles the connection. This leads to a reduced process number at once and makes space for a new one. A clean solution would implement my complete script in the mta itself – basically adding the IP to the firewall and terminating the smtp handling process. But this works also stable so far until something like this is added to courier (if it ever will be)
.

Install – Iptables stuff

First load the iptables module somewhere at boot time with an option to allow more IP addresses stored.


modprobe ipt_recent ip_list_tot=1000


I think 1000 is quite on the low end, as the spam waves easily reach them. Then add following iptables commands to your firewall script.


# build sub chain
$iptables -N SPAMMER
# move all incomming smtp traffic there
$iptables -A INPUT -p tcp --dport 25 -j SPAMMER
# check if the source ip is already in the list, if so give it another 600 sec and drop the packages
$iptables -A SPAMMER -m recent --name spammer --update --seconds 600 -j DROP


As you can see we drop the packets for 10 minutes. If a packet is send within that 10min the time period starts again.

You can test your setup by doing the following.


echo +1.1.1.1 >/proc/net/ipt_recent/spammer


and take a look at


cat /proc/net/ipt_recent/spammer


Replace 1.1.1.1 by an IP address of a spammer (just look in your logfile 😉 ) and see it working.

Install Daemon

Ok, now that we have the kernel/iptables part we need the script which adds the IP addresses of spammers on a DNS RBL after the first 5xx to the ipt_recent list.

Download the script here: smtpIptablesBlocker-0.1.tar.bz2

Copy the smtpIptablesBlocker.py deamon.py to /usr/local/sbin/ and edit the smtpIptablesBlocker.py file to reflect the location of your mail.err log file.

Now we need to link it to the init.d and get it in the runlevels:


#ln -s /usr/local/sbin/smtpIptablesBlocker.py /etc/init.d/smtpIptablesBlocker


on Debian/Ubuntu use following


# update-rc.d smtpIptablesBlocker defaults 21
Adding system startup for /etc/init.d/smtpIptablesBlocker ...
/etc/rc0.d/K21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc1.d/K21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc6.d/K21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc2.d/S21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc3.d/S21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc4.d/S21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker
/etc/rc5.d/S21smtpIptablesBlocker -> ../init.d/smtpIptablesBlocker

Take it to a test ride with


/etc/init.d/smtpIptablesBlocker start
tail -f /var/log/smtpIptablesBlocker.log


On Debian/Ubuntu you need to tell the syslogd that it should not do the log rotate, as this should be handled by logrotate, which is able to inform us about it.

Replace in /etc/cron.daily/sysklogd:

logs=$(syslogd-listfiles)

by

logs=$(syslogd-listfiles -s "(mail\.err|smtpIptablesBlocker\.log)")

and in /etc/cron.weekly/sysklogd

logs=$(syslogd-listfiles --weekly)

by

logs=$(syslogd-listfiles --weekly -s "(mail\.err|smtpIptablesBlocker\.log)")

Now we can add the logrotate config (/etc/logrotate.d/courier)

/var/log/mail.err /var/log/smtpIptablesBlocker.log {
    notifempty
    delaycompress
    sharedscripts
    prerotate
        /etc/init.d/smtpIptablesBlocker stop
    endscript
    postrotate
        /etc/init.d/sysklogd reload-or-restart
        /etc/init.d/smtpIptablesBlocker start
    endscript
}

And the result? Just look at this really big spam wave – and we’re able to scale it.


Tue Jul 28 14:36:44 CEST 2009 1
Tue Jul 28 14:36:49 CEST 2009 1
Tue Jul 28 14:36:54 CEST 2009 34
Tue Jul 28 14:36:59 CEST 2009 56
Tue Jul 28 14:37:04 CEST 2009 52
Tue Jul 28 14:37:09 CEST 2009 60
Tue Jul 28 14:37:14 CEST 2009 87
Tue Jul 28 14:37:19 CEST 2009 126
Tue Jul 28 14:37:24 CEST 2009 128
Tue Jul 28 14:37:29 CEST 2009 140
Tue Jul 28 14:37:34 CEST 2009 138
Tue Jul 28 14:37:39 CEST 2009 143
Tue Jul 28 14:37:44 CEST 2009 161
Tue Jul 28 14:37:49 CEST 2009 198
Tue Jul 28 14:37:54 CEST 2009 208
....
Tue Jul 28 14:41:11 CEST 2009 131
Tue Jul 28 14:41:16 CEST 2009 147
Tue Jul 28 14:41:21 CEST 2009 134
Tue Jul 28 14:41:26 CEST 2009 133
Tue Jul 28 14:41:31 CEST 2009 128
Tue Jul 28 14:41:36 CEST 2009 103
Tue Jul 28 14:41:41 CEST 2009 74
Tue Jul 28 14:41:47 CEST 2009 98
Tue Jul 28 14:41:52 CEST 2009 91
....
Tue Jul 28 14:43:27 CEST 2009 28
Tue Jul 28 14:43:32 CEST 2009 26
Tue Jul 28 14:43:37 CEST 2009 20
Tue Jul 28 14:43:42 CEST 2009 15
Tue Jul 28 14:43:47 CEST 2009 14
Tue Jul 28 14:43:52 CEST 2009 14
Tue Jul 28 14:43:57 CEST 2009 15
Tue Jul 28 14:44:02 CEST 2009 11
Tue Jul 28 14:44:07 CEST 2009 6
Tue Jul 28 14:44:12 CEST 2009 9
Tue Jul 28 14:44:17 CEST 2009 7


As you see 300 is never reached. Our server is the whole time reachable for our customers.