The Search for reduced SPAM load – Part 3

July 29, 2009

Take a look at

• The Search for reduced SPAM load – Part 1
• The Search for reduced SPAM load – Part 2

to understand purpose of this series and what I’m looking for. As only firewalling spammers which are in a DNS RBL after they got a 5xx didn’t worked as hoped I had an other idea.

Instead of only firewalling the spammer I thought as there will be no packet from the spammer within the 30sec timeout why not just terminate the process which handles the connection. This leads to a reduced process number at once and makes space for a new one. A clean solution would implement my complete script in the mta itself – basically adding the IP to the firewall and terminating the smtp handling process. But for a mere test it would be easy to extend my script to kill the current process.

This python script (watchForSpammers2.py) does exactly that – It extends the old script by searching through the process list for a submit progress which handles the spammer connection. It follows the ppid and kills with a SIGTERM the parent courieresmtpd process.

You say thats a hard method? And you asked yourself if it works – yes it does.

After some testing I implemented it on the productive server and I’ve it running for a few days now and I didn’t reach the maximum of 300 connections since.

e.g. take a look at this spam wave

Tue Jul 28 08:30:51 CEST 2009 5
Tue Jul 28 08:30:56 CEST 2009 8
Tue Jul 28 08:31:01 CEST 2009 7
Tue Jul 28 08:31:06 CEST 2009 2
Tue Jul 28 08:31:11 CEST 2009 3
Tue Jul 28 08:31:16 CEST 2009 4
Tue Jul 28 08:31:21 CEST 2009 13
Tue Jul 28 08:31:26 CEST 2009 100
Tue Jul 28 08:31:31 CEST 2009 77
Tue Jul 28 08:31:36 CEST 2009 48
Tue Jul 28 08:31:41 CEST 2009 31
Tue Jul 28 08:31:46 CEST 2009 32
Tue Jul 28 08:31:51 CEST 2009 39
Tue Jul 28 08:31:56 CEST 2009 34
Tue Jul 28 08:32:01 CEST 2009 19
Tue Jul 28 08:32:06 CEST 2009 19
Tue Jul 28 08:32:11 CEST 2009 18
Tue Jul 28 08:32:16 CEST 2009 18
Tue Jul 28 08:32:21 CEST 2009 16
Tue Jul 28 08:32:26 CEST 2009 15
Tue Jul 28 08:32:31 CEST 2009 13
Tue Jul 28 08:32:36 CEST 2009 14
Tue Jul 28 08:32:41 CEST 2009 19
Tue Jul 28 08:32:46 CEST 2009 53
Tue Jul 28 08:32:51 CEST 2009 58
Tue Jul 28 08:32:56 CEST 2009 38
Tue Jul 28 08:33:01 CEST 2009 30
Tue Jul 28 08:33:06 CEST 2009 28


or the biggest in the last days


Tue Jul 28 14:36:19 CEST 2009 3
Tue Jul 28 14:36:24 CEST 2009 4
Tue Jul 28 14:36:29 CEST 2009 2
Tue Jul 28 14:36:34 CEST 2009 1
Tue Jul 28 14:36:39 CEST 2009 3
Tue Jul 28 14:36:44 CEST 2009 1
Tue Jul 28 14:36:49 CEST 2009 1
Tue Jul 28 14:36:54 CEST 2009 34
Tue Jul 28 14:36:59 CEST 2009 56
Tue Jul 28 14:37:04 CEST 2009 52
Tue Jul 28 14:37:09 CEST 2009 60
Tue Jul 28 14:37:14 CEST 2009 87
Tue Jul 28 14:37:19 CEST 2009 126
Tue Jul 28 14:37:24 CEST 2009 128
Tue Jul 28 14:37:29 CEST 2009 140
Tue Jul 28 14:37:34 CEST 2009 138
Tue Jul 28 14:37:39 CEST 2009 143
Tue Jul 28 14:37:44 CEST 2009 161
Tue Jul 28 14:37:49 CEST 2009 198
Tue Jul 28 14:37:54 CEST 2009 208
Tue Jul 28 14:37:59 CEST 2009 187
Tue Jul 28 14:38:04 CEST 2009 175
Tue Jul 28 14:38:09 CEST 2009 140
Tue Jul 28 14:38:15 CEST 2009 144
Tue Jul 28 14:38:20 CEST 2009 150
Tue Jul 28 14:38:25 CEST 2009 190
Tue Jul 28 14:38:30 CEST 2009 182
Tue Jul 28 14:38:35 CEST 2009 167
Tue Jul 28 14:38:40 CEST 2009 176
Tue Jul 28 14:38:45 CEST 2009 190
Tue Jul 28 14:38:50 CEST 2009 206
Tue Jul 28 14:38:55 CEST 2009 199
Tue Jul 28 14:39:00 CEST 2009 197
Tue Jul 28 14:39:05 CEST 2009 199
Tue Jul 28 14:39:10 CEST 2009 168
Tue Jul 28 14:39:15 CEST 2009 199
Tue Jul 28 14:39:20 CEST 2009 210
Tue Jul 28 14:39:25 CEST 2009 201
Tue Jul 28 14:39:30 CEST 2009 195
Tue Jul 28 14:39:35 CEST 2009 216
Tue Jul 28 14:39:40 CEST 2009 203
Tue Jul 28 14:39:45 CEST 2009 200
Tue Jul 28 14:39:50 CEST 2009 196
Tue Jul 28 14:39:56 CEST 2009 189
Tue Jul 28 14:40:01 CEST 2009 180
Tue Jul 28 14:40:06 CEST 2009 176
Tue Jul 28 14:40:11 CEST 2009 173
Tue Jul 28 14:40:16 CEST 2009 177
Tue Jul 28 14:40:21 CEST 2009 165
Tue Jul 28 14:40:26 CEST 2009 170
Tue Jul 28 14:40:31 CEST 2009 164
Tue Jul 28 14:40:36 CEST 2009 167
Tue Jul 28 14:40:41 CEST 2009 151
Tue Jul 28 14:40:46 CEST 2009 147
Tue Jul 28 14:40:51 CEST 2009 139
Tue Jul 28 14:40:56 CEST 2009 140
Tue Jul 28 14:41:01 CEST 2009 136
Tue Jul 28 14:41:06 CEST 2009 131
Tue Jul 28 14:41:11 CEST 2009 131
Tue Jul 28 14:41:16 CEST 2009 147
Tue Jul 28 14:41:21 CEST 2009 134
Tue Jul 28 14:41:26 CEST 2009 133
Tue Jul 28 14:41:31 CEST 2009 128
Tue Jul 28 14:41:36 CEST 2009 103
Tue Jul 28 14:41:41 CEST 2009 74
Tue Jul 28 14:41:47 CEST 2009 98
Tue Jul 28 14:41:52 CEST 2009 91
Tue Jul 28 14:41:57 CEST 2009 75
Tue Jul 28 14:42:02 CEST 2009 75
Tue Jul 28 14:42:07 CEST 2009 88
Tue Jul 28 14:42:12 CEST 2009 89
Tue Jul 28 14:42:17 CEST 2009 83
Tue Jul 28 14:42:22 CEST 2009 81
Tue Jul 28 14:42:27 CEST 2009 64
Tue Jul 28 14:42:32 CEST 2009 55
Tue Jul 28 14:42:37 CEST 2009 71
Tue Jul 28 14:42:42 CEST 2009 52
Tue Jul 28 14:42:47 CEST 2009 45
Tue Jul 28 14:42:52 CEST 2009 40
Tue Jul 28 14:42:57 CEST 2009 41
Tue Jul 28 14:43:02 CEST 2009 40
Tue Jul 28 14:43:07 CEST 2009 36
Tue Jul 28 14:43:12 CEST 2009 35
Tue Jul 28 14:43:17 CEST 2009 32
Tue Jul 28 14:43:22 CEST 2009 32
Tue Jul 28 14:43:27 CEST 2009 28
Tue Jul 28 14:43:32 CEST 2009 26
Tue Jul 28 14:43:37 CEST 2009 20
Tue Jul 28 14:43:42 CEST 2009 15
Tue Jul 28 14:43:47 CEST 2009 14
Tue Jul 28 14:43:52 CEST 2009 14
Tue Jul 28 14:43:57 CEST 2009 15
Tue Jul 28 14:44:02 CEST 2009 11
Tue Jul 28 14:44:07 CEST 2009 6
Tue Jul 28 14:44:12 CEST 2009 9
Tue Jul 28 14:44:17 CEST 2009 7
Tue Jul 28 14:44:22 CEST 2009 7
Tue Jul 28 14:44:27 CEST 2009 11
Tue Jul 28 14:44:32 CEST 2009 14
Tue Jul 28 14:44:37 CEST 2009 11
Tue Jul 28 14:44:42 CEST 2009 9
Tue Jul 28 14:44:47 CEST 2009 9
Tue Jul 28 14:44:52 CEST 2009 2
Tue Jul 28 14:44:58 CEST 2009 3


If you compare that to the values from my first post you see that it really works. Currently it is only a test script which is not tuned for performance, on a big wave I’ve problems to kill the processes as fast as they are forked but a better algorithm would help here. And I will look also into the possibility of limiting the amount of new connections per seconds I can handle per iptables.

But the biggest advantage would be if courier would be extended in a way that the smtp handle process adds the IP to iptables and terminates itself.

Anyway, I’ll try to make my code more than only a test script, but one that I can run in production 24/7. I’ll keep you posted – any ideas on your part?

The Search for reduced SPAM load – Part 2

July 28, 2009

As my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to withstand these waves. If packets and connections should not reach courier, I would need to use a smtp proxy or to something with the Linux kernel. As the OpenBSD spamd is not available for Linux,  I looked than through the iptables documentation and found the ipt_recent module.

It provides a userspace interface which enables a script/program to add IP addresses to a list which get them drop/reject for a given time. I thought this is exactly what I need. Why?

I cannot just drop/reject packages of IPs which are in DNS RBL, as maybe there is a false positive and he needs to know that there is a problem. The mail server needs therefore to send a 5xx the first time, but it is quite ok I think to not except connections for him some minutes after this.

This setup should at least give courier time to close the connections within the timeout, and denies a spammer trying to deliver more than one mail, or keep the connection up by ignoring the 5xx. So I went ahead. First I loaded the iptables module with an option to allow more IP addresses stored.


modprobe ipt_recent ip_list_tot=1000


I think 1000 is quite on the low end, as the spam waves easily reach them. Than I added following iptables commands to my firewall script.


# build sub chain
$iptables -N SPAMMER
# move all incomming smtp traffic there
$iptables -A INPUT -p tcp --dport 25 -j SPAMMER
# check if the source ip is already in the list, if so give it another 60 sec and drop the packages
$iptables -A SPAMMER -m recent --name spammer --update --seconds 600 -j DROP


As you can see we drop the packets for 10 minutes. If a packet is send within that 10min the time period starts again.

You can test your setup by doing following.


echo +1.1.1.1 >/proc/net/ipt_recent/spammer


and take a look at


cat /proc/net/ipt_recent/spammer


Replace 1.1.1.1 by an IP address of a spammer (just look in your logfile 😉 ) and see it working. Ok, now that we have the kernel/iptables part we need a script which adds the IP addresses of spammers on a DNS RBL after the first 5xx to the ipt_recent list.

As this is only for testing at this point I wrote a small script which watches the mail.log file and looks for 511 errors (the courier error code for DNS RBL hits) and add the IP addresses of the sending servers/zombies to ipt_recent.

Here is the python script: watchForSpammers.py. I don’t go into any details it is quite easy anyway. Start it like this within screen (apt-get install screen) to keep it alive even after logout.


./watchForSpammers.py /var/log/mail.log


The script does its work and adds IP addresses to the ipt_recent list which blocks them also nicely. Just type following to verify it.


iptables -L -xvn


You will see a line like this.


Chain SPAMMER (1 references)
pkts bytes target prot opt in out source destination
15213 724935 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 recent: UPDATE seconds: 600 name: spammer side: source


But the big question is, does it help against the spam waves and its length and heights? More No than Yes ;-). It does almost nothing against the height, but it allows courier to go back a little faster to normal. But still my mail server is maxed out.

So I continue to search for another way. Any ideas?

The Search for reduced SPAM load – Part 1

July 27, 2009

This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.

One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.

The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.


Mon Jul 27 19:12:17 CEST 2009     4
Mon Jul 27 19:12:22 CEST 2009     5
Mon Jul 27 19:12:27 CEST 2009     5
Mon Jul 27 19:12:32 CEST 2009     7
Mon Jul 27 19:12:37 CEST 2009     13
Mon Jul 27 19:12:42 CEST 2009     9
Mon Jul 27 19:12:47 CEST 2009     8
Mon Jul 27 19:12:52 CEST 2009     50
Mon Jul 27 19:12:57 CEST 2009     88
Mon Jul 27 19:13:02 CEST 2009     101
Mon Jul 27 19:13:07 CEST 2009     120
Mon Jul 27 19:13:12 CEST 2009     129
Mon Jul 27 19:13:17 CEST 2009     135
Mon Jul 27 19:13:23 CEST 2009     145
Mon Jul 27 19:13:28 CEST 2009     150
Mon Jul 27 19:13:33 CEST 2009     163
Mon Jul 27 19:13:38 CEST 2009     167
Mon Jul 27 19:13:43 CEST 2009     175
Mon Jul 27 19:13:48 CEST 2009     180
Mon Jul 27 19:13:53 CEST 2009     183
Mon Jul 27 19:13:58 CEST 2009     190
Mon Jul 27 19:14:03 CEST 2009     202
Mon Jul 27 19:14:08 CEST 2009     204
Mon Jul 27 19:14:13 CEST 2009     204
Mon Jul 27 19:14:18 CEST 2009     203
Mon Jul 27 19:14:23 CEST 2009     206
Mon Jul 27 19:14:28 CEST 2009     207
Mon Jul 27 19:14:33 CEST 2009     211
Mon Jul 27 19:14:38 CEST 2009     216
Mon Jul 27 19:14:43 CEST 2009     218
Mon Jul 27 19:14:48 CEST 2009     220
Mon Jul 27 19:14:53 CEST 2009     222
Mon Jul 27 19:14:58 CEST 2009     227
Mon Jul 27 19:15:03 CEST 2009     229
Mon Jul 27 19:15:08 CEST 2009     228
Mon Jul 27 19:15:13 CEST 2009     229
Mon Jul 27 19:15:18 CEST 2009     232
Mon Jul 27 19:15:23 CEST 2009     234
Mon Jul 27 19:15:28 CEST 2009     234
Mon Jul 27 19:15:34 CEST 2009     237
Mon Jul 27 19:15:39 CEST 2009     240
Mon Jul 27 19:15:44 CEST 2009     241
Mon Jul 27 19:15:49 CEST 2009     243
Mon Jul 27 19:15:54 CEST 2009     244
Mon Jul 27 19:15:59 CEST 2009     244
Mon Jul 27 19:16:04 CEST 2009     246
Mon Jul 27 19:16:09 CEST 2009     245
Mon Jul 27 19:16:14 CEST 2009     246
Mon Jul 27 19:16:19 CEST 2009     247
Mon Jul 27 19:16:24 CEST 2009     246
Mon Jul 27 19:16:29 CEST 2009     244
Mon Jul 27 19:16:34 CEST 2009     250
Mon Jul 27 19:16:39 CEST 2009     253
Mon Jul 27 19:16:44 CEST 2009     252
Mon Jul 27 19:16:49 CEST 2009     253
Mon Jul 27 19:16:54 CEST 2009     256
Mon Jul 27 19:16:59 CEST 2009     258
Mon Jul 27 19:17:04 CEST 2009     260
Mon Jul 27 19:17:09 CEST 2009     264
Mon Jul 27 19:17:14 CEST 2009     262
Mon Jul 27 19:17:19 CEST 2009     273
Mon Jul 27 19:17:24 CEST 2009     301
Mon Jul 27 19:17:30 CEST 2009     301
Mon Jul 27 19:17:35 CEST 2009     301
Mon Jul 27 19:17:40 CEST 2009     301


It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.


Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.


At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.

You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.

First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:


root@mail:~# cat /etc/courier/esmtptimeout
30
root@mail:~# cat /etc/courier/esmtptimeoutconnect
30
root@mail:~# cat /etc/courier/esmtptimeoutdata
180
root@mail:~# cat /etc/courier/esmtptimeouthelo
30


Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.

Second Idea:

I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.

Sam Varshavchik the lead programmer of courier told me following:

Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.

So ok not good and he said following about my idea to drop the connection ….

And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.

The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.

This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.

So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.

Firefox Addons I use and recommend

July 25, 2009

I thought I share the Firefox plugins I use and which make me use Firefox in the first place. Without these plugins Firefox would be just a browser of many and the webkit browsers render faster on my Kubuntu ;-).  So these Plugins make the difference for me.

• NoScript: Without I feel “unsafe” in the internet. It allows JavaScript, Java and other executable content to run only from trusted domains of your choice. You can activate JS only for a session also, so you’re save again next time.
• Cookie Monster: This plugin allows me to manage my cookies. I can set from which domains I accept which kind. e.g. I accept cookies only for the session from a domain if it is needed.
• Xmarks: I use this plugin to sync my bookmarks between systems and also to have a backup at all times of them. You can also use it do sync/save your stored passworts securely.  You can also use your own server.
• DownloadHelper: You never know when you would like to download a flash movie or something like this onto your PC. This plugin will enable you to do so.
• Yip: If you’re using something like meebo.com for instant messaging you surely would like to get notifications of a new message also outside the tab in your browser, as it mostlikely happens that you’re working with an other program or in an other tab. If so, take a look at Yip, as its supports  supports Fluid and Prism which cover the large majority (100%?) of currently implemented notifications.

First look at Kubuntu 9.10 (Karmic) and KDE 4.3

Yesterday  I installed the Alpha 3 of Karmic which will become Kubuntu 9.10. It ships with KDE 4.3 (currently the RC3) and KMS, which stands for Kernel Mode Setting. And whats this? Kernel mode-setting (KMS) shifts responsibility for selecting and setting up the graphics mode from X.org to the kernel. When X.org is started, it then detects and uses the mode without any further mode changes. This promises to make booting faster, more graphical, and less flickery.

Buts whats the first impression?

KDE 4.3 really looks cool and polished with the new Air theme. Much better than the old one. Sure as widget style KDE 4.3 still uses Oxygen, but with Air it got rid of the black on black stuff. Maybe this is done by the  Kubuntu guys and not by the KDE ones, but I like it. If you know how a dialog/window looked in KDE 4.2 you see everywhere small things that got improved. Sure they’re small things but it makes it a much improved user experience.

Karmic starts even faster than Jaunty did and it was not slow. I even think that booting is as fast now as shutting down now. Please improve this also. 😉

When you boot into Karmic the first time it asks you for your openDesktop username and password, for the plasma widget on your desktop. I went ahead to the openDesktop homepage and created a user to test it.  At this point I know that it works, as I see other people in my area online.  But I’m not really sure what I should do with it. Should this replace my ICQ/GoogleTalk? Maybe someone can enlighten me.

One thing that did not work out of the box was the sound in flash videos, in KDE and mplayer everything worked. But it was simple to solve it. Just open KMix and raise the PCM volume, which was on my system defaulted to zero.

So far my first impressions – basically I really like it and it runs solid – no crashes with the KDE 4.3 RC3 stuff.

I’ve also updated my additional packages install script for karmic and you can download it here.

ps: Take a look at http://kubuntuguide.org/Karmic for some nice ideas and tips around Karmic.