The Search for reduced SPAM load – Part 1

July 27, 2009

This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.

One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.

The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.


Mon Jul 27 19:12:17 CEST 2009     4
Mon Jul 27 19:12:22 CEST 2009     5
Mon Jul 27 19:12:27 CEST 2009     5
Mon Jul 27 19:12:32 CEST 2009     7
Mon Jul 27 19:12:37 CEST 2009     13
Mon Jul 27 19:12:42 CEST 2009     9
Mon Jul 27 19:12:47 CEST 2009     8
Mon Jul 27 19:12:52 CEST 2009     50
Mon Jul 27 19:12:57 CEST 2009     88
Mon Jul 27 19:13:02 CEST 2009     101
Mon Jul 27 19:13:07 CEST 2009     120
Mon Jul 27 19:13:12 CEST 2009     129
Mon Jul 27 19:13:17 CEST 2009     135
Mon Jul 27 19:13:23 CEST 2009     145
Mon Jul 27 19:13:28 CEST 2009     150
Mon Jul 27 19:13:33 CEST 2009     163
Mon Jul 27 19:13:38 CEST 2009     167
Mon Jul 27 19:13:43 CEST 2009     175
Mon Jul 27 19:13:48 CEST 2009     180
Mon Jul 27 19:13:53 CEST 2009     183
Mon Jul 27 19:13:58 CEST 2009     190
Mon Jul 27 19:14:03 CEST 2009     202
Mon Jul 27 19:14:08 CEST 2009     204
Mon Jul 27 19:14:13 CEST 2009     204
Mon Jul 27 19:14:18 CEST 2009     203
Mon Jul 27 19:14:23 CEST 2009     206
Mon Jul 27 19:14:28 CEST 2009     207
Mon Jul 27 19:14:33 CEST 2009     211
Mon Jul 27 19:14:38 CEST 2009     216
Mon Jul 27 19:14:43 CEST 2009     218
Mon Jul 27 19:14:48 CEST 2009     220
Mon Jul 27 19:14:53 CEST 2009     222
Mon Jul 27 19:14:58 CEST 2009     227
Mon Jul 27 19:15:03 CEST 2009     229
Mon Jul 27 19:15:08 CEST 2009     228
Mon Jul 27 19:15:13 CEST 2009     229
Mon Jul 27 19:15:18 CEST 2009     232
Mon Jul 27 19:15:23 CEST 2009     234
Mon Jul 27 19:15:28 CEST 2009     234
Mon Jul 27 19:15:34 CEST 2009     237
Mon Jul 27 19:15:39 CEST 2009     240
Mon Jul 27 19:15:44 CEST 2009     241
Mon Jul 27 19:15:49 CEST 2009     243
Mon Jul 27 19:15:54 CEST 2009     244
Mon Jul 27 19:15:59 CEST 2009     244
Mon Jul 27 19:16:04 CEST 2009     246
Mon Jul 27 19:16:09 CEST 2009     245
Mon Jul 27 19:16:14 CEST 2009     246
Mon Jul 27 19:16:19 CEST 2009     247
Mon Jul 27 19:16:24 CEST 2009     246
Mon Jul 27 19:16:29 CEST 2009     244
Mon Jul 27 19:16:34 CEST 2009     250
Mon Jul 27 19:16:39 CEST 2009     253
Mon Jul 27 19:16:44 CEST 2009     252
Mon Jul 27 19:16:49 CEST 2009     253
Mon Jul 27 19:16:54 CEST 2009     256
Mon Jul 27 19:16:59 CEST 2009     258
Mon Jul 27 19:17:04 CEST 2009     260
Mon Jul 27 19:17:09 CEST 2009     264
Mon Jul 27 19:17:14 CEST 2009     262
Mon Jul 27 19:17:19 CEST 2009     273
Mon Jul 27 19:17:24 CEST 2009     301
Mon Jul 27 19:17:30 CEST 2009     301
Mon Jul 27 19:17:35 CEST 2009     301
Mon Jul 27 19:17:40 CEST 2009     301

It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.


Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.

At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.

You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.

First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:


root@mail:~# cat /etc/courier/esmtptimeout
30
root@mail:~# cat /etc/courier/esmtptimeoutconnect
30
root@mail:~# cat /etc/courier/esmtptimeoutdata
180
root@mail:~# cat /etc/courier/esmtptimeouthelo
30

Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.

Second Idea:

I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.

Sam Varshavchik the lead programmer of courier told me following:

Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.

So ok not good and he said following about my idea to drop the connection ….

And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.

The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.

This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.

So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.

4 Comments »

RSS feed for comments on this post. TrackBack URI

  1. Did you try to setup “spamd” already?

    From the Man-Page:
    > spamd is a fake sendmail(8)-like daemon
    > which rejects false mail. It is
    > designed to be very efficient so that
    > it does not slow down the receiving
    > machine.

    Cheers
    -Richard

    Comment by Richard — July 27, 2009 #

  2. Forget to mention it. I’m running Linux, to be exact Ubuntu, and I couldn’t find a port to Linux of it.

    Comment by robert — July 27, 2009 #

  3. [...] my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to [...]

    Pingback by The Search for reduced SPAM load – Part 2 | Robert Penz Blog — July 28, 2009 #

  4. [...] The Search for reduced SPAM load – Part 1 [...]

    Pingback by The Search for reduced SPAM load – Part 3 | Robert Penz Blog — July 29, 2009 #

Leave a comment


XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 70 queries. 1.022 seconds.