The Search for reduced SPAM load – Part 1
July 27, 2009
This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.
One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.
The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.
Mon Jul 27 19:12:17 CEST 2009 4
Mon Jul 27 19:12:22 CEST 2009 5
Mon Jul 27 19:12:27 CEST 2009 5
Mon Jul 27 19:12:32 CEST 2009 7
Mon Jul 27 19:12:37 CEST 2009 13
Mon Jul 27 19:12:42 CEST 2009 9
Mon Jul 27 19:12:47 CEST 2009 8
Mon Jul 27 19:12:52 CEST 2009 50
Mon Jul 27 19:12:57 CEST 2009 88
Mon Jul 27 19:13:02 CEST 2009 101
Mon Jul 27 19:13:07 CEST 2009 120
Mon Jul 27 19:13:12 CEST 2009 129
Mon Jul 27 19:13:17 CEST 2009 135
Mon Jul 27 19:13:23 CEST 2009 145
Mon Jul 27 19:13:28 CEST 2009 150
Mon Jul 27 19:13:33 CEST 2009 163
Mon Jul 27 19:13:38 CEST 2009 167
Mon Jul 27 19:13:43 CEST 2009 175
Mon Jul 27 19:13:48 CEST 2009 180
Mon Jul 27 19:13:53 CEST 2009 183
Mon Jul 27 19:13:58 CEST 2009 190
Mon Jul 27 19:14:03 CEST 2009 202
Mon Jul 27 19:14:08 CEST 2009 204
Mon Jul 27 19:14:13 CEST 2009 204
Mon Jul 27 19:14:18 CEST 2009 203
Mon Jul 27 19:14:23 CEST 2009 206
Mon Jul 27 19:14:28 CEST 2009 207
Mon Jul 27 19:14:33 CEST 2009 211
Mon Jul 27 19:14:38 CEST 2009 216
Mon Jul 27 19:14:43 CEST 2009 218
Mon Jul 27 19:14:48 CEST 2009 220
Mon Jul 27 19:14:53 CEST 2009 222
Mon Jul 27 19:14:58 CEST 2009 227
Mon Jul 27 19:15:03 CEST 2009 229
Mon Jul 27 19:15:08 CEST 2009 228
Mon Jul 27 19:15:13 CEST 2009 229
Mon Jul 27 19:15:18 CEST 2009 232
Mon Jul 27 19:15:23 CEST 2009 234
Mon Jul 27 19:15:28 CEST 2009 234
Mon Jul 27 19:15:34 CEST 2009 237
Mon Jul 27 19:15:39 CEST 2009 240
Mon Jul 27 19:15:44 CEST 2009 241
Mon Jul 27 19:15:49 CEST 2009 243
Mon Jul 27 19:15:54 CEST 2009 244
Mon Jul 27 19:15:59 CEST 2009 244
Mon Jul 27 19:16:04 CEST 2009 246
Mon Jul 27 19:16:09 CEST 2009 245
Mon Jul 27 19:16:14 CEST 2009 246
Mon Jul 27 19:16:19 CEST 2009 247
Mon Jul 27 19:16:24 CEST 2009 246
Mon Jul 27 19:16:29 CEST 2009 244
Mon Jul 27 19:16:34 CEST 2009 250
Mon Jul 27 19:16:39 CEST 2009 253
Mon Jul 27 19:16:44 CEST 2009 252
Mon Jul 27 19:16:49 CEST 2009 253
Mon Jul 27 19:16:54 CEST 2009 256
Mon Jul 27 19:16:59 CEST 2009 258
Mon Jul 27 19:17:04 CEST 2009 260
Mon Jul 27 19:17:09 CEST 2009 264
Mon Jul 27 19:17:14 CEST 2009 262
Mon Jul 27 19:17:19 CEST 2009 273
Mon Jul 27 19:17:24 CEST 2009 301
Mon Jul 27 19:17:30 CEST 2009 301
Mon Jul 27 19:17:35 CEST 2009 301
Mon Jul 27 19:17:40 CEST 2009 301
It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.
Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.
At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.
You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.
First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:
root@mail:~# cat /etc/courier/esmtptimeout
30
root@mail:~# cat /etc/courier/esmtptimeoutconnect
30
root@mail:~# cat /etc/courier/esmtptimeoutdata
180
root@mail:~# cat /etc/courier/esmtptimeouthelo
30
Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.
Second Idea:
I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.
Sam Varshavchik the lead programmer of courier told me following:
Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.
So ok not good and he said following about my idea to drop the connection ….
And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.
The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.
This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.
So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.
4 Comments »
RSS feed for comments on this post. TrackBack URI
Leave a comment
Powered by WordPress
Entries and comments feeds.
Valid XHTML and CSS.
37 queries. 0.077 seconds.
Did you try to setup “spamd” already?
From the Man-Page:
> spamd is a fake sendmail(8)-like daemon
> which rejects false mail. It is
> designed to be very efficient so that
> it does not slow down the receiving
> machine.
Cheers
-Richard
Comment by Richard — July 27, 2009 #
Forget to mention it. I’m running Linux, to be exact Ubuntu, and I couldn’t find a port to Linux of it.
Comment by robert — July 27, 2009 #
[…] my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to […]
Pingback by The Search for reduced SPAM load – Part 2 | Robert Penz Blog — July 28, 2009 #
[…] The Search for reduced SPAM load – Part 1 […]
Pingback by The Search for reduced SPAM load – Part 3 | Robert Penz Blog — July 29, 2009 #