Robert Penz Blog

Workaround for routing WOL (Wake on LAN) packets with Linux

robert — Sat, 16 Jan 2010 15:37:41 +0000

If you want to send a WOL packet to a PC within your subnet it is really easy. Just install a program like wakeonlan (apt-get install wakeonlan) and type something like:

wakeonlan 01:02:03:04:05:06

But how to you send a WOL packet to an other subnet? Basically you use a UDP packet and send it to the broadcast address of the other network. e.g. with wakeonlan it looks like this

wakeonlan -i 192.168.1.255 01:02:03:04:05:06

But you need support for this from your router, as normally they don’t allow sending to the broadcast address from other networks. Professional routers/layer3 switches have support for this (you just need to enable it), but you’ve a Linux router at home? (e.g. one with Openwrt or Debian/Ubuntu)

The simplest way to get it working is to enter following on the router (rerun it at every boot):

arp -s 192.168.1.254 FF:FF:FF:FF:FF:FF

This tells the router that the given IP has a MAC address which is used for broadcasts. Now you only need to send the packet to this new “broadcast” address instead of the real one. So your wakeup call looks like this:

wakeonlan -i 192.168.1.254 01:02:03:04:05:06

ps: you should only enable something like this on a trusted network and the IP address you use should be not used by any other device.

KDE 4.x and Autostart

robert — Thu, 03 Sep 2009 19:17:44 +0000

I was told by some fellow Linux guys that with KDE 4.x there is no Autostart possible anymore. I just wanted to write this blog post to show them that it is still possible even if it not in the KDE menu. Just create following directory if it does not exists

mkdir ~/.kde/Autostart

Some distributions may use .kde4 instead of .kde. Now just place a symlink into this directory with a command like this one for dropbox.

ln -s ~/.dropbox-dist/dropbox ~/.kde/Autostart/dropbox

At least on my Kubuntu systems with KDE 4.3.x that works.

Mini-Howto: Restore Windows MBR/Bootloader with Linux

robert — Wed, 26 Aug 2009 13:31:21 +0000

I’m often, at least more than I care, asked how to restore a Windows MBR/bootloader without having a windows install cd or a dos boot disk at hand. It’s quite easy you need just a Linux live cd like (the Ubuntu live cd or Knoppix) or an installed Linux you want get rid of. I really don’t know why you want to do the second, but anyway here are the 2 solutions I know of.

Boot Linux and make sure you’ve a working Internet connection and type following on the terminal/konsole.

1. Solution

sudo apt-get install syslinux

if the package got installed use following to write the MBR.

sudo dd if=/usr/lib/syslinux/mbr.bin of=/dev/sda

2. Solution

sudo apt-get install mbr

if the package got installed use following to write the MBR.

sudo install-mbr -i n -p D -t 0 /dev/sda

Common for both

Replace sda if you want to install the MBR to a different drive. Take a look at your hard disks with sudo fdisk -l if you’re unsure. Finally reboot and your windows should boot.

Workaround for the Ubuntu problem with KVM switches

robert — Wed, 26 Aug 2009 12:45:17 +0000

It seems that Ubuntu not only Karmic (9.10), but also older versions have a problem with KVM (Keyboard, Video, Mouse) switches. To be exact the problem is the auto-detection of the capabilities of the monitor. If you connect the monitor directly to the computer everything works, if you use a KVM switch you get only 800×600 as the maximum resolution.

The workaround is to tell the xserver the Horizsync and Vertrefresh the monitor really supports. With older Ubuntu versions you could just add following lines (for a 1280×1024 LCD) to your /etc/X11/xorg.conf in the monitor section:


Section "Monitor"
        .....
        Option          "DPMS"
	Horizsync 31.5-64.0
	Vertrefresh 56.0 - 65.0
        .....
EndSection

But starting with Karmic Ubuntu has no /etc/X11/xorg.conf file by default anymore. So what we need is a complete minimal xorg.conf file so we can include our 3 lines, but we don’t want to mess anything else up. This is the minimal config I came up with.


Section "InputDevice"
	Identifier	"Generic Keyboard"
	Driver		"kbd"
	Option		"XkbRules"	"xorg"
	Option		"XkbModel"	"pc105"
	Option		"XkbLayout"	"de"
	Option		"XkbVariant"	"nodeadkeys"
EndSection

Section "InputDevice"
	Identifier	"Configured Mouse"
	Driver		"mouse"
EndSection

Section "Device"
	Identifier	"Configured Video Device"
EndSection

Section "Monitor"
	Identifier	"Configured Monitor"
        Option          "DPMS"
	Horizsync 31.5-64.0
	Vertrefresh 56.0 - 65.0
EndSection

Section "Screen"
	Identifier	"Default Screen"
	Monitor		"Configured Monitor"
	Device		"Configured Video Device"
        SubSection "Display"
                Depth           24
                Modes           "1280x1024" "1024x768"
        EndSubSection
EndSection

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "Configured Mouse"
EndSection

Except the Keyboard stuff the should be nothing thats not minimal. Maybe it is also possible to remove some lines there, but I didn’t test it. I was happy that it worked this way . If you’ve an even more minimal config write a comment please!

Fast test if local mail works on a server

robert — Tue, 25 Aug 2009 19:18:54 +0000

I just helped a friend whose server did not send cron mails to his mail server, which is this case led almost to data loss as the backup didn’t work correctly. I looked at the setup and I though I found the problem and correct it, but now I wanted to test it as easily as possible. Therefore I typed following command:

echo "Subject: test" | sendmail -v root

Then I looked if the lokal MTA, in this cache ssmtp delivered the mail to mail server. I though at least I find this line again if I search my blog and maybe it helps someone else too.

Free SSL certificates will be supported in IE/Windows [Update]

robert — Fri, 21 Aug 2009 14:42:12 +0000

In the StartCom Blog the head of the company writes that its CA will be included by Microsoft. He states, “Starting approximately the 22nd of September, Microsoft intends to distribute a non-security update package to the Windows operating systems which includes the trusted StartCom root certificate and the automatic root certificate update service will update the cryptographic certificates root store on those systems whenever a StartCom issued certificate is encountered.”

Why is this worth reporting? This CA is with the above mentioned date the first CA which provides a Free SSL certificate which is supported by Microsoft and therefore by all Internet Explorer browsers. Sure this is “only” a SSL certificate which only assures the domain name or email address, but this is much more than now. You find many small mail servers where the webmail/IMAP/POP3/SMTP or a small homepage for a few users is self signed. All these can now be protected much better, the only problem seems to be that the CA is not supported by Firefox and other open source players. The other authority I know which provides free SSL certificates is CAcert. They are much stronger in the open source world, but with the inclusion of the StartSSL CA by Microsoft, they will have a much lower install base of browsers than StartSSL.

[Update] Mozilla and Apple support this CA for years already. My error – sorry. So these Free SSL certs are really something for a small website or mail server.[/Update]

Courier-MTA reacts badly if own DNS server goes down [Update]

robert — Tue, 18 Aug 2009 18:29:46 +0000

Today my DNS registrar and in my case also the provider of the DNS servers for my domains had a total blackout. He was down for about an hour, none of the 3 DNS servers was reachable. Ok, this was bad, no system could get the IPs for domain names – nothing I can to there except using a 4th DNS server operated by me in the future. But this was not the biggest problem, some DNS servers had the IP for my mail servers stored and tried to connect via SMTP to my courier-mta. The problem now is that courier didn’t accept the connections it tried to lookup its own name via the DNS servers in /etc/resolve.conf and ignored the settings in /etc/hosts (basically its own name). I thought surely I’ve a configuration error until I found following in the courier-mta FAQ.

NOTE: The Courier mail server does not read the hosts file. It needs a DNS server (although it is possible to have a working the Courier mail server configuration in a completely DNS-free environment, this excersize requires changing many configuration files, and perhaps will be its own FAQ entry some day).

What the fuck? Courier stops working if the DNS server for the own domain/hostname is not reachable? That must be a bug and not a feature. I will talk to the author and report back what he says.

[update] I talked with the author and other courier experts. The way courier is written it uses one internal function to all his DNS stuff and as it needs MX,TXT queries it uses a glibc function in it with does ignore the hosts file. There is currently no way around it, and so I’ll need to deploy an DNS server on my mail server to provide a DNS which can always resolve it’s own hostname. [/update]

The Search for reduced SPAM load – Part 3

robert — Wed, 29 Jul 2009 17:35:32 +0000

Take a look at

to understand purpose of this series and what I’m looking for. As only firewalling spammers which are in a DNS RBL after they got a 5xx didn’t worked as hoped I had an other idea.

Instead of only firewalling the spammer I thought as there will be no packet from the spammer within the 30sec timeout why not just terminate the process which handles the connection. This leads to a reduced process number at once and makes space for a new one. A clean solution would implement my complete script in the mta itself – basically adding the IP to the firewall and terminating the smtp handling process. But for a mere test it would be easy to extend my script to kill the current process.

This python script (watchForSpammers2.py) does exactly that – It extends the old script by searching through the process list for a submit progress which handles the spammer connection. It follows the ppid and kills with a SIGTERM the parent courieresmtpd process.

You say thats a hard method? And you asked yourself if it works – yes it does.

After some testing I implemented it on the productive server and I’ve it running for a few days now and I didn’t reach the maximum of 300 connections since.

e.g. take a look at this spam wave
Tue Jul 28 08:30:51 CEST 2009 5 Tue Jul 28 08:30:56 CEST 2009 8 Tue Jul 28 08:31:01 CEST 2009 7 Tue Jul 28 08:31:06 CEST 2009 2 Tue Jul 28 08:31:11 CEST 2009 3 Tue Jul 28 08:31:16 CEST 2009 4 Tue Jul 28 08:31:21 CEST 2009 13 Tue Jul 28 08:31:26 CEST 2009 100 Tue Jul 28 08:31:31 CEST 2009 77 Tue Jul 28 08:31:36 CEST 2009 48 Tue Jul 28 08:31:41 CEST 2009 31 Tue Jul 28 08:31:46 CEST 2009 32 Tue Jul 28 08:31:51 CEST 2009 39 Tue Jul 28 08:31:56 CEST 2009 34 Tue Jul 28 08:32:01 CEST 2009 19 Tue Jul 28 08:32:06 CEST 2009 19 Tue Jul 28 08:32:11 CEST 2009 18 Tue Jul 28 08:32:16 CEST 2009 18 Tue Jul 28 08:32:21 CEST 2009 16 Tue Jul 28 08:32:26 CEST 2009 15 Tue Jul 28 08:32:31 CEST 2009 13 Tue Jul 28 08:32:36 CEST 2009 14 Tue Jul 28 08:32:41 CEST 2009 19 Tue Jul 28 08:32:46 CEST 2009 53 Tue Jul 28 08:32:51 CEST 2009 58 Tue Jul 28 08:32:56 CEST 2009 38 Tue Jul 28 08:33:01 CEST 2009 30 Tue Jul 28 08:33:06 CEST 2009 28

or the biggest in the last days

Tue Jul 28 14:36:19 CEST 2009 3 Tue Jul 28 14:36:24 CEST 2009 4 Tue Jul 28 14:36:29 CEST 2009 2 Tue Jul 28 14:36:34 CEST 2009 1 Tue Jul 28 14:36:39 CEST 2009 3 Tue Jul 28 14:36:44 CEST 2009 1 Tue Jul 28 14:36:49 CEST 2009 1 Tue Jul 28 14:36:54 CEST 2009 34 Tue Jul 28 14:36:59 CEST 2009 56 Tue Jul 28 14:37:04 CEST 2009 52 Tue Jul 28 14:37:09 CEST 2009 60 Tue Jul 28 14:37:14 CEST 2009 87 Tue Jul 28 14:37:19 CEST 2009 126 Tue Jul 28 14:37:24 CEST 2009 128 Tue Jul 28 14:37:29 CEST 2009 140 Tue Jul 28 14:37:34 CEST 2009 138 Tue Jul 28 14:37:39 CEST 2009 143 Tue Jul 28 14:37:44 CEST 2009 161 Tue Jul 28 14:37:49 CEST 2009 198 Tue Jul 28 14:37:54 CEST 2009 208 Tue Jul 28 14:37:59 CEST 2009 187 Tue Jul 28 14:38:04 CEST 2009 175 Tue Jul 28 14:38:09 CEST 2009 140 Tue Jul 28 14:38:15 CEST 2009 144 Tue Jul 28 14:38:20 CEST 2009 150 Tue Jul 28 14:38:25 CEST 2009 190 Tue Jul 28 14:38:30 CEST 2009 182 Tue Jul 28 14:38:35 CEST 2009 167 Tue Jul 28 14:38:40 CEST 2009 176 Tue Jul 28 14:38:45 CEST 2009 190 Tue Jul 28 14:38:50 CEST 2009 206 Tue Jul 28 14:38:55 CEST 2009 199 Tue Jul 28 14:39:00 CEST 2009 197 Tue Jul 28 14:39:05 CEST 2009 199 Tue Jul 28 14:39:10 CEST 2009 168 Tue Jul 28 14:39:15 CEST 2009 199 Tue Jul 28 14:39:20 CEST 2009 210 Tue Jul 28 14:39:25 CEST 2009 201 Tue Jul 28 14:39:30 CEST 2009 195 Tue Jul 28 14:39:35 CEST 2009 216 Tue Jul 28 14:39:40 CEST 2009 203 Tue Jul 28 14:39:45 CEST 2009 200 Tue Jul 28 14:39:50 CEST 2009 196 Tue Jul 28 14:39:56 CEST 2009 189 Tue Jul 28 14:40:01 CEST 2009 180 Tue Jul 28 14:40:06 CEST 2009 176 Tue Jul 28 14:40:11 CEST 2009 173 Tue Jul 28 14:40:16 CEST 2009 177 Tue Jul 28 14:40:21 CEST 2009 165 Tue Jul 28 14:40:26 CEST 2009 170 Tue Jul 28 14:40:31 CEST 2009 164 Tue Jul 28 14:40:36 CEST 2009 167 Tue Jul 28 14:40:41 CEST 2009 151 Tue Jul 28 14:40:46 CEST 2009 147 Tue Jul 28 14:40:51 CEST 2009 139 Tue Jul 28 14:40:56 CEST 2009 140 Tue Jul 28 14:41:01 CEST 2009 136 Tue Jul 28 14:41:06 CEST 2009 131 Tue Jul 28 14:41:11 CEST 2009 131 Tue Jul 28 14:41:16 CEST 2009 147 Tue Jul 28 14:41:21 CEST 2009 134 Tue Jul 28 14:41:26 CEST 2009 133 Tue Jul 28 14:41:31 CEST 2009 128 Tue Jul 28 14:41:36 CEST 2009 103 Tue Jul 28 14:41:41 CEST 2009 74 Tue Jul 28 14:41:47 CEST 2009 98 Tue Jul 28 14:41:52 CEST 2009 91 Tue Jul 28 14:41:57 CEST 2009 75 Tue Jul 28 14:42:02 CEST 2009 75 Tue Jul 28 14:42:07 CEST 2009 88 Tue Jul 28 14:42:12 CEST 2009 89 Tue Jul 28 14:42:17 CEST 2009 83 Tue Jul 28 14:42:22 CEST 2009 81 Tue Jul 28 14:42:27 CEST 2009 64 Tue Jul 28 14:42:32 CEST 2009 55 Tue Jul 28 14:42:37 CEST 2009 71 Tue Jul 28 14:42:42 CEST 2009 52 Tue Jul 28 14:42:47 CEST 2009 45 Tue Jul 28 14:42:52 CEST 2009 40 Tue Jul 28 14:42:57 CEST 2009 41 Tue Jul 28 14:43:02 CEST 2009 40 Tue Jul 28 14:43:07 CEST 2009 36 Tue Jul 28 14:43:12 CEST 2009 35 Tue Jul 28 14:43:17 CEST 2009 32 Tue Jul 28 14:43:22 CEST 2009 32 Tue Jul 28 14:43:27 CEST 2009 28 Tue Jul 28 14:43:32 CEST 2009 26 Tue Jul 28 14:43:37 CEST 2009 20 Tue Jul 28 14:43:42 CEST 2009 15 Tue Jul 28 14:43:47 CEST 2009 14 Tue Jul 28 14:43:52 CEST 2009 14 Tue Jul 28 14:43:57 CEST 2009 15 Tue Jul 28 14:44:02 CEST 2009 11 Tue Jul 28 14:44:07 CEST 2009 6 Tue Jul 28 14:44:12 CEST 2009 9 Tue Jul 28 14:44:17 CEST 2009 7 Tue Jul 28 14:44:22 CEST 2009 7 Tue Jul 28 14:44:27 CEST 2009 11 Tue Jul 28 14:44:32 CEST 2009 14 Tue Jul 28 14:44:37 CEST 2009 11 Tue Jul 28 14:44:42 CEST 2009 9 Tue Jul 28 14:44:47 CEST 2009 9 Tue Jul 28 14:44:52 CEST 2009 2 Tue Jul 28 14:44:58 CEST 2009 3

If you compare that to the values from my first post you see that it really works. Currently it is only a test script which is not tuned for performance, on a big wave I’ve problems to kill the processes as fast as they are forked but a better algorithm would help here. And I will look also into the possibility of limiting the amount of new connections per seconds I can handle per iptables.

But the biggest advantage would be if courier would be extended in a way that the smtp handle process adds the IP to iptables and terminates itself.

Anyway, I’ll try to make my code more than only a test script, but one that I can run in production 24/7. I’ll keep you posted – any ideas on your part?

The Search for reduced SPAM load – Part 2

robert — Tue, 28 Jul 2009 17:05:39 +0000

As my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to withstand these waves. If packets and connections should not reach courier, I would need to use a smtp proxy or to something with the Linux kernel. As the OpenBSD spamd is not available for Linux, I looked than through the iptables documentation and found the ipt_recent module.

It provides a userspace interface which enables a script/program to add IP addresses to a list which get them drop/reject for a given time. I thought this is exactly what I need. Why?

I cannot just drop/reject packages of IPs which are in DNS RBL, as maybe there is a false positive and he needs to know that there is a problem. The mail server needs therefore to send a 5xx the first time, but it is quite ok I think to not except connections for him some minutes after this.

This setup should at least give courier time to close the connections within the timeout, and denies a spammer trying to deliver more than one mail, or keep the connection up by ignoring the 5xx. So I went ahead. First I loaded the iptables module with an option to allow more IP addresses stored.

modprobe ipt_recent ip_list_tot=1000

I think 1000 is quite on the low end, as the spam waves easily reach them. Than I added following iptables commands to my firewall script.

# build sub chain $iptables -N SPAMMER # move all incomming smtp traffic there $iptables -A INPUT -p tcp --dport 25 -j SPAMMER # check if the source ip is already in the list, if so give it another 60 sec and drop the packages $iptables -A SPAMMER -m recent --name spammer --update --seconds 600 -j DROP

As you can see we drop the packets for 10 minutes. If a packet is send within that 10min the time period starts again.

You can test your setup by doing following.

echo +1.1.1.1 >/proc/net/ipt_recent/spammer

and take a look at

cat /proc/net/ipt_recent/spammer

Replace 1.1.1.1 by an IP address of a spammer (just look in your logfile ) and see it working. Ok, now that we have the kernel/iptables part we need a script which adds the IP addresses of spammers on a DNS RBL after the first 5xx to the ipt_recent list.

As this is only for testing at this point I wrote a small script which watches the mail.log file and looks for 511 errors (the courier error code for DNS RBL hits) and add the IP addresses of the sending servers/zombies to ipt_recent.

Here is the python script: watchForSpammers.py. I don’t go into any details it is quite easy anyway. Start it like this within screen (apt-get install screen) to keep it alive even after logout.

./watchForSpammers.py /var/log/mail.log

The script does its work and adds IP addresses to the ipt_recent list which blocks them also nicely. Just type following to verify it.

iptables -L -xvn

You will see a line like this.

Chain SPAMMER (1 references) pkts bytes target prot opt in out source destination 15213 724935 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 recent: UPDATE seconds: 600 name: spammer side: source

But the big question is, does it help against the spam waves and its length and heights? More No than Yes . It does almost nothing against the height, but it allows courier to go back a little faster to normal. But still my mail server is maxed out.

So I continue to search for another way. Any ideas?

The Search for reduced SPAM load – Part 1

robert — Mon, 27 Jul 2009 17:50:31 +0000

This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.

One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.

The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.

Mon Jul 27 19:12:17 CEST 2009 4 Mon Jul 27 19:12:22 CEST 2009 5 Mon Jul 27 19:12:27 CEST 2009 5 Mon Jul 27 19:12:32 CEST 2009 7 Mon Jul 27 19:12:37 CEST 2009 13 Mon Jul 27 19:12:42 CEST 2009 9 Mon Jul 27 19:12:47 CEST 2009 8 Mon Jul 27 19:12:52 CEST 2009 50 Mon Jul 27 19:12:57 CEST 2009 88 Mon Jul 27 19:13:02 CEST 2009 101 Mon Jul 27 19:13:07 CEST 2009 120 Mon Jul 27 19:13:12 CEST 2009 129 Mon Jul 27 19:13:17 CEST 2009 135 Mon Jul 27 19:13:23 CEST 2009 145 Mon Jul 27 19:13:28 CEST 2009 150 Mon Jul 27 19:13:33 CEST 2009 163 Mon Jul 27 19:13:38 CEST 2009 167 Mon Jul 27 19:13:43 CEST 2009 175 Mon Jul 27 19:13:48 CEST 2009 180 Mon Jul 27 19:13:53 CEST 2009 183 Mon Jul 27 19:13:58 CEST 2009 190 Mon Jul 27 19:14:03 CEST 2009 202 Mon Jul 27 19:14:08 CEST 2009 204 Mon Jul 27 19:14:13 CEST 2009 204 Mon Jul 27 19:14:18 CEST 2009 203 Mon Jul 27 19:14:23 CEST 2009 206 Mon Jul 27 19:14:28 CEST 2009 207 Mon Jul 27 19:14:33 CEST 2009 211 Mon Jul 27 19:14:38 CEST 2009 216 Mon Jul 27 19:14:43 CEST 2009 218 Mon Jul 27 19:14:48 CEST 2009 220 Mon Jul 27 19:14:53 CEST 2009 222 Mon Jul 27 19:14:58 CEST 2009 227 Mon Jul 27 19:15:03 CEST 2009 229 Mon Jul 27 19:15:08 CEST 2009 228 Mon Jul 27 19:15:13 CEST 2009 229 Mon Jul 27 19:15:18 CEST 2009 232 Mon Jul 27 19:15:23 CEST 2009 234 Mon Jul 27 19:15:28 CEST 2009 234 Mon Jul 27 19:15:34 CEST 2009 237 Mon Jul 27 19:15:39 CEST 2009 240 Mon Jul 27 19:15:44 CEST 2009 241 Mon Jul 27 19:15:49 CEST 2009 243 Mon Jul 27 19:15:54 CEST 2009 244 Mon Jul 27 19:15:59 CEST 2009 244 Mon Jul 27 19:16:04 CEST 2009 246 Mon Jul 27 19:16:09 CEST 2009 245 Mon Jul 27 19:16:14 CEST 2009 246 Mon Jul 27 19:16:19 CEST 2009 247 Mon Jul 27 19:16:24 CEST 2009 246 Mon Jul 27 19:16:29 CEST 2009 244 Mon Jul 27 19:16:34 CEST 2009 250 Mon Jul 27 19:16:39 CEST 2009 253 Mon Jul 27 19:16:44 CEST 2009 252 Mon Jul 27 19:16:49 CEST 2009 253 Mon Jul 27 19:16:54 CEST 2009 256 Mon Jul 27 19:16:59 CEST 2009 258 Mon Jul 27 19:17:04 CEST 2009 260 Mon Jul 27 19:17:09 CEST 2009 264 Mon Jul 27 19:17:14 CEST 2009 262 Mon Jul 27 19:17:19 CEST 2009 273 Mon Jul 27 19:17:24 CEST 2009 301 Mon Jul 27 19:17:30 CEST 2009 301 Mon Jul 27 19:17:35 CEST 2009 301 Mon Jul 27 19:17:40 CEST 2009 301

It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.

Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections. Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections. Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.

At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.

You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.

First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:

root@mail:~# cat /etc/courier/esmtptimeout 30 root@mail:~# cat /etc/courier/esmtptimeoutconnect 30 root@mail:~# cat /etc/courier/esmtptimeoutdata 180 root@mail:~# cat /etc/courier/esmtptimeouthelo 30

Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.

Second Idea:

I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.

Sam Varshavchik the lead programmer of courier told me following:

Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.

So ok not good and he said following about my idea to drop the connection ….

And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.

The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.

This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.

So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.