Howto install TeamSpeak 3 server on Ubuntu 10.04 (Lucid)

May 15, 2010

It has been a long time since my last post – I’m sorry for that but I didn’t have the time. Anyway I just installed TeamSpeak 3 on a Ubuntu 10.04 for a friend and want to share that info. Getting TeamSpeak running is mostly not the problem but you don’t want to start it after every boot by hand or run it as root. This Howto shows what I did. I assume that all user actions shown in this howto are performed as root or after executing sudo bash.

First you need to create a user under which the TeamSpeak server should run by executing following command:

adduser --disabled-login teamspeak

Now we need to get the software (64bit in my case)

wget http://ftp.4players.de/pub/hosted/ts3/releases/beta-22/teamspeak3-server_linux-amd64-3.0.0-beta22.tar.gz (Take also a look if a new version is out when you install your server)

and extract it

tar xzf teamspeak3-server_linux-amd64-3.0.0-beta22.tar.gz

We move it to a nice place with

mv teamspeak3-server_linux-amd64 /opt/ts3

and give it to the user teamspeak

chown -R teamspeak /opt/ts3

If you take a look into the /opt/ts3 directory you’ll see that there is a already a start/stop script (ts3server_startscript.sh), we will utilize it. Create a init.d file with pasting the content after executing cat > /etc/init.d/teamspeak :


#! /bin/sh
### BEGIN INIT INFO
# Provides:          teamspeak
# Required-Start:    networking
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      S 0 1 6
# Short-Description: TeamSpeak Server Daemon
# Description:       Starts/Stops/Restarts the TeamSpeak Server Daemon
### END INIT INFO

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="TeamSpeak Server"
NAME=teamspeak
USER=teamspeak
DIR=/opt/ts3
DAEMON=$DIR/ts3server_startscript.sh
#PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

cd $DIR
sudo -u teamspeak ./ts3server_startscript.sh $1

Now press ENTER and CTRL-D and you’ve inserted the content into the file. Set the permission correctly with

chmod 755 /etc/init.d/teamspeak

and now you can try it out by calling

/etc/init.d/teamspeak start

Take note of the login and token as you will need them later. You can also look for them in the log files in /opt/ts3/logs/. The last thing you need to do now is to make sure the init script is executed at boot time by using following command:

update-rc.d teamspeak defaults

At last if you’ve a firewall running on your system you need to make sure that you open all your ports. To find out which ports are used by teamspeak use following command:


# netstat -lnp | grep ts3
tcp        0      0 0.0.0.0:10011           0.0.0.0:*               LISTEN      30232/ts3server_lin
tcp        0      0 0.0.0.0:30033           0.0.0.0:*               LISTEN      30232/ts3server_lin
udp        0      0 0.0.0.0:9987            0.0.0.0:*                           30232/ts3server_lin

I hope this howto helped someone and write a comment if you found an error or a better way to do something. Now you just need to point your TeamSpeak client to the server and go to the menu entry “permissions | use token” and copy and past the token from above into the edit box. (only insert the chars behind “token=”)

Workaround for routing WOL (Wake on LAN) packets with Linux

January 16, 2010

If you want to send a WOL packet to a PC within your subnet it is really easy. Just install a program like wakeonlan (apt-get install wakeonlan) and type something like:


wakeonlan 01:02:03:04:05:06

But how to you send a WOL packet to an other subnet? Basically you use a UDP packet and send it to the broadcast address of the other network. e.g. with wakeonlan it looks like this


wakeonlan -i 192.168.1.255 01:02:03:04:05:06

But you need support for this from your router, as normally they don’t allow sending to the broadcast address from other networks. Professional routers/layer3 switches have support for this (you just need to enable it), but you’ve a Linux router at home? (e.g. one with Openwrt or Debian/Ubuntu)

The simplest way to get it working is to enter following on the router (rerun it at every boot):


arp -s 192.168.1.254 FF:FF:FF:FF:FF:FF

This tells the router that the given IP has a MAC address which is used for broadcasts. Now you only need to send the packet to this new “broadcast” address instead of the real one. So your wakeup call looks like this:


wakeonlan -i 192.168.1.254 01:02:03:04:05:06

ps: you should only enable something like this on a trusted network and the IP address you use should be not used by any other device.

KDE 4.x and Autostart

September 3, 2009

I was told by some fellow Linux guys that with KDE 4.x there is no Autostart possible anymore. I just wanted to write this blog post to show them that it is still possible even if it not in the KDE menu. Just create following directory if it does not exists

mkdir ~/.kde/Autostart

Some distributions may use .kde4 instead of .kde. Now just place a symlink into this directory with a command like this one for dropbox.

ln -s ~/.dropbox-dist/dropbox ~/.kde/Autostart/dropbox

At least on my Kubuntu systems with KDE 4.3.x that works. 😉

Mini-Howto: Restore Windows MBR/Bootloader with Linux

August 26, 2009

I’m often, at least more than I care, asked how to restore a Windows MBR/bootloader without having a windows install cd or a dos boot disk at hand. It’s quite easy you need just a Linux live cd like (the Ubuntu live cd or Knoppix) or an installed Linux you want get rid of. I really don’t know why you want to do the second, but anyway here are the 2 solutions I know of.

Boot Linux and make sure you’ve a working Internet connection and type following on the terminal/konsole.

1. Solution


sudo apt-get install syslinux

if the package got installed use following to write the MBR.


sudo dd if=/usr/lib/syslinux/mbr.bin of=/dev/sda

2. Solution


sudo apt-get install mbr

if the package got installed use following to write the MBR.


sudo install-mbr -i n -p D -t 0 /dev/sda

Common for both

Replace sda if you want to install the MBR to a different drive. Take a look at your hard disks with sudo fdisk -l if you’re unsure. Finally reboot and your windows should boot.

 

On request I put this comment from a guy (whats to be anonymous) up to the main article.

Dear Robert,

You have given an excellent article. And I know how much useful it may be to a frustrated person trying to get their work done.

But I sincerely request you to do one last job of giving them a last revealing information about alleged non friendliness of linux derivatives. I think at the moment when people are visiting your post they are frustrated and they vent their ire on linux without getting the other side of the story.

Please consider my suggestions below:

In response to your post, people have been wrongly complaining (in the comments) about the user unfriendliness of ubuntu and other linux variants etc in favour of Microsoft Windows. They think that after installing a linux variant they are unable to install the Microsoft’s Windows back, even after deleting all the partitions.

This actually happens because as soon as a Microsoft Windows OS installer sees a non Microsoft bootloader (for example, GRUB), it refuses to install and says something like the environment is incorrect. Sadly further, it does not explain the ‘environment’ (which could allow a user may look up a solution). And neither does it provide there, a straightforward option to overwrite the other bootloader with its own (even if you delete all the partitions on the disk from the Windows Installer or from the outside!).

I think the Windows implementers chose not to overwrite a foreign boot loader because they did not want to support a non Microsoft OS alongside their own. But the unfortunate thing is they should not do this when not even a single partition is allocated (then obviously there can be no OS installed).

Please let the readers know that it is the Microsoft windows which does not support any non Microsoft OS sitting along side itself and not necessarily the other way round. Please put this information in the main post itself, since otherwise it gets buried under the volume of the comments.

Workaround for the Ubuntu problem with KVM switches

It seems that Ubuntu not only Karmic (9.10), but also older versions have a problem with KVM (Keyboard, Video, Mouse) switches. To be exact the problem is the auto-detection of the capabilities of the monitor. If you connect the monitor directly to the computer everything works, if you use a KVM switch you get only 800×600 as the maximum resolution.

The workaround is to tell the xserver the Horizsync and Vertrefresh the monitor really supports. With older Ubuntu versions you could just add following lines (for a 1280×1024 LCD) to your /etc/X11/xorg.conf in the monitor section:


Section "Monitor"
        .....
        Option          "DPMS"
	Horizsync 31.5-64.0
	Vertrefresh 56.0 - 65.0
        .....
EndSection

But starting with Karmic Ubuntu has no /etc/X11/xorg.conf file by default anymore. So what we need is a complete minimal xorg.conf file so we can include our 3 lines, but we don’t want to mess anything else up. This is the minimal config I came up with.


Section "InputDevice"
	Identifier	"Generic Keyboard"
	Driver		"kbd"
	Option		"XkbRules"	"xorg"
	Option		"XkbModel"	"pc105"
	Option		"XkbLayout"	"de"
	Option		"XkbVariant"	"nodeadkeys"
EndSection

Section "InputDevice"
	Identifier	"Configured Mouse"
	Driver		"mouse"
EndSection

Section "Device"
	Identifier	"Configured Video Device"
EndSection

Section "Monitor"
	Identifier	"Configured Monitor"
        Option          "DPMS"
	Horizsync 31.5-64.0
	Vertrefresh 56.0 - 65.0
EndSection

Section "Screen"
	Identifier	"Default Screen"
	Monitor		"Configured Monitor"
	Device		"Configured Video Device"
        SubSection "Display"
                Depth           24
                Modes           "1280x1024" "1024x768"
        EndSubSection
EndSection

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "Configured Mouse"
EndSection

Except the Keyboard stuff the should be nothing thats not minimal. Maybe it is also possible to remove some lines there, but I didn’t test it. I was happy that it worked this way ;-). If you’ve an even more minimal config write a comment please!

Fast test if local mail works on a server

August 25, 2009

I just helped a friend whose server did not send cron mails to his mail server, which is this case led almost to data loss as the backup didn’t work correctly. I looked at the setup and I though I found the problem and correct it, but now I wanted to test it as easily as possible. Therefore I typed following command:

echo "Subject: test" | sendmail -v root

Then I looked if the lokal MTA, in this cache ssmtp delivered the mail to mail server. I though at least I find this line again if I search my blog and maybe it helps someone else too.

Courier-MTA reacts badly if own DNS server goes down [Update]

August 18, 2009

Today my DNS registrar and in my case also the provider of the DNS servers for my domains had a total blackout. He was down for about an hour, none of the 3 DNS servers was reachable. Ok, this was bad, no system could get the IPs for domain names – nothing I can to there except using a 4th DNS server operated by me in the future. But this was not the biggest problem, some DNS servers had the IP for my mail servers stored and tried to connect via SMTP to my courier-mta. The problem now is that courier didn’t accept the connections it tried to lookup its own name via the DNS servers in /etc/resolve.conf and ignored the settings in /etc/hosts (basically its own name). I thought surely I’ve a configuration error until I found following in the courier-mta FAQ.

NOTE: The Courier mail server does not read the hosts file. It needs a DNS server (although it is possible to have a working the Courier mail server configuration in a completely DNS-free environment, this excersize requires changing many configuration files, and perhaps will be its own FAQ entry some day).

What the fuck? Courier stops working if the DNS server for the own domain/hostname is not reachable? That must be a bug and not a feature. I will talk to the author and report back what he says.

[update] I talked with the author and other courier experts. The way courier is written it uses one internal function to all his DNS stuff and as it needs MX,TXT queries it uses a glibc function in it with does ignore the hosts file. There is currently no way around it, and so I’ll need to deploy an DNS server on my mail server to provide a DNS which can always resolve it’s own hostname. [/update]

The Search for reduced SPAM load – Part 3

July 29, 2009

Take a look at

to understand purpose of this series and what I’m looking for. As only firewalling spammers which are in a DNS RBL after they got a 5xx didn’t worked as hoped I had an other idea.

Instead of only firewalling the spammer I thought as there will be no packet from the spammer within the 30sec timeout why not just terminate the process which handles the connection. This leads to a reduced process number at once and makes space for a new one. A clean solution would implement my complete script in the mta itself – basically adding the IP to the firewall and terminating the smtp handling process. But for a mere test it would be easy to extend my script to kill the current process.

This python script (watchForSpammers2.py) does exactly that – It extends the old script by searching through the process list for a submit progress which handles the spammer connection. It follows the ppid and kills with a SIGTERM the parent courieresmtpd process.

You say thats a hard method? And you asked yourself if it works – yes it does.

After some testing I implemented it on the productive server and I’ve it running for a few days now and I didn’t reach the maximum of 300 connections since.

e.g. take a look at this spam wave

Tue Jul 28 08:30:51 CEST 2009 5
Tue Jul 28 08:30:56 CEST 2009 8
Tue Jul 28 08:31:01 CEST 2009 7
Tue Jul 28 08:31:06 CEST 2009 2
Tue Jul 28 08:31:11 CEST 2009 3
Tue Jul 28 08:31:16 CEST 2009 4
Tue Jul 28 08:31:21 CEST 2009 13
Tue Jul 28 08:31:26 CEST 2009 100
Tue Jul 28 08:31:31 CEST 2009 77
Tue Jul 28 08:31:36 CEST 2009 48
Tue Jul 28 08:31:41 CEST 2009 31
Tue Jul 28 08:31:46 CEST 2009 32
Tue Jul 28 08:31:51 CEST 2009 39
Tue Jul 28 08:31:56 CEST 2009 34
Tue Jul 28 08:32:01 CEST 2009 19
Tue Jul 28 08:32:06 CEST 2009 19
Tue Jul 28 08:32:11 CEST 2009 18
Tue Jul 28 08:32:16 CEST 2009 18
Tue Jul 28 08:32:21 CEST 2009 16
Tue Jul 28 08:32:26 CEST 2009 15
Tue Jul 28 08:32:31 CEST 2009 13
Tue Jul 28 08:32:36 CEST 2009 14
Tue Jul 28 08:32:41 CEST 2009 19
Tue Jul 28 08:32:46 CEST 2009 53
Tue Jul 28 08:32:51 CEST 2009 58
Tue Jul 28 08:32:56 CEST 2009 38
Tue Jul 28 08:33:01 CEST 2009 30
Tue Jul 28 08:33:06 CEST 2009 28

or the biggest in the last days


Tue Jul 28 14:36:19 CEST 2009 3
Tue Jul 28 14:36:24 CEST 2009 4
Tue Jul 28 14:36:29 CEST 2009 2
Tue Jul 28 14:36:34 CEST 2009 1
Tue Jul 28 14:36:39 CEST 2009 3
Tue Jul 28 14:36:44 CEST 2009 1
Tue Jul 28 14:36:49 CEST 2009 1
Tue Jul 28 14:36:54 CEST 2009 34
Tue Jul 28 14:36:59 CEST 2009 56
Tue Jul 28 14:37:04 CEST 2009 52
Tue Jul 28 14:37:09 CEST 2009 60
Tue Jul 28 14:37:14 CEST 2009 87
Tue Jul 28 14:37:19 CEST 2009 126
Tue Jul 28 14:37:24 CEST 2009 128
Tue Jul 28 14:37:29 CEST 2009 140
Tue Jul 28 14:37:34 CEST 2009 138
Tue Jul 28 14:37:39 CEST 2009 143
Tue Jul 28 14:37:44 CEST 2009 161
Tue Jul 28 14:37:49 CEST 2009 198
Tue Jul 28 14:37:54 CEST 2009 208
Tue Jul 28 14:37:59 CEST 2009 187
Tue Jul 28 14:38:04 CEST 2009 175
Tue Jul 28 14:38:09 CEST 2009 140
Tue Jul 28 14:38:15 CEST 2009 144
Tue Jul 28 14:38:20 CEST 2009 150
Tue Jul 28 14:38:25 CEST 2009 190
Tue Jul 28 14:38:30 CEST 2009 182
Tue Jul 28 14:38:35 CEST 2009 167
Tue Jul 28 14:38:40 CEST 2009 176
Tue Jul 28 14:38:45 CEST 2009 190
Tue Jul 28 14:38:50 CEST 2009 206
Tue Jul 28 14:38:55 CEST 2009 199
Tue Jul 28 14:39:00 CEST 2009 197
Tue Jul 28 14:39:05 CEST 2009 199
Tue Jul 28 14:39:10 CEST 2009 168
Tue Jul 28 14:39:15 CEST 2009 199
Tue Jul 28 14:39:20 CEST 2009 210
Tue Jul 28 14:39:25 CEST 2009 201
Tue Jul 28 14:39:30 CEST 2009 195
Tue Jul 28 14:39:35 CEST 2009 216
Tue Jul 28 14:39:40 CEST 2009 203
Tue Jul 28 14:39:45 CEST 2009 200
Tue Jul 28 14:39:50 CEST 2009 196
Tue Jul 28 14:39:56 CEST 2009 189
Tue Jul 28 14:40:01 CEST 2009 180
Tue Jul 28 14:40:06 CEST 2009 176
Tue Jul 28 14:40:11 CEST 2009 173
Tue Jul 28 14:40:16 CEST 2009 177
Tue Jul 28 14:40:21 CEST 2009 165
Tue Jul 28 14:40:26 CEST 2009 170
Tue Jul 28 14:40:31 CEST 2009 164
Tue Jul 28 14:40:36 CEST 2009 167
Tue Jul 28 14:40:41 CEST 2009 151
Tue Jul 28 14:40:46 CEST 2009 147
Tue Jul 28 14:40:51 CEST 2009 139
Tue Jul 28 14:40:56 CEST 2009 140
Tue Jul 28 14:41:01 CEST 2009 136
Tue Jul 28 14:41:06 CEST 2009 131
Tue Jul 28 14:41:11 CEST 2009 131
Tue Jul 28 14:41:16 CEST 2009 147
Tue Jul 28 14:41:21 CEST 2009 134
Tue Jul 28 14:41:26 CEST 2009 133
Tue Jul 28 14:41:31 CEST 2009 128
Tue Jul 28 14:41:36 CEST 2009 103
Tue Jul 28 14:41:41 CEST 2009 74
Tue Jul 28 14:41:47 CEST 2009 98
Tue Jul 28 14:41:52 CEST 2009 91
Tue Jul 28 14:41:57 CEST 2009 75
Tue Jul 28 14:42:02 CEST 2009 75
Tue Jul 28 14:42:07 CEST 2009 88
Tue Jul 28 14:42:12 CEST 2009 89
Tue Jul 28 14:42:17 CEST 2009 83
Tue Jul 28 14:42:22 CEST 2009 81
Tue Jul 28 14:42:27 CEST 2009 64
Tue Jul 28 14:42:32 CEST 2009 55
Tue Jul 28 14:42:37 CEST 2009 71
Tue Jul 28 14:42:42 CEST 2009 52
Tue Jul 28 14:42:47 CEST 2009 45
Tue Jul 28 14:42:52 CEST 2009 40
Tue Jul 28 14:42:57 CEST 2009 41
Tue Jul 28 14:43:02 CEST 2009 40
Tue Jul 28 14:43:07 CEST 2009 36
Tue Jul 28 14:43:12 CEST 2009 35
Tue Jul 28 14:43:17 CEST 2009 32
Tue Jul 28 14:43:22 CEST 2009 32
Tue Jul 28 14:43:27 CEST 2009 28
Tue Jul 28 14:43:32 CEST 2009 26
Tue Jul 28 14:43:37 CEST 2009 20
Tue Jul 28 14:43:42 CEST 2009 15
Tue Jul 28 14:43:47 CEST 2009 14
Tue Jul 28 14:43:52 CEST 2009 14
Tue Jul 28 14:43:57 CEST 2009 15
Tue Jul 28 14:44:02 CEST 2009 11
Tue Jul 28 14:44:07 CEST 2009 6
Tue Jul 28 14:44:12 CEST 2009 9
Tue Jul 28 14:44:17 CEST 2009 7
Tue Jul 28 14:44:22 CEST 2009 7
Tue Jul 28 14:44:27 CEST 2009 11
Tue Jul 28 14:44:32 CEST 2009 14
Tue Jul 28 14:44:37 CEST 2009 11
Tue Jul 28 14:44:42 CEST 2009 9
Tue Jul 28 14:44:47 CEST 2009 9
Tue Jul 28 14:44:52 CEST 2009 2
Tue Jul 28 14:44:58 CEST 2009 3

If you compare that to the values from my first post you see that it really works. Currently it is only a test script which is not tuned for performance, on a big wave I’ve problems to kill the processes as fast as they are forked but a better algorithm would help here. And I will look also into the possibility of limiting the amount of new connections per seconds I can handle per iptables.

But the biggest advantage would be if courier would be extended in a way that the smtp handle process adds the IP to iptables and terminates itself.

Anyway, I’ll try to make my code more than only a test script, but one that I can run in production 24/7. I’ll keep you posted – any ideas on your part?

The Search for reduced SPAM load – Part 2

July 28, 2009

As my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to withstand these waves. If packets and connections should not reach courier, I would need to use a smtp proxy or to something with the Linux kernel. As the OpenBSD spamd is not available for Linux,  I looked than through the iptables documentation and found the ipt_recent module.

It provides a userspace interface which enables a script/program to add IP addresses to a list which get them drop/reject for a given time. I thought this is exactly what I need. Why?

I cannot just drop/reject packages of IPs which are in DNS RBL, as maybe there is a false positive and he needs to know that there is a problem. The mail server needs therefore to send a 5xx the first time, but it is quite ok I think to not except connections for him some minutes after this.

This setup should at least give courier time to close the connections within the timeout, and denies a spammer trying to deliver more than one mail, or keep the connection up by ignoring the 5xx. So I went ahead. First I loaded the iptables module with an option to allow more IP addresses stored.


modprobe ipt_recent ip_list_tot=1000

I think 1000 is quite on the low end, as the spam waves easily reach them. Than I added following iptables commands to my firewall script.


# build sub chain
$iptables -N SPAMMER
# move all incomming smtp traffic there
$iptables -A INPUT -p tcp --dport 25 -j SPAMMER
# check if the source ip is already in the list, if so give it another 60 sec and drop the packages
$iptables -A SPAMMER -m recent --name spammer --update --seconds 600 -j DROP

As you can see we drop the packets for 10 minutes. If a packet is send within that 10min the time period starts again.

You can test your setup by doing following.


echo +1.1.1.1 >/proc/net/ipt_recent/spammer

and take a look at


cat /proc/net/ipt_recent/spammer

Replace 1.1.1.1 by an IP address of a spammer (just look in your logfile 😉 ) and see it working. Ok, now that we have the kernel/iptables part we need a script which adds the IP addresses of spammers on a DNS RBL after the first 5xx to the ipt_recent list.

As this is only for testing at this point I wrote a small script which watches the mail.log file and looks for 511 errors (the courier error code for DNS RBL hits) and add the IP addresses of the sending servers/zombies to ipt_recent.

Here is the python script: watchForSpammers.py. I don’t go into any details it is quite easy anyway. Start it like this within screen (apt-get install screen) to keep it alive even after logout.


./watchForSpammers.py /var/log/mail.log

The script does its work and adds IP addresses to the ipt_recent list which blocks them also nicely. Just type following to verify it.


iptables -L -xvn

You will see a line like this.


Chain SPAMMER (1 references)
pkts bytes target prot opt in out source destination
15213 724935 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 recent: UPDATE seconds: 600 name: spammer side: source

But the big question is, does it help against the spam waves and its length and heights? More No than Yes ;-). It does almost nothing against the height, but it allows courier to go back a little faster to normal. But still my mail server is maxed out.

So I continue to search for another way. Any ideas?

The Search for reduced SPAM load – Part 1

July 27, 2009

This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.

One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.

The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.


Mon Jul 27 19:12:17 CEST 2009     4
Mon Jul 27 19:12:22 CEST 2009     5
Mon Jul 27 19:12:27 CEST 2009     5
Mon Jul 27 19:12:32 CEST 2009     7
Mon Jul 27 19:12:37 CEST 2009     13
Mon Jul 27 19:12:42 CEST 2009     9
Mon Jul 27 19:12:47 CEST 2009     8
Mon Jul 27 19:12:52 CEST 2009     50
Mon Jul 27 19:12:57 CEST 2009     88
Mon Jul 27 19:13:02 CEST 2009     101
Mon Jul 27 19:13:07 CEST 2009     120
Mon Jul 27 19:13:12 CEST 2009     129
Mon Jul 27 19:13:17 CEST 2009     135
Mon Jul 27 19:13:23 CEST 2009     145
Mon Jul 27 19:13:28 CEST 2009     150
Mon Jul 27 19:13:33 CEST 2009     163
Mon Jul 27 19:13:38 CEST 2009     167
Mon Jul 27 19:13:43 CEST 2009     175
Mon Jul 27 19:13:48 CEST 2009     180
Mon Jul 27 19:13:53 CEST 2009     183
Mon Jul 27 19:13:58 CEST 2009     190
Mon Jul 27 19:14:03 CEST 2009     202
Mon Jul 27 19:14:08 CEST 2009     204
Mon Jul 27 19:14:13 CEST 2009     204
Mon Jul 27 19:14:18 CEST 2009     203
Mon Jul 27 19:14:23 CEST 2009     206
Mon Jul 27 19:14:28 CEST 2009     207
Mon Jul 27 19:14:33 CEST 2009     211
Mon Jul 27 19:14:38 CEST 2009     216
Mon Jul 27 19:14:43 CEST 2009     218
Mon Jul 27 19:14:48 CEST 2009     220
Mon Jul 27 19:14:53 CEST 2009     222
Mon Jul 27 19:14:58 CEST 2009     227
Mon Jul 27 19:15:03 CEST 2009     229
Mon Jul 27 19:15:08 CEST 2009     228
Mon Jul 27 19:15:13 CEST 2009     229
Mon Jul 27 19:15:18 CEST 2009     232
Mon Jul 27 19:15:23 CEST 2009     234
Mon Jul 27 19:15:28 CEST 2009     234
Mon Jul 27 19:15:34 CEST 2009     237
Mon Jul 27 19:15:39 CEST 2009     240
Mon Jul 27 19:15:44 CEST 2009     241
Mon Jul 27 19:15:49 CEST 2009     243
Mon Jul 27 19:15:54 CEST 2009     244
Mon Jul 27 19:15:59 CEST 2009     244
Mon Jul 27 19:16:04 CEST 2009     246
Mon Jul 27 19:16:09 CEST 2009     245
Mon Jul 27 19:16:14 CEST 2009     246
Mon Jul 27 19:16:19 CEST 2009     247
Mon Jul 27 19:16:24 CEST 2009     246
Mon Jul 27 19:16:29 CEST 2009     244
Mon Jul 27 19:16:34 CEST 2009     250
Mon Jul 27 19:16:39 CEST 2009     253
Mon Jul 27 19:16:44 CEST 2009     252
Mon Jul 27 19:16:49 CEST 2009     253
Mon Jul 27 19:16:54 CEST 2009     256
Mon Jul 27 19:16:59 CEST 2009     258
Mon Jul 27 19:17:04 CEST 2009     260
Mon Jul 27 19:17:09 CEST 2009     264
Mon Jul 27 19:17:14 CEST 2009     262
Mon Jul 27 19:17:19 CEST 2009     273
Mon Jul 27 19:17:24 CEST 2009     301
Mon Jul 27 19:17:30 CEST 2009     301
Mon Jul 27 19:17:35 CEST 2009     301
Mon Jul 27 19:17:40 CEST 2009     301

It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.


Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.

At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.

You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.

First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:


root@mail:~# cat /etc/courier/esmtptimeout
30
root@mail:~# cat /etc/courier/esmtptimeoutconnect
30
root@mail:~# cat /etc/courier/esmtptimeoutdata
180
root@mail:~# cat /etc/courier/esmtptimeouthelo
30

Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.

Second Idea:

I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.

Sam Varshavchik the lead programmer of courier told me following:

Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.

So ok not good and he said following about my idea to drop the connection ….

And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.

The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.

This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.

So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 34 queries. 0.073 seconds.