Information Leak due OEBB Railjet WiFi

November 4, 2013

Today I traveled with the OEBB Railjet which provides a free WiFi. As the journey took some hours I had time to look at my networks traces and found something. After the captive portal with the Terms of Services was acknowledged, a page with some infos is shown. One of the infos is the original URL the user requested. If the users clicks on the link a separate tab opens with the page. The problem is that the URL the browser was given to access this info page has following format:

http://10.20.43.1:4990/www/portal/index.html?res=notyet&uamip=10.20.43.1&uamport=3990&challenge=7e623f476a0cd811b414ad6aa9cfce13&called=00-C0-3A-9D-55-E8&mac=xx-xx-xx-xx-xx-xx&ip=10.20.43.33&nasid=nas01&sessionid=5277d1c60000000b&userurl=http%3a%2f%2fwww.orf.at%2f

Which is sent as referrer to the original requested page if you click onto the link. As you see this referrer contains the full MAC address of the requesting device. Normally the MAC address is only visible via Layer 2 but with the information leak in my case www.orf.at knows my MAC address and if I have already gotten a cookie, they could add now my MAC to the list of know IDs. Ok, I guess the ORF doesn’t do that, but others might.

A solution would be simple for the OEBB, but until then don’t click on this link – type the URL again.

Session verification fails after update from PHP 5.3 to 5.4 with suhosin

September 15, 2013

I’ve just upgraded my PHP install from 5.3.25 to 5.4.19 and I ran into the problem that some PHP programs on my server stopped working. The first I found to have a problem was Tiny Tiny RSS as I use it myself. 🙂 I was not able to login into it anymore and in the log file I found following:

[Sun Sep 15 11:00:31 2013] [warn] [client xxx.xxx.xxx.xxx] mod_fcgid: stderr: PHP Warning:  Unknown: Failed to write session data (user). Please verify that the current setting of session.save_path is correct (/var/xxxxxx/sessions/) in Unknown on line 0, referer: https://xxxxxxxx/

After searching really long I found out that it worked again if I disabled suhosin (which is a module to harden PHP) by editing /etc/php.d/suhosin.ini and putting a ; in front of

extension=suhosin.so

But this is not secure way to handle this, therefore I search further and found a pull request on GitHup which solves the problem. Ok you need to patch and compile the module … but technically it is fixed 😉

 

Simple way to secure a MySQL installation

May 27, 2013

I now know this script was there for years but I just found it some month ago, when I installed MySQL on a CentOS and it told me to use it. But I checked, it is also present on an old Ubuntu 8.04 LTS (released 5 years ago). The script is called:

/usr/bin/mysql_secure_installation

Here are the questions you get asked by it:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You really should run this program as it makes the basic securing of MySQL really easy.

How to get Tiny Tiny RSS running on a half-way secure PHP setup

May 20, 2013

As Google is discontinuing their Google Reader I needed to look for alternatives and after some looking I went for Tiny Tiny RSS, which I can run on one of my web servers. At least this was the theory :-). The problems I faced were that Tiny Tiny RSS seems to have problems with a half-way secure PHP setup, and I needed therefore to make some changes to get it working. As I didn’t find that much documentation on this topic and it seams the Tiny Tiny RSS developers are not really security affine.

1. open_basedir is not supported

I did as described in the Install Notes and went to the http://your.site.com/tt-rss/install/ URL to use the installer to configure my Tiny Tiny RSS, but I was prompted with this error message:

PHP configuration option open_basedir is not supported. Please disable this in PHP settings file (php.ini).

I searched through the Internet and it seams that some obscure errors can occur if you use open_basedir. But to me it seem more like: I (=developer) don’t have the problem and the user has it and he has open_basedir activated so this must be the problem. Anyway, open_basedir is an imported security measure for PHP hosting, do not ever disabled it, specially if you have more than one vhost. I running with enabled open_basedir for 2 weeks now without getting any obscure errors (or any errors at all).  At the time of writing this blog post I was running PHP 5.3.23 and 5.3.25 on this server.

After these preliminary words – how did I get rid of this error messages? Simple I just commented raising of the exceptions out. For this open the install/index.php in a text editor and search for the line containing “option open_basedir is not supported“, comment it out like this:

And you need to do the same in the include/sanity_check.php file:

2. passthru disabled

I wanted to call the ./update.php --daemon command from the command line but I got following error message:

PHP Warning:  passthru() has been disabled for security reasons in /var/www/xxxx/html/update.php on line 150

This is because a secure configured default php.ini has following set:

disable_functions = exec, passthru, shell_exec, system, proc_open, popen, curl_exec, curl_multi_exec, parse_ini_file, show_source

I remove disabled functions per vhost so I don’t have the least common denominator for security. In this case needed to make sure that the update.php called via the command line uses a custom made php.ini. I therefore created this bash script.

#!/bin/bash
export PHPRC="/var/www/php/xxxx/"
./update.php --daemon

In this directory a special php.ini is located which has passthru enabled.

3. undefined function posix_getpid

I wanted to use the update_daemon2.php script as it is recommended but I got following error message:

[09:48:30] [MASTER] installing shutdown handlers
PHP Fatal error: Call to undefined function posix_getpid() in /var/www/xxxx/html/update_daemon2.php on line 223
PHP Fatal error: Call to undefined function posix_getpid() in /var/www/xxxx/html/update_daemon2.php on line 236
PHP Fatal error: Call to undefined function posix_getpid() in /var/www/xxxx/html/update_daemon2.php on line 99

The solution on Centos/RHEL/SL is easy, just install the posix package for php:

yum install php-posix

I hope this post helps others which move to Tiny Tiny RSS and still want a reasonably secure PHP setup.

List of IEEE 802.1x hotfixes for Windows 7

February 16, 2013

Most companies which want to enable 802.1x for their wired network run Windows 7 SP1. After activating 802.1x you’ll run into various problems with your Windows 7 boxes. It will work fine in lap tests but it will fail in the real world. Why is that? Because there are many 802.1x bugs in it.

Normally I blog about Linux, networking and security and not Windows related stuff. I’m not an Windows expert but I needed to get it authenticated with my network so needed to look into the matter. I hope it helps other networks guys. I’m working for over 8 month (no not all the time 😉 ) now to get Windows 7 SP1 100% of the time working with 802.1x. Its working 99% of the time, but there are still errors (under rare and special condition) that occur. ;-(

At the time of writing this list I didn’t find any other site that lists the available hotfixes, so I though I start a list. And the description texts from Microsoft to these hotfixes are sometimes brain dead.  Anyway some entries are the result of working with the Microsoft Premier Support on cases. If you’ll find any other hotfix, fixit, … please let me know.

ps: We’re using EAP-TLS so I can only write about patches I needed for it.

1. KB2481614
   If you’re configuring your 802.1x settings via Group Policy you’ll see sometimes EAP-PEAP request from clients in your radius server log during booting even if you’ll set EAP-TLS. This error happened in our case with 1/3 of the boots with some models. The error is caused by a timing problem during startup. Sometimes the 802.1x is faster and sometimes the Group Policy is, and if the 802.1x is faster than the default configuration is taken, which is PEAP. Which lead to a EAP-NAK by the radius server.
2. KB980295
   If an initial 802.1x authentication is passed, but a re-authentication fails, Windows 7 will ignore all later 802.1x requests. This hotfix should also fix a problem with computers waking up from sleep or hibernation – but we’ve disabled these features so I can’t comment on them.
3. KB976373
   This hotfix is called “A computer that is connected to an IEEE 802.1x-authenticated network via another 802.1x enabled device does not connect to the correct network”. I can’t comment on this, as we’ve not deployed 802.1x for our VoIP phones at this point. But it solves one other problem, which is described here. The Windows Vista hotfix for the same problem, linked in the article (There is a third hotfix related but not linked in the article, its for XP – so it seems the problem is through the whole product line), states that if an error occurs Windows is normally hard-coded to ignore EAPOL packets for 20min. I would guess it is the same for Windows 7 too. The linked article tells you to install the patch and set some registry key to lower the value.
4. KB2769121
   A short time ago I found this one: “802.1X authentication fails on a Windows 7-based or Windows 2008 R2-based computer that has multiple certificates”. At time of writing I’m not sure if it helps for something in my setup. According to the symptoms list of the hotfix, it does not, but maybe it helps for something else, as the one before does.
5. KB2736878
   An other error during booting – this time it happens if the read process starts before the network adapter is initialized. Really seems that they wanted to get faster boot times, no matter the costs.
6. KB2494172
   This hotfix fixes a problem if you’ve installed a valid and invalid certificate for 802.1x authentication. The workaround is just deleting the invalid certificate. I’m not sure at this point if it affects also wired authentication.
7. KB976210
   This problem occurs only during automated build processes and if you use an EAP method which needs user interaction – as I don’t do that I can’t comment on this hotfix.

So far this is my list – with the list you should get running Windows 7 and 802.1x nicely, but it is not perfect – Do you know any other patches or workarounds?

 

FreeRADIUS and CRLs – Done the right way [Update]

May 28, 2012

Update: I changed the FreeRADIUS in-line CRL verification to an external program – running it now for several month at it works without restart of FreeRADIUS.

While I was implementing 802.1x EAP-TLS with FreeRADIUS I googled for documentation on how to implement Certificate Revocation Lists (CRL) in FreeRADIUS. The first text was in the eap.conf:


# Check the Certificate Revocation List
#
# 1) Copy CA certificates and CRLs to same directory.
# 2) Execute 'c_rehash '.
# 'c_rehash' is OpenSSL's command.
# 3) uncomment the line below.
# 5) Restart radiusd
check_crl = yes
CA_path = /etc/freeradius/certs/CA/


Which is basically correct, but it is so short on information that I googled on and found following posts:

• A solution where you need to combine the CA and the CRL in one File (and you need to restart FreeRADIUS to reload the CRL)
• Also ignoring the c_rehash stuff and using a variable crl_file (and I guess, as in this case as didn’t try it, restart FreeRADIUS to reload the CRL)

Both where not the solution I needed as I didn’t want to restart FreeRADIUS all the time and I have multiple sub CAs. I than started googling for more information on c_rehash, as the version on RHEL5/CentOS5/SL5 don’t need to have a man page. Take a look at this online man page.

And just as info as it is not that clear which package has c_rehash: yum provides "*/c_rehash" --> yum install openssl-perl. Basically c_rehash needs to be provided a directory with .pem files in it (also the CRLs need to be name .pem) and it than creates symlinks with the hashes of the files as names. After I got a prototype working but before I wrote this blog, Erik Inge Bolso wrote this blog post describing the same thing.

You need at least FreeRADIUS 2.1.10 (shipped with Centos/RHEL 5 (inc. updates) and later) for this solution to work.

After I got the prototype working I wrote a script which does download multiple CRLs, converts them from DER (e.g. used by Windows CAs) to PEM, verifies them and than uses c_rehash to hash them for FreeRadius. You need to do following steps to get it working for you.

1. Create some directories
mkdir /var/tmp/cacheCRLs
mdir /etc/pki/crl/

2. download this script cacheCRLs4FreeRadius.py to /usr/local/sbin/
cd /usr/local/sbin/
wget http://robert.penz.name/wp-content/uploads/2012/05/cacheCRLs4FreeRadius.py

3. edit /usr/local/sbin/cacheCRLs4FreeRadius.py and change the URLs and names to your CAs
4. run /usr/local/sbin/cacheCRLs4FreeRadius.py, no output means no error and check the content of /etc/pki/crl/
5. check that your radius config contains following and restart FreeRADIUS after the change
# we're using our own code for checking the CRL
# check_crl = yes
CA_path = /etc/pki/crl/
....
tls {
....
verify {
tmpdir = /var/tmp/radiusd
client = "/usr/local/sbin/checkcert.sh ${..CA_path} %{TLS-Client-Cert-Filename}"
}
}

6. /usr/local/sbin/checkcert.sh should contain following
   #!/bin/sh
   output=`/usr/bin/openssl verify -CApath $1 -crl_check $2`

   if [ -n "`echo $output | /bin/grep error`" ]; then
RC=1
else
RC=0
fi
echo $output
exit $RC


7. If you need more performance replace the shell script a C program, as this program is started at every authentication request.
8. Try to authenticate with an revoked certificate and you should get following. If you’re running FreeRADIUS with -X you should see following (this log looks a little bit different when using the external program, which is the method I’m using now) :
   
[eap] Request found, released from the list
[eap] EAP/tls
[eap] processing type tls
[tls] Authenticate
[tls] processing EAP-TLS
[tls] eaptls_verify returned 7
[tls] Done initial handshake
[tls] < << TLS 1.0 Handshake [length 05f8], Certificate --> verify error:num=23:certificate revoked
[tls] >>> TLS 1.0 Alert [length 0002], fatal certificate_revoked
TLS Alert write:fatal:certificate revoked
TLS_accept: error in SSLv3 read client certificate B
rlm_eap: SSL error error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
SSL: SSL_read failed in a system call (-1), TLS session fails.
TLS receive handshake failed during operation
[tls] eaptls_process returned 4
[eap] Handler failed in EAP/tls
[eap] Failed in EAP select

9. run the script via cron in intervals that are required in your setup. e.g. once a day or once every hour

Howto connect multiple networks over the Internet the cheap way

October 17, 2010

I’m quit often asked by 2 types of people how to connect cheaply multiple networks securely over the Internet. The first type are owners of small companies which have more than one office and want to connect them to their central office. And the other type are people who are the de facto IT guy for their family and friends and need an easy way to get into the the other networks.

In the beginning most of them start with with remote connection software like Teamview, VNC, but at some point thats not enough anymore, when the responsibilities grow. The solution that I implement for them is based on the OpenVPN which is a well know, free and secure Open Source VPN solution, which is able to run on cheap hardware. Why do I say it needs hardware when you see on their homepage that it runs on Windows too?

1. You want a system with which nobody messes around and so it works for years without ever touching it again. And yes that is possible with the proposed solution. If you can’t reach the router it normally a ISP problem or power outage.
2. You don’t have only one Computer in each network and one VPN for each computer is not a good idea if you don’t need it.
3. There are also devices in the network where you can’t or won’t install a VPN software. e.g. printer, TV, receiver, acess point, ….

Anyway you can also use this setup to let road warriors into your network over the Internet. I’ll also show how to do that, but it is not the main focus of this blog post.

So about what hardware I’m talking?

1. An Accesspoint/DSL/Cable Router which is able to run OpenWRT, or any other system for which OpenVPN is available. I basically stick with systems that can run OpenWRT as I want similar systems to minimize my efforts. To be precise I’m almost always use a Linksys WRT54GL which you get under 40 Euros. But you’re free to use anything else. e.g. a friend of mine has a setup working with a SheevaPlug and Debian on it, an other has a setup running with DD-WRT.
2. If you have a local server in the network, be it Linux or Windows based, you can use that do. But don’t use clients if its not for road warriors.

The Setup

You need one OpenVPN node per network/location. In the following example we have

• one central office network with a server and the router is from Internet service provider and we cannot install software on it, the server is also used as DHCP server. (I’ve specially selected that as it shows what additionally needs to be done if the OpenVPN server is not the default gateway for the devices in the network. If for your setup thats in a remote location its the same setup there.)
• one remote office which is connected via cable the Internet. An OpenWRT based router is connected to the cable modem and masquerades the devices behind it.
• one remote office which is connected via DSL to the Internet. An OpenWRT based router inserted after the provider router, both masquerades and to make the life interesting you cannot change anything on the provider router, e.g. add a port forwarding

In this howto we’re working with following subnets:

• 10.23.0.0/24 – central office
• 10.23.1.0/24 – remote office 1
• 10.23.2.0/24 – remote office 2
• 10.23.255.0/24 – transport network remote offices
• 10.23.254.0/24 – transport network road warriors (not in the figure)

In the figure above you see that each Network has a different subnet – this is important as we need to route packets! Advise: Always use different subnets for different locations you set up. Even if you don’t connect them together at the beginning, make a list which subnets you’ve already used. There are plenty subnets in the 10.0.0.0 - 10.255.255.255 range for a single guy or a small company. Use a separate C class networks for each location (e.g. 10.42.0.0/24 for the first, 10.42.1.0/24 for the second and so on).

One subnet is needed as transit network within the VPN to connect the all remote offices to the central one. One subnet is only needed if you also want road warriors to connect to your network.

Preconditions

Common to all locations:

Get OpenVPN installed on all 3 systems. The version can differ a little between the servers which makes it easy if your OpenWRT router has an older version as you use for your main server.

Central office:

• You need to forward 2 UDP ports on the ISP router from the Internet to your server. The example ports I use in this howto are 10000 for the VPN connection of the remote offices and 10001 for the road warriors.
• Create an DNS entry for the external IP address of your router. If you’ve a dynamic IP address or no own DNS server/domain you can use for example the dyndns.com service; during this howto I assume centraloffice.dyndns.org as the host name.
• You need to add a routing rule in your DHCP config which tells the clients that 10.23.0.0/16 is reachable behind the IP of your sever. And yes the 10.23.0.0/16 also includes the 10.23.0.0/24 of the central office, but it is important to remember that the more specific rule counts. The usage of one 10.23.0.0/16 and not a rule for ech C Class subnet allows you to add remote offices later without changing the DHCP. If your DHCP server is not able to add routing rules,  a workaround is possible if your default gateway used by the clients (in this case the ISP router) is able to add the 10.23.0.0/16 rule to its routing table. In this case every traffic goes first to the default gateway and then back to the server. Thats not nice but for a small network with only a low bandwidth connection to the other networks it is ok. In a big network the security / network guy will cry…..
• Make sure that routing is enable on the server. On Linux systems you can check that easily with cat /proc/sys/net/ipv4/ip_forward. If the result is 1 it is enabled otherwise not.
• Make sure that if you’ve a local firewall enabled that UDP ports 10000 and 10001 are open.

Remote offices:

All requirements are meet if you’ve installed a modern version of OpenVPN and if the systems have the correct time. This maybe harder than it sounds as most DSL routers don’t have an internal clock that survives reboot. But it is important that the time of the router is bigger than the time the certificates where created, as otherwise the systems will not accept the certificate of the node in the central office. An easy way to make sure that at least this requirement is fulfilled is to set the date at boot to something fixed. I’ll show a simple way to do this later.

Firewall considerations

OpenVPN uses tun* device names for the VPN tunnels. On the central office server you’ve have tun0 for the remote offices and tun1 for the road warrios. For testing purposes you should let any traffice from and to tun0,tun1 and eth0 (LAN interface) through. After you made sure that everything works you should limit that to traffic which is really needed. On the remote offices nodes I recommend to open the traffic permanently in both directions (eth0 to tun0 and tun0 to eth0) as otherwise you need to administer the firewall rules on on multiple systems.

Import: You need also firewalls rules (on the central office server) that allow the traffic from one remote office to the other if you wish that kind of traffic. OpenVPN will be configured in a way that lets that kind of traffic through but without firewall rules that also allow the traffic it will not work.

Setting up a Certicate Authority (CA)

So what is  a CA? I copied the first paragraph and linked the explanation of wikipedia as a starting point.

In cryptography, a certificate authority or certification authority (CA) is an entity that issues digital certificates. The digital certificate certifies the ownership of a public key by the named subject of the certificate. This allows others (relying parties) to rely upon signatures or assertions made by the private key that corresponds to the public key that is certified. In this model of trust relationships, a CA is a trusted third party  that is trusted by both the subject (owner) of the certificate and the party relying upon the certificate. CAs are characteristic of many public key infrastructure (PKI) schemes.

In more common words each VPN nodes needs to have a one certificate (*.crt – containing the public key and the signature of the CA) and one private key. Each node needs to trust the CA and for this they have the certificate of the CA stored. For each node you need therefore at least 3 cryptography files. The certificate file also contains a flag describing a node as server or client. We’ll configure the OpenVPN clients in a way that will make sure that they connect only to nodes with the server flag.

The CA should be not on one of the VPN nodes but on separate computer. The easiest way to reach a working and secure enough solution for a small company is to put the CA onto a USB stick and only insert it on a secure PC if you’ll need it for generating new certificates.

Now lets get to the doing part, I’ll will assume that you’ve a Linux or Linux like system as your CA system but it will also work on others. In your OpenVPN package you’ll find a directory or .gz file named something like easy-rsa-2.0. On an Ubuntu/Debian system you’ll find it under /usr/share/doc/openvpn/examples/easy-rsa/2.0/ after installing the package openvpn.

1. Copy or extract easy-rsa to a directory of your liking.
2. Open the vars file in a text editor and scroll down to the end of the file. There you can change the key length to something higher than 1024 but keep in mind that you’ve not the fastest systems if you’re using OpenWRT on a DSL Router. You should also change the default country, city, … entries. The default lifetime of the certificates with 10 years should be enouch 😉
3. Type . vars which will load the variable in the environment of your current shell. You need to this step every time you start with an fresh shell and want todo any CA operation.
4. Call ./clean-all – Important: Only do that in the beginning as it will delete your complete CA and certificates if you’ll use it later.
5. Call ./build-ca mycompany-ca to generate the CA, you can leave most questions to the default answers. The important part the common name is already specified a command line parameter.
6. Call ./build-dh to get a special file for the OpenVPN node in the central office
7. Call ./build-key-server centralofficenode to generate the certificate for your central office node. Change the name to something meaningfull. Important: The name (= common name) needs to be unique for all nodes and I recommend to use only A-Z,a-z,0-9,-,_ as common name.
8. Call ./build-key remoteofficenode for each remote location you want to connect. Important: Use a system to choose the names otherwise you’ll loose overview of it if you’ll get more nodes and subnets. e.g. the name of the location/network you typically use to refer to it.

You’ll find all files in the keys subdirectory and you need to copy the .crt, .key of each node and the CA file to the node (and not the files for an other node!) For the node in the central office you need also the dh1024.pem.

road warriors:

If you also want to setup a VPN service for road warriors I recommend to use a separate CA – so make a second copy of the easy-rsa and start again. Import use other names otherwise you will loose the overview.

OpenVPN configuration

Central office:

I assume in this howto that it is a Linux server (Ubuntu/Debian based to be exact) in your central office, but the configuration on for example Windows is the basically the same – just the paths, starting methods and logging stuff are different.

1. Create the directory /etc/openvpn/certs/ and copy the centraloffice.crt, centraloffice.key, ca.crt and the dh1024.pem to this directory and change the permissions for the directory to 700 and for all files to 600 and the owner should be root for both.
2. Create a config file /etc/openvpn/0_remote_offices.conf with following content:
   
port 10000
proto udp
dev tun
ca /etc/openvpn/certs/ca.crt
cert /etc/openvpn/certs/centraloffice.crt
key /etc/openvpn/certs/centraloffice.key
dh /etc/openvpn/certs/dh1024.pem
server 10.23.255.0 255.255.255.0
client-to-client
client-config-dir /etc/openvpn/remoteoffice_networks
verb 3
# we can here operate with the same B class network as the more specify rules counts
route 10.32.0.0 255.255.0.0
push "route 10.32.0.0 255.255.0.0"
keepalive 10 120
cipher AES-128-CBC   # AES
comp-lzo
max-clients 100
persist-key
persist-tun

3. Create the directory /etc/openvpn/remoteoffice_networks and insert one file for each remote relocation with the name of the common name you choose for it. The files contain only one line with following content for the first remote location iroute 10.23.1.0 255.255.255.0 and iroute 10.23.2.0 255.255.255.0 for the second remote location. This tells OpenVPN which remote network is reachable behind which node.
4. Optional: Copy the road warrior CA stuff needed to the certs directory. Important: Make sure that you use an other name as ca.crt, as you’ve that one for the remote offices.
   5. Optional: Create a configuration file /etc/openvpn/1_remote_offices.conf with following content for the road warriors:
   
port 10001
proto udp
dev tun
ca /etc/openvpn/certs/ca_roadwarrior.crt
cert /etc/openvpn/certs/centraloffice_roadwarrior.crt
key /etc/openvpn/certs/centraloffice_roadwarrior.key
dh /etc/openvpn/certs/centraloffice_dh1024.pem
server 10.23.254.0 255.255.255.0
verb 3
push "route 10.23.0.0 255.255.0.0"
keepalive 10 120
# Maybe you need to comment this out, as your e.g. Windows client uses an slightly other encoding and so you can't transfer data. But if possible AES is the best.
cipher AES-128-CBC   # AES
comp-lzo
max-clients 100
persist-key
persist-tun

5. Restart the OpenVPN server with /etc/init.d/openvpn restart and check if the configuration was at least that much correct to start up.
6. Run tail -f /var/log/syslog (or your distribution equivalent) to check on entries if the OpenVPN clients try to connect.

Remote Offices:

You need to do following steps for each remote office and replace the filenames with the correct ones for the specify remote location.

1. Create the directory /etc/openvpn/certs/ and copy the remoteoffice*.crt, remoteoffice*.key and ca.crt to this directory and change the permissions for the directory to 700 and for all files to 600 and the owner should be root for both.
2. Create a config file /etc/openvpn/0_to_the_central_office.conf with following content:
   
client
dev tun
proto udp
remote centraloffice.dyndns.org 10000
resolv-retry infinite
ns-cert-type server
cipher AES-128-CBC
ping 15
ping-restart 45
ping-timer-rem
persist-tun
persist-key
comp-lzo
verb 3
ca /etc/openvpn/certs/ca.crt
cert /etc/openvpn/certs/remoteoffice*.crt
key /etc/openvpn/certs/remoteoffice*.key

3. Create an init.d script /etc/init.d/openvpn with the permissions 755 and the content:
   
# change the date to something bigger than the time you used to generate the certificates
date -s "2010-03-01 10:00:00"
cd /etc/openvpn
openvpn --config /etc/openvpn/0_to_the_central_office.conf --daemon

4. Create an symlink to start the init.d script automatically.
   
cd /etc/rc.d/
ln -s ../init.d/openvpn S65openvpn

5. For the first test you should do the steps in the init.d script by hand to check if they work and to see potential errors. For this don’t copy the --daemon parameter for the OpenVPN as this way you’ve see all errors and can easily stop OpenVPN with CRTL-C. Look also at the OpenVPN log entries on the central office node as it may also contain important error message.

Road Warriors:

The configuration file looks similar to the remote offices ones, just use the 10001 port and use the files from the road warrior CA. On Windows you should use the GUI version of OpenVPN which needs the configuration file normally in c:\programs\openvpn\conf with the extension .ovpn. Copy also the files into the same directory and use no absolute paths as it may varies on different clients and you don’t want to change it for every client.

Conclusion

I hope this howto has shown how easily you can connect various networks via the Internet with Open Source tools and cheap hardware. I’ve you’ve any questions please write them in the comments below that post, I’ll try to answer them. This setup is only the beginning, if there is interest I can show some stuff I use to extend this setup. I’ve also written a blog post some time ago which is interesting for you if you use the same CA for different VPN Server and want to make sure that only a some clients are allowed to connect to a specific server.

Free SSL certificates will be supported in IE/Windows [Update]

August 21, 2009

In the StartCom Blog the head of the company writes that its CA will be included by Microsoft. He states, “Starting approximately the 22nd of September, Microsoft intends to distribute a non-security update package to the Windows operating systems which includes the trusted StartCom root certificate and the automatic root certificate update service will update the cryptographic certificates root store on those systems whenever a StartCom issued certificate is encountered.”

Why is this worth reporting? This CA is with the above mentioned date the first CA which provides a Free SSL certificate which is supported by Microsoft and therefore by all Internet Explorer browsers. Sure this is “only” a SSL certificate which only assures the domain name or email address, but this is much more than now. You find many small mail servers where the webmail/IMAP/POP3/SMTP or a small homepage for a few users is self signed. All these can now be protected much better, the only problem seems to be that the CA is not supported by Firefox and other open source players. The other authority I know which provides free SSL certificates is CAcert. They are much stronger in the open source world, but with the inclusion of the StartSSL CA by Microsoft, they will have a much lower install base of browsers than StartSSL.

[Update] Mozilla and Apple support this CA for years already. My error – sorry. So these Free SSL certs are really something for a small website or mail server.[/Update]

The Search for reduced SPAM load – Part 2

July 28, 2009

As my first ideas discussed in the first part of this series didn’t work out as I liked it, I went ahead and looked for other means to withstand these waves. If packets and connections should not reach courier, I would need to use a smtp proxy or to something with the Linux kernel. As the OpenBSD spamd is not available for Linux,  I looked than through the iptables documentation and found the ipt_recent module.

It provides a userspace interface which enables a script/program to add IP addresses to a list which get them drop/reject for a given time. I thought this is exactly what I need. Why?

I cannot just drop/reject packages of IPs which are in DNS RBL, as maybe there is a false positive and he needs to know that there is a problem. The mail server needs therefore to send a 5xx the first time, but it is quite ok I think to not except connections for him some minutes after this.

This setup should at least give courier time to close the connections within the timeout, and denies a spammer trying to deliver more than one mail, or keep the connection up by ignoring the 5xx. So I went ahead. First I loaded the iptables module with an option to allow more IP addresses stored.


modprobe ipt_recent ip_list_tot=1000


I think 1000 is quite on the low end, as the spam waves easily reach them. Than I added following iptables commands to my firewall script.


# build sub chain
$iptables -N SPAMMER
# move all incomming smtp traffic there
$iptables -A INPUT -p tcp --dport 25 -j SPAMMER
# check if the source ip is already in the list, if so give it another 60 sec and drop the packages
$iptables -A SPAMMER -m recent --name spammer --update --seconds 600 -j DROP


As you can see we drop the packets for 10 minutes. If a packet is send within that 10min the time period starts again.

You can test your setup by doing following.


echo +1.1.1.1 >/proc/net/ipt_recent/spammer


and take a look at


cat /proc/net/ipt_recent/spammer


Replace 1.1.1.1 by an IP address of a spammer (just look in your logfile 😉 ) and see it working. Ok, now that we have the kernel/iptables part we need a script which adds the IP addresses of spammers on a DNS RBL after the first 5xx to the ipt_recent list.

As this is only for testing at this point I wrote a small script which watches the mail.log file and looks for 511 errors (the courier error code for DNS RBL hits) and add the IP addresses of the sending servers/zombies to ipt_recent.

Here is the python script: watchForSpammers.py. I don’t go into any details it is quite easy anyway. Start it like this within screen (apt-get install screen) to keep it alive even after logout.


./watchForSpammers.py /var/log/mail.log


The script does its work and adds IP addresses to the ipt_recent list which blocks them also nicely. Just type following to verify it.


iptables -L -xvn


You will see a line like this.


Chain SPAMMER (1 references)
pkts bytes target prot opt in out source destination
15213 724935 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 recent: UPDATE seconds: 600 name: spammer side: source


But the big question is, does it help against the spam waves and its length and heights? More No than Yes ;-). It does almost nothing against the height, but it allows courier to go back a little faster to normal. But still my mail server is maxed out.

So I continue to search for another way. Any ideas?

The Search for reduced SPAM load – Part 1

July 27, 2009

This is the first part of a series of articles which show the various ways I try to reduce the memory and CPU usage on one of my mail server due spammer. Why a series? The first and obvious ideas didn’t work …… but lets start from the beginning.

One of the mail server I manage is for a small software company which sells products mainly to Windows power users. As the email addresses are in every product/readme and on the homepage the server get a lot of spam. I use courier as MTA and IMAP backend for it. To fight the spam I’m using DNS RBL as first line of defense, followed by greylisting, spam assassin and some other stuff.

The ratio of spam which gets through is quite low but thats not the problem at this point. The problem are the spam waves that hit the server several times a day. Normally the system has under 30 open SMTP connections to/from others server, but within minutes that value raises drastically. For example look these values which show the amount of open SMTP connections at the given moment.


Mon Jul 27 19:12:17 CEST 2009     4
Mon Jul 27 19:12:22 CEST 2009     5
Mon Jul 27 19:12:27 CEST 2009     5
Mon Jul 27 19:12:32 CEST 2009     7
Mon Jul 27 19:12:37 CEST 2009     13
Mon Jul 27 19:12:42 CEST 2009     9
Mon Jul 27 19:12:47 CEST 2009     8
Mon Jul 27 19:12:52 CEST 2009     50
Mon Jul 27 19:12:57 CEST 2009     88
Mon Jul 27 19:13:02 CEST 2009     101
Mon Jul 27 19:13:07 CEST 2009     120
Mon Jul 27 19:13:12 CEST 2009     129
Mon Jul 27 19:13:17 CEST 2009     135
Mon Jul 27 19:13:23 CEST 2009     145
Mon Jul 27 19:13:28 CEST 2009     150
Mon Jul 27 19:13:33 CEST 2009     163
Mon Jul 27 19:13:38 CEST 2009     167
Mon Jul 27 19:13:43 CEST 2009     175
Mon Jul 27 19:13:48 CEST 2009     180
Mon Jul 27 19:13:53 CEST 2009     183
Mon Jul 27 19:13:58 CEST 2009     190
Mon Jul 27 19:14:03 CEST 2009     202
Mon Jul 27 19:14:08 CEST 2009     204
Mon Jul 27 19:14:13 CEST 2009     204
Mon Jul 27 19:14:18 CEST 2009     203
Mon Jul 27 19:14:23 CEST 2009     206
Mon Jul 27 19:14:28 CEST 2009     207
Mon Jul 27 19:14:33 CEST 2009     211
Mon Jul 27 19:14:38 CEST 2009     216
Mon Jul 27 19:14:43 CEST 2009     218
Mon Jul 27 19:14:48 CEST 2009     220
Mon Jul 27 19:14:53 CEST 2009     222
Mon Jul 27 19:14:58 CEST 2009     227
Mon Jul 27 19:15:03 CEST 2009     229
Mon Jul 27 19:15:08 CEST 2009     228
Mon Jul 27 19:15:13 CEST 2009     229
Mon Jul 27 19:15:18 CEST 2009     232
Mon Jul 27 19:15:23 CEST 2009     234
Mon Jul 27 19:15:28 CEST 2009     234
Mon Jul 27 19:15:34 CEST 2009     237
Mon Jul 27 19:15:39 CEST 2009     240
Mon Jul 27 19:15:44 CEST 2009     241
Mon Jul 27 19:15:49 CEST 2009     243
Mon Jul 27 19:15:54 CEST 2009     244
Mon Jul 27 19:15:59 CEST 2009     244
Mon Jul 27 19:16:04 CEST 2009     246
Mon Jul 27 19:16:09 CEST 2009     245
Mon Jul 27 19:16:14 CEST 2009     246
Mon Jul 27 19:16:19 CEST 2009     247
Mon Jul 27 19:16:24 CEST 2009     246
Mon Jul 27 19:16:29 CEST 2009     244
Mon Jul 27 19:16:34 CEST 2009     250
Mon Jul 27 19:16:39 CEST 2009     253
Mon Jul 27 19:16:44 CEST 2009     252
Mon Jul 27 19:16:49 CEST 2009     253
Mon Jul 27 19:16:54 CEST 2009     256
Mon Jul 27 19:16:59 CEST 2009     258
Mon Jul 27 19:17:04 CEST 2009     260
Mon Jul 27 19:17:09 CEST 2009     264
Mon Jul 27 19:17:14 CEST 2009     262
Mon Jul 27 19:17:19 CEST 2009     273
Mon Jul 27 19:17:24 CEST 2009     301
Mon Jul 27 19:17:30 CEST 2009     301
Mon Jul 27 19:17:35 CEST 2009     301
Mon Jul 27 19:17:40 CEST 2009     301


It is important to notice these are the values after some tuning on my side already and running a special iptables filter, which I describe in a later articel. They where worse before. The current limit are 300 active connections which lead to following entries by courier.


Jul 27 19:17:23 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:18:24 mail courieresmtpd: 300 maximum active connections.
Jul 27 19:21:14 mail courieresmtpd: 300 maximum active connections.


At this point my monitoring reports that the SMTP service is not reachable as courier does not handle new connections. After some minutes the wave got past my server and the active connections go down again and the monitoring reports the the SMTP service is up again.

You may ask why I don’t just raise the limit of 300 active connections? Because with 300 connections I need already 1GB of RAM only for SMTP stuff. This is because courier forks for every connection a “small” process which handles it. Almost all of the connecting servers are in a DNS RBL so it should be quite easy to just block them. But it is not that easy.

First Idea:
courier is waiting a log time before it closes a connection after sending the 5xx verdict. So I looked at a way to reduce the time, by setting following values in seconds:


root@mail:~# cat /etc/courier/esmtptimeout
30
root@mail:~# cat /etc/courier/esmtptimeoutconnect
30
root@mail:~# cat /etc/courier/esmtptimeoutdata
180
root@mail:~# cat /etc/courier/esmtptimeouthelo
30


Which are quit on the lower end and they helped a little bit, specially to get down to normal business after the top of the spam wave has past. But the wave still maxed my system out.

Second Idea:

I thought at first why does courier wait for the spammer to send a mail before sending a 5xx as the IP is on a DNS RBL? A special process could just drop the connections after doing asynchron DNS RBL check and sending a 5xx to the other server.

Sam Varshavchik the lead programmer of courier told me following:

Except that dropping a connection at this point has proven not to work, as soon as you run into a real mail server, but a broken one that interprets an immediately disconnected connection as an invitation to reconnect immediately.

So ok not good and he said following about my idea to drop the connection ….

And watch your bandwidth get eaten up by a broken server that’s trying to dump on you a load of spam, using multiple connections which immediatly try to reconnect as soon as you drop them.

The most reliable way to reject unwanted mail is by RCPT TO. This is why even when it’s a blacklisted sender, it doesn’t get rejected until it actually sends an RCPT TO. Which means that you have to fully implement SMTP up to that point.

This means that you’ll wind up with a model of using a monolithic, huge process for all SMTP connections, or a single lightweight process per connection.

So this is the starting point of my search for a solution to fix my problem with the exception of replacing courier as MTA at this point (if there is nothing really better out there) and specially no hardware upgrades. I hope some readers have some ideas (please write comments!). I had already one which invokes iptables, but it is a more complex setup which I’ll write in the next article of this series.