The fallout of the Debian OpenSSL security problem

May 24, 2008

Today I don’t want to write about how the Debian security problem occurred, if it was the fault of the Debian maintainer or the OpenSSL guys. We’re all human so errors can occur, but this shows a much bigger problem we have. Our SSL infrastructure!!

I’m quit sure you read about the weak SSL whitehouse.gov had and that the white house does not handle their SSL stuff by them self, instead the use Akamai for this. If you don’t know Akamai, it is a major content distribution network. They have tens of thousands servers distributed all over the world and the content is served by the closest server to provide higher download speeds for the customs and make a DDOS attack much harder if not impossible. Their customer list includes Microsoft, the New York Times and so on.

So basically the SSL keys of the Akamai servers where weak, and it was possible to get their public keys (it is sent as part of the SSL handshake), and calculate the private ones out of it (there are only 32k possible keys). I know that at least one did it and sent the keys to the CCC, which verify the authenticity of the keys. Sure Akamai replaced their keys immediately. BUT. There is no way to revoke the SSL keys!!

What most people don’t seem to understand so far is that these keys are signed by a CA which is in any browser. This means that a man in the middle attack can easily performed with this. As ATI is also a customer of Akamai, someone could send you a Trojan horse instead of the newest ATI you wanted. SSL has in theory two defenses against this:

• Keys expire, the Akamai key in October 2008
• Originally SSL had the idea that CAs publish a list of compromised keys (revoke list) and as part of the SSL handshake the browser should check if a key is on the list. The problem with it was that this does not scale and is a privacy problem too. Browsers don’t implement this or have not activated it by default.

So we’re out in the open for this key until this fall, but that won’t be the end, as e.g. godaddy at least allows to the sign keys for a longer period of time. e.g. 3 years. And the same problem occurs if a private key is leak by other means. The whole foundation of our web security infraction is build on sand. We need something new!!

PS: I want to stress that Akamai did nothing wrong here. They did everything right and still have a problem!

Improving the security of an installed WordPress

May 18, 2008

This weekend was a busy one, due the openssl security problem, I needed to regenerate ssh keys and openvpn certs. After this was I done I thought it is time to improve also the security of my blog.

I looked a little bit around and found the wp-security-scan wordpress plugin, which does a basic scanning of the security of the installation. It found some stuff I had not changed from the default install. For example, I tried to use the plugin to change the tables prefix, but that didn’t work for me, it complained always about missing alter privileges of the db user which was not correct. I then clicked on the link which explained how to do it by hand, but that broke my system a little bit. But I found following blog entry which show how to do it correctly, even how to get the cryptographp plugin working again afterwards.

I know not every blogger has the technical knowledge to secure their blog, but than they should look maybe for a hosted version. For all others with their self hosted blog: Take really a look at the plugin and correct the stuff it reports.

iptables dynamic port script for NFS

May 10, 2008

Some days ago I talked with a friend (here a link to his homepage) about firewalls and file servers and he told me he has a iptables script which adapts to the NFS ports automatically. I asked him for this script and here is it. Thx Hannes for the script.


# rpcinfo -p prints a list of all registered RPC programs
# sed -e '1D' removes the headline
# tr -s ' ' '\t' replaces repeated spaces with a single tab
# cut -f 4,5 we only need the protocol- and port-columns
# sort | uniq removes the duplicate lines
# now we have lines with the needed protocol and port but for splits
# this lines to single words so we have to store the protocol
for l in `rpcinfo -p | sed -e '1D' | tr -s ' ' '\t' | cut -f 4,5 | sort | uniq`
do
  case $l in
    tcp)
      SYN=--syn
      PROTOCOL=$l
        ;;
    udp)
      SYN=
      PROTOCOL=$l
        ;;
    *)
      iptables -A INPUT -p $PROTOCOL --dport $l $SYN -j ACCEPT
    ;;
  esac
done

Online VM builder for VMware Player

Every used VMware Player to “play” precreated VMs? I did, but I thought when I need to create VMs by my own I need VMware Workstation or Virtual PC if was running Windows and not Linux. Ok and there is now VirtualBox, but I never used it before, but as it comes now with Ubuntu 8.04 its changes are raising (no need to compile anything like kernel modules for every security update of the kernel). Anyway I found some thing that allows you to use Vmware Player with your own VMs. Following Websites allow you to create images for VMware Player:

EasyVMX!
VM Builder
vmx-builder.cmd

I think the first is the best one. Maybe this info helps also others, as most of the time VMware Player is enough and someone does not need the Workstation version, and Virtual PC 2007 is bad product. Ever tried to run a current Linux kernel on it. It crashes the kernel. I learned that the hard way with Ubuntu 8.04 server within a Virtual PC 2007. Which was not easy to install in the first place, but booting the current kernel after the installation was the end point of the journey, no problem with VMware Player however.

Kubuntu 8.04 hardy addition packages install script

This script is for my friends, who most know the previous versions already. It installs additional packages for kubuntu 8.04 hardy. I use it for the initial setup of a desktop system. First install Kubuntu from CD and than use this script to get the system which, has all codecs and commonly used programs (be it free or non free software) installed. So this blog entry is for my own reference and for my friends. Basically after running this script you’ll have a system which is ready for usage by a standard user.

Insecurity of Virtual Appliances and some thoughts on 7-zip compression

May 3, 2008

This week I looked for a Ubuntu server 8.04 LTS virtual appliance for Vmware – I found one here. But before I could start testing it I needed to extract the .7z file on my VMware server. The first thing I though was, why the hack 7-zip? Why not use bzip2, which is standard on Linux (beside the faster, but less compressing gzip)?

But I was shown wrong by the first entries at my google search – 7-zip has most of the time the better compression and is not much slower than bzip2. And there is even an open source command line tool on Linux, it is called p7zip. The only thing which prevents me from using it, is that it is not supported by tar so far, as soon that happens I will start using it.

But now to something security related. Almost every virtual appliance I download has openssh as sshd daemon installed. Am I the only guy who things this is a bad idea? The host keys are the same for all virtual appliances. So anyone who knows which virtual appliances I used to setup my server, can use this knowledge to perform a man in the middle attack and get my login name and password. This bad habit seems to occur by almost all virtual appliances I got my hands on. My solutions so far is following on Ubuntu and Debian Systems:


apt-get --purge remove openssh-server && apt-get install ssh


This way I’ve a clean config and new keys. (ssh is a meta package for openssh-client and openssh-server). So there is a easy work around but how many administrators will think about that? I think virtual appliances are made to ease the life of the administrators or to allow even non expert to provide a service based on the appliance. With this goal comes also the responsibility to make the system save by default.

Plausibility checks

April 28, 2008

Before I get to the actual topic of this post I want to write some sentences about some user reactions to my last post. I was asked why I called the hacker a professional one, as thats what he does is not complicated or cutting edge. That’s true, but he makes a living with it and that defines professional for me in this case.

So now to the plausibility checks I already talked about. Soon after I posted the link for the blog post to the hacker in the query I got an access to the page from the IP address 75.125.44.xxx, which I believe is the VPN gateway he talked about. This server seems to be a hacked dedicated at US ISP The Planet. This IP address accessed my blog the first time in the last weeks so thats a dead end. But I found something else, the browser agent variable “Mozilla/5.0 (Windows; U; Windows NT 5.1; nb-NO; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14”. OK thats the newest Firefox version but the other stuff doesn’t sound that common (e.g. the language). So I did a check in my logfiles and found following:

Users with the same User agent did access following files at the provided times:

• 84.215.58.xxx [22. Apr. 2008 21:55:54 +0200] “/wp-content/uploads/2008/03/perl_bot.pl"
• 84.215.58.xxx [26. Apr. 2008 18:56:07 +0200] “/44/a-tale-of-searching-for-a-hacker-and-his-supporter-the-idiot-programmer/”
• 195.219.156.xxx [23. Apr. 2008 17:36:36 +0200] “/wp-content/uploads/2008/03/perl_bot.pl”

Both IP addresses did not access anything else in the last 2 weeks, thats really unusual. The first of the above addresses is an IP address from the Norwegian ISP Get AS. Hmm … could that be the real IP the hacker uses? The first access was before the DDOS and the second time it was under 30min after I talked with the hacker. Thats too much for coincidence.

But lets take a look at the other IP address. The reverse lookup of that IP points to a mail server, which looks like a hacked server as the 75.125.44.xxx address did. So the second server with Norwegian browser installed even if it is not located in that country. So even more coincidences, specially as the access happened minutes before the DDOS attack started.

But there is still more. This time the browser provided even attentional information. The reference entry: http://www.google.no/search?hl=no&q=%40SqlTable%2BxeQt&btnG=S%C3%B8k&meta=""
And what do we see? “@SqlTable+xeQt” is the search term … would that not be the name of the hacker?

Now I can say it – I don’t believe him. He lied about no knowing my blog and as he accesses it minutes before the attack happened. I believe he attacked my blog! But why? only for posting the source of his malware?

PS: If you ask yourself why I didn’t post the full IP addresses. Thats because it is only a guess (a good one, but still a guess) and it could be someone indecent behind a given address. And even if it’s the hacker and he said the truth about his internet connection, that IP would lead only to a neighbor of himself.

Interview with a professional hacker

April 26, 2008

After the DDOS attack against my blog this week , I decided to go to the channel I wrote in my initial hacker post about, as I believed that the most likely attacker is hacker I wrote about. After I joined the channel, the hacker opened a query to identify me as he thought I’m a bot. I wrote him that I’ve some questions and that I want to talk to him. He agreed to it and this post contains the important parts of the discussion and some thoughts off mine. He calls himself xeQt.

The first question I asked was if he did the DDOS attack against my blog. He said that he doesn’t do DDOS attacks and that my blog is no challenge for him. He told me that he has other methods to get even. After some discussion he said me that he don’t even know my blog, you will see in a later post that this is most likely not entire true. Within that discussion I also posted a link to my initial post, he said that he won’t click onto it but later said that he has an VPN for this anyway. As I will write in a second post he clicked onto it.

He than was interested if he got one of my servers, which I could decline as it was a server of a friend. This discussion leaded than to the point that he said that I should get used to DDOS attacks as he gets them daily, as he writes bad about other hacker groups which than attack him.

I asked him than if I could use this discussion in my blod and he said yes, as he has nothing to hide and that only a miracle will get him busted. I asked him if he has nothing to loose. He told me that he has no life so it doesn’t matter anyway and that he does not have a own internet connection, and therefore he beliefs he is safe. I guess he is using a open WLAN of one of his neighbors.

My next question was what he gains from the hacking that servers. He answered following: “I sell them to scammers, spammers”, this leaded to the question how long a hacked server stays online normally. He told me that this can vary from one day to one year, and that it depends what is done with the server. Which I can tell is quite true, as most of the time I get only called if the machine has a unusual high CPU usage, generates too much traffic or a mail server administrator detects spam mails from one of the servers in his network.

He than said that most server administrator don’t have much knowledge about Linux and that they don’t secure the systems and that he secures the servers for them and sells them to spammers or people who need root or botnet clients. With securing he means that he closes the attack vector that he used to gain access to the system so no other hacker can take that machine from him. To get a better picture of the size of his operations. He said to me that the hacks 500 servers daily. This means that he does not look for special target but for lowest hanging fruits for which he can gain automatic or semi automatic access to make a living.

We had also some other points (more technical) but these where the most interesting parts for my readers. I want to say thanks to xeQt for talking with me and allowing me to write about our discussion. I will write a second post with some plausibility checks as already written above, so stay tuned.

UDP Flood DDOS attack against my blog

April 24, 2008

Starting 18:00 CET (23.04.2008) someone started with a distributed denial of service attack against my blog. The UDP Flood attack was carried out, as showed my investigation by hacked servers and not zombie windows clients. At the time of writing the attack is still underway but got weaker after the first 24h.

The traffic accounting reports so far >750gb incoming traffic, but in reality it will be even higher as not every packet was counted in the beginning of the attack as it consumed large amounts of network resources. The data center my server is located at removed the route for the sub network from the border gateways, so the operation of the whole data cents was not affected. After I guess some network admins detected that some of their machines got misused for a DDOS and did shut them down, the traffic went down. After this happened the subnetwork has been reactivated again, and the blog is online again.

But why should someone attack my little blog in the first place? I didn’t post in the last 14 days. The only idea I’ve is that the hacker I found at the server of a friend and wrote about it wanted to get even. What counts for this theory is that it is carried out by hacked servers from and to random UDP ports – a feature the found bot also has.

I’ll investigate further and report in my blog about it.

Update: Following IP are still attacking me after >30h … it seems to be time to try to contact the admins.


202.147.170.5 (Pakistan) - informed - not active anymore after 48h
222.122.46.92 (Korea) - informed - not active anymore after 48h
72.36.175.98 (USA) - informed - reacted within 12h
85.214.100.202 (Germany) - informed - reacted within 12h
62.112.193.46 (Hungary) - informed - still active after 3 days
195.219.156.98 (Spain) - informed - reacted within 24h
211.174.182.13 (Korea) - informed - still active after 3 days


Update2: 3 days after the start of the attack it still continues. ok only with lonely 2 systems, whose admins don’t seem to care about the attack and my mail. whats the reason for this? did the hacker lose control over them? what does he gain with it – the side is online without any problems for the users. Has anyone an idea?

LED blinking on your switch

April 9, 2008

Did you ever have the problem that you didn’t know to which switch port a given ethernet port /cable is connected to? Wouldn’t it be cool if the LED of the switch port would blink so you know which one is the correct one?

You’re lucky – it is possible with Linux. There are even two ways. With some chipsets ethtool -p eth0 works but not with all. But following script also helps in any case:

#/bin/bash
# usage example: blink.sh eth0


while true ; do
  ifconfig $1 down
  sleep 2
  ifconfig $1 up
  sleep 2
done


Put that script into /usr/local/sbin/blink.sh and set the execution permissions. Call it with the device as parameter. Don’t set the blinking below 2sec as it is possible that the connection negation takes up to that amount of time.