Plausibility checks

April 28, 2008

Before I get to the actual topic of this post I want to write some sentences about some user reactions to my last post. I was asked why I called the hacker a professional one, as thats what he does is not complicated or cutting edge. That’s true, but he makes a living with it and that defines professional for me in this case.

So now to the plausibility checks I already talked about. Soon after I posted the link for the blog post to the hacker in the query I got an access to the page from the IP address 75.125.44.xxx, which I believe is the VPN gateway he talked about. This server seems to be a hacked dedicated at US ISP The Planet. This IP address accessed my blog the first time in the last weeks so thats a dead end. But I found something else, the browser agent variable “Mozilla/5.0 (Windows; U; Windows NT 5.1; nb-NO; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14”. OK thats the newest Firefox version but the other stuff doesn’t sound that common (e.g. the language). So I did a check in my logfiles and found following:

Users with the same User agent did access following files at the provided times:

• 84.215.58.xxx [22. Apr. 2008 21:55:54 +0200] “/wp-content/uploads/2008/03/perl_bot.pl"
• 84.215.58.xxx [26. Apr. 2008 18:56:07 +0200] “/44/a-tale-of-searching-for-a-hacker-and-his-supporter-the-idiot-programmer/”
• 195.219.156.xxx [23. Apr. 2008 17:36:36 +0200] “/wp-content/uploads/2008/03/perl_bot.pl”

Both IP addresses did not access anything else in the last 2 weeks, thats really unusual. The first of the above addresses is an IP address from the Norwegian ISP Get AS. Hmm … could that be the real IP the hacker uses? The first access was before the DDOS and the second time it was under 30min after I talked with the hacker. Thats too much for coincidence.

But lets take a look at the other IP address. The reverse lookup of that IP points to a mail server, which looks like a hacked server as the 75.125.44.xxx address did. So the second server with Norwegian browser installed even if it is not located in that country. So even more coincidences, specially as the access happened minutes before the DDOS attack started.

But there is still more. This time the browser provided even attentional information. The reference entry: http://www.google.no/search?hl=no&q=%40SqlTable%2BxeQt&btnG=S%C3%B8k&meta=""
And what do we see? “@SqlTable+xeQt” is the search term … would that not be the name of the hacker?

Now I can say it – I don’t believe him. He lied about no knowing my blog and as he accesses it minutes before the attack happened. I believe he attacked my blog! But why? only for posting the source of his malware?

PS: If you ask yourself why I didn’t post the full IP addresses. Thats because it is only a guess (a good one, but still a guess) and it could be someone indecent behind a given address. And even if it’s the hacker and he said the truth about his internet connection, that IP would lead only to a neighbor of himself.

Interview with a professional hacker

April 26, 2008

After the DDOS attack against my blog this week , I decided to go to the channel I wrote in my initial hacker post about, as I believed that the most likely attacker is hacker I wrote about. After I joined the channel, the hacker opened a query to identify me as he thought I’m a bot. I wrote him that I’ve some questions and that I want to talk to him. He agreed to it and this post contains the important parts of the discussion and some thoughts off mine. He calls himself xeQt.

The first question I asked was if he did the DDOS attack against my blog. He said that he doesn’t do DDOS attacks and that my blog is no challenge for him. He told me that he has other methods to get even. After some discussion he said me that he don’t even know my blog, you will see in a later post that this is most likely not entire true. Within that discussion I also posted a link to my initial post, he said that he won’t click onto it but later said that he has an VPN for this anyway. As I will write in a second post he clicked onto it.

He than was interested if he got one of my servers, which I could decline as it was a server of a friend. This discussion leaded than to the point that he said that I should get used to DDOS attacks as he gets them daily, as he writes bad about other hacker groups which than attack him.

I asked him than if I could use this discussion in my blod and he said yes, as he has nothing to hide and that only a miracle will get him busted. I asked him if he has nothing to loose. He told me that he has no life so it doesn’t matter anyway and that he does not have a own internet connection, and therefore he beliefs he is safe. I guess he is using a open WLAN of one of his neighbors.

My next question was what he gains from the hacking that servers. He answered following: “I sell them to scammers, spammers”, this leaded to the question how long a hacked server stays online normally. He told me that this can vary from one day to one year, and that it depends what is done with the server. Which I can tell is quite true, as most of the time I get only called if the machine has a unusual high CPU usage, generates too much traffic or a mail server administrator detects spam mails from one of the servers in his network.

He than said that most server administrator don’t have much knowledge about Linux and that they don’t secure the systems and that he secures the servers for them and sells them to spammers or people who need root or botnet clients. With securing he means that he closes the attack vector that he used to gain access to the system so no other hacker can take that machine from him. To get a better picture of the size of his operations. He said to me that the hacks 500 servers daily. This means that he does not look for special target but for lowest hanging fruits for which he can gain automatic or semi automatic access to make a living.

We had also some other points (more technical) but these where the most interesting parts for my readers. I want to say thanks to xeQt for talking with me and allowing me to write about our discussion. I will write a second post with some plausibility checks as already written above, so stay tuned.

UDP Flood DDOS attack against my blog

April 24, 2008

Starting 18:00 CET (23.04.2008) someone started with a distributed denial of service attack against my blog. The UDP Flood attack was carried out, as showed my investigation by hacked servers and not zombie windows clients. At the time of writing the attack is still underway but got weaker after the first 24h.

The traffic accounting reports so far >750gb incoming traffic, but in reality it will be even higher as not every packet was counted in the beginning of the attack as it consumed large amounts of network resources. The data center my server is located at removed the route for the sub network from the border gateways, so the operation of the whole data cents was not affected. After I guess some network admins detected that some of their machines got misused for a DDOS and did shut them down, the traffic went down. After this happened the subnetwork has been reactivated again, and the blog is online again.

But why should someone attack my little blog in the first place? I didn’t post in the last 14 days. The only idea I’ve is that the hacker I found at the server of a friend and wrote about it wanted to get even. What counts for this theory is that it is carried out by hacked servers from and to random UDP ports – a feature the found bot also has.

I’ll investigate further and report in my blog about it.

Update: Following IP are still attacking me after >30h … it seems to be time to try to contact the admins.


202.147.170.5 (Pakistan) - informed - not active anymore after 48h
222.122.46.92 (Korea) - informed - not active anymore after 48h
72.36.175.98 (USA) - informed - reacted within 12h
85.214.100.202 (Germany) - informed - reacted within 12h
62.112.193.46 (Hungary) - informed - still active after 3 days
195.219.156.98 (Spain) - informed - reacted within 24h
211.174.182.13 (Korea) - informed - still active after 3 days


Update2: 3 days after the start of the attack it still continues. ok only with lonely 2 systems, whose admins don’t seem to care about the attack and my mail. whats the reason for this? did the hacker lose control over them? what does he gain with it – the side is online without any problems for the users. Has anyone an idea?

LED blinking on your switch

April 9, 2008

Did you ever have the problem that you didn’t know to which switch port a given ethernet port /cable is connected to? Wouldn’t it be cool if the LED of the switch port would blink so you know which one is the correct one?

You’re lucky – it is possible with Linux. There are even two ways. With some chipsets ethtool -p eth0 works but not with all. But following script also helps in any case:

#/bin/bash
# usage example: blink.sh eth0


while true ; do
  ifconfig $1 down
  sleep 2
  ifconfig $1 up
  sleep 2
done


Put that script into /usr/local/sbin/blink.sh and set the execution permissions. Call it with the device as parameter. Don’t set the blinking below 2sec as it is possible that the connection negation takes up to that amount of time.

Howto setup Asterisk for recording a podcast over the Internet

April 6, 2008

Some friends and I are planing a to make a podcast and as we IT guys we needed something to support our distributed recordings over the Internet. One of my friend lives about 200km away but also with the others it would be not that easy to get all into one room at the same time. When we looked around for various ways to do distributed recordings we found mostly Skype howto’s and we knew only a few podcasts which did the recording with Asterisk (in a not that good audio quality I think). But we didn’t find what we really wanted and so I started to look into that topic. At first I wrote our requirements down:

• the recording should be done centrally on a server automatically without user interaction (so it is not forgotten and to minimize the lag and sound quality problems)
• the recorded files should be easy available as OGG files to all podcast members after the recording, as the post production is maybe not done always by the same people
• various client operation systems and VoIP clients should be supported, therefore an open standard protocol should be used
• possibility to record each participant separately to allow changes in the volume or exclusively applying audio filters
• it should be possible to invite guest for interviews via the same system without requiring more than a VoIP client which support the choosen protocol. (no registration somewhere)
• optional it should be possible to connect the system to the POTS (plain old telephone service) for interviews with people which cannot use a VoIP client.
• And as a requirement by my knowledge and existing infrastructure, the server should run on one of my Linux servers I’ve running in a big data center, within a OpenVZ virtual environment.

After some research I decided to go with Asterisk. This howto describes what I’ve done to reach the above goals. After the completion of this howto you should have following:

• A SIP server where you participants can connect to and talk with each other.
• As soon as they go into a special virtual conference room everything they say will be recorded
• After they leave the conference room a background process will reencode the recorded WAV file to OGG and make it available via web.
• Every participant gets its own OGG file with the starting timestamp in the filename, you need to use the recording at the correct place in the post production.
• The inclusion of a SIP provider with connection to the POTS network is not described in this howto as there are may others describing it.

Some points of this howto are specific to my OpenVZ setup and the chosen distribution but most are generic and should work for any setup. Anyway here is the software I used.

• OpenVZ for virtualization
• Ubuntu 8.04 Hardy (x86_64) as distribution for the virtual environment
• Asterisk and Zaptel for VoIP part (Ubuntu packages)
• sox for translating the WAV files to OGG (Ubuntu package)
• lighttpd as small and fast web server for the OGG files (Ubuntu package)
• Twinkle as SIP client under Linux. (apt-get install twinkle)

Part 1: Hardware node setup

You can ignore that part if you don’t use OpenVZ and your kernel/distribution comes with ztdummy modules. The hardware node in my case runs on a Centos4 (x86_64). This is important as Asterisk needs the ztdummy kernel module, which comes with zaptel, for the meetme Asterisk module which is used for the conference rooms. As it is not possible to load kernel modules within a VE (virtual environment) (that’s a security feature!) I needed it on my hardware node. As the kernel of the hardware is a OpenVZ patched kernel and also Centos 4 does not come with a ztdummy module anyway, I needed to compile it.

I used the same version of zaptel as Ubuntu 8.04 does and it is also very important that you use a 64bit VE if you hardware node is 64bit, otherwise the device cannot be accessed correctly.

The install is quite straightforward

# tar xzf zaptel-1.4.8.tar.gz
# cd zaptel-1.4.8
# ./install_prereq test


Install the required packages and then continue with:

# ./configure
# make
# make install


but no “make config”, as we don’t need init scripts or that stuff. Now load the kernel module with modprobe ztdummy (and make sure that this is done after very boot, before the VEs start). Make sure the device is working with:

# ztcfg -v


The output should be something like:

Zaptel Version: 1.4.8
Echo Canceller: MG2
Configuration
======================
0 channels to configure.


At last we need the VE be able to access the ztdummy device, so we need to tell openVZ this.

# for x in `ls /dev/zap`; do /usr/sbin/vzctl set XXX --devnodes zap/${x}:rw --save; done


Replace the XXX with the ID of your podcast VE. Now we’re done with the hardware node and we can take a look at the user space stuff.

Part 2: Virtual Environment setup

At first we install the packages we need with following command:

# apt-get install asterisk-h323 asterisk-doc speex vpb-utils sox libsox-fmt-all lighttpd zaptel


Now we configure zaptel and check if it works:

# genzaptelconf
# ztcfg -v
# ztcfg -d


No error should be given. If a device is not found check if they got created by vzctl. After that make the devices in /dev/zap read and writable for the asterisk user:

# chown root:asterisk /dev/zap/*
# chmod 660 /dev/zap/*

Now we can work on the Asterisk configuration. We set following values in /etc/default/asterisk:

RUNASTERISK=yes
AST_REALTIME=no


The real time stuff does not work in a VE and gives audio problems. Now we need to do some configuration for NAT users in /etc/asterisk/sip.conf:

externip = you're_external_IP ; this is needed as asterisk has problem with the venet0 stuff otherwise
localnet=192.168.0.0/255.255.0.0; All RFC 1918 addresses are local networks
localnet=10.0.0.0/255.0.0.0 ; Also RFC1918
localnet=172.16.0.0/12 ; Another RFC1918 with CIDR notation
localnet=169.254.0.0/255.255.0.0 ;Zero conf local network


nat=yes
qualify=yes
canreinvite=no


After this global setup we configure for each of our podcasters one section, as shown here:

[firstPodcaster] ; this is also the user name
type=friend
context=sip
secret=the_password_of_this_user
callerid="Your_Name" <1> ; it is recommended to use no spaces here, as we use this as part of the filename. You need the “, “ and < ,> exactly as show here
host=dynamic
dtmfmode=info
disallow=all
allow=alaw
callingpres=allowed_passed_screen


We only support alaw so every client uses G.711a and we don’t need to translate. I believe in the US you need to use G.711u and therefore ulaw. Now we need a conference room for which we add following line to /etc/asterisk/meetme.conf:

conf => 10
conf => 20


Now we need to tie that together with the dial plan in /etc/asterisk/extensions.conf:

[globals]
MONITOR_EXEC=/usr/local/bin/wavIn2ogg.sh


And add following section:

[sip]


; so we can talk directly with another
exten => 31,1,Dial(SIP/firstPodcaster,20,tr)
exten => 32,1,Dial(SIP/secondPodcaster,20,tr)
exten => 33,1,Dial(SIP/thirdPodcaster,20,tr)


; this conference room is not recorded, for preparations
exten => 10,1,Answer
exten => 10,2,Wait(1)
exten => 10,3,Meetme(10,s)


; this conference room records automatically
exten => 20,1,Answer
exten => 20,2,Wait(1)
exten => 20,3,Set(CALLFILENAME=podcast_X_${CALLERID(name)}-${STRFTIME(${EPOCH},,%Y%m%d-%H%M%S)})
exten => 20,4,Monitor(wav,${CALLFILENAME},m)
exten => 20,5,Meetme(20,s)


; from the demo code – useful to look at the quality of your connection
; Create an extension, 60, for evaluating echo latency.
exten => 60,1,Playback(demo-echotest) ; Let them know what's going on
exten => 60,n,Echo ; Do the echo test
exten => 60,n,Playback(demo-echodone) ; Let them know it's over
exten => 60,n,Goto(s,60) ; Start over


Now you can restart Asterisk and connect with you SIP client. Call 60 to check if the audio stream works in both directions (try it without firewall on the Asterisk server if don’t hear anything). After that go into the conference room 10 with 1 or 2 friends and test it. If that all works you can work on the recording stuff. Create following script /usr/local/bin/wavIn2ogg.sh (don’t forget chmod 755):

#!/bin/bash
# wavIn2ogg.sh - creates ogg of the input mono stream
# used for recording each participant in a meeting room separately
# Written by Robert Penz


SOX=/usr/bin/sox
NICE=/usr/bin/nice


InFile="$1"
OutFile="$2"
OggFile=`echo $3|sed s/.wav//`.ogg
FinalDir=/var/www/production


#test if input files exist
test ! -r $InFile && exit
test ! -r $OutFile && exit


$NICE -n 19 $SOX -t wav "$InFile" -t vorbis "$OggFile"


#remove input files if successfull
test -r "$OggFile" && rm "$InFile" "$OutFile"


# at last set the permissions and move it in an atomic way
chmod 644 "$OggFile"
mv "$OggFile" "$FinalDir/."


Create the directory /var/www/production and set the correct permissions:

# mkdir /var/www/production
# chown asterisk:www-data /var/www/production


Now go the the conference room 20 and say something and disconnect. If it worked you should see with your browser under http:///production/ the recorded OGG file(s). If there are none, take a look at /var/spool/asterisk/monitor/ if there are 2 WAV files. If so call the wavIn2ogg.sh script by hand and look for any errors.

So thats the end of the story – you’ve now a system for recording podcasts over the internet in a cool way! Any comments, ideas or questions? Post them here.