Filter the output of command line programs e.g. run by cron

September 14, 2008

Ever had to run a program by cron which writes always stuff to stdout and you therefore get every time a mail from cron? You did an > /dev/null to get rid of the messages but errors are not written to stderr by this program but also to stdout?

I have sometimes that problem and therefore I’ve written a small python script which is run by cron and launches itself the real program. It takes any output of that program and filters it by configured regular expression rules. Everything that matches a rules is not reported. As soon as one line does not match, this line is reported and followed by the full output of the program to make the error finding easier.

I often let programs run in verbose mode, as it is filtered anyway by the python script and if an error occurs it is nice to have more information at hand. The python script also forwards any provided parameter to the real program.

There is the python script filterOutput.py. Just download it, set the execute permissions and open it in an file editor. following 3 variables are interesting for you:

• programPath: the path to which the script should change before starting the program
• programCommand: the full path of the program to launch and filter
• regexList: the list of regular expressions rules which are used to filter the output of the called programm

Hidden Firefox feature and the plugin Pencil

September 3, 2008

I found something interesting, which I guess some already know, but still I though its a nice feature. Copy following into your Firefox browser address bar and hit enter.

javascript:document.designMode='on'; void 0


You can now change a homepage by deleting stuff or replacing some text to test something or making a faked screenshot. I think this can be really handy sometimes and it is build in. Something more sophisticated is following Firefox 3.x plugin, called Pencil (You can also download it here). Pencil is a Firefox add-on to do GUI prototyping and simple sketching, take a look at this blog post which shows what Pencil can do. As a teaser I put a screenshot at the end to this post 😉

Howto install rtorrent and wtorrent within an Ubuntu Hardy VE

August 13, 2008

I didn’t look at rtorrent and wtorrent at first. The first software I found for the web based Bittorrent client I searched was TorrentFlux (to be exact Torrentflux-b4rt). The installing was not complicated, but the software is really bad. For once the php part starts for every torrent a new Bittorrent command line program process which takes really really much memory if you’ve like 10 torrents.Their is also the problem that such processes did tend to hang and it was not possible to solve all problems via the web GUI. The next big part was that the web GUI put a really high CPU load on my browsers if the AJAX update was activated (even if it was set to 30 sec). Basically it was an non integrated set of peaces which made more work than it helped me to save, and as I could not set the overall bandwidth I started to search for something else.

It was not easy to find rtorrrent and wtorrent, don’t know why I didn’t find it when I searched for “web based bittorrent client”. I hope my howto helps somewhat in this regard. The combination of these two applications work for me now without any problems. Low CPU load on the client and Server, small footprint on the server and stable. And it works with torrents which didn’t work with torrentflux.

This Howto describes how to setup rtorrent with the web GUI wtorrent (on lighttpd) on an Ubuntu Hardy and in my case within an OpenVZ VE. Of course this Howto works also if you don’t use visualization and it should also work with the current Debian release. Whats special of my setup is that I want to run the rtorrent and wtorrent within a VE, but the finished data should be stored on an remote NFS server (e.g. a small NAS).

1. OpenVZ VE Setup

First we create the VE from a template, you can download one from OpenVZ:

vzctl create XXX --ostemplate ubuntu-8.04-i386-minimal --hostname torrent

After this we add an IP address and activate the NFS client support within the VE:


vzctl set XXX --ipadd XXX.XXX.XXX.XXX --save
vzctl set XXX --features "nfs:on" --save


After this I recommend to change the limits of your VE, I use following (configured via /etc/openvz/conf/XXX.conf)

KMEMSIZE="11055923:11377049"
LOCKEDPAGES="256:256"
PRIVVMPAGES="525000:572715"
SHMPAGES="21504:21504"
NUMPROC="240:240"
PHYSPAGES="0:2147483647"
VMGUARPAGES="33792:2147483647"
OOMGUARPAGES="26112:2147483647"
NUMTCPSOCK="1440:1440"
NUMFLOCK="1880:2060"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="1720320:2703360"
TCPRCVBUF="1720320:2703360"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="360:360"
DCACHESIZE="3409920:3624960"
NUMFILE="9312:9312"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

Check them on you system that the are not too low. Now you start your VE with


vzctl start XXX
vzctl enter XXX


Take a look at configured DNS servers and that you can reach the Internet with this VE. Put your NFS server into /etc/fstab like this:

yyy.yyy.yyy.yy:/nfsshare /media/nfs nfs soft,udp,auto,user,rsize=32768,wsize=32768 0 0

Don’t forget to create the directory and to apt-get install nfs-common. After this try to mount the share.

2. Configure your firewall

This part of the setup is based on following assumption:

• Your Linux system is used as DSL/Cable Router and
• you’ve only one wordwide IP address and
• your VE has a private IP address.

It is assumed that your VE can connect to the internet, as only the special setup for the bittorrent ports is shown. The following iptables code will redirect the required ports to your VE, where xxx.xxx.xxx.xxx is the IP of your VE.

iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 63963:63981 -j DNAT --to-destination xxx.xxx.xxx.xxx
iptables -A FORWARD -d xxx.xxx.xxx.xxx -p tcp --dport 63963:63981 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 63982 -j DNAT --to-destination xxx.xxx.xxx.xxx
iptables -A FORWARD -d xxx.xxx.xxx.xxx -p tcp --dport 63982 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i ppp0 -p udp --dport 63982 -j DNAT --to-destination xxx.xxx.xxx.xxx
iptables -A FORWARD -d xxx.xxx.xxx.xxx -p udp --dport 63982 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT


3. Install the base and rtorrent

At first the packages we need: (As more packages we can use the less work we’ll have)

apt-get install rtorrent screen mc wget lighttpd gawk php5-cgi php5-common php5-sqlite php5-xmlrpc sqlite unzip php5-curl

Now download following file and save it as /etc/init.d/rtorrent in your VE and activated for automatic startup at boot time with

update-rc.d rtorrent defaults 25

After this we need to add an user under which the rtorrrent process runs and


# adduser --disabled-login rt
# usermod -aG tty rt
# su - rt
$ wget http://robert.penz.name/wp-content/uploads/2008/08/rtorrent.rc -O .rtorrent.rc
$ mkdir /home/rt/session
$ logout


change the /home/rt/.rtorrent.rc file to your needs. You should try out rtrorrent after any config file change, to avoid problems with syntax errors or that stuff.


# su - rt
$ rtorrent


If it started press CTRL-Q to get out.


$ logout


If all worked start it with /etc/init.d/rtorrent start

4. lighttpd setup

This sections shows how to setup lighttpd for rtorrent XML RPC and for wtorrent. Add "mod_scgi" to the server.modules in /etc/lighttpd/lighttpd.conf and add following there too:

url.access-deny = ("~", ".inc", ".db", ".tpl.php", ".cls.php",)

Create following file /etc/lighttpd/conf-available/10-scgi.conf with following content:


scgi.server = (
"/RPC2" => # RT_DIR
( "127.0.0.1" =>
(
"host" => "127.0.0.1", # Ip where rtorrent is listening
"port" => 5000, # Port specified in .rtorrent.rc
"check-local" => "disable"
)
)
)


Enable following two configs by setting a symlink:


# cd /etc/lighttpd/conf-enabled/
# ln -s ../conf-available/10-cgi.conf .
# ln -s ../conf-available/10-scgi.conf .


Restart the lighttpd:

/etc/init.d/lighttpd restart

5. Install the wtorrent

Get the newest version and configure it:

# cd /var/www/
# rm index.lighttpd.html
# wget "http://www.wtorrent-project.org/trac/changeset/latest/trunk/?old_path=%2F&format=zip" -O wtorrent.zip
#unzip wtorrent.zip
#mv trunk/wtorrent/* .
#rm -rf trunk
#mkdir tpl_c/
#cd conf
# cp sample.user.conf.php user.conf.php


Change this file according your settings, specially the lines.

define( 'DIR_TORRENTS', 'torrents/');
define( 'DIR_EXEC', '/var/www/');
define( 'DIR_DOWNLOAD', '/home/rt/doing');

After this make everything belong to www-data and read/writable.

#cd /var/www
#chown -R www-data:www-data *
#chmod -R 755 *


Call now

http://xxx.xxx.xxx.xxx/install2.php

After the install is finished do following:

# mv install.php install.php_deactivated
# mv install2.php install2.php_deactivated

You should be able to login via http://xxx.xxx.xxx.xxx/

Ever did some design and wanted blind / dummy text?

August 4, 2008

I found a very nice and easy homepage which generates that kind of text for you. Take a look at Lorem Ipsum. And here is a text generated by it 😉

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nullam at erat ac sem aliquam tempus. Vivamus nisl odio, euismod quis, tempus quis, porta eu, ipsum. Vivamus est erat, commodo tempus, interdum et, elementum ac, nulla. Vestibulum pede tortor, eleifend at, mattis eget, dignissim eget, lorem. Mauris eu leo. Cras sodales, enim sed faucibus ultricies, tellus tortor blandit nunc, quis lacinia quam est a felis. Ut ac lorem non nunc consectetuer rhoncus. Aenean erat lacus, mollis nec, aliquam et, laoreet quis, neque. Fusce urna lorem, posuere et, molestie eu, facilisis eget, dolor. Integer erat erat, bibendum et, viverra quis, dapibus at, erat. Vestibulum metus. Ut diam erat, sollicitudin sed, ullamcorper at, commodo luctus, eros. Nulla vitae erat. Cras hendrerit commodo mauris.

Aliquam aliquet turpis vitae odio. Etiam luctus orci vitae leo. In et augue eu justo convallis adipiscing. Integer ultricies. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Nam ante lacus, pellentesque quis, tempor eget, malesuada eget, dui. Proin nec libero. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Sed scelerisque facilisis justo. Suspendisse semper turpis et pede. Pellentesque erat libero, sodales ac, iaculis sed, commodo non, velit. Donec commodo pellentesque elit.

Suspendisse sagittis molestie quam. Cras at dui a magna pharetra ornare. Sed interdum felis quis augue. Maecenas turpis. Suspendisse dui. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Sed vulputate mauris id est. In consectetuer neque nec ipsum. Maecenas consectetuer tellus at pede. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Aenean ante risus, cursus vel, porta ultricies, gravida quis, dolor. Mauris blandit velit ut enim. Pellentesque enim. Praesent vehicula. Maecenas pharetra sollicitudin eros.

Kpartx: a tool for mounting partitions within an image file

July 12, 2008

Kpartx can be used to set up device mappings for the partitions of any partitioned block device. It is part of the Linux multipath-tools. With kpartx -l imagefile you get an overview of the partitions in the image file and with kpartx -a imagefile the partitions will accessible via /dev/mapper/loop0pX (X is the number of the partition). You can mount it now with mount /dev/mapper/loop0pX /mnt/ -o loop,ro. After unmounting you can disconnect the mapper devices with kpartx -d imagefile.

There are packages for Debian and Ubuntu.

• Ubuntu: http://packages.ubuntu.com/de/hardy/kpartx
• Debian: http://packages.debian.org/search?keywords=kpartx

Fix for Kopete 0.12.7 to work again with ICQ

July 2, 2008

Yesterday my Kopete stopped working with the ICQ network. The ICQ told me that my client version is too old. Here is the fix to make it work again. Look in the ~/.kde/share/config/kopeterc file and change the values of the variables to following (which are from trunk):

[ICQVersion]
Build=0x17AB
ClientId=0x010A
ClientString=ICQ Client
Country=us
Lang=en
Major=0x0006
Minor=0x0000
Other=0x00007535
Point=0x0000


After a restart of Kopete everthing works again for me.

iptables dynamic port script for NFS

May 10, 2008

Some days ago I talked with a friend (here a link to his homepage) about firewalls and file servers and he told me he has a iptables script which adapts to the NFS ports automatically. I asked him for this script and here is it. Thx Hannes for the script.


# rpcinfo -p prints a list of all registered RPC programs
# sed -e '1D' removes the headline
# tr -s ' ' '\t' replaces repeated spaces with a single tab
# cut -f 4,5 we only need the protocol- and port-columns
# sort | uniq removes the duplicate lines
# now we have lines with the needed protocol and port but for splits
# this lines to single words so we have to store the protocol
for l in `rpcinfo -p | sed -e '1D' | tr -s ' ' '\t' | cut -f 4,5 | sort | uniq`
do
  case $l in
    tcp)
      SYN=--syn
      PROTOCOL=$l
        ;;
    udp)
      SYN=
      PROTOCOL=$l
        ;;
    *)
      iptables -A INPUT -p $PROTOCOL --dport $l $SYN -j ACCEPT
    ;;
  esac
done

Online VM builder for VMware Player

Every used VMware Player to “play” precreated VMs? I did, but I thought when I need to create VMs by my own I need VMware Workstation or Virtual PC if was running Windows and not Linux. Ok and there is now VirtualBox, but I never used it before, but as it comes now with Ubuntu 8.04 its changes are raising (no need to compile anything like kernel modules for every security update of the kernel). Anyway I found some thing that allows you to use Vmware Player with your own VMs. Following Websites allow you to create images for VMware Player:

EasyVMX!
VM Builder
vmx-builder.cmd

I think the first is the best one. Maybe this info helps also others, as most of the time VMware Player is enough and someone does not need the Workstation version, and Virtual PC 2007 is bad product. Ever tried to run a current Linux kernel on it. It crashes the kernel. I learned that the hard way with Ubuntu 8.04 server within a Virtual PC 2007. Which was not easy to install in the first place, but booting the current kernel after the installation was the end point of the journey, no problem with VMware Player however.

LED blinking on your switch

April 9, 2008

Did you ever have the problem that you didn’t know to which switch port a given ethernet port /cable is connected to? Wouldn’t it be cool if the LED of the switch port would blink so you know which one is the correct one?

You’re lucky – it is possible with Linux. There are even two ways. With some chipsets ethtool -p eth0 works but not with all. But following script also helps in any case:

#/bin/bash
# usage example: blink.sh eth0


while true ; do
  ifconfig $1 down
  sleep 2
  ifconfig $1 up
  sleep 2
done


Put that script into /usr/local/sbin/blink.sh and set the execution permissions. Call it with the device as parameter. Don’t set the blinking below 2sec as it is possible that the connection negation takes up to that amount of time.

Howto setup Asterisk for recording a podcast over the Internet

April 6, 2008

Some friends and I are planing a to make a podcast and as we IT guys we needed something to support our distributed recordings over the Internet. One of my friend lives about 200km away but also with the others it would be not that easy to get all into one room at the same time. When we looked around for various ways to do distributed recordings we found mostly Skype howto’s and we knew only a few podcasts which did the recording with Asterisk (in a not that good audio quality I think). But we didn’t find what we really wanted and so I started to look into that topic. At first I wrote our requirements down:

• the recording should be done centrally on a server automatically without user interaction (so it is not forgotten and to minimize the lag and sound quality problems)
• the recorded files should be easy available as OGG files to all podcast members after the recording, as the post production is maybe not done always by the same people
• various client operation systems and VoIP clients should be supported, therefore an open standard protocol should be used
• possibility to record each participant separately to allow changes in the volume or exclusively applying audio filters
• it should be possible to invite guest for interviews via the same system without requiring more than a VoIP client which support the choosen protocol. (no registration somewhere)
• optional it should be possible to connect the system to the POTS (plain old telephone service) for interviews with people which cannot use a VoIP client.
• And as a requirement by my knowledge and existing infrastructure, the server should run on one of my Linux servers I’ve running in a big data center, within a OpenVZ virtual environment.

After some research I decided to go with Asterisk. This howto describes what I’ve done to reach the above goals. After the completion of this howto you should have following:

• A SIP server where you participants can connect to and talk with each other.
• As soon as they go into a special virtual conference room everything they say will be recorded
• After they leave the conference room a background process will reencode the recorded WAV file to OGG and make it available via web.
• Every participant gets its own OGG file with the starting timestamp in the filename, you need to use the recording at the correct place in the post production.
• The inclusion of a SIP provider with connection to the POTS network is not described in this howto as there are may others describing it.

Some points of this howto are specific to my OpenVZ setup and the chosen distribution but most are generic and should work for any setup. Anyway here is the software I used.

• OpenVZ for virtualization
• Ubuntu 8.04 Hardy (x86_64) as distribution for the virtual environment
• Asterisk and Zaptel for VoIP part (Ubuntu packages)
• sox for translating the WAV files to OGG (Ubuntu package)
• lighttpd as small and fast web server for the OGG files (Ubuntu package)
• Twinkle as SIP client under Linux. (apt-get install twinkle)

Part 1: Hardware node setup

You can ignore that part if you don’t use OpenVZ and your kernel/distribution comes with ztdummy modules. The hardware node in my case runs on a Centos4 (x86_64). This is important as Asterisk needs the ztdummy kernel module, which comes with zaptel, for the meetme Asterisk module which is used for the conference rooms. As it is not possible to load kernel modules within a VE (virtual environment) (that’s a security feature!) I needed it on my hardware node. As the kernel of the hardware is a OpenVZ patched kernel and also Centos 4 does not come with a ztdummy module anyway, I needed to compile it.

I used the same version of zaptel as Ubuntu 8.04 does and it is also very important that you use a 64bit VE if you hardware node is 64bit, otherwise the device cannot be accessed correctly.

The install is quite straightforward

# tar xzf zaptel-1.4.8.tar.gz
# cd zaptel-1.4.8
# ./install_prereq test


Install the required packages and then continue with:

# ./configure
# make
# make install


but no “make config”, as we don’t need init scripts or that stuff. Now load the kernel module with modprobe ztdummy (and make sure that this is done after very boot, before the VEs start). Make sure the device is working with:

# ztcfg -v


The output should be something like:

Zaptel Version: 1.4.8
Echo Canceller: MG2
Configuration
======================
0 channels to configure.


At last we need the VE be able to access the ztdummy device, so we need to tell openVZ this.

# for x in `ls /dev/zap`; do /usr/sbin/vzctl set XXX --devnodes zap/${x}:rw --save; done


Replace the XXX with the ID of your podcast VE. Now we’re done with the hardware node and we can take a look at the user space stuff.

Part 2: Virtual Environment setup

At first we install the packages we need with following command:

# apt-get install asterisk-h323 asterisk-doc speex vpb-utils sox libsox-fmt-all lighttpd zaptel


Now we configure zaptel and check if it works:

# genzaptelconf
# ztcfg -v
# ztcfg -d


No error should be given. If a device is not found check if they got created by vzctl. After that make the devices in /dev/zap read and writable for the asterisk user:

# chown root:asterisk /dev/zap/*
# chmod 660 /dev/zap/*

Now we can work on the Asterisk configuration. We set following values in /etc/default/asterisk:

RUNASTERISK=yes
AST_REALTIME=no


The real time stuff does not work in a VE and gives audio problems. Now we need to do some configuration for NAT users in /etc/asterisk/sip.conf:

externip = you're_external_IP ; this is needed as asterisk has problem with the venet0 stuff otherwise
localnet=192.168.0.0/255.255.0.0; All RFC 1918 addresses are local networks
localnet=10.0.0.0/255.0.0.0 ; Also RFC1918
localnet=172.16.0.0/12 ; Another RFC1918 with CIDR notation
localnet=169.254.0.0/255.255.0.0 ;Zero conf local network


nat=yes
qualify=yes
canreinvite=no


After this global setup we configure for each of our podcasters one section, as shown here:

[firstPodcaster] ; this is also the user name
type=friend
context=sip
secret=the_password_of_this_user
callerid="Your_Name" <1> ; it is recommended to use no spaces here, as we use this as part of the filename. You need the “, “ and < ,> exactly as show here
host=dynamic
dtmfmode=info
disallow=all
allow=alaw
callingpres=allowed_passed_screen


We only support alaw so every client uses G.711a and we don’t need to translate. I believe in the US you need to use G.711u and therefore ulaw. Now we need a conference room for which we add following line to /etc/asterisk/meetme.conf:

conf => 10
conf => 20


Now we need to tie that together with the dial plan in /etc/asterisk/extensions.conf:

[globals]
MONITOR_EXEC=/usr/local/bin/wavIn2ogg.sh


And add following section:

[sip]


; so we can talk directly with another
exten => 31,1,Dial(SIP/firstPodcaster,20,tr)
exten => 32,1,Dial(SIP/secondPodcaster,20,tr)
exten => 33,1,Dial(SIP/thirdPodcaster,20,tr)


; this conference room is not recorded, for preparations
exten => 10,1,Answer
exten => 10,2,Wait(1)
exten => 10,3,Meetme(10,s)


; this conference room records automatically
exten => 20,1,Answer
exten => 20,2,Wait(1)
exten => 20,3,Set(CALLFILENAME=podcast_X_${CALLERID(name)}-${STRFTIME(${EPOCH},,%Y%m%d-%H%M%S)})
exten => 20,4,Monitor(wav,${CALLFILENAME},m)
exten => 20,5,Meetme(20,s)


; from the demo code – useful to look at the quality of your connection
; Create an extension, 60, for evaluating echo latency.
exten => 60,1,Playback(demo-echotest) ; Let them know what's going on
exten => 60,n,Echo ; Do the echo test
exten => 60,n,Playback(demo-echodone) ; Let them know it's over
exten => 60,n,Goto(s,60) ; Start over


Now you can restart Asterisk and connect with you SIP client. Call 60 to check if the audio stream works in both directions (try it without firewall on the Asterisk server if don’t hear anything). After that go into the conference room 10 with 1 or 2 friends and test it. If that all works you can work on the recording stuff. Create following script /usr/local/bin/wavIn2ogg.sh (don’t forget chmod 755):

#!/bin/bash
# wavIn2ogg.sh - creates ogg of the input mono stream
# used for recording each participant in a meeting room separately
# Written by Robert Penz


SOX=/usr/bin/sox
NICE=/usr/bin/nice


InFile="$1"
OutFile="$2"
OggFile=`echo $3|sed s/.wav//`.ogg
FinalDir=/var/www/production


#test if input files exist
test ! -r $InFile && exit
test ! -r $OutFile && exit


$NICE -n 19 $SOX -t wav "$InFile" -t vorbis "$OggFile"


#remove input files if successfull
test -r "$OggFile" && rm "$InFile" "$OutFile"


# at last set the permissions and move it in an atomic way
chmod 644 "$OggFile"
mv "$OggFile" "$FinalDir/."


Create the directory /var/www/production and set the correct permissions:

# mkdir /var/www/production
# chown asterisk:www-data /var/www/production


Now go the the conference room 20 and say something and disconnect. If it worked you should see with your browser under http:///production/ the recorded OGG file(s). If there are none, take a look at /var/spool/asterisk/monitor/ if there are 2 WAV files. If so call the wavIn2ogg.sh script by hand and look for any errors.

So thats the end of the story – you’ve now a system for recording podcasts over the internet in a cool way! Any comments, ideas or questions? Post them here.