Howto setup a Debian 9 with Proxmox and containers using as few IPv4 and IPv6 addresses as possible

August 4, 2017

My current Linux Root-Server needs to be replaced with a newer Linux version and should also be much cheaper then the current one. So at first I did look what I don’t like about the current one:

  • It is expensive with about 70 Euros / months. Following is responsible for that
    • My own HPE hardware with 16GB RAM and a software RAID (hardware raid would be even more expensive) – iLo (or something like it) is a must for me 🙂
    • 16 additional IPv4 addresses for the visualized container and servers
    • Large enough backup space to get back some days.
  • A base OS which makes it hard to run newer Linux versions in the container (sure old ones like CentOS6 still get updates, but that will change)
    • Its time to move to newer Linux versions in the containers
  • OpenVZ based containers which are not mainstream anymore

Then I looked what surrounding conditions changed since I did setup my current server.

  • I’ve IPv6 at home and 70% of my traffic is IPv6 (thx to Google (specially Youtube) and Cloudflare)
  • IPv4 addresses got even more expensive for Root-Servers
  • I’m now using Cloudflare for most of the websites I host.
  • Cloudflare is reachable via IPv4 and IPv6 and can connect back either with IPv4 or IPv6 to my servers
  • With unprivileged containers the need to use KVM for security lessens
  • Hosting providers offer now KVM servers for really cheap, which have dedicated reserved CPUs.
  • KVM servers can host containers without a problem

This lead to the decision to try following setup:

  • A KVM based Server for less than 10 Euro / month at Netcup to try the concept
  • No additional IPv4 addresses, everything should work with only 1 IPv4 and a /64 IPv6 subnet
  • Base OS should be Debian 9 (“Stretch”)
  • For ease of configuration of the containers I will use the current Proxmox with LXC
  • Don’t use my own HTTP reverse proxy, but use exclusively Cloudflare for all websites to translate from IPv4 to IPv6

After that decision was reached I search for Howtos which would allow me to just set it up without doing much research. Sadly that didn’t work out. Sure, there are multiple Howtos which explain you how to setup Debian and Proxmox, but if you get into the nifty parts e.g. using only minimal IP addresses, working around MAC address filters at the hosting providers (which is quite a important security function, BTW) and IPv6, they will tell you: You need more IP addresses, get a really complicated setup or just ignore that point at all.

As you can read that blog post you know that I found a way, so expect a complete documentation on how to setup such a server. I’ll concentrate on the relevant parts to allow you to setup a similar server. Of course I did also some security harding like making a secure ssh setup with only public keys, the right ciphers, …. which I won’t cover here.

Setting up the OS

I used the Debian 9 minimal install, which Netcup provides, and did change the password, hostname, changed the language to English (to be more exact to C) and moved the SSH Port a non standard port. The last one I did not so much for security but for the constant scans on port 22, which flood the logs.

passwd
vim /etc/hosts
vim /etc/hostname
dpkg-reconfigure locales
vim /etc/ssh/sshd_config
/etc/init.d/ssh restart

I followed that with making sure no firewall is active and installed the net-tools so I got netstat and ifconfig.

apt install net-tools

At last I did a check if any packages needs an update.

apt update
apt upgrade

Installing Proxmox

First I checked if the IP address returns the correct hostname, as otherwise the install fails and you need to start from scratch.

hostname --ip-address

Adding the Proxmox Repos to the system and installing the software:

echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg
apt update && apt dist-upgrade
apt install proxmox-ve postfix open-iscsi

After that I did a reboot and booted the Proxmox kernel, I removed some packages I didn’t need anymore

apt remove os-prober linux-image-amd64 linux-image-4.9.0-3-amd64

Now I did my first login to the admin GUI to https://<hostname>:8006/ and enabled the Proxmox firewall

Than set the firewall rules for protecting the host (I did that for the whole datacenter even if I only have one server at this moment). Ping is allowed, the Webgui and ssh.

I mate sure with

iptables -L -xvn

that the firewall was running.

BTW, if you don’t like the nagging windows at every login that you need a license and if this is only a testing machine as mine is currently, type following:

sed -i.bak 's/NotFound/Active/g' /usr/share/perl5/PVE/API2/Subscription.pm && systemctl restart pveproxy.service

Now we need to configure the network (vmbr0) for our virtual systems and this is the point where my Howto will go an other direction. Normally you’re told to configure the vmbr0 and put the physical interface into the bridge. This bridging mode is the easiest normally, but won’t work here.

Routing instead of bridging

Normally you are told that if you use public IPv4 and IPv6 addresses in containers you should bridge it. Yes thats true, but there is one problem. LXC containers have their own MAC addresses. So if they send traffic via the bridge to the datacenter switch, the switch sees the virtual MAC address. In a internal company network on a physical host that is normally not a problem. In a datacenter where different people rent their servers thats not good security practice. Most hosting providers will filter the MAC addresses on the switch (sometimes additional IPv4 addresses come with the right to use additional MAC addresses, but we want to save money here 🙂 ). As this server is a KVM guest OS the filtering is most likely part of the virtual switch (e.g. for VMware ESX this is the default even).

With ebtables it is possible to configure a SNAT for the MAC addresses, but that will get really complicated really fast – trust me with networking stuff – when I say complicated it is really complicated fast. 🙂

So, if we can’t use bridging we need to use routing. Yes the routing setup on the server is not so easy, but it is clean and easy to understand.

First we configure the physical interface in the admin GUI

Two configurations are different than at normal setups. The provider gave you most likely a /23 or /24, but I use a subnet mask /32 (255.255.255.255), as I only want to talk to the default gateway and not the other servers from other customers. If the switch thinks traffic is ok, he can reroute it for me. The provider switch will defend its IP address against ARP spoofing, I’m quite sure as otherwise a incorrect configuration of a customer will break the network for all customer – the provider will make that mistake only once. For IPv6 we do basically the same with /128 but in this case we also want to reuse the /64 subnet on our second interface.

As I don’t have additional IPv4 addresses, I’ll use a local subnet to provide access to IPv4 addresses to the containers (via NAT), the IPv6 address gets configured a second time with the /64 subnet mask. This setup allows use to route with only one /64 – we’re cheap … no extra money needed.

Now we reboot the server so that the /etc/network/interfaces config gets written. We need to add some additional settings there, so it looks like this

The first command in the red frame is needed to make sure that traffic from the containers pass the second rule. Its some kind lxc specialty. The second command is just a simple SNAT to your public IPv4 address. The last 2 are for making sure that the iptable rules get deleted if you stop the network.

Now we need to make sure that the container traffic gets routed so we put following lines into /etc/sysctl.conf

And we should also enable following lines

Now we’re almost done. One point remains. The switch/router which is our default gateway needs to be able to send packets to our containers. For this he does for IPv6 something similar to an ARP request. It is called neighbor discovery and as the network of the container is routed we need to answer the request on the host system.

Neighbor Discovery Protocol (NDP) Proxy

We could now do this by using proxy_ndp, the IPv6 variant of proxy_arp. First enable proxy_ndp by running:

sysctl -w net.ipv6.conf.all.proxy_ndp=1

You can enable this permanently by adding the following line to /etc/sysctl.conf:

net.ipv6.conf.all.proxy_ndp = 1

Then run:

ip -6 neigh add proxy 2a03:5000:3d:1ee::100 dev ens3

This means for the host Linux system to generate Neighbor Advertisement messages in response to Neighbor Solicitation messages for 2a03:5000:3d:1ee::100 (e.g. our container with ID 100) that enters through ens3.

While proxy_arp could be used to proxy a whole subnet, this appears not to be the case with proxy_ndp. To protect the memory of upstream routers, you can only proxy defined addresses. That’s not a simple solution, if we need to add an entry for every container. But we’re saved from that as Debian 9 ships with an daemon that can proxy a whole subnet, ndppd. Let’s install and configure it:

apt install ndppd
cp /usr/share/doc/ndppd/ndppd.conf-dist /etc/ndppd.conf

and write a config like this

route-ttl 30000
proxy ens3 {
router no
timeout 500
ttl 30000
rule 2a03:5000:3d:1ee::/64 {
auto
}
}

now enable it by default and start it

update-rc.d ndppd defaults
/etc/init.d/ndppd start

Now it is time to boot the system and create you first container.

Container setup

The container setup is easy, you just need to use the Proxmox host as default gateway.

As you see the setup is quite cool and it allows you to create containers without thinking about it. A similar setup is also possible with IPv4 addresses. As I don’t need it I’ll just quickly describe it here.

Short info for doing the same for an additional IPv4 subnet

Following needs to be added to the /etc/network/interfaces:

iface ens3 inet static
pointopoint 186.143.121.1

iface vmbr0 inet static
address 186.143.121.230 # Our Host will be the Gateway for all container
netmask 255.255.255.255
# Add all single IP's from your /29 subnet
up route add -host 186.143.34.56 dev br0
up route add -host 186.143.34.57 dev br0
up route add -host 186.143.34.58 dev br0
up route add -host 186.143.34.59 dev br0
up route add -host 186.143.34.60 dev br0
up route add -host 186.143.34.61 dev br0
up route add -host 186.143.34.62 dev br0
up route add -host 186.143.34.63 dev br0
.......

We’re reusing the ens3 IP address. Normally we would add our additional IPv4 network e.g. a /29. The problem with this straight forward setup would be that we would lose 2 IP addresses (netbase and broadcast). Also the pointopoint directive is important and tells our host to send all requests to the datacenter IPv4 gateway – even if we want to talk to our neighbors later.

The for the container setup you just need to replace the IPv4 config with following

auto eth0
iface eth0 inet static
address 186.143.34.56 # Any IP of our /29 subnet
netmask 255.255.255.255
gateway 186.143.121.13 # Our Host machine will do the job!
pointopoint 186.143.121.1

How that saved you some time setting up you own system!

WannaCry happened and nobody called me during my vacation – I tell you why

May 18, 2017

I was since last Wednesday on a biking trip through Austria and Bavaria, when on Friday reading main stream media the world broke down with WannaCry. Ok, I thought sensationalism by the main media but now as I’m at home, I cannot believe what I read in tech blogs and the IT media. I won’t link all of them here, just the one I plainly can’t understand and to which I disagree in the strongest way possible – telling plainly patching is hard and we can’t do anything.

Lets start with how a WannaCry infection spreads through a company.

  • The malware needed to get into the company network – be it via open SMB ports (445 TCP) to the Internet and via Email. As I read through the articles its not 100% clear how the infection – lets assume both methods have been used for this post
  • In the second case a user clicks onto the attachment and the malware gets executed
  • Than it searches through the company network and tries with a RCE (remote code execution) to infect other PCs
  • It encrypted the local hard drive

Now lets talk why this should not have been any problem in your organization

  • Port 445 TCP reachable from the Internet? Really? If you’re unsure, quickly go to Shodan and type into the search field net:"xxx.xxx.xxx.xxx/yy" with xxx.xxx.xxx.xxx you IP address range followed by the yy subnet mask and take a look if you know about open ports and services.
  • And now lets take a look at all the stuff I wrote over a year ago, what you should have done before the Locky malware happened (yes this is not the first ransomware making big waves), to be not affected:Stop panicking about the Locky ransomware [Update 2]
    • For the Email infection vector:
      • Block EXE attachments in emails
      • Remove active code from Word, Excel and Powerpoint files by default
      • Block EXE downloads on the Proxy
    • For both  infection vectors:
      • Use application white-listing – we moved to whitelisting for firewall rules a long time ago, its past time to do that for applications. Guess why there is not so much iPhone malware – Apple is effectively white-listing software.
      • Block client 2 client traffic – Even if that is not possible on a day2day basis, it should be prepared to be enabled in a case of emergency.

    With one of the last two alone an widespread infection would not have been possible.

  • Microsoft provided a patch on March 14 and called the vulnerability critical. Lets take a look when Microsoft calls some vulnerabilities  critical and when important. The difference is that with important the user gets asked and than infected, with critical there is nothing, just infection. So important is remote code execute and critical is wormable remote code execution.
    And at last take a look at following text from Microsoft: “Mitigating Factors: Microsoft has not identified any mitigating factors for this vulnerability.” To make it short if you read about such a vulnerability in Windows and know that an exploit is in the wild, drop everything and start patching that hole at once.

Looking at the above ways the malware/worm could have easily been blocked. Anyway at last I really want now to take a look at the post linked above from the SMBlog by Steven M. Bellovin.

  • Because patching is very hard and very risk, and the more complex your systems are, the harder and riskier it is.
    Thats not true in this case.

    • Port 445 open to the Internet, no real network separation, deactivated local Windows Firewall and still have SMB1 activated on Windows Client Systems (see Microsoft recommendation from 2016) – thats not at patching problem, thats a security policy failure (e.g. base hardening of operating systems)
    • standard client PCs (for the normal employee) not patched  – not talking about special systems – we patch thousand PCs every month after the Windows patches are released without any problems in years. The special systems needed to get infected after all by something.
    • If non company managed client PCs got connected to the network and infected special systems, its a failure in network access control – plane and simple
    • no mitigation prepared for a case like a worm breakout – Just to make a point, we prepared a client2client block  ACLs for all edge switches, which could be activated within a few minutes, in 2011 – as you newer can know. This is a missing emergency plan like required in ISO 27001.
    • 2 month window for patching an remote code execution wormable vulnerability. If it was not possible in 2 month to patch something like that, than the company has a high technical/security debt. This is a management failure.
    • still running non supported software – that is a management failure, by not making correct contracts with the vendor or ignoring the problem like a Ostrich.
  • So—if you’re the CIO, what do you do? Break the company, or risk an attack? (Again, this is an imaginary conversation.)
    Thats the wrong question – if the CIO is at the questing he has done a bad job before:

    • All your critical software should have a maintenance contract which specially handles security updates (and specially the timeline) of the underlaying operating system and the software itself and there must be contractual penalty in it. Done that for year now with “call for bids” – Big IT companies provide that security handling without you asking for it – so this is mainly for special software.
    • If the IT department has not the time to patch everything a Triage needs to be done. The vulnerabilities with the highest probability and potential for damage need to be patched first – this vulnerability must have been on the top of any list.
    • The systems are not as in German is called “Stand der Technik” which can be translated as “state of the art” – an Windows XP system is not state of the art, no meaningful network separation is not state of the art, …..
  • That patching is so hard is very unfortunate. Solving it is a research question. Vendors are doing what they can to improve the reliability of patches, but it’s a really, really difficult problem.
    • Ok patching might be not a easy as it could be but
      • A big institution that got caught by this malware did leave the back door open and is now complaining that a herd of wild boars went through the house and did damage.
      • the security and IT department just failed at their job – just do a postmortem without finger pointing and fix the problems. I’m quite sure the affected IT departments got caught also by the Locky malware and didn’t learn a thing.
      • Vendors doing not enough, sure in this case Microsoft did patch it but specially with IoT devices vendor to nothing.
      • Searching through Google for problems after installing MS17-010 reviles only a few post after billions of updated PCs –> there are no problems with this patch –> no reason to not install it

So thats my view onto the WannaCry stuff after being on vacation ….. tell me your views – did I miss something?

Mitigating application layer (HTTP(S)) DDOS attacks

April 23, 2017

DDOS attacks seem to be new norm on the Internet. Years before only big websites and web applications got attacked but nowadays also rather small and medium companies or institutions get attacked. This makes it necessary for administrators of smaller sites to plan for the time they get attacked. This blog post shows you what you can do yourself and for what stuff you need external help. As you’ll see later you most likely can only mitigate DDOS attacks against the application layer by yourself and need help for all other attacks. One important part of a successful defense against a DDOS attack, which I won’t explain here in detail, is a good media strategy. e.g. If you can convince the media that the attack is no big deal, they may not report sensational about it and make the attack appear bigger and more problematic than it was. A classic example is a DDOS against a website that shows only information and has no impact on the day to day operation. But there are better blogs for this non technical topic, so lets get into the technical part.

different DDOS attacks

From the point of an administrator of a small website or web application there are basically 3 types of attacks:

  • An Attack that saturates your Internet or your providers Internet connection. (bandwidth and traffic attack)
  • Attacks against your website or web application itself. (application attack)

saturation attacks

Lets take a closer look at the first type of attack. There are many different variations of this connection saturation attacks and it does not matter for the SME administrator. You can’t do anything against it by yourself. Why? You can’t do anything on your server as the good traffic can’t reach your server as your Internet connection or a connection/router of your Internet Service Provider (ISP) is already saturated with attack traffic. The mitigation needs to take place on a system which is before the part that is saturated. There are different methods to mitigate such attacks.

Depending on the type of website it is possible to use a Content Delivery Networks (CDN). A CDN basically caches the data of your website in multiple geographical distributed locations. This way each location gets only attacked by a part of the attacking systems. This is a nice way to also guard against many application layer attacks but does not work (or not easily) if the content of your site is not the same for every client / user. e.g. an information website with some downloads and videos is easily changed to use a CDN but a application like a Webmail system or an accounting system will be hard to adapt and will not gain 100% protection even than. An other problem with CDNs is that you must protect each website separately, thats ok if you’ve only one big website that is the core of your business, but will be a problem if attacker can choose from multiple sites/applications. An classic example is that a company does protect its homepage with an CDN but the attacker finds via Google the Webmail of the companies Exchange Server. Instead of attacking the CDN, he attacks the Internet connection in front of the Qebmail. The problem will now most likely be that the VPN site-2-site connections to the remote offices of the company are down and working with the central systems is not possible anymore for the employees in the remote locations.

So let assume for the rest of the document that using a CDN is not possible or not feasible. In that case you need to talk to your ISPs. Following are possible mitigations a provider can deploy for you:

  • Using a dedicated DDOS mitigation tool. These tools take all traffic and will filter most of the bad traffic out. For this to work the mitigation tool needs to know your normal traffic patterns and the DDOS needs to be small enough that the Internet connections of the provider are able to handle it. Some companies sell on on-premise mitigation tools, don’t buy it, its wasting money.
  • If the DDOS attack is against an IP address, which is not mission critical (e.g. attack is against the website, but the web application is the critical system) let the provider block all traffic to that IP address. If the provider as an agreement with its upstream provider it is even possible to filter that traffic before it reaches the provider and so this works also if the ISPs Internet connection can not handle the attack.
  • If you have your own IP space it is possible for your provider(s) to stop announcing your IP addresses/subnet to every router in the world and e.g. only announce it to local providers. This helps to minimize the traffic to an amount which can be handled by a mitigation tool or by your Internet connection. This is specially a good mitigation method, if you’re main audience is local. e.g. 90% of your customers/clients are from the same region or country as you’re – you don’t care during an attack about IP address from x (x= foreign far away country).
  • A special technique of the last topic is to connect to a local Internet exchange which maybe also helps to reduce your Internet costs but in any case raises your resilience against DDOS attacks.

This covers the basics which allows you to understand and talk with your providers eye to eye. There is also a subsection of saturation attacks which does not saturate the connection but the server or firewall (e.g. syn floods) but as most small and medium companies will have only up to a one Gbit Internet connection it is unlikely that a descend server (and its operating system) or firewall is the limiting factor, most likely its the application on top of it.

application layer attacks

Which is a perfect transition to this chapter about application layer DDOS. Lets start with an example to describe this kind of attacks. Some years ago a common attack was to use the ping back feature of WordPress installations to flood a given URL with requests. I’ve seen such an attack which requests on a special URL on an target system, which did something CPU and memory intensive, which let to a successful DDOS against the application with less than 10Mbit traffic. All requests were valid requests and as the URL was an HTTPS one (which is more likely than not today) a mitigation in the network was not possible. The solution was quite easy in this case as the HTTP User Agent was WordPress which was easy to filter on the web server and had no side effects.

But this was a specific mitigation which would be easy to bypassed if the attacker sees it and changes his requests on his botnet. Which also leads to the main problem with this kind of attacks. You need to be able to block the bad traffic and let the good traffic through. Persistent attackers commonly change the attack mode – an attack is done in method 1 until you’re able to filter it out, than the attacker changes to the next method. This can go on for days. Do make it harder for an attacker it is a good idea to implement some kind of human vs bot detection method.

I’m human

The “I’m human” button from Google is quite well known and the technique behind it is that it rates the connection (source IP address, cookies (from login sessions to Google, …) and with that information it decides if the request is from a human or not. If the system is sure the request is from a human you won’t see anything. In case its sightly unsure a simple green check-mark will be shown, if its more unsure or thinks the request is by a bot it will show a CAPTCHA.  So the question is can we implement something similar by ourself. Sure we can, lets dive into it.

peace time

Set an special DDOS cookie if an user is authenticated correctly, during peace time. I’ll describe the data in the cookie later in detail.

war time

So lets say, we detected an attack manually or automatically by checking the number of requests eg. against the login page. In that case the bot/human detection gets activated. Now the web server checks for each request the presence of the DDOS cookie and if the cookie can be decoded correctly. All requests which don’t contain a valid DDOS cookie get redirected temporary to a separate host e.g. https://iamhuman.example.org. The referrer is the original requested URL. This host runs on a different server (so if it gets overloaded it does not effect the normal users). This host shows a CAPTCHA and if the user solves it correctly the DDOS cookie will be set for example.org and a redirect to the original URL will be send.

Info: If you’ve requests from some trusted IP ranges e.g. internal IP address or IP ranges from partner organizations you can exclude them from the redirect to the CAPTCHA page.

sophistication ideas and cookie

An attacker could obtain a cookie and use it for his bots. To guard against it write the IP address of the client encrypted into the cookie. Also put the timestamp of the creation of the cookie encrypted into it. Also storing the username, if the cookie got created by the login process, is a good idea to check which user got compromised.

Encrypt the cookie with an authenticated encryption algorithm (e.g. AES128 GCM) and put following into it:

  • NONCE
  • typ
    • L for Login cookie
    • C for Captcha cookie
  • username
    • Only if login cookie
  • client IP address
  • timestamp

The key for the encryption/decryption of the cookie is static and does not leave the servers. The cookie should be set for the whole domain to be able to protected multiple websites/applications. Also make it HttpOnly to make stealing it harder.

implementation

On the normal web server which checks the cookie following implementations are possible:

  • The apache web server provides a module called mod_session_* which provides some functionality but not all
  • The apache module rewriteMap (https://httpd.apache.org/docs/2.4/rewrite/rewritemap.html) and using „prg: External Rewriting Program“ should allow everything. Performance may be an issue.
  • Your own Apache module

If you know about any other method, please write a comment!

The CAPTCHA issuing host is quite simple.

  • Use any minimalistic website with PHP/Java/Python to create cookie
  • Create your own CAPTCHA or integrate a solution like Recaptcha

pro and cons

  • Pro
    • Users than accessed authenticated within the last weeks won’t see the DDOS mitigation. Most likely these are your power users / biggest clients.
    • Its possible to step up the protection gradually. e.g. the IP address binding is only needed when the attacker is using valid cookies.
    • The primary web server does not need any database or external system to check for the cookie.
    • The most likely case of an attack is that the cookie is not set at all which does take really few CPU resources to check.
    • Sending an 302 to the bot does create only a few bytes of traffic and if the bot requests the redirected URL it on an other server and there no load on the server we want to protect.
    • No change to the applications is necessary
    • The operations team does not to be experts in mitigating attacks against the application layer. Simple activation is enough.
    • Traffic stats local and is not send to external provider (which may be a problem for a bank or with data protections laws in Europe)
  • Cons
    • How to handle automatic requests (API)? Make exceptions for these or block them in case of an attack?
    • Problem with non browser Clients like ActiveSync clients.
    • Multiple domains need multiple cookies

All in all I see it as a good mitigation method for application layer attacks and I hope the blog post did help you and your business. Please leave feedback in the comments. Thx!

How to brute force a MySQL DB

December 29, 2016

There are many articles on how to use Metasploit or some other mighty stuff that is fine if you work with it all day. But if you just found a MySQL server on an appliance listening in your network and need to do a fast small security check there is something easier. First find the MySQL server and check the version – maybe there is a exploit available and you don’t need to try passwords. The first choice for this is nmap, just install it with sudo apt-get install nmap and call it like this:

# nmap -sV -O <IP>

Starting Nmap 7.01 ( https://nmap.org ) at 2016-xx-xx xx:xx CET
Nmap scan report for hostname (<IP>)
Host is up (0.020s latency).
Not shown: 986 closed ports
PORT STATE SERVICE VERSION
....
3306/tcp open mysql MySQL 5.6.33-79.0-log
....
Device type: general purpose
Running: Linux 3.X|4.X
OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4
OS details: Linux 3.2 - 4.0
Network Distance: x hops

You need to call it as root with these options. The -sV shows the versions of the listening services and -O guesses the operating system.  For brute forcing we need 3 things

  • a list of usernames to try
  • a list of passwords to try
  • a software that does the trying

The first is for one thing quite easy as the default users are known and you maybe know something about the system .. like software name or vendor name or the online download-able manual shows the username. So lets write the file:

$ cat > usernames.txt
admin
root
mysql
db
test
user

Now we need a list of likely passwords .. sure we could think about some by our own, but it is easier to download them. A good source is Skull Security. Choose your list and download it and extract it with bunzip2 xxxxx.txt.bz2. Now we only need the software … we’ll use THC Hydra, but you don’t need to download it there and compile it, as Ubuntu ships with it. Just type sudo apt-get install hydra. Now we just need to call it.

$ hydra -L usernames.txt -P xxxxx.txt <ip> mysql
Hydra v8.1 (c) 2014 by van Hauser/THC - Please do not use in military or secret service organizations, or for illegal purposes.
Hydra (http://www.thc.org/thc-hydra) starting at 2016-12-29 14:19:38
[INFO] Reduced number of tasks to 4 (mysql does not like many parallel connections)
[DATA] max 4 tasks per 1 server, overall 64 tasks, 86066388 login tries (l:6/p:14344398), ~336196 tries per task
[DATA] attacking service mysql on port 3306
[STATUS] 833.00 tries/min, 833 tries in 00:01h, 86065555 todo in 1721:60h, 4 active
.....

I think this password list is too long 😉 Choose a shorter one 😉

Accessing Mikrotik RouterOS via MAC Telnet from a Linux box

November 18, 2016

If you know Mikrotik Routers you know that you’re able to access them via MAC Telnet (see here for more details) via Layer2 with Winbox. But running Winbox via Wine on a Linux is not that great for using MAC Telnet, and there is a better way .. just use MAC-Telnet from Håkon Nessjøen. On Ubuntu/Debian you can just install the package with

sudo apt-get install mactelnet-client

and you see its feature like this:

$ mactelnet -h
MAC-Telnet 0.4.2
Usage: mactelnet <MAC|identity> [-h] [-n] [-a <path>] [-A] [-t <timeout>] [-u <user>] [-p <password>] [-U <user>] | -l [-B] [-t <timeout>]

Parameters:
MAC MAC-Address of the RouterOS/mactelnetd device. Use mndp to
discover it.
identity The identity/name of your destination device. Uses
MNDP protocol to find it.
-l List/Search for routers nearby (MNDP). You may use -t to set timeout.
-B Batch mode. Use computer readable output (CSV), for use with -l.
-n Do not use broadcast packets. Less insecure but requires
root privileges.
-a <path> Use specified path instead of the default: ~/.mactelnet for autologin config file.
-A Disable autologin feature.
-t <timeout> Amount of seconds to wait for a response on each interface.
-u <user> Specify username on command line.
-p <password> Specify password on command line.
-U <user> Drop privileges to this user. Used in conjunction with -n
for security.
-q Quiet mode.
-h This help.

So lets give it a try, first with searching for my home router

$ mactelnet -l
Searching for MikroTik routers... Abort with CTRL+C.
IP MAC-Address Identity (platform version hardware) uptime
10.x.x.x 0:xx:xx:xx:xx:xx jumpgate (MikroTik x.x.x. xxxx) up 139 days 5 hours XXXXX-XXXX vlanInternal

and then we’ll connect

$ mactelnet 0:xx:xx:xx:xx:xx

and we’re connected.

Howto live-sniffer traffic on a remote Linux system with Wireshark

October 2, 2016

You ask why you should need this at all? Easy, sometimes a tcpdump is not enough or not that easy to use:

Sure, it’s quite easy to sniffer on a remote Linux box with tcpdump into an file and copy that that over via scp to the local system and take a closer look at the traffic. But getting used to the feature of my Mikrotik routers to stream traffic live to my local Wireshark, I thought something similar must also be possible with normal Linux boxes. And sure it is.

We just use ssh to pipe the captured traffic through to the local Wireshark. Sure this is not the perfect method for GBytes of traffic but often you just need a few packets to check something or monitor some low volume traffic. Anyway first we need to make sure that Wireshark is able to execute the dumpcap command with our current user. So we need to check the permissions

ll /usr/bin/dumpcap
-rwxr-xr-- 1 root wireshark 88272 Apr 8 11:53 /usr/bin/dumpcap*

So on Ubuntu/Debian we need to add ourself to the wireshark group and check that it got applied with the id command (You need to logoff or start a new sesson with su - $user beforehand). Now you can simply call:

ssh [email protected] 'tcpdump -f -i eth0 -w - not port 22' | wireshark -k -i -

wireshark

And now the really cool part comes. I’m using Ubiqity Unifi access points in multiple setups and I sometimes need to look at the traffic a station communicates with the access point on the wireless interface. With that commands I’m able to ssh into the access point and look at the live traffic of an access point and a station which is hundreds of kilometres way. You can ssh into the AP with your normal web GUI user (if not configured differently) and the bridge config looks like this

BZ.v3.7.8# brctl show
bridge name bridge id STP enabled interfaces
br0 ffff.00272250d9cf no ath0
ath1
ath2
eth0

You can choose one of that interfaces (or the bridge) for normal IP traffic or go one level deeper with wifi0, which looks like this

ssh [email protected] 'tcpdump -f -i wifi0 -w -' | wireshark -k -i -

wireshark2

That’s cool!?! 😉

Howto protect BGP router against DOS and injection attacks

September 17, 2016

It is good practice to configure an individual MD5 password for each BGP peer, but this is not enough. Why?

  • Resource consumption attacks against TCP connections protected with MD5 as the router must verify the MD5 signature of packets it receives
  • Many routers are based on Linux as there base operating system and there is a weakness which allows an attacker to insert arbitrary data into TCP connection. For more details click on this link.

The classical BGP TTL security is based on using a low TTL, generally 1, for single-hop BGP connections. The measure is effective in preventing a BGP connection from being established from a peer more than one hop away. Why? routers decrement the TTL when routing the packets and won’t route packets with a TTL of 1. So if your BGP session is a multi hop connection over one router a TTL of 2 makes sure the packet travels only over one router and not more. That sounds fine but has still has a drawback, as packets with a TTL of 1 are trivial to spoof, so rogue packets will still reach the router – leading to the problems described above.

GTSM (Global TTL Security Mechanism; RFC 5082, which obsolated RFC 3682) suggests the opposite approach. Instead of using a TTL value of 1, it suggests a value of 255 and discarding any packets received with have a TTL lower than 255 minus the hop count for this BGP session. Doing that an attacker is not able to perform the attack as the TTL gets decremented by every router (something the attacker can’t prevent). So setting the TTL on the router to 255 and having no multi hop BGP session allows to drop all packets on the receiving router which are lower than 255. This way only an attack in the same subnet is possible.

After the theory here the actual doing:

For Cisco routers is quite easy, just add ttl-security hops 1 (for a BGP session in the local subnet) to the peering config neighbor x.x.x.x command. For Mikrotik routers it is a little more complicated. For IPv4 connections configure following (should be default anyway):

/routing bgp peer set 0 ttl=255

For dropping incoming packets just use the firewall on the Mikrotik with a command like this:

/ip firewall filter add action=drop chain=input log=yes log-prefix="RFC 5082 block" src-address=xxx.xxx.xxx.xxx ttl=less-than:255

For IPv6 it is a little more complicated as setting the TTL for bgp peer configuration does have no effect (I’ve reported the bug already) but there is a simple workaround possible. Just use the mangle function of the firewall to set the correct hop count:

/ipv6 firewall mangle add action=change-hop-limit chain=output dst-address=xxx:xxx:xxx::xxx/128 new-hop-limit=set:255

Now you need only to filter the IPv6 packets:

/ipv6 firewall filter add action=drop chain=input log=yes log-prefix="RFC 5082 block" src-address=xxx:xxx:xxx::xxx/128 hop-limit=less-than:248

This configuration is quite easy and minimal invasive but should help a lot against attacks on your BGP routers.

Block Ransomware botnet C&C traffic with a Mikrotik router

March 14, 2016

In my last blog post I wrote about blocking, detecting and mitigating the Locky Ransomware. I’ve referenced to a earlier blog post of mine which allows to block traffic to/from the Tor network. This blog post combines both – a way to block Ransomware botnet C&C traffic on a Mikrotik router. The base are the block lists from Abuse.ch, which also provide a nice statistic. Locky is not the most common Ransomware today.

ransomware

Linux part

You need also a small Linux/Unix server to help. This server needs to be trustworthy one as the router executes a script this server generates. This is required as RouterOS is only able to parse text files up to 4096 by itself, and the IP address and domain list is longer.

So first we create the script /usr/local/sbin/generateMalwareBlockScripts.py on the Linux server by downloading following Python script. Open the file and change the paths to your liking. The filename path works on CentOS, on Ubuntu you need to remove the html directory. Now make the file executable

chmod 755 /usr/local/sbin/generateMalwareBlockScripts.py

and execute it

/usr/local/sbin/generateMalwareBlockScripts.py

No output is good. Make sure that the file is reachable via HTTP (e.g. install httpd on CentOS) from the router. If everything works make sure that the script is called once every hour to update the list. e.g. place a symlink in /etc/cron.hourly:

ln -s /usr/local/sbin/generateMalwareBlockScripts.py /etc/cron.hourly/generateMalwareBlockScripts.py

Mikrotik part

Copy and paste following to get the IP address script onto the router:

/system script
add name=scriptUpdateMalwareIPs owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source="# Script which will download a script which adds the malware IP addresses to an address-list\
\n# Using a script to add this is required as RouterOS can only parse 4096 byte files, and the list is longer\
\n# Written by Robert Penz <[email protected]> \
\n# Released under GPL version 3\
\n\
\n# get the \"add script\"\
\n/tool fetch url=\"http://10.xxx.xxx.xxx/addMalwareIPs.rsc\" mode=http\
\n:log info \"Downloaded addMalwareIPs.rsc\"\
\n\
\n# remove the old entries\
\n/ip firewall address-list remove [/ip firewall address-list find list=addressListMalware]\
\n\
\n# import the new entries\
\n/import file-name=addMalwareIPs.rsc\
\n:log info \"Removed old IP addresses and added new ones\"\
\n"

and copy and paste following for the DNS filtering script – surely you can combine them … I let them separated as maybe someone needs only one part:

/system script
add name=scriptUpdateMalwareDomains owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source="# Script which will download a script which adds the malware domains as static DNS entry\
\n# Using a script to add this is required as RouterOS can only parse 4096 byte files, and the list is longer\
\n# Written by Robert Penz <[email protected]> \
\n# Released under GPL version 3\
\n\
\n# get the \"add script\"\
\n/tool fetch url=\"http://10.xxx.xxx.xxx/addMalwareDomains.rsc\" mode=http\
\n:log info \"Downloaded addMalwareDomains.rsc\"\
\n\
\n# remove the old entries\
\n/ip dns static remove [/ip dns static find comment~\"addMalwareDomains\"]\
\n\
\n# import the new entries\
\n/import file-name=addMalwareDomains.rsc\
\n:log info \"Removed old domains and added new ones\"\
\n"

To make the first try run use following command

/system script run scriptUpdateMalwareIPs

and

/system script run scriptUpdateMalwareDomains

if you didn’t get an error

/ip firewall address-list print

and

/ip dns static print

should show many entries. Now you only need to run the script once a hour which following command does:

/system scheduler add interval=1h name=schedulerUpdateMalwareIPs on-event=scriptUpdateMalwareIPs start-date=nov/30/2014 start-time=00:05:00

and

/system scheduler add interval=1h name=schedulerUpdateMalwareDomains on-event=scriptUpdateMalwareDomains start-date=nov/30/2014 start-time=00:10:00

You can use the address list and DNS blacklist now in various ways .. the simplest is following

/ip firewall filter
add chain=forward comment="just the answer packets --> pass" connection-state=established
add chain=forward comment="just the answer packets --> pass" connection-state=related
add action=reject chain=forward comment="no Traffic to malware IP addresses" dst-address-list=addressListMalware log=yes log-prefix=malwareIP out-interface=pppoeDslInternet
add action=reject chain=forward comment="report Traffic to DNS fake IP address" dst-address=10.255.255.255 log=yes log-prefix=malwareDNS out-interface=pppoeDslInternet
add chain=forward comment="everything from internal is ok --> pass" in-interface=InternalInterface

If a clients generates traffic to such DNS names or IP address you’ll get following in your log (and the traffic gets blocked):

20:07:16 firewall,info malwareIP forward: in:xxx out:pppoeDslInternet, src-mac xx:xx:xx:xx:xx:xx, proto ICMP (type 8, code 0), 10.xxx.xxx.xxx->104.xxx.xxx.xxx, len 84

or

20:09:34 firewall,info malwareDNS forward: in:xxx out:pppoeDslInternet, src-mac xx:xx:xx:xx:xx:xx, proto ICMP (type 8, code 0), 10.xxx.xxx.xxx ->10.255.255.255, len 84

ps: The Python script is done in a way that it easily allows you to add also other block lists … e.g. I added the Feodo blocklist from Abuse.ch.

Setup Let’s encrypt TLS certificate for Proxmox [Update]

February 21, 2016

The graphical interface of Proxmox runs on port 8006 and uses HTTPS. By default this is a self signed certificate, which is a problem if you login in from a client the first time. In this case you’re not sure if there is no MitM attack going on. But there is a solution for this by using Let’s encrypt.

First you need to install git

apt-get install git

download the client software

cd /root
git clone https://github.com/letsencrypt/letsencrypt

and now you need to “patch” it as with running containers the check for a free port 80 most likely fails as a container with a running a web server is quite common. Open this file /root/letsencrypt/letsencrypt/plugins/util.py with an editor and place

return False

as first command in the already_listening method. Make sure that it is aligned with the line above and below as Python requires that. It should look this:

letsencrypt

ps: If you started to read this article after calling ./letsencrypt-auto the first time you also need to patch following file the same way.

/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/plugins/util.py

[Update] In the newer version the client is called certbot and so the path got changed to

/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/plugins/util.py

[/Update]

Now make sure that Port 80 is not firewalled and therefore reachable from the Internet and call following command:

cd /root/letsencrypt/
./letsencrypt-auto certonly --standalone --standalone-supported-challenges http-01 -d <dnsname>

Replace with the <dnsname> with the dns name you use to connect to the management Web GUI. If all worked, you should see following:

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at
/etc/letsencrypt/live//fullchain.pem. Your cert
will expire on 2016-05-21. To obtain a new version of the
certificate in the future, simply run Let's Encrypt again.
- If you like Let's Encrypt, please consider supporting our work by:
....

Now only saving the old certificate with these commands …

mv /etc/pve/pve-root-ca.pem /etc/pve/pve-root-ca.pem.orig
mv /etc/pve/local/pve-ssl.key /etc/pve/local/pve-ssl.key.orig
mv /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.pem.orig

and copying the new ones to the correct place …

cp /etc/letsencrypt/live/<hostname>/chain.pem /etc/pve/pve-root-ca.pem
cp /etc/letsencrypt/live/<hostname>/privkey.pem /etc/pve/local/pve-ssl.key
cp /etc/letsencrypt/live/<hostname>/cert.pem /etc/pve/local/pve-ssl.pem

followed by a restart of the processes:

service pveproxy restart
service pvedaemon restart

is open.

The last 5 commands are needed, because the special file system Proxmox uses for /etc/pve does not support symlinks. You need to execute that 5 commands after each renewal. The option for this is “renew”. You can/should automate the renew process.

Tips on how to provide a secure public WiFi hotspot – Part 3

February 4, 2016

This the last part of the the series “tips on how to provide a secure public WiFi hotspot”. In the first two parts we concentrated on the wireless and layer 2 and 3 part, this time we take a look at the application layer stuff you should tune to provide a secure hotspot.

Captive Portal and non HTTP traffic

If you want or need to show the user a website before he is able to surf the web, you should not only think about redirecting HTTP to your landing page.  You should reject (send an ICMP error) for all other TCP connection attempts. Why? Many pages are nowadays HTTPS and letting the user wait for a time-out is not nice. Sure technical minded user will not wait, but others may do. Sure this is not a security advice but an usability one. But it does work nice together with following: Don’t try to provide your own TLS certificate to redirect traffic to the captive portal. Users should not be trained to accept error messages and ignore them and it will give your hotspot an bad look security-wise.

There is just nothing you can do to redirect HTTPS URLs. Just make sure that the captive portal detection of modern operating systems work and so the user is told there is a captive portal.

HTTPS for captive portal

If you require any data from the user except accepting the terms of service your captive portal should be HTTPS.  Get an official DNS name for your portal, e.g. portal.<company>.com and a valid TLS certificate. As there are multiple methods for getting free TLS certificate e.g. startssl or let’s encryt, you should get a real one. One thing that is important to whitelist for your captive portal are the OCSP and CRL URLs of your certificate, issuing CA and root CA. To check for the URLs look at the certificate details for all certificates in the chain of your captive portal. Following images show the 2 places for google.com (you need to check also the Google Internet Authority G2 and GeoTrust Gobal CA in this case.)

crl1

crl2

Why, you ask, you should whitelist these URLs? The browsers will check them before showing your captive portal site. Some/Most browsers will fail open, but some will fail close.

Limit and monitor the DNS

If the user has not past the captive portal his only way though the firewall will be the DNS server. You normally need to resolve the requests to the true IP address to guard against problems with false cached DNS entries after the user past the captive portal. As the user is able to craft the requests as he likes it is possible to send data through to a 3rd party DNS server. This setup can be used as DNS tunnel – it will be not fast but its possible to work through it. As nowadays simple howtos like following and OpenSource tools are provided, consider it easily possible for most power users – its not a expert thing any more.

So now comes the question what you can do against it. In this case you don’t need to block the covert channel of sophisticated malware but just script kiddies that want to tunnel through. Your typical user will only need A and AAAA query types. Tunneling is mostly done via TXT as it allows big packets. Following query types are common:

  • Any
  • A
  • AAAA
  • CNAME
  • MX
  • NS
  • SOA

I won’t go into too much detail, but blocking can be done on the DNS server … e.g. if it is a Windows DNS Server (Allow only certain QTypes) or with the Firewall in this case iptables.

Content Filtering via DNS

Often you’re required to filter certain traffic but don’t want to setup and maintain a transparent proxy or with all the HTTPS it does not help that much any more. An easy way is to use the OpenDNS system, which allows to select certain categories which are then blocked on the DNS level (Query will resolve to an error page) . If you redirect all port 53 traffic to your DNS server which uses the OpenDNS ones as recursive DNS servers you should be good for most cases.

SMTP

Just reject (blocking lets the client run into timeouts) TCP port 25, otherwise infected client systems will flood your bandwidth with SPAM mails.  A mail agent should not send mails to the mail server via port 25 anyway.

I hope this article series helped you and leads to more secure hot spots I’m also able to enjoy. 😉

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 91 queries. 0.208 seconds.