Access Mikrotik Router OS via SSH Public Key authentication

July 12, 2014

Sometimes you need to execute various commands on a Mikrotik automatically from a Server. Surely it is possible to store the password in the script, but there is a better way – it is called Public Key authentication for SSH. Basically a pair of files is generated and the public one is copied to the Mikrotik and the private key stays on the PC. If you encrypt this key on the PC (which is useful if not a script does use  it but a person) you get a 2-factor authentication. An attacker needs that private file and the password to decrypt it to administer the router. There are two types supported by SSH RSA and DSA. RSA is more commonly used but Mikrotik does only support DSA so we need to create a DSA key pair.

The first step is to generate the key pair as the user on the Linux system which is than used to access it. If it is a script it maybe a separate user just for this purpose is a good idea.

$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/<user>/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_dsa.
Your public key has been saved in id_dsa.pub.
The key fingerprint is:
.....

If you just press enter on the file question, the default one will be used. If you want to use some separate directory that’s fine, you just need to provide the location later at the ssh call. If you press just enter for the passphrase the private key will not be encrypted. Now we copy the public key (.pub extension) to the Mikrotik:

scp /home/<user>/.ssh/id_dsa.pub [email protected]:

And after that we need to import the key. If we choose the user admin, which we use our self to login, no password login will be possible anymore for that user. So if you don’t like that you should create a special user for the script. As my script needs only to read stuff I’m okay with the group “read” and create a user like this:

/user add name=scriptUser group=read comment="user for our readonly scripts" disabled=no

Now we import that public key to the scriptUser with following command:

/user ssh-keys import public-key-file=id_dsa.pub user=scriptUser

We’re done .. just testing is open …. if you used not the default directory to store the key files you need to provide them via the -i parameter, if its the default location you don’t need to provide it. This command logs into the router and gets you some basic data without entering a password.

$ ssh -i <pathTo/id_dsa> scriptUser@xxx.xxx.xxx.xxx “/system resource print”

You should also try to login as this user without the key file (e.g. from an other computer) and it should not be possible.

Howto convert Outlook .msg files to normal MIME mails

June 29, 2014

Just got a Microsoft Outlook .msg file and I needed to convert it to something I could read on the console. Its quite simple to convert it to a normal MIME mail .. here for Ubuntu 14.04 but it should be as simple on other distributions:

sudo apt-get install libemail-outlook-message-perl libemail-localdelivery-perl
wget http://www.matijs.net/software/msgconv/msgconvert.pl
perl msgconvert.pl filename.msg

the result file name is filename.msg.mime.

Howto install and use the Burp Suite as HTTPS Proxy on Ubuntu 14.04

June 26, 2014

This article shows you how to intercept and analyze HTTPS traffic. This of course only works if you can add your CA to the client system or the client system does not check the key chain. But you would not believe how many clients, specially embedded devices and mobiles apps don’t check it. The software I use in this post is the Free version of the Burp Suite – from the homepage of the Burp Suite:

Burp Suite is an integrated platform for performing security testing of web applications. Its various tools work seamlessly together to support the entire testing process, from initial mapping and analysis of an application’s attack surface, through to finding and exploiting security vulnerabilities.

Installation

As it is written in Java it is really easy to install and run. Just these 3 commands

sudo apt-get install openjdk-7-jre
wget http://portswigger.net/burp/burpsuite_free_v1.6.jar
sudo java -jar -Xmx2g burpsuite_free_v1.6.jar

I use sudo for the Java process as it needs in my cases to listen on ports lower than 1024 and this requires root permissions. After entering the last command and accepting the EULA you get following window:

burp

Now go to the Proxy | Intercept tab and click onto “Intercept is on” button to disabled it. Otherwise you’ll need to acknowledge every request.

intercept

The proxy is already running but only on localhost, we need it to listen on all to look at traffic from other devices.

burp_interfaces

And in this case I want to see the traffic to a specific host from my mobile, so I set Burp to port 443 and to invisible mode and define a redirect IP address (the original host IP address).

burp_redirect

After this, you only need to set on my local DNS Server the wished host name to my desktop IP address and the traffic runs over it and if the client accepts the faked certificate you can look at the traffic. Which looks e.g. like this:

burp_history

For other devices or requirements it is also possible to use Burp as a HTTP Proxy, just configured it on the client. ARP spoofing is also an idea…..

 

Howto setup a Mikrotik RouterOS with Suricata as IDS

June 1, 2014

Lets say you’ve a Mikrotik router as your internet router and you would like to detect bad traffic that is going over it, so basically you would like to have an IDS (Intrusion detection system). This article shows how you can setup a IDS with a Mikrotik router and Suricata running on a Ubuntu 14.04 (but it runs on any other Linux). This is no high performance setup for 10Gbit links, as in this case you would use a SPAN port a manged switch and you would need to tune the drivers, Linux and Suricata a bit (a lot actually for 10Gbit … and the server hardware is not cheap/small). This setup is for the SOHO (small office home office) space where you use the Mikrotik as your Internet router, be it for Cable or DSL. I use this setup at home and I’ve installed Ubuntu 14.04 64bit Server as virtual machine on my home server. No special switch or hardware is required as we facilitate the TaZmen Sniffer Protocol (TZSP) which is supported by Mikrotik. It is even possible to sent the data over a Layer-3 connection, you just need the bandwidth for the traffic you want to sent to Suricata.

I assume that Ubuntu 14.04 is installed (minimal server install is recommended).

Mikrotik Setup

You just need to configure it to copy the traffic to the Linux server with these commands:

/tool sniffer set filter-stream=yes streaming-enabled=yes streaming-server=xxx.xxx.xxx.xxx
/tool sniffer start

xxx.xxx.xxx.xxx is the IP of the Linux server

Basic Suricata Setup

First we change into the super user mode for every command we execute later:

sudo bash

Now we add the stable Suricata PPA to our system. At the time of writing the stable version is 2.0.1.

add-apt-repository ppa:oisf/suricata-stable
apt-get update
apt-get install suricata oinkmaster

Now we download the open/free Emerging Threats rules for the first tests. (There are also commercial rules available.)

cd /etc/suricata/
wget https://rules.emergingthreatspro.com/open/suricata/emerging.rules.tar.gz
tar -xzf emerging.rules.tar.gz

We use the reference.config from ET:

mv reference.config reference.config.orig
ln -s /etc/suricata/rules/reference.config /etc/suricata/reference.config

And we need to create one file and one directory:

touch /etc/suricata/threshold.config
mkdir /var/log/suricata

In the 2.0.1 packages one file got forgotten, so check if it is also missing in your version and if so download it manually.

ll /etc/suricata/rules/dns-events.rules

If it is missing … do following:

cd /etc/suricata/rules
wget https://raw.githubusercontent.com/inliniac/suricata/master/rules/dns-events.rules

Test Suricata alone

To make the first test call it like this (wait for some minutes):

suricata -c /etc/suricata/suricata.yaml -i eth0

you should get some files in /var/log/suricata

root@nids:/var/log/suricata# ll
total 364
drwxr-xr-x  2 root root     4096 Jun  1 13:45 ./
drwxrwxr-x 10 root syslog   4096 Jun  1 13:32 ../
-rw-r--r--  1 root root    46195 Jun  1 13:51 eve.json
-rw-r--r--  1 root root    25138 Jun  1 13:51 fast.log
-rw-r--r--  1 root root        0 Jun  1 13:36 http.log
-rw-r--r--  1 root root   236014 Jun  1 13:51 stats.log
-rw-r--r--  1 root root     1846 Jun  1 13:36 unified2.alert.1401622567
-rw-r--r--  1 root root    42445 Jun  1 13:51 unified2.alert.1401623113

Test the Mikrotik sniffer stream

Check if you get traffic by first downloading trafr from here and copy it to /usr/local/bin and extract and test it like this:

cd /usr/local/bin/
tar xzf trafr.tgz
./trafr

If you get -bash: ./trafr: No such file or directory, take a look at this post.  Now check with tcpdump if you’re getting any packets (hit CTRL-C to stop)

trafr -s | tcpdump -r - -n

If you don’t get packets something is wrong with the Mikrotik setup or the packets getting filtered/blocked.

Connect Suricata with Mikrotik sniffer stream

I assume that you got packets and are now ready for your first run of Suricata with the Mikrotik sniffer stream. Just enter following command …

trafr -s | suricata -c /etc/suricata/suricata.yaml -r -

.. and open a second console and type following

tail -f /var/log/suricata/fast.log

your should see output like this (this are just examples)

06/01/2014-14:50:22.703188 [**] [1:2012648:3] ET POLICY Dropbox Client Broadcasting [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} xxx.xxx.xxx.xxx:17500 -> 255.255.255.255:17500
06/01/2014-14:57:16.608473 [**] [1:2210021:2] SURICATA STREAM ESTABLISHED retransmission packet before last ack [**] [Classification: (null)] [Priority: 3] {TCP} xxx.xxx.xxx.xxx:443 -> xxx.xxx.xxx.xxx:49503
06/01/2014-15:26:03.601539 [**] [1:2006380:13] ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} xxx.xxx.xxx.xxx:58188 -> xxx.xxx.xxx.xxx:8080

If you got also some entries its good, it works – we need to make it permanent now.

Auto-Update ET Rules

Now that every works we need to configure it in a way to get new rules every night. For this we add following line at the end of /etc/oinkmaster.conf

url = https://rules.emergingthreatspro.com/open/suricata/emerging.rules.tar.gz

To test it we run following command:

oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules

So the config works we create /etc/cron.daily/suricataUpdateRules with following content:

#!/bin/bash
/usr/sbin/oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules |& grep -i "error" > /dev/null
/bin/kill -USR2 `pidof suricata`

After we created it we need to make it executable:

chmod 755 /etc/cron.daily/suricataUpdateRules

Init Script

The last step is to make a init script so everything is started automatically, for this create a file /etc/init/suricata.conf with following content:

# suricata
description "Intruder Detection System Daemon"
start on runlevel [2345]
stop on runlevel [!2345]
expect fork
exec /usr/local/bin/trafr -s | /usr/bin/suricata -c /etc/suricata/suricata.yaml -r - &

Stop the test Suricata command and try it with the init script:

start suricata

You should get something like this:

suricata start/running, process 8003

But to make sure everything is really running, call

ps aux | grep -E "(suri|trafr)"

and you should get following output:

root      8003  0.1  0.0   2020   272 ?        S    14:50   0:00 trafr -s
root      8005  8.8 17.5 841852 360556 ?       Sl   14:50   0:15 suricata -c /etc/suricata/suricata.yaml -r -

And that’s it. Your IDS is running in a basic configuration. You now need to tune your rules and maybe you want to install a Web GUI for your IDS. I can recommend following to look at:

Howto fix -bash: ./trafr: No such file or directory?

Trafr is a program from Mikrotik which receives mirrored traffic from Mikrotik RouterOS systems. The linked blog article shows how to use it with Wireshark. This blog article shows how to get it working if you have following problem:

root@nids:/usr/local/bin# ll
total 12
drwxr-xr-x 2 root root 4096 Jun 1 13:59 ./
drwxr-xr-x 10 root root 4096 Jun 1 12:09 ../
-rw-r--r-- 1 root root 2629 Jun 1 13:53 trafr.tgz
root@nids:/usr/local/bin# tar xzf trafr.tgz
root@nids:/usr/local/bin# ll
total 20
drwxr-xr-x 2 root root 4096 Jun 1 13:59 ./
drwxr-xr-x 10 root root 4096 Jun 1 12:09 ../
-rwxr-xr-x 1 1003 root 4764 Mär 17 2004 trafr*
-rw-r--r-- 1 root root 2629 Jun 1 13:53 trafr.tgz
root@nids:/usr/local/bin# ./trafr
-bash: ./trafr: No such file or directory

The problem is that the trafr program is a 32 bit application and you’re trying this on a 64bit OS without 32bit compatibility libraries. To verify that you can use following command:

file ./trafr
./trafr: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.0, stripped

On Ubuntu 14.04 you can that simple by doing following:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install libc6:i386

and you get

sudo ./trafr
usage: trafr <file | -s> [ip_addr]
-s write output to stdout. pipe it into tcpdump for example:
./trafr -s | /usr/sbin/tcpdump -r -
ip_addr use to filter one source router by ip address

For older Ubuntu Version following helps (didn’t check at which version it changed):

apt-get install ia32-libs

For other distribution its similar. e.g. for CentOS6 or RHEL6 its

sudo yum install glibc.i686

Howto filter “No VR found on VLAN xxx with VR Id xxx” on Extreme XOS switches

May 25, 2014

If your Extreme Networks switches are using VRRP and other devices are using it also in the same VLAN, the Exterme XOS switches will complain loudly about that … one log line per broadcast. In my case it were two per second and as the switch stores only 1000 log lines .. the log soon contained only these entries:

05/22/2014 17:29:41.11 Slot-1: No VR found on VLAN xxx with VR Id xxx
05/22/2014 17:29:40.48 Slot-2: No VR found on VLAN xxx with VR Id xxx
05/22/2014 17:29:40.11 Slot-1: No VR found on VLAN xxx with VR Id xxx
05/22/2014 17:29:39.48 Slot-2: No VR found on VLAN xxx with VR Id xxx
05/22/2014 17:29:39.11 Slot-1: No VR found on VLAN xxx with VR Id xxx

The one pitfall with using the exclude match string variable is that the VRIDs must not be treated as string variable. This does not work because Extreme XOS does not treat the VRID as a string variable, but rather as a integer. To determine the valid variables available for the specific event you’ll need to type following:

Slot-1 xxxxxxx.1 # show log events "VRRP.UnkVR" details
Component   SubComponent Condition               Severity      Parameters
----------- ------------ ----------------------- ------------- ----------
VRRP                     UnkVR                   Warning        2 Total
0 - string
1 - number (32-bit unsigned int)
No VR found on VLAN %0% with VR Id %1%

This tells us that to filter on the VRRP.UnkVR messages, there is a string variable (%0%) equal to the VLAN name, and a integer (%1%) equal to the VRID itself. Because Extreme XOS interprets the VRID itself as a number and not a string, doing an exclude match string will not work. You must use the number variable as follows:

configure log filter "DefaultFilter" add exclude events "VRRP.UnkVR" match number xxx

From the Concept Guide:

The filter can be associated with one or more targets using the command to control the messages sent to those targets. The system has one built-in filter named DefaultFilter, which itself may be customized. Therefore, the if a filter other than DefaultFilter is desired. As its name implies, DefaultFilter initially contains the default level of logging in which every Extreme XOS component and subcomponent has a pre-assigned severity level.

PS: You can use this solution to filter out any other event, just check with show log events "xxxx" details

How to configure SNMPv3 securely on Mikrotik RouterOS [Update]

May 11, 2014

In the last post I wrote on how to configure SNMPv3 for CentOS/RHEL/SL. I thought it might be a good idea to post the SNMPv3 configuration for various systems/devices – so this post is about Mikrotik RouterOS.

This time it is even easier than on Linux, just one line:

/snmp community set [ find default=yes ] name=snmpv3user security=private authentication-password=snmpv3authPass authentication-protocol=SHA1 encryption-password=snmpv3encPass encryption-protocol=DES read-access=yes write-access=no addresses=10.0.0.0/24

If you want to keep the default SNMP configuration and add just another, use this:

/snmp community add name=snmpv3user security=private authentication-password=snmpv3authPass authentication-protocol=SHA1 encryption-password=snmpv3encPass encryption-protocol=DES read-access=yes write-access=no addresses=10.0.0.0/24

But the RouterOS has also one weakness, it only supports DES and not AES. So the test command looks like this:

snmpwalk -u snmpv3user -A snmpv3authPass -a SHA -X snmpv3encPass -x DES -l authPriv 10.7.7.1 -v3

If you now look at the answer of a request in Wireshark you’ll only see encrypted text:

snmpv3answer

Update: Starting with RouterOS 6.16rc17 (2014-Jul-09 09:52) AES encryption (rfc3826) for SNMP is supported according to the changelog.

How to configure SNMPv3 securely in CentOS/RHEL/SL

May 4, 2014

Looking through the Internet I found more often than not insecure SNMP setups explained in Howtos, maybe the authors where happy to got running in the first place – don’t know. The Simple Network Management Protocol (SNMP) is used to monitor (and configure in the case of network equipment) systems via the network in a standardized way. There are 3 versions of it in the wild:

Version 1

The first version, it is really old and only uses community strings to “protect” access. If you look with Wireshark what the packet for the request contains you’ll see following:

snmpv1

Yes, thats the community in clear text …. so don’t use it, if you must not. If you must (old devices which supports only this) use it on a secure network and make sure that the device only accepts quests from a given source IP address.

Version 2c

Not better in any security related way. The e.g. traffic counters are just bigger, as on fast interfaces it was possible with v1 that the counter rotated more than ones within the interval you queried so you didn’t know how many bytes really got transfered over the link.

Version 3

Now we’re talking. This versions supports authentication passwords, that get not transfered in the clear and it also supports encryption passwords so a attacker does not see what is transfered. I’ll show you now how you configure your CentOS/RHEL/SL to let it be queried via SNMPv3 in a secure way.

  1. We need to install the package we need:
    yum install net-snmp

  2. We create our user. SHA1 is used for the authentication and AES for the encryption of the traffic. And for security it is important to use two separated passwords.
    service snmpd stop
    net-snmp-create-v3-user -ro -A snmpv3authPass -a SHA -X snmpv3encPass -x AES snmpv3user
    service snmpd start

    This will print something like this

    adding the following line to /var/lib/net-snmp/snmpd.conf:
    createUser snmpv3user SHA "snmpv3authPass" AES snmpv3encPass
    adding the following line to /etc/snmp/snmpd.conf:
    rouser snmpv3user

  3. Secure the SNMP daemon with a host firewall and only allow requests from source IP addresses you know and which need to query the system.
  4. Verify that everything works with following command:
    snmpwalk -u snmpv3user -A snmpv3authPass -a SHA -X snmpv3encPass -x AES -l authPriv 127.0.0.1 -v3

So as you see it is quite easy, so there is now reason to use anything less secure!

If you want to delete the user, just edit the two files mentioned above. In /etc/snmp/snmpd.conf delete the line rouser snmpv3user and in /var/lib/net-snmp/snmpd.conf look for a line like this and remove it:

usmUser 1 3 0x80001f88801e15113fbf17bf5100000000 0x736e6d7076337573657200 0x736e6d7076337573657200 NULL .1.3.6.1.6.3.10.1.1.3 0xaaa3bc2aa941d932a581bbb11956699a7d807238 .1.3.6.1.6.3.10.1.2.4 0xcd6eed84f9d1aa903e92162f0e3fcb26 ""

restart the daemon with service snmpd restart and the user is gone.

I get only a A- on Qualys SSL Labs tests – Why? and what can I do?

April 19, 2014

In the last months more and more sysadmins started looking into their SSL configuration of their HTTPS websites. And one of the major sites that is used to rate/check the quality of the SSL configuration on a given HTTPS server is the Qualys SSL Labs SSL Server Test which can be reached via this link. If a sysadmin gets a not so good rating he search through the Internet and uses something like this settings (Apache 2.2 on Centos 6) to fix it:

SSLEngine on
SSLProtocol All -SSLv2 -SSLv3
Header add Strict-Transport-Security "max-age=15768000"
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4
SSLHonorCipherOrder On

This leads at the time of writing (Quality SSL Labs changes the rating from time to time to following) to:

ssl_rating

And now you are wondering why you get only a A- and what the problem with your configuration is. To make your journey shorter, the problem is most likely not the SSL configuration, it is the software you’re running.  As you see on the screenshot the test reports that Forward Secrecy is not supported by all browsers and if you take a look at the details,

pfs

you’ll see that the problem is the Internet Explorer and that Forward Secrecy works for all other browsers.

(Perfect) Forward Secrecy

But what is (Perfect) Forward Secrecy in the first plage and why should you care.  PFS ensures the integrity of a session key in the event that the private key of the server gets compromised. This is done by generating a separate session key for every new HTTPS session/connection.

Why should you care?

An attacker could record the SSL traffic for some time and later he got the private key and now without PFS he would be able to extract all the SSL traffic he was not able to look into before.  Basically without PFS if a private key gets compromised you not only need to look the now and the future but also a the past and consider everything that was encrypted/signed by this key as compromised. With PFS you’re sure that an attacker is not able to extract data from before he got the private key. With the Heartbleed Bug in OpenSSL such an attack was possible or by hacking the server.

The cipher suites (the ones you choose with SSLCipherSuite in the Apache configuration) that provide Perfect Forward Secrecy are those that use an ephemeral form of the Diffie-Hellman key exchange. The disadvantage of them is that they have a performance overhead, but the security is worth it and it is not that much overhead. With some elliptic curve variants the performance would be better.

A- and the workarounds

And how to the problem with the A- rating  – I’ll quote Shadow Zhang who described it nicely in his post:

With Apache 2.2.x you have only DHE suites to work with, but they are not enough. Internet Explorer (in all versions) does not support the required DHE suites to achieve Forward Secrecy. (Unless youre using DSA keys, but no one does; that’s a long story.) Apache does not support configurable DH parameters in any version, but there are patches you could use if you can install from source. Even if openssl can provide ECDHE the apache 2.2 in debian stable does not support this mechanism. You need apache 2.4 to fully support forward secrecy.

Patching and compiling Apache is not the best idea, as you need to do it again for every security update. I see following options:

  • Use a distribution version which supports Apache 2.4
  • Use Nginx as reverse proxy in front of the Apache because it fully supports ECDHE.
  • Change to a web server that is shipped with your distribution and that does support ECDHE.

I hope this post helped and saved you some time looking through the internet for a solution.

Howto setup a redundant and secure BGP (full table) Internet connection with Mikrotik Routers

March 22, 2014

Looking through the Internet, there are much howto’s specially in the OpenSource field but a guide line for a redundant and secure internet connection based on BGP (full table) is not something you find on many sites. So I thought I write such a documentation and I’m hoping it helps some networks admins in setting up their company internet connection. BGP is not that hard ;-).

General conditions

Following points are the general conditions for this howto:

  • Two Internet Uplinks to two different providers, each connected via one fibre link
  • One provides the BGP peer in the same VLAN and one peer is only reachable via a routing hop (to show the different configuration)
  • One provider hands the customer only one peering IP address and the other two (to show the different configuration)
  • We use 2 BGP routers on our side for redundancy
  • Both provide IPv4 and IPv6 Full Tables
  • No traffic engineering to steer traffic to one provider over the other is done
  • A failure of
    • one router must not change anything for the user/customer
    • one switch is allowed to lose one Uplink but not both, so traffic for the user/customers needs to be unaffected
    • one fibre link leads to one Uplink down, but the traffic for the user/customers needs to be unaffected
  • Secure setup
  • Setting up the layer 2 switches and the redundant firewall behind the routers is not part of this howto
  • Using Mikrotik RouterOS devices as the routers in the config part, but the same setup would also work with Cisco or Vayatta routers, which I’ve also used for BGP based Internet connections.

 

Setup

Following drawing shows the setup for the BGP Internet connection.

bgp_logical_blogpost-03

As you see I’m using 2 switches as media converters and to distribute the provider transit networks to both routers. Why I do this as there are Mikrotik routers with SPF and SPF+ modules? First using a Mikrotik on a x86 provides you with  no switching (just bridging). Secondly even if you use a Mikrotik Hardware router with switching support, a switch that is only used for layer 2 stuff and has no IP interface in the public networks (only in the management network) will be more stable specially concerning firmware updates than routers which are used for active interaction with other systems. No update for multiple years is not uncommon for switches in this scenario, which is not valid for the routers, specially if you use some special features on the routers. This means you can update a router without the Ethernet link to the provider going down and as the Mikrotik boots under 30 seconds its a minimal impact. The default switching time for BGP is 180 seconds (3*60 seconds) which is much longer than a boot after a firmware update.

Configuration of the routers

If not specified the configuration is the same for both routers and the syntax works with RouterOS 6.10, but it does not change that much normally, at least not since version 4 when I started using Mikrotiks.

First we start with the names of the routers

BGP1:
/system identity set name=bgp1

BGP2:
/system identity set name=bgp2

And now to the actual work – we need to configure our interfaces. We create a loopback interface for at least following reasons:

  • This interface is always up, so the IP address is always up – good for monitoring the node vs interfaces
  • We use the IP address on this interface as our OSFP and BGP ID
  • We use it to blackhole routed traffic .. more later in this post

/interface bridge add name=loopback
/interface ethernet
set [ find default-name=ether1 ] name=ether1vlanTransitProvider2
set [ find default-name=ether2 ] name=ether2vlanCrossConnection
set [ find default-name=ether3 ] name=ether3vlanTransitFirewall
set [ find default-name=ether4 ] name=ether4vlanMgmt
set [ find default-name=ether5 ] name=ether5vlanTransitProvider1

As Mikrotik allows to rename the interface we do so as it makes configuration lines which use these interfaces much easier to understand … believe me I’ve routers with > 100 interfaces :-). For the transit network to the firewall we’ll setup a VRRP and to be somewhat more secure than normal VRRP we also set a long and random password. We configure also a no default VRID, as most system use 1 as default and who knows what the firewalls use. ;-)

BGP1:
/interface vrrp add interface=ether3vlanTransitFirewall name=vrrpTransitFirewall password=XXXXXXX priority=250 vrid=10

BGP2:
/interface vrrp add interface=ether3vlanTransitFirewall name=vrrpTransitFirewall password=XXXXXXX priority=200 vrid=10

So if the BGP1 is up, it always will be the master. Now we need to configure the IP addresses …. lets start with IPv4

BGP1:
/ip address
add address=1.0.0.244/32 interface=loopback
add address=3.0.5.11/29 interface=ether1vlanTransitProvider2
add address=1.0.0.241/30 interface=ether2vlanCrossConnection
add address=1.0.0.250/29 interface=ether3vlanTransitFirewall
add address=1.0.0.249/29 interface=vrrpTransitFirewall
add address=10.0.0.1/24 interface=ether4vlanMgmt
add address=2.0.5.11/29 interface=ether5vlanTransitProvider1

BGP2:
/ip address
add address=1.0.0.245/32 interface=loopback
add address=3.0.5.12/29 interface=ether1vlanTransitProvider2
add address=1.0.0.242/30 interface=ether2vlanCrossConnection
add address=1.0.0.251/29 interface=ether3vlanTransitFirewall
add address=1.0.0.249/29 interface=vrrpTransitFirewall
add address=10.0.0.2/24 interface=ether4vlanMgmt
add address=2.0.5.12/29 interface=ether5vlanTransitProvider1

And now we do the same for IPv6 … just the internal management is kept IPv4 only, as you don’t need it there normally (at least I’m not)

BGP1:
/ipv6 address
add address=2001:1::1/64 interface=loopback
add address=2001:3:3::11/64 interface=ether1vlanTransitProvider2
add address=2001:1:2::1/64 interface=ether2vlanCrossConnection
add address=2001:1:3::2/64 interface=ether3vlanTransitFirewall
add address=2001:1:3::1/64 interface=vrrpTransitFirewall
add address=2001:2:3::11/64 interface=ether5vlanTransitProvider1

BGP2:
/ipv6 address
add address=2001:1:1::1/64 interface=loopback
add address=2001:3:3::12/64 interface=ether1vlanTransitProvider2
add address=2001:1:2::2/64 interface=ether2vlanCrossConnection
add address=2001:1:3::3/64 interface=ether3vlanTransitFirewall
add address=2001:1:3::2/64 interface=vrrpTransitFirewall
add address=2001:2:3::12/64 interface=ether5vlanTransitProvider1

Now we add our static routes we need. We need to set one for our management network, so we can be reached via the admin computers and set the route for provider 1 as the BGP routers are not in the same subnet. Also the router to the firewalls for our internal network is clear, but we need one more feature which needs some explaining. If the link to the firewalls goes down on a router, the IP address / network also goes down and its routes over this interface. As the router redistributes the connected and static routes via BGP it will not anymore send it out.  This is basically ok, but now something comes into play that is called “BGP Route Flap Damping“, which can lead to the problem that everything is running again but some AS are not setting traffic to you for some time. So it is paramount to keep the announcing running as stable as possible, which leads us to black hole routes. As in IPv6 Mikrotik does not support it (as of yet) we use a workaround to accomplish the same. PS: you can use the same to black hole an attacker .. really fast and without much load on the system … just saying :-)

/ip route
add distance=1 dst-address=10.0.0.0/8 gateway=10.0.0.254
add distance=1 dst-address=2.0.1.1/32 gateway=2.0.5.10
add distance=1 dst-address=2.0.2.1/32 gateway=2.0.5.10
add distance=1 dst-address=1.0.0.0/22 gateway=1.0.0.254
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=1.0.0.0/22 type=blackhole

/ipv6 route
add distance=1 dst-address=2001:2:1::1/128 gateway=2001:2:3::1
add distance=1 dst-address=2001:2:2::1/128 gateway=2001:628:1400:1003::1
add distance=1 dst-address=2001:1::0/48 gateway=2001:1:3::10

BGP1:
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=2001:1::0/48 gateway=2001:1::ffff

BGP2:
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=2001:1::0/48 gateway=2001:1:1::1:ffff

After the IP addresses and static routes are configured we need to secure our setup before doing anything else. As the BGP routers are in front of the firewalls they can get attacked directly from the Internet, sure, but traffic (e.g. attacks, P2P, …) to systems behind it can also make problems for the routers, so we’ll do something that we normally don’t do. We’ll disable connection tracking – we are a plain and stupid router … let the firewall track connections, we don’t care. This takes much work from the router if you’ve many many connections over it. Sure it makes the firewall settings on the router harder but as said, let the router focus on its single task – route traffic as much and as fast as possible. I sometimes see BGP routers overloaded with other tasks and than people complain that they have problems with high loads. If your network/uplinks is so small, that it does not matter, sticking with connection tracking is also ok – you’ll just can change the firewall rules to use connection awareness.

/ip firewall connection tracking set enabled=no
/ip settings set tcp-syncookies=yes

Now we create a address list of our BGP peers which we will allow to connect to our BGP daemon. Don’t forget your routers, as they talk also between themselves:

/ip firewall address-list
add address=2.0.1.1 list=listBgpIPv4Peers
add address=2.0.2.1 list=listBgpIPv4Peers
add address=3.0.5.10 list=listBgpIPv4Peers
add address=1.0.0.241 list=listBgpIPv4Peers
add address=1.0.0.242 list=listBgpIPv4Peers

/ipv6 firewall address-list
add address=2001:2:1::1 list=listBgpIPv6Peers
add address=2001:2:2::1 list=listBgpIPv6Peers
add address=2001:3:3::1 list=listBgpIPv6Peers
add address=2001:1:2::1 list=listBgpIPv6Peers
add address=2001:1:2::2 list=listBgpIPv6Peers

And now to actual firewall rules:

/ip firewall filter
add chain=input comment="BGP incomming is ok on all interfaces from our peers" src-address-list=listBgpIPv4Peers dst-port=179 protocol=tcp
add chain=input comment="without conntrack we need to allow that" dst-port=1024-65535 protocol=tcp src-address-list=listBgpIPv4Peers
add chain=input comment="OSFP is on the crosslink ok" in-interface=ether2vlanCrossConnection protocol=ospf
add chain=input comment="VRRP is ok on the interface to the firewalls" dst-address=224.0.0.18 in-interface=ether3vlanTransitFirewall protocol=vrrp
add chain=input comment="everyone can ping us" protocol=icmp
add action=drop chain=input comment="we drop any request from not from the Mgmt Interface" in-interface=!ether4vlanMgmt

/ipv6 firewall filter
add chain=input comment="BGP incomming is ok on all interfaces from our peers" src-address-list=listBgpIPv4Peers dst-port=179 protocol=tcp
add chain=input comment="without conntrack we need to allow that" dst-port=1024-65535 protocol=tcp src-address-list=listBgpIPv6Peers
add chain=input comment="OSFP is on the crosslink ok" in-interface=ether2vlanCrossConnection protocol=ospf
add chain=input comment="VRRP is ok on the interface to the switches" dst-address=ff02::12/128 in-interface=ether3vlanTransitFirewall protocol=vrrp
add chain=input comment="everyone can ping us" protocol=icmpv6
add action=drop chain=input comment="we drop any request from not from the Mgmt Interface" in-interface=!ether4vlanMgmt

Looks like a secure setup .. hopefully it also is :-). Now we’re ready to configure the BGP part, starting with the configuration of the instance.

BGP1:
/routing bgp instance set default as=1000 redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes router-id=1.0.0.244

BGP2:
/routing bgp instance set default as=1000 redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes router-id=1.0.0.245

Now we need to set our networks to announce:

/routing bgp network
add network=1.0.0.0/22
add network=2001:1::0/48

And now we configure our peers. For the 2 BGP routers which are reachable only via an other router we need to set multihop to yes. We need also to make a link between our 2 routers if one sees a peer the other does not but he still is the the VRRP master.

/routing bgp peer

BGP1:
add in-filter=filterIpv4AS2000in multihop=yes name=p1_bgp1 out-filter=filterIpv4GLOBALout remote-address=2.0.1.1 remote-as=2000 tcp-md5-key=xxxxxxxx
add in-filter=filterIpv4AS3000in name=p2_bgp1 out-filter=filterIpv4GLOBALout remote-address=3.0.5.10 remote-as=3000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS2000in multihop=yes name=p1_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:2:1::1 remote-as=2000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS3000in name=p2_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:3:3::1 remote-as=3000 tcp-md5-key=xxxxxxx
add name=bgp2 remote-address=1.0.0.242 remote-as=1000
add name=bgp2ipv6 remote-address=2001:1:2::2 remote-as=1000

BGP2:
add in-filter=filterIpv4AS2000in multihop=yes name=p1_bgp2 out-filter=filterIpv4GLOBALout remote-address=2.0.2.1 remote-as=2000 tcp-md5-key=xxxxxxxx
add in-filter=filterIpv4AS3000in name=p2_bgp1 out-filter=filterIpv4GLOBALout remote-address=3.0.5.10 remote-as=3000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS2000in multihop=yes name=p1_bgp2 out-filter=filterIpv6GLOBALout remote-address=2001:2:2::1 remote-as=2000 tcp-md5-key=xxxxx
add address-families=ipv6 in-filter=filterIpv6AS3000in name=p2_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:3:3::1 remote-as=3000 tcp-md5-key=xxxxxxx
add name=bgp1 remote-address=1.0.0.241 remote-as=1000
add name=bgp1ipv6 remote-address=2001:1:2::1 remote-as=1000

That was not that hard, but what are all this filter names? As I told you in the beginning we’re paranoid so we don’t trust anyone so we’re filtering all routes going in and out.  So lets start with the out filters as they are much easier. They just let us announce our own networks, so we won’t account networks of the one provider to the other and therefore make a link for them over us.

/routing filter
add action=accept chain=filterIpv4GLOBALout prefix=1.0.0.0/22
add action=discard chain=filterIpv4GLOBALout

add action=accept chain=filterIpv6GLOBALout prefix=2001:1::0/48
add action=discard chain=filterIpv6GLOBALout

The in filters are at little bit more complicated, but not that hard. We make sure that every AS path we get from the provider starts with his AS. It had happened that some provider are a little bit messy there.

add action=jump chain=filterIpv4AS2000in jump-target=filterIpv4Nomartians
add action=accept bgp-as-path="^2000(,[0-9]+)*\$" chain=filterIpv4AS2000in
add action=accept chain=filterIpv4AS2000in

add action=jump chain=filterIpv4AS3000in jump-target=filterIpv4Nomartians
add action=accept bgp-as-path="^3000(,[0-9]+)*\$" chain=filterIpv4AS3000in
add action=accept chain=filterIpv4AS3000in

add action=jump chain=filterIpv6AS2000in jump-target=filterIpv6Nomartians
add action=accept bgp-as-path="^2000(,[0-9]+)*\$" chain=filterIpv6AS2000in
add action=accept chain=filterIpv6AS2000in

add action=jump chain=filterIpv6AS3000in jump-target=filterIpv6Nomartians
add action=accept bgp-as-path="^3000(,[0-9]+)*\$" chain=filterIpv6AS3000in
add action=accept chain=filterIpv6AS3000in

After this is clear, I only need to explain the reason for the filterIpv4Nomartians and filterIpv6Nomartians filters. Its quite easy, these lists contain IP subnets that we should not get via BGP, because they are not used on the Internet (at least not by good people) so we’ll filter them.

add action=discard chain=filterIpv4Nomartians prefix=0.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=127.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=192.0.2.0/24
add action=discard chain=filterIpv4Nomartians prefix=10.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=172.16.0.0/12
add action=discard chain=filterIpv4Nomartians prefix=192.168.0.0/16
add action=discard chain=filterIpv4Nomartians prefix=192.168.0.0/15
add action=discard chain=filterIpv4Nomartians prefix=168.254.0.0/16
add action=discard chain=filterIpv4Nomartians prefix=240.0.0.0/4
add action=return chain=filterIpv4Nomartians

add action=discard chain=filterIpv6Nomartians prefix=::/96
add action=discard chain=filterIpv6Nomartians prefix=::/128
add action=discard chain=filterIpv6Nomartians prefix=::1/128
add action=discard chain=filterIpv6Nomartians prefix=::ffff:0.0.0.0/96
add action=discard chain=filterIpv6Nomartians prefix=::224.0.0.0/100
add action=discard chain=filterIpv6Nomartians prefix=::/104
add action=discard chain=filterIpv6Nomartians prefix=::255.0.0.0/104
add action=discard chain=filterIpv6Nomartians prefix=::/8
add action=discard chain=filterIpv6Nomartians prefix=200::/7
add action=discard chain=filterIpv6Nomartians prefix=3ffe::/16
add action=discard chain=filterIpv6Nomartians prefix=2001:db8::/32
add action=discard chain=filterIpv6Nomartians prefix=2002:e000::/20
add action=discard chain=filterIpv6Nomartians prefix=2002:7f00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:ff00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:a00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:ac10::/28
add action=discard chain=filterIpv6Nomartians prefix=2002:c0a8::/32
add action=discard chain=filterIpv6Nomartians prefix=fc00::/7
add action=discard chain=filterIpv6Nomartians prefix=fe80::/10
add action=discard chain=filterIpv6Nomartians prefix=fec0::/10
add action=discard chain=filterIpv6Nomartians prefix=ff00::/8
add action=return chain=filterIpv6Nomartians

Now we’re done with the BGP setup, only some OSFP stuff is left open. Why OSFP? We want to reach our loopback interfaces via the other router, as only one can be the VRRP master. BGP will only redistribute our complete network and the networks from the provides between our 2 routers, but not some parts of our networks – for this we need OSFP.

BGP1:
/routing ospf instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.244
/routing ospf-v3 instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.244

BGP2:
/routing ospf instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.245
/routing ospf-v3 instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.245

If you wonder why we use an IPv4 address for the OSFPv3, its because even if its an IPv6 protocol no IPv6 address can be used there … its more like an ID field. Now we only need to set our interfaces and network (only for IPv4 needed):

/routing ospf interface
add interface=loopback network-type=point-to-point passive=yes
add interface=ether3vlanTransitFirewall network-type=point-to-point
add interface=ether2vlanCrossConnection network-type=point-to-point
/routing ospf network
add area=backbone network=1.0.0.240/30
add area=backbone network=1.0.0.248/30
/routing ospf-v3 interface
add area=backbone interface=loopback network-type=point-to-point passive=yes
add area=backbone interface=ether3vlanTransitFirewall network-type=point-to-point
add area=backbone interface=ether2vlanCrossConnection network-type=point-to-point

Basically we’re done …. just some standard setups I’m setting on any Mikrotik and recommend you to set it also:

Change the SNMP Community to something long and not guessable:

/snmp
set contact="Robert Penz" enabled=yes location="datacenter" trap-community=xxxxxxxx trap-generators=interfaces trap-target=10.x.x.x trap-version=2
/snmp community set [ find default=yes ] name=XXXXXXXXXX

Set the clock to the correct timezone and set a NTP server as otherwise the log entries are hard to read:

/system clock set time-zone-name=Europe/Vienna
/system ntp client set enabled=yes mode=unicast primary-ntp=10.x.x.x secondary-ntp=10.x.x.x

Configure a syslog server to have some logs if a Mikrotik booted:

/system logging
add action=remote topics=info
add action=remote topics=error
add action=remote topics=warning
add action=remote topics=critical
/system logging action set 3 bsd-syslog=yes remote=10.x.x.x src-address=<ip of the mikrotik ether4vlanMgmt>

Setting the internal DNS Servers is also a good idea:

/ip dns set servers=10.x.x.x,10.x.x.x

Disable some Services on the Mikrotiks we don’t need:

/ip service
set telnet address=0.0.0.0/0 disabled=yes
set ftp address=0.0.0.0/0 disabled=yes
set www address=0.0.0.0/0 disabled=yes
set winbox address=0.0.0.0/0 disabled=yes

And at last we send traffic samples to our SFlow server …. I would recommend you to have also a good SFlow server for your BGP routers.

/ip traffic-flow target add address=10.x.x.x:9996 version=9

Now you could test your routers, but one last thing I recommend you to install on your router is following script written by MarkB. With one command you get something that looks like show ip bgp summary on Cisco or Vayatta and that makes looking at the BGP stuff much easier on a Mikrotik. Get the script from here.

Any questions or improvement ideas on this setup/howto?

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 24 queries. 0.206 seconds.