Howto setup a redundant and secure BGP (full table) Internet connection with Mikrotik Routers

March 22, 2014

Looking through the Internet, there are much howto’s specially in the OpenSource field but a guide line for a redundant and secure internet connection based on BGP (full table) is not something you find on many sites. So I thought I write such a documentation and I’m hoping it helps some networks admins in setting up their company internet connection. BGP is not that hard ;-).

General conditions

Following points are the general conditions for this howto:

  • Two Internet Uplinks to two different providers, each connected via one fibre link
  • One provides the BGP peer in the same VLAN and one peer is only reachable via a routing hop (to show the different configuration)
  • One provider hands the customer only one peering IP address and the other two (to show the different configuration)
  • We use 2 BGP routers on our side for redundancy
  • Both provide IPv4 and IPv6 Full Tables
  • No traffic engineering to steer traffic to one provider over the other is done
  • A failure of
    • one router must not change anything for the user/customer
    • one switch is allowed to lose one Uplink but not both, so traffic for the user/customers needs to be unaffected
    • one fibre link leads to one Uplink down, but the traffic for the user/customers needs to be unaffected
  • Secure setup
  • Setting up the layer 2 switches and the redundant firewall behind the routers is not part of this howto
  • Using Mikrotik RouterOS devices as the routers in the config part, but the same setup would also work with Cisco or Vayatta routers, which I’ve also used for BGP based Internet connections.

 

Setup

Following drawing shows the setup for the BGP Internet connection.

bgp_logical_blogpost-03

As you see I’m using 2 switches as media converters and to distribute the provider transit networks to both routers. Why I do this as there are Mikrotik routers with SPF and SPF+ modules? First using a Mikrotik on a x86 provides you with  no switching (just bridging). Secondly even if you use a Mikrotik Hardware router with switching support, a switch that is only used for layer 2 stuff and has no IP interface in the public networks (only in the management network) will be more stable specially concerning firmware updates than routers which are used for active interaction with other systems. No update for multiple years is not uncommon for switches in this scenario, which is not valid for the routers, specially if you use some special features on the routers. This means you can update a router without the Ethernet link to the provider going down and as the Mikrotik boots under 30 seconds its a minimal impact. The default switching time for BGP is 180 seconds (3*60 seconds) which is much longer than a boot after a firmware update.

Configuration of the routers

If not specified the configuration is the same for both routers and the syntax works with RouterOS 6.10, but it does not change that much normally, at least not since version 4 when I started using Mikrotiks.

First we start with the names of the routers

BGP1:
/system identity set name=bgp1

BGP2:
/system identity set name=bgp2

And now to the actual work – we need to configure our interfaces. We create a loopback interface for at least following reasons:

  • This interface is always up, so the IP address is always up – good for monitoring the node vs interfaces
  • We use the IP address on this interface as our OSFP and BGP ID
  • We use it to blackhole routed traffic .. more later in this post

/interface bridge add name=loopback
/interface ethernet
set [ find default-name=ether1 ] name=ether1vlanTransitProvider2
set [ find default-name=ether2 ] name=ether2vlanCrossConnection
set [ find default-name=ether3 ] name=ether3vlanTransitFirewall
set [ find default-name=ether4 ] name=ether4vlanMgmt
set [ find default-name=ether5 ] name=ether5vlanTransitProvider1

As Mikrotik allows to rename the interface we do so as it makes configuration lines which use these interfaces much easier to understand … believe me I’ve routers with > 100 interfaces :-). For the transit network to the firewall we’ll setup a VRRP and to be somewhat more secure than normal VRRP we also set a long and random password. We configure also a no default VRID, as most system use 1 as default and who knows what the firewalls use. ;-)

BGP1:
/interface vrrp add interface=ether3vlanTransitFirewall name=vrrpTransitFirewall password=XXXXXXX priority=250 vrid=10

BGP2:
/interface vrrp add interface=ether3vlanTransitFirewall name=vrrpTransitFirewall password=XXXXXXX priority=200 vrid=10

So if the BGP1 is up, it always will be the master. Now we need to configure the IP addresses …. lets start with IPv4

BGP1:
/ip address
add address=1.0.0.244/32 interface=loopback
add address=3.0.5.11/29 interface=ether1vlanTransitProvider2
add address=1.0.0.241/30 interface=ether2vlanCrossConnection
add address=1.0.0.250/29 interface=ether3vlanTransitFirewall
add address=1.0.0.249/29 interface=vrrpTransitFirewall
add address=10.0.0.1/24 interface=ether4vlanMgmt
add address=2.0.5.11/29 interface=ether5vlanTransitProvider1

BGP2:
/ip address
add address=1.0.0.245/32 interface=loopback
add address=3.0.5.12/29 interface=ether1vlanTransitProvider2
add address=1.0.0.242/30 interface=ether2vlanCrossConnection
add address=1.0.0.251/29 interface=ether3vlanTransitFirewall
add address=1.0.0.249/29 interface=vrrpTransitFirewall
add address=10.0.0.2/24 interface=ether4vlanMgmt
add address=2.0.5.12/29 interface=ether5vlanTransitProvider1

And now we do the same for IPv6 … just the internal management is kept IPv4 only, as you don’t need it there normally (at least I’m not)

BGP1:
/ipv6 address
add address=2001:1::1/64 interface=loopback
add address=2001:3:3::11/64 interface=ether1vlanTransitProvider2
add address=2001:1:2::1/64 interface=ether2vlanCrossConnection
add address=2001:1:3::2/64 interface=ether3vlanTransitFirewall
add address=2001:1:3::1/64 interface=vrrpTransitFirewall
add address=2001:2:3::11/64 interface=ether5vlanTransitProvider1

BGP2:
/ipv6 address
add address=2001:1:1::1/64 interface=loopback
add address=2001:3:3::12/64 interface=ether1vlanTransitProvider2
add address=2001:1:2::2/64 interface=ether2vlanCrossConnection
add address=2001:1:3::3/64 interface=ether3vlanTransitFirewall
add address=2001:1:3::2/64 interface=vrrpTransitFirewall
add address=2001:2:3::12/64 interface=ether5vlanTransitProvider1

Now we add our static routes we need. We need to set one for our management network, so we can be reached via the admin computers and set the route for provider 1 as the BGP routers are not in the same subnet. Also the router to the firewalls for our internal network is clear, but we need one more feature which needs some explaining. If the link to the firewalls goes down on a router, the IP address / network also goes down and its routes over this interface. As the router redistributes the connected and static routes via BGP it will not anymore send it out.  This is basically ok, but now something comes into play that is called “BGP Route Flap Damping“, which can lead to the problem that everything is running again but some AS are not setting traffic to you for some time. So it is paramount to keep the announcing running as stable as possible, which leads us to black hole routes. As in IPv6 Mikrotik does not support it (as of yet) we use a workaround to accomplish the same. PS: you can use the same to black hole an attacker .. really fast and without much load on the system … just saying :-)

/ip route
add distance=1 dst-address=10.0.0.0/8 gateway=10.0.0.254
add distance=1 dst-address=2.0.1.1/32 gateway=2.0.5.10
add distance=1 dst-address=2.0.2.1/32 gateway=2.0.5.10
add distance=1 dst-address=1.0.0.0/22 gateway=1.0.0.254
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=1.0.0.0/22 type=blackhole

/ipv6 route
add distance=1 dst-address=2001:2:1::1/128 gateway=2001:2:3::1
add distance=1 dst-address=2001:2:2::1/128 gateway=2001:628:1400:1003::1
add distance=1 dst-address=2001:1::0/48 gateway=2001:1:3::10

BGP1:
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=2001:1::0/48 gateway=2001:1::ffff

BGP2:
add comment="if interface to firewall goes down, this route is used" distance=254 dst-address=2001:1::0/48 gateway=2001:1:1::1:ffff

After the IP addresses and static routes are configured we need to secure our setup before doing anything else. As the BGP routers are in front of the firewalls they can get attacked directly from the Internet, sure, but traffic (e.g. attacks, P2P, …) to systems behind it can also make problems for the routers, so we’ll do something that we normally don’t do. We’ll disable connection tracking – we are a plain and stupid router … let the firewall track connections, we don’t care. This takes much work from the router if you’ve many many connections over it. Sure it makes the firewall settings on the router harder but as said, let the router focus on its single task – route traffic as much and as fast as possible. I sometimes see BGP routers overloaded with other tasks and than people complain that they have problems with high loads. If your network/uplinks is so small, that it does not matter, sticking with connection tracking is also ok – you’ll just can change the firewall rules to use connection awareness.

/ip firewall connection tracking set enabled=no
/ip settings set tcp-syncookies=yes

Now we create a address list of our BGP peers which we will allow to connect to our BGP daemon. Don’t forget your routers, as they talk also between themselves:

/ip firewall address-list
add address=2.0.1.1 list=listBgpIPv4Peers
add address=2.0.2.1 list=listBgpIPv4Peers
add address=3.0.5.10 list=listBgpIPv4Peers
add address=1.0.0.241 list=listBgpIPv4Peers
add address=1.0.0.242 list=listBgpIPv4Peers

/ipv6 firewall address-list
add address=2001:2:1::1 list=listBgpIPv6Peers
add address=2001:2:2::1 list=listBgpIPv6Peers
add address=2001:3:3::1 list=listBgpIPv6Peers
add address=2001:1:2::1 list=listBgpIPv6Peers
add address=2001:1:2::2 list=listBgpIPv6Peers

And now to actual firewall rules:

/ip firewall filter
add chain=input comment="BGP incomming is ok on all interfaces from our peers" src-address-list=listBgpIPv4Peers dst-port=179 protocol=tcp
add chain=input comment="without conntrack we need to allow that" dst-port=1024-65535 protocol=tcp src-address-list=listBgpIPv4Peers
add chain=input comment="OSFP is on the crosslink ok" in-interface=ether2vlanCrossConnection protocol=ospf
add chain=input comment="VRRP is ok on the interface to the firewalls" dst-address=224.0.0.18 in-interface=ether3vlanTransitFirewall protocol=vrrp
add chain=input comment="everyone can ping us" protocol=icmp
add action=drop chain=input comment="we drop any request from not from the Mgmt Interface" in-interface=!ether4vlanMgmt

/ipv6 firewall filter
add chain=input comment="BGP incomming is ok on all interfaces from our peers" src-address-list=listBgpIPv4Peers dst-port=179 protocol=tcp
add chain=input comment="without conntrack we need to allow that" dst-port=1024-65535 protocol=tcp src-address-list=listBgpIPv6Peers
add chain=input comment="OSFP is on the crosslink ok" in-interface=ether2vlanCrossConnection protocol=ospf
add chain=input comment="VRRP is ok on the interface to the switches" dst-address=ff02::12/128 in-interface=ether3vlanTransitFirewall protocol=vrrp
add chain=input comment="everyone can ping us" protocol=icmpv6
add action=drop chain=input comment="we drop any request from not from the Mgmt Interface" in-interface=!ether4vlanMgmt

Looks like a secure setup .. hopefully it also is :-). Now we’re ready to configure the BGP part, starting with the configuration of the instance.

BGP1:
/routing bgp instance set default as=1000 redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes router-id=1.0.0.244

BGP2:
/routing bgp instance set default as=1000 redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes router-id=1.0.0.245

Now we need to set our networks to announce:

/routing bgp network
add network=1.0.0.0/22
add network=2001:1::0/48

And now we configure our peers. For the 2 BGP routers which are reachable only via an other router we need to set multihop to yes. We need also to make a link between our 2 routers if one sees a peer the other does not but he still is the the VRRP master.

/routing bgp peer

BGP1:
add in-filter=filterIpv4AS2000in multihop=yes name=p1_bgp1 out-filter=filterIpv4GLOBALout remote-address=2.0.1.1 remote-as=2000 tcp-md5-key=xxxxxxxx
add in-filter=filterIpv4AS3000in name=p2_bgp1 out-filter=filterIpv4GLOBALout remote-address=3.0.5.10 remote-as=3000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS2000in multihop=yes name=p1_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:2:1::1 remote-as=2000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS3000in name=p2_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:3:3::1 remote-as=3000 tcp-md5-key=xxxxxxx
add name=bgp2 remote-address=1.0.0.242 remote-as=1000
add name=bgp2ipv6 remote-address=2001:1:2::2 remote-as=1000

BGP2:
add in-filter=filterIpv4AS2000in multihop=yes name=p1_bgp2 out-filter=filterIpv4GLOBALout remote-address=2.0.2.1 remote-as=2000 tcp-md5-key=xxxxxxxx
add in-filter=filterIpv4AS3000in name=p2_bgp1 out-filter=filterIpv4GLOBALout remote-address=3.0.5.10 remote-as=3000 tcp-md5-key=xxxxxxx
add address-families=ipv6 in-filter=filterIpv6AS2000in multihop=yes name=p1_bgp2 out-filter=filterIpv6GLOBALout remote-address=2001:2:2::1 remote-as=2000 tcp-md5-key=xxxxx
add address-families=ipv6 in-filter=filterIpv6AS3000in name=p2_bgp1 out-filter=filterIpv6GLOBALout remote-address=2001:3:3::1 remote-as=3000 tcp-md5-key=xxxxxxx
add name=bgp1 remote-address=1.0.0.241 remote-as=1000
add name=bgp1ipv6 remote-address=2001:1:2::1 remote-as=1000

That was not that hard, but what are all this filter names? As I told you in the beginning we’re paranoid so we don’t trust anyone so we’re filtering all routes going in and out.  So lets start with the out filters as they are much easier. They just let us announce our own networks, so we won’t account networks of the one provider to the other and therefore make a link for them over us.

/routing filter
add action=accept chain=filterIpv4GLOBALout prefix=1.0.0.0/22
add action=discard chain=filterIpv4GLOBALout

add action=accept chain=filterIpv6GLOBALout prefix=2001:1::0/48
add action=discard chain=filterIpv6GLOBALout

The in filters are at little bit more complicated, but not that hard. We make sure that every AS path we get from the provider starts with his AS. It had happened that some provider are a little bit messy there.

add action=jump chain=filterIpv4AS2000in jump-target=filterIpv4Nomartians
add action=accept bgp-as-path="^2000(,[0-9]+)*\$" chain=filterIpv4AS2000in
add action=accept chain=filterIpv4AS2000in

add action=jump chain=filterIpv4AS3000in jump-target=filterIpv4Nomartians
add action=accept bgp-as-path="^3000(,[0-9]+)*\$" chain=filterIpv4AS3000in
add action=accept chain=filterIpv4AS3000in

add action=jump chain=filterIpv6AS2000in jump-target=filterIpv6Nomartians
add action=accept bgp-as-path="^2000(,[0-9]+)*\$" chain=filterIpv6AS2000in
add action=accept chain=filterIpv6AS2000in

add action=jump chain=filterIpv6AS3000in jump-target=filterIpv6Nomartians
add action=accept bgp-as-path="^3000(,[0-9]+)*\$" chain=filterIpv6AS3000in
add action=accept chain=filterIpv6AS3000in

After this is clear, I only need to explain the reason for the filterIpv4Nomartians and filterIpv6Nomartians filters. Its quite easy, these lists contain IP subnets that we should not get via BGP, because they are not used on the Internet (at least not by good people) so we’ll filter them.

add action=discard chain=filterIpv4Nomartians prefix=0.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=127.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=192.0.2.0/24
add action=discard chain=filterIpv4Nomartians prefix=10.0.0.0/8
add action=discard chain=filterIpv4Nomartians prefix=172.16.0.0/12
add action=discard chain=filterIpv4Nomartians prefix=192.168.0.0/16
add action=discard chain=filterIpv4Nomartians prefix=192.168.0.0/15
add action=discard chain=filterIpv4Nomartians prefix=168.254.0.0/16
add action=discard chain=filterIpv4Nomartians prefix=240.0.0.0/4
add action=return chain=filterIpv4Nomartians

add action=discard chain=filterIpv6Nomartians prefix=::/96
add action=discard chain=filterIpv6Nomartians prefix=::/128
add action=discard chain=filterIpv6Nomartians prefix=::1/128
add action=discard chain=filterIpv6Nomartians prefix=::ffff:0.0.0.0/96
add action=discard chain=filterIpv6Nomartians prefix=::224.0.0.0/100
add action=discard chain=filterIpv6Nomartians prefix=::/104
add action=discard chain=filterIpv6Nomartians prefix=::255.0.0.0/104
add action=discard chain=filterIpv6Nomartians prefix=::/8
add action=discard chain=filterIpv6Nomartians prefix=200::/7
add action=discard chain=filterIpv6Nomartians prefix=3ffe::/16
add action=discard chain=filterIpv6Nomartians prefix=2001:db8::/32
add action=discard chain=filterIpv6Nomartians prefix=2002:e000::/20
add action=discard chain=filterIpv6Nomartians prefix=2002:7f00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:ff00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:a00::/24
add action=discard chain=filterIpv6Nomartians prefix=2002:ac10::/28
add action=discard chain=filterIpv6Nomartians prefix=2002:c0a8::/32
add action=discard chain=filterIpv6Nomartians prefix=fc00::/7
add action=discard chain=filterIpv6Nomartians prefix=fe80::/10
add action=discard chain=filterIpv6Nomartians prefix=fec0::/10
add action=discard chain=filterIpv6Nomartians prefix=ff00::/8
add action=return chain=filterIpv6Nomartians

Now we’re done with the BGP setup, only some OSFP stuff is left open. Why OSFP? We want to reach our loopback interfaces via the other router, as only one can be the VRRP master. BGP will only redistribute our complete network and the networks from the provides between our 2 routers, but not some parts of our networks – for this we need OSFP.

BGP1:
/routing ospf instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.244
/routing ospf-v3 instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.244

BGP2:
/routing ospf instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.245
/routing ospf-v3 instance set [ find default=yes ] redistribute-connected=as-type-2 redistribute-static=as-type-2 router-id=1.0.0.245

If you wonder why we use an IPv4 address for the OSFPv3, its because even if its an IPv6 protocol no IPv6 address can be used there … its more like an ID field. Now we only need to set our interfaces and network (only for IPv4 needed):

/routing ospf interface
add interface=loopback network-type=point-to-point passive=yes
add interface=ether3vlanTransitFirewall network-type=point-to-point
add interface=ether2vlanCrossConnection network-type=point-to-point
/routing ospf network
add area=backbone network=1.0.0.240/30
add area=backbone network=1.0.0.248/30
/routing ospf-v3 interface
add area=backbone interface=loopback network-type=point-to-point passive=yes
add area=backbone interface=ether3vlanTransitFirewall network-type=point-to-point
add area=backbone interface=ether2vlanCrossConnection network-type=point-to-point

Basically we’re done …. just some standard setups I’m setting on any Mikrotik and recommend you to set it also:

Change the SNMP Community to something long and not guessable:

/snmp
set contact="Robert Penz" enabled=yes location="datacenter" trap-community=xxxxxxxx trap-generators=interfaces trap-target=10.x.x.x trap-version=2
/snmp community set [ find default=yes ] name=XXXXXXXXXX

Set the clock to the correct timezone and set a NTP server as otherwise the log entries are hard to read:

/system clock set time-zone-name=Europe/Vienna
/system ntp client set enabled=yes mode=unicast primary-ntp=10.x.x.x secondary-ntp=10.x.x.x

Configure a syslog server to have some logs if a Mikrotik booted:

/system logging
add action=remote topics=info
add action=remote topics=error
add action=remote topics=warning
add action=remote topics=critical
/system logging action set 3 bsd-syslog=yes remote=10.x.x.x src-address=<ip of the mikrotik ether4vlanMgmt>

Setting the internal DNS Servers is also a good idea:

/ip dns set servers=10.x.x.x,10.x.x.x

Disable some Services on the Mikrotiks we don’t need:

/ip service
set telnet address=0.0.0.0/0 disabled=yes
set ftp address=0.0.0.0/0 disabled=yes
set www address=0.0.0.0/0 disabled=yes
set winbox address=0.0.0.0/0 disabled=yes

And at last we send traffic samples to our SFlow server …. I would recommend you to have also a good SFlow server for your BGP routers.

/ip traffic-flow target add address=10.x.x.x:9996 version=9

Now you could test your routers, but one last thing I recommend you to install on your router is following script written by MarkB. With one command you get something that looks like show ip bgp summary on Cisco or Vayatta and that makes looking at the BGP stuff much easier on a Mikrotik. Get the script from here.

Any questions or improvement ideas on this setup/howto?

Howto capture traffic from a Mikrotik router on Linux

February 15, 2014

If you as I need to get some traffic from a Mikrotik router and /tool sniffer quick doesn’t cut it, as you need not just the headers the best way is stream the traffic to the a Linux box. The Mikrotik configuration is easy, just set the server you want to stream to:

/tool sniffer set streaming-enabled=yes streaming-server=<ip_of_the_server>

Configure a filter as you don’t want to stream everything:

/tool sniffer set filter-ip-address=<an_example_filter_ip>

and now you need only to start it with

/tool sniffer start

and check with

/tool sniffer print

if everything is running.

But now comes the part that is not documented that well. Searching through the internet I found some posts/articles on how to use Wireshark for capturing, but that does not work correctly – at least not for me.

capture_filter

If you configure the capture filter to udp port 37008 to get everything the router sends via TZSP you will see following lines

fragmented_ip_protocol2

If you now set the display filter to show only TZSP these packets are not displayed any more. This packets contain information we need and I was not able to configure Wireshark 1.10.2 to work correctly. If you know how to get it to work, please write a comment. I changed my approach to use an other program to write the packets to disk and look at them later with Wireshark. And I found a program from Mikrotik directly which does that.  Go to the download page and download Trafr and extract and use it like this:

$ tar xzf trafr.tgz
$ ./trafr
usage: trafr <file | -s> [ip_addr]
-s      write output to stdout. pipe it into tcpdump for example:
./trafr -s | /usr/sbin/tcpdump -r -
ip_addr use to filter one source router by ip address
$ ./trafr test.pcap <ip_of_the_router>

After you stopped the program you can open the file in Wireshark and no packets are missing.

At the Tone, the Time will be.

January 12, 2014

Last week we at work got a mail from CERT.at that 2 IP addresses in our AS where probably running misconfigured NTP Servers, which can be abused for DDoS attacks via NTP Reflection. But first we need to start with the background.

Background

In the last weeks multiple DDoS attacks were using NTP Reflection. The attackers are making use of the monlist commands, which is enabled on older versions of the NTP daemon. With that command it is possible to get a list of up to the last 600 hosts / ip address which connected to the NTP daemon. As NTP is UDP based, an attacker fakes its source IP address and the answer packet from the NTP daemon is send to the victim. Beside hiding the attackers IP addresses to the victim it amplifies the attack as the request packet is much smaller than the answer packet. The other problem with this monlist command is, that it releases potential sensitive information (the IP address of the clients using NTP)

How to verify you’re vulnerable?

First you need to find your NTP servers – and thats not so easy as it seams. E.g. our 2 reported NTP servers where not our official NTP servers … but more about that later. To find NTP Servers which are reachable from the Internet use e.g. nmap in a way like this:

sudo nmap -p 123 -sV -sU -sC -P0 <your_network/subnet_mask>

This will return for a linux ntp server something like this

Nmap scan report for xxxxx (xxxxxxxx)
Host is up (0.00016s latency).
PORT    STATE SERVICE VERSION
123/udp open  ntp     NTP v4
| ntp-info:
|   receive time stamp: 2014-01-12T11:02:30
|   version: ntpd [email protected] Wed Nov 24 19:02:17 UTC 2010 (1)
|   processor: x86_64
|   system: Linux/2.6.32-358.18.1.el6.x86_64
|   leap: 0
|   stratum: 3
|   precision: -24
|   rootdelay: 20.807
|   rootdispersion: 71.722
|   peer: 56121
|   refid: 91.206.8.36
|   reftime: 0xd67cedcd.b514b142
|   poll: 10
|   clock: 0xd67cf4be.9a6959a7
|   state: 4
|   offset: 0.042
|   frequency: -3.192
|   jitter: 0.847
|   noise: 1.548
|   stability: 0.163
|_  tai: 0

But you may find also something like this

Nmap scan report for xxxxx
Host is up (0.00017s latency).
PORT    STATE SERVICE VERSION
123/udp open  ntp     NTP v4
| ntp-info:
|_  receive time stamp: 2014-01-12T11:02:55

from a system you had not on the list. After this deactivate and/or filter the services you don’t need – a running service which is not needed is always a bad idea. But surely you also want to know how to probe the NTP daemon for the monlist command – just like this:

ntpdc -n -c monlist <ip_address>

If the daemon is vulnerable you’ll get a list of ip address which connected to the daemon. If the NTP daemon is running on a Linux, Cisco or Juniper System take look at this page which describes how to configure it correctly.

But I guess you’re curious, which systems where running on the 2 ip addresses we got reported? They where Alcatel Lucent Switches which have the NTP daemon activated by default it seams.  So its really important to check all your IP addresses not only the known NTP Servers.

Multiple consumer routers contain a backdoor

January 5, 2014

As hopefully many of my readers have already heard/read multiple consumer routers contain a backdoor, which allows the attacker to get the configuration of the router, which also contains the administrator password. I won’t rewrite here everything big IT news sites have already written. Here just the basics to get you up to speed if you didn’t hear it before:

  1. Eloi Vanderbeken found on his Linksys router WAG200G a process what was listening on TCP port 32764. After analyzing the code he figured out that it was possible to extract the configuration from the router over this process without knowing the password. The configuration contains also the password.
  2. After hey posted the information to the net, other users stepped forward and told him that other manufactures and models have the same backdoor. Don’t say “conspiracy theory” now. ;-)
  3. On some routers the process is “only” listening on the internal network (which is also attackable over the users browser) but some are also reachable on the Internet. Scanning for this in the internet is easy with zmap .. only 45min for the whole IPv4 Internet address space.
  4. Click here to get the current list of affected routers – its a long list containing vendors like Cisco, Linksys, Netgear, Diamond, LevelOne
  5. To verify if your router is also affected download this Python Script (Linux has normally Python preinstalled on Windows you need to install it). And call it like this:  python poc.py --ip <IP-Address of your router>. If it found something you can extract the configuration by adding --print_conf to the command line.
  6. To check if the process is also reachable from the Internet use a Website like this.

Possible workarounds to get the hole fixed fast:

  1. On some routers you can configure a local firewall which allows you  to block the Port 32764. Depending on your router this is possible for the Internet interface and/or the internal interface.
  2. Install a OpenSource software like OpenWRT.
  3. Install the new firmware release of your vendor when and if it is released … I wouldn’t wait for this. ;-)

Thoughts on IPv6 /64 scanning and NDP cache size

December 24, 2013

Today I was talking with some friends about the possibility to make a DOS attack against an IPv6 router/switch if I was in the same /64 subnet by simply sending IPv6 NDP Packets to fill the neighbour cache on the router. But the question I was thinking than about was how many packets can I send e.g. over an 1Gbit link per second? How many entries will the neighbour cache need to hold if the timeout is e.g. set to 120 sec? How long would it take to scan the whole /64? So I sat down and looked at the questions.

How man packets can I send in one direction send over an 1Gbit Ethernet link?

The amount of packets which can be sent over a link depends on the size of the packets. The smallest ones used for calculation are 64byte in the IP world.  We need to put that into a Ethernet frame which adds up to 84 octets Details can be found here. Which leads to following formula:

1000MbitPerSec / 8 Bits  / 84 OctetsPerFrame= 1.488.095 FramesPerSec

As only one packet can be in a frame we can send 1.488.095 packets per second (often called: pps), which is also often called line speed or wire speed. The calculation is true for pure Ethernet, but I changes if you use VLAN Tags, QinQ or MPLS … in these cases take a look at this article.

How many entries will the neighbour cache need to hold if the timeout is e.g. 120 sec?

So now we know how many packets a can send at most and forget that we need some additional bytes for the NDP, which makes it easy to set the limit for the neigbour cache of our router.

1.488.095 PacketsPerSecond * 120 SecondsTimeout = 178571400 entries = 178  Million Entries

Lets say that this is only a RAM problem and everything else would work. Each entry contains a least the IP address and the MAC address. (There would be an optimization possible in only to store the host part of the IP address). An IPv6 address has 128bit = 16byte and the MAC address has 48bit = 6byte which leads to a total of 22byte per entry. A router needs 3,6Gbyte of RAM to store this table … not impossible but not common also. ;-)

How long would it take to scan the whole /64?

And as bonus question we talk on how long it would take to scan that many IP addresses. First we need to get the amount of IP addresses a /64 can hold.

2^64 = 18.446.744.073.709.551.616 = 1,844674407×10¹⁹ IP Addresses

We know that we can scan 1.488.095 IP addresses per second which leads to

1,844674407×10¹⁹IPaddresses / 1488095 packetsPerSec / 60/ 60 / 24 / 365 = 393081 years

Ok not practical. But wait … we need only to scan for /48 IP addresses as the host part is derived from the MAC … this makes only 2,814749767×10¹⁴ IP addresses

2,814749767×10¹⁴IPaddreses / 1488095 packetsPerSec / 60/ 60 / 24 / 365 = 6 years

Much smaller but still too long for my spare time. ;-)

 

Howto get your external IP address via command line

September 28, 2013

I just had to find out the external IP address (as seen from the Internet) of a Linux server which is behind a NAT router. The normal way to goto WhatsMyIP didn’t work as I was only connected via SSH to this server.  But the solution is quite easy thanks to the guys from ipecho, just type:

wget http://ipecho.net/plain -O - -q ; echo

Thats so easy! And even faster than using a browser in the first way ….. :-)

Howto to quick test a DSCP based QoS system?

August 29, 2013

You’ve just completed your QoS system, which is based on DSCP for classifying and managing network traffic? Sure there are many sophisticated methods to validate your configuration, but there is also a really simple one which you can do from every Windows or Linux PC as a first check.

And you won’t believe it – the program is called ping. On Linux use the option -Q to set the DSCP value of the packets. From the manual:

-Q tos Set Quality of Service -related bits in ICMP datagrams.  tos can be either decimal or hex number.  Traditionally (RFC1349), these have been interpreted as: 0 for reserved (currently being redefined as congestion control), 1-4 for Type of Service and 5-7 for Precedence.  Possible settings for Type of Service are: minimal cost: 0x02, reliability: 0x04, throughput: 0x08, low delay: 0x10.  Multiple TOS  bits  should not  be set simultaneously.  Possible settings for special Precedence range from priority (0x20) to net control (0xe0).  You must be root (CAP_NET_ADMIN capability) to use Critical or higher precedence value. You cannot set bit 0x01 (reserved) unless ECN has been enabled in the kernel.  In RFC2474, these fields has been redefined as 8-bit Differentiated Services (DS), consisting of: bits 0-1 of separate data  (ECN will be used, here), and bits 2-7 of Differentiated Services Codepoint (DSCP).

On Windows the same is achieved with -v

In both cases you need to provide the Type of Service (TOS) byte. While this is not the wished DSCP value, the ToS byte (or 8-bits) encompasses DSCP. DSCP only uses the first 6 bits of the ToS byte and ignores bits 7 and 8. You’re asking ??hey?? ;-)
There is a quite easy way to get from one to the other: DSCP * 4 = TOS, or you can use following table.

Most VoIP systems use AF31(DSCP 26) for signaling (e.g. SIP) and EF (DSCP 46) for voice/media (e.g. RTP). This means for testing we use

ping -Q 104 <ip_address> # for DSCP 26
ping -Q 184 <ip_address> # for DSCP 46

After calling these commands you can easily check your counters if the increment correctly. After this put some load on the connection/link e.g. with FTP or SCP and let the ping run, it should be stable and with a low latency. If not the VoIP stuff with also not work. ;-)

This quick test can also help you by an other problem. You need to deploy a system which relies on the fact that the DSCP value in not being stripped away in transit. For this you use the above command and let Wireshark run.

First you need to add the DSCP colum. Just select a packet and then select the DSCP header and use the right mouse button to get to the “Apply as Column” menu entry.

Add DSCP column

After this you can just look at the DSCP values. If they travel across the network everything is Ok.

Capture DSCP

Howto configure a Mikrotik as central DHCP server with switches as DHCP relays

April 27, 2013

I’ve found many articles about how to configure a Mikrotik RouterOS as DHCP relay or as simple DHCP server, but I didn’t find an article about following setup:

  • central Mikrotik DHCP Server (e.g. in the data center)
  • multiple VLANs/subnets for clients which are connected via a Layer3 Switch (or even more hops) to the central data center

According to the Mikrotik Wiki and the described options it is possible but there is no example shown and everyone uses the setup wizard to configure a DHCP Server. I’ll get most people don’t know what happens in the background … I’ll show this the old fashioned way. ;-)

Setup for this example/howto

For this example we assume that we’ve 4 VLANs with following subnets:

  • 10.88.0.0/24 (data center VLAN for the servers and the DHCP server)
  • 10.88.10.0/24 (clients network location 1)
  • 10.88.11.0/24 (clients network location 2)
  • 10.88.12.0/24 (clients network location 3)

The Mikrotik has the IP 10.88.0.100 and connected via ether1 to the data center VLAN. The Layer3 switches use always the .1 IP address in the clients networks.

Layer3 Switches / DHCP Relay

Most (if not all) switches which are able to perform at least simple layer 3 tasks (often also labeled as Layer2+ switches) are able to forward DHCP requests. Check the manual of the switch for this. One setting I came across sometimes leads to problems. It is called “DHCP Relay delay” and is sometimes set to 1 or 2 seconds in the the default configuration. This setting allows a local DHCP Server to answer faster, but sometimes (specially embedded clients) don’t wait that long and run into an error. If there is no local DHCP server set this timer to 0 seconds.

You’ll need to set the DHCP Server IP on the switch to the IP of the Mikrotik.

Mikrotik as DHCP Server

First we configure our pools of the client networks, the mikrotik will give out IPs from this ranges:

/ip pool
add name=poolClientsLocation1 ranges=10.88.10.10-10.88.10.250
add name=poolClientsLocation1 ranges=10.88.11.10-10.88.11.250
add name=poolClientsLocation1 ranges=10.88.12.10-10.88.12.250

Now we need to set the configuration the DHCP Server will handout the clients:

/ip dhcp-server network
add address=10.88.10.0/24 dns-server=10.88.0.100 gateway=10.88.10.1
add address=10.88.11.0/24 dns-server=10.88.0.100 gateway=10.88.11.1
add address=10.88.11.0/24 dns-server=10.88.0.100 gateway=10.88.12.1

And at last we configure which DHCP Relay gets which configuration/pool:

/ip dhcp-server
add address-pool=poolClientsLocation1 authoritative=yes disabled=no interface=ether1 lease-time=1w name=dhcpClientsLocation1 relay=10.88.10.1
add address-pool=poolClientsLocation2 authoritative=yes disabled=no interface=ether1 lease-time=1w name=dhcpClientsLocation2 relay=10.88.11.1
add address-pool=poolClientsLocation3 authoritative=yes disabled=no interface=ether1 lease-time=1w name=dhcpClientsLocation3 relay=10.88.12.1

This is all … ;-)

How to get IPv6 with a Mikrotik router via an IPv6 tunnel broker

March 2, 2013

You want to try IPv6 but you’re provider doesn’t give you IPv6 addresses? If your router is a Mikrotik this howto will show you how to use an IPv6 tunnel brocker. This setup will also work for guys with a dynamic IP as most home users do.

Signup with an IPv6 tunnel broker

Go to this Wikipedia page for a list of IPv6 tunnel brokers. I’ve chosen the guys at Hurricane Electric as its free and works also with dynamic IP addresses. Other brokers which provide also 6in4 (in RouterOS called /interface 6to4) which provide a method to update of your IP address via a web URL will work too. If you’ve a static IPv4 address the choosing gets even easier.

Following shows whats needs to be done if you use Hurricane Electric as tunnel broker. The tunnel broker wants to be able to ping your router via IPv4 … lets enable that (only if you disabled ping in the first place)

/ip firewall filter add chain=input icmp-options=8:0 protocol=icmp src-address=66.220.2.74

Move this rule to a place before the blocking rule. This rule is made in a way that it opens the least possible hole which is needed for the tunnel broker to work. Now go to the broker website and do following:

  • Click on “Create Regular Tunnel”
  • Under “IPv4 Endpoint (Your side):” enter your current IPv4 address
  • Choose a server location, for better performance choose one thats not that far away from you. ;-)
  • After Creating the Tunnel you’ll be shown a web page with following important information you’ll need later:
    • Tunnel ID
    • Server IPv4 Address
    • Server IPv6 Address
    • Client IPv4 Address
    • Client IPv6 Address
    • Routed /64

For the commands I’ll use “Tunnel ID” if you should use your Tunnel Id and so on.

Mikrotik Tunnel Setup

First make sure that you’ve the IPv6 package enabled -use following command:

[admin@mikrotik] > /system package print
Flags: X - disabled
# NAME VERSION SCHEDULED
0 X ipv6 6.0rc11
.....

In this example it is not enabled, so we need to enable it and reboot.

[admin@mikrotik] > /system package enable ipv6
[admin@mikrotik] > /system reboot

Now it should look like this:

[admin@mikrotik] > /system package print
Flags: X - disabled
# NAME VERSION SCHEDULED
0 ipv6 6.0rc11

.....

After this is done we can start configuring. First we need to create the tunnel endpoint on our side.

[admin@mikrotik] /ip firewall filter> /interface 6to4 add comment="Hurricane Electric IPv6 Tunnel Broker" disabled=no local-address="Client IPv4 Address" mtu=1280 name=sit1 remote-address="Server IPv4 Address"

After that we need to tell the router this is his new default gateway.

[admin@mikrotik] /ip firewall filter> /ipv6 route add dst-address=2000::/3 gateway="Server IPv6 Address"

And of course we need also an IPv6 address.

[admin@mikrotik] /ip firewall filter> /ipv6 address add address="Client IPv6 Address"/64 advertise=yes eui-64=no interface=sit1

Now we can test our setup the first time. Lets check if the router is able to ping an IPv6 address in the Internet … we use for this the Google DNS servers.

[admin@mikrotik] > ping 2001:4860:4860::8844
HOST SIZE TTL TIME STATUS
2001:4860:4860::8844 56 58 36ms echo reply
2001:4860:4860::8844 56 58 35ms echo reply
2001:4860:4860::8844 56 58 46ms echo reply
sent=3 received=3 packet-loss=0% min-rtt=35ms avg-rtt=39ms max-rtt=46ms

If this does not work .. try to ping “Server IPv6 Address”. If this does work you got something wrong which the default gateway. If this also does not work .. check your tunnel setup and your IPv6 address.

Mikrotik Setup for the Clients

Now we got it working for the Mikrotik but it would be even better to have IPv6 for the clients. To achieve this we need to configure an IPv6 address on the LAN interface and enable
the advertisement to the clients. Take the “Routed /64″ address and add an 1 between the :: and the “/” – this give you the first IP address in the routed subnet.

e.g. 2001:xxx:xxx:xxx::/64 –> 2001:xxx:xxx:xxx::1/64

This leads to following command:

/ipv6 address add interface="your LAN interface" address="IP address in your routed IPv6 subnet" advertise=yes

Now wait a little bit (some minutes) and check if your client operation system got an IPv6 address from the routed subnet. I’ll show it here on an Ubuntu 12.04:

$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:xx:xx
inet addr:10.xx.xx.xx Bcast:10.xx.xx.xx Mask:255.255.255.0
inet6 addr: 2001:470:xxx:xxx:xxx:xxx:xxx:xxx/64 Scope:Global
inet6 addr: 2001:470:xxx:xxx:xxx:xxx:xxx:xxx/64 Scope:Global
inet6 addr: fe80::xxx:xxx:xxx:xxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5733100 errors:0 dropped:0 overruns:0 frame:0
TX packets:4191113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4271914251 (4.2 GB) TX bytes:841997928 (841.9 MB)
Interrupt:22 Memory:f6ae0000-f6b00000

You’ll see 3 IPv6 addresses, which are used for following:

  • Scope: Link: Only used for communication within the local VLAN … is always there if IPv6 is enabled on the client
  • Scope:Global:  The second one has the MAC address of this interface in it. This is already a worldwide IP address, but it will tell everyone your MAC address .. which is not that good. So there is the first global one which uses a random part. This feature is called “privacy extension” and the IP is changed by the system after some time. If the privacy extension is activated this IP will be used for communicating.

If you’ve 2 or better 3 IPv6 addresses you can try to ping the Google DNS server:

$ ping6 2001:4860:4860::8844
PING 2001:4860:4860::8844(2001:4860:4860::8844) 56 data bytes
64 bytes from 2001:4860:4860::8844: icmp_seq=1 ttl=57 time=36.7 ms
64 bytes from 2001:4860:4860::8844: icmp_seq=2 ttl=57 time=39.6 ms
64 bytes from 2001:4860:4860::8844: icmp_seq=3 ttl=57 time=48.6 ms
64 bytes from 2001:4860:4860::8844: icmp_seq=4 ttl=57 time=55.3 ms

So this seems to work .. now we need to check if your DNS server reports IPv4 and IPv6 addresses back .. if this is not the case your need to configure your DNS correctly or change to an other server … e.g. the Google DNS Servers.

We’ll verify the correct working with following command:

$ host robert.penz.name
robert.penz.name has address 108.162.198.82
robert.penz.name has address 108.162.199.82
robert.penz.name has IPv6 address 2400:cb00:2048:1::6ca2:c652
robert.penz.name has IPv6 address 2400:cb00:2048:1::6ca2:c752
robert.penz.name mail is handled by 10 mail.penz.name.

This looks good … now you’ll only need to configure a firewall on your Mikrotik as you’re clients now have IPv6 address which can be reached worldwide. Your IPv4 firewall is not enough – you need to filter in /ipv6 firewall.

Mikrotik Setup for users with a dynamic IPv4 address

Now we need to make sure the tunnel broker knows our new IPv4 address after each change. For this Hurricane Electric’s provides a web URL which is described here. Thats nice as there are already some DynDNS update scripts available in the Mikrotik Wiki.

I’ve changed the 5.x one in following parts:

  • “username”: Your Hurricane Electric’s username
  • “password”: Your Hurricane Electric’s password
  • “hostname”: Your “Tunnel ID”
  • I’ve changed the /tool fetch line by replacing members.dyndns.org with ipv4.tunnelbroker.net
  • And at last I added  /interface 6to4 set 0 local-address=$currentIP after :log info ("UpdateDynDNS: Dyndns Update Result: ".$result) line as we need also to change our local tunnel IP address.

Thats it .. call this script every few minutes and you’ll have IPv6 connectivity even after your IP address changes. I’ll hope this article gets others also in the IPv6 world.

IPv6 OpenVZ VEs and Debian/Proxmox as host system

February 24, 2013

A friend of mine got a new root server and asked me to help him set it up. And of course I helped and as he got a free IPv6 subnet I thought lets configure it.  He is running Proxmox as his host system, which is based on Debian.So this guide is also true for Debian systems which have OpenVZ installed.

  1. You want to use IPv6 for the host system, you need to add at least following to  /etc/network/interfaces
    iface vmbr0 inet6 static
    address 2001:xxxx:xxxx:xxx::1
    netmask 64
      up ip -6 route add default via 2001:xxxxx:beef::1 dev vmbr0
      down ip -6 route del default via 2001:xxxxx:beef::1 dev vmbr0

    Replace vmbr0 with eth0 if you’re not using Proxmox and only OpenVZ on Debian.

  2. The ISP my friend has the server located at uses a default gateway, which is not in his IPv6 subnet, you need therefore add a host route. Which is done be following 2 lines
      up ip -6 route add 2001:4ba0:fff7:1:beef::1 dev vmbr0
      down ip -6 route del 2001:4ba0:fff7:1:beef::1 dev vmbr0
  3. Restart the networking with: /etc/init.d/networking restart
  4. Test it with following:
  5. Add following to /etc/sysctl.conf and make sure it is not defined two times
    # IPv6 Packet Forwarding and Proxy NDP
    net.ipv6.conf.default.forwarding = 1
    net.ipv6.conf.all.forwarding = 1
    net.ipv6.conf.default.proxy_ndp = 1
    net.ipv6.conf.all.proxy_ndp = 1
  6. Call sysctl -p to activate it at once (so no reboot is required)
  7. Go to /etc/vz/vz.conf and make sure following is in there
    IPV6="yes"
  8. You currently can’t add an IPv6 address through the web GUI of Proxmox – so you need to use the CLI (replace with a IP adresse from your IPv6 range
    vzctl set <VEID> --ipadd 2001:xxx:xxx::xxx --save
  9. restart the VE, and try the ping stuff from step 4 in the VE. (ps: It can take up to 5 pings before you get a response the first time.)

More is not needed … its quite easy!

ps: don’t forget to add an IPv6 DNS server, if it is a IPv6 only VE. This can currently not be done via Proxmox web GUI.

pps: iptables does not filter IPv6 .. you need ip6tables for this. don’t think that you’re secure, if you didn’t open anything with iptables.

Powered by WordPress
Entries and comments feeds. Valid XHTML and CSS. 24 queries. 0.195 seconds.