Configure Remotely Triggered Black Hole (RTBH) using Blackhole Community to mitigate DDoS

This post is dedicated to Phúc Toàn, a friendly and joyful guy from Ho Chi Minh City, Vietnam. He was  one of the participants in IPv4/IPv6 Routing Workshop at APRICOT 2017. He created his own lab to test RTBH using blackhole community but for some reason it wasn’t working and he showed it to me after the class. We could find the problem, and finally it worked as expected. Later, he took me to a cafe and we had some seasonal fresh fruit juice along with one of his colleagues and then he gave me a ride on his bike around the Saigon river, it was really a beautiful view of HCM City in the evening. Thanks to Phúc Toàn, I wish to meet him someday again.

What is RTBH?

Remotely Triggered Black Hole (RTBH) is a technique where configuration performed in your network triggers an action to be executed automatically in a remote network to black hole traffic that are intended to reach to the destination in your network. RTBH can be used (in fact widely used to blackhole DDoS traffic).

Why and when you may need it?

If you want to filter out unwanted incoming traffic (towards a certain IP or IP block in your network or in your customer network) closer to the source or as far away as possible from your network, RTBH can do that for you. A very popular example would be filtering Distribute Denial of Service (DDoS) traffic. You can filter out the incoming packet with other methods like using inbound ACL in your gateway router. But, the traffic actually already reached your gateway router (before getting dropped) using your uplink circuit. Which means, your valuable uplink bandwidth is already used by those unwanted packet and your gateway router would process the packet before discarding. If those traffic is of large amount enough to chock your uplink bandwidth or to utilize router’s full process, the whole network and services will be affected. All the clients would start complaining. If those unwanted packets could have been filtered out far away from the network, those consequences can be avoided.

But the problem is…

While the unwanted incoming packets are filtered out by triggering remotely, it will discard the valid packets as well. Because, the remote networks are only triggered to filter out traffics towards a certain IP address or a certain network, there is now way to instruct them to allow valid packets based on valid source addresses.

How RTBH can help during a DDoS attack?

Suppose one of your servers is under a DDoS attack. The attacking devices are residing all over the Internet and participating in the attack from different locations. The torrents of incoming traffic can be blocked in your gateway router using an inbound ACL but that would not solve the problem. As I explained before, the huge volume of traffic is already congesting your uplink bandwidth and may have consumed router’s whole process. All your other clients and services must suffer in this case. Otherwise, what you can do is, you can blackhole those traffic in your network destined to that affected server’s IP address and ask your upstream routers to do so. BGP blackhole community is used to trigger this instruction. Then, your upstream would blackhole traffic towards your server’s IP and ask their upstreams/peers to do so. They would ask their upstreams/peers and so on. This way you can trigger and propagate blackhole instruction as far away as possible from your network. However, as said earlier, all the valid traffic would also get backholed due to this action. Which means, eventually the attackers win as your services running on that server would temporarily remain unavailable until you restore/resume it using another IP address.

Configuration Example:

Here is our lab scenario where we have an ISP (or enterprise), a national or regional transit provider (depicting a Tier-2 ISP or so) and a large ISP or Global Transit Provider (depicting a Tier-1 ISP). The reason why this topology is chosen is to show the probable configuration of BGP blackhole community at each level and how it works throughout the whole Internet. The target is to blackhole the incoming traffic towards Victim Server (192.168.0.2) using blackhole community.

Considerations:

  • Private IP address blocks are used for this lab instead of public IP addresses.
  • BGP is configured between Autonomous Systems and community sharing is permitted (send-community is configured between neighbors)
  • Regional-Transit-1 and Regional-Transit-2 do not filter customer prefixes automatically based on RPKI/ROA/RADB. Rather, they use route map, prefix list and AS filter list to do so.
  • Global-Transit-1 and Global-Transit-2 filters customer prefixes automatically based on RPKI/ROA/RADB.
  • XYZ-ISP knows the blackhole community of all its upstream (i.e 65502:222 for Regional-Transit-1 and 65503:333 for Regional Transit-2).
  • Regional-Transit-1 also knows the blackhole community of all its upstream. In our lab, the ultimate upstream are Global-Transit-1 (blackhole community is 65504:444) and Global-Transit-2 (blackhole community is 65505:555).

Basic Configuration:

Before configuring the RTBH, the basic BGP and interface configurations are done in each router. For example, the configuration of XYZ-ISP may look like this:

interface fa0/0
 description Connected to Server
 ip address 192.168.0.1 255.255.255.0
!
interface gi1/0
 description Connected to Regional-Transit-1
 ip address 10.10.12.1 255.255.255.252
!
interface gi2/0
 description Connected to Regional-Transit-2
 ip address 10.10.13.1 255.255.255.252
!
router bgp 65501
 network 192.168.0.0 mask 255.255.0.0
 neighbor 10.10.12.2 remote-as 65502
 neighbor 10.10.12.2 send-community
 neighbor 10.10.13.2 remote-as 65503
 neighbor 10.10.13.2 send-community

Other routers are also configured accordingly.

RTBH Specific Configuration:

In XYZ-ISP:

At first, XYZ-ISP need to prepare a route-map to set all its upstream’s blackhole community there.

ip bgp-community new-format
!
route-map DDOS-COMM permit 10
 set community 65502:222 65503:333

Configure Regional-Transit-1 (similar configuration is applied to Regional-Transit-2):

The transit providers of XYZ-ISP need to perform two tasks. Task# 1 is, they need to ripple the trigger of RTBH towards their upstream. Which means if XYZ-ISP advertises a prefix attaching the blackhole community of Regional-Transit-1 and/or Regional-Transit-2, the transit routers will match their community and attach their upstream’s blackhole community just to move the trigger a bit further away from the XYZ-ISP. And Task# 2 is, Regional-Transit-1 and Regional-Transit-2 must be able to blackhole locally originated traffic or its customer traffic that are intended to reach the Victim-Server.

For Task# 1, XYZ-ISP’s transit provider must know their upstream’s blackhole community. Here in our example, Regional-Transit-1 would ask its upstream Global-Transit-1 and Global-Transit-2 to get their blackhole community 65504:444 and 65505:555. Similar tasks for Regional-Transit-2. If they have multiple upstream, they need to know the blackhole community of each of their upstream. Now, in order to set upstream’s blackhole community for any prefix coming from the downstream, Regional-Transit-1 needs to add Global-Transit-1’s blackhole community to that prefix. Here’s how to do that:

Configure a loopback interface to nullify blackhole traffic that may comes to Regional-Transit-1 network (Task# 2 mentioned above). This traffic may be generated by Regional-Transit-1’s local network or by its clients. (BTW, why I used loopback? See the Note at the bottom for the answer).

interface Loopback0
 ip address 100.100.100.100 255.255.255.255

For Task# 1, configure a community list permitting its blackhole community:

ip bgp-community new-format
ip community-list 1 permit 65502:222

Now as we considered that the regional transit network is filtering clients’ prefixes manually, we need to configure AS filter list, prefix list and route map as shown below:

ip as-path access-list 5 permit ^65500$
!
ip prefix-list CLIENT1-LE24 seq 10 permit 192.168.0.0/16 le 24
!
ip prefix-list CLIENT1-LE32 seq 10 permit 192.168.0.0/16 le 32
!
route-map CLIENT1-IN permit 10
 match ip address prefix-list CLIENT1-LE32
 match as-path 5
 match community 1
 set community 65504:444 65505:555 additive
 set ip next-hop 100.100.100.100
!
route-map CLIENT1-IN permit 20
 match ip address prefix-list CLIENT1-LE24
 match as-path 1

You may be thinking, why we need two prefix lists to permit the same client prefix. Prefixes with subnet length grater than /24 is not allowed in global BGP table. However, during DDoS, you may need to advertise a single IP address (/32) with blackhole community. For that reason, two prefix lists for the same client prefix are created, prefix-list CLIENT1-LE24 permits up to /24 prefixes only and prefix-list CLIENT1-LE32 permits up to /32 prefixes of the client IP block.

route-map CLIENT1-IN permit 10 matches XYZ-ISP’s valid prefix of any size (192.168.0.0/16 up to /32), if and only if the prefix originates from AS 65501 (XYZ-ISP) and have the blackhole community tag of Regional-Transit-1. Once there is a match, the route-map sets that incoming prefix with blackhole community of all the upstream of Regional-Transit-1 and also it forces to set the next-hop of the prefix to loopback 0 (100.100.100.100) so that traffic towards that traffic gets blackholed.

However, under normal condition when no DDoS is affecting XYZ-ISP network, it won’t send any prefix with the blackhole community. Hence, under normal condition route-map CLIENT1-IN permit 10 won’t be used. Instead, route-map CLIENT1-IN permit 20 will be used where it simply permits XYZ-ISP’s valid prefixes with a maximum subnet length of /24.

Now, apply the route-map to inbound bgp peering with XYZ-ISP:

router bgp 65502
 neighbor 10.10.12.1 route-map CLIENT1-IN in

Regional-Transit-1 and Regional-Transit-2 need to do similar configurations for each of its clients.

Configuration in Global-Transit-1 network (similar configuration is applied to Global-Transit-2):

The configuration of Global-Transit-1 is quite similar to the Regional-Transit-1. First, configure a loopback interface to nullify blackhole traffic.

interface Loopback0
 ip address 100.100.100.100 255.255.255.255

Configure a community list permitting its blackhole community:

ip bgp-community new-format
ip community-list 10 permit 65504:444

Configure a route-map to match blackhole community sent by clients (i.e. Regional-Transit-1) and force set the next-hop to the loopback 0.

route-map CLIENT-PREF-IN permit 10
 match community 10
 set ip next-hop 100.100.100.100
!
route-map CLIENT-PREF-IN permit 20

Now, apply the route-map to inbound bgp peering with all the clients:

router bgp 65504
 neighbor 100.100.24.1 route-map CLIENT-PREF-IN in

Global-Transit-1 and Global-Transit-2 need to do similar configurations for each of its clients.

Analysis:

Before experiencing any DDoS:

Prefix 192.168.0.0/24 is originated and announced by XYZ-ISP which is visible in global routing table.

Global-Transit-1# show ip bgp
<output omitted for brevity>
     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.0.0      10.10.24.2                           0 65502 65501 i

Global-Transit-1# show ip route
<output omitted for brevity>
B        192.168.0.0/24 [20/0] via 10.10.24.2, 00:06:33

The server (192.168.0.2) is reachable from remote end (across the Internet).

DDoS-Attack-Source-1> ping 192.168.0.2
84 bytes from 192.168.0.2 icmp_seq=1 ttl=61 time=70.330 ms
84 bytes from 192.168.0.2 icmp_seq=2 ttl=61 time=62.430 ms
84 bytes from 192.168.0.2 icmp_seq=3 ttl=61 time=66.811 ms
84 bytes from 192.168.0.2 icmp_seq=4 ttl=61 time=64.521 ms
84 bytes from 192.168.0.2 icmp_seq=5 ttl=61 time=64.094 ms

DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
 1   10.0.0.1   6.156 ms  11.616 ms  11.482 ms
 2   10.10.24.2   23.274 ms  23.295 ms  22.743 ms
 3   10.10.12.1   71.202 ms  46.970 ms  47.438 ms
 4   *192.168.0.2   60.071 ms

DDoS-Attack-Source-2> ping 192.168.0.2
84 bytes from 192.168.0.2 icmp_seq=1 ttl=61 time=131.122 ms
84 bytes from 192.168.0.2 icmp_seq=2 ttl=61 time=42.788 ms
84 bytes from 192.168.0.2 icmp_seq=3 ttl=61 time=43.041 ms
84 bytes from 192.168.0.2 icmp_seq=4 ttl=61 time=65.200 ms
84 bytes from 192.168.0.2 icmp_seq=5 ttl=61 time=60.756 ms

DDoS-Attack-Source-2> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
 1   172.16.0.1   9.881 ms  10.605 ms  11.692 ms
 2   10.10.35.3   34.238 ms  23.966 ms  24.412 ms
3   10.10.13.1   45.684 ms  47.363 ms  46.607 ms
 4   *192.168.0.2   69.863 ms 

During the DDoS attack to Victim-Server (192.168.0.2):

Once the XYZ-ISP realised that one of their servers (192.168.0.2) is experiencing DDoS attack, all they need is to announce that single IP address using BGP blackhole community that was prepared beforehand.

router bgp 65501
 network 192.168.0.2 mask 255.255.255.255 route-map DDOS-COMM
!
ip route 192.168.0.2 255.255.255.255 Null0

Now, two prefixes are visible in global table (one is the /24 prefix and other one is DDoS affected /32 IP address). The next hope of that /32 prefix is the loopback in the router which means that it will blackhole any traffic targeting 192.168.0.2.

Global-Transit-1# show ip bgp
<output omitted for brevity>
     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.0.0      10.10.24.2                           0 65502 65501 i
 *>  192.168.0.2/32   100.100.100.100                        0 65502 65501 i

Global-Transit-1# show ip route
<output omitted for brevity>
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
B        192.168.0.0/24 [20/0] via 10.10.24.2, 00:06:33
B        192.168.0.2/32 [20/0] via 100.100.100.100, 00:00:08

Global-Transit-1#show ip bgp 192.168.0.2
BGP routing table entry for 192.168.0.2/32, version 4
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Refresh Epoch 1
  65502 65501
    100.100.100.100 from 10.10.24.2 (10.10.24.2)
      Origin IGP, localpref 100, valid, external, best
      Community: 65502:222 65503:333 65504:444
      rx pathid: 0, tx pathid: 0x0

Global-Transit-1#show ip bgp 192.168.0.0
BGP routing table entry for 192.168.0.0/24, version 3
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Refresh Epoch 1
  65502 65501
    10.10.24.2 from 10.10.24.2 (10.10.24.2)
      Origin IGP, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0

Hence, any traffic towards the Victim-Server will be dropped far away from the XYZ-ISP network.

DDoS-Attack-Source-1> ping 192.168.0.2
192.168.0.2 icmp_seq=1 timeout
192.168.0.2 icmp_seq=2 timeout
192.168.0.2 icmp_seq=3 timeout
192.168.0.2 icmp_seq=4 timeout
192.168.0.2 icmp_seq=5 timeout

DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
 1   10.0.0.1   7.274 ms  11.580 ms  11.437 ms
 2   *10.0.0.1   11.322 ms (ICMP type:3, code:3, Destination port unreachable)

DDoS-Attack-Source-2> ping 192.168.0.2
192.168.0.2 icmp_seq=1 timeout
192.168.0.2 icmp_seq=2 timeout
192.168.0.2 icmp_seq=3 timeout
192.168.0.2 icmp_seq=4 timeout
192.168.0.2 icmp_seq=5 timeout

DDoS-Attack-Source-2> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1   172.16.0.1   5.012 ms  11.501 ms  11.689 ms
2   *172.16.0.1   11.322 ms (ICMP type:3, code:3, Destination port unreachable)

Note:

All the references in the Internet about RTBH has a static route towards null interface to blackhole the traffic. I tried to use it but it didn’t work. As you can see below, 100.100.100.100 pointing towards null0 show inaccessible hence the router won’t install the /32 route in the routing table. So, packets towards the Victim-Server are not actually blackholed. Hence, I used the configuration with loopback interface instead of null interface. I understood, it may increase the process a little bit compared to using null0 interface, but I am okay with that as the process is not that much increased.

Global-Transit-1(config)#ip route 100.100.100.100 255.255.255.255 null 0

Global-Transit-1#show ip bgp 192.168.0.2
BGP routing table entry for 192.168.0.2/32, version 9
Paths: (1 available, no best path)
  Not advertised to any peer
  Refresh Epoch 2
  65502 65501
    100.100.100.100 (inaccessible) from 10.10.24.2 (10.10.24.2)
      Origin IGP, localpref 100, valid, external
      Community: 65502:222 65503:333 65504:444
      rx pathid: 0, tx pathid: 0

Global-Transit-1# show ip route
<output omitted for brevity>
B        192.168.0.0/24 [20/0] via 10.10.24.2, 00:18:03

DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
 1   10.0.0.1   12.678 ms  10.237 ms  11.455 ms
 2   100.100.24.2   57.416 ms  34.630 ms  21.331 ms
 3   100.100.12.1   34.156 ms  35.722 ms  34.258 ms
 4   *100.100.12.1   34.008 ms 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.