This post is dedicated to Phúc Toàn, a friendly and joyful guy from Ho Chi Minh City, Vietnam. He was one of the participants in IPv4/IPv6 Routing Workshop at APRICOT 2017. He created his own lab to test RTBH using blackhole community but for some reason it wasn’t working and he showed it to me after the class. We could find the problem, and finally it worked as expected. Later, he took me to a cafe and we had some seasonal fresh fruit juice along with one of his colleagues and then he gave me a ride on his bike around the Saigon river, it was really a beautiful view of HCM City in the evening. Thanks to Phúc Toàn, I wish to meet him someday again.
What is RTBH?
Remotely Triggered Black Hole (RTBH) is a technique where configuration performed in your network triggers an action to be executed automatically in a remote network to black hole traffic that are intended to reach to the destination in your network. RTBH can be used (in fact widely used to blackhole DDoS traffic).
Why and when you may need it?
If you want to filter out unwanted incoming traffic (towards a certain IP or IP block in your network or in your customer network) closer to the source or as far away as possible from your network, RTBH can do that for you. A very popular example would be filtering Distribute Denial of Service (DDoS) traffic. You can filter out the incoming packet with other methods like using inbound ACL in your gateway router. But, the traffic actually already reached your gateway router (before getting dropped) using your uplink circuit. Which means, your valuable uplink bandwidth is already used by those unwanted packet and your gateway router would process the packet before discarding. If those traffic is of large amount enough to chock your uplink bandwidth or to utilize router’s full process, the whole network and services will be affected. All the clients would start complaining. If those unwanted packets could have been filtered out far away from the network, those consequences can be avoided.
But the problem is…
While the unwanted incoming packets are filtered out by triggering remotely, it will discard the valid packets as well. Because, the remote networks are only triggered to filter out traffics towards a certain IP address or a certain network, there is now way to instruct them to allow valid packets based on valid source addresses.
How RTBH can help during a DDoS attack?
Suppose one of your servers is under a DDoS attack. The attacking devices are residing all over the Internet and participating in the attack from different locations. The torrents of incoming traffic can be blocked in your gateway router using an inbound ACL but that would not solve the problem. As I explained before, the huge volume of traffic is already congesting your uplink bandwidth and may have consumed router’s whole process. All your other clients and services must suffer in this case. Otherwise, what you can do is, you can blackhole those traffic in your network destined to that affected server’s IP address and ask your upstream routers to do so. BGP blackhole community is used to trigger this instruction. Then, your upstream would blackhole traffic towards your server’s IP and ask their upstreams/peers to do so. They would ask their upstreams/peers and so on. This way you can trigger and propagate blackhole instruction as far away as possible from your network. However, as said earlier, all the valid traffic would also get backholed due to this action. Which means, eventually the attackers win as your services running on that server would temporarily remain unavailable until you restore/resume it using another IP address.
Configuration Example:
Here is our lab scenario where we have an ISP (or enterprise), a national or regional transit provider (depicting a Tier-2 ISP or so) and a large ISP or Global Transit Provider (depicting a Tier-1 ISP). The reason why this topology is chosen is to show the probable configuration of BGP blackhole community at each level and how it works throughout the whole Internet. The target is to blackhole the incoming traffic towards Victim Server (192.168.0.2) using blackhole community.
Considerations:
- Private IP address blocks are used for this lab instead of public IP addresses.
- BGP is configured between Autonomous Systems and community sharing is permitted (send-community is configured between neighbors)
- Regional-Transit-1 and Regional-Transit-2 do not filter customer prefixes automatically based on RPKI/ROA/RADB. Rather, they use route map, prefix list and AS filter list to do so.
- Global-Transit-1 and Global-Transit-2 filters customer prefixes automatically based on RPKI/ROA/RADB.
- XYZ-ISP knows the blackhole community of all its upstream (i.e 65502:222 for Regional-Transit-1 and 65503:333 for Regional Transit-2).
- Regional-Transit-1 also knows the blackhole community of all its upstream. In our lab, the ultimate upstream are Global-Transit-1 (blackhole community is 65504:444) and Global-Transit-2 (blackhole community is 65505:555).
Basic Configuration:
Before configuring the RTBH, the basic BGP and interface configurations are done in each router. For example, the configuration of XYZ-ISP may look like this:
interface fa0/0
description Connected to Server
ip address 192.168.0.1 255.255.255.0
!
interface gi1/0
description Connected to Regional-Transit-1
ip address 10.10.12.1 255.255.255.252
!
interface gi2/0
description Connected to Regional-Transit-2
ip address 10.10.13.1 255.255.255.252
!
router bgp 65501
network 192.168.0.0 mask 255.255.0.0
neighbor 10.10.12.2 remote-as 65502
neighbor 10.10.12.2 send-community
neighbor 10.10.13.2 remote-as 65503
neighbor 10.10.13.2 send-community
Other routers are also configured accordingly.
RTBH Specific Configuration:
In XYZ-ISP:
At first, XYZ-ISP need to prepare a route-map to set all its upstream’s blackhole community there.
ip bgp-community new-format
!
route-map DDOS-COMM permit 10
set community 65502:222 65503:333
Configure Regional-Transit-1 (similar configuration is applied to Regional-Transit-2):
The transit providers of XYZ-ISP need to perform two tasks. Task# 1 is, they need to ripple the trigger of RTBH towards their upstream. Which means if XYZ-ISP advertises a prefix attaching the blackhole community of Regional-Transit-1 and/or Regional-Transit-2, the transit routers will match their community and attach their upstream’s blackhole community just to move the trigger a bit further away from the XYZ-ISP. And Task# 2 is, Regional-Transit-1 and Regional-Transit-2 must be able to blackhole locally originated traffic or its customer traffic that are intended to reach the Victim-Server.
For Task# 1, XYZ-ISP’s transit provider must know their upstream’s blackhole community. Here in our example, Regional-Transit-1 would ask its upstream Global-Transit-1 and Global-Transit-2 to get their blackhole community 65504:444 and 65505:555. Similar tasks for Regional-Transit-2. If they have multiple upstream, they need to know the blackhole community of each of their upstream. Now, in order to set upstream’s blackhole community for any prefix coming from the downstream, Regional-Transit-1 needs to add Global-Transit-1’s blackhole community to that prefix. Here’s how to do that:
Configure a loopback interface to nullify blackhole traffic that may comes to Regional-Transit-1 network (Task# 2 mentioned above). This traffic may be generated by Regional-Transit-1’s local network or by its clients. (BTW, why I used loopback? See the Note at the bottom for the answer).
interface Loopback0
ip address 100.100.100.100 255.255.255.255
For Task# 1, configure a community list permitting its blackhole community:
ip bgp-community new-format
ip community-list 1 permit 65502:222
Now as we considered that the regional transit network is filtering clients’ prefixes manually, we need to configure AS filter list, prefix list and route map as shown below:
ip as-path access-list 5 permit ^65500$
!
ip prefix-list CLIENT1-LE24 seq 10 permit 192.168.0.0/16 le 24
!
ip prefix-list CLIENT1-LE32 seq 10 permit 192.168.0.0/16 le 32
!
route-map CLIENT1-IN permit 10
match ip address prefix-list CLIENT1-LE32
match as-path 5
match community 1
set community 65504:444 65505:555 additive
set ip next-hop 100.100.100.100
!
route-map CLIENT1-IN permit 20
match ip address prefix-list CLIENT1-LE24
match as-path 1
You may be thinking, why we need two prefix lists to permit the same client prefix. Prefixes with subnet length grater than /24 is not allowed in global BGP table. However, during DDoS, you may need to advertise a single IP address (/32) with blackhole community. For that reason, two prefix lists for the same client prefix are created, prefix-list CLIENT1-LE24 permits up to /24 prefixes only and prefix-list CLIENT1-LE32 permits up to /32 prefixes of the client IP block.
route-map CLIENT1-IN permit 10 matches XYZ-ISP’s valid prefix of any size (192.168.0.0/16 up to /32), if and only if the prefix originates from AS 65501 (XYZ-ISP) and have the blackhole community tag of Regional-Transit-1. Once there is a match, the route-map sets that incoming prefix with blackhole community of all the upstream of Regional-Transit-1 and also it forces to set the next-hop of the prefix to loopback 0 (100.100.100.100) so that traffic towards that traffic gets blackholed.
However, under normal condition when no DDoS is affecting XYZ-ISP network, it won’t send any prefix with the blackhole community. Hence, under normal condition route-map CLIENT1-IN permit 10 won’t be used. Instead, route-map CLIENT1-IN permit 20 will be used where it simply permits XYZ-ISP’s valid prefixes with a maximum subnet length of /24.
Now, apply the route-map to inbound bgp peering with XYZ-ISP:
router bgp 65502
neighbor 10.10.12.1 route-map CLIENT1-IN in
Regional-Transit-1 and Regional-Transit-2 need to do similar configurations for each of its clients.
Configuration in Global-Transit-1 network (similar configuration is applied to Global-Transit-2):
The configuration of Global-Transit-1 is quite similar to the Regional-Transit-1. First, configure a loopback interface to nullify blackhole traffic.
interface Loopback0
ip address 100.100.100.100 255.255.255.255
Configure a community list permitting its blackhole community:
ip bgp-community new-format
ip community-list 10 permit 65504:444
Configure a route-map to match blackhole community sent by clients (i.e. Regional-Transit-1) and force set the next-hop to the loopback 0.
route-map CLIENT-PREF-IN permit 10
match community 10
set ip next-hop 100.100.100.100
!
route-map CLIENT-PREF-IN permit 20
Now, apply the route-map to inbound bgp peering with all the clients:
router bgp 65504
neighbor 100.100.24.1 route-map CLIENT-PREF-IN in
Global-Transit-1 and Global-Transit-2 need to do similar configurations for each of its clients.
Analysis:
Before experiencing any DDoS:
Prefix 192.168.0.0/24 is originated and announced by XYZ-ISP which is visible in global routing table.
Global-Transit-1# show ip bgp
<output omitted for brevity>
Network Next Hop Metric LocPrf Weight Path
*> 192.168.0.0 10.10.24.2 0 65502 65501 i
Global-Transit-1# show ip route
<output omitted for brevity>
B 192.168.0.0/24 [20/0] via 10.10.24.2, 00:06:33
The server (192.168.0.2) is reachable from remote end (across the Internet).
DDoS-Attack-Source-1> ping 192.168.0.2
84 bytes from 192.168.0.2 icmp_seq=1 ttl=61 time=70.330 ms
84 bytes from 192.168.0.2 icmp_seq=2 ttl=61 time=62.430 ms
84 bytes from 192.168.0.2 icmp_seq=3 ttl=61 time=66.811 ms
84 bytes from 192.168.0.2 icmp_seq=4 ttl=61 time=64.521 ms
84 bytes from 192.168.0.2 icmp_seq=5 ttl=61 time=64.094 ms
DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1 10.0.0.1 6.156 ms 11.616 ms 11.482 ms
2 10.10.24.2 23.274 ms 23.295 ms 22.743 ms
3 10.10.12.1 71.202 ms 46.970 ms 47.438 ms
4 *192.168.0.2 60.071 ms
DDoS-Attack-Source-2> ping 192.168.0.2
84 bytes from 192.168.0.2 icmp_seq=1 ttl=61 time=131.122 ms
84 bytes from 192.168.0.2 icmp_seq=2 ttl=61 time=42.788 ms
84 bytes from 192.168.0.2 icmp_seq=3 ttl=61 time=43.041 ms
84 bytes from 192.168.0.2 icmp_seq=4 ttl=61 time=65.200 ms
84 bytes from 192.168.0.2 icmp_seq=5 ttl=61 time=60.756 ms
DDoS-Attack-Source-2> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1 172.16.0.1 9.881 ms 10.605 ms 11.692 ms
2 10.10.35.3 34.238 ms 23.966 ms 24.412 ms
3 10.10.13.1 45.684 ms 47.363 ms 46.607 ms
4 *192.168.0.2 69.863 ms
During the DDoS attack to Victim-Server (192.168.0.2):
Once the XYZ-ISP realised that one of their servers (192.168.0.2) is experiencing DDoS attack, all they need is to announce that single IP address using BGP blackhole community that was prepared beforehand.
router bgp 65501
network 192.168.0.2 mask 255.255.255.255 route-map DDOS-COMM
!
ip route 192.168.0.2 255.255.255.255 Null0
Now, two prefixes are visible in global table (one is the /24 prefix and other one is DDoS affected /32 IP address). The next hope of that /32 prefix is the loopback in the router which means that it will blackhole any traffic targeting 192.168.0.2.
Global-Transit-1# show ip bgp
<output omitted for brevity>
Network Next Hop Metric LocPrf Weight Path
*> 192.168.0.0 10.10.24.2 0 65502 65501 i
*> 192.168.0.2/32 100.100.100.100 0 65502 65501 i
Global-Transit-1# show ip route
<output omitted for brevity>
192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
B 192.168.0.0/24 [20/0] via 10.10.24.2, 00:06:33
B 192.168.0.2/32 [20/0] via 100.100.100.100, 00:00:08
Global-Transit-1#show ip bgp 192.168.0.2
BGP routing table entry for 192.168.0.2/32, version 4
Paths: (1 available, best #1, table default)
Not advertised to any peer
Refresh Epoch 1
65502 65501
100.100.100.100 from 10.10.24.2 (10.10.24.2)
Origin IGP, localpref 100, valid, external, best
Community: 65502:222 65503:333 65504:444
rx pathid: 0, tx pathid: 0x0
Global-Transit-1#show ip bgp 192.168.0.0
BGP routing table entry for 192.168.0.0/24, version 3
Paths: (1 available, best #1, table default)
Not advertised to any peer
Refresh Epoch 1
65502 65501
10.10.24.2 from 10.10.24.2 (10.10.24.2)
Origin IGP, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
Hence, any traffic towards the Victim-Server will be dropped far away from the XYZ-ISP network.
DDoS-Attack-Source-1> ping 192.168.0.2
192.168.0.2 icmp_seq=1 timeout
192.168.0.2 icmp_seq=2 timeout
192.168.0.2 icmp_seq=3 timeout
192.168.0.2 icmp_seq=4 timeout
192.168.0.2 icmp_seq=5 timeout
DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1 10.0.0.1 7.274 ms 11.580 ms 11.437 ms
2 *10.0.0.1 11.322 ms (ICMP type:3, code:3, Destination port unreachable)
DDoS-Attack-Source-2> ping 192.168.0.2
192.168.0.2 icmp_seq=1 timeout
192.168.0.2 icmp_seq=2 timeout
192.168.0.2 icmp_seq=3 timeout
192.168.0.2 icmp_seq=4 timeout
192.168.0.2 icmp_seq=5 timeout
DDoS-Attack-Source-2> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1 172.16.0.1 5.012 ms 11.501 ms 11.689 ms
2 *172.16.0.1 11.322 ms (ICMP type:3, code:3, Destination port unreachable)
Note:
All the references in the Internet about RTBH has a static route towards null interface to blackhole the traffic. I tried to use it but it didn’t work. As you can see below, 100.100.100.100 pointing towards null0 show inaccessible hence the router won’t install the /32 route in the routing table. So, packets towards the Victim-Server are not actually blackholed. Hence, I used the configuration with loopback interface instead of null interface. I understood, it may increase the process a little bit compared to using null0 interface, but I am okay with that as the process is not that much increased.
Global-Transit-1(config)#ip route 100.100.100.100 255.255.255.255 null 0
Global-Transit-1#show ip bgp 192.168.0.2
BGP routing table entry for 192.168.0.2/32, version 9
Paths: (1 available, no best path)
Not advertised to any peer
Refresh Epoch 2
65502 65501
100.100.100.100 (inaccessible) from 10.10.24.2 (10.10.24.2)
Origin IGP, localpref 100, valid, external
Community: 65502:222 65503:333 65504:444
rx pathid: 0, tx pathid: 0
Global-Transit-1# show ip route
<output omitted for brevity>
B 192.168.0.0/24 [20/0] via 10.10.24.2, 00:18:03
DDoS-Attack-Source-1> trace 192.168.0.2
trace to 192.168.0.2, 8 hops max, press Ctrl+C to stop
1 10.0.0.1 12.678 ms 10.237 ms 11.455 ms
2 100.100.24.2 57.416 ms 34.630 ms 21.331 ms
3 100.100.12.1 34.156 ms 35.722 ms 34.258 ms
4 *100.100.12.1 34.008 ms