Jack
Active Member
Hey;
Just wondering if this has happened to any other providers on here as it was a first for me when I read the reason for the outage;.
https://my.securedragon.net/announcements.php?id=313
Today we received some network disruption in our Tampa data center due to our IPs being hijacked by a data center in Latin America. While we do not know if this was a malicious attempt to impact our network or an honest mistake (fat fingered an IP?), it was still disruptive none-the-less. Luckily our data center reacted quickly to resolve this issue and the downtime was limited to about 30 minutes according to our external monitoring. In reality, some of our upstream providers should have been routing normally so the outage was not a complete outage (check drgn.biz and you'll see no downtime reported but our other monitoring service shows downtime for certain nodes in Tampa).
The "fix".
Our data center had to announce our IPs as a /24 to overwrite their announcement.
So how is this possible?
(An excellent question from a longtime client of ours that prompted me to do more research about it.)
It's possible because BGP requires no authentication or confirmation of IP ownership. It's up to the upstream providers (Level3, Cogent, HE, nLayer, Verizon, Comcast, etc...) to verify IP ownership but some of them accept any routes provided to them without even manual intervention (Cogent lets all data centers who pay for a commit to announce any IP via a web interface, Level3 requires a phone call or an e-mail/ticket along with a Letter of Authority from the IP owners).
Unfortunately there is no real method to prevent this which is why BGP monitoring services are pretty popular although they just send an e-mail AFTER the damage is done and the IPs are announced. Hopefully with the smaller announcements it will prevent future hijacks.
-The Secure Dragon Staff
Just wondering if this has happened to any other providers on here as it was a first for me when I read the reason for the outage;.
https://my.securedragon.net/announcements.php?id=313
Today we received some network disruption in our Tampa data center due to our IPs being hijacked by a data center in Latin America. While we do not know if this was a malicious attempt to impact our network or an honest mistake (fat fingered an IP?), it was still disruptive none-the-less. Luckily our data center reacted quickly to resolve this issue and the downtime was limited to about 30 minutes according to our external monitoring. In reality, some of our upstream providers should have been routing normally so the outage was not a complete outage (check drgn.biz and you'll see no downtime reported but our other monitoring service shows downtime for certain nodes in Tampa).
The "fix".
Our data center had to announce our IPs as a /24 to overwrite their announcement.
So how is this possible?
(An excellent question from a longtime client of ours that prompted me to do more research about it.)
It's possible because BGP requires no authentication or confirmation of IP ownership. It's up to the upstream providers (Level3, Cogent, HE, nLayer, Verizon, Comcast, etc...) to verify IP ownership but some of them accept any routes provided to them without even manual intervention (Cogent lets all data centers who pay for a commit to announce any IP via a web interface, Level3 requires a phone call or an e-mail/ticket along with a Letter of Authority from the IP owners).
Unfortunately there is no real method to prevent this which is why BGP monitoring services are pretty popular although they just send an e-mail AFTER the damage is done and the IPs are announced. Hopefully with the smaller announcements it will prevent future hijacks.
-The Secure Dragon Staff
