amuck-landowner

Providers: Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss?

DomainBop

Dormant VPSB Pathogen
Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss on Some Routes? How long of a network outage, or packet loss, is required before you send out an RFO.

The reason I'm asking is because SeFlow had a short 5 minute packet loss on some international routes (not a total network outage) today due to a 40Gbps DDoS attack and sent out an RFO within the hour (which I thought was very nice).

Meanwhile (close your eyes if you cringe when seeing the word ColoCrossing), when I had VPS's on ColoCrossing's network (specifically CVPS/UGVPS in Buffalo, Atlanta, Los Angeles) packet loss was a very common occurence and typically lasted much longer than 5 minutes and RFO's were not sent out (unless it was a total network outage that lasted for several hours...and even then RFO's were not always sent)

The RFO SeFlow sent (posted for educational purposes because a few  LEB providers may not know what an RFO looks like)

Dear Customer,

                             as a result of a violent attack over 40Gbps happened a few minutes ago, we had packet loss on international uplinks for about 5 minutes. In last period attacks have increased size due to the NTP amplification bug.

In order to prevent further disruptions , we have implemented a series of scripts inside seguard that put automatic blackhole if we receive attack other 40Gbps.

The change has already been implemented and is already working.

blackholing means the we block , at transit level, target DDoS IP. This means that in case of attack other SeFlow capacity, only theinvolved ip address will be locked, and all other services will remain active without any problematic or packet loss.

With this solution we are certain to avoid any future packet loss in our network
+1 for SeFlow
 

KuJoe

Well-Known Member
Verified Provider
If there is no outage then sending a Reason For Outage is not necessary. If there is packet loss on some routes and it goes unnoticed then nothing is done (that's the idea of being multi-homed). If there is packet loss and it is noticed then an announcement is posted.
 

TruvisT

Server Management Specialist
Verified Provider
We typically do only due to the fact that we offer all clients hosted with us a monitoring service that monitors all their servers from a remote location down to the services and because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.
 

DomainBop

Dormant VPSB Pathogen
Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )

They're in Milan ( <1ms from Prometeus).  I think a lot of data centers have had 40+ Gbps attacks in recent days :).  OVH started filtering all NTP traffic through its "VAC" DDoS system today regardless of whether there has been a DDoS from/to the server.

http://status.ovh.com/?do=details&id=6321

because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.

reducing the number of "what happened" support tickets following an incident is one of the primary reasons to send out an RFO as soon as possible.
 

KS_Phillip

New Member
Verified Provider
Anything that triggers a "down" or "unresponsive" notification from our monitoring results in an RFO email to our enterprise customers.  Gaming customers get an update on the gaming site.
 

Francisco

Company Lube
Verified Provider
Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )
So about that.

I kinda get the feeling that level3 took a full nose dive when that happened. For a good 10 - 15 minutes I had 90%+ packetloss to 4.2.2.1 from my home and some sites simply didn't load. There was a thead on LET about 'what broke in Buffalo', and the broken trace went over Level3.

I think it's safe to assume that something freakin' MASSIVE bitch slapped L3.

Francisco
 
Packet loss doesn't necessairly lead to a full outage. Packet loss also doesn't necessarily affect all customers. You could have a congested node on Comcast's network that affects a lot of end users and leads to packet loss, but you certainly wouldn't issue an RFO for that.
 

hostinghouston

Member
Verified Provider
Sorry to be late to the party on this thread, but I think it's important to send an RFO / Announcement to customers when something happens. Be that a forum post, emial, status update on social media - they have a right to know and it's wrong for a company to think that such events go un-noticed. It's also a good way to prevent tons of "WTF" tickets from angry customers because they were in the middle of a Forex trade or got kicked from their game, radio stream, whatever it may be!
 

Echelon

New Member
Verified Provider
Sending an RFO even when all quiet on the support front doesn't hurt, since customers prefer that you're proactively ensuring their service is working as opposed to reactively fixing based on demands, requests, and questions.
 

Exelion

New Member
I actually like it when ISPs admit things have gone wrong and were fixed before anyone noticed. It shows how honest a provider is. It also means you could easily charge way more for your service just to have that.
 
Top
amuck-landowner