Providers: Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss?

[DomainBop]

Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss on Some Routes? How long of a network outage, or packet loss, is required before you send out an RFO.

The reason I'm asking is because SeFlow had a short 5 minute packet loss on some international routes (not a total network outage) today due to a 40Gbps DDoS attack and sent out an RFO within the hour (which I thought was very nice).

Meanwhile (close your eyes if you cringe when seeing the word ColoCrossing), when I had VPS's on ColoCrossing's network (specifically CVPS/UGVPS in Buffalo, Atlanta, Los Angeles) packet loss was a very common occurence and typically lasted much longer than 5 minutes and RFO's were not sent out (unless it was a total network outage that lasted for several hours...and even then RFO's were not always sent)

The RFO SeFlow sent (posted for educational purposes because a few  LEB providers may not know what an RFO looks like)

Dear Customer,

                             as a result of a violent attack over 40Gbps happened a few minutes ago, we had packet loss on international uplinks for about 5 minutes. In last period attacks have increased size due to the NTP amplification bug.

In order to prevent further disruptions , we have implemented a series of scripts inside seguard that put automatic blackhole if we receive attack other 40Gbps.

The change has already been implemented and is already working.

blackholing means the we block , at transit level, target DDoS IP. This means that in case of attack other SeFlow capacity, only theinvolved ip address will be locked, and all other services will remain active without any problematic or packet loss.

With this solution we are certain to avoid any future packet loss in our network

+1 for SeFlow

 

[KuJoe]

If there is no outage then sending a Reason For Outage is not necessary. If there is packet loss on some routes and it goes unnoticed then nothing is done (that's the idea of being multi-homed). If there is packet loss and it is noticed then an announcement is posted.

 

[danni]

Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )

 

[Amitz]

SeFlow is in Milan/Italy. Knownhost only has US locations. So - No.

 

[TruvisT]

We typically do only due to the fact that we offer all clients hosted with us a monitoring service that monitors all their servers from a remote location down to the services and because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.

 

[DomainBop]

Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )

They're in Milan ( <1ms from Prometeus).  I think a lot of data centers have had 40+ Gbps attacks in recent days .  OVH started filtering all NTP traffic through its "VAC" DDoS system today regardless of whether there has been a DDoS from/to the server.

http://status.ovh.com/?do=details&id=6321

because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.

reducing the number of "what happened" support tickets following an incident is one of the primary reasons to send out an RFO as soon as possible.

 

[KS_Phillip]

Anything that triggers a "down" or "unresponsive" notification from our monitoring results in an RFO email to our enterprise customers.  Gaming customers get an update on the gaming site.

 

[Francisco]

Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )

So about that.

I kinda get the feeling that level3 took a full nose dive when that happened. For a good 10 - 15 minutes I had 90%+ packetloss to 4.2.2.1 from my home and some sites simply didn't load. There was a thead on LET about 'what broke in Buffalo', and the broken trace went over Level3.

I think it's safe to assume that something freakin' MASSIVE bitch slapped L3.

Francisco

 

[blergh]

Yes, the only issue lies with our status-page derping from time to time.

 

[imperio]

My favourite is blaming internap notices.

 

[BrianHarrison]

Packet loss doesn't necessairly lead to a full outage. Packet loss also doesn't necessarily affect all customers. You could have a congested node on Comcast's network that affects a lot of end users and leads to packet loss, but you certainly wouldn't issue an RFO for that.

 

[hostinghouston]

Sorry to be late to the party on this thread, but I think it's important to send an RFO / Announcement to customers when something happens. Be that a forum post, emial, status update on social media - they have a right to know and it's wrong for a company to think that such events go un-noticed. It's also a good way to prevent tons of "WTF" tickets from angry customers because they were in the middle of a Forex trade or got kicked from their game, radio stream, whatever it may be!

 

[Echelon]

Sending an RFO even when all quiet on the support front doesn't hurt, since customers prefer that you're proactively ensuring their service is working as opposed to reactively fixing based on demands, requests, and questions.

 

[Exelion]

I actually like it when ISPs admit things have gone wrong and were fixed before anyone noticed. It shows how honest a provider is. It also means you could easily charge way more for your service just to have that.