# Providers: Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss?



## DomainBop (Feb 13, 2014)

Do You Send An RFO Email If Your Network Has 5 Minutes of Packet Loss on Some Routes? How long of a network outage, or packet loss, is required before you send out an RFO.

The reason I'm asking is because SeFlow had a short 5 minute packet loss on some international routes (not a total network outage) today due to a 40Gbps DDoS attack and sent out an RFO within the hour (which I thought was very nice).

Meanwhile (close your eyes if you cringe when seeing the word ColoCrossing), when I had VPS's on ColoCrossing's network (specifically CVPS/UGVPS in Buffalo, Atlanta, Los Angeles) packet loss was a very common occurence and typically lasted much longer than 5 minutes and RFO's were not sent out (unless it was a total network outage that lasted for several hours...and even then RFO's were not always sent)

The RFO SeFlow sent (posted for educational purposes because a few  LEB providers may not know what an RFO looks like)



> Dear Customer,
> 
> as a result of a violent attack over 40Gbps happened a few minutes ago, we had packet loss on international uplinks for about 5 minutes. In last period attacks have increased size due to the NTP amplification bug.
> 
> ...


+1 for SeFlow


----------



## KuJoe (Feb 13, 2014)

If there is no outage then sending a Reason For Outage is not necessary. If there is packet loss on some routes and it goes unnoticed then nothing is done (that's the idea of being multi-homed). If there is packet loss and it is noticed then an announcement is posted.


----------



## danni (Feb 13, 2014)

Is seflow within same datacenter as KnownHost ?

( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )


----------



## Amitz (Feb 13, 2014)

SeFlow is in Milan/Italy. Knownhost only has US locations. So - No.


----------



## TruvisT (Feb 13, 2014)

We typically do only due to the fact that we offer all clients hosted with us a monitoring service that monitors all their servers from a remote location down to the services and because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.


----------



## DomainBop (Feb 13, 2014)

danni said:


> Is seflow within same datacenter as KnownHost ?
> 
> ( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )



They're in Milan ( <1ms from Prometeus).  I think a lot of data centers have had 40+ Gbps attacks in recent days .  OVH started filtering all NTP traffic through its "VAC" DDoS system today regardless of whether there has been a DDoS from/to the server.

http://status.ovh.com/?do=details&id=6321



> because they would notice that little gap we send out an e-mail to effected users before we get e-mails asking.



reducing the number of "what happened" support tickets following an incident is one of the primary reasons to send out an RFO as soon as possible.


----------



## KS_Phillip (Feb 14, 2014)

Anything that triggers a "down" or "unresponsive" notification from our monitoring results in an RFO email to our enterprise customers.  Gaming customers get an update on the gaming site.


----------



## Francisco (Feb 14, 2014)

danni said:


> Is seflow within same datacenter as KnownHost ?
> 
> ( http://forums.knownhost.com/threads/2-12-14-tx-network-connectivity-issues.2901/ <-- the last post )


So about that.

I kinda get the feeling that level3 took a full nose dive when that happened. For a good 10 - 15 minutes I had 90%+ packetloss to 4.2.2.1 from my home and some sites simply didn't load. There was a thead on LET about 'what broke in Buffalo', and the broken trace went over Level3.

I think it's safe to assume that something freakin' MASSIVE bitch slapped L3.

Francisco


----------



## blergh (Feb 14, 2014)

Yes, the only issue lies with our status-page derping from time to time.


----------



## imperio (Feb 15, 2014)

My favourite is blaming internap notices.


----------



## BrianHarrison (Feb 20, 2014)

Packet loss doesn't necessairly lead to a full outage. Packet loss also doesn't necessarily affect all customers. You could have a congested node on Comcast's network that affects a lot of end users and leads to packet loss, but you certainly wouldn't issue an RFO for that.


----------



## hostinghouston (Mar 26, 2014)

Sorry to be late to the party on this thread, but I think it's important to send an RFO / Announcement to customers when something happens. Be that a forum post, emial, status update on social media - they have a right to know and it's wrong for a company to think that such events go un-noticed. It's also a good way to prevent tons of "WTF" tickets from angry customers because they were in the middle of a Forex trade or got kicked from their game, radio stream, whatever it may be!


----------



## Echelon (Mar 28, 2014)

Sending an RFO even when all quiet on the support front doesn't hurt, since customers prefer that you're proactively ensuring their service is working as opposed to reactively fixing based on demands, requests, and questions.


----------



## Exelion (Mar 29, 2014)

I actually like it when ISPs admit things have gone wrong and were fixed before anyone noticed. It shows how honest a provider is. It also means you could easily charge way more for your service just to have that.


----------

