# URPad down, Wholesale Internet and Datashack also down.



## drmike (Aug 6, 2013)

(we need a section for outages)

Stopped in to check on things and I see:

URPad (UK)

Wholesale Internet (Kansas City)

Datashack (Kansas City)

are all offline.

Lots of price conscious folks offline.


----------



## Mun (Aug 6, 2013)

Hmmm, wonder why?

Mun


----------



## Mun (Aug 6, 2013)

As per Datashack, a fiber was cut. No ETA.

Mun


----------



## drmike (Aug 6, 2013)

Well in Kansas City, I am betting on a network failure due to hardware failure.  Their network runs well, but have been increasing number of outages in the past year.  Either aging hardware or growth up to capacity of devices and humans on staff.


----------



## drmike (Aug 6, 2013)

Someone just said that that KC outage is a fiber cut.


----------



## Mun (Aug 6, 2013)

buffalooed said:


> Someone just said that that KC outage is a fiber cut.



I wonder who dat could have been, silly !



Mun said:


> As per Datashack, a fiber was cut. No ETA.
> 
> Mun


----------



## MCH-Phil (Aug 6, 2013)

Confirmed, at WHT, fiber cut by AT&T.  Good job AT&T


----------



## drmike (Aug 6, 2013)

Mun said:


> I wonder who dat could have been, silly !


Well it wasn't me   It was you Mun  Hadn't seen your post here (didn't refresh prior to post).

Wii-Aaron - owner of WSI confirmed it is an external fiber cut by AT&T workers and supervisors of all sorts are on the site of breakage.

Going to be a long day for them.  I don't expect being online for another 6 hours.  Let's see if they can beat that.


----------



## Slownode (Aug 6, 2013)

AT&T splicing more taps in fibre for spying huh.


----------



## Mun (Aug 6, 2013)

Hmm, still down. This is rather crappy.

Mun


----------



## Centaur (Aug 6, 2013)

Yeah, this is getting a little crappy. Their support is fast, however, there is still now mass email or any message on their Facebook or Twitter page. Which to me is really bad. 

Hopefully, everything will be back online soon. So much for redundancy. Having both fiber links next to each other.


----------



## drmike (Aug 6, 2013)

Aaron posted to WHT saying it should be back up soon.

AT&T idiots cut with a big saw municipal fiber. Big mess there.


----------



## HalfEatenPie (Aug 6, 2013)

Yep.  He estimated 1 hour to 2 hours more wait, but it seems to be close to the 2.5 hours mark right now (unless my math is off which is a high possibility because I haven't been really following it too closely due to... well...  I don't have services there anymore).  

*Edit:*  Basically to explain what happened.

Apparently they were cutting the pavement in a location for a manhole (using a manhole drill) and that cut the fibers for the datacenter (owned by the city).  They did have redundant fibers, it was just that the fibers were near each other.  They'll look into later if the fiber was in the wrong location (or if the construction crew drilled the hole in the wrong place (that's what she said?)) but basically they took out the "redundant network".  To fix this they're moving one of the fiber cables to the other side of the street.  Also they do have a backup Fiber network in place but switching over to that would probably take longer than just waiting for this to be fixed.

That's all assuming my interpretation of what WSI-Aaron told us on WHT. 

Not trying to divert away from the situation, but WSI-Aaron's name reminds me of this:






Hopefully that'll lighten the mood a little bit!


----------



## JayCawb (Aug 6, 2013)

There's not really much any customers of WSI/Datashack can do at the moment, I'm still up at 2:28AM as I had some other work to do but I guess we'll just have to wait it out.

Goodluck to them on getting this up asap anyhow, I've just been getting canned replies, and the tech I know personally isn't on shift right now.


----------



## ThePrimeHost (Aug 6, 2013)

Bah. Unfortunately this kind of incident happens from time to time in this industry. I've seen it repeatedly over the years with various providers on WHT. It's a "grin and bear" it situation all the while eu's are screaming "this is unacceptable!!!!!! - I wan't you to waive your magic wand and fix it nowwww!!!!"  Or something like that


----------



## Magiobiwan (Aug 6, 2013)

So far the thread on WHT hasn't filled up with crying raging people. Give it time though.


----------



## drmike (Aug 6, 2013)

It is a big, long and nasty outage. Going to be perhaps costly for a bunch of companies (SLAs, lost customers, etc.).

There really needs to be some official insurance one can buy against things like this directly.


----------



## HalfEatenPie (Aug 6, 2013)

buffalooed said:


> It is a big, long and nasty outage.


That's what she said.  Or would that be that's what he said?

Inappropriate jokes aside, the city did own the fiber lines and I feel like they placed the infrastructure there with the assumption/understanding the City had their systems taken care of (redundancy wise as well).  When this happened, well... I feel like the city in itself should be responsible (I mean WSI-Aaron did state the City owned the fiber lines that were cut) for setting up redundancy.  But then now it comes to the point of how much redundancy do we actually need?  Like how far up the "chain" do we have to go to advertise true redundancy?


----------



## drmike (Aug 6, 2013)

Yeah Aaron did mention that their own fiber was pulled from same street side.  Meaning redundancy could be taken out with one vault compromise.

He also said they were going to go and have new fiber pulled from other direction in the future.

The outage didn't effect Oak high rise location, but instead their North Datashack location which is backhauling fiber from Oak to there.  Meaning packets flow into Oak and then out through city fiber to North Datashack location.   The issue and redundancy is specific to Datashack and not WSI itself.  WSI customers appear to be online throughout this outage.

I've been love/hate about Datashack for eons.  Love the price, pretty good network.  But there has been breakage/outages more than I'd like (and packet loss and weird changes here and there in routing).  Glad this wasn't WSI/DS but outside issue for a change.  Last outage out of their grasp was extended weekend day where they were dealing with power upgrades in Oak.  That was another big nasty outage.


----------



## MartinD (Aug 7, 2013)

To be down one when a pipe is cut is bad news. Where's the second from the other side of the building?

Tut tut.


----------



## imperio (Aug 7, 2013)

buffalooed said:


> The outage didn't effect Oak high rise location, but instead their North Datashack location which is backhauling fiber from Oak to there.  Meaning packets flow into Oak and then out through city fiber to North Datashack location.   The issue and redundancy is specific to Datashack and not WSI itself.  WSI customers appear to be online throughout this outage.


I have a dedicated from WSI(support mails are from datashack,billing mails are from WSI) and it was offline due this outage.


----------



## drmike (Aug 7, 2013)

Thanks @imperio.   That's kind of news since the North facility was pretty much 90% of what was spoken about.

Upside to this outage is routes appear to be over Cogent for now.   Haven't been watching the network much lately, but do remember and often groaning about the Sprint transit there instead (which also was adding a bit more latency).

Ping times for me are up and down - pretty big deviations in short period of time 35-54 ms (tested 2-3 minutes ago).  Saw higher deviations last night.


----------

