URPad down, Wholesale Internet and Datashack also down.

drmike

100% Tier-1 Gogent
(we need a section for outages)

Stopped in to check on things and I see:

URPad (UK)

Wholesale Internet (Kansas City)

Datashack (Kansas City)

are all offline.

Lots of price conscious folks offline.
 

drmike

100% Tier-1 Gogent
Well in Kansas City, I am betting on a network failure due to hardware failure.  Their network runs well, but have been increasing number of outages in the past year.  Either aging hardware or growth up to capacity of devices and humans on staff.
 

drmike

100% Tier-1 Gogent
I wonder who dat could have been, silly :D!
Well it wasn't me :)  It was you Mun :) Hadn't seen your post here (didn't refresh prior to post).

Wii-Aaron - owner of WSI confirmed it is an external fiber cut by AT&T workers and supervisors of all sorts are on the site of breakage.

Going to be a long day for them.  I don't expect being online for another 6 hours.  Let's see if they can beat that. :)
 

Centaur

New Member
Yeah, this is getting a little crappy. Their support is fast, however, there is still now mass email or any message on their Facebook or Twitter page. Which to me is really bad. 

Hopefully, everything will be back online soon. So much for redundancy. Having both fiber links next to each other.
 

drmike

100% Tier-1 Gogent
Aaron posted to WHT saying it should be back up soon.

AT&T idiots cut with a big saw municipal fiber. Big mess there.
 

HalfEatenPie

The Irrational One
Retired Staff
Yep.  He estimated 1 hour to 2 hours more wait, but it seems to be close to the 2.5 hours mark right now (unless my math is off which is a high possibility because I haven't been really following it too closely due to... well...  I don't have services there anymore).  

Edit:  Basically to explain what happened.

Apparently they were cutting the pavement in a location for a manhole (using a manhole drill) and that cut the fibers for the datacenter (owned by the city).  They did have redundant fibers, it was just that the fibers were near each other.  They'll look into later if the fiber was in the wrong location (or if the construction crew drilled the hole in the wrong place (that's what she said?)) but basically they took out the "redundant network".  To fix this they're moving one of the fiber cables to the other side of the street.  Also they do have a backup Fiber network in place but switching over to that would probably take longer than just waiting for this to be fixed.

That's all assuming my interpretation of what WSI-Aaron told us on WHT. 

Not trying to divert away from the situation, but WSI-Aaron's name reminds me of this:


Hopefully that'll lighten the mood a little bit! 
 
Last edited by a moderator:

JayCawb

New Member
There's not really much any customers of WSI/Datashack can do at the moment, I'm still up at 2:28AM as I had some other work to do but I guess we'll just have to wait it out.

Goodluck to them on getting this up asap anyhow, I've just been getting canned replies, and the tech I know personally isn't on shift right now.
 
Last edited by a moderator:

ThePrimeHost

New Member
Verified Provider
Bah. Unfortunately this kind of incident happens from time to time in this industry. I've seen it repeatedly over the years with various providers on WHT. It's a "grin and bear" it situation all the while eu's are screaming "this is unacceptable!!!!!! - I wan't you to waive your magic wand and fix it nowwww!!!!"  Or something like that :D
 

Magiobiwan

Insert Witty Statement Here
Verified Provider
So far the thread on WHT hasn't filled up with crying raging people. Give it time though.
 

drmike

100% Tier-1 Gogent
It is a big, long and nasty outage. Going to be perhaps costly for a bunch of companies (SLAs, lost customers, etc.).

There really needs to be some official insurance one can buy against things like this directly.
 

HalfEatenPie

The Irrational One
Retired Staff
It is a big, long and nasty outage. 
That's what she said.  Or would that be that's what he said?

Inappropriate jokes aside, the city did own the fiber lines and I feel like they placed the infrastructure there with the assumption/understanding the City had their systems taken care of (redundancy wise as well).  When this happened, well... I feel like the city in itself should be responsible (I mean WSI-Aaron did state the City owned the fiber lines that were cut) for setting up redundancy.  But then now it comes to the point of how much redundancy do we actually need?  Like how far up the "chain" do we have to go to advertise true redundancy? 
 
Last edited by a moderator:

drmike

100% Tier-1 Gogent
Yeah Aaron did mention that their own fiber was pulled from same street side.  Meaning redundancy could be taken out with one vault compromise.

He also said they were going to go and have new fiber pulled from other direction in the future.

The outage didn't effect Oak high rise location, but instead their North Datashack location which is backhauling fiber from Oak to there.  Meaning packets flow into Oak and then out through city fiber to North Datashack location.   The issue and redundancy is specific to Datashack and not WSI itself.  WSI customers appear to be online throughout this outage.

I've been love/hate about Datashack for eons.  Love the price, pretty good network.  But there has been breakage/outages more than I'd like (and packet loss and weird changes here and there in routing).  Glad this wasn't WSI/DS but outside issue for a change.  Last outage out of their grasp was extended weekend day where they were dealing with power upgrades in Oak.  That was another big nasty outage.
 

MartinD

Retired Staff
Verified Provider
Retired Staff
To be down one when a pipe is cut is bad news. Where's the second from the other side of the building?

Tut tut.
 
Top