URPad down, Wholesale Internet and Datashack also down.
drmike
100% Tier-1 Gogent
Well it wasn't meI wonder who dat could have been, silly!
It was you Mun Hadn't seen your post here (didn't refresh prior to post).Wii-Aaron - owner of WSI confirmed it is an external fiber cut by AT&T workers and supervisors of all sorts are on the site of breakage.
Going to be a long day for them. I don't expect being online for another 6 hours. Let's see if they can beat that.
Yeah, this is getting a little crappy. Their support is fast, however, there is still now mass email or any message on their Facebook or Twitter page. Which to me is really bad.
Hopefully, everything will be back online soon. So much for redundancy. Having both fiber links next to each other.
Hopefully, everything will be back online soon. So much for redundancy. Having both fiber links next to each other.
Yep. He estimated 1 hour to 2 hours more wait, but it seems to be close to the 2.5 hours mark right now (unless my math is off which is a high possibility because I haven't been really following it too closely due to... well... I don't have services there anymore).
Edit: Basically to explain what happened.
Apparently they were cutting the pavement in a location for a manhole (using a manhole drill) and that cut the fibers for the datacenter (owned by the city). They did have redundant fibers, it was just that the fibers were near each other. They'll look into later if the fiber was in the wrong location (or if the construction crew drilled the hole in the wrong place (that's what she said?)) but basically they took out the "redundant network". To fix this they're moving one of the fiber cables to the other side of the street. Also they do have a backup Fiber network in place but switching over to that would probably take longer than just waiting for this to be fixed.
That's all assuming my interpretation of what WSI-Aaron told us on WHT.
Not trying to divert away from the situation, but WSI-Aaron's name reminds me of this:
Hopefully that'll lighten the mood a little bit!
Edit: Basically to explain what happened.
Apparently they were cutting the pavement in a location for a manhole (using a manhole drill) and that cut the fibers for the datacenter (owned by the city). They did have redundant fibers, it was just that the fibers were near each other. They'll look into later if the fiber was in the wrong location (or if the construction crew drilled the hole in the wrong place (that's what she said?)) but basically they took out the "redundant network". To fix this they're moving one of the fiber cables to the other side of the street. Also they do have a backup Fiber network in place but switching over to that would probably take longer than just waiting for this to be fixed.
That's all assuming my interpretation of what WSI-Aaron told us on WHT.
Not trying to divert away from the situation, but WSI-Aaron's name reminds me of this:
Hopefully that'll lighten the mood a little bit!
Last edited by a moderator:
There's not really much any customers of WSI/Datashack can do at the moment, I'm still up at 2:28AM as I had some other work to do but I guess we'll just have to wait it out.
Goodluck to them on getting this up asap anyhow, I've just been getting canned replies, and the tech I know personally isn't on shift right now.
Goodluck to them on getting this up asap anyhow, I've just been getting canned replies, and the tech I know personally isn't on shift right now.
Last edited by a moderator:
Bah. Unfortunately this kind of incident happens from time to time in this industry. I've seen it repeatedly over the years with various providers on WHT. It's a "grin and bear" it situation all the while eu's are screaming "this is unacceptable!!!!!! - I wan't you to waive your magic wand and fix it nowwww!!!!" Or something like that
So far the thread on WHT hasn't filled up with crying raging people. Give it time though.
That's what she said. Or would that be that's what he said?
Inappropriate jokes aside, the city did own the fiber lines and I feel like they placed the infrastructure there with the assumption/understanding the City had their systems taken care of (redundancy wise as well). When this happened, well... I feel like the city in itself should be responsible (I mean WSI-Aaron did state the City owned the fiber lines that were cut) for setting up redundancy. But then now it comes to the point of how much redundancy do we actually need? Like how far up the "chain" do we have to go to advertise true redundancy?
Last edited by a moderator:
drmike
100% Tier-1 Gogent
Yeah Aaron did mention that their own fiber was pulled from same street side. Meaning redundancy could be taken out with one vault compromise.
He also said they were going to go and have new fiber pulled from other direction in the future.
The outage didn't effect Oak high rise location, but instead their North Datashack location which is backhauling fiber from Oak to there. Meaning packets flow into Oak and then out through city fiber to North Datashack location. The issue and redundancy is specific to Datashack and not WSI itself. WSI customers appear to be online throughout this outage.
I've been love/hate about Datashack for eons. Love the price, pretty good network. But there has been breakage/outages more than I'd like (and packet loss and weird changes here and there in routing). Glad this wasn't WSI/DS but outside issue for a change. Last outage out of their grasp was extended weekend day where they were dealing with power upgrades in Oak. That was another big nasty outage.
He also said they were going to go and have new fiber pulled from other direction in the future.
The outage didn't effect Oak high rise location, but instead their North Datashack location which is backhauling fiber from Oak to there. Meaning packets flow into Oak and then out through city fiber to North Datashack location. The issue and redundancy is specific to Datashack and not WSI itself. WSI customers appear to be online throughout this outage.
I've been love/hate about Datashack for eons. Love the price, pretty good network. But there has been breakage/outages more than I'd like (and packet loss and weird changes here and there in routing). Glad this wasn't WSI/DS but outside issue for a change. Last outage out of their grasp was extended weekend day where they were dealing with power upgrades in Oak. That was another big nasty outage.
