amuck-landowner

Fiberhub running on generator power

iWF-Jacob

New Member
Verified Provider
Just got this lovely email from Fiberhub. Glad their backup system is working so far, I'll be curious to see if it continues to do so!

Emergency Maintenance Notification

Start Time:  Tuesday, July 22 2014 4:00PM Pacific Time

Estimated Completion Time:  Unknown, current estimate 18-12 hours

Expected Service Impact:  None

Possible Service Impact:  High

Maintenance Information:  Fiberhub LAS1 is currently operating on emergency  backup generator power.  The electric utility company which provides power  to our LAS1 datacenter (NV Energy) has determined that our main service tr ansformer has suffered a critical failure, resulting in a complete loss of utility power to the datacenter.  We are actively working with their engine ers to complete a repair or replacement of the unit ASAP.

Our facility is currently operating normally on generator backup power.  Al

l redundant power systems have operated flawlessly thus far, and customer p

ower has not been impacted in any way.  Our fuel suppliers have been notifi

ed and are on standby should we require additional fuel at any time.

Please stand by for updates, as this situation is rapidly evolving.
 

iWF-Jacob

New Member
Verified Provider
I already talked to Rob about it and they were ready for this due to some upgrades their power company was already working on.


Francisco
That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.
 

Francisco

Company Lube
Verified Provider
That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.
I found spamming him on google hangouts works great :D

Honestly the only things he really made me wait on was IPV6 & an IPMI drop. I learned to start talking to Don/etc directly to speed things up if I'm impatient.

Francisco
 

iWF-Jacob

New Member
Verified Provider
You're lucky. We had to grab two /24's from them because we were in a bind back in April, and they still have not been SWIP'd. Last reply was over a month ago.

We have another ticket where we have some spam listings around said IPs that are causing us issues. That's been open since June 28th with no reply. 
 
Last edited by a moderator:

Francisco

Company Lube
Verified Provider
Fiberhub and power issues... The fun continues....
They've not had any since last year when they had an ATS crap out.

Honestly, i prefer the honesty and heads up. I've been with a few providers over the years

that had been on/off of generator power w/o telling any customers. I've had at least 1 other

that said they had generators when they didn't.

Francisco
 

drmike

100% Tier-1 Gogent
Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.

Hopefully utility upgrades along with facility investment put the power issue to bed permanently.

As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.
 

Francisco

Company Lube
Verified Provider
Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.

Hopefully utility upgrades along with facility investment put the power issue to bed permanently.

As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.
They installed a fairly beefy 2nd generator/UPS system when I was there last :) It's for their B side legs.

Fran
 

qps

Active Member
Verified Provider
They've not had any since last year when they had an ATS crap out.
You mean on May 15, 2014 when they replaced the ATS during business hours?  Temps in the data center spiked significantly because they had the AC offline for "a few minutes" - to the point where our router shutdown multiple times due to the heat.  

I will say that their remote hands has significantly improved recently.  
 

devonblzx

New Member
Verified Provider
You're lucky. We had to grab two /24's from them because we were in a bind back in April, and they still have not been SWIP'd. Last reply was over a month ago.

We have another ticket where we have some spam listings around said IPs that are causing us issues. That's been open since June 28th with no reply. 
That's not surprising to me, unless a server was down, I waited an average of 2 weeks for replies to tickets with Fiberhub when we were with them last year.   They just weren't very professional with communication from the whole process from sales to support.   Rob never even provided me with the proper Master Service Agreement then expected me to abide by it when we cancelled our services.

I had to call them repeatedly to get additional servers and a new switch setup.  After 3-4 weeks and ~4 phone calls and 2 tickets I was told they misplaced my switch.  They did credit my account, but the communication was just poor.
 
Last edited by a moderator:

Francisco

Company Lube
Verified Provider
I dunno.

Maybe having a cage makes things easier since there's plenty of space for storage so things

get taken right from the shipper right into our rack.

Francisco
 

devonblzx

New Member
Verified Provider
I dunno.


Maybe having a cage makes things easier since there's plenty of space for storage so things


get taken right from the shipper right into our rack.


Francisco
Possibly.  It probably would have worked better for me being closer and being able to do things myself.  However, I relied on them to do the work, I didn't mind paying the remote hands / rack n stack rates, but it was near impossible to get a response from them unless it was an emergency.  They did have good uptime when I was with them, so I guess I can't fault them for that.
 

VPSCorey

New Member
Verified Provider
the last datacenter I built we put in 25MW wroth of generators.  I have some generator porn somewhere of it.  10 2.5MW CAT gen's.  Nice and yellow.

It's a pretty cool setup as well.  Power fails, all start up and then the system starts shutting them down one at a time until the remaining generators are running at the proper load.
 

Rob T

New Member
Verified Provider
Just got this lovely email from Fiberhub. Glad their backup system is working so far, I'll be curious to see if it continues to do so!
I find your lack of faith disturbing.  :D

We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.

Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we choose to be a bit more transparent with our customers when we have something going on that most others do.
 
Last edited by a moderator:

iWF-Jacob

New Member
Verified Provider
I find your lack of faith disturbing.  :D

We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.

Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we chose to be a bit more transparent with our customers when we have something going on that most others do.
Hah! I did not know you frequented these forums, Rob! I lost no faith in you and your team, simply an idle question. Good job with the successful transfer!
 

Steven

New Member
That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.
I have to say in the past 6 months his communication has really stepped up. I haven't really had a single issue.

They've not had any since last year when they had an ATS crap out.


Honestly, i prefer the honesty and heads up. I've been with a few providers over the years


that had been on/off of generator power w/o telling any customers. I've had at least 1 other


that said they had generators when they didn't.


Francisco
We had a minor outage on B power due to an electrical contractor turning off the POST ups power. It affected one of my racks because the rack level ats is lazy and I run on B primary but it literally was under 10 seconds.

Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.

Hopefully utility upgrades along with facility investment put the power issue to bed permanently.

As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.
Most of their power issues in 2013 was bad luck IMHO. Ive been with them since Jan 2012 as a colo customer and the entire year of 2012 was flawless, not a single outage. The year 2013 had some issues but they were ALWAYS fast to respond to the power issues. There are providers who endured multi hour outages several times last year, so they are doing pretty good. With that said they have some massive infrastructure and cooling upgrades and we are continuously growing out foot print at Fiberhub.

They installed a fairly beefy 2nd generator/UPS system when I was there last :) It's for their B side legs.


Fran
Indeed. I can confirm I have physically seen this with my eyes. They were also installing additional cooling into their power room when I was there on the 3rd.

You mean on May 15, 2014 when they replaced the ATS during business hours?  Temps in the data center spiked significantly because they had the AC offline for "a few minutes" - to the point where our router shutdown multiple times due to the heat.  

I will say that their remote hands has significantly improved recently.  
No. Not that day. He is talking about 6-26-13 I am pretty sure. I will say this however, I checked our temperature monitoring right now and I do not see a significant increase in temperature that day that affected any of my gear. Not debating they may have done this but we didn't experience any issues with our cage space. It may have been specific to area in the DC. IDK.

I dunno.


Maybe having a cage makes things easier since there's plenty of space for storage so things


get taken right from the shipper right into our rack.


Francisco
Agreed. Cage space is awesome. I also use rack drawers in some of my racks to store small parts.

I find your lack of faith disturbing.  :D

We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.

Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we choose to be a bit more transparent with our customers when we have something going on that most others do.
Just want to say thanks for the updates. I was seriously contemplating driving to vegas while I was eating dinner incase there was an interruption, but I was relieved with your updates. Keep up the good work as usual.
 
Last edited by a moderator:

Francisco

Company Lube
Verified Provider
When we were with HE we had multiple multi hour outages because they screwed up their systems and once was on generator power for multiple days. Was incredibly stressful and is what caused EGI to pull most of their users out of there.

They recently hired on Anthony and he's a huge help. He's quite skilled in his work and has done some pretty annoying projects for us with minimal issues.

Network is solid, facility is clean, & everyone is a hoot to talk to & generous. When I was there to install the new router, my cab never showed up even though we called multiple times and gave almost 3 hours of lead up time. In the end Rob took time out of his afternoon to drive me over to the terminal.

Francisco
 

Steven

New Member
When we were with HE we had multiple multi hour outages because they screwed up their systems and once was on generator power for multiple days. Was incredibly stressful and is what caused EGI to pull most of their users out of there.


They recently hired on Anthony and he's a huge help. He's quite skilled in his work and has done some pretty annoying projects for us with minimal issues.


Network is solid, facility is clean, & everyone is a hoot to talk to & generous. When I was there to install the new router, my cab never showed up even though we called multiple times and gave almost 3 hours of lead up time. In the end Rob took time out of his afternoon to drive me over to the terminal.


Francisco
I refuse to take cabs there when I fly in. I rent a car, and rental return is literally around the block from the airport.
 
Last edited by a moderator:

Francisco

Company Lube
Verified Provider
I refuse to take cabs there when I fly in. I rent a car, and rental return is literally around the block from the airport.
I can't drive, but if I could I'd do that for sure.

Come next year i'll probably be spending 3 - 4 months of the year in LA which means trips to LV are quick enough that Matt & I can drive over for a weekend of whatever.

Francisco
 
Top
amuck-landowner