# Fiberhub running on generator power



## iWF-Jacob (Jul 22, 2014)

Just got this lovely email from Fiberhub. Glad their backup system is working so far, I'll be curious to see if it continues to do so!



> Emergency Maintenance Notification
> 
> Start Time:  Tuesday, July 22 2014 4:00PM Pacific Time
> 
> ...


----------



## Francisco (Jul 22, 2014)

I already talked to Rob about it and they were ready for this due to some upgrades their power company was already working on.

Francisco


----------



## iWF-Jacob (Jul 22, 2014)

Francisco said:


> I already talked to Rob about it and they were ready for this due to some upgrades their power company was already working on.
> 
> 
> Francisco


That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.


----------



## Francisco (Jul 22, 2014)

iWF-Jacob said:


> That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.


I found spamming him on google hangouts works great 

Honestly the only things he really made me wait on was IPV6 & an IPMI drop. I learned to start talking to Don/etc directly to speed things up if I'm impatient.

Francisco


----------



## iWF-Jacob (Jul 22, 2014)

You're lucky. We had to grab two /24's from them because we were in a bind back in April, and they still have not been SWIP'd. Last reply was over a month ago.

We have another ticket where we have some spam listings around said IPs that are causing us issues. That's been open since June 28th with no reply.


----------



## drmike (Jul 22, 2014)

Fiberhub and power issues... The fun continues....


----------



## Francisco (Jul 22, 2014)

drmike said:


> Fiberhub and power issues... The fun continues....


They've not had any since last year when they had an ATS crap out.

Honestly, i prefer the honesty and heads up. I've been with a few providers over the years

that had been on/off of generator power w/o telling any customers. I've had at least 1 other

that said they had generators when they didn't.

Francisco


----------



## drmike (Jul 22, 2014)

Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.

Hopefully utility upgrades along with facility investment put the power issue to bed permanently.

As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.


----------



## Francisco (Jul 22, 2014)

drmike said:


> Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.
> 
> Hopefully utility upgrades along with facility investment put the power issue to bed permanently.
> 
> As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.


They installed a fairly beefy 2nd generator/UPS system when I was there last  It's for their B side legs.

Fran


----------



## qps (Jul 22, 2014)

Francisco said:


> They've not had any since last year when they had an ATS crap out.


You mean on May 15, 2014 when they replaced the ATS during business hours?  Temps in the data center spiked significantly because they had the AC offline for "a few minutes" - to the point where our router shutdown multiple times due to the heat.  

I will say that their remote hands has significantly improved recently.


----------



## devonblzx (Jul 22, 2014)

iWF-Jacob said:


> You're lucky. We had to grab two /24's from them because we were in a bind back in April, and they still have not been SWIP'd. Last reply was over a month ago.
> 
> We have another ticket where we have some spam listings around said IPs that are causing us issues. That's been open since June 28th with no reply.


That's not surprising to me, unless a server was down, I waited an average of 2 weeks for replies to tickets with Fiberhub when we were with them last year.   They just weren't very professional with communication from the whole process from sales to support.   Rob never even provided me with the proper Master Service Agreement then expected me to abide by it when we cancelled our services.

I had to call them repeatedly to get additional servers and a new switch setup.  After 3-4 weeks and ~4 phone calls and 2 tickets I was told they misplaced my switch.  They did credit my account, but the communication was just poor.


----------



## Francisco (Jul 22, 2014)

I dunno.

Maybe having a cage makes things easier since there's plenty of space for storage so things

get taken right from the shipper right into our rack.

Francisco


----------



## devonblzx (Jul 22, 2014)

Francisco said:


> I dunno.
> 
> 
> Maybe having a cage makes things easier since there's plenty of space for storage so things
> ...


Possibly.  It probably would have worked better for me being closer and being able to do things myself.  However, I relied on them to do the work, I didn't mind paying the remote hands / rack n stack rates, but it was near impossible to get a response from them unless it was an emergency.  They did have good uptime when I was with them, so I guess I can't fault them for that.


----------



## VPSCorey (Jul 22, 2014)

the last datacenter I built we put in 25MW wroth of generators.  I have some generator porn somewhere of it.  10 2.5MW CAT gen's.  Nice and yellow.

It's a pretty cool setup as well.  Power fails, all start up and then the system starts shutting them down one at a time until the remaining generators are running at the proper load.


----------



## Rob T (Jul 23, 2014)

iWF-Jacob said:


> Just got this lovely email from Fiberhub. Glad their backup system is working so far, I'll be curious to see if it continues to do so!


I find your lack of faith disturbing.  

We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.

Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we choose to be a bit more transparent with our customers when we have something going on that most others do.


----------



## iWF-Jacob (Jul 23, 2014)

Rob T said:


> I find your lack of faith disturbing.
> 
> We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.
> 
> Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we chose to be a bit more transparent with our customers when we have something going on that most others do.


Hah! I did not know you frequented these forums, Rob! I lost no faith in you and your team, simply an idle question. Good job with the successful transfer!


----------



## Steven (Jul 23, 2014)

iWF-Jacob said:


> That's good to know. I like Rob, I really do, but his hit and miss communication style sometimes drives me batty.


I have to say in the past 6 months his communication has really stepped up. I haven't really had a single issue.



Francisco said:


> They've not had any since last year when they had an ATS crap out.
> 
> 
> Honestly, i prefer the honesty and heads up. I've been with a few providers over the years
> ...


We had a minor outage on B power due to an electrical contractor turning off the POST ups power. It affected one of my racks because the rack level ats is lazy and I run on B primary but it literally was under 10 seconds.



drmike said:


> Fiberhub indeed has had a better year this current one.... Just reoccurring power related issues no matter how far we stretch the calendar.
> 
> Hopefully utility upgrades along with facility investment put the power issue to bed permanently.
> 
> As for other facilities and their crazy stories, yeppers, plenty of bad actors.  We should demolish such story tellers for lousy fiction when they fail and get busted in the process.


Most of their power issues in 2013 was bad luck IMHO. Ive been with them since Jan 2012 as a colo customer and the entire year of 2012 was flawless, not a single outage. The year 2013 had some issues but they were ALWAYS fast to respond to the power issues. *There are providers who endured multi hour outages several times last year*, so they are doing pretty good. With that said they have some massive infrastructure and cooling upgrades and we are continuously growing out foot print at Fiberhub.



Francisco said:


> They installed a fairly beefy 2nd generator/UPS system when I was there last  It's for their B side legs.
> 
> 
> Fran


Indeed. I can confirm I have physically seen this with my eyes. They were also installing additional cooling into their power room when I was there on the 3rd.



qps said:


> You mean on May 15, 2014 when they replaced the ATS during business hours?  Temps in the data center spiked significantly because they had the AC offline for "a few minutes" - to the point where our router shutdown multiple times due to the heat.
> 
> I will say that their remote hands has significantly improved recently.


No. Not that day. He is talking about 6-26-13 I am pretty sure. I will say this however, I checked our temperature monitoring right now and I do not see a significant increase in temperature that day that affected any of my gear. Not debating they may have done this but we didn't experience any issues with our cage space. It may have been specific to area in the DC. IDK.



Francisco said:


> I dunno.
> 
> 
> Maybe having a cage makes things easier since there's plenty of space for storage so things
> ...


Agreed. Cage space is awesome. I also use rack drawers in some of my racks to store small parts.



Rob T said:


> I find your lack of faith disturbing.
> 
> We're 100% back on utility power with ZERO impact to customer power delivery.  All backup power systems functioned as expected.
> 
> Datacenters lose utility power all the time, folks - that's why they have UPS and Generator backups.  The only difference is that we choose to be a bit more transparent with our customers when we have something going on that most others do.


Just want to say thanks for the updates. I was seriously contemplating driving to vegas while I was eating dinner incase there was an interruption, but I was relieved with your updates. Keep up the good work as usual.


----------



## Francisco (Jul 23, 2014)

When we were with HE we had multiple multi hour outages because they screwed up their systems and once was on generator power for multiple days. Was incredibly stressful and is what caused EGI to pull most of their users out of there.

They recently hired on Anthony and he's a huge help. He's quite skilled in his work and has done some pretty annoying projects for us with minimal issues.

Network is solid, facility is clean, & everyone is a hoot to talk to & generous. When I was there to install the new router, my cab never showed up even though we called multiple times and gave almost 3 hours of lead up time. In the end Rob took time out of his afternoon to drive me over to the terminal.

Francisco


----------



## Steven (Jul 23, 2014)

Francisco said:


> When we were with HE we had multiple multi hour outages because they screwed up their systems and once was on generator power for multiple days. Was incredibly stressful and is what caused EGI to pull most of their users out of there.
> 
> 
> They recently hired on Anthony and he's a huge help. He's quite skilled in his work and has done some pretty annoying projects for us with minimal issues.
> ...


I refuse to take cabs there when I fly in. I rent a car, and rental return is literally around the block from the airport.


----------



## Francisco (Jul 23, 2014)

Steven said:


> I refuse to take cabs there when I fly in. I rent a car, and rental return is literally around the block from the airport.


I can't drive, but if I could I'd do that for sure.

Come next year i'll probably be spending 3 - 4 months of the year in LA which means trips to LV are quick enough that Matt & I can drive over for a weekend of whatever.

Francisco


----------



## Rob T (Jul 23, 2014)

Getting a cab from the airport to the datacenter - piece of cake.

Getting a cab back to the airport - you'd be better off hitchhiking.

Vegas cabbies are a complete pain in the butt.  Bring on Uber!!!


----------



## concerto49 (Jul 23, 2014)

Francisco said:


> I can't drive, but if I could I'd do that for sure.
> 
> 
> Come next year i'll probably be spending 3 - 4 months of the year in LA which means trips to LV are quick enough that Matt & I can drive over for a weekend of whatever.
> ...


What are you staying there for?


----------



## drmike (Jul 23, 2014)

concerto49 said:


> What are you staying there for?


Global domination I suspect  Or tired of the frosty north.


----------



## DomainBop (Jul 23, 2014)

concerto49 said:


> What are you staying there for?


new location:

New York basement   New Jersey - shared underground space with Jimmy Hoffa in the Meadowlands

Vegas - casino parking garage

Los Angeles - San Andreas fault line


----------



## Francisco (Jul 23, 2014)

concerto49 said:


> What are you staying there for?


Honestly? Change of scenery.

A friend of mine is going to be moving down there anyway so he's wanting me to come stay at

his place for the summer(s) so I figure why not? Personal costs come out about the same

and it makes it a lot cheaper & easy if I want to take a trip over to Vegas.

I'm going down his way for my birthday in the fall and i'll also be taking a trip over to Vegas

to install another PSU & management card in the router.



Rob T said:


> Getting a cab from the airport to the datacenter - piece of cake.
> 
> Getting a cab back to the airport - you'd be better off hitchhiking.
> 
> Vegas cabbies are a complete pain in the butt.  Bring on Uber!!!


Yep. Leaving the airport is easy since there's a huge stream of cabs waiting to pick you up.

Getting anything outside of the strip is a total pain in the ass. Hell, when I stay off strip

when I go for DC trips I usually get out of the hotel by 8:30 AM but don't get to the DC

till almost lunch time once I get breakfast & a cab ride.

If you're on strip you can always just walk to one of the casino's and they all have rings

in the front.

Francisco


----------



## MannDude (Jul 23, 2014)

Francisco said:


> Honestly? Change of scenery.
> 
> 
> A friend of mine is going to be moving down there anyway so he's wanting me to come stay at
> ...


Just be aware of the bus line 

Last year Miller and I got on the wrong bus. We only had like 8 blocks to walk but the bus went the opposite direction and dropped us off at the bus stop... where we had to await a new bus. Highlight was talking to someone about how hardcore Detroit was and being offered drugs as I frequently get offered when on vacation... must be the hair.


----------

