# NY CC blowed away



## Neo (Mar 13, 2014)

Hi,

Seems like CC in NY faced by some problems, lowendbox/lowendtalk/ipxcore is down.


----------



## Nett (Mar 13, 2014)

lol not surprised with CC


----------



## Francisco (Mar 13, 2014)

I thought LE was hosted in Chicago? or did *the cluster* require too much computing power?


----------



## Neo (Mar 13, 2014)

Seems like Cloudshards NY (Buffalo) is also down.


----------



## NodePing (Mar 13, 2014)

SSDvps NY (Buffalo) also down.


----------



## DomainBop (Mar 13, 2014)

*OMFG THE DAY CARE IS DOWN TOO!!!*

It's not just you! http://countrypark.com looks down from here.

Check another site?

...the site is hosted in Chicago so Buffalo isn't the only CC location having problems.


----------



## serverian (Mar 13, 2014)

Go go power rangers


----------



## manacit (Mar 13, 2014)

My dedicated server @ cc buf looks like it's been wiped off of the map

Sucks


----------



## GVH-Jon (Mar 13, 2014)

Looks like all GVH buffalo nodes are down as well. *sigh*


----------



## drmike (Mar 13, 2014)

What happened? Pretzel warmer in the food court burn the DC down?


----------



## Aldryic C'boas (Mar 13, 2014)

Can't wait to hear the excuse for this one


----------



## Awmusic12635 (Mar 13, 2014)

I have been calling for updates, nothing to report yet


----------



## serverian (Mar 13, 2014)

It's network. We know that much.


----------



## Neo (Mar 13, 2014)




----------



## AMDbuilder (Mar 13, 2014)

Nice, a movie to go with my dinner!

opcorn:


----------



## Nett (Mar 13, 2014)

An outsourced tech tripped over on the spaghetti cables...and it takes ages to find the cable


----------



## Francisco (Mar 13, 2014)

Rumor is that @scv poured lemonade on the cisco.

Francisco


----------



## Nett (Mar 13, 2014)

Let's wait for drama and excuses.


----------



## serverian (Mar 13, 2014)




----------



## drmike (Mar 13, 2014)

Hoodlum escaping from one of the many Foot Lockers, while fleeing with ill gotten sneakers, tripped on the extension cord for the datacenter.


----------



## GVH-Jon (Mar 13, 2014)

Confirmed CC buffalo is back up.


----------



## Francisco (Mar 13, 2014)

GVH-Jon said:


> Confirmed CC buffalo is back up.


Yep. Nodeping has a monitoring node there and it blasted off alerts for Jersey for us.

I don't expect much of an 'excuse', most likely just a hardware failure or possibly a screwup on an upstreams prefix lists/ACL's.

Francisco


----------



## Aldryic C'boas (Mar 13, 2014)

Spamhaus got tired of the BS and hired a 2bit booter


----------



## bzImage (Mar 13, 2014)

Now I know why I keep getting false alarm SMS messages from node ping.

23.94.101.96 - - [09/Mar/2014:09:06:09 -0700] "GET / HTTP/1.1" 302 2352 "-" "NodePing"
23.94.101.96 - - [09/Mar/2014:09:07:10 -0700] "GET / HTTP/1.1" 302 2352 "-" "NodePing"
23.94.101.96 - - [09/Mar/2014:09:08:09 -0700] "GET / HTTP/1.1" 302 2352 "-" "NodePing"
23.94.101.96 - - [09/Mar/2014:09:09:10 -0700] "GET / HTTP/1.1" 302 2352 "-" "NodePing"
 

ColoCrossing CC-16 (NET-23-94-0-0-1) 23.94.0.0 - 23.95.255.255


----------



## ocitysolutions (Mar 13, 2014)

Our network up there was offline for over an hour. This will be added to the list of reasons we are moving next weekend.


----------



## MannDude (Mar 13, 2014)

Cfab said on Twitter there will be a RFO posted, so lets wait for that. https://twitter.com/chicagovps/status/444289517156302848


----------



## drmike (Mar 13, 2014)

ocitysolutions said:


> Our network up there was offline for over an hour. This will be added to the list of reasons we are moving next weekend.


63 minutes of downtime.


----------



## raindog308 (Mar 13, 2014)

Francisco said:


> I thought LE was hosted in Chicago?


I though LET was running host on vanillaforums.com


----------



## Francisco (Mar 13, 2014)

raindog308 said:


> I though LET was running host on vanillaforums.com


No that's what "the cluster" thing that we poke at. They were hosted there but ate some fairly large floods.

On top of that, vanillaforums.com got hacked due to LET being hosted there.

Francisco


----------



## NodePing (Mar 13, 2014)

Francisco said:


> Yep. Nodeping has a monitoring node there and it blasted off alerts for Jersey for us.
> 
> 
> I don't expect much of an 'excuse', most likely just a hardware failure or possibly a screwup on an upstreams prefix lists/ACL's.
> ...


We don't have a production probe server in New York.


----------



## NodePing (Mar 13, 2014)

bzImage said:


> Now I know why I keep getting false alarm SMS messages from node ping.
> 
> 23.94.101.96 - - [09/Mar/2014:09:06:09 -0700] "GET / HTTP/1.1" 302 2352 "-" "NodePing"
> 
> ...


We don't have a probe in New York.  Those logs are from March 9.  Are they related to this outage?

We work hard to not send false alerts. If you're getting some, please let us know via a support ticket and we'll look into it.


----------



## Awmusic12635 (Mar 13, 2014)

> Tonight, while a technician was in the building meet me room completing the installation of additional fiber connections to service our Buffalo datacenter, an error which impacted a master patch panel occurred which caused an interruption of service to all segments of our network. ColoCrossing understands that you rely on our services and that any outage, for any reason, including human error is absolutely unacceptable. Immediately upon receiving notice of the network-down alert ColoCrossing's facilities team began investigating and took action to resolve the outage. In the coming days we will be conducting a full review and will institute changes to our policies to ensure that a similar event cannot impact our network in the future. We appreciate your business and regret any frustration caused.


----------



## concerto49 (Mar 13, 2014)

Apparently someone tripped over the fiber room.


----------



## peterw (Mar 14, 2014)

concerto49 said:


> Apparently someone tripped over the fiber room.


Or needed power for his coffee machine.


----------



## Nett (Mar 14, 2014)

Nah, Biloh forgot to pay the tech salary and the tech decided to cut the network.


----------



## joshj889 (Mar 14, 2014)

drmike said:


> What happened? Pretzel warmer in the food court burn the DC down?


Probably a "former" BuyVM employee taking a leak on their network gear.


----------



## shovenose (Mar 14, 2014)

Wait... is the Buffalo, NY location actually located IN a mall?


----------



## DomainBop (Mar 14, 2014)

shovenose said:


> Wait... is the Buffalo, NY location actually located IN a mall?


store directory http://www.mainliberty.com/public/files/MPM%20Tenant%20Directory%202.pdf



> Main Place Tower is home to Downtown Buffalo’s only indoor mall and food court
> 
> *Main Place*
> 350 Main Street
> ...


----------



## MannDude (Mar 14, 2014)

shovenose said:


> Wait... is the Buffalo, NY location actually located IN a mall?


Kind of. In the same building, but not like... in a store "in" the mall.



DomainBop said:


> store directory http://www.mainliberty.com/public/files/MPM%20Tenant%20Directory%202.pdf


Shitty mall.


----------



## Aldryic C'boas (Mar 14, 2014)

MannDude said:


> Shitty mall.


Understatement of the year -_-;


----------



## shovenose (Mar 14, 2014)

Haha I can just imagine seeing ColoCrossing on the list..

http://www.shopatnorthgate.com/Map/

That's the mall nearest to me.

Would be funny!

But if it's not a store in the mall where is it? Like attached?


----------



## ocitysolutions (Mar 14, 2014)

shovenose said:


> But if it's not a store in the mall where is it? Like attached?


In the base floor of the office tower the mall is attached to. Walk from the mall into the tower lobby and you're at CC's door.


----------



## NodeWest-Dan (Mar 15, 2014)

shovenose said:


> Haha I can just imagine seeing ColoCrossing on the list..
> 
> http://www.shopatnorthgate.com/Map/
> 
> ...


Disregard. Read the address.


----------

