# Colocrossing Network down in Los Angeles and Other Locations



## drmike (Oct 15, 2014)

So, ColoCrossing appears to be having a rather large outage.

Roughly an hour or more downtime.   

Definitely out in Los Angeles at Quadranet.

Other locations folks are mumbling about being down in as well.

What are you seeing?


----------



## k0nsl (Oct 15, 2014)

Yep, I'm seeing it too. They don't know any details themselves, at least that's what I'm being told..


----------



## QuadraNet_Adam (Oct 15, 2014)

To confirm: QuadraNet is not having any power or network outages within any of our datacenters. Please keep in mind some colocation customers run their own network infrastructure.


----------



## k0nsl (Oct 15, 2014)

Here is one of their Los Angeles location:


----------



## Munzy (Oct 15, 2014)

I am seeing that as well.

http://uptime.munroenet.com/outages.php?name=Torrents


----------



## drmike (Oct 15, 2014)

k0nsl said:


> Yep, I'm seeing it too. They don't know any details themselves, at least that's what I'm being told..


Which location?



QuadraNet_Adam said:


> Please keep in mind some colocation customers run their own network infrastructure.


That's right kids, CC dumped Quadra's BW a few months back and has at last check almost single homed self rolled network.  I believe it was nLayer in one direction barely at times and mostly Zayo everywhere else.


----------



## aggressivenetworks (Oct 15, 2014)

Dam must be their switch they are using for a supposed router? Just speculation!


----------



## MannDude (Oct 15, 2014)

BlueVM is down in Chicago, Atlanta, LA, and New York: http://bluevm.linuxthefish.net/

Pretty much everywhere but Texas and Jersey.

GVH as well according to LET, unsure on locations from them but I'm sure someone can chime in. Probably the same.

Strange for it to be across multiple locations.


----------



## drmike (Oct 15, 2014)

aggressivenetworks said:


> Dam must be their switch they are using for a supposed router? Just speculation!




http://bayimg.com/iaaJIAaFN

http://bayimg.com/IAajKaAFn

http://bayimg.com/iAajlAAFn

http://bayimg.com/iaaJMAaFN

Question always has been if other locations were as poorly wired.  No doubt that they ran the same old school routers with same round robin balancing on uplinks.

They supposedly bought / upgraded in the past year though.  So the story went and photo op on their social stuff.  But I only saw one box, router... didn't see stuff for all their locations....   and I know they are still ghetto 100Mbit ports in Cali even though Fabozzi lies on sales of San Jose claiming Gbit.


----------



## drmike (Oct 15, 2014)

MannDude said:


> BlueVM is down in Chicago, Atlanta, LA, and New York: http://bluevm.linuxthefish.net/
> 
> Pretty much everywhere but Texas and Jersey.
> 
> ...



Fair and honest question, but are CHI, ATL, LA, and NY all locations where they converted to their own rolled network instead of using facility bandwidth?

I wager my bet on those being own rolled network locations.

I know LA is own rolled network as per earlier research and routes compared to the Quadra house blends.


----------



## k0nsl (Oct 15, 2014)

One thing's for sure, I must deal with this issue tomorrow when I'm fresh, after a proper sleep of four hours. Set up the stuff I use there at some reliable (read: reputable) provider - probably BuyBM or RamNode. It's just the main IRC node + KiwiIRC and some other insignificant services, but the main node can't cope with this each and every time. Sometimes it works, the next day it's rubbish.

The little user base I have are all complaining over unstable network. Sometimes its not that easy to run 25+ IRC nodes on a one man operation...at least not for me, I could use that time for productive stuff rather than having to deal with constant nonsense.

I'm going to force myself to re-do the entire set up tomorrow, shouldn't take so long. In the end, it'll be worth the hassle


----------



## DomainBop (Oct 15, 2014)

MannDude said:


> BlueVM is down in Chicago, Atlanta, LA, and New York: http://bluevm.linuxthefish.net/


BlueVM is apparently always down in those locations.  Every single one of their nodes in all 6 locations on that uptime page have under 99% uptime.


----------



## TruvisT (Oct 15, 2014)

Ah, so this is why there was a big drop in spam.


----------



## Aldryic C'boas (Oct 15, 2014)

Maybe dropping entire locations is how they were able to get some of those SBLs removed.


----------



## DomainBop (Oct 15, 2014)

Maybe the high volume of SPAM that ROKSO Yair Shalev sent out from his /25 broke their network today.


----------



## drmike (Oct 15, 2014)

Weird, one CC provider said NTT was responsible for this.

I don't know CC to use NTT.  They certainly haven't recently at least in LA.  Blend out there for them looks to be HE and Zayo.


----------



## drmike (Oct 15, 2014)

DomainBop said:


> Maybe the high volume of SPAM that ROKSO Yair Shalev sent out from his /25 broke their network today.


How many times is he going to appear on their network  ?

http://www.spamhaus.org/rokso/sbl_archived/SPM1134/yair-shalev-kobeni-solutions

 velocity-servers.net  = 65 matches.


----------



## HalfEatenPie (Oct 15, 2014)

QuadraNet_Adam said:


> To confirm: QuadraNet is not having any power or network outages within any of our datacenters. Please keep in mind some colocation customers run their own network infrastructure.


I can confirm Quadranet's Network is perfectly fine and has not had any power outages either.  

I have a dedi with a provider that owns a rack in Quadranet and no network or power failure since I've been with him (which has been a while).


----------



## HalfEatenPie (Oct 15, 2014)

Maybe CC is pulling a BurstNET+VolumeDrive!!!


----------



## DomainBop (Oct 15, 2014)

drmike said:


> Weird, one CC provider said NTT was responsible for this.
> 
> I don't know CC to use NTT.  They certainly haven't recently at least in LA.  Blend out there for them looks to be HE and Zayo.


The CC provider who said that is a 16 yr old habitual liar with very limited technical knowledge.   NTT wasn't responsible.






> *Bye-Low on WHT*
> 
> 
> 
> ...


----------



## trewq (Oct 16, 2014)

"occurred simultaneously in about 9% of our cabinets, across 5 different sites."


Wait, what? Did he just say pigs could fly or am I reading it wrong?


----------



## zafouhar (Oct 16, 2014)

Where is the FOR IMMEDIATE RELEASE? seems he forgot it


----------



## rds100 (Oct 16, 2014)

trewq said:


> "occurred simultaneously in about 9% of our cabinets, across 5 different sites."
> 
> 
> Wait, what? Did he just say pigs could fly or am I reading it wrong?


Maybe they started using a bunch of home belkin routers as access switches or something ;-)


----------



## serverian (Oct 16, 2014)

I'm new to this. Can someone explain?


----------



## Aldryic C'boas (Oct 16, 2014)

serverian said:


> I'm new to this. Can someone explain?


 "Business as Usual."


----------



## drmike (Oct 16, 2014)

"_access layer switch issue that occurred simultaneously in about 9% of our cabinets, across 5 different sites. The problem is almost fully resolved at this time."_

I am new here too 

What the hark is an access layer switch and why is someone poking a mass of them in cabinets all over the country?

Is he talking about those switches they had doing routing previously?!?!?!?!?!?!?


----------



## Francisco (Oct 16, 2014)

Only way I could see that happening is if they have a config file distro system and someone FUBAR'd a VLAN/port and broke things pretty nasty.

Francisco


----------



## tdale (Oct 16, 2014)

@Francisco

That is probably what happened. You remember how the CC panel works, it was probably some bad config that was pushed through the API. I'm sure Alex was in an uproar over that one.

-Tom


----------



## concerto49 (Oct 17, 2014)

9%? Like 10.12 cabinets?


----------



## Francisco (Oct 17, 2014)

concerto49 said:


> 9%? Like 10.12 cabinets?





concerto49 said:


> 9%? Like 10.12 cabinets?


I know Jon had 1/4 cabs for sale in Buffalo...Maybe he has 1/8th cabs? 

Francisco


----------

