Colocrossing's Core "Router" is a Switch

[drmike]

Some fun info about Colocrossing for today.

The reason Colocrossing can't offer IPv6 and has delayed setting up BGP sessions
(See Crissic's LA departure thread) is because they don't actually have a large 
core router.

Instead, they use switches similar to the photo below as their "edge network". These
routers can bring up a BGP session but they can't hold "an internet worth of routes".
These switches top out around ~10,000 total routes.



So how does Colocrossing get around this? They use multipath between all of their peers.
Multipath does round-robin blasting packets down each peer in equal weight. Ever
wonder why you have really ugly traceroutes like this?










3 host.colocrossing.com (192.3.94.137) 0.652 ms host.colocrossing.com (192.3.94.133) 0.708 ms host.colocrossing.com (192.3.94.137) 0.826 ms
4 207.86.157.13 (207.86.157.13) 0.288 ms 0.287 ms buf-b1-link.telia.net (213.248.96.41) 0.326 ms
5 nyk-bb1-link.telia.net (80.91.246.37) 9.606 ms 9.604 ms 9.545 ms
6 tinet.yyz02.atlas.cogentco.com (154.54.13.74) 12.793 ms 12.846 ms nyk-b3-link.telia.net (80.91.245.80) 9.808 ms
7 tmobile-ic-302276-war-b1.c.telia.net (213.248.83.118) 27.436 ms xe-4-3-0.atl11.ip4.tinet.net (141.136.108.134) 49.765 ms xe-8-0-0.atl11.ip4.tinet.net (141.136.108.142) 49.862 ms
8 eth2-1.r1.ash1.us.atrato.net (78.152.34.117) 21.874 ms total-server-solutions-gw.ip4.tinet.net (173.241.130.54) 49.968 ms eth2-1.r1.ash1.us.atrato.net (78.152.34.117) 23.592 ms
9 eth3-1.r1.atl1.us.atrato.net (78.152.34.181) 46.245 ms total-server-solutions-gw.ip4.tinet.net (173.241.130.54) 61.983 ms


Notice how you have Cogent & Telia on the same hop, as well as Atrato once you start getting to Colo@? That's because each packet of the traceroute is going through a different peer in round robin.

If they were running full routing tables they would have a single path through a single provider to get to a
destination (since that ISP/path would have been picked to be the best, assuming they aren't doing cost based balancing).

It's pretty sad and this setup actually causes random speed issues for some users, especially Comcast customers. You'll have good speed one one transfer then garbage the next.

I'm not sure where they "invested" $1,000,000 into their network but it's obviously a lie.

 

[Tux]

The daycare center in Buffalo has an awful setup... it's over an bonded DSL line

 

[zzrok]

I don't understand.  If what you say is true, why is hop 5 always the same?  It looks to me like telia is to blame, but I'm not a networking expert.

 

[Francisco]

I don't understand.  If what you say is true, why is hop 5 always the same?  It looks to me like telia is to blame, but I'm not a networking expert.

It isn't.

3 host.colocrossing.com (192.3.94.137) 1.743 ms host.colocrossing.com (192.3.94.133) 1.162 ms . (172.245.12.225) 1.756 ms
4 buf-b1-link.telia.net (213.248.96.41) 0.422 ms buff-b1-link.telia.net (62.115.34.137) 0.419 ms te7-4.ccr01.buf02.atlas.cogentco.com (38.122.36.45) 10.177 ms
5 te8-8.ccr02.cle04.atlas.cogentco.com (154.54.31.237) 23.280 ms nyk-bb1-link.telia.net (80.91.246.37) 9.988 ms te8-8.ccr02.cle04.atlas.cogentco.com (154.54.31.237) 23.260 ms
There's nothing wrong with multipath and it was going to be what we did in LV, but we have a much smaller deployment than what makes up all of CC's Buffalo setup.
I'd really hope they had full tables but I dunno...

Francisco

 

[TheLinuxBug]

Usually I am on board with your threads, however, what you have said here is a bit confusing and I am not sure if you understand correctly.  If they are indeed doing round robin, your first hop outside their network is what would change, not routes throughout a traceroute.  When it picks a path you would see the route directly after their core "router" would change on a per provider basis.  What you are showing above just looks like poor routing on Telia's behalf.

Please correct me if I am wrong.

Cheers!

 

[Francisco]

Usually I am on board with your threads, however, what you have said here is a bit confusing and I am not sure if you understand correctly.  If they are indeed doing round robin, your first hop outside their network is what would change, not routes throughout a traceroute.  When it picks a path you would see the route directly after their core "router" would change on a per provider basis.  What you are showing above just looks like poor routing on Telia's behalf.

Please correct me if I am wrong.

Cheers!

Nope.

If they have a full table in place formed from all of their upstreams then you'd never see the same hop have different upstreams like we do in this.

With multipath you are actually taking multiple default routes and then the switching/routing platform will round robin it.

Sometimes you'll see all the packets go out the same path but that's just luck. Remember, CC is pushing a lot of transit and a ton of PPS so the hashing/round robin'ing is constantly hammering away.

Francisco

 

[zzrok]

I still don't understand how, in the first post, hop 5 is the same each time, but it is different after that point.  Isn't it up to telia which route is taken after hop 5?  How can CC determine (influence) the route of the packet beyond the first router?

 

[Francisco]

I still don't understand how, in the first post, hop 5 is the same each time, but it is different after that point.  Isn't it up to telia which route is taken after hop 5?  How can CC determine the route of the packet beyond the first router?

In a normal BGP setup? Yes. In multipath? Maybe.

Normally multipath is used as an alternative to bonding interfaces. You have 40gbit from a single upstream? You'd setup 4 different BGP sessions with them and the packets would multipath between it giving you your 40gbit/sec upstream without the cost of running 100gbit connections and crap like that.

When you multipath between different providers, though, you end up getting really funny routes since packets will get switched between all members.

Notice my later trace which shows hop 5 hit 2 - 3 different providers. The original trace showing hop 5 all hitting telia was just a luck of the round-robin draw.

Again, I can't confirm what they're using, all I know is I got a BGP session with them for our ASN.

Francisco

 

[weservit]

Actually when you use multiple TIER1 carriers most paths will have the same length. Example:

When you do a BGP lookup for this IP 194.25.0.125 (an IP from Deutsche Telekom) at all TIER1 carriers you will get a direct path to AS3320. So when you use multiple TIER1 carriers like they do (Cogent, Telia) you will get 2 direct routes to AS3320. Without multipath BGP it's very hard to balance traffic at your uplinks without traffic engineering. So you don't make optimal use of your transit capacity.

Multipath BGP doesn't mean that you always get balanced routes, when the AS path is shorter at carrier1 than carrier2, traffic will go over carrier1. But when the path has the same AS length at carrier1 and carrier2 multipath BGP will balance it between these carriers.

 

[Francisco]

True, multipath is a sweet feature

But I can't see 4 different ISP's all having the same length route to COLO@.

Francisco

 

[weservit]

True, multipath is a sweet feature


But I can't see 4 different ISP's all having the same length route to COLO@.


Francisco

I did some BGP lookups to COLO@

Telia:

AS path: 5580 46562 40426 

Cogent:

AS path: 3257 46562 40426

As you can see the AS length is the same, when one of these had a shorter AS path you probably wouldn't have a multipath route.

Another advantage of multipath BGP is that you won't have a 100% outage to a specific route when one of your carriers fails. As soon as one carrier fails BGP has to relearn the routes to find another path, when you don't use multipath BGP you will experience a complete loss for a short time depending on how fast your router will relearn the routes. 

 

[MannDude]

So... pardon my ignorance on this one, how does this negatively impact the customer? Does it? Or is this just proof that their 'million dollar network upgrade' didn't happen or is more hot air from CC?

I'm not happy that we're not offering IPV6 yet, but its not as if there is no reason behind it. About six months ago we completed a million dollar upgrade to our switch infrastructure installing all new TOR devices and connected each back to our distribution layer by between 20 or 40 gbit. Those devices, Brocade ICX6450 are waiting for a software update to support OSPFv3 (for ipv6). Once that is released, which is currently a few months late, we'll be pushing forward for IPV6. People ask us why not just deliver IPv6 to colocation customers now (which we could, because we use Junipers at the distribution layer) the thing is that wouldn't be fair to our 60% of dedicated customers.

Src: http://lowendtalk.com/discussion/comment/316739/#Comment_316739

 

[concerto49]

The Juniper's do support IPv6 though? I don't think that part is the problem. I mean they can always take a default route from 1 carrier even and get IPv6 up IF that was the problem.

 

[Francisco]

So... pardon my ignorance on this one, how does this negatively impact the customer? Does it? Or is this just proof that their 'million dollar network upgrade' didn't happen or is more hot air from CC?

Src: http://lowendtalk.com/discussion/comment/316739/#Comment_316739

It just leads to wonky routes and possibly wonky speeds if you're on an ISP that's too cheap to pay for ports or unwilling to work out a peering deal (Comcast/Verizon/Rogers/Bell pretty much).

2 different traces to google.com show hops against all 4 upstreams with a latency different from 30ms <> 63ms at the final destination.

It's possible BUF is still due a big ass upgrade and is delayed due to how much of a headache it'll be. I'm pretty sure CC finished BUF up at the start of the year though.

Do they use their own upstream contracts in any other locations? Maybe CHI? I figure the others are just defaults from coloat/quadranet?.

Francisco

 

[drmike]

I'm sure glad and quiet on this   Beyond my knowledge base.  I don't monkey at that level of the network.

I've seen the issues in CC's network for eons and posted ample exampls of the wonkiness.  Never could quite figure out what the gremlin in the machine was.

Thanks to everyone participating with the technical.

 

[Tux]

Do they use their own upstream contracts in any other locations? Maybe CHI? I figure the others are just defaults from coloat/quadranet?

I think this is only the case in Atlanta and perhaps San Jose. Everywhere else (excluding Buffalo) gets the DC blend.

 

[Francisco]

I think this is only the case in Atlanta and perhaps San Jose. Everywhere else (excluding Buffalo) gets the DC blend.

Yep I remember Jon mentioning having a lot of nlayer in SJ.

Coloat's inhouse blend is nice so it wouldn't be a bad thing to use.

Francisco

 

[MannDude]

Yep I remember Jon mentioning having a lot of nlayer in SJ.

Coloat's inhouse blend is nice so it wouldn't be a bad thing to use.

Francisco

Which of Colocrossing's locations are actually Colo@? I see Colo@ operates a lot of locations CC does, but unsure which locations of their's Colocrossing rents from.

I only ask because I know in the past people have listed 'Quadranet' as their DC when they're using Colocrossing in LA, which I guess isn't technically untrue, just a bit misleading. I don't think I've seen someone say "Colo@" in place of CC, though. (Yet)

 

[drmike]

Colo@ in.... Atlanta, Dallas....  I think.... 

 

[Francisco]

Dallas is the odd one. From what was said the bandwidth is with Quadranet where as the racks are directly with...colo4? colo4dallas? Something like that.

Francisco