Colocrossing's Core "Router" is a Switch

Discussion in 'The Pub (Off topic discussion)' started by drmike, Sep 11, 2013.

  1. drmike

    drmike 100% Tier-1 Gogent

    8,573
    2,717
    May 13, 2013
    Some fun info about Colocrossing for today.

    The reason Colocrossing can't offer IPv6 and has delayed setting up BGP sessions
    (See Crissic's LA departure thread) is because they don't actually have a large 
    core router.

    Instead, they use switches similar to the photo below as their "edge network". These
    routers can bring up a BGP session but they can't hold "an internet worth of routes".
    These switches top out around ~10,000 total routes.

    [​IMG]

    So how does Colocrossing get around this? They use multipath between all of their peers.
    Multipath does round-robin blasting packets down each peer in equal weight. Ever
    wonder why you have really ugly traceroutes like this?










    3 host.colocrossing.com (192.3.94.137) 0.652 ms host.colocrossing.com (192.3.94.133) 0.708 ms host.colocrossing.com (192.3.94.137) 0.826 ms
    4 207.86.157.13 (207.86.157.13) 0.288 ms 0.287 ms buf-b1-link.telia.net (213.248.96.41) 0.326 ms
    5 nyk-bb1-link.telia.net (80.91.246.37) 9.606 ms 9.604 ms 9.545 ms
    6 tinet.yyz02.atlas.cogentco.com (154.54.13.74) 12.793 ms 12.846 ms nyk-b3-link.telia.net (80.91.245.80) 9.808 ms
    7 tmobile-ic-302276-war-b1.c.telia.net (213.248.83.118) 27.436 ms xe-4-3-0.atl11.ip4.tinet.net (141.136.108.134) 49.765 ms xe-8-0-0.atl11.ip4.tinet.net (141.136.108.142) 49.862 ms
    8 eth2-1.r1.ash1.us.atrato.net (78.152.34.117) 21.874 ms total-server-solutions-gw.ip4.tinet.net (173.241.130.54) 49.968 ms eth2-1.r1.ash1.us.atrato.net (78.152.34.117) 23.592 ms
    9 eth3-1.r1.atl1.us.atrato.net (78.152.34.181) 46.245 ms total-server-solutions-gw.ip4.tinet.net (173.241.130.54) 61.983 ms


    Notice how you have Cogent & Telia on the same hop, as well as Atrato once you start getting to [email protected]? That's because each packet of the traceroute is going through a different peer in round robin.

    If they were running full routing tables they would have a single path through a single provider to get to a
    destination (since that ISP/path would have been picked to be the best, assuming they aren't doing cost based balancing).

    It's pretty sad and this setup actually causes random speed issues for some users, especially Comcast customers. You'll have good speed one one transfer then garbage the next.

    I'm not sure where they "invested" $1,000,000 into their network but it's obviously a lie.
     
    Last edited by a moderator: Apr 30, 2017
    InertiaNetworks-John likes this.
  2. Tux

    Tux DigitialOcean? lel

    217
    37
    May 18, 2013
    The daycare center in Buffalo has an awful setup... it's over an bonded DSL line
     
    drmike likes this.
  3. zzrok

    zzrok New Member

    104
    41
    Jul 14, 2013
    I don't understand.  If what you say is true, why is hop 5 always the same?  It looks to me like telia is to blame, but I'm not a networking expert.
     
    Last edited by a moderator: Sep 11, 2013
  4. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    It isn't.

    3 host.colocrossing.com (192.3.94.137) 1.743 ms host.colocrossing.com (192.3.94.133) 1.162 ms . (172.245.12.225) 1.756 ms
    4 buf-b1-link.telia.net (213.248.96.41) 0.422 ms buff-b1-link.telia.net (62.115.34.137) 0.419 ms te7-4.ccr01.buf02.atlas.cogentco.com (38.122.36.45) 10.177 ms
    5 te8-8.ccr02.cle04.atlas.cogentco.com (154.54.31.237) 23.280 ms nyk-bb1-link.telia.net (80.91.246.37) 9.988 ms te8-8.ccr02.cle04.atlas.cogentco.com (154.54.31.237) 23.260 ms
    There's nothing wrong with multipath and it was going to be what we did in LV, but we have a much smaller deployment than what makes up all of CC's Buffalo setup.
    I'd really hope they had full tables but I dunno...

    Francisco
     
    Last edited by a moderator: Sep 11, 2013
  5. TheLinuxBug

    TheLinuxBug New Member

    402
    356
    May 15, 2013
    Usually I am on board with your threads, however, what you have said here is a bit confusing and I am not sure if you understand correctly.  If they are indeed doing round robin, your first hop outside their network is what would change, not routes throughout a traceroute.  When it picks a path you would see the route directly after their core "router" would change on a per provider basis.  What you are showing above just looks like poor routing on Telia's behalf.

    Please correct me if I am wrong.

    Cheers!
     
  6. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    Nope.

    If they have a full table in place formed from all of their upstreams then you'd never see the same hop have different upstreams like we do in this.

    With multipath you are actually taking multiple default routes and then the switching/routing platform will round robin it.

    Sometimes you'll see all the packets go out the same path but that's just luck. Remember, CC is pushing a lot of transit and a ton of PPS so the hashing/round robin'ing is constantly hammering away.

    Francisco
     
    Last edited by a moderator: Sep 11, 2013
    TheLinuxBug likes this.
  7. zzrok

    zzrok New Member

    104
    41
    Jul 14, 2013
    I still don't understand how, in the first post, hop 5 is the same each time, but it is different after that point.  Isn't it up to telia which route is taken after hop 5?  How can CC determine (influence) the route of the packet beyond the first router?
     
    Last edited by a moderator: Sep 11, 2013
    mtwiscool likes this.
  8. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    In a normal BGP setup? Yes. In multipath? Maybe.

    Normally multipath is used as an alternative to bonding interfaces. You have 40gbit from a single upstream? You'd setup 4 different BGP sessions with them and the packets would multipath between it giving you your 40gbit/sec upstream without the cost of running 100gbit connections and crap like that.

    When you multipath between different providers, though, you end up getting really funny routes since packets will get switched between all members.

    Notice my later trace which shows hop 5 hit 2 - 3 different providers. The original trace showing hop 5 all hitting telia was just a luck of the round-robin draw.

    Again, I can't confirm what they're using, all I know is I got a BGP session with them for our ASN.

    Francisco
     
  9. weservit

    weservit New Member Verified Provider

    39
    10
    May 16, 2013
    Actually when you use multiple TIER1 carriers most paths will have the same length. Example:

    When you do a BGP lookup for this IP 194.25.0.125 (an IP from Deutsche Telekom) at all TIER1 carriers you will get a direct path to AS3320. So when you use multiple TIER1 carriers like they do (Cogent, Telia) you will get 2 direct routes to AS3320. Without multipath BGP it's very hard to balance traffic at your uplinks without traffic engineering. So you don't make optimal use of your transit capacity.

    Multipath BGP doesn't mean that you always get balanced routes, when the AS path is shorter at carrier1 than carrier2, traffic will go over carrier1. But when the path has the same AS length at carrier1 and carrier2 multipath BGP will balance it between these carriers.
     
    Last edited by a moderator: Sep 11, 2013
    TheLinuxBug and drmike like this.
  10. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    True, multipath is a sweet feature :)

    But I can't see 4 different ISP's all having the same length route to [email protected]

    Francisco
     
  11. weservit

    weservit New Member Verified Provider

    39
    10
    May 16, 2013
    I did some BGP lookups to [email protected]

    Telia:

    AS path: 5580 46562 40426 

    Cogent:

    AS path: 3257 46562 40426

    As you can see the AS length is the same, when one of these had a shorter AS path you probably wouldn't have a multipath route.

    Another advantage of multipath BGP is that you won't have a 100% outage to a specific route when one of your carriers fails. As soon as one carrier fails BGP has to relearn the routes to find another path, when you don't use multipath BGP you will experience a complete loss for a short time depending on how fast your router will relearn the routes. 
     
    Last edited by a moderator: Sep 11, 2013
    drmike likes this.
  12. MannDude

    MannDude Just a dude vpsBoard Founder Moderator

    5,036
    2,634
    Mar 8, 2013
    MannDude
    So... pardon my ignorance on this one, how does this negatively impact the customer? Does it? Or is this just proof that their 'million dollar network upgrade' didn't happen or is more hot air from CC?

    Src: http://lowendtalk.com/discussion/comment/316739/#Comment_316739
     
    Last edited by a moderator: Sep 11, 2013
  13. concerto49

    concerto49 New Member Verified Provider

    960
    200
    May 5, 2013
    The Juniper's do support IPv6 though? I don't think that part is the problem. I mean they can always take a default route from 1 carrier even and get IPv6 up IF that was the problem.
     
  14. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    It just leads to wonky routes and possibly wonky speeds if you're on an ISP that's too cheap to pay for ports or unwilling to work out a peering deal (Comcast/Verizon/Rogers/Bell pretty much).

    2 different traces to google.com show hops against all 4 upstreams with a latency different from 30ms <> 63ms at the final destination.

    It's possible BUF is still due a big ass upgrade and is delayed due to how much of a headache it'll be. I'm pretty sure CC finished BUF up at the start of the year though.

    Do they use their own upstream contracts in any other locations? Maybe CHI? I figure the others are just defaults from coloat/quadranet?.

    Francisco
     
  15. drmike

    drmike 100% Tier-1 Gogent

    8,573
    2,717
    May 13, 2013
    I'm sure glad and quiet on this :)  Beyond my knowledge base.  I don't monkey at that level of the network.

    I've seen the issues in CC's network for eons and posted ample exampls of the wonkiness.  Never could quite figure out what the gremlin in the machine was.

    Thanks to everyone participating with the technical.
     
    Last edited by a moderator: Sep 11, 2013
    kro likes this.
  16. Tux

    Tux DigitialOcean? lel

    217
    37
    May 18, 2013
    I think this is only the case in Atlanta and perhaps San Jose. Everywhere else (excluding Buffalo) gets the DC blend.
     
  17. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    Yep I remember Jon mentioning having a lot of nlayer in SJ.

    Coloat's inhouse blend is nice so it wouldn't be a bad thing to use.

    Francisco
     
  18. MannDude

    MannDude Just a dude vpsBoard Founder Moderator

    5,036
    2,634
    Mar 8, 2013
    MannDude
    Which of Colocrossing's locations are actually [email protected]? I see [email protected] operates a lot of locations CC does, but unsure which locations of their's Colocrossing rents from.

    I only ask because I know in the past people have listed 'Quadranet' as their DC when they're using Colocrossing in LA, which I guess isn't technically untrue, just a bit misleading. I don't think I've seen someone say "[email protected]" in place of CC, though. (Yet)
     
  19. drmike

    drmike 100% Tier-1 Gogent

    8,573
    2,717
    May 13, 2013
  20. Francisco

    Francisco Company Lube Verified Provider

    2,476
    1,770
    May 15, 2013
    Dallas is the odd one. From what was said the bandwidth is with Quadranet where as the racks are directly with...colo4? colo4dallas? Something like that.

    Francisco