Colocrossing's Core "Router" is a Switch

[RyanD]

Los Angeles is Quadranet, but wondering if directly out there or Colo@ in the middle...  Let me look....

I can easily clear that up for you

ColoCrossing is only a client in our Atlanta location as is publicly known. We are not a "man in the middle" in any facility for them, they are only with us in the facility in which we wholly operate it. I think they work solely with facility operators. IE us (ATL), ServerCentral (CHI), Quadranet (LA),  Centralogic (Buffalo), and I have no idea where they are elsewhere.

We have a private suite in the Penthouse @ Quadranet, it was originally owned by another company and got bought by Quadranet. Only thing that changed for us is we picked up a port of IP transit from them to add into our blend

 

[RyanD]

As I was the principle person responsible for setting up the network during it's birth, I am going to go out on a limb and defend taking default routes. Routers/switches with large amounts of memory for routing are not cheap, years and years ago when the routing table was ~200k routes (with /24's mixed in there), a 6500 series with a decent sup and nice backplane bandwidth was about $40k plus (direct from my old vendor). 

On the other hand, a smaller, faster switch (high PPS) was cheaper and could do more for less money (trunking, hsrp, etc). We went the cheaper route. While I /wanted/ a fully loaded 6506 with sup720 (they were $25k new), but we could not justify the cost JUST for full bgp routes. 95% of dedicated customers did not care about if we had full BGP or not, as long as their service worked (and didn't lag).

Back in the 90s when I worked for AboveNet, the motto at IAD4 was 'Keep it simple', which is exactly what I used when I designed the existing network. OSPF with eBGP (default routes), we used communities to alter our localpref in some cases, though. It worked. 

One thing I did NOT agree with is mixing vendor equipment, it's nice to do layer2 on different vendors (most of the time) but firmware bugs, broken RFC agreements made it difficult. I despised foundry's gear as being total shit made by people who wear leafs as shoes. 

It's simple numbers at the end of the day, the needs of the many (simple setup, default routes) outweigh the needs of the few (full blown routing, MPLS, et al)

Not trying to discredit you or anything but those prices are high even by the standards of back then.

I mean in 2006 when we got our first 6500 with 2 x 16-port 1g (gbic) blades and 2 x 48-port 1g (copper) and sup720-3bxl it was only $19k.

Later when we added on 2 more SUP720's ran about 7k and WS-X6704's ran about 7k as well.  Now days the SUP720-3bxl is still a very serviceable sup (we still run about a dozen of them). Convergence times stink but othewise they chug right along. 

Even if you were looking for cost effective edge routing the 6500s w/7203bxl still aren't a bad choice today. You can deploy a 6506 w/8 x 10G and 96 x 1G (copper) for about $8k. If you wanted to load it out with 5 x 4-Port 6704 and use CFCs on the 6074's you could deploy a 7203bxl w/20x10G for about $13k

We use 3bxl's on all our 67xx line cards so it drives cost up a bit but it's worth it as you scale.

 

[RyanD]

We setup default routing long ago and did not take full tables since we were not really multihomed, years ago a sup720 costs an arm and a leg to do full BGP tables, so it was just more worthwhile to take defaults and use multihop with run-of-the-mill switching gear. 

Giving users BGP isn't really hard, but any layer3 customers wanted full tables from us, and we can't give people full routing tables with the gear we had at the time, so we had to turn them down. Is it worthwhile to spend $30k+ on a 6503 + sup720 for a customer that might do $1000 a month of transit costs? Is it worthwhile to accept all those prefixes if you're singled homed to the same AS (with different links)? 

Since I no longer work for CC, I cannot comment on their switching topology. At the beginning we used OSPF for our internal topology which is great for scaling, vlans, et al. I do not like broadcade stuff (their firmware is shit), I've always been partial to Juniper/Cisco. 

Certainly if you are single homed to a carrier (even if you have multiple uplinks to them) it is pointless to take full tables unless you really want to mess with aspaths for some odd reason (maybe tag different communities based upon prefix destination as?) 

Now, any time you move to more than one carrier, I don't see any reason why you wouldn't want full tables, the level of granular control you gain and ability to manipulate the otherwise 'dumb' decision making of the standard bgp path selection algorithms enables you to largely improve the quality of your network and delivery of traffic through it.

We've had Internap's FCP platform in place since 2008, I don't know what we would do without it. Now there are other options in the market such as Noction that do similar functionality but it takes a huge amount of work off our network team to have to worry about manual path manipulation and performance issues.  I am a huge fanboy  

 

[Deleted]

Not trying to discredit you or anything but those prices are high even by the standards of back then.

I mean in 2006 when we got our first 6500 with 2 x 16-port 1g (gbic) blades and 2 x 48-port 1g (copper) and sup720-3bxl it was only $19k.

Later when we added on 2 more SUP720's ran about 7k and WS-X6704's ran about 7k as well.  Now days the SUP720-3bxl is still a very serviceable sup (we still run about a dozen of them). Convergence times stink but othewise they chug right along. 

Even if you were looking for cost effective edge routing the 6500s w/7203bxl still aren't a bad choice today. You can deploy a 6506 w/8 x 10G and 96 x 1G (copper) for about $8k. If you wanted to load it out with 5 x 4-Port 6704 and use CFCs on the 6074's you could deploy a 7203bxl w/20x10G for about $13k

We use 3bxl's on all our 67xx line cards so it drives cost up a bit but it's worth it as you scale.

Trust me, that is exactly what they were when the 720's first came out. We priced a 6506-E with 3bxl, 2 6724's and 2 copper gig blades, it was $39,000USD (I can probably find the email from my vendor). 

We looked at the sup2's as well, due to their raw PPS (and competitive pricing), the sup32's were total shit and not as featured as the sup2's (which made me laugh).. This also included the tier1 smartnet contract as well. I also looked at Force10, as they have very good switching fabric latency (almost an order of magnitude better than the cisco stuff), but the cost was outrageous. 

 

[RyanD]

Trust me, that is exactly what they were when the 720's first came out. We priced a 6506-E with 3bxl, 2 6724's and 2 copper gig blades, it was $39,000USD (I can probably find the email from my vendor). 

We looked at the sup2's as well, due to their raw PPS (and competitive pricing), the sup32's were total shit and not as featured as the sup2's (which made me laugh).. This also included the tier1 smartnet contract as well. I also looked at Force10, as they have very good switching fabric latency (almost an order of magnitude better than the cisco stuff), but the cost was outrageous. 

That explains it then, new pricing + smartnet without much of a discount.  

As an aside I've worked with hardware placement in the cisco broker channel for the better part of 10 years now with very large cisco partners so I understand the discount levels and what things can actually be purchased for with proper negotiations

Cisco itself and many purchasers are still stuck in the mindset that they need to buy new and carry huge warranties and have 4 hour replacement, blah blah.  Now days, equipment is so cheap everything should be treated with the Google/Utility model. Forget the warranties, buy a bunch more of it. When it dies, throw it out and replace it.  It's significantly cheaper to keep on-site spares and use your own staff for 15-minute replacement than deal with 4-hour replacement support contracts. I've never seen dell replace something in 4 hours, that 4 hour is always more like a 10 hour total process.

 

[Shados]

I've never seen dell replace something in 4 hours, that 4 hour is always more like a 10 hour total process.

Yeah, I can vouch for that too.

 

[Deleted]

I lost all respect for cisco the day they threatened OpenBSD with a patent lawsuit over VRRP (aka CARP). Their products are overrated. 

 

[VPSCorey]

In Cisco's defense 4 hours is somtimes 30 minutes, though they send cards to us via Taxi sometimes.  However that 4 hour warranty can even make fortune 100 companies cringe every once in awhile.

EX4500's are not meant to be BGP routers if that photo is true, fine for TOR switches though.

FCP and Nocticon to me are awesome I agree with RyanD about that.  These devices can figure out things that may take a lot of time and research for a human to get correct in seconds and can improve over time.  Networks are getting more complex and a lot of optimizing is being done via computers now.

Trust me, that is exactly what they were when the 720's first came out. We priced a 6506-E with 3bxl, 2 6724's and 2 copper gig blades, it was $39,000USD (I can probably find the email from my vendor). 

We looked at the sup2's as well, due to their raw PPS (and competitive pricing), the sup32's were total shit and not as featured as the sup2's (which made me laugh).. This also included the tier1 smartnet contract as well. I also looked at Force10, as they have very good switching fabric latency (almost an order of magnitude better than the cisco stuff), but the cost was outrageous. 

Cisco gives discounts the more you buy from them.  It's crap for the small guys though.

 

[peterw]

CC needs some EdgeMax power.