Colocrossing's Core "Router" is a Switch
- Thread starter drmike
- Start date
nunim
VPS Junkie
They've been promising IPv6 is coming "in a month" since October 2011 at least, probably longer, this is the main reason I won't use any VPS provider in a CC location.
That's pretty silly, they've been saying there technically capable of rolling out IPv6 for quite some time but wouldn't do it for this reason or that reason, if you have the capability and the demand why wouldn't you at least satisfy the customers you can? Since when does CC care about what's fair?
Last edited by a moderator:
Well what I don't understand then is this:
Source: http://lowendtalk.com/discussion/comment/316739/#Comment_316739Jon Biloh said:
How is it possible that they could setup IPv6 for colo customers, but not their clients who are renting servers?
Last edited by a moderator:
They said in that thread that the switches they use for their dedicated servers don't support OSPFv3, which is required for IPv6 (if you are going to use OSPF). They're waiting for a software update to enable this feature.
Last edited by a moderator:
Not sure why they don't use iBGP instead.
It's possible they somehow have more than 10k routes internally?
It's also most likely because brocade screws people pretty hard for licensing. OSPF is almost always supported in the base images I think where as BGP always tacks a few grand onto the sticker. Same thing in Junipers.
Francisco
That is NOT what multipath is for. Multipath load balances equal-path links and by default, only when the AS paths match exactly. As Francisco mentioned this is typically for balancing links with an upstream that you have multiple connections with at the same router to avoid bonding the ports instead. In CC's case, they must have intentionally disabled matching of the entire AS-path to force traffic out two different carriers. Load balancing across multiple carriers is a bad idea due to traceroute being a mess as well as latency difference, your transfer speeds will suffer drastically due to packets arriving at different timings. However, it is the fastest and easiest way to balance your outgoing traffic.
Reason is simple, if they claim to be using the Brocade ICX 6450 then it's not a mystery. This switch likely does their customer VLAN routing and it doesn't support BGP, only OSPFv2 (ipv4) and RIPv2 (ipv4). It supports IPv6 in hardware, but static routes only. Based on this information, OSPF is critical to their operation because they are redistributing default from their core routers to the Brocade, as well as VLAN routes from the Brocade to their core routers. They can't enable IPv6 OSPF since the firmware doesn't support it (yet?) and they need two way OSPF announcements if they want to route IPv6 properly, I cannot imagine them wanting to configure static routes all over the place just to get IPv6 running.
The use of the Brocade for customer facing ports also explains why CC doesn't want to offer BGP sessions, the switch doesn't support BGP and it's where they terminate customer VLAN and routing. In order to offer BGP sessions, they have to plug you into a switch/router that has BGP, and assuming these switches are used as top-of-rack switches I cannot imagine them wanting to pull new cross connects just for you. Their Brocades are likely running pure L3 uplinks to avoid doing L2 spanning tree for redundancy as well as to isolate VLAN numbers to the switch itself, so they can't just configure your port as a VLAN back to their core routers without making things overly complex.
There'll be probably people complaining why the use of Brocade if it doesn't support IPv6/BGP, blah blah. Brocade (ex-Foundry) is well known for reliable/high performance switches (L2 only) at cheap prices, in this case with basic layer 3 slapped on as a bonus. They are used extensively in L2 deployment especially for high port count setups and the equipment itself is rock solid. They are not known for their L3 deployments, so it's not surprising they lack OSPFv3 on these.
*Disclaimer* I do not work for CC. I just happen to have years of experience running Foundry/Brocade equipment so I'm familiar with the limitations of their platforms and can piece together the puzzle on a network level.
Yep.
We have a superX that we use in LV for our switching now. It's garbage for layer3 but for layer 2? no sweat.
We have most of our blades filled and then a 10gig link from that to the router.
We never have issues with it and it's a nice change from the TOR's w/ LACP like we used to have.
Francisco
We have a superX that we use in LV for our switching now. It's garbage for layer3 but for layer 2? no sweat.
We have most of our blades filled and then a 10gig link from that to the router.
We never have issues with it and it's a nice change from the TOR's w/ LACP like we used to have.
Francisco
I used to have Foundry Bigirons as core routers that terminate VLANs for L2 TOR Foundry Fastiron switches. Great L2 performance, L3 completely iffy especially at high pps. Now switched to Juniper EX for L3 VLAN routing and HP/Force10 for cheap gigabit L2 with 10G upgrade if necessary in future.
mpkossen
New Member
They run OSPF according to the post you linked, not BGP. Also, Jon says in there they use Brocade, not Juniper. So, the OP pulled a switch photo from their website and made up a story around it, I assume
Anyway, nothing else I can make of it.
We setup default routing long ago and did not take full tables since we were not really multihomed, years ago a sup720 costs an arm and a leg to do full BGP tables, so it was just more worthwhile to take defaults and use multihop with run-of-the-mill switching gear.
Giving users BGP isn't really hard, but any layer3 customers wanted full tables from us, and we can't give people full routing tables with the gear we had at the time, so we had to turn them down. Is it worthwhile to spend $30k+ on a 6503 + sup720 for a customer that might do $1000 a month of transit costs? Is it worthwhile to accept all those prefixes if you're singled homed to the same AS (with different links)?
Since I no longer work for CC, I cannot comment on their switching topology. At the beginning we used OSPF for our internal topology which is great for scaling, vlans, et al. I do not like broadcade stuff (their firmware is shit), I've always been partial to Juniper/Cisco.
Single homed, sure default routing makes perfect sense. Now there's Telia + Cogent, doesn't make sense anymore (and I think previously L3?). One's a decent provider globally, the other is a decent provider within their own network only. Assuming there's no change in the equipment, all CC needs to do now is to take customer routes from Cogent and default Telia, that would be a simple and efficient setup without needing to load a full table.
I don't think full table is the problem here, most people taking service from CC will likely not bother picking up another transit and running cross connects, most of them just want to route their own IPs with their own ASN. If they run a pure OSPF setup, this is troublesome to arrange alongside.
I'm not surprised they maintained the same OSPF setup, like you said it's easy as hell to setup and maintain. Add VLAN, assign IP subnet, tag to port, redistribute into OSPF, done. It's not a wrong solution, just that Brocade's lack of IPv6 OSPF made it difficult for CC to implement IPv6. Brocade's firmware always had issues and they're pretty slow to fix stuff, there was plenty of horror stories in WHT with providers who used primarily Brocade and got hit by the Brocade/Foundry specific DOS attack or providers who ran into firmware bugs (Sagonet I think, many years ago). The perfect use case for Brocade is pure L2 with VLANs, period. Firmware wouldn't even matter anymore.
Problem here is that their equipment (if it is true they are using EX-series switches to face their providers) isn't capable of much more than default routes. Pretty much all they can do is default everything to one and use the other as a failover, or use things like they have them setup now.
Last edited by a moderator:
EX4500 would be 12K FIB won't be sufficient, but I built my assumption that they still used the CIsco 6500 which should be able to do 200k FIB.
But god this day and age if they're moving multi-10Gs, with the MX series at decent price point I don't see why they would need to cut corners on router hardware. Full 10G Telia would be probably what, 10k/month or so? Cogent another 5k/month? I'm sure there's budget for network hardware there somewhere.
As I was the principle person responsible for setting up the network during it's birth, I am going to go out on a limb and defend taking default routes. Routers/switches with large amounts of memory for routing are not cheap, years and years ago when the routing table was ~200k routes (with /24's mixed in there), a 6500 series with a decent sup and nice backplane bandwidth was about $40k plus (direct from my old vendor).
On the other hand, a smaller, faster switch (high PPS) was cheaper and could do more for less money (trunking, hsrp, etc). We went the cheaper route. While I /wanted/ a fully loaded 6506 with sup720 (they were $25k new), but we could not justify the cost JUST for full bgp routes. 95% of dedicated customers did not care about if we had full BGP or not, as long as their service worked (and didn't lag).
Back in the 90s when I worked for AboveNet, the motto at IAD4 was 'Keep it simple', which is exactly what I used when I designed the existing network. OSPF with eBGP (default routes), we used communities to alter our localpref in some cases, though. It worked.
One thing I did NOT agree with is mixing vendor equipment, it's nice to do layer2 on different vendors (most of the time) but firmware bugs, broken RFC agreements made it difficult. I despised foundry's gear as being total shit made by people who wear leafs as shoes.
It's simple numbers at the end of the day, the needs of the many (simple setup, default routes) outweigh the needs of the few (full blown routing, MPLS, et al)
On the other hand, a smaller, faster switch (high PPS) was cheaper and could do more for less money (trunking, hsrp, etc). We went the cheaper route. While I /wanted/ a fully loaded 6506 with sup720 (they were $25k new), but we could not justify the cost JUST for full bgp routes. 95% of dedicated customers did not care about if we had full BGP or not, as long as their service worked (and didn't lag).
Back in the 90s when I worked for AboveNet, the motto at IAD4 was 'Keep it simple', which is exactly what I used when I designed the existing network. OSPF with eBGP (default routes), we used communities to alter our localpref in some cases, though. It worked.
One thing I did NOT agree with is mixing vendor equipment, it's nice to do layer2 on different vendors (most of the time) but firmware bugs, broken RFC agreements made it difficult. I despised foundry's gear as being total shit made by people who wear leafs as shoes.
It's simple numbers at the end of the day, the needs of the many (simple setup, default routes) outweigh the needs of the few (full blown routing, MPLS, et al)
I don't think anyone would have faulted you for using a default route setup when there's low bandwidth usage and what you did fit the business needs at that point of time.
My point is that looking at the prices today and the amount of bandwidth I'd estimate CC doing, justifying the equipment would be far easier compared to years ago. Network hardware prices for Juniper went down heavily since the MX platform stablized, their EX series have also pushed Cisco into a good price fight for performance. $40k was a lot of money years back and you needed 2 for redundancy. $40-60k today can get you a pair of MX80 which will do 2x40G all day long, and plug in nicely to the EX4500 (yey same vendor!).
I hear you on the Brocade/Foundry part. I believe you were at the time where RSTP/MSTP was still in development and every vendor had their own implementation that refused to talk to the other guy. Even today Cisco vs Juniper L2 issues are still existing, so while slightly better today but not totally gone.
My point is that looking at the prices today and the amount of bandwidth I'd estimate CC doing, justifying the equipment would be far easier compared to years ago. Network hardware prices for Juniper went down heavily since the MX platform stablized, their EX series have also pushed Cisco into a good price fight for performance. $40k was a lot of money years back and you needed 2 for redundancy. $40-60k today can get you a pair of MX80 which will do 2x40G all day long, and plug in nicely to the EX4500 (yey same vendor!).
I hear you on the Brocade/Foundry part. I believe you were at the time where RSTP/MSTP was still in development and every vendor had their own implementation that refused to talk to the other guy. Even today Cisco vs Juniper L2 issues are still existing, so while slightly better today but not totally gone.