amuck-landowner

How to gauge the quality of a provider's network

D. Strout

Resident IPv6 Proponent
Disclaimer: I do not pretend to be an expert on this subject, but this is how I do it. Feel free to add or critique anything I say here.

Not all VPSes are created equal. There are many ways two VPSes with identical specs can differ, making one better than the other. Overselling, a provider's reviews/ratings, what hardware they're running on, and other factors can make a $7/mo 512MB VPS better than a $3/mo VPS from another company. The factor I'm considering here is network quality. Your provider will always have what are called "upstream" providers that carry the network traffic going to and from your VPS. The number and quality of these will affect network latency (how long it takes traffic to get to and from your VPS), as well as the number of other computers and routers your request has to go through.

So, how do you do it? The biggest thing you need is a provider's test IP address. If the provider you're considering has multiple locations, be sure you get the test IP for that location, as test IPs differ between locations. With the test IP in hand, you should go to bgp.he.net and plug it in to the search box up top. For an example, I am going to take the test IP of one of the providers I really like: Versatile IT. The IP is 50.2.39.239 (even looks cool :)). A search on the above site should give you this:

V1vfzW0.png

So what does this mean? As you can see, the "Origin AS" is the same on all three entries. This IP is announced and originated from AS 30693. AS stands for "Autonomous System", which is basically a self-contained network with certain IP address blocks assigned to it. It is these we're interested in. Click the link to AS30693. This will give you lots of information about the AS, including how many IPs are assigned to it, where it originates from, and, importantly, the number of "peers" it has. Arguably the most important part is below the box with all the numbers, down where the pie chart(s) are/is. Next to the pie chart is a list of peers. In the case of AS30693, here are the pie charts. In this case there are two, one for IPv4 and one for IPv6. Providers that only have IPv4 will usually have just one.

oaSDRUq.png

Fundamentally, what you're looking for here is number and quality of the peers. A peer is another AS that this one exchanges traffic with. On one level, it's good for your provider's AS to have a lot of peers, so that if one's network goes down, traffic can be routed over others. So if you're lazy, just check that your provider's AS has a lot of peers. But if you're more diligent, check the quality of the peers. How do you do that? Traceroutes, a lot of them. Bring up a terminal emulator (command prompt, terminal, whatever) and type tracert (Windows) or traceroute (Unix variants), then the test IP. Starting around the third "hop", you should see that the IP addresses resolve to hostnames that indicate what provider the traffic is going through at that hop. First you should see your ISP's domain, then your ISP will "hand off" to other providers, eventually ending up at the IP. The last few hops should look familiar to you - they will be going through some of the providers you saw in the lists on the AS info page.

Other than that, though, what you're looking for is that the routing isn't "weird". Here's what my traceroute to 50.2.39.239 looks like:


Tracing route to 50.2.39.239 over a maximum of 30 hops

1 2 ms 2 ms 3 ms Wireless_Broadband_Router.home [192.168.1.1]
2 227 ms 250 ms 236 ms xx.xx.xx.xx
3 284 ms 308 ms 249 ms xx.xx.xx.xx
4 479 ms 376 ms 324 ms burl-lnk-70-109-168-138.ngn.east.myfairpoint.net [70.109.168.138]
5 380 ms 425 ms 347 ms te8-5.ccr01.alb02.atlas.cogentco.com [38.104.52.29]
6 465 ms 415 ms 460 ms te8-1.ccr01.jfk01.atlas.cogentco.com [154.54.5.37]
7 552 ms 491 ms 402 ms te0-3-0-7.ccr21.jfk02.atlas.cogentco.com [154.54.7.69]
8 251 ms 217 ms 155 ms te0-4-0-6.ccr21.dca01.atlas.cogentco.com [154.54.42.21]
9 112 ms 253 ms 122 ms te0-2-0-6.ccr21.atl01.atlas.cogentco.com [154.54.2.45]
10 125 ms 129 ms 107 ms te0-2-0-1.ccr21.iah01.atlas.cogentco.com [154.54.29.6]
11 131 ms 328 ms 129 ms te2-1.ccr01.sat01.atlas.cogentco.com [154.54.80.149]
12 140 ms 143 ms 142 ms te7-8.ccr01.elp01.atlas.cogentco.com [154.54.27.117]
13 201 ms 172 ms * te8-1.ccr01.phx02.atlas.cogentco.com [154.54.80.209]
14 325 ms 331 ms 257 ms 154.54.89.106
15 80 ms 81 ms 190 ms phx1.200-1.bb1.g1-3.eonixco.com [38.122.88.46]
16 199 ms 204 ms 82 ms 50.3.0.117
17 241 ms 238 ms 188 ms node03.versatileit.com.au [173.44.240.188]
18 151 ms 159 ms 158 ms 50.2.39.239

Trace complete.

I've "censored" the 2nd and 3rd hops since they're similar to my IP, which you don't need to know :p After that, though, you can see where the traffic is going. The fourth hop is in Burlington, Vermont with (the "burl" part at the beginning) with Fairpoint (my ISP, the "myfairpoint.net" part), then it hands off to Cogent ("cogentco.com"). To find out what locations the following hops are in, you can do a geoip lookup on the IP, or you can look for the bits that look like cities. The fifth hop has "alb02" in it, which refers to Albany. In 6 and 7 "jfk" is New York City, because jfk is the airport code for J.F.K. International. A lot of providers label their switching points this way. If you were to look up all of these IPs, you would see that from Albany, the traffic goes to New York City (hops 6-7), Washington D.C. (hop 8), Atlanta (9), Houston (10), San Antonio (11), El Paso (12), then Phoenix, which is where 50.2.39.239 is. Between hops 14-15, Cogent hands off to Eonix, which is the network that Versatile IT's test IP is in. With a few more hops in their internal network, we're at the test IP.

So what does all this tell us? Well, let's map it. I've not put the map here, because this is long enough as is, but you can see how this routing maps out here. As you can tell, it's a bit of a curve. Ideally, all routings would be a straight line, but they're not. What you don't want is weird routing where there are "zigzags", for instance, routing down to D.C. then back up to New York, then over to Chicago, then Kansas City, then St. Louis, then Dallas, then Phoenix. A good provider has a network with efficient routing. The routing from my house to 50.2.39.239 could be better, but it could be worse. And this is just one traceroute. Do more from your own VPSes, using looking glass tools, and if necessary ask other people to do them for you. If a server is just for your own personal use, you can just do a traceroute from your own connection, but usually the more the merrier. Over time, you'll begin to see which providers and upstream providers (usually called "carriers) have good routing and which ones don't. Then once you have a test IP, you can do a quick lookup of the ASN, see which carriers peer with it, and decide if this VPS likely has good enough routing to make it a good buy.
 

D. Strout

Resident IPv6 Proponent
Great info. One of my favorite blends is at colostore... http://bgp.he.net/AS12260

I get pure level3 from quite a few locations.
Glad you liked it. I'm sure this is familiar to many of us, but we all learned it at one point, possibly slowly. This will hopefully help people to figure it out all at once. Nice bandwidth blend there. Reminds me - consider this an "appendix":



Upstream providers I like:

Level3

Hurricane Electric

Cogent (I especially like Cogent and Hurricane Electric together)

That I dislike:

nLayer

Tinet

Telia
 
Last edited by a moderator:

Tux

DigitialOcean? lel
Glad you liked it. I'm sure this is familiar to many of us, but we all learned it at one point, possibly slowly. This will hopefully help people to figure it out all at once. Nice bandwidth blend there. Reminds me - consider this an "appendix":

Upstream providers I like:

Level3

Hurricane Electric

Cogent (I especially like Cogent and Hurricane Electric together)

That I dislike:

nLayer

Tinet

Telia
Tinet in Atlanta is basically hauling down data to Miami. It's not bad in Seattle though. It's GT-T now.

nLayer is actually a pretty good network. It was also sold off to GT-T.

Telia? Yeah, Telia does really suck.

Another good point to bring up: BGP doesn't mean everything. Just because your ISP peers with say Level3 and Atratro and your provider peers with say Cogent and nLayer, you will usually take the shortest route in terms of hops (AS and routing). Your ISP might have a shorter route with Atratro, and your provider might have a shorter route through Cogent.

Example: Charter (AS20115) and RamNode (AS3842) have a PoP in ATL. Both are peered with AS3257 (Tinet) in some form: AS3842 has a direct route with AS3257, and AS20115 is indirectly connected via AS174/AS1299/AS4436/AS7922/AS19151 - these involve an extra AS hop and more routing hops (sometimes outside the region). However, both ASes peer with AS4436 (nLayer) and it's just 2 hops. It's going to choose the nLayer route, as it's shorter on both sides.

Just a thought.
 
Last edited by a moderator:

Amfy

New Member
As more as you guys are getting into the BGP thing, you will notice that everything shown at bgp.he.net can be "manipulated". For example, it's possible to let users think that the provider has Telia, Level3 and other not really budget providers, but route everything through HE.

Next question would be, how are the guys here determining what's a budget and what's a premium provider or with @D. Strout words: Upstreams you like or dislike. In my understanding it depends more on how many PoPs a provider has and how much they're overselling their links.

Telia? Yeah, Telia does really suck.
Why should they?

Less hops != shorter route (in the way bgp routers look at it)

There are *sooo many* things you need to think about with global routing :)
 
Last edited by a moderator:

TheHackBox

New Member
I've seen screwy routes from Atrato, Chicago --> New York --> Amsterdam ---> London --> New York --> Chicago
 

Tux

DigitialOcean? lel
As more as you guys are getting into the BGP thing, you will notice that everything shown at bgp.he.net can be "manipulated". For example, it's possible to let users think that the provider has Telia, Level3 and other not really budget providers, but route everything through HE.

Next question would be, how are the guys here determining what's a budget and what's a premium provider or with @D. Strout words: Upstreams you like or dislike. In my understanding it depends more on how many PoPs a provider has and how much they're overselling their links.

Why should they?

Less hops != shorter route (in the way bgp routers look at it)

There are *sooo many* things you need to think about with global routing :)
Yes - another great point that needs to be brought up.

bgp.he.net is a simple breakdown. You really want to double-check it with the other provider's looking glass. If your provider peers with Level3, check it with Level3's looking glass. Make sure to use a location close to you/the provider.
 

Amfy

New Member
Hi,

Thanks for your answer!

bgp.he.net is a simple breakdown. You really want to double-check it with the other provider's looking glass. If your provider peers with Level3, check it with Level3's looking glass. Make sure to use a location close to you/the provider.
 

If we're going to the point "what is technically possible", I can even manipulate that. Outbound per /32, inbound per /24; checking with the looking glass of the provider also doesn't help, I can set a community tag not to export the route to other peers of the transit :)

But however, if we're at that point, you should ask yourself again, if you can trust the provider with your data...
 

D. Strout

Resident IPv6 Proponent
This is getting beyond what I know, which is good. Keep in mind this is a general overview, I certainly know and understand that things can be much more complex.
 

EarthVPN

New Member
Good Topic.Let me add some clarification.

I will use Global-Layer AS57172 for the example.

When you look at global-layer's peers via bgp.he.net you will notice that %50 peers are via Atrato-IP and %49 peers are via Switch Communications.You can decide based on the bgp.he.net results that global-layer have configured their upstreams to share traffic equally.

http://bgp.he.net/AS57172#_asinfo

OFQluAT.png

However when you check via robtex you can notice that reality is different.%93 via Atrato-IP , %7 via switch communications.

http://as.robtex.com/as57172.html#asinfo

7mNoZMJ.png

To double verify your findings you can check your destination IP via below site doing multiple traceroutes worldwide.

http://www.ping.ms/index.php?act=trace
 

D. Strout

Resident IPv6 Proponent
Thanks @EarthVPN, that makes sense, and also clears up some stuff I've seen on my own home connection. How and why do companies configure their network this way?
 

EarthVPN

New Member
How ?

Via their bgp configuration on their border router (as-path-prepend/export/communities etc).

Why?

1) Cost.Every upstream providers cost per mbps is different.If you are multi-homed means you are using more than one upstream you can optimize your traffic i.e: you can set one of your /24 a premium network(including your level3,ntt,gtt upstreams) or low cost network(including HE,cogent upstreams).

2) There are also internet exchange(IX) points which is more cheaper than buying from upstream providers.You can route some traffic via IX to reduce latency and cost.
 

D. Strout

Resident IPv6 Proponent
How ?

Via their bgp configuration on their border router (as-path-prepend/export/communities etc).
The "why" is as I expected, but this I still don't understand. Where does the 49%/51% that Hurricane Electric shows come from? If really the traffic is being divided so unequally in your example, what do those numbers mean?
 

concerto49

New Member
Verified Provider
The "why" is as I expected, but this I still don't understand. Where does the 49%/51% that Hurricane Electric shows come from? If really the traffic is being divided so unequally in your example, what do those numbers mean?
It can mean a lot of things though. It could be that HE is purely outbound and that's what the ratio is (or purely inbound). It could be balanced using odd/even routing policy (if you remember the incident earlier) or just BGP. Who knows? It could also be a particular customer takes up 49% of the bandwidth and is Cogent only so in reality they are single-homed to HE. It only gives you an idea, not a true answer.
 

kaniini

Beware the bunny-rabbit!
Verified Provider
Upstream providers I like: Level3 Hurricane Electric Cogent (I especially like Cogent and Hurricane Electric together) That I dislike: nLayer Tinet Telia
Well... Cogent and HE are both really useful as one-trick ponies that hop off to a peer.  But for example, using Cogent to reach AT&T is really bad right now because of the Netflix dispute (AT&T is deprioritizing Cogent traffic right now).

nLayer/GT-T can be useful to supplement for some routes.  For example, in LA, it is really useful for reaching HiNET/ChinaUnicom.  Beyond that, Cox and Comcast localpref nLayer over some other routes due in part to having a lower ASN.

It should also be noted that nLayer/Tinet (Inteliquent) are going to be the same network soon, as they merged operations.  Telia is actually very good for reaching Europe as is Inteliquent -- the merger between Inteliquent and nLayer is likely to create a very interesting international network that may actually wind up becoming a Tier 1.

The bottom line is that you should look for a well-balanced network that combines transit and local peering (or at least transit providers which have extensive peering like HE does).

Another network to look at for European reach is Sparkle.  Atrato is also interesting in this regard, but not as interesting as Sparkle.
 

concerto49

New Member
Verified Provider
It should also be noted that nLayer/Tinet (Inteliquent) are going to be the same network soon, as they merged operations.  Telia is actually very good for reaching Europe as is Inteliquent -- the merger between Inteliquent and nLayer is likely to create a very interesting international network that may actually wind up becoming a Tier 1.

Another network to look at for European reach is Sparkle.  Atrato is also interesting in this regard, but not as interesting as Sparkle.
The merger isn't happening soon. They will exist separately for as long as possible at the moment. Same as level3 and Global Crossing. There will be some shared peering and that's it.

Atrato is only good because it's cheap.
 

Magiobiwan

Insert Witty Statement Here
Verified Provider
I personally get the best latency through HE where I live. Cogent is also reasonably good for me too. Even coming from servers I have all the way in Atlanta or in Illinois, with the pings travelling all the way to Oregon.
 

kaniini

Beware the bunny-rabbit!
Verified Provider
I personally get the best latency through HE where I live. Cogent is also reasonably good for me too. Even coming from servers I have all the way in Atlanta or in Illinois, with the pings travelling all the way to Oregon.
It's because HE is a really good one-hop transit provider for dumping you onto a peered ISP quickly, such as Cox or Comcast.

Everyone peers with HE.
 

InfiniteTech

New Member
Verified Provider
IPs are not "assigned" to ASNs. They are "announced".

Peer in OP's scenario really means upstream providers who have an active BGP session. Peer is usually a network that excahnges traffic for free (in a bilateral agreement). Transit is paid upstream traffic.
 

egihosting

New Member
Verified Provider
Very nice info.  Definitely would like thoughts on what everyone on VPSBoard thinks EGI should add to our mix.

AS18779
 
Top
amuck-landowner