Routing weirdness with Above.net IPv6

[D. Strout]

So I've got a VPS in Dallas, TX with Versatile IT. Brendan's upstream there is Virtbiz Internet Services, AS40395. They peer heavily with Above.net, and tonight there seems to be something fishy with those routes. A traceroute out to a VPS with BudgetVM in Chicago (also a big Above.net peer over IPv6) can't get through, here's the trace:


1 node01.versatileit.com.au (2607:1c00:a:9::2) 0.08 ms 0.137 ms 0.028 ms
2 g5-0.brdr2.dal1.virtbiz.com (2607:1c00:1:1::4:1) 0.713 ms * 0.595 ms
3 ge-11-2-7.er1.dfw2.us.above.net (2001:438:fffe::29) 0.746 ms 0.776 ms 0.692 ms
4 2001:438:ffff::407d:1e56 (2001:438:ffff::407d:1e56) 0.956 ms 1.423 ms 1.052 ms
5 2001:438:ffff::407d:15e6 (2001:438:ffff::407d:15e6) 1.827 ms 1.23 ms 1.222 ms
6 2001:438:ffff::407d:1f4d (2001:438:ffff::407d:1f4d) 28.787 ms 28.572 ms 28.662 ms
7 2001:438:ffff::407d:142a (2001:438:ffff::407d:142a) 29.295 ms 28.704 ms 28.652 ms
8 2001:438:fffe::6ae (2001:438:fffe::6ae) 28.608 ms 28.612 ms 28.544 ms
9 * * *
10 * * *

Everything seems to be OK at first, it starts off through Above.net, but then after several hops it just stops. I cancelled it at that point, but believe me, it doesn't work even if left until the 30 hop limit is up. I tried several other IPv6 addresses, but very few worked. ipv6.google.com did, going out over Above.net and stopping at the 7th hop, presumably at some Google DC in Dallas. One very weird one is when I tried an apt-get update. Naturally it tried to use IPv6 and failed, but I tried a traceroute on the address it attempted to use: 2001:67c:1360:8c01::18. That one went through Above.net, then handed off to Level3, which took it from Dallas to NYC to London, then stopped dead again on the ninth hop, which looked like this:


9 SOURCE-MANA.edge5.London1.Level3.net (2001:1900:5:2:2::131a) 105.929 ms 105.27 ms 105.224 ms

That's some strange looking RDNS on that record. And of course the fact remains that it doesn't frickin' work. I've opened a ticket with Versatile IT, but since it seems likely to me that this is beyond Brendan's control, I'm posting this to see if anyone else knows what could be going on. I was going to try some stuff with the Above.net looking glass, but apparently it doesn't support IPv6. Really?

Edit: Same RDNS with Level3 right before the handoff to Canonical when I go through a provider with a different network mix (Ramnode). With the Above.net origin, though, it stops dead there. Still, weird RDNS there, any thoughts on that?

 

[wlanboy]

Really strange. If I look to my he tunnel everything looks fine:


2 gige-g3-8.core1.nyc4.he.net (2001:470:0:5d::1) 21.4 ms 15.785 ms 19.493 ms
3 level3.gige-g3-5.core1.nyc4.he.net (2001:470:0:202::2) 15.994 ms 15.753 ms 16.158 ms
4 2001:1900:19:6::7 (2001:1900:19:6::7) 15.819 ms 15.746 ms 15.807 ms
5 vl-4086.edge3.London1.Level3.net (2001:1900:6:1::11) 84.146 ms 92.944 ms 96.399 ms
6 vl-51.edge5.London1.Level3.net (2001:1900:101:1::a) 84.508 ms 84.568 ms 84.465 ms
7 SOURCE-MANA.edge5.London1.Level3.net (2001:1900:5:2:2::131a) 85.122 ms 85.249 ms 85.251 ms
8 obake.canonical.com (2001:67c:1360:8c01::18) 85.119 ms 85.237 ms 86.629 ms

So why is only the last hop missing on your traces? Because SOURCE-MANA is the last one before obake.

If I look e.g. to SecureDragons network:


3 * * *
4 * * *
5 * * *
6 * * *
7 * vl-4060.car2.Atlanta2.Level3.net (2001:1900:4:1::a6) 53.832 ms 206.683 ms
8 ae-1-4067.edge1.Washington12.Level3.net (2001:1900:4:1::3b5) 30.731 ms 30.341 ms 30.486 ms
9 2001:1900:4:1::3da (2001:1900:4:1::3da) 30.831 ms 30.724 ms 30.805 ms
10 vl-4047.car1.NewYork1.Level3.net (2001:1900:4:1::3ba) 36.270 ms 36.627 ms 54.478 ms
11 vl-4086.edge3.London1.Level3.net (2001:1900:6:1::11) 104.554 ms 104.496 ms 107.625 ms
12 vl-51.edge5.London1.Level3.net (2001:1900:101:1::a) 104.806 ms 105.262 ms 104.970 ms
13 SOURCE-MANA.edge5.London1.Level3.net (2001:1900:5:2:2::131a) 105.312 ms 105.442 ms 105.506 ms
14 obake.canonical.com (2001:67c:1360:8c01::18) 105.502 ms 105.389 ms 105.647 ms

Did not read anything about Above.net issues yet.

 

[D. Strout]

So why is only the last hop missing on your traces? Because SOURCE-MANA is the last one before obake.

That's exactly what I'd like to know. But all I can tell you is that it does not make that final hop, just hangs, giving empty hops with the three asterisks until the 30 hop limit is reached. 

 

[kaniini]

I am starting to suspect that some AboveNet POPs may have an incomplete IPv6 BGP table.  For example, 6to4 to/from AboveNet MIA is unreachable.  But Dallas and Chicago are fine.
 

Regarding obake.canonical.com, it is possible (and the most likely case) that there is an incomplete IPv6 BGP table or outdated prefix filter on the return path.  Likely at Canonical's demarc, which would be the SOURCE-MANA hop.

 

[D. Strout]

I am starting to suspect that some AboveNet POPs may have an incomplete IPv6 BGP table.  For example, 6to4 to/from AboveNet MIA is unreachable.  But Dallas and Chicago are fine.

Regarding obake.canonical.com, it is possible (and the most likely case) that there is an incomplete IPv6 BGP table or outdated prefix filter on the return path.  Likely at Canonical's demarc, which would be the SOURCE-MANA hop.

So how would such a thing ever get fixed? Not a big deal for me personally (though it could mean slow DNS for a fifth of my IPv6 users), but surely other people are going to have trouble here. Will Above.net just notice this eventually and fix it? I hope so.

 

[Tux]

So how would such a thing ever get fixed? Not a big deal for me personally (though it could mean slow DNS for a fifth of my IPv6 users), but surely other people are going to have trouble here. Will Above.net just notice this eventually and fix it? I hope so.

I'd bring it to the NOC's attention.

 

[DearLeaderJohn]

Bit of a bump however I've investigated this issue (I work for Cloud Shards, new owners of Versatile IT) and we've worked with Virtbiz + Above to get this sorted and we have been informed this has just been fixed.

 

[D. Strout]

Yup, looks good to me. According to the ticket response I got:

The issue revolved around the upstream peering with a carrier that we have recently made a circuit change with. This impacted the return path on some remote IPv6 hosts, depending upon the return path routing as determined by their BGP calculations.

 

[concerto49]

Yup, looks good to me. According to the ticket response I got:

Great that we sorted all this for you. Upon taking over, our number 1 priority was to fix all the outstanding issues that Versatile IT were having. Customers first.