How to test quality of server internet connection?

[ICPH]

Hi,

i got server and the SSH shell really lagging for whole like 1,5 days.

i wait even seconds before commands are written on the screen after i stop typing

the top -c, iotop, dstat shows server is not bussy

when tried to download something especially from europe (server is in USA), there are repeated moments connection stop and SCP command shows "stalled" message next to transfer..

Is there any simple way on how i can test the connection so i know it is issue of provider?

this is example traceroute from server (USA)  to my pc (Europe):



traceroute to myPCIPaddress (myPCIPaddress), 30 hops max, 40 byte packets
1 * * *
2 * * *
3 199.191.57.193 (199.191.57.193) 2.751 ms * 199.191.57.193 (199.191.57.193) 2.760 ms
4 * * *
5 * * *
6 ffm-bb1-link.telia.net (80.91.246.61) 112.766 ms 112.948 ms 112.952 ms
7 prag-bb1-link.telia.net (62.115.139.222) 130.490 ms * *
8 prag-b3-link.telia.net (213.155.131.63) 648.990 ms prag-b3-link.telia.net (213.155.137.19) 769.470 ms prag-b3-link.telia.net (80.91.249.224) 128.735 ms
9 rio-ic-143573-prag-b3.c.telia.net (213.248.87.98) 130.407 ms 133.575 ms *
10 sitK9-bb-jsw01-xe-0-1-0-0-188-175-255-198.net.myprovider.cz (188.175.255.198) 651.029 ms 653.168 ms 741.071 ms
11 myPCIPaddress.client.myprovider.cz (myPCIPaddress) 741.105 ms 741.105 ms 741.166 ms

I have similar server also in USA, and this one at same time works ok (dont see any lag working on command prompt)

 

[ICPH]

there is same time two server comparison by linux tool "mtr":



isnt there an issue on above one (the one that lagging for me), but its not only 1 second lag, there are even 2 second lags

 

[Munzy]

Look at the packet loss, it is clear there is something wrong with your first MTRs server. This would explain your issue with SSH.

Wait you run a company?

 

[zzrok]

The first server has about 20% packet loss on its first hop.  The loss is roughly the same on all subsequent hops, which leads me to believe the problem is with that server or your provider's network.

I'm with @Munzy, you run a company?  This is pretty basic networking problem on which to get hung up.

 

[aggressivenetworks]

He runs the company with the good old point and click method. Which anybody can do now a days to be a hosting provider. All the @ops threads he requires technical help of some kind. I mean its OK to ask for help. But, he wants us to do his sys admin work too!

 

[DomainBop]

, it is clear there is something wrong with your first MTRs server.

It's in Lenoir so I'm not surprised there is packet loss and 710ms pings to CZ.

There is also minor packet loss on incoming connections to that server from both Nuremberg, Hetzner DC1) and Paris (online.net) when it hits the server (about 1.5%-2.5% loss).

tl;dr contact the provider, the server's port is probably either saturated/overloaded or the server has a few abusers on it.

edited to add:

How to test quality of server internet connection?

smokeping

 

[Munzy]

@Aldryic C'boas we need your assistance!

 

[MartinD]

Wait you run a company?

Thought this after the first thread a few days ago.

 

[TurnkeyInternet]

We all have to start somewhere  

definatley give your provider a ring, on the bright side you have the data they will ask you for to qualify for them to escalate it to their engineering dept to figure out what's going on.  Perhaps your provider or server is maxing bandwidth, or on a bad wire/switch.

 

[ICPH]

Thank you, i have contacted the provider, but apparently after 7 days it did not changed, here is todays 1 hour of pinging&traceroute for two USA servers in different datacenters. Traffic goes thru Telia sonera network to the Europe for both monitored servers, but one server has latency spikes

 

[zzrok]

You need to check the mtr for the reverse path.  I'm guessing the one with high latency has a different, much worse return path than the one with good latency.  Do you have access to the host you are tracing on the screenshots?  Run mtrs from that host to your two servers and post the results.

 

[TurnkeyInternet]

Due to it coming and going, 2 thoughts pop into my head

#1) some sort of DoS / DDoS against others in the datacenter/rack/virtualized server that impacts your network (or parts of the upstream network connection points that connect to the heavier source of the attacks)

#2) possibly some over-bandwidth-subscription (backups to a cloud source filling up the bandwidth creating congestion points).

either way - you have an issue there and if they wont resolve, you certainly can find an alternative.

 

[ICPH]

Thank You for the advices TurnkeyInternet and zzrok.

I may update screenshot for oposite way pinging as zzrok suggested (at least i think he want me to do tracerouting from destination?), but i need to do it on my other PC where i have virtualized Linux with "mtr" tool.

in the meantime im updating with todays 5 hours pinging/tracerouting, two servers with different datacenter, comparison. same servers as before...

left server is from bigbrainglobal, right server from servercomplete:

 

[zzrok]

I'm not sure what you are pointing out in this latest image.  You are looking at the worst column, which can only tell us that packets were hung up at least one time, which isn't a very strong indicator of anything.

Standard deviation is more useful.  The part that caught my eye was looking at the standard deviation of the first few hops for each server.  The BBG server has values 40 and above, and the SC server has values 15 and below.  This pattern hold for all hops, actually.  A high standard deviation indicates that a large number of packets are taking considerably longer than the average round trip time (RTT).  High RTT will cause ssh to lag.

This tells me the BBG server or internal network is much less stable than SC.  This is an issue with the server or BBG's internal network.  You have to work with them to figure it out.