iwStack, a Prometeus project - who is using them and network throughout

[drmike]

I've been testing an iwStack cloud instance tonight for a friend.

I am a fan of Prometeus and have had good experiences in the past there.

Ran one of the freevps derivative scripts and found the network performance everywhere super slow.  This is a fresh empty container, so nothing competing there for resources.


===== NORTH AMERICA =====

Download from Cachefly: 7.09MB/s
Download speed from OVH(BHS), Canada: 1.41MB/s
Download speed from Datashack/WSI, Kansas City: 1.45MB/s
Download speed from TMS, Dallas, TX: 998KB/s
Download speed from Softlayer, Dallas, TX: 1.36MB/s
Download speed from Softlayer, Seattle, WA: 1.08MB/s
Download speed from Softlayer, San Jose, CA: 1.08MB/s
Download speed from Softlayer, Washington, DC: 1.80MB/s
Download speed from BurstNET, Scranton, PA: 1.10MB/s
Download speed from BurstNET, Los Angeles, CA: 703KB/s
Download speed from BurstNET, Miami, FL: 941KB/s
Download speed from BurstNET, Dallas, TX: 1005KB/s
Download speed from Volumedrive, Northern, PA: 1.40MB/s
Download speed from Turnkey Internet, New York Tech Valley (??): 1.14MB/s
Download speed from PhotonVPS, Los Angeles, CA: 762KB/s
Download speed from Fiberhub, Los Vegas, Nevada: 1004KB/s

Download speed from ServerComplete, Jacksonville, Florida: 1.19MB/s
Download speed from IOFlood, Phoenix, AZ: 749KB/s
Download speed from Dacentec, Lenoir, NC: 1.28MB/s

========= EUROPE ========

Download speed from Linode, London, UK: 6.56MB/s
Download speed from Rapidswitch, UK: 4.28MB/s
Download speed from Serverius, NL: 7.27MB/s
Download speed from i3d.net, NL: 4.67MB/s
Download speed from Leaseweb, Haarlem, NL: 5.94MB/s
Download from OVH (RBX), France: 1.47MB/s
Download speed from Hetzner, Germany: 930KB/s

========== ASIA =========

Download speed from Linode, Tokyo, JP: 669KB/s
Download speed from Leapswitch, Pune, India: 1.19MB/s
Download speed from HKcolocation, Hong Kong : 530KB/s





I removed WANSecurity speed tests, as WANSec pulled their test files and speeds are invalid / script doesn't error check for that condition.

Can someone with iwStack or Prometeus VPS run some tests and compare notes here?

 

[drmike]

Guess I should amend this some...

Is anyone aware of a network issue or perhaps speed limitation with filtering?  Realizing they are including "filtered" IPs at this point / what is being used?

Is there any current network degradation going on there that anyone knows of?

 

[DomainBop]

Guess I should amend this some...

Is anyone aware of a network issue or perhaps speed limitation with filtering?  Realizing they are including "filtered" IPs at this point / what is being used?

Is there any current network degradation going on there that anyone knows of?

Your tests differ greatly from mine.   These are the results from a production IWStack instance

wget freevps.us/downloads/bench.sh -O - -o /dev/null|bash


CPU model :  QEMU Virtual CPU version (cpu64-rhel6)


Number of cores : 2


CPU frequency :  1999.999 MHz


Total amount of ram : 1002 MB


Total amount of swap : 465 MB


System uptime :   47 days, 5:16,       


Download speed from CacheFly: 51.5MB/s


Download speed from Coloat, Atlanta GA: 11.9MB/s


Download speed from Softlayer, Dallas, TX: 12.2MB/s


Download speed from Linode, Tokyo, JP: 5.21MB/s


Download speed from i3d.net, Rotterdam, NL: 24.1MB/s


Download speed from Leaseweb, Haarlem, NL: 60.5MB/s


Download speed from Softlayer, Singapore: 5.45MB/s


Download speed from Softlayer, Seattle, WA: 9.07MB/s


Download speed from Softlayer, San Jose, CA: 8.73MB/s


Download speed from Softlayer, Washington, DC: 18.0MB/s


I/O speed :  131 MB/s

wget http://cachefly.cachefly.net/100mb.test -O /dev/null


--2014-01-14 04:35:55--  http://cachefly.cachefly.net/100mb.test


Resolving cachefly.cachefly.net... 205.234.175.175


Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.


HTTP request sent, awaiting response... 200 OK


Length: 104857600 (100M) [application/octet-stream]


Saving to: `/dev/null'


100%[======================================>] 104,857,600 63.1M/s   in 1.6s    


2014-01-14 04:35:57 (63.1 MB/s) - `/dev/null' saved [104857600/104857600]

 

[cubixcloud]

Have your friend opened a ticket with them and find out. That's probably the best source right now.

 

[drmike]

Your tests differ greatly from mine.   These are the results from a production IWStack instance

Very different and strange.

What OS / ISO image are you using?

 

[budi1413]

@drmike Do you do this?

nano /etc/sysctl.conf

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

sysctl -p

 

[drmike]

@drmike Do you do this?

nano /etc/sysctl.conf





net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216


sysctl -p

Nope, giving that a spin now  

Reboot required?

 

[DomainBop]

Very different and strange.

What OS / ISO image are you using?

Debian Squeeze 64-bit

 

[drmike]

Running Debian 32-bit minimal....

I made the tweaks recommended by @budi1413 (thanks friend!)


Download speed from CacheFly: 7.29MB/s
Download speed from Coloat, Atlanta GA: 6.84MB/s
Download speed from Softlayer, Dallas, TX: 7.32MB/s
Download speed from Linode, Tokyo, JP: 3.40MB/s
Download speed from i3d.net, Rotterdam, NL: 8.08MB/s
Download speed from Leaseweb, Haarlem, NL: 7.94MB/s
Download speed from Softlayer, Singapore: 4.20MB/s
Download speed from Softlayer, Seattle, WA: 4.75MB/s
Download speed from Softlayer, San Jose, CA: 6.13MB/s
Download speed from Softlayer, Washington, DC: 7.69MB/s


Those are "better". Still far short of speeds @DomainBop is seeing

 

[wlanboy]

It states that it is a "CPU model :  QEMU Virtual CPU version (cpu64-rhel6)".

Did you activate virtio drivers for the network?

Whould be nice to see some traceroutes to NL/UK/NY/DAL/ATL/SEA/LAS.

What upstream providers are they using?

 

[drmike]

Looking around their panel control for the VIRTIO here in a bit.

Upstreams I am unsure of.

 

[Dylan]

I just spun up two brand new instances, one with a filtered IP and one without. Based on these results I'm guessing your instance is filtered, drmike, and DomainBop's isn't?

Regular:


Download speed from CacheFly: 76.4MB/s

Download speed from Coloat, Atlanta GA: 8.93MB/s

Download speed from Softlayer, Dallas, TX: 14.4MB/s

Download speed from Linode, Tokyo, JP: 5.10MB/s

Download speed from i3d.net, Rotterdam, NL: 27.8MB/s

Download speed from Leaseweb, Haarlem, NL: 47.5MB/s

Download speed from Softlayer, Singapore: 6.87MB/s

Download speed from Softlayer, Seattle, WA: 14.8MB/s

Download speed from Softlayer, San Jose, CA: 9.04MB/s

Download speed from Softlayer, Washington, DC: 20.7MB/s


 

Filtered:

Download speed from CacheFly: 8.16MB/s

Download speed from Coloat, Atlanta GA: 6.47MB/s

Download speed from Softlayer, Dallas, TX: 4.66MB/s

Download speed from Linode, Tokyo, JP: 4.95MB/s

Download speed from i3d.net, Rotterdam, NL: 7.90MB/s

Download speed from Leaseweb, Haarlem, NL: 8.06MB/s

Download speed from Softlayer, Singapore: 4.28MB/s

Download speed from Softlayer, Seattle, WA: 7.21MB/s

Download speed from Softlayer, San Jose, CA: 4.93MB/s

Download speed from Softlayer, Washington, DC: 7.48MB/s

 

I wonder if this speed difference is normal. I'd expect less on the filtered, but this much?

 

[MannDude]

I'd tag prometeus and Mao_Member_no_signature for a comment. Either one should be able to touch base on this.

Also, I didn't know they did DDoS filtering... so that's neat. Haven't really looked at iwStack much but may give it a peak.

 

[maounique]

Hello !

Due to the semi-ban (rMember special group where people which do not agree with the admins are placed), wont be able to edit my post, so please forgive the typos and multiple posts (eventual).

1. We used to have a 200 mbps limit so the 1 gbps port of each node will not be hogged by 1-2 people. That is removed now because we havent detected such a behaviour.

2. I have seen that happening before, usually for small instances (384 MB). Those are not routed differently, it is probably an issue generated from the lack of virtio drivers, I was never able to replicate it.

3. @DomainBop, we upgraded te storage, now we have about 700 MB/s and more than double IOPS, you will need to re-deploy the instance or open a ticket to have it migrated to the new storage. Only the smallest instances are still on the old storage (that will probably be kept and offered at a lower price, we did not make a decision yet).

4. SSD local storage will probably be added this month.

 

[wlanboy]

I just spun up two brand new instances, one with a filtered IP and one without. Based on these results I'm guessing your instance is filtered, drmike, and DomainBop's isn't?

Regular:


Download speed from CacheFly: 76.4MB/s

Download speed from i3d.net, Rotterdam, NL: 27.8MB/s

Filtered:

Download speed from CacheFly: 8.16MB/s

Download speed from i3d.net, Rotterdam, NL: 7.90MB/s
 

I wonder if this speed difference is normal. I'd expect less on the filtered, but this much?

Yup that's an upstream problem.

Do a "wget http://cachefly.cach....net/100mb.test -O /dev/null" on both.

We will see two different ips.

Would be interesting to see who is filtering the traffic.

 

[budi1413]

Nope, giving that a spin now

Reboot required?

Don't need reboot.

 

[maounique]

Hello again !

The filtered IPs are indeed slower, we have partenered with seflow.it for them. They should be used only if you are under DDoS risk.

Our DDoS protection is mostly aimed to protect our network and as an insurance it will not happen that we have to suspend or nullroute production services. It is not for DDoS refugees operating game-servers, for example.

If you do not have VIrtio drivers, the performance will be hit.

You can enable them by selecting our templates (they have it), our ISOs (also included) or by selecting other PV 64/32 when inserting your own ISO.

 

[drmike]

So to biz side of things...

This is a small instance like you described #2. 

Where is the virtio driver setting?  Take it is in the Cloud Panel right?  Haven't looked yet.

Can you push the /etc/sysctl.conf optimizations to the templates?  That certainly helped and might be fine for lots of folks as is...?

 

[maounique]

Would be interesting to see who is filtering the traffic.

We have partnered w Seflow.it for the DDoS protection, that is 100 mbps per IP guaranteed. It might be that their protection is triggered on tests due to very big speeds from cachefly and all the rest of the tests are over the filtering machines.

 

[DomainBop]

We have partnered w Seflow.it for the DDoS protection, that is 100 mbps per IP guaranteed. It might be that their protection is triggered on tests due to very big speeds from cachefly and all the rest of the tests are over the filtering machines.

For comparison.

Dylan's filtered IP via SeFlow tests:

Filtered:

Download speed from CacheFly: 8.16MB/s

Download speed from Coloat, Atlanta GA: 6.47MB/s

Download speed from Softlayer, Dallas, TX: 4.66MB/s

Download speed from Linode, Tokyo, JP: 4.95MB/s

Download speed from i3d.net, Rotterdam, NL: 7.90MB/s

Download speed from Leaseweb, Haarlem, NL: 8.06MB/s

Download speed from Softlayer, Singapore: 4.28MB/s

Download speed from Softlayer, Seattle, WA: 7.21MB/s

Download speed from Softlayer, San Jose, CA: 4.93MB/s

Download speed from Softlayer, Washington, DC: 7.48MB/s

 

Test on a SeFlow dedicated without filtering:

 

wget freevps.us/downloads/bench.sh -O - -o /dev/null|bash


CPU model :  Intel® Core i7-3770 CPU @ 3.40GHz


Number of cores : 8


CPU frequency :  1600.000 MHz


Total amount of ram : 31942 MB


Total amount of swap : 18536 MB


System uptime :   81 days, 4:21,       


Download speed from CacheFly: 80.8MB/s


Download speed from Coloat, Atlanta GA: 9.00MB/s


Download speed from Softlayer, Dallas, TX: 10.1MB/s


Download speed from Linode, Tokyo, JP: 5.00MB/s


Download speed from i3d.net, Rotterdam, NL: 29.3MB/s


Download speed from Leaseweb, Haarlem, NL: 47.9MB/s


Download speed from Softlayer, Singapore: 5.43MB/s


Download speed from Softlayer, Seattle, WA: 11.1MB/s


Download speed from Softlayer, San Jose, CA: 8.51MB/s


Download speed from Softlayer, Washington, DC: 14.4MB/s


I/O speed :  739 MB/s