# iwStack, a Prometeus project - who is using them and network throughout



## drmike (Jan 13, 2014)

I've been testing an iwStack cloud instance tonight for a friend.

I am a fan of Prometeus and have had good experiences in the past there.

Ran one of the freevps derivative scripts and found the network performance everywhere super slow.  This is a fresh empty container, so nothing competing there for resources.


===== NORTH AMERICA =====

Download from Cachefly: 7.09MB/s 
Download speed from OVH(BHS), Canada: 1.41MB/s 
Download speed from Datashack/WSI, Kansas City: 1.45MB/s 
Download speed from TMS, Dallas, TX: 998KB/s 
Download speed from Softlayer, Dallas, TX: 1.36MB/s 
Download speed from Softlayer, Seattle, WA: 1.08MB/s 
Download speed from Softlayer, San Jose, CA: 1.08MB/s 
Download speed from Softlayer, Washington, DC: 1.80MB/s
Download speed from BurstNET, Scranton, PA: 1.10MB/s
Download speed from BurstNET, Los Angeles, CA: 703KB/s
Download speed from BurstNET, Miami, FL: 941KB/s
Download speed from BurstNET, Dallas, TX: 1005KB/s
Download speed from Volumedrive, Northern, PA: 1.40MB/s
Download speed from Turnkey Internet, New York Tech Valley (??): 1.14MB/s
Download speed from PhotonVPS, Los Angeles, CA: 762KB/s
Download speed from Fiberhub, Los Vegas, Nevada: 1004KB/s

Download speed from ServerComplete, Jacksonville, Florida: 1.19MB/s
Download speed from IOFlood, Phoenix, AZ: 749KB/s
Download speed from Dacentec, Lenoir, NC: 1.28MB/s

========= EUROPE ========

Download speed from Linode, London, UK: 6.56MB/s 
Download speed from Rapidswitch, UK: 4.28MB/s 
Download speed from Serverius, NL: 7.27MB/s 
Download speed from i3d.net, NL: 4.67MB/s
Download speed from Leaseweb, Haarlem, NL: 5.94MB/s 
Download from OVH (RBX), France: 1.47MB/s 
Download speed from Hetzner, Germany: 930KB/s 

========== ASIA =========

Download speed from Linode, Tokyo, JP: 669KB/s 
Download speed from Leapswitch, Pune, India: 1.19MB/s 
Download speed from HKcolocation, Hong Kong : 530KB/s 





I removed WANSecurity speed tests, as WANSec pulled their test files and speeds are invalid / script doesn't error check for that condition.

Can someone with iwStack or Prometeus VPS run some tests and compare notes here?


----------



## drmike (Jan 13, 2014)

Guess I should amend this some...

Is anyone aware of a network issue or perhaps speed limitation with filtering?  Realizing they are including "filtered" IPs at this point / what is being used?

Is there any current network degradation going on there that anyone knows of?


----------



## DomainBop (Jan 13, 2014)

drmike said:


> Guess I should amend this some...
> 
> Is anyone aware of a network issue or perhaps speed limitation with filtering?  Realizing they are including "filtered" IPs at this point / what is being used?
> 
> Is there any current network degradation going on there that anyone knows of?


Your tests differ greatly from mine.   These are the results from a production IWStack instance



> wget freevps.us/downloads/bench.sh -O - -o /dev/null|bash
> 
> 
> CPU model :  QEMU Virtual CPU version (cpu64-rhel6)
> ...






> wget http://cachefly.cachefly.net/100mb.test -O /dev/null
> 
> 
> --2014-01-14 04:35:55--  http://cachefly.cachefly.net/100mb.test
> ...


----------



## cubixcloud (Jan 13, 2014)

Have your friend opened a ticket with them and find out. That's probably the best source right now.


----------



## drmike (Jan 13, 2014)

DomainBop said:


> Your tests differ greatly from mine.   These are the results from a production IWStack instance


Very different and strange.

What OS / ISO image are you using?


----------



## budi1413 (Jan 13, 2014)

@drmike Do you do this?

nano /etc/sysctl.conf



```
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
```

sysctl -p


----------



## drmike (Jan 13, 2014)

budi1413 said:


> @drmike Do you do this?
> 
> nano /etc/sysctl.conf
> 
> ...


Nope, giving that a spin now   

Reboot required?


----------



## DomainBop (Jan 13, 2014)

drmike said:


> Very different and strange.
> 
> What OS / ISO image are you using?


Debian Squeeze 64-bit


----------



## drmike (Jan 13, 2014)

Running Debian 32-bit minimal....

I made the tweaks recommended by @budi1413 (thanks friend!)


Download speed from CacheFly: 7.29MB/s 
Download speed from Coloat, Atlanta GA: 6.84MB/s 
Download speed from Softlayer, Dallas, TX: 7.32MB/s 
Download speed from Linode, Tokyo, JP: 3.40MB/s 
Download speed from i3d.net, Rotterdam, NL: 8.08MB/s
Download speed from Leaseweb, Haarlem, NL: 7.94MB/s 
Download speed from Softlayer, Singapore: 4.20MB/s 
Download speed from Softlayer, Seattle, WA: 4.75MB/s 
Download speed from Softlayer, San Jose, CA: 6.13MB/s 
Download speed from Softlayer, Washington, DC: 7.69MB/s 


Those are "better". Still far short of speeds @DomainBop is seeing


----------



## wlanboy (Jan 13, 2014)

It states that it is a "CPU model :  QEMU Virtual CPU version (cpu64-rhel6)".

Did you activate virtio drivers for the network?

Whould be nice to see some traceroutes to NL/UK/NY/DAL/ATL/SEA/LAS.

What upstream providers are they using?


----------



## drmike (Jan 13, 2014)

Looking around their panel control for the VIRTIO here in a bit.

Upstreams I am unsure of.


----------



## Dylan (Jan 13, 2014)

I just spun up two brand new instances, one with a filtered IP and one without. Based on these results I'm guessing your instance is filtered, drmike, and DomainBop's isn't?

Regular:


Download speed from CacheFly: 76.4MB/s

Download speed from Coloat, Atlanta GA: 8.93MB/s

Download speed from Softlayer, Dallas, TX: 14.4MB/s

Download speed from Linode, Tokyo, JP: 5.10MB/s

Download speed from i3d.net, Rotterdam, NL: 27.8MB/s

Download speed from Leaseweb, Haarlem, NL: 47.5MB/s

Download speed from Softlayer, Singapore: 6.87MB/s

Download speed from Softlayer, Seattle, WA: 14.8MB/s

Download speed from Softlayer, San Jose, CA: 9.04MB/s

Download speed from Softlayer, Washington, DC: 20.7MB/s


 

Filtered:

Download speed from CacheFly: 8.16MB/s

Download speed from Coloat, Atlanta GA: 6.47MB/s

Download speed from Softlayer, Dallas, TX: 4.66MB/s

Download speed from Linode, Tokyo, JP: 4.95MB/s

Download speed from i3d.net, Rotterdam, NL: 7.90MB/s

Download speed from Leaseweb, Haarlem, NL: 8.06MB/s

Download speed from Softlayer, Singapore: 4.28MB/s

Download speed from Softlayer, Seattle, WA: 7.21MB/s

Download speed from Softlayer, San Jose, CA: 4.93MB/s

Download speed from Softlayer, Washington, DC: 7.48MB/s

 

I wonder if this speed difference is normal. I'd expect less on the filtered, but this much?


----------



## MannDude (Jan 13, 2014)

I'd tag prometeus and Mao_Member_no_signature for a comment. Either one should be able to touch base on this.

Also, I didn't know they did DDoS filtering... so that's neat. Haven't really looked at iwStack much but may give it a peak.


----------



## maounique (Jan 13, 2014)

Hello !

Due to the semi-ban (rMember special group where people which do not agree with the admins are placed), wont be able to edit my post, so please forgive the typos and multiple posts (eventual).

1. We used to have a 200 mbps limit so the 1 gbps port of each node will not be hogged by 1-2 people. That is removed now because we havent detected such a behaviour.

2. I have seen that happening before, usually for small instances (384 MB). Those are not routed differently, it is probably an issue generated from the lack of virtio drivers, I was never able to replicate it.

3. @DomainBop, we upgraded te storage, now we have about 700 MB/s and more than double IOPS, you will need to re-deploy the instance or open a ticket to have it migrated to the new storage. Only the smallest instances are still on the old storage (that will probably be kept and offered at a lower price, we did not make a decision yet).

4. SSD local storage will probably be added this month.


----------



## wlanboy (Jan 13, 2014)

Dylan said:


> I just spun up two brand new instances, one with a filtered IP and one without. Based on these results I'm guessing your instance is filtered, drmike, and DomainBop's isn't?
> 
> Regular:
> 
> ...


Yup that's an upstream problem.

Do a "wget http://cachefly.cach....net/100mb.test -O /dev/null" on both.

We will see two different ips.

Would be interesting to see who is filtering the traffic.


----------



## budi1413 (Jan 13, 2014)

drmike said:


> Nope, giving that a spin now
> 
> Reboot required?


Don't need reboot.


----------



## maounique (Jan 13, 2014)

Hello again !

The filtered IPs are indeed slower, we have partenered with seflow.it for them. They should be used only if you are under DDoS risk.

Our DDoS protection is mostly aimed to protect our network and as an insurance it will not happen that we have to suspend or nullroute production services. It is not for DDoS refugees operating game-servers, for example.

If you do not have VIrtio drivers, the performance will be hit.

You can enable them by selecting our templates (they have it), our ISOs (also included) or by selecting other PV 64/32 when inserting your own ISO.


----------



## drmike (Jan 13, 2014)

So to biz side of things...

This is a small instance like you described #2. 

Where is the virtio driver setting?  Take it is in the Cloud Panel right?  Haven't looked yet.

Can you push the /etc/sysctl.conf optimizations to the templates?  That certainly helped and might be fine for lots of folks as is...?


----------



## maounique (Jan 13, 2014)

wlanboy said:


> Would be interesting to see who is filtering the traffic.


We have partnered w Seflow.it for the DDoS protection, that is 100 mbps per IP guaranteed. It might be that their protection is triggered on tests due to very big speeds from cachefly and all the rest of the tests are over the filtering machines.


----------



## DomainBop (Jan 14, 2014)

Mao_Member_no_signature said:


> We have partnered w Seflow.it for the DDoS protection, that is 100 mbps per IP guaranteed. It might be that their protection is triggered on tests due to very big speeds from cachefly and all the rest of the tests are over the filtering machines.



For comparison.

Dylan's filtered IP via SeFlow tests:



> Filtered:
> 
> Download speed from CacheFly: 8.16MB/s
> 
> ...


 

Test on a SeFlow dedicated without filtering:

 




> wget freevps.us/downloads/bench.sh -O - -o /dev/null|bash
> 
> 
> CPU model :  Intel® Core i7-3770 CPU @ 3.40GHz
> ...


----------



## maounique (Jan 14, 2014)

Yes, as I said, we have the 100 mbps version over the filtered IPs, that should be enough of filtered traffic under DDoS. I am guessing that speed tests, due to very high speed (since cachefly is in same DC) are triggering the DDoS protection so for a while everything is kept under 100 mbps.

I will have to ask uncle to chec logs see if that is the case (protection triggering too fast/easy) and see what can be done, we are still tweaking it.


----------



## peterw (Jan 14, 2014)

I always buy additional filtered ip. Other filtered hosters like Liquidsolutions and Buyvm do have bad download speads on filtered ips too.


----------



## MannDude (Jan 14, 2014)

Thread has been cleaned up to remove the drama as it's being handled privately.


----------



## drmike (Jan 14, 2014)

Thanks to all.

Guess I'll try to get a container allocated fresh and steer clear of filtering.   Filtered IP is the default selection currently on installs.  That's typically what I recommend - default stuff until familiar with quirks of provider.


----------



## maounique (Jan 14, 2014)

Hum, well, it is default because was the latest added, this will need to change.

Thanks for your input.


----------



## maounique (Jan 14, 2014)

Update (sorry, cannot edit as you know):

We will be adding another range when it will be possible and needed so DDoS protected services will not be the default because this cna create problems later on in march when those will be charged.

The income (download) stream is over seflow, outgoing (upload) is from prometeus, so, the total incoming over filtered pipe is 200 mbps. It looks like too many people took the DDoS protected IPs and direct a lot of traffic over them. I am sorry for this situation, if you have this issue, please change the network offering by making a template and redeploying it.

The IP will change (please leave on DHCP all the time, never hardcode the IP in templates) and remove any mention of a MAC address. If you are using our templates, that is not necessary in most of them, if you installed from ISO it is most likely needed as you will get another MAC address.


----------

