vpsBoard - Community input and feedback on our infrastructure and future

[drmike]

Choose a reliable host and you won't need some hacked up HA setup with donated virtual servers from shovehost sniffing for login data. 

Well, this sort of HA setup as proposed will work and does work.  There are MANY HUGE sites that do essentially this.  Cloudflare does this essentially.

The sniffing for login data part, that's a real concern though.

Shouldn't be plugging these front end nodes on to just any network anywhere.  I'd start with BuyVM and SecureDragon since both reputable and both have DDoS protection services.  But, both are purely US West Coast so far.

Choose a reliable host and you won't need some hacked up HA setup with donated virtual servers from shovehost sniffing for login data. 

Reliable hosting is just one part of this.  Choosing Telx or Equinix solely isn't going to make everything run right and provide redundancy and geo-balancing --- unless you get setups in multiple geographic locations from them ==== $$$.  Plus the DDoS protection.

Varnish caches MySQL requests as well s

Does IPB support such or is Varnish being ran as a MySQL proxy to facilitate that?  Caching on database layer creates tons of issues unless plugins/mods exist in IPB to handle and control such.

Toss CloudFlare in front of that, you're good. 

CloudFlare for all it's success has plenty of failings.  See what they do when someone tosses a DDoS at the site for 12 hours.   Their service is good, but whatever the top advertised package is, they'll have you up there paying in no time for being a hassle

.

The minute someone finds out where any of your non-ddos protected assets are and takes them down. Or when one of them goes down. If you host it with one SQL server, you still have a single point of failure. If you make some six-servered beast for a small forum, it's not going to be a good time. Sure, one of your load balancers might be able to go down, but when latency gets ahold of you and it's taking 500ms just to get all of the queries for a page-load, you might as well just not. 

Where to start with this one...  How would someone find the front ends?  I suppose they could.  But you shouldn't be advertising them, so kind of hard.

If a front end node goes down (Varnish + proxy) you will end up needing to pull that node from the pool.  That will be best done at DNS level with a real API enabled monitoring service.

Databases over the internet = bad and high latency.  Don't even bother if you are thinking that.

 The Varnish + proxy stack should have MySQL bundled and the webserver also.   So each and every location can fully do everything.   It's a many to many distribution.  Complicated, yes.

backend which are in close proximity to the frontend

Close proximity better be the same datacenter.  Better be < 5ms.  Still high as heck and big delay.

advertising off the site

Unsure why the advertising hate exists.   I HATE ADVERTISING, because it is usually off topic and not relative to me.  Ads here?  Well, they would be relative to my interests and much of what I discuss.  So it is not evil.  Plus ads would be from productive community members.

 

[drmike]

I've set up MySQL clusters before and watched network latency or downtime cause them to go haywire.

^ this.  

 

[TheLinuxBug]

@Ruchirablog, Not really, already have 90% of the config already done and available for this.  Granted we may not need as many servers as I had proposed originally, in fact as stated this could be reduced to:  BuyVM DDOS Protected server with Varnis in front of 2 different KVMs in round robin with 256Mb ram that pull from BuyVMs offloaded MySQL server.  My goal once again is to show that expansion actually isn't that hard.  Also, it would allow multiple people to donate services to help this out, and in case of an issue where someone can no longer provide such services, would not cause an issue with the site.   As I have stated a few time, please, make suggestions I am open to them.  However, please understand that I really do have most of this down pat and it really isn't as hard to setup as you may  think with a little bit of experience and some trial and error to figure out the idiosyncrasies of iPB.

Edit:  Okay everyone, thanks for participating, and please keep up your suggestions, however it is time for me to get some sleep.  I don't make any final decisions, everything contributed here will be weighed out by MannDude and he will have final say on anything that happens.  My attempts here are just to allow the community to drive this site instead of the advertising (as a preference).

Once again, thanks to everyone who participates in this discussion.

Cheers!

 

[manacit]

The idea of the setup is good, sure, but large sites that do this generally have dedicated hardware and full-time people managing the setup and solving issues that arise. 

CloudFlare might not be a magic pill, sure, but it's at least more battle-tested than most (all) of the LEB hosts around here are, not that it matters, since this is a tiny forum. 

The fact that you say Varnish caches MySQL queries is troubling - varnish is a reverse *HTTP* proxy. 

My worry is that people will be able to figure out what is hosted where based on the downtime of LEB hosts. If we have hosts publicly donating things, when something goes down (pulled out of rotation, etc), it wouldn't take a genius to figure out which hosts were down and then just DDoS the crap out of them. We've seen more cunning things happen recently. 

 

[Ruchirablog]

@Ruchirablog, Not really, already have 90% of the config already done and available for this.  Granted we may not need as many servers as I had proposed originally, in fact as stated this could be reduced to:  BuyVM DDOS Protected server with Varnis in front of 2 different KVMs in round robin with 256Mb ram that pull from BuyVMs offloaded MySQL server.  My goal once again is to show that expansion actually isn't that hard.  Also, it would allow multiple people to donate services to help this out, and in case of an issue where someone can no longer provide such services, would not cause an issue with the site.   As I have stated a few time, please, make suggestions I am open to them.  However, please understand that I really do have most of this down pat and it really isn't as hard to setup as you may  think with a little bit of experience and some trial and error to figure out the idiosyncrasies of iPB.

Cheers!

Still this wont be a HA setup because if BuyVM DDOS protected IP and server or offloaded mysql goes down, vpsb will go down. Achieving 100% HA isn't that important for a community web site. I don't understand why even 99% availability wouldn't suffice for a site like vpsb. 

 

[drmike]

it wouldn't take a genius to figure out which hosts were down and then just DDoS the crap out of them.

True to an extent.  But there are multiple front ends.   

They would have to know all the front ends and DDoS multiple networks.

I've ran a similar solutions for 4+ years without any major issues.

Biggest piece is monitoring the top nodes and getting them pulled out of the pool quickly when downtime happens.

 

[drmike]

Still this wont be a HA setup because if BuyVM DDOS protected IP and server or offloaded mysql goes down, vpsb will go down.

True.

For this to work right each and every front nodey should have full stack to serve the site plus DDoS protection from that provider/network.

 

[vanarp]

While I am not against this idea, I am concerned of the amount of time and effort required from Sysadmin point of view. And depending on volunteers for the sysadmin/support is probably not a good idea.

I propose to take one step at a time. A bigger KVM with Pure SSD to run MySQL/Varnish locally + CloudFlare to be the first step. See how it goes for a while before bringing in more components into the setup.

 

[manacit]

True to an extent.  But there are multiple front ends.   

They would have to know all the front ends and DDoS multiple networks.

I've ran a similar solutions for 4+ years without any major issues.

Biggest piece is monitoring the top nodes and getting them pulled out of the pool quickly when downtime happens.

My worry is less about the front ends and more about the back ends. If you're hosting the SQL in one place like many have said, you can put as many DDoS protected servers up front as you want, the minute someone DDoSes your SQL server, it's toast. Same with any intermediary servers, etc. 

 

[Ruchirablog]

CloudFlare might not be a magic pill, sure, but it's at least more battle-tested than most (all) of the LEB hosts around here are, not that it matters, since this is a tiny forum. 

My worry is that people will be able to figure out what is hosted where based on the downtime of LEB hosts. If we have hosts publicly donating things, when something goes down (pulled out of rotation, etc), it wouldn't take a genius to figure out which hosts were down and then just DDoS the crap out of them. We've seen more cunning things happen recently. 

It wouldn't take a genius to find the real ip of vpsb even its masked behind the cloudflare.

@Manndude Remove the function of uploading profiles pics via a url. It only takes few seconds to find the real ip of vpsb by using a simple service like iplogger.org 

 

[manacit]

I propose to take one step at a time. A bigger KVM with Pure SSD to run MySQL/Varnish locally + CloudFlare to be the first step. See how it goes for a while before bringing in more components into the setup.

This is *exactly* what I'm saying. But much more succinct. 

Also, Varnish is nice and all, but you'll likely end up not caching a lot for logged-in users, which makes it not quite as worthwhile as you'd want. 

 

[blergh]

Rent a dedicated server and then set up one large KVM container that uses all of the available resources, so we get the power of a dedicated server while 'technically' still using a VPS

Why?

 

[peterw]

For this to work right each and every front nodey should have full stack to serve the site plus DDoS protection from that provider/network.

A lot of money you are talking about.

 

[drmike]

A lot of money you are talking about.

Well, perhaps.

1GB VPSes should be doable.  2GB should be plenty of overkill space.  Separating the software stack on a site like this will be unnecessarily complicated and add unnecessary latency.

Frankly we need to revisit the affordable DDoS protection options out there.  CNServers seems to be what many use.   So SecureDragon + BuyVM would have common CNServers issue.

There aren't many options for DDoS protection.  At affordable rates, well even less.

 

[KuJoe]

http://www.lowendtalk.com/discussion/3825/raymii-org-got-reddited-lwn-ed-and-news-ycomb-ed-on-a-128mb-leb-with-stats

^Why not do this?

Additionally, I've be happy to donate a VPS from Tampa, Denver, and Portland (DDOS Protected) for geographic diversity. While we also use CNServers like BuyVM, we are hosted inside of CNServers so the network layouts are different.

 

[TheLinuxBug]

@, There are a few different options as I see it for DDOS protection:

Europe:

Girenet:

http://lowendtalk.com/discussion/11725/ginernet-4-99-month-ssd-openvz-4-gb-ram-ddos-protected#latest 

OVH:

Supposedly if their DDOS protection is any good, any $3/month OVH OpenVZ VPS should be able to function with DDOS protection.

USA:

BuyVM

ddosprotect.us (Secure Dragon)

Staminus

Really the "issue" if there is one is finding a service that is on the east coast of USA to handle DDOS, but with at least 2 locations it shouldn't be a huge issue. I haven't tried Girenet, but I would be interested to see how they perform.

A lot of money you are talking about.

The largest output of money would be for the DDOS protected frontend servers. The hope is that the backend servers could be contributed.  

1GB VPSes should be doable.  2GB should be plenty of overkill space.  Separating the software stack on a site like this will be unnecessarily complicated and add unnecessary latency.

I am completely open to building out the backend servers with Nginx+php-fpm and MySQL on the same server.  We could use some decent 2GB SSD powered servers and there shouldn't be an issue with having MySQL on the backend instead of separated.

I will say this again for everyone's benefit:

What I listed above is a general IDEA, the goal of this thread is to find the BEST idea.  This could mean changing what I have proposed.  This is the information that we are seeking to get in this thread, not the "It can't be done" but the "This is how we can make it happen". 

It seems no providers are participating (well other than @Kenshin (thanks btw)), I was hoping a few of the guys who were offering to help out before would have some input on this as well.  I hope your not holding out just so you get ads plastered all over the place.... 

Cheers!

 

[TheLinuxBug]

Additionally, I've be happy to donate a VPS from Tampa, Denver, and Portland (DDOS Protected) for geographic diversity. While we also use CNServers like BuyVM, we are hosted inside of CNServers so the network layouts are different.

Thanks for being a team player @KuJoe  

Edit: My post was after yours, but I was still working on it when you posted.  I am happy to see you participating here, Thanks again!  

 

[manacit]

http://www.lowendtalk.com/discussion/3825/raymii-org-got-reddited-lwn-ed-and-news-ycomb-ed-on-a-128mb-leb-with-stats

^Why not do this?

Additionally, I've be happy to donate a VPS from Tampa, Denver, and Portland (DDOS Protected) for geographic diversity. While we also use CNServers like BuyVM, we are hosted inside of CNServers so the network layouts are different.

That setup is a lot simpler because his site is all static content without databases. A round robin DNS setup with a bunch of HTML files is easy-peasy. 

I still think we shouldn't jump into anything crazy, just get a big KVM that's DDoS protected, run the site on that, and then go up from there if we need it. 

 

[Kenshin]

I used to run a large local (SG) community forum, as well as currently running MySQL replication (master-slave) for both a stable distributed qmail setup as well as for shared hosting use (slave for quick failover + backing up without locking). I've read up a lot on master-master setups but never actually put it into testing or production, simply because for every 1 success story I read, there are 9 others with issues. It might work on the qmail (vpopmail) setup since the data is pretty straightforward, but for forum data it's practically suicide.

If Cloudflare is an issue, then just re-create the Cloudflare component with the use of multiple reverse proxies on DDOS protected IPs at various locations. Performance of the MySQL server is an issue, and if you take into account Layer7 attacks I'd rather investment in a dedicated server with SSD to ensure MySQL or Nginx can handle the load. Dedicated servers aren't even that expensive anymore, but having the available CPU & I/O would allow not just growth but also tanking of simple L7 attacks.

My proposed setup:

1-2 DDOS protected IPs running Varnish, ideally west coat + east coast/EU

1x Primary web server located in east coast or central USA. Dedicated server with SSD drives. MySQL master.

1x Secondary web server located in west coast or EU. VPS with SSD drives. MySQL slave.

1x Backup server. Daily backup of web files and database to enable at least 7 day complete roll back.

CDN for static elements to reduce http load and increase loading speed

By speeding up the page generation via dedicated server w/SSD, it can offset the additional latency required for the Varnish <-> web server, which would be the main traffic. Moving all static elements to CDN would speed things up for users like me who are in Asia and are always at a disadvantage. My aim is to reduce the overall page completion time, as well as have sufficient resources for growth/L7 attacks.

I left Manndude a message on IRC about offering OnApp CDN (pull based) for the static elements as long as he can integrate it into IPB. I have my own asia CDN POP so technically it's "free" for me to provide, and pricing for US/EU CDN POPs are cheap it won't hurt me much financially.

 

[TheLinuxBug]

Sooo... does anyone else have any input on this?  I was hoping to see a few more suggestions or ideas?

Cheers!