vpsBoard - Community input and feedback on our infrastructure and future

[TheLinuxBug]

Hello Everyone,

Over the past month there have been several topics going back and forth, from selling Ad space to finding a way that we can make this place fully community supported. MannDude and I have been going back and forth in private message for a few weeks now on this topic and we thought that it would be something that we should allow the community to have some input on.

Currently vpsBoard's infrastructure is relatively small, as the site is hosted on a BuyVM 256Mb KVM instance and using BuyVM's offloaded MySQL services for MySQL. So far this has been a more than reasonable start, however, going forward as this community grows, this will need to change to keep things moving and working at reasonable speed.

My proposal to MannDude was to create a cluster of servers, instead of using just one or two large servers. Now there are probably many ways to do this, but we are discussing using not only the services that are the topic of this board (VPS) but also trying to keep this as low cost and community driven as possible. There has recently been a few different posts on Varnish here, so as you can imagine this is one of the parts that was discussed.

So, my proposed method was to use several DDOS protected VPS server as reverse proxy entry ways, with several backend web servers to provide the non-static content and use at least two reasonably sized SSD VPS servers in a Master-Master replication to provide the MySQL services needed by the cluster. On the reverse proxy using Varnish and Haproxy (to provide better session control as forums are very dependent on session cookies).

When discussing it with MannDude I threw together a quick Visio drawing of what this might look like, to give you a better idea of what I am talking about graphically. You can find the drawing here. 

About  this point, you are probably wondering, what does this have to do with me? Right? 

One of the reason I suggested  this clustered method is because we would be able to allow providers and community members alike who would like to contribute to this infrastructure an opportunity to do just that.  For the back end servers we could allow the community to provide these instances, allowing the cost to MannDude to be reduced.  Okay, so now your next question is likely. "How do we do this while being provider neutral?"  Well, using this type of setup would allow people to contribute resources in an anonymous manner as the only ips the members see when coming to the site would be those of our entry proxy servers and not those of the backend resources. Meaning, anyone who is willing to contribute is not going to be directly advertised, however, MannDude has said he will give an honorable mention to those who do decide to help (we have not yet determined how that would be displayed).  Those who do contribute also can not claim to host the site them selves, as it would be a distributed infrastructure.  In a case where say one contributor is not able to provide said service any longer, we could remove it from the cluster and still continue to run without much immediate effect.  Then, as we can, replace the node with a new contributed server.

All this said, maybe the way I have mentioned is not the best way and is certainly not the only way, so at the same time I am bringing this option to you, we are still very much open to some more suggestions on how this can be done or even made better.  So please, leave feedback, make suggestions and lets come up with a way to keep this a community initiative instead of having to have Ads plastered everywhere.  

Also, if this is an interest to you and you wish to help build this infrastructure or help to maintain it and you have some skill sets you would be willing to offer for free to help us out, please speak up and let us know.

 

[Kenshin]

Having Master-Master MySQL servers across west/east coast just sounds like a nuclear bomb waiting to blow up. It'll make more sense if you just found a primary central US server with good routing between the west/east coast reverse proxies, then run slave replication (and backups) for quick failover in case crap happens. Much more deterministic failover rather than having to deal with master-master breakage.

 

[texteditor]

Rent a dedicated server and then set up one large KVM container that uses all of the available resources, so we get the power of a dedicated server while 'technically' still using a VPS

 

[shawn_ky]

I'd like to help, but at this point don't have the skillset... I'd like to learn more because this is something of what I'd like to do myself even. Don't really want to depend on one provider..   Count me onboard for whatever I can do to help, even if it meant buying a premium membership or something to aid in the cause...

 

[drmike]

Master-Master... Can't say I've tried that. Who has done that in MySQL without an issue?

I won't kick this idea, as I implement the very same general idea

I see Varnish to HAProxy.  Why both?   Varnish can do whatever HAProxy can for a project like this (I think).

How is the determination being made as to where a user ends up east or west?  GEO-DNS I take it....?

 

[TheLinuxBug]

Master-Master... Can't say I've tried that. Who has done that in MySQL without an issue?   I won't kick this idea, as I implement the very same general idea   I see Varnish to HAProxy.  Why both?   Varnish can do whatever HAProxy can for a project like this (I think).   How is the determination being made as to where a user ends up east or west?  GEO-DNS I take it....?

I have done Master-Master replication with MySQL 5.5, it actually works pretty well.  I suppose in a forum with a lot of activity that @Kenshin may be correct that having Master-Slave may be a bit more quick and reliable.  If the replication lags you could see a new item on one server and it not yet show up on another.  In my experience though, running one server in San Fran, CA and one is Ashburn, VA I have only seen this be an issue a few times (and usually it corrects in less than half a minute).  Also if one of the servers fails it can sometimes be a delicate process to get things resynced, but I haven't had that many issues with it. 

Most forums provide session cookies and such specific to said server you are accessing.  Unless you share the session directory on each backend server (using NFS, SSHFS, etc). For the best redundancy it isn't advised to use such remote mounts though, because if the mount server dies.. well then you have a huge issue.  Using HAproxy you can have it continue to send said user to the same backend (using a special cookie it inserts) which will allow you to keep your session while still distributing load between servers.  In the case where one backend fails, then it would provide you with the other backend with the only caveat being if you have not checked "Remember me" you will have to re-login to the forum. 

Depending on the setup, yes, GEO-DNS would be used to route the client to the correct reverse proxy for their region.

Hopefully that helps clarify things a bit.  If you have any other questions I can answer for you, please let me know.

Cheers!

 

[Epidrive]

Why not just go w the new OVH infa. It is certainly a great solution to this though you must have to wait until its implemented. (P.s. they now offer ddos protected servers for almost the same normal amount you pay them per month)

 

[TheLinuxBug]

Why not just go w the new OVH infa. It is certainly a great solution to this though you must have to wait until its implemented. (P.s. they now offer ddos protected servers for almost the same normal amount you pay them per month)

 

Part of the goal (in my opinion) is to use VPS (as this is a board about VPS).  Also, we are trying to keep this low cost, and we will be asking providers if they wish to contribute some resources.  Another consideration is we would like to have availability in more than a single location (redundancy) in case something does happen to the server like the issues we have seen in the past with BuyVM. 

Cheers!

 

[drmike]

Most forums provide session cookies and such specific to said server you are accessing.  Using HAproxy you can have it continue to send said user to the same backend (using a special cookie it inserts) which will allow you to keep your session while still distributing load between servers.  In the case where one backend fails, then it would provide you with the other backend with the only caveat being if you have not checked "Remember me" you will have to re-login to the forum. 

Typically server determination from a pool of available backends is done via simple hashing.  Hashing of the requesting party's IP address or of the URL being requested.

There has to be a way to accomplish that inside of Varnish.  Been years since I dabbled with Varnish.  Less pieces involved, the better always.

Rest of the plan sounds pretty good to me.

 

[manacit]

This is probably the most over-engineered setup possible for running an extremely simple, and relatively small, forum.

You're changing something that would be as simple as 

Request <-> Internet <-> (Server | Database) - Traditional, quick, easy. 

to

Request <-> Internet <-> Backend Server <-> Internet <-> Database <-> Other Database - Downright unwieldy. 

For what, to save 80ms of latency at most? It's going to cause you nothing but headache and time wasted. Not to mention the security concerns... 

Sure, you want it to be cheap. Know what else is cheap? Hosting it on a VPS. You don't even need a dedicated server at this point, just a VPS with a couple of gigs of RAM. Set up some caching, and it'd be quick (on a nice SSD or even HDD VPS, it'll probably still be quick!).

Hacky HA is only going to cause more problems than it'll fix.

You could pick up a small dedicated server that would host this until it dies, I guarantee it. You could likely even host it on a medium sized VPS for a lot of time to come. 

 

[TheLinuxBug]

There has to be a way to accomplish that inside of Varnish.  Been years since I dabbled with Varnish.  Less pieces involved, the better always.

If you know of a way to accomplish  this I am all ears, I was actually trying to figure this out my self and hadn't had much luck yet. (On a setup I already have running with a clients forum)

@manacit, Part of the goal is redundancy, you can't accomplish that with a single server.  Also, once again the other idea is to reduce cost, purchasing dedicated servers does not help with this.  Though the stats on my drawing are extremely high end examples (as I stated I did it VERY quickly), as it is currently running on a 256Mb VM, the setup once fully configured would probably be made of 4 backend servers (2 backends for each frontend) with 256-512MB ram,     10-20Gb Hard drive and ~500-1000MB bandwidth.  The MySQL situation is still up for debate but I believe that could be done on 1-2 SSD VPS with 512Mb-1Gb ram. 

Edit: Also one important thing you may not know is that Varnish caches MySQL requests as well so unless it is brand new content, you will not see as much action to the MySQL server as you might think. 

Cheers!

 

[manacit]

You guys are talking about "sticky sessions" - http://mesmor.com/2012/02/15/varnish-client-director-with-sticky-session/

You're turning a ~300 active-user forum into a six-server setup. The only reason you're "saving" money is because you want people to give it to you for free. If you actually had to pay for your setup, it'd be as much as an OVH server that would end up being more powerful and able to serve more people.

That HA setup is going to go down the trash the minute one host starts acting up and your SQL cluster gets messed up and someone needs to go in and fix it manually, causing the site to query a database that's 120ms away 15 times a page load slowing everything down even more. 

Choose a reliable host and you won't need some hacked up HA setup with donated virtual servers from shovehost sniffing for login data. 

 

[TheLinuxBug]

@manacit, Once again, how does a single server provide redundancy?   Honestly, this could run with 1 MySQL server and 2 backend servers running in round-robin with out issue.  The idea here is to show that expansion is easy and that as a COMMUNITY we can come together to provide the resources needed so we don't need to turn this place into an advertising outlet.  

Edit:  Thanks for the link to that article on sticky sessions, however in my works with vBulletin I was not able to perfect that, maybe with some help we could make it happen for here and remove a step. 

That HA setup is going to go down the trash the minute one host starts acting up and your SQL cluster gets messed up and someone needs to go in and fix it manually, causing the site to query a database that's 120ms away 15 times a page load slowing everything down even more. 

Actually I have this type of setup already in place for a clients forum and it is very reliable.  Also I think @Kenshin made a valid point that Master-Slave may be a better way to go.  This is all up for discussion, nothing is in stone, heck this hasn't even been started.  Maybe you can offer a better solution that works with redundancy and provides the high availability needed?  Also, please read my previous reply to you about Varnish and it caching MySQL requests, thus there is no need for the "15 times a page load" you are talking about.

Cheers!

 

[manacit]

My oh-so-not-subtle point was that you don't need redundancy, just get a server somewhere that's not a super-budget-lowend host that get's DDoSed all the time and is hosted in a facility held together by zipties (not literally, sorry FiberHub, but you need to work on that whole power thing), and you won't go down very often/at all. Toss CloudFlare in front of that, you're good. 

You know what will negate any "redundancy" - the minute someone finds out where any of your non-ddos protected assets are and takes them down. Or when one of them goes down. If you host it with one SQL server, you still have a single point of failure. If you make some six-servered beast for a small forum, it's not going to be a good time. Sure, one of your load balancers might be able to go down, but when latency gets ahold of you and it's taking 500ms just to get all of the queries for a page-load, you might as well just not. 

 

[TheLinuxBug]

My oh-so-not-subtle point was that you don't need redundancy, just get a server somewhere that's not a super-budget-lowend host that get's DDoSed all the time and is hosted in a facility held together by zipties (not literally, sorry FiberHub, but you need to work on that whole power thing), and you won't go down very often/at all. Toss CloudFlare in front of that, you're good. 

You mean like when BuyVM has planned maintenance so the site goes down?  Cloudflare has also been excessively flakey, and I believe MannDude is considering removing it all together for something different at this point.  If someone has the ability to flood out multiple locations which provide DDOS protection then there is a bigger problem outside the actual hosting setup, as I imagine this would mean great amount of data being pushed at these locations.  I highly doubt hosting it at OVH would help with that, I am sure even with their protection after a certain point they likely null route you (we will see once people start using it). 

Part of the goal here would be to get servers for the backend which are in close proximity to the frontend to avoid the large amount of latency, thus my wanting two backends for each front end.  

At the end of the day, there are a lot of things we could do about this, hell we can sell out just like Chief on LET to some faceless corporation and have them pay all the bills, or we can plaster the whole site with advertising?  However, my goal, once again, is to keep this a COMMUNITY driven site and to do that we need to keep the costs down.  Right now this whole forum is running on a 256Mb KVM  from BuyVM w/ MySQL offload, meaning we do not need a dedicated server amount of resources to run it.  Even with double the clients we could probably handle the load with just an extra KVM somewhere and a reverse proxy in front. 

I invite you to come up with a solution that is better than this which keeps costs of running low, advertising off the site, and a way for the community here to contribute.  If you have can some up with something better, I am more than happy to listen

Cheers!

 

[MannDude]

For the record, I am a fan of simplicity. I'm still not sold on this idea presented in this thread but am open to it.

Basically, all I want is the site to remain online with good performance to the rest of the world. Initially I was going to sell ad-space so I can cover the cost of an east coast USA KVM or dedicated, put up Cloudflare Business DDoS protection and call it a day. But in an effort to attempt to keep vpsBoard ad-free, and to keep it a community supported (and not advertiser supported, even if the advertisers are the community), I am open to this option as well.

 

[manacit]

You've gone from a six+ vps set up to "an extra KVM" - that's pretty much my point, it's a small site, there's no need for so much infrastructure. That's how you keep it cheap. 

The idea would be to host somewhere slightly more expensive than BuyVM to avoid ending up somewhere like FiberHub which lacks any sort of power redundancy, etc. 

My solution? People have offered to help the site out with a KVM. Use that to host the site, not as one of a million different cobbled together servers. 

Clustering your x near your y is great until your y goes down and then you have to communicate from x to z a bunch, that's not very available. 

 

[TheLinuxBug]

@manacit, Now I am curious.  Have you ever hosted a forum?  Do you understand the use of Varnish? Do you have experience to say that this is really as complicated as you say it is going to be?  I actively manage a setup similar to this for a forum, and I have never seen the dooms day issues you are describing?   

Cheers!

 

[Ruchirablog]

This gotta be the most craziest hosting idea I ever seen for this type of web site. Jeez dude! You are making this over complicated. vpsb doesnt really need this at the moment nor future. Managing this type of cluster is too much work and too many points of failure. Before jumping in to conclusions you should think about weak points of current setup. And that list is small and there are many ways to improve on that before investing money and more importantly time for implementing an idea like yours. 

 

[manacit]

Yes, I have hosted a forum (lol), and yes, I have used Varnish (I was the one who linked to you sticky sessions, which I have successfully implemented, actually). 

I've set up MySQL clusters before and watched network latency or downtime cause them to go haywire.

What does your setup look like? I'm happy it works for you, but it's total overkill in this situation.