amuck-landowner

lowendbox.com and lowendtalk.com down

wlanboy

Content Contributer
Because I got a lot PMs, emails and queries on IRC:


The image of the uptime is a screenshot of my statuscake.com account.
 

mikho

Not to be taken seriously, ever!
Can anyone please explain what the deal with LET and cluster is?
The tl;dr version


CC installed a cluster to serve LE*, problems like databases are not synced, data was lost(whole days) were lost, then found.


It's been up, then down. Everytime someone makes an update somewhere, slmething else breaks.


Everyone blames the "cluster".
 

drmike

100% Tier-1 Gogent
Cluster f*cked.

They have --- or did ProxMox running.  1 IP was database, 1 was LEB, 1 was LET.

They were attacked and offlined in the past few weeks and jumping IPs.   Pretty sure for  a time they botched something or CloudFlare booted/nulled them... Exposed real IPs... Fun continued again.

At some point someone at CC thought if they put up more servers they could weather the attack.  Thus the cluster.

Biloh talked crap on some "contractor" who was working on this cluster for weeks or months prior.  Then chatted up his own team who implemented the cluster in a few days.     Pretty sure Kossen is the one who implemented this cluster she-bang.

I won't go into how to make this work right, they'll figure it out eventually.  It still with crumple when/if someone throws love at them. 

Did I read 7 node cluster for  LET? Hehe, I could run that site an ARM processor in a phone.   Can't fix stupid.   Like tanking DDoS because we have big pipe to absorb it.  Same mentality.
 

Raymii

New Member
More hardware isn't always the solution, but in this case it might be. Altough it better be combined with an upstream provider (or a good network admin) who knows and has the equipment to handle this.

I've set up quite a few clusters these days, and the database part is the hardest. Especially MySQL in multi-master setup is a crime. Postgres is a breeze nowdays (>9.1) with the WAL replication. Also, HAProxy is hard and Corosync/pacemaker between HAProxy is also not that difficult...

If they need help they can contact me :p
 

mpkossen

New Member
I really, really think they should be on separate physical servers to prevent both being down at once.
They are. Each have their own cluster.

We've had CloudFlare issues earlier this week, which I belive was limited to The Netherlands or an area around that. I'm not sure what has caused this outage, though.

More hardware isn't always the solution, but in this case it might be. Altough it better be combined with an upstream provider (or a good network admin) who knows and has the equipment to handle this.

I've set up quite a few clusters these days, and the database part is the hardest. Especially MySQL in multi-master setup is a crime. Postgres is a breeze nowdays (>9.1) with the WAL replication. Also, HAProxy is hard and Corosync/pacemaker between HAProxy is also not that difficult...

If they need help they can contact me :p
Galera could be used for multi-master MySQL.
 

RLT

Active Member
Looks down again. Got one cloudflare always up page then just the normal problem page after that.

Something has gone wrong.
We've run into a problem and are unable to handle this request right now. 
Please check back in a little while.
 
Last edited by a moderator:
Top
amuck-landowner