earl
Active Member
Yes.. but LEB without summer host is like fries without gravy.. and some of the drama we get it's like ketchup on top! lol
ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]
- Thread starter Magiobiwan
- Start date
-
- Tags
- chicagovps cvps exploit solusvm
- Status
Haha yea... Too bad LEB doesn't post offers from only reputable hosts... They pick and choose who they post...
Aldryic C'boas
The Pony
Alright, not to be a dick about this, but if you want to nitpick on words, fine... when exactly have you gone through a situation like this? What relevant experience do you have regarding this level of meltdown? When's the last majorly stressful disaster you went through where you were able to keep a cool head and not let the overwhelming gravity of the situation seriously skew your mental faculty?
You speak of "amateur", and yet show no evidence that you are anything to the contrary.
No, I'm not defending that there seems to have been little update; if I were a client, I'd be pissed about being in the dark as well - I'm simply providing a little bit of insight as to what a situation like this is like for a provider for those of you that have absolutely no clue, and seem to think it's as simple as "Press button, restore node, run to twitter". Perhaps instead of just being indignant, you could step back, be honest, and tell yourself "Wow, that's one hell of a situation they got shoved in... I honestly couldn't say if I could do it better since I've never been thrown under the bus like that. Yeah, it really sucks that there might not be backups or an ETA yet; but it is what it is, and my yelling won't magically make things go faster".
Last edited by a moderator:
jer
New Member
Offtopic: Wanted to add that a company I worked for did the same thing - moved boxes from CA in the back of a truck to their 2nd DC in a different state. There was an issue with adult related sites not having the right licenses for the new state, and the customers didn't have enough notice to correct it. An after issue that made an already hetic issue worse with the 2nd DC unable to handle the new electric load.
On Topic: I understand they're up to their elbows, that's why I haven't bugged 'em. I also don't expect updates each hour if an outage is known. I did expect my vps to be responding after a few hours of the node being up. When it wasn't, I thought I'd sign up and post to let others know their vps may not be up even if the node reports it is.
Offtopic: We were all frineds through the ranks in my shop, even had female flight commanders twice. Most of us lived off base and hung out together. Rotc guys rarely joined, but were always funny to watch.
Last edited by a moderator:
drmike
100% Tier-1 Gogent
Thanks for some insight @XFS_Duke. Didn't realize they were splurging for outside help. Uggh! That's, umm, well, bad.
The updates are mandatory for customer sake. Again, there is a reason why a 2GB VPS isn't $2-$7 a month, unless something is wrong (i.e. little to no staff behind the product)
Normal everyday people, the average customer doesn't care why something isn't working. Customers get mighty miffed when a big storm rips through a town downing utility lines. A day, maybe two and that's all the tolerance people have. Think we are at
4 days now.
To that point:
The updates are mandatory for customer sake. Again, there is a reason why a 2GB VPS isn't $2-$7 a month, unless something is wrong (i.e. little to no staff behind the product)
Normal everyday people, the average customer doesn't care why something isn't working. Customers get mighty miffed when a big storm rips through a town downing utility lines. A day, maybe two and that's all the tolerance people have. Think we are at
4 days now.
To that point:
Last edited by a moderator:
Aldryic C'boas
The Pony
Yeah, that aspect is a bit worrying. And honestly, without administrative experience themselves clients don't understand how something like that would happen.
Just to help folks understand (and this is just an example, it may not be what's happening with those nodes at all): just because an OpenVZ node is online doesn't mean the VPSes on it automatically are. Think of the VPSes like programs you have installed in Windows - if certain config files are missing, the programs may not run at all. It's very likely that they got the node online, and there are simply some VPSes who's "config files" were trashed/damaged, and need to be repaired before that VPS can function again.
I work in a similar technology service field and I completely understand how difficult it can be to recover from issues. However, in our business we notify customers as soon as we detect any customer impacting issues. The more serious the issue, the more frequent the updates (regardless of the amount of information we have uncovered). Our customers understand that there can be major situations that are started by things outside of our control but it is still no excuse for us to leave them in the dark.
Personally this just they way i feel and it doesnt go for anyone else but dam its only a VPS . Dam ppl we got kids that starve at night. People complaining about something so MINOR! You paid money i understand that but they are doing what they can. They are only HUMAN! If the VPS was hosting something so big like a business website or it was making your money. Well you should of planned some kind of redundancy! Peace Out cause its summer time summer time!
Didn't they update everyone via email and their website. What more you want them to do? Pull a bunny out of their a$$. It is just a bunch of nit picking. So lets enjoy the day and move on!
Last edited by a moderator:
The first email update I received was ~12 hours after my VPS went down. There has been 1 twitter/facebook update which was ~10 hours after and the client area notice went up around the same time. There have only been 3 additional email updates in the nearly 60 hours since the beginning. However, the entire time this has been going on their "Network Status" page reports no incidents.
drmike
100% Tier-1 Gogent
Summer... Host.
Tee hee... Summer time and a host that has been around for years that acts like a summer host
Mighty Big Fall
I remember a thread where CVPS_Chris jumped on @Prometeus and talked all sorts of smackola. That thread and some other mouthing about being the biggest VPS company in this segment to an equine host. How the mighty fall and the data shows lies.
Contingency Planning:
I wonder how other folks would recover from something like this? What steps are reputable providers taking to deal with backups on a regular basis? How could you mass migrate backups to new nodes? Think about it providers. Get your contingency plans in place and stand out from the pack.
Social Media and Keeping YOUR Customers
Love social media and all the power to quickly say something to everyone listening. Too bad folks in this situation can't find a human, a secretary or the janitor to post to such. They never seem to have issues pushing the cheap snake oil out via such though Buy Buy, buy our sh!t!
Minor Complaints or Minor Threat
@SgtZinn said this is all minor compared to real world problems and he's right.
The big butt is many customers lack backups and too many aren't tech-knowledgeable enough to setup their own VPS again from scratch. People have hours / outsourced / paid others to config and get working. Many fail to document things adequately to replicate what they had. These folks are in a world of pain right now.
jer
New Member
Yep, that's why I opened the ticket (after several hours of the box reporting up). There may be a problem and they may not be aware if the host is reporting up. I'm also not expecting them to even see the ticket for hours.
Aldryic C'boas
The Pony
You're 100% correct. If I came across as defending the lack of updates, my apologies for the miscommunication (pretty much anyone that knows Chris and I knows that either one of us defending the other would likely result in hell freezing over
). I was more giving a experienced viewpoint for the clients that don't realize what all a situation like this entails. Aye, for sure there should be more updates (I'm not one of their clients, so I have no clue about the frequency on notifications going on), but maybe now some of them at least know there's more to the situation than them simply being left in the dark for no reason. My posts were definitely more for the clients' sake than anything.
jer
New Member
I thought I'd read more from those folks in this thread. If they didn't know to have backups, they do now. If their projects were commercial, that's a hard lesson..
Exactly. Unless you notify your customers they have no idea what the situation entails and what timeline they should expect.You're 100% correct. If I came across as defending the lack of updates, my apologies for the miscommunication (pretty much anyone that knows Chris and I knows that either one of us defending the other would likely result in hell freezing over
upsetcvps
New Member
Yes, and Chris was posting here and at LET before even sending out the first e-mail. He could easily have put a notice on the client page or on twitter. And these e-mails they are providing are just too vague. I have no idea if I'll be up soon or only next month. They don't need to update me every hour, but actually give me some useful information with the updates. I'm glad you're aware of the issue, that's a good start. But when am I going to be able to use my vps again?
infinityhosting
New Member
8230
This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.
We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.
Regards
The ChicagoVPS Team
This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.
We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.
Regards
The ChicagoVPS Team
I wonder what "servers" are in this email. Do servers mean each virtual server so it is going to take hundreds of hours from now or are "servers" each node and everything should be restored in a few hours?
Last edited by a moderator:
Was wondering the same thing, but I am assuming they're talking about nodes...otherwise we could be looking at being back online sometime next month....
- Status