How often do you update your servers?

[Wambo]

Just wondering how often you update your servers and if you should really be doing it very frequently or once every couple months is okay or not.

 

[tonyg]

Setup a cron job to run every day and update as necessary.

If not confortable with auto updates, then at least set up a script to query for udpates daily and update as needed manually.

 

[Munzy]

Unattended upgrades is amazing

 

[telephone]

Setup a cron job to run every day and update as necessary.

If not confortable with auto updates, then at least set up a script to query for udpates daily and update as needed manually.

On your first point, unattended updates should never be done in a production environment. Even on a non-production server, you risk breaking changes or an unoptimized config if a major version is released (not all packages try for non-BC).

What I tend to do, is tune my RSS reader for major packages I use within my stack. If an update is out I'll know about it rather quickly  .

Also, I run a local VM for development/testing. All my servers use the same OS, so if I see there's updates locally, I'll issue the command to all my servers.

Lastly, you could just do what Ubuntu does with "update-motd" and "update-notifier-common", where they'll update your MOTD to include the number of packages.

 

[drmike]

I update regularly, as in daily... However I am brutally slapped by breakage all the time.  This applies to Debian as much as it does to apps for Android.

QA testing has gone down the sewer in recent years where people are all about rapid flushing the toilet with code.

 

[telephone]

Another good suggestion for your MOTD:
 
Pair "cowsay" with istheinternetonfire.


Edit your ~/.bashrc or ~/.zshrc. Add the following:

# Location may vary. On Arch, cowsay is located at "/usr/bin/cowsay"
# and cows are located at "/usr/share/cows"
if [ -x /usr/games/cowsay ]; then
host -t txt istheinternetonfire.com | cut -f 2 -d '"' | cowsay -f $(ls /usr/share/cowsay/cows/ | shuf -n1)
# alternative: dig +short txt istheinternetonfire.com | cowsay -f $(ls /usr/share/cowsay/cows/ | shuf -n1)
fi
 
Extra credit for @Francisco and @Aldryic C'boas:
 
Ponysay! 
 

 

[splitice]

Servers running customer purchased services? 3 monthly - with updates of specificly vulnerably services / libraries in the interim. Feature updates as needed (with deployment plans).

Servers with IP based ACLs doing backend processing work (i.e log processing or database serving)? 9 - 9 monthly at most. Again with specific updates for OpenSSL and any other vulnerabilities

I would like to make it more frequent but trusting unattended updates? Never.

Time? Limited. Interruption or unexpected behaviour? Likely. QA? A must.

 

[mitgib]

I run ksplice/kernelkare patched kernels on all production servers, so reboots are a rarity, then update packages once a month unless I read about a critical update, or there is some benefit added by patching earlier. I am the master of set it and forget it

 

[danielm]

+1 for KernelCare. We run it on all of our VPS nodes and cPanel servers. Like others, we update everything a few times a month or when we are notified of a vulnerability. Recommend the Rack911 HostingSecList for keeping in the loop on vulnerabilities.

 

[Wambo]

Thanks. I do not know if I should run automated updates as sometimes updates break stuff or will stop a service until it has user input so I do not want to run automated on a cron job but will check it more often.

 

[tonyg]

Just to add to the unattended update discussion.

I run unattended updates on all my servers, but unattended does not mean not monitoring the update.

All my unattended updates are run at a time when I know I will be present to intervene if needed.

I use screen for all updates so if the udpate requires manual intervention, I can log in and intervene.

Thus, I run a daily script that monitors and sends me emails to update each of the points below:

1- If there are udates available

2- If there are updates, the names of the software packages to be updated

3- The success or failure of an update.

4- If the update requires manual intervention, the screen command to use to login in and intervene.

 

[Munzy]

Unattended upgrades are becoming more and more a must due to the high volume of security updates.

 

[MannDude]

Now I feel like a slacker since I usually update things once a week unless I am notified of an urgent need to to push out an update immediately.

What do you guys recommend for automating the updates on a VPS?

I don't like the idea of just running: "apt-get update && apt-get upgrade -y" on a cron job once a day in a production environment but am curious now what other options exist.

 

[DomainBop]

What do you guys recommend for automating the updates on a VPS?

No need for that.  It's the provider's job to make sure his damn nodes are secure. Here's my low end upgrade strategy:

1. only upgrade when a major version comes out: WordPress 2, WordPress 3, WordPress 4, etc.

2. if I get hacked, I file a chargeback and start a thread on WHT saying it wasn't my fault because I have an upgrade policy in place and the blame for the hack lies fully with my provider's poor security which is why I'm lawyering up to sue him for the tens of thousands in lost income I suffered when he suspended my $10 annual VPS after it was hacked.

If you only have a couple of VPS's my suggestion would be to skip automating updates and instead put a little recurring reminder on your calendar to update on days x, y, z..and also subscribe to your distros and scripts (IPBoard, etc) security lists so you'll know when there's a critical update.

That said, I use Puppet for automation.

unattended updates should never be done in a production environment. Even on a non-production server, you risk breaking changes or an unoptimized config if a major version is released (not all packages try for non-BC).

GitHub suffering a major DDoS in the middle of an unattended upgrade and returning 403 errors is another thing that can go wrong (see http://news.softpedia.com/news/GitHub-Has-Been-Under-a-Continuous-DDoS-Attack-in-the-Last-24-Hours-476902.shtml).  Of course, I just had that problem while doing a manual upgrade (of OpenProject) so manual upgrades are no guarantee that murphy won't get you either.

 

[drmike]

I see folks recommending KernelCare, haven't seen directly (am not a provider) but witnessed admins cussing about KernelCare breaking.  

Ksplice, was also recommended, I run the free version on my VPS instances.  That does work good and haven't been slapped that I am aware of by that.

As-is updates are a bit of a mess and need rethought.   There are major releases which should be optional and slow to adopt and there are the brutal security fixes that should be prioritized updates.  The security ones should be rapid to deploy.  In userland, say with apt-get default, everything is just an update on equal footing.  Unsure if their is a bastardized method of skimming just security updates, but that conceptually would be superior and something to actively pull, be aware of and ideally deploy.  Mailing lists with such are nice, but can eat up life with the 2000 other things an admin has to do.

 

[Francisco]

Extra credit for @Francisco and @Aldryic C'boas:

Ponysay! 

Poooonnyyyyy

There may have been a mishap in San Jose during our LV migration where an unknown staffer tried to pay a cab driver to run over a long group of furries in the area for a convention (think of the Elvis impersonators in the early GTA games >_>)

Francisco

 

[Aldryic C'boas]

I tried to give the cabbie like 300$ to swerve and take out a few fursuits, but he wouldn't go for it

 

[Munzy]

I tried to give the cabbie like 300$ to swerve and take out a few fursuits, but he wouldn't go for it

WHYYYYYY?

 

[RLT]

Why not? Personally I think the offer was waaaayyyy too cheap.

 

[k0nsl]

I try to keep my servers updated as frequent as I can, which is to say at least check for updates every week — for the less important servers, and decide what to update, or if I should hold off (in case of breaking stuff). So unless it is a security related issue, I try to keep it cool and assess the situation before actually upgrading. As for the servers I care for, I handle them very gently. As you would with a child!  :lol:

I've got KernelCare running on some of the major ones which has been working fine for me, no issues, but it worked fine before KC too. So I'm not sure if I'm wasting my shekels or not, oh well: it's just $4, or about that much, I forget. In the end, it makes me feel a little more at ease.

Thanks for the Ponysay tip, @telephone. I've employed that on my personal VPS that I use for all sorts of tasks. Funny  :wub: