Use logwatch to keep an eye on your logfiles

wlanboy · Aug 21, 2013

It is one of the tasks I want to do but never try to finish: Check logfiles.

Look through all the logfiles to see if something happend that should not happen.

It is a job which is boring and so a single important line can splip through your scrolling.

But linux-like: There is a tool for it - called logwatch.

Installation

sudo apt-get install logwatch libdate-manip-perl (yum: perl-DateManip)
sudo mkdir /var/cache/logwatch
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

Thats all. The cache dir and the default config are not part of the installer because logwatch can be easily used by command line. You do need them only if you want to run a cronjob.

perl-DateManip is a easy option to define date ranges.
Configuration
Code:
```
sudo nano /etc/logwatch/conf/logwatch.conf
```
Things you should alter:

Output = mail
Format = html
MailTo = [email protected]
MailFrom = [email protected]
Encode = base64

#Range = yesterday
Range = between -7 days and -1 days
Detail = High
Service = All

Service can be all daemons like sshd.
Cron

00 20 * * * /usr/sbin/logwatch --mailto [email protected]

A simple call for the command line would be:

logwatch --service sshd --range today --detail 10

output is something like:

################### Logwatch 7.4.0 (03/01/11) ####################
Processing Initiated: Wed Aug 21 07:19:18 2013
Date Range Processed: today
( 2013-Aug-21 )
Period is day.
Detail Level of Output: 10
Type of Output/Format: stdout / text
Logfiles for Host: servnl
##################################################################

--------------------- SSHD Begin ------------------------

SSHD Killed: 2 Time(s)

Users logging in through sshd:
aname:
8.8.8.8 (a-host-name): 5 times
bname:
9.9.9.9 (another-host-name): 2 times
---------------------- SSHD End -------------------------

###################### Logwatch End #########################

Or postfix:

logwatch --service postfix --range today --detail 10

Code:

 --------------------- Postfix Begin ------------------------

 ****** Summary *************************************************************************************

    7.760K  Bytes accepted                               7,946
    7.760K  Bytes delivered                              7,946
 ========   ==================================================

       20   Accepted                                   100.00%
 --------   --------------------------------------------------
       20   Total                                      100.00%
 ========   ==================================================

       20   Removed from queue
       20   Delivered

 ****** Detail (1) **********************************************************************************

       20   Delivered -------------------------------------------------------------------------------
       20      wlanboy.com

 === Delivery Delays Percentiles ============================================================
                     0%       25%       50%       75%       90%       95%       98%      100%
 --------------------------------------------------------------------------------------------
 Before qmgr       0.00      0.00      4.00      9.50     10.10     11.00     11.00     11.00
 In qmgr           0.00      0.00      0.01      0.03      0.04      0.04      0.04      0.04
 Conn setup        0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
 Transmission      0.01      0.01      0.01      0.02      0.02      0.03      0.03      0.03
 Total             0.01      0.02      4.07      9.53     10.10     11.00     11.00     11.00
 ============================================================================================
 ---------------------- Postfix End -------------------------

A great tool to get weekly reports about all services which are running on a vps.

clarity · Aug 21, 2013

This:

sudo nano /etc/logwatch/logwatch.conf
Should be:

Code:

sudo nano /etc/logwatch/conf/logwatch.conf

imperio · Aug 21, 2013

Is it configurable like if specific pattern is found on the log file alert via mail ?

wlanboy · Aug 21, 2013

dclardy said:
Should be:

sudo nano /etc/logwatch/conf/logwatch.conf

Fixed it. Thank you for pointing to the missing subfolder.

imperio said:
Is it configurable like if specific pattern is found on the log file alert via mail ?

You can customize logwatch: http://www.stellarcore.net/logwatch/tabs/docs/HOWTO-Customize-LogWatch.html

E.g.:

I want to watch a new service:

So I create a new file in:

/etc/logwatch/conf/services/

Content would be like:

Title = "mynewservice"
LogFile = messages
*OnlyService = serviceD #filter everything that is not logged via this name
*RemoveHeaders # filter time stamp, hostname, ...

Line two is a head cracker...

You have to set the log file group that is used, which tells logwatch which file it should look in the /etc/logwatch/logfiles/.

Conent of the messages.conf is:

# What actual file? Defaults to LogPath if not absolute path....
LogFile = messages

# If the archives are searched, here is one or more line
# (optionally containing wildcards) that tell where they are...
#If you use a "-" in naming add that as well -mgt
Archive = messages.*
Archive = archiv/messages.*
Archive = messages-*
Archive = archiv/messages-*

# Expand the repeats (actually just removes them now)
*ExpandRepeats

# Now, lets remove the services we don't care about at all...
# Comma separated list works best -mgt
*RemoveService = talkd,telnetd,inetd,nfsd,/sbin/mingetty,netscreen,NetScreen

# Keep only the lines in the proper date range...
*ApplyStdDate

# vi: shiftwidth=3 tabstop=3 et

So the "messages" do include all "/var/log/messages.*" files.

Now we need a perl script saved in .../scripts/services/mynewservice.pl

Content would be something like: (pam for example)

#!/usr/bin/perl

##########################################################################
# $Id: pam,v 1.11 2008/03/24 23:31:26 kirk Exp $
##########################################################################

#####################################################
## Copyright (c) 2008 Kirk Bauer
## Covered under the included MIT/X-Consortium License:
## http://www.opensource.org/licenses/mit-license.php
## All modifications and contributions by other persons to
## this script are assumed to have been donated to the
## Logwatch project and thus assume the above copyright
## and licensing terms. If you want to make contributions
## under your own copyright or a different license this
## must be explicitly stated in the contribution an the
## Logwatch project reserves the right to not accept such
## contributions. If you have made significant
## contributions to this script and want to claim
## copyright please contact [email protected].
#########################################################

$Detail = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;

while (defined($ThisLine = <STDIN>)) {
if ( ( $ThisLine =~ /^pam_get_user: no username obtained$/ ) or
( $ThisLine =~ /^pam_end: NULL pam handle passed/ ) ) {
# We don't care about these
}
elsif ( $ThisLine =~ s/^FAILED LOGIN SESSION FROM ([^ ]+) FOR .*$/$1/ ) {
$FailedLogins{$ThisLine}++;
}
else {
# Report any unmatched entries...
push @OtherList,$ThisLine;
}
}

if ( (keys %FailedLogins) and ($Detail >= 10) ) {
print "\nFailed Login Sessions:\n";
foreach $ThisOne (keys %FailedLogins) {
print " " . $FailedLogins{$ThisOne} . " from " . $ThisOne;
}
}

if ($#OtherList >= 0) {
print "\n**Unmatched Entries**\n";
print @OtherList;
}

exit(0);

# vi: shiftwidth=3 tabstop=3 syntax=perl et
# Local Variables:
# mode: perl
# perl-indent-level: 3
# indent-tabs-mode: nil
# End:

There are a lot of examples in /usr/share/logwatch/scripts/services.

I like to work with if statements to search for events:

if (
( $TheLine =~ m/new connection/ )
) {
print "a new connectiont!"
}

The scripts are quite simple. You get one STDIN and you have to print to STDOUT. Just print every information you want to.

drmike · Aug 21, 2013

You are on a roll @wlanboy.

scv · Aug 21, 2013

The default logwatch on FreeBSD is great. If only more Linux distributions offered this sort of functionality out of the box.

Maximum_VPS · Aug 22, 2013

@wlanboy another excellent Guide

wlanboy · Aug 22, 2013

buffalooed said:
You are on a roll @wlanboy.

Just write about things that easy my work.

Maximum_VPS said:
@wlanboy another excellent Guide

Thank you.

I start this series of short guides to draw our attention to this cool little helpers.

johng · Aug 22, 2013

scv said:
The default logwatch on FreeBSD is great. If only more Linux distributions offered this sort of functionality out of the box.

**Noob alert**

What is the default logwatch on FreeBSD? Is it logwatch or a BSD equivalent?

Thanks.

**Noob alert over. You may now return to your regularly scheduled programming.**

scv · Aug 22, 2013

johng said:
**Noob alert**

What is the default logwatch on FreeBSD? Is it logwatch or a BSD equivalent?

Thanks.

**Noob alert over. You may now return to your regularly scheduled programming.**

They aren't the same service but they offer similar functionality. By default it gives you a security log and a general log. Here's an example from a desktop machine I use at my day job:

Daily run

Removing stale files from /var/preserve:

Cleaning out old system announcements:

Removing stale files from /var/rwho:

Backup passwd and group files:

Verifying group file syntax:

/etc/group is fine

Backing up mail aliases:

Backing up package db directory:

Disk status:

Filesystem     Size    Used   Avail Capacity Mounted on

/dev/ada0p2    9.7G    2.1G    6.8G    24%    /

devfs          1.0k    1.0k      0B   100%    /dev

/dev/ada0p3    125G    2.8G    112G     2%    /usr

Network interface status:

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs Coll Drop

em0    1500 <Link#1>      00:21:9b:45:e3:85    84272     0     0     5594     0     0    0

em0    1500 fe80::221:9bf fe80::221:9bff:fe        0     -     -        6     -     -    -

em0    1500 2001:470:1f11 2001:470:1f11:cb0        0     -     -        0     -     -    -

em0    1500 10.1.10.0     10.1.10.80           12613     -     -     5578     -     -    -

usbus     0 <Link#2>                               0     0     0        0     0     0    0

usbus     0 <Link#3>                               0     0     0        0     0     0    0

usbus     0 <Link#4>                               0     0     0        0     0     0    0

usbus     0 <Link#5>                               0     0     0        0     0     0    0

usbus     0 <Link#6>                               0     0     0        0     0     0    0

usbus     0 <Link#7>                               0     0     0        0     0     0    0

usbus     0 <Link#8>                               0     0     0        0     0     0    0

xl0*   1500 <Link#9>      00:0a:5e:77:13:80        0     0     0        0     0     0    0

plip0 1500 <Link#10>                              0     0     0        0     0     0    0

lo0   16384 <Link#11>                              6     0     0        6     0     0    0

lo0   16384 localhost     ::1                      0     -     -        0     -     -    -

lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    -

lo0   16384 your-net      localhost                6     -     -        6     -     -    -

Local system status:

3:01AM up 23:16, 2 users, load averages: 0.00, 0.00, 0.00

Mail in local queue:

/var/spool/mqueue is empty

                Total requests: 0

Mail in submit queue:

/var/spool/clientmqueue is empty

                Total requests: 0

Security check:

    (output mailed separately)

Checking for rejected mail hosts:

Checking for denied zone transfers (AXFR and IXFR):

-- End of daily output --

Security check:

Checking setuid files and devices:

Checking negative group permissions:

Checking for uids of 0:

root 0

Checking for passwordless accounts:

Checking login.conf permissions:

Checking for ports with mismatched checksums:

fgsfds kernel log messages:

+++ /tmp/security.mNDCbj0h 2012-09-06 03:01:08.000000000 -0400

fgsfds login failures:

fgsfds refused connections:

Since the machine's mostly idle there isn't anything very exciting or relevant in there but I am sure you can get the general gist of its purpose.

CraigA · Aug 22, 2013

Thanks for this, I'm going to check it out.

VPSCorey · Aug 24, 2013

I'll evangelize Splunk.com for playing with logs as well. There's a free edition for 500MB a day of logs.

trexos · Sep 3, 2013

*noob alert*

I get this error:

Can not open HTML Header at /usr/share/logwatch/default.conf/html/header.html: No such file or directory

I read that this is a debian error. How can I fix it?

Edit #1: I fixed it with downloading and copying the html files from the newest release.

But know I have this error:

root@s1:~# logwatch --service sshd --range today --detail 10
root@s1:~# logwatch --service sshd --range today --detail 10

I don't get a result.

wlanboy · Sep 3, 2013

trexos said:
But know I have this error:

root@s1:~# logwatch --service sshd --range today --detail 10
root@s1:~# logwatch --service sshd --range today --detail 10

I don't get a result.

If you did all the steps the default output of logwatch is "send via email".

Do you have a running smtp server?

Run following command to see if mail delivery is working:

sudo less /var/log/mail.log

Don't want to install a whole mailserver?

Install nullmailer and use an existing mail account for that. Look at this tutorial.

If you want to see the output (for testeing add following parameter --output):

logwatch --service sshd --range today --detail 10 --output stdout

It will forward the output to the console.

tonyg · Sep 9, 2013

Any way to correct the misspelling on the title?

logfiles not logiles.

wlanboy · Sep 9, 2013

tonyg said:
Any way to correct the misspelling on the title?

I asked the admin ... cannot edit anything.

peterw · Nov 5, 2013

Missed this great tutorial. Thank you.

wlanboy · Jan 12, 2014

FRCorey said:
I'll evangelize Splunk.com for playing with logs as well. There's a free edition for 500MB a day of logs.

And there is an open source one doing the same: Fluentd.

Use logwatch to keep an eye on your logfiles

wlanboy

Content Contributer

clarity

Active Member

imperio

New Member

wlanboy

Content Contributer

drmike

100% Tier-1 Gogent

scv

Massive Nerd

Maximum_VPS

New Member

wlanboy

Content Contributer

johng

New Member

scv

Massive Nerd

CraigA

New Member

VPSCorey

New Member

trexos

New Member

wlanboy

Content Contributer

tonyg

New Member

wlanboy

Content Contributer

peterw

New Member

wlanboy

Content Contributer