Observium - Monitoring Server and Client Installation Tutorial

HalfEatenPie · Feb 9, 2015

There are a wide range of server and network monitoring software available out there. Just to name a few, you have Munin, Nagios/Icinga, Zabbix, PRTG, and of course ServerStatus by Mojeda and Mun.

All those alternatives are fantastic. I could talk about the key benefits of every single monitoring software. However, for this tutorial we'll be jumping into something more general: Observium. Observium is "an autodiscovering network monitoring platform supporting a wide range of hardware platforms and operating systems..." While Observium's main focus is network monitoring, it also includes some hardware monitoring components available making it a pretty well-rounded monitoring platform.

If you're already an Observium veteran, then fantastic! At the very bottom I'll be including some minor changes to the configs and additional modules I'm using in addition to Observium. Feel free to take a gander if you wish.

Before we start, shoutout to @mitgib for getting me started on this several years ago when I was first fiddling around with monitoring systems! You're the man!

The contents of this tutorial will be broken down into multiple posts due to certain limitations. We'll start with setting up the Observium server, setting up the Observium client, then end with minor tweaks and additional modules available. However, this tutorial will not touch upon the Unix Agent since... ehh... I think it's incredibly finicky and there's not a whole lot of documentation available. The instructions for the Observium Server is also available here on Observium's Wiki.

[SIZE= 24px]Observium Server[/SIZE]

Jumping right in, we're going to be install Observium on a Debian 7 server. This is because Observium is actually developed on Ubuntu and Debian systems. However, RHEL and CentOS instructions are available here for those of you who are interested, and for the monitoring portion we'll include instructions on how to monitor RHEL and CentOS Servers. Just note Observium doesn't provide assistance on RHEL/CentOS or any other installations that aren't Ubuntu or Debian. For the purpose of this tutorial, we're going to assume you're running as the root user (because permission and whatnot).

Begin by running:

apt-get update
While it may sound trivial, you want to download the latest package lists from the repositories. Anyways now install the packages required to run Observium:

Code:

apt-get install libapache2-mod-php5 php5-cli php5-mysql php5-gd php5-mcrypt php5-json php-pear snmp fping mysql-server mysql-client python-mysqldb rrdtool subversion whois mtr-tiny ipmitool graphviz imagemagick

Observium runs on top of Apache, MySQL, PHP, RRD, and NetSNMP (as well as Graphviz and fping). During the package installation process, you're going to receive a prompt to provide the MySQL Root password. Provide a secure password since that's pretty important and make sure you don't forget it!
Create the directory Observium is going to operate out of:

mkdir -p /opt/observium && cd /opt
For the purpose of this tutorial, we're going to be using the Community/Open Source Edition of Observium. Download and unpack it.

Code:

wget http://www.observium.org/observium-community-latest.tar.gz
 tar zxvf observium-community-latest.tar.gz

You're going to have a new folder in your /opt/ folder named observium. Change to that folder:

Code:

cd observium

Login to the MySQL Command Line by typing:

Code:

mysql -u root -p

Provide the MySQL Root Password you set earlier. From here you'll notice the mysql>. This is the MySQL shell. From here, we're going to be creating our database and assigning a new user all permissions to the new database. From the MySQL Shell, enter:

Code:

https://paste.ee/p/mr1Wy

Link (Note: Moved to Paste.ee due to IPB not accepting SQL Commands)
Now exit the MySQL Shell by typing:

exit
Now we'll find ourselves back in Bash and in /opt/observium folder. Lets copy the default configuration and edit it for our system.

Code:

cp config.php.default config.php
nano config.php

Update the config.php file with the proper MySQL database information.
Let's setup the default schema for the MySQL Database:

php includes/update/update.php
We're going to create the directory Observium will store it's logs. In addition, we'll also be creating the directory to store the RRD data files as well as modify the permissions:

Code:

mkdir logs
mkdir rrd
chown www-data:www-data rrd

Now this tutorial is assuming your server will only be running Observium for the webserver. This can be modified by using vHosts, however that's outside the scope of this tutorial. Open the default apache configuration file:

Code:

nano /etc/apache2/sites-available/default

and I'd suggest changing it to this:

Code:

<VirtualHost *:80>
       ServerAdmin webmaster@localhost
       DocumentRoot /opt/observium/html
       <Directory />
               Options FollowSymLinks
               AllowOverride None
       </Directory>
       <Directory /opt/observium/html/>
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Order allow,deny
               allow from all
       </Directory>
       ErrorLog  ${APACHE_LOG_DIR}/error.log
       LogLevel warn
       CustomLog  ${APACHE_LOG_DIR}/access.log combined
       ServerSignature On
</VirtualHost>

Note: For those of you who are using Ubuntu 14.04, use this Apache2 Config...

Spoiler

Code:

<VirtualHost *:80>
       ServerAdmin webmaster@localhost
       DocumentRoot /opt/observium/html
       <Directory />
               Options FollowSymLinks
               AllowOverride None
       </Directory>
       <Directory /opt/observium/html/>
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Require all granted
       </Directory>
       ErrorLog  ${APACHE_LOG_DIR}/error.log
       LogLevel warn
       CustomLog  ${APACHE_LOG_DIR}/access.log combined
       ServerSignature On
</VirtualHost>

With the apache2 config files edited, we're going to enable a few modules. Enable the PHP mcrypt module:

php5enmod mcrypt
Now enable the Apache module rewrite to "prettify" Observium's URLs:

Code:

a2enmod rewrite
apache2ctl restart

Now add the administrator account (level 10) to Observium:

Code:

cd /opt/observium
./adduser.php <username> <password> 10

Finally, setup the cronjob so that it discovers new hardware and polls our servers regularly by:

Code:

nano /etc/cron.d/observium

and entering this as the contents:

Code:

33  */6   * * *   root    /opt/observium/discovery.php -h all >> /dev/null 2>&1
*/5 *     * * *   root    /opt/observium/discovery.php -h new >> /dev/null 2>&1
*/5 *     * * *   root    /opt/observium/poller-wrapper.py 2 >> /dev/null 2>&1

Note: The last line of the above cronjob shows "/opt/observium/poller-wrapper.py 2". Older versions of Observium used the outdated poller.php which only created a single poller instance. This was great for initial testing or just a low number of servers, but for a large volume of servers this wasn't enough. poller-wrapper.py was then included with more recent Observium installations which created however many processes defined (in this case, 2). Change the number after poller-wrapper.py to the number of cores or instances you wish to run/use (e.g. for a VPS with four CPUs you can change the number to 4).
Great! You've installed Observium Server! Now point your browser to http://<Server IP> and be on your way!

[SIZE= 24px]Observium Client[/SIZE]

Observium mainly utilizes two types of pollers, SNMP and the Unix Agent. Only SNMP will be covered in this tutorial. The Unix Agent can/will be featured in a future post, or someone else can do it who knows.

This tutorial will help you install and configure SNMP for CentOS, RHEL, Debian, and Ubuntu servers. This tutorial will not help you configure SNMP for Windows Server or other clients, however there are resources available to help you with that.

To start, install the SNMPD package:

For CentOS and RHEL:

yum install net-snmp net-snmp-libs net-snmp-utils
For Debian and Ubuntu:

Code:

apt-get install snmpd

To make life easier, we're simply going to scrap the default SNMPd Configurations:

Code:

echo "" > /etc/snmp/snmpd.conf

Now open the blank SNMPd configuration:

Code:

nano /etc/snmp/snmpd.conf

Enter the following configurations:

Code:

rocommunity COMMUNITYNAME <OBSERVIUM SERVER IP>
    syslocation LOCATION
    syscontact  [email protected]

SNMP operates with the community strings, therefore you can change COMMUNITYNAME to something else (a single word though, no spaces or punctuations are accepted). For the purpose of this tutorial I'll be using vpsBoard. After you type in the community name enter your server IP (to prevent reflection attacks). syslocation is metadata used by Observium and other snmp services. Under LOCATION, enter the System's physical location, for the purpose of this tutorial I'll be using "Dallas, Texas, United States". syscontact is additional metadata required by SNMP. Frequently I just enter one of my own email addresses. In the spoilers is a sample configuration of snmpd.conf.

Spoiler

Code:

rocommunity vpsBoard 8.8.8.8
    syslocation Dallas, Texas, United States
    syscontact  [email protected]

With the SNMPd configurations done, we have to restart the service!

service snmpd restart
We're not out of the woods yet! Make sure you check on the Firewall to allow Incoming UDP on Port 161! Simply for tutorial's sake, here's the IPtables for it:

Code:

iptables -I INPUT -p udp --dport 161 -j ACCEPT

Congrats! You've setup SNMP properly on the client server! Time to have Observium monitor it.
Go into Observium's web interface (http://<Observium Server IP>). Login, and from the navigations go Devices -> Add Device.

Enter the information you've configured SNMP to listen for (in this case, my sample configuration):

Press "Add Device" and then wait for the next cron to run.

Congratulations! You've added a server to your Observium installation! Now wait for data collection to occur!

Here's a sample of one of my utility server (BuyVM VPS).

Spoiler

[SIZE= 24px]What Else?[/SIZE]

So that's the tutorial for the vanilla Observium installation. However, I personally recommend these minor changes to help with your use of Observium.

[SIZE= 18px]Timeout Configuration[/SIZE]

Observium was originally created for ISPs and to monitor networks, not servers. Therefore, vanilla Observium has almost no tolerance for even the smallest network blip (such as a single packet not making it to the destination). So to help with that, we're going to add a few extra lines to the config file.

Open up the configuration file:

nano /opt/observium/config.php
Add the following to the end of the configuration file:

Code:

// Timeout Config
$config['snmp']['timeout'] = 20; // timeout in seconds
$config['snmp']['retries'] = 5; // how many times to retry the query
$config['ping']['retries'] = 10; // How many times to retry ping
$config['ping']['timeout'] = 1500; // Timeout in milliseconds

The descriptions are pretty straight forward. With this configuration, Observium will now continually retry polling the server until a predetermined number of times before considering it "down". This is especially helpful if you have set Observium to email you during server downtimes (Note: You can enable this by editing the config.php file and installing sendmail or configuring smiliar mail services on the server).
[SIZE= 18px]External Application Integration - Collectd[/SIZE]

So Observium is pretty awesome that it can also integrate with External Applications such as smokeping, RANCID, syslog, etc. For this tutorial I'm simply going to address Collectd, but a full list is available here. Please note the application monitoring section (such as monitoring Apache, nginx, MySQL, etc.) of Observium requires the Unix Agent which, again, is not covered in this tutorial (but maybe in the future).

Collectd is "a daemon which collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, for example in RRD files." To be perfectly honest, it's very similar to the data collected by the SNMP poller, however Collectd comes with numerous plugins you can also monitor (and therefore monitor with Observium). Pretty awesome and keeps your life simpler.

There's two parts to Collectd that we have to consider for Observium. The server and the client. Let's begin with the Server.

[SIZE= 14px]Collectd Server[/SIZE]

For the server, install collectd:

For CentOS and RHEL:

Collectd is unavailable in RHEL and CentOS repositories, therefore you can either download the collectd RPM from collectd's website or build from the source package. Building from source or downloading the RPM and installing from collectd's website is outside the scope of this tutorial. However there are resources available online that can help you with installing collectd on CentOS and RHEL servers.

For Debian and Ubuntu:

apt-get install collectd
Once you have collectd installed, edit the collectd configuration file at /etc/collectd/collectd.conf.

Code:

nano /etc/collectd/collectd.conf

Configure that file in any way you see fit, however make sure hostname and the network plugin is loaded. Observium watches for the hostname when determining if the server has collectd enabled:

Code:

Hostname "observium.tutorial.vpsboard"

LoadPlugin network
<Plugin network>
Listen "0.0.0.0" "25826"
</Plugin>

Restart the collectd service:

Code:

service collectd restart

Now we're going to have to edit Observium's configuration file to tell it where collectd has the RRD files. First, open config.php:

Code:

nano /opt/observium/config.php

Add the following configuration argument:

Code:

$config['collectd_dir']         = "/mnt/rrdcached/db/collectd/";

That's it! The collectd tab should automatically appear for any servers that collectd is receiving the graphs for (assuming the hostnames match).
[SIZE= 14px]Collectd Client[/SIZE]

The client and the server are very similar. The only major difference is the network plugin configuration.

For CentOS and RHEL:

Collectd is unavailable in RHEL and CentOS repositories, therefore you can either download the collectd RPM from collectd's website or build from the source package. Building from source or downloading the RPM and installing from collectd's website is outside the scope of this tutorial. However there are resources available online that can help you with installing collectd on CentOS and RHEL servers.

For Debian and Ubuntu:

apt-get install collectd
Once you have collectd installed, edit the collectd configuration file at /etc/collectd/collectd.conf.

Code:

nano /etc/collectd/collectd.conf

Configure that file in any way you see fit, however make sure hostname and the network plugin is loaded. Observium watches for the hostname when determining if the server has collectd enabled:

Code:

Hostname "observium.tutorial.vpsboard"

LoadPlugin network
<Plugin network>
Listen "1.2.3.4" "25826"
</Plugin>

The IP (1.2.3.4) is the IP of the Observium Server, not the IP of the server being monitored!
Restart the collectd service:

service collectd restartThat's it! The collectd tab should automatically appear for any servers that collectd is receiving the graphs for (assuming the hostnames match).
[SIZE= 24px]Final Thoughts[/SIZE]

Hope you've enjoyed this giant crash-course on Observium! It doesn't cover everything about it but it covers majority of it. If you have any questions, comments, or concerns feel free to reply. If I don't get to them then I'm sure someone else will come along to help! If you have any awesome changes to your Observium installation feel free to let us know here!

HalfEatenPie · Feb 9, 2015

Additions to the Tutorial

This section is reserved to provide direct links to some awesome changes some other people have for Observium! Check it out!

@DomainBop posted:

Using php5-fpm
Using and Configuring nginx
Automated Observium Client Installation
Addition to SNMP Configurations
Using php5-mysqlnd instead of php5-mysql

The Link to his comment is here:

DomainBop · Feb 9, 2015

Observium

A few suggestions for much better performance and lower server loads (and an optional tip for using SSL)...

#1 use php5-fpm and apache2-mpm-event instead of libapache2-mod-php5 and apache2-mpm-prefork

#2 use nginx as a front end proxy for apache and also install memcached

a. change apache ports from 80 to 8xxx and then restart apache

b. add dotdeb repos and then apt-get install nginx memcached

nginx conf file for virtual host (name it observium and put it in sites-enabled )

server {

    listen              80 default_server;

#listen [::]:80 default_server ipv6only=on;

    server_name         my.farkingserver.su;

#   return 301 https://$host$request_uri;

location ~* \.(jpg|jpeg|gif|png|ico|css|css|bmp|js)$ {

root /opt/observium/html;

access_log off;

expires 360d;

set $memcached_key "$uri?$args";

memcached_pass 127.0.0.1:11211;

error_page 404 502 504 = @fallback;

proxy_pass http://127.0.0.1:82;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location / {

root /opt/observium/html;

index index.html index.htm index.php;

access_log off;

set $memcached_key "$uri?$args";

memcached_pass 127.0.0.1:11211;

error_page 404 502 504 = @fallback;

proxy_pass http://127.0.0.1:82;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

}

#3 optional, if you want to use SSL, create the directory /etc/nginx/ssl, then buy a SSL certificate or use openssl to create one, and use the configuration file below instead of the one in #2.

server {

listen 443 ssl default;

#listen [::]:443 ssl default ipv6only=on;

server_name my.farkingsecureserver.su;

server_name_in_redirect on;

resolver 127.0.0.1;

access_log off;

#SSL

ssl_certificate     /etc/nginx/ssl/yoursslcert.com.crt;

ssl_certificate_key /etc/nginx/ssl/yoursslcert.com.key.nopass;

ssl_session_cache shared:SSL:10m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_prefer_server_ciphers on;

ssl_ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS;

location ~* \.(jpg|jpeg|gif|png|ico|css|css|bmp|js)$ {

root /opt/observium/html;

access_log off;

expires 360d;

set $memcached_key "$uri?$args";

memcached_pass 127.0.0.1:11211;

error_page 404 502 504 = @fallback;

proxy_pass http://127.0.0.1:82;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location / {

root /opt/observium/html;

index index.html index.htm index.php;

access_log off;

set $memcached_key "$uri?$args";

memcached_pass 127.0.0.1:11211;

error_page 404 502 504 = @fallback;

proxy_pass http://127.0.0.1:82;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

}

server {

    listen              80 default_server;

#listen [::]:80 default_server ipv6only=on;

    server_name         my.farkingsecureserver.su;

    return 301 https://$host$request_uri;

}

Observium Client

LOL the lazy among us just use nunim's automated script to install the client

#1 https://www.sonicboxes.com/observium-client-install-script/#more-37

recommended but optional #2:

add the 3 lines below the dashed line to /etc/snmp/snmpd.conf and then restart snmpd

Code:

view all    included  .1                               80
-------------------------------
view all    excluded  .1.3.6.1.2.1.4.21
view all    excluded  .1.3.6.1.2.1.4.24
view system included  .iso.org.dod.internet.mgmt.mib-2.system

HalfEatenPie · Feb 9, 2015

Aaaannddd... that's beautiful! I love you!

Yeah those modifications could help out. I've never actually used nunim's automated script (nor knew it existed since I've always just configured stuff manually). That's useful!

Thanks for the additional content @DomainBop!

MannDude · Feb 9, 2015

Ddaammnn son, you hit it out of the park with this. Great write up.

DomainBop · Feb 9, 2015

php5-mysql

One more small tip

: use php5-mysqlnd instead of php5-mysql when you install Observium

The advantages of using php5-mysqlnd instead of the stock php5-mysql: (from the MySQL site):

The mysqlnd library is highly optimized for and tightly integrated into PHP. The MySQL Client Library cannot offer the same optimizations because it is a general-purpose client library.

The mysqlnd library is using PHP internal C infrastructure for seamless integration into PHP. In addition, it is using PHP memory management, PHP Streams (I/O abstraction) and PHP string handling routines. The use of PHP memory management by mysqlnd allows, for example, memory savings by using read-only variables (copy on write) and makes mysqlnd apply to PHP memory limits.

...and a chart of the difference between the two at php.net

http://php.net/manual/en/mysqlinfo.library.choosing.php

danielm · Feb 9, 2015

We've been using Observium (subscription edition) for over a year now and I think its one of the best tools we have.

One thing I liked from this guide is the Collectd portion as this is something we've never gotten around to using. Hopefully someone will post a straightforward guide to the Unix-agent because thats also a pain.

HalfEatenPie · Feb 9, 2015

DomainBop said:
One more small tip : use php5-mysqlnd instead of php5-mysql when you install Observium

The advantages of using php5-mysqlnd instead of the stock php5-mysql: (from the MySQL site):

The mysqlnd library is highly optimized for and tightly integrated into PHP. The MySQL Client Library cannot offer the same optimizations because it is a general-purpose client library.

The mysqlnd library is using PHP internal C infrastructure for seamless integration into PHP. In addition, it is using PHP memory management, PHP Streams (I/O abstraction) and PHP string handling routines. The use of PHP memory management by mysqlnd allows, for example, memory savings by using read-only variables (copy on write) and makes mysqlnd apply to PHP memory limits.

...and a chart of the difference between the two at php.net

http://php.net/manual/en/mysqlinfo.library.choosing.php

Damnn chief you're on fire!

I'll also add this to the list.

HalfEatenPie · Feb 9, 2015

danielm said:
We've been using Observium (subscription edition) for over a year now and I think its one of the best tools we have.

One thing I liked from this guide is the Collectd portion as this is something we've never gotten around to using. Hopefully someone will post a straightforward guide to the Unix-agent because thats also a pain.

Yeah the Unix Agent honestly isn't that hard to setup. The only major problem is that it's pretty finicky and a little more complicated than setting up SNMP. However, Unix Agent does allow for easier customization on the data being sent over (as well as using a totally different port).

blergh · Feb 10, 2015

From my experiences it seems that trying to use the agent with the current community edition means almost all of the actually usable plugins are "broken" or require serious php-code-ninja'ing to "work".

HalfEatenPie · Feb 10, 2015

blergh said:
From my experiences it seems that trying to use the agent with the current community edition means almost all of the actually usable plugins are "broken" or require serious php-code-ninja'ing to "work".

Hm. So I guess the Commercial/Enterprise version has a much better unix agent?

blergh · Feb 16, 2015

HalfEatenPie said:
Hm. So I guess the Commercial/Enterprise version has a much better unix agent?

I don´t think the agent itself differs, it's more about the modules/parts/base-version that doesn't offer the same features/flexibility.

nunim · Feb 18, 2015

HalfEatenPie said:
Aaaannddd... that's beautiful! I love you!

Yeah those modifications could help out. I've never actually used nunim's automated script (nor knew it existed since I've always just configured stuff manually). That's useful!

Thanks for the additional content @DomainBop!

Knowledge is power

The client setup is the annoying part so I felt it was best to automate that, although I do have a master install script that I should publish..

I think I made a VPSBoard post with the client setup script before.

HalfEatenPie · Feb 19, 2015

nunim said:
Knowledge is power The client setup is the annoying part so I felt it was best to automate that, although I do have a master install script that I should publish..

I think I made a VPSBoard post with the client setup script before.

Ahh that could be a possibility. I didn't really check. My bad! But yeah man you have some great stuff there!

stealthyhosting · Feb 24, 2015

Very good post!

If you use Observium, do consider purchasing the professional version for the newest version and bug/security updates. http://www.observium.org/subscribe/

For more information on the version differences http://www.observium.org/wiki/Edition_Split

Onra Host · Feb 25, 2015

You should also try writing a quick guide for users on how to setup SNMP on the common routers like Cisco/Jupiner

key900 · Mar 13, 2015

Great tutorial i'm running Observium as well for private use.

lolitseasy · Apr 2, 2015

Aaaah, this helps a lot. Thanks.

PacketPunks · Apr 3, 2015

Excellent share halfeatenpie! Will be Implementing this soon.

DomainBop · Apr 6, 2015

Another tip: installing rrdcached will reduce disk i/o (tradeoff is a slight increase in memory usage)

graph: rrdcached was enabled Friday and in/out was reduced by about 50%:

It's late and I'm too lazy to write a guide to installing rrdcached so use this one --> http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html

Observium - Monitoring Server and Client Installation Tutorial

The Irrational One

The Irrational One

Dormant VPSB Pathogen

The Irrational One

Just a dude

Dormant VPSB Pathogen

New Member

The Irrational One

The Irrational One

New Member

The Irrational One

New Member

VPS Junkie

The Irrational One

New Member

New Member

New Member

New Member

New Member

Dormant VPSB Pathogen