For this tutorial I will be showing how to get Zabbix to automatically create items and triggers for you... sound too good to be true? Its entirely possible, we just have to tell Zabbix how to create those triggers and for this we use Low Level Discovery (LLD) rules - not to be confused with Network Discovery for hosts.
For this tutorial I am going to guide you through the creation of a discovery rule for the automatic monitoring of any GRE/IPIP tunnels on your server. Assumptions have been made that your tunnel is a /30 where the endpoint is one IP address up from yours (e.g monitoring a gateway server of 10.0.x.1's ability to communication with 10.0.x.2).
An example of what this might look like in "ip addr" is:
[...]
32: x198: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN
link/gre 103.x.x.x peer 109.x.x.x
inet 10.0.3.21/30 scope global x198
Where in this case the endpoint of this tunnel is 10.0.0.22 .
Step 1 - Build a script that outputs the checks to create in the format described at: https://www.zabbix.com/documentation/2.0/manual/discovery/low_level_discovery
For this example a script like the following will suffice - Do not concern yourself too much with what the poorly coded awk and grep script does, this is an example for your benefit... I would not recommend using it in production (this is not the script I use any longer).
#!/bin/bash
echo "{"
echo "\"data\":["
ip addr | grep "inet 10.0." | awk '{ print $2 }' | awk -F\. '{print ($4)+($3*256)+($2*256*256)+($1*256*256*256)+1}' | awk '{ print int($1 / 16777216) "." int($1 % 16777216 / 65536) "." int($1 % 65536 / 256) "." int($1 % 256) }' | awk '{ print "{\"{#SIP}\":\""$0"\"}," }' | head -c -2
echo "]"
echo "}"
This script generates outputs data in the following format -
{
"data":[
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"}]
}
{#SIP} is the macro we are going to use in the creation of the items and triggers. We can define more than one macro as additional key/values in the JSON object representing the row of data.
Step 2 - Create the Discover item
You now need to create the item to run this script on the server. This script will be run at the defined interval, at which time it will make any changes (create or delete) items, triggers and graphs as necessary.
The configuration of the discoverer should look something like below, ensure that the script is executable by the zabbix user and that EnableRemoteCommands is set to 1. If you do not want to enable this, define a custom UserParameter it will also work.
Step 3 - Create an Item prototype
Next we need to create the prototype for the item, this is the template from which an item will be created for each row found. In this case a suitable item could be one that pings the backend to confirm that is online. As agents do not provide ping checks (this is the responsibility of the server and / or proxies) you will need to define a UserParameter / Custom check for this.
An item with a key like "system.run[sudo ping {#SIP} -c 1 -w 1 -q | grep rtt | wc -l]" should suffice for our purposes. Of course a more advanced check is recommended for use in production as UDP can and is often lost on networks (and GRE is classed the same).
Of course you will need to ensure zabbix has sudo (NOPASSWORD) for that item to work, else use fping which is more secure than granting zabbix sudo. fping can probably be found in your disro repo and is what I recommend if you plan to use this tutorial in production.
Step 4 - Create a Trigger prototype
Creating a trigger is simple, just like a normal trigger. Nothing special here, except that like with Item Prototypes you can utilize macro's from the LLD data in the trigger description etc.
---
I hope I haven't gone too fast for people. This scenario may not be too useful for everyone, but I figured it was something many of you could relate to (and it relates to my work I have been doing today).
Other examples of possible LLD rules include the discovery of containers on openvz or KVM's etc. Or even just a generic network interface discovery rule for those too lazy to maintain a macro with the interface name (such as eth0) in it.
For this tutorial I am going to guide you through the creation of a discovery rule for the automatic monitoring of any GRE/IPIP tunnels on your server. Assumptions have been made that your tunnel is a /30 where the endpoint is one IP address up from yours (e.g monitoring a gateway server of 10.0.x.1's ability to communication with 10.0.x.2).
An example of what this might look like in "ip addr" is:
[...]
32: x198: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN
link/gre 103.x.x.x peer 109.x.x.x
inet 10.0.3.21/30 scope global x198
Where in this case the endpoint of this tunnel is 10.0.0.22 .
Step 1 - Build a script that outputs the checks to create in the format described at: https://www.zabbix.com/documentation/2.0/manual/discovery/low_level_discovery
For this example a script like the following will suffice - Do not concern yourself too much with what the poorly coded awk and grep script does, this is an example for your benefit... I would not recommend using it in production (this is not the script I use any longer).
#!/bin/bash
echo "{"
echo "\"data\":["
ip addr | grep "inet 10.0." | awk '{ print $2 }' | awk -F\. '{print ($4)+($3*256)+($2*256*256)+($1*256*256*256)+1}' | awk '{ print int($1 / 16777216) "." int($1 % 16777216 / 65536) "." int($1 % 65536 / 256) "." int($1 % 256) }' | awk '{ print "{\"{#SIP}\":\""$0"\"}," }' | head -c -2
echo "]"
echo "}"
This script generates outputs data in the following format -
{
"data":[
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"},
{"{#SIP}":"10.0.x.x"}]
}
{#SIP} is the macro we are going to use in the creation of the items and triggers. We can define more than one macro as additional key/values in the JSON object representing the row of data.
Step 2 - Create the Discover item
You now need to create the item to run this script on the server. This script will be run at the defined interval, at which time it will make any changes (create or delete) items, triggers and graphs as necessary.
The configuration of the discoverer should look something like below, ensure that the script is executable by the zabbix user and that EnableRemoteCommands is set to 1. If you do not want to enable this, define a custom UserParameter it will also work.
Step 3 - Create an Item prototype
Next we need to create the prototype for the item, this is the template from which an item will be created for each row found. In this case a suitable item could be one that pings the backend to confirm that is online. As agents do not provide ping checks (this is the responsibility of the server and / or proxies) you will need to define a UserParameter / Custom check for this.
An item with a key like "system.run[sudo ping {#SIP} -c 1 -w 1 -q | grep rtt | wc -l]" should suffice for our purposes. Of course a more advanced check is recommended for use in production as UDP can and is often lost on networks (and GRE is classed the same).
Of course you will need to ensure zabbix has sudo (NOPASSWORD) for that item to work, else use fping which is more secure than granting zabbix sudo. fping can probably be found in your disro repo and is what I recommend if you plan to use this tutorial in production.
Step 4 - Create a Trigger prototype
Creating a trigger is simple, just like a normal trigger. Nothing special here, except that like with Item Prototypes you can utilize macro's from the LLD data in the trigger description etc.
---
I hope I haven't gone too fast for people. This scenario may not be too useful for everyone, but I figured it was something many of you could relate to (and it relates to my work I have been doing today).
Other examples of possible LLD rules include the discovery of containers on openvz or KVM's etc. Or even just a generic network interface discovery rule for those too lazy to maintain a macro with the interface name (such as eth0) in it.
Last edited by a moderator:
I am no AWK god as can be seen by the multiple chained awk calls full of slightly modified snippets or simple operations.