The task of monitoring all vhosts on a web-server in addition to the host and service statuses is necessary because vhosts can fail at the application level where the web-server is operational. In a web development shop like Radical Designs vhosts come up and are taken down all of the time, SLAs mandate that operations staff need to make sure certain HTTP response codes are returned and pages appropriately redirected when new redirects are added. Operations staff need to make sure specific vhosts and urls on those vhosts return specific content and assure all of these do not overwhelm the target web servers or Nagios. At Radical Designs we wanted the update status of web applications associated with vhosts integrated into nagios.

nagios_vhost.pl is capable of polling web servers for their current list of vhosts, vhost alias, comments that act as monitoring directives, and web application type (WordPress and Drupal) and update status. Each host that is configured to have its vhosts monitored will have its own process associated with the checks to help control the rate of requests per server. Its written in Perl. With this in place the following reports across all servers in the farm are always available via the Nagios interface: Vhost status report, and available web application updates (further broken down in web application type).

The main loop looks like this:

$SIG{USR1} = sub { print_statistics(); };

Proc::Daemon::Init;
initDB();

# Loop across all of the vhosts and alias' in the database and submit passive checks for 
my $sth = $DBH->prepare("SELECT host_id,name from host") || die "$DBI::errstr";
$sth->execute();
while (my $host = $sth->fetchrow_hashref()) {
            my $pid = fork_server($host->{host_id}, $host->{name});
 }

Start monitoring vhosts, alias, and web application update status for a new serer

# Update the config for the host in nagios_vhost
puppet:~/# /usr/bin/perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl --verbose --update-web-servers huang.radicaldesigns.org

# restart nagios
puppet:~/# /etc/init.d/nagios3 restart

# restart nagios_vhost
puppet:~/# ps -ef|grep nagios_vhost|grep daemon
root     20153     1  0 11:51 ?        00:00:00 perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl --daemon
puppet:~/# kill -s TERM 20153
puppet:~/# perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl --daemon

Poll a host for its current vhosts, aliases, and web applications

puppet:/etc/nagios3/conf.d# /usr/bin/perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl --verbose --update-web-servers codepink.radicaldesigns.org
Initializing application database (/home/radicaldesigns/src/nagios_vhost/.nagios_vhost.pl.db)
Getting vhost information from codepink.radicaldesigns.org...
Creating vhost config file for codepink.radicaldesigns.org(/etc/nagios3/conf.d/codepink.radicaldesigns.org_vhosts.cfg...

# If you do not see the "Things look ok" message below something went wrong
puppet:/etc/nagios3/conf.d# nagios3 -v /etc/nagios3/nagios.cfg
...
Things look okay - No serious problems were detected during the pre-flight check

# Reload nagios
puppet:/etc/nagios3/conf.d# /etc/init.d/nagios3 reload
Reloading nagios3 monitoring daemon configuration files: nagios3.

Configure a check for a vhost or url to verify it is returning a specific string

puppet:/etc/nagios3/conf.d# /usr/bin/perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl --add-vhost-query-string codepink.org --query-string "CODEPINK : Index"

Configure a check for a vhost to make sure it is password protected

select * from vhost where name like '%human%';

17|2133|2016-01-21 22:50:37|humanfuture.huang.radicaldesigns.org|80||104.200.21.192|301
17|2154|2016-06-21 18:22:51|ahumanfuture.org|80||104.200.21.192|301

update vhost set response='401' where name like '%human%';

Configure a check for a vhost or url to verify it is redirecting to the expected url

# Change the expected response code and expected redirection URL after getting the vhost_id

mysql> select * from vhost where name="codepink.rdsecure.org";
6|1904|2014-08-28 16:48:33|codepink.rdsecure.org|443|https://codepink.myshopify.com/|173.255.251.234|302

mysql> update vhost set response=302, query_string='https://codepink.myshopify.com/' where vhost_id=1904;

Get status on the current functionality of the nagios_vhost.pl daemon

# get the process id as above and send the USR1 signal to it
puppet:/etc/nagios3/conf.d# kill -s USR1 9139

puppet:/etc/nagios3/conf.d# grep "nagios_vhost.pl: INFO" /var/log/syslog
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - huang.radicaldesigns.org 133 checks (45 vhosts, 88 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - corpethics.radicaldesigns.org 11 checks (2 vhosts, 9 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - blog-psaonline.radicaldesigns.org 1 checks (1 vhosts, 0 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - codepink.radicaldesigns.org 25 checks (5 vhosts, 20 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - rockwood.radicaldesigns.org 7 checks (5 vhosts, 2 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - app-m-smartvoter.radicaldesigns.org 7 checks (2 vhosts, 5 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - grace.radicaldesigns.org 393 checks (121 vhosts, 272 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - jacqui.radicaldesigns.org 730 checks (220 vhosts, 510 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - fca.radicaldesigns.org 5 checks (1 vhosts, 4 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - charlotte.radicaldesigns.org 2 checks (1 vhosts, 1 aliases)
Sep  3 15:20:46 puppet nagios_vhost.pl: INFO - revent-app.radicaldesigns.org 10 checks (1 vhosts, 9 aliases)

Add a new server to have its vhosts and urls monitored

radicaldesigns@puppet:~/$ /usr/bin/perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl –verbose –add-web-servers newwebserver.radicaldesigns.org

# Get the ssh key for the puppet host on newwebserver.radicaldesigns.org and configure ssh key authentication for the root user

# then load the vhosts from the new webserver as above

Why are there vhosts getting checked with "meow" as the subdomain?

Often a vhost config in apache will look something like this:

ServerName radicaldesigns.org
ServerAlias *.radicaldesigns.org

This is so that the stanza that configures radicaldesigns.org will handle all subdomains for radicaldesigns.org. In this instance we check to make sure that the subdomain ''meow.radicaldesigns.org'' is handled because it is a highly unlikely subdomain to actually exist, and this is what we have decided to check for whenever we come across a wild card in the ServerAlias directive.

Update nagios with the web application update status across our farm

The web application status is scheduled to be updated every day from cron on puppet at midnight. You can manually run this process with:

radicaldesigns@puppet:~/$ /usr/bin/perl /home/radicaldesigns/src/nagios_vhost/nagios_vhost.pl –verbose --web-application-status-to-nagios

nagios_vhost.pl is managed in version control here.

Employer: 
Skills: 
Project: 
Start: 
2014