Website Monitoring

August 6, 2021 - 5 mins read

Prologue:

Monitoring a website for any downtime has always been crucial and it’s even more important to get a notification of such occurences for prompt remediation actions and log such issues to analyze and avoid them in future.

This post discusses about actively monitoring a website availability by sending requests to the website URL. We will be using urllib.request module to make the request in a Python script and the script will be invoked at regular intervals via cron job.

The script will be proactively checking the website availability instead of actually capturing users interactions. The proactive monitoring of the website is called synthtic monitoring. A frequencey of 10 minutes would be good enough for a site with moderate traffic. However, the frequency can be easily changed using the cron expression that we will be discussing in the upcoming sections.

Checking the Website

Following is a basic code snippet which uses request.urlopen method to check the website:

import urllib.request as request

url = 'http://example.com'

try:
    responseCode = request.urlopen(url).getcode()
except Exception as err:
    logging.error('Could not check the website due to error: %s', err)

Notifying website admin of downtime

After checking the website, we can extract the response code followed by sending a notification if the response code is 5xx - showing website server has issues. There can be numerous ways to notify the admin; for example you can leverage AWS services like AWS SES or AWS SNS or other providers like One Signal and Pushnami. However, in this post, we are going to use a very basic smtplib module in this post.

The following snippet gives an idea of sending mail using smtplib with SMTP_SSL class.

import smtplib

def sendMail():
    emailId = 'sender-mail-id'
    emailPassword = 'LoginPassword'

    sentFrom = "Web Monitor"
    to = ['reciever1-emailId', 'reciever2-emailId'] 
    subject = 'A test mail'
    body = 'Website Monitoring demo'

    emailText = (
        "From: %s\r\n"
        "To: %s\r\n"
        "Subject: %s\n\n"
        "%s"
        % (sentFrom, ", ".join(to), subject, body))

    try: 
        #Create your SMTP session
        smtp = smtplib.SMTP_SSL('smtp.gmail.com', 465) 

        #User Authentication
        smtp.login(emailId,emailPassword)
        
        #Sending the Email
        smtp.sendmail(sentFrom, to, emailText) 

        #Terminating the session 
        smtp.quit() 
        print('Email sent successfully!')
    except Exception as err:
        print('Failed to send Email due to %s', err)

Extra Shots on Secured SMTP Connections
In the above snippet, the SMTP connection is created with SSL, so it’s encrypted from the begining. The smtplib module provides another method SMTP.starttls, but it first creates the conncetion using smtplib.SMTP and uses SMTP.starttls to apply SSL on the connection. So, SMTP_SSL class is preferred for secured SMTP connections.

Logging for future Root Cause Analysis

Notifying the website admins will help in getting back the site available faster, but it’s important that we log such incidents so that a thorough investigation can be done later to find the root cause and avoid such issues in future.

We will be using logging module which can be used to log the 5XX responses from the website. By default, the logging module prints the logging content on the console. However, we can specify a file using basicConfig module which will be used for logging.

Following snippet shows how to set the logging file. Here, the path of the logging will be relative to the script path.

import logging 

# Setting logging configuration
logging.basicConfig(level=logging.DEBUG, filename='pathToLogFile.log', \
    filemode='a', format='%(asctime)s %(levelname)s %(message)s')

logging.info('Logging at Information level.')

# Lodding in the logfile: 2021-07-20 13:00:31,354 INFO Logging at Information level.

Logging tutorials can be found at basic and advanced tutorials.

Tying all together

Using all the above discussed sections, we can synthtic monitoring of a website, notify website admin and log for issues' future analysis and I had tested it using following script.

import logging 
import urllib.request as request
import smtplib

# Setting logging configuration
logging.basicConfig(level=logging.DEBUG, filename='pathToLogFile.log', \
    filemode='a', format='%(asctime)s %(levelname)s %(message)s')

def sendMail(responseCode):
    emailId = 'sender emailId'
    emailPassword = 'sender email login password'

    sentFrom = "Web Monitor"
    to = ['web-admin-email-id-group', 'operations-team-email-id-group'] 
    subject = 'WARNING! Website down!'
    body = 'Website is down!! Response Code: ', responseCode

    emailText = (
        "From: %s\r\n"
        "To: %s\r\n"
        "Subject: %s\n\n"
        "%s"
        % (sentFrom, ", ".join(to), subject, body))

    logging.info('Sending mail to the website admin and operations team.')
    try: 
        #Create your SMTP session
        smtp = smtplib.SMTP_SSL('smtp.gmail.com', 465) 

        #User Authentication
        smtp.login(emailId,emailPassword)
        
        #Sending the Email
        smtp.sendmail(sentFrom, to, emailText) 

        #Terminating the session 
        smtp.quit() 
        logging.info('Email sent successfully!') 

    except Exception as err:
        logging.info('Failed to send Email due to %s', err)


# Check if the website is available
url = 'https://techiehustle.com'

try:
    logging.info('Reaching the website %s', url)
    responseCode = request.urlopen(url).getcode()
    if responseCode in [500, 501, 502, 503, 504]:
        logging.warning('Website DOWN!!\nResponse Code received = %s', responseCode)
        logging.info('Notifying website admin and Operations team.')
        sendMail(responseCode)
except Exception as err:
    logging.error('Could not check the website due to error: %s', err)

Note: The above script logs and notifies only for the common server-side 5XX responses.

Cron job to invoke the script

We have reached at the last step of creating cron job which will be triggered every 10 minutes. On Linux machines, make sure that the crond service is running using command $ service crond status and command $ service crond start can be used to run crond. If cron is not installed, it can be installed using:

CentOS/RHEL: $ sudo yum install cronie
Debian/Ubuntu: $ sudo apt-get install cron

Then a cron job can be created using crontab -e command and following in the file.

*/10 * * * * python3 /relativePathOfPythonScript/webMonitor.py

The frequency of the cron job can be set by changing the cron expression in the above command. You can find cron expressions with the help of this site.

In case of Windows machine, we can use Register-ScheduledJob with RunEvery flag to specify the frequency.