Skip to main content

Monitoring with Amazon CloudWatch

This article describes how to configure Halon Engage & Protect (runnning in an EC2 instance) to publish custom metrics such as queue sizes to Amazon CloudWatch.

Prerequisites​

Grant permissions to the EC2 instance​

We're going to create a new IAM role that will include just the bare-minimum amount of permissions required to publish custom metrics to CloudWatch.

note

If you already have an IAM role attached to your Halon EC2 instance you can skip this step and instead modify that one to include the necessary permissions.

Create IAM role​

Halon-Trust-Policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
aws iam create-role \
    --role-name Halon \
    --assume-role-policy-document file://Halon-Trust-Policy.json

Create policy​

HalonCloudWatchPutMetricData-Policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["cloudwatch:PutMetricData"],
      "Resource": "*"
    }
  ]
}
aws iam create-policy \
    --policy-name HalonCloudWatchPutMetricData \
    --policy-document file://HalonCloudWatchPutMetricData-Policy.json

Assign policy to IAM role​

aws iam attach-role-policy \
    --role-name Halon \
    --policy-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/HalonCloudWatchPutMetricData

Create IAM instance profile and attach IAM role to it​

aws iam create-instance-profile \
    --instance-profile-name Halon-Instance-Profile
aws iam add-role-to-instance-profile \
    --instance-profile-name Halon-Instance-Profile \
    --role-name Halon

Associate IAM instance profile with your EC2 instance​

aws ec2 associate-iam-instance-profile \
    --iam-instance-profile Name=Halon-Instance-Profile \
    --instance-id YourInstanceId \
    --region YourRegion
note

Replace YourInstanceId with the ID of your EC2 instance and YourRegion with the region the EC2 instance is located in.

Install AWS CLI on the EC2 instance​

We're going to use the aws command-line tool directly from the EC2 instance to publish our custom metrics to CloudWatch.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
note

You might need to install unzip for the above commands to work.

sudo apt-get install unzip

Install jq on the EC2 instance​

This command-line tool is needed to parse and transform the output from halonctl process-stats --json so that it can be used in combination with the aws command-line tool.

sudo apt-get install jq

Publish custom metrics to CloudWatch​

Now we're ready to start publishing our custom metrics to CloudWatch. Below are some examples which can easily be run once per minute as cronjobs directly on the EC2 instance.

note

It's recommended to only publish the custom metrics you need to keep AWS costs down.

Examples​

Active queue size​

QueueActiveSize.sh
#!/bin/bash
set -e
ProcessStats=$(halonctl process-stats --json)
Value=$(echo "$ProcessStats" | jq '.queue.queue.active.size | tonumber')
InstanceId=$(ec2metadata --instance-id)
aws cloudwatch put-metric-data \
    --metric-name QueueActiveSize \
    --namespace Halon \
    --unit Count \
    --value $Value \
    --dimensions InstanceId=$InstanceId

Defer queue size​

QueueDeferSize.sh
#!/bin/bash
set -e
ProcessStats=$(halonctl process-stats --json)
Value=$(echo "$ProcessStats" | jq '.queue.queue.defer.size | tonumber')
InstanceId=$(ec2metadata --instance-id)
aws cloudwatch put-metric-data \
    --metric-name QueueDeferSize \
    --namespace Halon \
    --unit Count \
    --value $Value \
    --dimensions InstanceId=$InstanceId

License expiration​

LicenseExpiration.sh
#!/bin/bash
set -e
ProcessStats=$(halonctl process-stats --json)
Value=$(echo "$ProcessStats" | jq '.process.license.expiration | tonumber')
InstanceId=$(ec2metadata --instance-id)
aws cloudwatch put-metric-data \
    --metric-name LicenseExpiration \
    --namespace Halon \
    --unit Seconds \
    --value $(($Value - $(date +%s))) \
    --dimensions InstanceId=$InstanceId

Create CloudWatch alarms​

Once we have our custom metrics inside CloudWatch we can start creating alarms for them. Below are some examples showing how to create various types of alarms.

Examples​

Active queue size exceeds one million​

aws cloudwatch put-metric-alarm \
    --alarm-name "QueueActiveSize" \
    --alarm-description "Active queue size exceeds one million" \
    --namespace "Halon" \
    --metric-name QueueActiveSize \
    --statistic Average \
    --period 60 \
    --evaluation-periods 1 \
    --threshold 1000000 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --dimensions Name=InstanceId,Value=YourInstanceId \
    --region YourRegion
note

Replace YourInstanceId with the ID of your EC2 instance and YourRegion with the region the EC2 instance is located in.

License expires in less than 7 days​

aws cloudwatch put-metric-alarm \
    --alarm-name "LicenseExpiration" \
    --alarm-description "License expires in less than 7 days" \
    --namespace "Halon" \
    --metric-name LicenseExpiration \
    --statistic Average \
    --period 60 \
    --evaluation-periods 1 \
    --threshold 604800 \
    --comparison-operator LessThanOrEqualToThreshold \
    --dimensions Name=InstanceId,Value=YourInstanceId \
    --region YourRegion
note

Replace YourInstanceId with the ID of your EC2 instance and YourRegion with the region the EC2 instance is located in.