AWS CloudWatch

Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. With Librato’s CloudWatch integration, users can get a lot more out of the data CloudWatch is collecting for them.

Creating read-only AWS credentials

Before you start using the integration, we recommend setting up a new read-only AWS user.

Step 1. Login to AWS IAM Management Console

Step 2. Create a New User

  1. Select ‘Generate Access Token for each user’
  2. Download CSV of credentials

Step 3. View details of New User created.

  1. Under Permission select the Inline Policy dropdown creation link
  2. Select Custom Policy
  3. Paste the below custom policy

(If you’re not using a service, feel free to omit it):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "dynamodb:ListTables",
        "ec2:DescribeInstances",
        "ec2:DescribeVolumes",
        "ecs:ListClusters",
        "ecs:DescribeClusters",
        "elasticache:describeCacheClusters",
        "elasticbeanstalk:DescribeEnvironments",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTags",
        "elasticmapreduce:ListClusters",
        "rds:DescribeDBInstances",
        "ses:GetSendQuota",
        "ses:GetSendStatistics",
        "sqs:ListQueues",
        "lambda:ListFunctions"
     ],
      "Resource": "*"
    }
  ]
}

You can then add your User to a Group and generate an Access Key for the user (or use the Access Key generated when you created the User) to use in the Librato AWS integration.

Why do we need so many permissions?

We obviously need the “cloudwatch” permissions to import the actual data, but what about all the others? The Cloudwatch list-metrics API call includes all metrics that have had any data reported for them in the last two weeks, even if the instance, database or other resource has been terminated or deleted. This can result in our integration continuing to make 100s of requests for things that haven’t had any new data in several days. To save on costs, both to our integration importer having to do extra work, and also for you having to pay for all those additional AWS API calls, we use the relevant “DescribeX” API call to determine which resources are actively running so we can limit our importing to only those. Additionally, for a few like EC2 and ELB, Cloudwatch does not provide us with much metadata such as EC2 Tags or Availability Zone. In those cases, we also make use of the data available to creating a more human-friendly source name when we import it.

Importing your CloudWatch metrics to Librato.

Prerequisite to start using this integration is that you have a Librato account. If you do not already have one, sign up for a free trial - it only requires an email address.

After you sign in to your Librato account, you will see a welcome screen that will guide you through the process of setting up your CloudWatch integration.

librato_onboarding_aws

Select the “Amazon Web Services” option and proceed to the next step. After providing a title for your new integration you will be prompted to enter the read-only AWS Access Key ID and Secret Access Key you created earlier. Select an AWS Region and check the AWS Services you’re interested in tracking.

You can import metrics from as many instances as you want but as a precautionary measure we have set a limit to 250 instances. If you want that limit removed just send us an email to support@librato.com.

NOTE: Once the integration is saved, it can take 10-15 minutes to start seeing metrics in our system. Then we will query back the previous 1 hour of data and all subsequent data.

integrations_aws_cloudwatch

Name and Source Prefix

There are two optional fields when configuring the integration. The first, “Name”, is a free text field you may use to help you find a particular integration in the list, but has no bearing the the operation of the import. The second, “Source Prefix” will be prepended to all source names created by the import which is particularly useful to disambiguate multiple AWS accounts.

For example, if you have enabled the “AWS/EC2” service and have an instance i-abcdef0 in the us-east-1a region, it will normally be imported with a source name of us-east-1a.i-abcdef0. However, since AWS makes no guarantees about the uniqueness of InstanceId’s across accounts, you will want to prefix each of your AWS accounts with a source prefix, e.g. “staging” or “production” or “web”. In that case, the integration will import source names like production.us-east-1a.i-abcdef0.

Regions and Services

Use the checkboxes for the regions and services to determine which metrics you import.

aws-integration-regions

aws-integration-services

Adding multiple AWS accounts

To add multiple AWS account just click on the “Add a new AWS CloudWatch Integration”. An empty form will appear. Save the form when you done and you will see a new integration show up in the list.

AWS Spaces and Charts

Once your credentials are set up and saved you will immediately see Spaces and charts we’ve automatically curated for each service. They are created as templates using the dynamic source feature. Simply enter a source in the dynamic source field, such as the server id or a wildcard ( * for all), and the graphs are created.

image3

Filtering Metrics

Librato supports the ability to filter by tag for EBS and EC2 resources. In the Cloudwatch integration, the EBS and EC2 service types have an additional input option next to each service type (when checked). This input is a toggle for two states: all or filtered.

integrations_aws_cloudwatch_filter

The default state is all, which is to collect all resource metrics from CloudWatch for the respective service type. If you click on the link it will toggle to filtered; in this state we whitelist metrics coming from resources with the following tag applied:

librato:import=true

... where “librato:import” is the key and “true” is the value. For example, if you are using the EC2 command line tool you would apply the tag to an instance like:

ec2tag <instance id> -t librato:import=true

Or in the AWS Console:

image5

Stop Importing CloudWatch Metrics

To stop importing CloudWatch metrics, either hit the delete button to remove your IAM credentials or un-check the AWS Regions and AWS Services boxes and save.

After Setup Is Complete

After you have completed your setup, you should start seeing metrics flow into your account within 10-15 minutes. If that is not the case, check the Integrations page for any errors that may be present. The error message will give you a hint on what needs to be corrected. If you need help please send us an email.

image10

Metric Namespaces

Depending on the CloudWatch services you have selected, Librato will create metrics in the namespaces following the pattern:

AWS.<service-name>

Predefined Spaces

Dynamic Source

When you first open a Space you will see the message: “You must specify a dynamic source”. All Spaces use the dynamic source feature. To see graphs enter a source pattern into the Dynamic Source Search box or just use the * wildcard for all sources.

image11

Service Specific Spaces

You will notice that two predefined Spaces are created per AWS service:

AWS <service name>
AWS <service name> Compact

The main Spaces are more detailed in that the charts show one line for each source (in other words the “breakout” of the sources). Note that if you only have one source, you’ll just see one line per metric.

The Compact Spaces are aggregated to the metric level, showing one line for a metric aggregated across all sources.

In both cases, the aggregated values displayed in Librato will vary depending on the use case, but in general will be one of min, max, average, or sum. For example, “breakout of the sum” means we’ll display the sum of measurements across the time window, broken out by source. Similarly, “average of sum” means we will display the sum of measurements across the time window, averaged across the sources.

You can easily recreate the graphs in a new space and define the aggregation methodology that best suits your needs.

Changing Predefined Spaces

The predefined Spaces we curate are immutable. In order create editable versions, you will have to use our Copy Spaces feature to clone them. Clones are customizable, so now you can create spaces with CloudWatch metrics from several services mixed with custom metrics so that all critical metrics are in one place.

EC2 Instance Tags

Librato’s CloudWatch integration provides support for EC2 instance tags. If you set either librato:displayName or Name on an instance, we’ll automatically use that value as a source display name. In the case both tags are set, librato:displayName takes precedence. Any changes to the tag’s value will be detected and applied automatically.

You can search or wildcard match on either the original instance ID or the human-readable name specified in one of these tags. Whichever value matches your search will be used in the display.

How this works: The CloudWatch integration uses source names of the format zone.instance-id for EC2 metrics as the actual source and then, as a separate step, creates a mapping of that zone.instance-id => name, where name is the value of the tag “Name” on the instance itself. We then use our sources/:name API to set the source name.

That then becomes what you see in the UI.

Elastic Beanstalk

To learn how to enable the Elastic Beanstalk service in your Librato CloudWatch Integration so that you can import all of your enhanced health metrics into your Librato account and preconfigured AWS Elastic Beanstalk space read: AWS CloudWatch: Elastic Beanstalk enhanced health metrics

EBS Volume Tags

Librato will retrieve the “Name” tag from an EBS volume (via the ec2-describe-volumes API) and use that as the source display name. For example, a VolumeId of vol-012e34e5 in us-east-1 with a Name tag of My Files will result in a Librato source name of us-east-1.vol-012e34e5 and a source display name of My Files. You can then filter volumes either by VolumeId or by Name tag.

Custom Metrics

Librato is a platform designed for custom metrics. You can use any of the pre-integrated collection agents or language bindings to add metrics to Librato and then add them to your AWS service graphs or Spaces. For example you can run Librato Agent on your AWS instances to pull in memory and other metrics that CloudWatch doesn’t provide.

You can also pull in CloudWatch custom metrics by checking the box under AWS Service in your AWS Configuration. To learn more about CloudWatch custom metrics, see Amazon’s “Publish Custom Metrics” article. Keep in mind that if you pull in CloudWatch custom metrics you will not only be charged by Amazon but also by Librato. Amazon currently charges $0.30 per custom metric per month and Librato’s cost for 5min metrics is $0.05 per month. As Librato was designed for custom metrics, you might consider sending the metrics directly to Librato’s RESTful API, thereby saving cost and gaining control over the metric resolution.

NOTE: Once a new custom metric is added to Cloudwatch, it may take up to 30 minutes to appear in Librato. This is because we maintain a cached enumerated list in Librato, that expires every 30 minutes.

A note on custom metrics and CloudWatch “dimensions”: Librato is currently only able to import CloudWatch custom metrics with a single dimension, therefore any custom metrics with multiple dimensions will be ignored.  The reason for this is that we translate the dimension “Value” directly into a Librato source name, and the ordering of CloudWatch dimensions is unspecified, which prevents us from naming sources in a predictable way.

As an example, the following metric will not be imported:

 {
    "Namespace": "Custom",
    "Dimensions": [
        {
            "Name": "MyTag1",
            "Value": "foo"
        },
        {
            "Name": "MyTag2",
            "Value": "bar"
        }
    ],
    "MetricName": "MyCustomMetric"
}

We are working to find the best solution to this problem which will likely include a user-specified set of tags and ordering.  Please contact support@librato.com if you have questions on this.

A note on custom metrics and source names: If you’re using the AWS monitoring scripts to send custom metrics, the source names for EC2 custom metrics are not likely to match the source names we import from the standard “AWS/EC2” namespace.  We retrieve the Availability Zone so that the source becomes zone.instance_id.  For example, the source for a custom metric might show i-12345 while the same instance will show a source of us-east-1a.i-12345 for AWS.EC2.CPUUtilization. This results in not being able to correlate sources across metrics, and is probably worth fixing on your installation.

Reporting Intervals

Depending on the service, CloudWatch metrics are reported at either a 5 minute frequency or a 1 minute frequency. If you want EC2 data at a 1 minute frequency, you’ll need to enable “Detailed Monitoring” (see http://aws.amazon.com/cloudwatch/details). Note that ELB, RDS and ElastiCache have 1 minute intervals by default. Using the AWS credentials you have provided, Librato will poll your CloudWatch metrics every 5 minutes and then write the resulting data into your Librato account, including data that is at 1 minute intervals.

FAQ:

Can I import data from multiple CloudWatch accounts?

Yes you can. Just click on the AWS Integration icon on of your Integrations page and you will see a list of all the current configurations. Click on the “Add Configuration” button to add another.

How much does it cost to add CloudWatch metrics?

AWS CloudWatch metrics vary in both the number of metrics reported per service and the reporting interval. The default configuration for Amazon CloudWatch EC2 instance metrics for example is to report 10 metrics at 5 minute intervals. Please note that we have to make API calls on your behalf which Amazon charges for (see the last paragraph in this article).

If you are subscribed to the Developer plan you can track up to 100 metrics for free, so you can track up to 10 EC2 instances.

The Production Plan does not limit the number of metrics. A single metric reporting every 5 minutes costs $0.05 per month and 1 minute resolution metric (used for detailed monitoring of EC2 instances) costs $0.10 per month.

Below is a list of what basic CloudWatch metrics cost for each service per instance per month:

AWS Service Metrics Cost/Instance Comment
API Gateway 7 $0.70 1min metrics
AWS Billing 1 $0.05 Reports every 4 hours
Cloudfront 6 $0.60 1min metrics
DynamoDB 22 $1.10 5min metrics
EC2 Container Service (ECS) 4 $0.40 1min metrics
Elastic Block Store (EBS) 10 $0.50 5min metrics
EBS Provisioned IOPS (SSD) 10 $1.00 1min metrics
Elastic File System (EFS) 8 $0.80 1min metrics
Elastic Compute Cloud (EC2) 12 $0.75 1min (3) & 5min (9) metrics
Elastic Load Balancing 13 $1.30 1min metrics
Elastic Beanstalk (Enhanced Health) 33 $3.30 1min metrics
ElastiCache Host-Level Metrics 5 $0.50 1min metrics
ElastiCache Redis 17 $1.70 1min metrics
ElastiCache Memcached 34 $3.40 1min metrics
Elastic MapReduce 26 $1.30 5min metrics
Elastic Search 20 $2.00 1min metrics
Internet of Things (IoT) 9 $0.90 1min metrics
Kinesis 17 $1.70 1min metrics
Kinesis Firehose 25 $2.50 1min metrics
Lambda 4 Variable* 1min & sparse metrics
OpsWorks 15 $0.75 5min metrics
Relational Database (RDS) 18 $1.70 1min (16) & 5min (2) metrics
Redshift 13 $1.30 1min metrics
Route53 2 $0.20 1min metrics
Simple Notification Service (SNS) 4 $0.20 5min metrics
Simple Queue Service (SQS) 8 $0.40 5min metrics
Simple Storage Service (S3) 2 $0.10 Reports every 24 hours
Simple Workflow Service (SWF) 21 $2.10 1min metrics
Storage Gateway 11 $0.55 5min metrics
Web Application Firewall (WAF) 3 $0.30 1min metrics
CloudWatch Custom Metrics 1 $0.10 1min resolution

* Cost of Lambda monitoring: If the job is running all the time, it will be charged as a 1 minute metric (4 metrics * $0.10 = $0.40). If it runs sparsely, the charge will be fractions of a cent.

A note about custom metrics...

Librato does not charge per Alert or API call and there is no limit to the amount of custom metrics you can send us. If you send custom metrics to AWS and then import them to Librato you are paying twice whereas if you only send them to Librato you only pay once. 5min metrics only cost $0.05/mo and 1min metrics cost $0.10/mo. If you’re an extensive user of Alerts and Custom Metrics, you may even save money by using Librato!

Activating billing metrics

Before you can see data on your estimated charges, you must enable monitoring in AWS, which creates the metric data. To learn how, see the article “Enabling the Monitoring of Your Estimated Charges ”. Note that AWS Billing reports every 4 hours so it may take a while for you to see your data.  Also note that you may need to activate the integration for us-east-1 (N. Virginia, US).  From the AWS documentation: “Billing metric data is stored in the US East (N. Virginia) region and represent worldwide charges”

Will I be charged for the AWS API requests?

Yes. Amazon charges a fee (currently $0.01 per 1,000 requests) for CloudWatch API access exceeding 1 million requests/month and these accrue on the account associated with the credentials you provide. Currently we poll the AWS services every 5 minutes and make one API call for each metric instance that we are tracking.

Do you support Auto Scaling Groups?

We do support Auto Scaling Group metrics such as “GroupMinSize”, “GroupMaxSize”, “GroupTotalInstances”, etc as shown in Amazon’s Use Amazon CloudWatch to Monitor Your Auto Scaling Instances article.

If you don’t care about differentiating between each instance, you can set the name tag or the librato:displayName tag; then every instance in the autoscaling group will have the same source display name. Even though they have the same source display name, they will show as separate lines on the graph because the underlying ID is different for each instance.

If you would like to be able to identify each individual instance, don’t set the name tag or the librato:displayName tag and we will append the autoscaling group name tag (aws:autoscaling:groupName) value to the instance id to create the source display name for each instance booted under the ASG.

Can I use Service-Side Aggregation (SSA) on CloudWatch metrics?

No. Because of the “polling” nature of the Librato / CloudWatch integration, there can be issues with the timing of data coming out of CloudWatch and into Librato. Since SSA essentially “waits” for measurements to come in during the aggregation period (say, 60s), and we pull data from CloudWatch based on the “last reported” values, timing issues may occur and the possibility of bad data (double counting) is introduced. For this reason, we do not recommend using SSA for your CloudWatch metrics.