New Relic currently provides independent integrations with AWS to collect performance metrics and metadata for more than 50 AWS services.
Learn how to set up Metric Streams in New Relic.
Why does this matter?
Our current system, which relies on individual integrations, runs on a polling fleet and calls multiple AWS APIs at regular intervals to retrieve the metrics and metadata. Using AWS CloudWatch significantly improves how metrics are gathered, overcoming some of the limitations of using the individual integrations.
API mode | Stream mode |
|---|---|
It requires an integration with each AWS service to collect the metrics. | All CloudWatch metrics from all AWS services and custom namespaces are available in New Relic at once, without needing a specific integration to be built or updated. There's one exception: metrics that are made available to CloudWatch with more than 2 hours delay are not included in the stream. |
It adds an additional delay to metrics being available in New Relic for alerting and dashboarding. The fastest polling interval we offer today is 5 minutes. | Latency is significantly improved, since metrics are streamed in less than two minutes since they are made available in AWS CloudWatch. |
It may lead to AWS API throttling for large AWS environments. | AWS API throttling is eliminated. |
Want to try out our Amazon CloudWatch Metric Streams integration? Sign up with New Relic for free, forever!
Cost considerations
Consider the following when evaluating the cost of the AWS CloudWatch metric streams integration with New Relic:
- AWS CloudWatch metric updates.
- AWS Kinesis Firehose ingest.
- AWS Kinesis Firehose data transfer.
- Optionally, custom tags and CloudWatch metrics enrichment with resource metadata is based on the AWS Config service.
Migrating from AWS API polling integrations
When metrics are sent via Metric Streams to New Relic, if the same metrics are being retrieved using the current poll-based integrations, those metrics will be duplicated. For example, alerts and dashboards that use sum or count will return twice the actual number. This includes alerts and dashboards that use metrics that have a .Sum suffix.
We recommend sending the data to a non-production New Relic account where you can safely do tests. If that is not an option, then AWS CloudWatch Metric Stream filters are available to include or exclude certain namespaces that can cause trouble.
Alternatively, you can use filtering on queries to distinguish between metrics that come from Metric Streams and those that come through polling. All metrics coming from Metric Streams are tagged with collector.name='cloudwatch-metric-streams'.
Migration steps
On a typical deployment, migrating from API polling to metric stream involves the following steps (we recommend trying this on a dev / staging environment first):
- Go through the AWS UI in New Relic (or use NerdGraph APIs) to link your AWS account with New Relic. This is currently needed even if your AWS account is already linked with polling integrations.
- Make sure you complete the last step in the onboarding, which involves enabling AWS CloudWatch metric stream and the AWS Kinesis Data Firehose to push metrics to New Relic.
- Repeat step 2 for any additional AWS region you want to monitor, since AWS CloudWatch requires one stream per region.
- Ensure metrics are received from all connected regions and namespaces. This may take several minutes.
- Disable all unnecessary polling integrations in the previous AWS provider account. Remember that some integrations still need to be enabled since they aren't fully replaced by metric streams.
Query, dashboard, and alert considerations
AWS Metric Streams integration uses the Metric API to push metrics in the dimensional metric format.
Poll-based integrations push metrics based on events (for example, ComputeSample event), and will be migrated to dimensional metrics in the future.
To assist in this transition, we provide a mechanism (known as shimming) that transparently lets you write queries in any format. Then these queries are processed as expected based on the source that's available (metrics or events). This mechanism works both ways, from events to metrics, and vice versa.
Please consider the following when migrating from poll-based integrations:
- Dashboards: Custom dashboards that use poll-based AWS integration events will still work as expected.
- Alerts: Alert conditions that use poll-based AWS events will still work. We recommend adapting those to the dimensional metric format (using NRQL as source).
- Entities: New Relic Explorer might show duplicated entities for up to 24 hours.
Integrations not fully replaced by metric streams
The AWS CloudWatch Metric Streams integration focuses on CloudWatch metrics. As a result, the following integrations still need to be configured and enabled to get complete visibility from AWS services.
Polling integrations based on service APIs:
- AWS Billing
- AWS CloudTrail
- AWS Health
- AWS Trusted Advisor
- AWS X-Ray
Integrations based on CloudWatch Logs, forwarded to New Relic via Lambda:
- AWS RDS Enhanced
- AWS VPC Flow Logs
Metrics not available with CloudWatch metric streams
AWS CloudWatch metric stream will not include metrics that are made available to CloudWatch with more than 2 hours delay. Examples of AWS namespaces that might contain metrics that are aggregated and exposed after 2 hours include: AWS DMS, AWS RDS, AWS DocDB, AWS S3 and AWS DAX.
Please refer to AWS documentation to learn more.