Data partitions are a way to group or organize log data for faster and more efficient querying. When a query targets a single partition, New Relic's Logs UI:
- Scans less unrelated data.
- Returns results faster.
Accounts can have multiple partitions, and multiple partitions can be queried at the same time.
Data partitions also allow data to be mapped to an alternative, or “secondary” namespace with a fixed 30-day retention. This is useful for maintaining compliance with privacy-centric regulations and standards like the General Data Protection Regulation (GDPR).
Plan your partition
Before you start creating partitions, make sure you have the required permissions and a plan for how to implement the partitions.
Important
Logs are routed to partitions during the ingestion process, before data is written to NRDB. Partition rules won't affect logs that were ingested before the rule was created.
Sizing and organizing a partition
You can gain significant performance improvements with proper use of data partitions. Organizing your data into discrete partitions enables you to query them separately or all together. The goals of partitioning your data should be:
- Create data partitions that align with concepts in your environment or organization that are static or change infrequently (for example, by business unit, team, environment, service, etc.).
- Ensure each partition remains below 1 TB of daily ingest for optimal performance.
Having more partitions allows for more targeted searches, but creating too many partitions can make logs hard to find and increase administrative overhead. Finding the right balance is important. We support 100 partitions maximum per account, but the optimal number for most accounts is 10 to 15 partitions.
Choosing a namespace
A partition's namespace determines its retention period. We offer two retention options:
- Standard: The account's default retention determined by your New Relic subscription. This is the maximum retention period available in your account and is the namespace you'll select for most of your partitions.
- Secondary: 30-day retention. All logs sent to a partition that's a member of the Secondary namespace will be purged on a rolling basis 30 days after having been ingested.
Secondary retention is not a cost control mechanism; data is billed on ingest.
Use NerdGraph API to manage data partitions
If you want to manage your data partitions programatically, you can use NerdGraph, our GraphQL-format API. The NerdGraph data partitions tutorial shows how to query, create, and delete data partitions using this API.
Create partition rules via UI
From the left nav in the Logs UI, click Data partitions, then create a Log_ partition name with the retention namespace, optional description, and matching criteria.
To create a new partition rule:
- Go to one.newrelic.com > Logs.
- From Manage data on the left nav of the Logs UI, click Data partitions, then click Create partition rule.
- Define a Partition name as an alphanumeric string that begins with
Log_. - Add an optional description.
- Select the retention namespace for the partition.
- Set your rule's Matching criteria: Select EQUALS to target logs that match your criteria exactly, or select LIKE to apply a fuzzy match.
- Click the Enable rule slider, and click Create.
To view the list of custom parsing rules: From Manage Data on the left nav of the Logs UI, click Parsing.
Search data partitions
The default partition for all Logs accounts is Log. Any log that is not affected by a partition rule will be stored in the Log partition by default.
You can query multiple partitions simultaneously. For best performance, select the smallest number of partitions possible.
To search data partitions:
- From Views and partitions on the left nav of the Logs UI, click Select partitions.
- Click one or more partitions you want to query, or search for a partition name.
- Click Query logs to search your selected partitions.