Rotate AWS Elasticsearch Service Data using Elasticsearch Curator.
Elasticsearch is normally used for application logs management and monitoring. Logs should be retained for a specific interval of time, based on the needs and later must be discarded to clean up the disk space.
The recommended way to clean up data is by using Elasticsearch Curator.
So in this story, we will create a lambda for curator and trigger it by using the CloudWatch event after a defined interval of time. Once lambda is triggered it will clean up the data using multiple filters.
Each of the above steps will be discussed in details later in this story.
It is better to have knowledge about these services:
I am assuming that you have a running AWS Elasticsearch cluster and application logs are being dumped in it.
In this section, I will explain each step of the solution in detail:
1. Curator’s Lambda
First of all, we will create a lambda for the curator. Follow the steps to do it:
- Create an IAM role and attach this inline policy:
This policy will allow lambda to perform Get and Delete operations on the Elasticsearch cluster.
2. Create a lambda and assign the above role to it.
3. Once lambda is created we need to package and publish its code. In this Github repository, you will find:
- lambda’s code.
- guidelines on how to use filters (it will be used to filter elasticsearch indices that need to be deleted).
- how to use different environment variables.
- guideline on packaging and publishing lambda’s code.
2. Curator’s Lambda
Once lambda is published, we need to implement a cron functionality to trigger the lambda on regular intervals, to do it we will use AWS CloudWatch Events. Follow the guidelines given below:
- Create an Event Rule.
- Choose the Schedule event source.
- Create and assign a Cron Expression based on your needs. AWS cron expression is a little bit different from the one we normally use. Details can be found on this link
- In the target select the lambda created above.
I hope that you like this story and please give feedback about anything that can be improved or I have missed. Thank you :)