Extend Lambda’s functionality using AWS Step Functions

How to use Step Functions to extend Lambda functionality.

Step Functions

Overview

AWS Lambda is a great service to run your code in a serverless environment. But it also has its own shortcomings like state management and limited execution time. Currently, Lambda’s max execution time is 15 mins, which means we cannot use it to run long-running tasks and there is no way to manage the state unless we use external services.

One question arises here that why use lambda for this purpose because it is not built for running long-running tasks and state management?

The reason to use it is due to its strong integration with other AWS services (i.e Kinesis, SQS, etc), the devs can focus on their code without worrying about the infrastructure. So, these benefits make it a strong contender for these types of tasks.

The article is about Step function utilization to overcome lambda’s limitations. I will use AWS RDS Disaster Recovery Plan (migrating manual snapshots to another region) as an example to demonstrate it. Snapshot creation and migration is a time-consuming task, so it will serve as a great example for this purpose.

If you are interested in the AWS RDS Disaster Recovery strategy read this article.

Pre Requisites

1. RDS instances must have Multi-AZ configuration enabled to avoid IO suspension.
2. Create an AWS KMS key in the destination region. It will be used during snapshot copy/migration operation.
3. Create Role and Policies for lambda.

Details

This section provides detailed guidelines on how to develop the whole workflow:

1. Develop a lambda function that will create and migrate the snapshots. Lambda needs to modify the lambda event object to add state variables and return the event object so that Step Function can decide what to do next. The code given below contains skeleton code for the lambda:

Lambda Skeleton

2. Once the lambda is developed, the next step will be to create a state machine for the lambda to manage its state so that we can run lambda again and again until all the snapshots are migrated to another region. The JSON given below contains the config for the state machine:

State Machine Configuration

The state machine will look like this:

State Machine

3. After lambda and state machine creation create an AWS Cloudwatch Event that will StepFunction trigger the state machine based on your requirements.

Final Thoughts

I hope you enjoyed this story. Please share your feedback about anything that can be improved or I missed. Thank you.