How to use Step Functions to extend Lambda functionality.

Step Functions

Overview

AWS Lambda is a great service to run your code in a serverless environment. But it also has its own shortcomings like state management and limited execution time. Currently, Lambda’s max execution time is 15 mins, which means we cannot use it to run long-running tasks and there is no way to manage the state unless we use external services.

One question arises here that why use lambda for this purpose because it is not built for running long-running tasks and state management?

The reason to use it is due to…


Overview

In this story, I will explain how to better utilize AWS Elasticsearch Service Shards. The approaches discussed are based on specific use cases, it might not be feasible for every use case.

Problem/Limitation

AWS Elasticsearch service has a hard limit of 1000 shards per data node. It can be increased but any update operation(storage increase, data nodes instance type change etc) on the cluster will revert the configuration back to the old state.

I have a data pipeline in which the data from the multiple sources are being processed and at the end dumped into Elasticsearch.

Elasticsearch cluster configuration is given…


Overview

There are multiple ways to secure the AWS API Gateway. I used the API Key to secure it. I integrated API Gateway with Lambda, so I will discuss things in Lambda’s context.

Go through this AWS API Gateway’s documentation to grasp the basic understanding.

  1. Create a Lambda and add a trigger:

Overview

This story is about how to distribute data among kinesis shards uniformly but before moving forward go through the kinesis documentation to grasp the basic understanding of kinesis data stream. I used Lambda as a consumer for Kinesis data stream, so I will discuss things in Lambda’s context.

Read this article about Kinesis stream and AWS Lambda.

Problem

A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence number that is assigned by Kinesis Data Streams.

To put a record in Kinesis stream Partition Key must be provided…


Overview

This story is about how to utilize Apache Zookeeper for Configuration Management of Distributed Systems.

There are a lot of problems in managing and designing Distributed Systems:

  • Security
  • Scalability
  • Failure handling
  • Configuration Management
  • Concurrency
  • Transparency
  • Quality of service

But in this story, we will focus on the configuration management of distributed system.

Assumptions

This story has been written by keeping the following assumptions in mind:

  • A distributed system’s services use configuration to operate.
  • Configuration can be changed at runtime and the services should not be restarted to load new configurations.

Details

Suppose we have a distributed system that has 100 services running…


https://www.zenitk.com/import-from-csv-to-elasticsearch-with-logstash

Overview

This story is not about dumping CSV data in AWS Elasticsearch using Logstash because there are already a lot of good articles. I personally like this one.

Instead, it is about an issue I faced while doing the above operation.

Issue

I deployed an Elasticsearch instance on AWS which was publically accessible. I was able to perform requests using CURL. But when I tried to access the dump data using Logstash with this configuration:

I was getting this error:

elasticsearch - Failed to install template. {:message=>"Got response code '401' contacting Elasticsearch at URL '<elasticsearch-url>:443/_xpack'"

By looking at the URL, I…


Rotate AWS Elasticsearch Service Data using Elasticsearch Curator.

Overview

Problem

Elasticsearch is normally used for application logs management and monitoring. Logs should be retained for a specific interval of time, based on the needs and later must be discarded to clean up the disk space.

Elasticsearch provides a feature that can be used to delete the old data. But it is not recommended due to this problem.

The recommended way to clean up data is by using Elasticsearch Curator.

So in this story, we will create a lambda for curator and trigger it by using the CloudWatch event after a defined interval…


Validate a URL/Domain using Node.js HTTPs Module.

Overview

This post provides guidelines on how to validate a xyz.com domain. By validation, I mean that the domain has a valid certificate signed by Certificate Authority.

Scenarios

The list given below contains scenarios in which you want to validate the domain/URL:

  1. You want to upload data to a server with a URL like this (xyz.com) and you are not sure whether this server is secure or not.
  2. You have developed a B2B service and you want to only serve requests from valid domains.

How to do it?

In node.js there are two ways to do it:

  1. https module

https://www.kindpng.com/picc/m/188-1882559_python-flask-hd-png-download.png

This story provides guidelines on how to configure CI/CD pipeline for a Flask Application using Travis CI, Github Actions, and Heroku.

The detailed guidelines are provided in the README.md of this repository.


This repository contains guidelines to call AWS API Gateway’s endpoint from EC2 instance using a node.js and python scripts. The scripts need to send sign requests to the endpoint, so therefore it needs AWS credentials. For security and automation purposes, AWS credentials are not provided as environment variables instead a role has been assigned to the EC2 instance, which allows EC2 to access the API Gateway Endpoint. In this story, I will explain how to load the credentials programmatically using node.js and python scripts.

Follow the step by step guidelines provided in the README.md of this repository for implementation details.

I hope that you liked this story, please give feedback regarding anything that can be improved.

Irtiza

DevSecOps Engineer https://irtizaali.com/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store