Reliable Webhooks processing with AWS Lambda and Amazon SQS

In this blog post, I propose an architecture using 2 AWS services, namely AWS Lambda and Amazon SQS, to achieve a notification pipeline using a webhook.

Before we dive into the solution, let's review the concepts we'll be using.

What are Webhooks?

Webhooks offer a way to send real time notifications to clients. Opposite to traditional APIs, where a client requests data from a server, Webhooks allow a client as a consumer to subscribe to server events. After the initial registration, no more interaction is needed.

What is Amazon SQS (Simple Queue Service)

Amazon SQS is message queuing service commonly used to decouple services: instead of having two or more services communicating synchronously, queues can be used to defer this communication asynchronously, allowing simpler scaling.

What is AWS Lambda?

AWS Lambda is a serverless compute platform. This means it uses AWS compute infrastructure, but all the instance setup is managed by AWS: the customer is only responsible for the business logic. AWS Lambda uses Lambda functions, which are applications that are triggered by events.

Proposed architecture

Here's a diagram describing the main components of the architecture. They will be discussed in more detail later.

Setting up the listener

This post assumes you have Node.js installed, as well as AWS CDK Toolkit. You should have an AWS account created.

Let's start by creating the customer webhook endpoint. This will serve as a target to the notifications.

Create a new file app.js with the following code:

const http = require('http');

const port = 3000;

const server = http.createServer((req, res) => {
    let body = '';
    req.on('data', chunk => {
        body += chunk.toString();
    });
    req.on('end', () => {
        console.log(body);
        res.end('ok');
    });
});

server.listen(port, () => {
    console.log(`Server running  on ${port}/`);
});

This is a simple listener that will wait for a request with a body, and will print the body contents to the console. We will use this to make sure we are getting our notifications, meaning the webhook is effectively working.

To start the listener, run:

node app.js

By default, the server will be running on localhost:3000. Since we need to communicate with our listener from AWS Lambda, we need to make this endpoint public. A really useful tool for this I often use is ngrok, you can use it to create a public proxy to your local port (you’re free to use any other method you’re familiar with). After exposing the endpoint publicly, you should have a URL that is reachable from the Internet.

Infrastructure

So we have a client endpoint waiting for notifications. Now we need the infrastructure to send the notifications.

Let's detail how each architecture component fits together.

Components

Amazon SQS

In this particular use case, Amazon SQS is used as a source of events for the notifications. The server business logic is abstracted here, for simplicity purposes; in a real life scenario, a service would send events, which eventually reach an Amazon SQS queue. Here, we'll push messages directly to the queue to model this behaviour.

AWS Lambda

For this particular use case, the Lambda function will be invoked to consume messages from the SQS queue and, for each message, will send a notification to the customer webhook endpoint.

Building the infrastructure

Using AWS CDK to deploy infrastructure

Amazon offers an Infrastructure-As-Code solution, called AWS Cloud Development Kit, or CDK for short. It uses AWS Cloudformation internally to define infrastructure in a reviewable, deployable manner. CDK is offered in multiple languages, we'll use Typescript in this example.

AWS CDK setup

Assuming you already have CDK Toolkit installed, create a folder for the CDK code and initialise the project:

mkdir notification-cdk
cd notification-cdk && cdk init app --language typescript

cdk init app builds some initial scaffolding for the project. It creates a Stack, which will contain the pieces of infrastructure required.

A final step for the setup is to go the generated code for the stack under bin/my-app-cdk.ts and uncomment the env: line and replace with your AWS account info:

// env: { account: '123456789012', region: 'us-east-1' },

Let's add the necessary pieces for building our infrastructure stack on 'lib/cdk-stack.ts'

import * as path from "path";
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import { SqsEventSource } from "aws-cdk-lib/aws-lambda-event-sources";

export class EcdevWebhookInfrastructureCdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const queue = new sqs.Queue(this, "EventsQueue", {
      queueName: "EventsQueue",
    });

    const lambdaFunction = new lambda.Function(this, "EventHandler", {
      runtime: lambda.Runtime.NODEJS_16_X,
      handler: "index.handler",
      code: lambda.Code.fromAsset(path.join(__dirname, "lambda-handler")),
    });

    lambdaFunction.addEventSource(new SqsEventSource(queue));
  }
}

Code walkthrough

Let's go through the code to make sure everything is clear.

1. Create an Amazon SQS queue.
2. Create an AWS Lambda function, using a local folder as source.
3. Set Amazon SQS queue as an event source for the function.

Edit code for AWS Lambda function

We'll use the CDK stack to include the Lambda function code, since it's a simple example.

In the root of the cdk project, run:

mkdir lambda-handler
touch lambda-handler/index.js

This creates a new folder and file where the AWS Lambda function code will live. Edit the file with the following code, replacing <LISTENER_URL>:

var http = require('https');

exports.handler = function (event, context) {
    event.Records.forEach(record => {
        const params = {
            host: "<LISTENER_URL>",
            body: record.body,
            method: "POST"
        }
        var post_req = http.request(params);

        post_req.write(record.body);
        post_req.end()
    })
}

What the lambda handler's code does is, for each event record (which will match an individual queue message), create a HTTP POST request and send it to the Listener endpoint that was setup earlier. Please note that the host mustn't include the protocol (http:// or https://).

Deploy the infrastructure

Now that we have the infrastructure ready to be deployed, let's execute the following command to do it:

cdk deploy

Sending the message to Amazon SQS

aws sqs get-queue-url --queue-name EventsQueue

aws sqs send-message --queue-url 'QUEUE_URL' --message-body 'This is an important and time-critical message'

Checking the notification is received

After sending the message to the Amazon SQS queue, if all went well, you should see the message in the client application console:

Server running  on 3000/
Received notification: This is an important and time-critical message

Cleanup

To clear the infrastructure resources just created, you can use CDK Toolkit to destroy the stack. Simply run:

cdk destroy

Conclusion

In this blog post, a simple architecture to use a webhook to send notifications to clients was presented. This shows the essential working blocks the solution. In real life scenarios, there are other concerns, such as security, reliability, cost, that are out of scope for this use case, but definitely crucial to a production service.

All code referenced in this blog post is available in the listener and cdk repositories.