We're big fans of Amazon Web Services. While we run our heaviest workloads directly on EC2 for cost efficiency reasons, we've adopted AWS API Gateway and AWS Lambda for on-demand workloads where flexibility is important. It's hard to get started, so we're sharing a template we use. It shows how to use CloudFormation to publish a REST API using:

• Lambda - to provide as-needed compute resources
• API Gateway - to expose our Java application to any HTTP client
• CloudWatch - for HTTP request/response logs from API Gateway and debug logs from the Lambda function
• Route 53 - to expose our API Gateway service on a custom domain name
• Certificate Manager - to automatically issue and renew the SSL certificate for our custom domain name
• IAM - to limit the privileges available to our Lambda

This stack is what powers our regex tester and Common Crawl search previews.

Just want the code? You can skip to the associated repository on GitHub, with Java and Nodejs 10.x examples.

The Goal

We're going to build an end-to-end service that echoes the user's input. You could use any of Lambda's supported languages, but we'll use Java [1]. When everything's done, you'll be able to send a request like this:

$ curl https://echo.code402.com/echo?message=hello
HELLO

We could do this entirely through the AWS UI, but instead we'd like to describe the infrastructure with a CloudFormation template. This delivers some real benefits: teammates can code review the infrastructure, developers can easily duplicate the stack to deploy a staging environment, and the ops team can easily replicate the stack to multiple regions for availability or performance improvements. We use CloudFormation because we like to stick to AWS-specific tooling, but you may also want to consider competing tools like Terraform or Serverless.

The Service

Since the actual service isn't too important, we'll write one that receives some text and responds with SOME TEXT. Say hello to the Shouty Echo Service! It's quite short:

Even with no prior experience with API Gateway, you can likely see what's going on. The APIGatewayProxyRequestEvent contains the properties of the HTTP request. Your service's response is written to the APIGatewayProxyResponseEvent object [2].

The service and its dependencies need to be packaged into a single, self-contained uberjar. Our GitHub repository includes a sample project that does just that.

The CloudFormation Template

CloudFormation translates a text file describing AWS resources into live instances of those resources. This "stack" of resources can be monitored using the AWS CloudFormation console. The text file itself can be YAML or JSON. Most people use YAML, since it supports comments.

We'll construct our template step-by-step from the ground up. First we'll create the AWS Lambda that runs the shouty echo service. Then we'll connect it to CloudWatch for logging. Next, we'll connect it to API Gateway so that anonymous Internet users can invoke it over HTTP. Finally, we'll integrate with Route 53 and AWS Certificate Manager to give it a custom domain name.

The Lambda Function

Start by provisioning a Lambda function that runs the service:

This template introduces two high-level CloudFormation concepts:

• Parameters let the user override default values. For example, to provision a Lambda function, you need to specify the S3 bucket that contains the code. You obviously won't have access to our bucket, so you can provide the S3Bucket parameter to specify your own.
• Resources are the meat of CloudFormation. They describe a resource you'd like to provision, like an S3 bucket or SQS queue. Almost every AWS service has at least some support for CloudFormation - you can refer to the Resource reference for more information.

If you have previous experience configuring Lambda functions, you'll see that this template describes the steps you would have otherwise taken in the UI. The !Ref and !GetAtt functions are intrinsic functions that CloudFormation provides. They're needed when a hardcoded value isn't enough, for example, to include a user-provided parameter value, or to reference the Amazon Resource Name of a resource that will be created as part of the stack.

You could provision this stack immediately through the UI, or via the command-line: aws cloudformation create-stack --region us-east-1 --stack-name lambda-demo --template-body file://stack-lambda.yaml --capabilities CAPABILITY_IAM --parameters ParameterKey=S3Bucket,ParameterValue=your-bucket-name

CloudWatch Logging

To ease debugging, we'd like to log output from the Lambda to CloudWatch. By default, Lambda tries to log to a CloudWatch log group named after the function name of the Lambda. In order for it to work, we need to update the IAM role's permissions to include a policy that grants it access to that log group:

Logs default to being kept forever. That adds up, so we'll explicitly limit them to 14 days:

Note that we've introduced the Sub intrinsic function to construct strings that contain your account ID, AWS region and Lambda function name.

API Gateway

To expose this service to the public Internet, we turn to API Gateway. There's a lot going on, so we'll use inline comments to break it down:

Lastly, we introduce a new top-level section, the Outputs section. Because CloudFormation discourages hardcoding values, the identifiers of the resources will change each time you rebuild the stack. That's a pain! Outputs let you collect those values from each run in a central place. They show up in the UI like this:

The shouty echo service is now available!

$ curl https://6jdt0ybq1e.execute-api.us-east-1.amazonaws.com/echo?message=hello
HELLO

Custom Domain Names

API Gateway domain names are hard to remember, so we'll configure a vanity subdomain, for example echo.code402.com. Since we host our domain with Route 53, this is almost painless! We'll just have to add some entries to our CloudFormation file. First, we'll add some parameters to capture the desired domain name and the ID of the Route 53 zone:

Next, we'll add some new resources to register the domain name, request an SSL certificate from AWS Certificate Manager, and tell API Gateway to respond on the domain name:

There's a weird quirk here. Before it issues an SSL certificate, AWS Certificate Manager requires proof of ownership of the domain name. You'd think that since we're using Route 53 to manage the domain, AWS could verify this without our assistance, but it turns out it can't. Instead, when we apply the template, CloudFormation pauses until we create the verification record ourself in Route 53:

Once the record is created, CloudFormation will continue provisioning the stack. When it's all done, we can do:

$ curl https://echo.code402.com/echo?message=hello
HELLO

Conclusion

We think that CloudFormation, while initially daunting, unlocks some real value. The template file we built can be peer-reviewed by other members of the team. It can be checked in to source control so future edits are subject to review. A developer new to the project can spin up their own entirely self-contained stack to use as a sandbox for testing.

The Java app and CloudFormation templates used in this article can also be found on our GitHub. The repo includes a bonus JavaScript example built on Node v10, too.

Notes

1. In fact, if you're comfortable writing some shell scripts and compiling code for Linux, you can target any language you like with a custom runtime.
2. This is using API Gateway in "proxy" mode, which exposes the underlying HTTP concepts. API Gateway also offers a legacy mode that tries to abstract these details away with a very opinionated set of design decisions. Most people prefer the proxy mode.