I believe there are many people who have used AWS for their projects or are learning AWS. We know that with cloud services like AWS, it’s not very easy to connect to the cloud when developing locally, not to mention that the staging/production environment will have security considerations. So when we create a project, how do we build its local development environment to facilitate our local development and debugging? That’s right! Use localstack!

LocalStack - A fully functional local AWS cloud stack

LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications.

Currently, the focus is primarily on supporting the AWS cloud stack.

How is it used?

Create an SNS service with LocalStack

Imagine we have a SpringBoot service that provides an interface to send an email to a user when the user submits a record, which we would normally do asynchronously by saving the database and sending the email. If we use AWS, we can call the SNS service after saving the database, publish an Event and wait for the downstream mail service to subscribe to the corresponding topic and then consume it.

So how do we create it? In the docker-compose.yml file, add localstack, SERVICES and specify the sns as follows

docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

version: "3.7"

services:
  localstack:
    image: localstack/localstack:0.12.1
    networks:
      - app_net
    ports:
      - "4566:4566"
    environment:
      - SERVICES=sns
      - DEFAULT_REGION=ap-southeast-2
      - DEBUG=1
    volumes:
      - ./auto/create-localstack-topic:/docker-entrypoint-initaws.d/create-localstack-topic.sh

create-localstack-topic.sh

In addition, you can see that we have a script in the volume, which is the script to create the SNS. In fact, this command is aws cli, which can be found on its official website, we just need to replace aws with awslocal

1
2
3
4
5
6

#!/bin/bash

REGION=${DEFAULT_REGION:-ap-southeast-2}
TOPIC_NAME=demo-events-topic

awslocal sns create-topic --name=${TOPIC_NAME} --region "${REGION}"

Startup Log

When docker-compose up localstack is started, the SNS service is created in that container for us to use.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

sns_1            | Waiting for all LocalStack services to be ready
sns_1            | 2020-12-27 07:02:36,554 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
sns_1            | 2020-12-27 07:02:36,559 INFO supervisord started with pid 15
sns_1            | 2020-12-27 07:02:37,566 INFO spawned: 'dashboard' with pid 21
sns_1            | 2020-12-27 07:02:37,571 INFO spawned: 'infra' with pid 22
sns_1            | 2020-12-27 07:02:37,577 INFO success: dashboard entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
sns_1            | 2020-12-27 07:02:37,577 INFO exited: dashboard (exit status 0; expected)
sns_1            | (. .venv/bin/activate; exec bin/localstack start --host)
sns_1            | 2020-12-27 07:02:38,591 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sns_1            | LocalStack version: 0.12.1
sns_1            | Starting local dev environment. CTRL-C to quit.
sns_1            | 2020-12-27T07:02:39:DEBUG:bootstrap.py: Loading plugins - scope "services", module "localstack": <function register_localstack_plugins at 0x7f963f120f70>
sns_1            | Waiting for all LocalStack services to be ready
sns_1            | 2020-12-27T07:02:43:INFO:localstack.utils.analytics.profiler: Execution of "load_plugin_from_path" took 4333.9550495147705ms
sns_1            | 2020-12-27T07:02:43:INFO:localstack.utils.analytics.profiler: Execution of "load_plugins" took 4334.24186706543ms
sns_1            | Starting edge router (https port 4566)...
sns_1            | Starting mock SNS service on http port 4566 ...
sns_1            | 2020-12-27T07:02:45:INFO:localstack.utils.analytics.profiler: Execution of "prepare_environment" took 2061.4540576934814ms
sns_1            | 2020-12-27T07:02:45:INFO:localstack.multiserver: Starting multi API server process on port 59903
sns_1            | [2020-12-27 07:02:45 +0000] [23] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
sns_1            | 2020-12-27T07:02:45:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
sns_1            | [2020-12-27 07:02:45 +0000] [23] [INFO] Running on http://0.0.0.0:59903 (CTRL + C to quit)
sns_1            | 2020-12-27T07:02:45:INFO:hypercorn.error: Running on http://0.0.0.0:59903 (CTRL + C to quit)
sns_1            | 2020-12-27 07:02:45,824:API:  * Running on http://0.0.0.0:57589/ (Press CTRL+C to quit)
sns_1            | Waiting for all LocalStack services to be ready
sns_1            | Ready.
sns_1            | 2020-12-27T07:02:50:INFO:localstack.utils.analytics.profiler: Execution of "start_api_services" took 5102.221965789795ms
sns_1            | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/create-localstack-topic.sh
sns_1            | {
sns_1            |     "TopicArn": "arn:aws:sns:ap-southeast-2:000000000000:demo-events-topic"
sns_1            | }

Calling

1

aws --endpoint-url=http://localhost:4566 sns publish --topic-arn arn:aws:sns:ap-southeast-2:000000000000:demo-events-topic --region ap-southeast-2 --message "Hello SNS"

Note that you need to override the endpoint-url when calling services in localstack locally with the aws command, otherwise the credentials will be used to call the services in the real environment.

Notes for use in SpringBoot

For use in SpringBoot or other codebases (such as node), you can create different SNSClients for different environments, taking care to override the endpoint for local environments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@Configuration
@Profile({"local", "docker"})
public class LocalSnsClientConfiguration {

    @Value("${aws.sns.endpoint}")
    private String awsSnsEndpoint;

    @Bean
    public SnsClient snsClient() {
        var clientBuilder = SnsClient.builder();
        if (!Strings.isNullOrEmpty(awsSnsEndpoint)) {
            clientBuilder.endpointOverride(URI.create(awsSnsEndpoint));
        }
        return clientBuilder.build();
    }
}

Start multiple services

The above is just an example of starting an SNS service. In practice, we will use multiple services in combination. For example, there will be an SQS service that subscribes to an SNS topic and then triggers a lambda to perform some tasks, so how can we implement these services locally to subscribe and trigger each other? In fact, you only need to start multiple services in a localstack and then execute some scripts to establish the relationship between them (the specific commands are the same as aws cli), as follows.

docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

version: "3.7"

services:
  localstack:
    image: localstack/localstack:0.12.1
    privileged: true
    container_name: localstack
    networks:
      - app_net
    ports:
      - "4566:4566"
    environment:
      - SERVICES=sns,sqs,kms,cloudwatch,lambda
      - DEFAULT_REGION=ap-southeast-2
      - LAMBDA_EXECUTOR=docker-reuse
      - LAMBDA_REMOTE_DOCKER=false
      - LAMBDA_DOCKER_NETWORK=host
      - DEBUG=1
      - HOST_TMP_FOLDER=${TMPDIR}
      - DOCKER_HOST=unix:///var/run/docker.sock
      - LOCAL_CODE_PATH=${PWD}
    volumes:
      - ${TMPDIR:-/tmp/localstack}:/tmp/localstack
      - /var/run/docker.sock:/var/run/docker.sock
      - ./auto/create-localstack:/docker-entrypoint-initaws.d/create-localstack.sh
      - ./kms/kms_seed.yaml:/init/seed.yaml

networks:
  app_net:

kms_seed.yaml

I have started sns,sqs,kms,cloudwatch,lambda here. It is worth mentioning that in addition to overriding the endpoint-url for local access to sqs, sns, kms, etc., you need to specify a seed.yml to encrypt and decrypt it for local use with kms.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Keys:
  Symmetric:
    Aes:
      - Metadata:
          KeyId: 832ac356-3c82-4c4d-a3dc-7489da152197
        BackingKeys:
          - 2bdaead27fe7da2de47945d34cd6d79e36494e73802f3cd3869f1d2cb0b5d74c
Aliases:
  - AliasName: alias/testing
    TargetKeyId: 832ac356-3c82-4c4d-a3dc-7489da152197

Create script create-localstack.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/bin/bash

QUEUE_NAME=demo-queue
TOPIC_NAME=demo-topic
FUNCTION_NAME=demo-function
APP_ENV=dev

awslocal sns create-topic --name=${TOPIC_NAME}
awslocal sqs create-queue --queue-name=${QUEUE_NAME}
awslocal sns subscribe \
    --topic-arn arn:aws:sns:ap-southeast-2:000000000000:${TOPIC_NAME} \
    --protocol sqs \
    --notification-endpoint http://localhost:4566/000000000000/${QUEUE_NAME}

awslocal lambda create-function \
    --code S3Bucket="__local__",S3Key="${LOCAL_CODE_PATH}" \
    --function-name ${FUNCTION_NAME} \
    --runtime nodejs12.x \
    --timeout 5 \
    --handler dist/index.handler \
    --role dev \
    --environment "{\"Variables\":{\"APP_ENV\":\"${APP_ENV}\"}}"

awslocal lambda create-event-source-mapping \
    --event-source-arn arn:aws:sqs:ap-southeast-2:000000000000:${QUEUE_NAME} \
    --function-name ${FUNCTION_NAME} \
    --enabled

Local Start

execution docker-compose up localstack

Now send an SNS message from the command line to trigger our Lambda execution.

1

aws --endpoint-url=http://localhost:4566 sns publish --topic-arn arn:aws:sns:ap-southeast-2:000000000000:demo-topic --region ap-southeast-2 --message "Hello SNS - SQS - Lambda"

Write a handler() method to Lambda’s index.ts.

1
2
3
4
5
6

require('./overwriteAwsLocalEndpoint'); //overwrite aws local endpoint,Please keep it here.
import { SQSEvent, SQSHandler } from 'aws-lambda';

export const handler: SQSHandler = (event: SQSEvent) => {
  console.log(JSON.stringify(event.Records));
}

Print the message body of the SQS.

1
2
3
4

localstack    | > START RequestId: ce5ae5ff-054d-16e0-dc62-71161118d3bd Version: $LATEST
localstack    | > 2020-12-27T09:32:56.373Z	ce5ae5ff-054d-16e0-dc62-71161118d3bd	INFO	[{"body":"{\"Type\": \"Notification\", \"MessageId\": \"04c12e03-66d0-474a-a60a-f0b3c2451456\", \"Token\": null, \"TopicArn\": \"arn:aws:sns:ap-southeast-2:000000000000:demo-topic\", \"Message\": \"Hello SNS - SQS - Lambda\", \"SubscribeURL\": null, \"Timestamp\": \"2020-12-27T09:32:52.202Z\", \"SignatureVersion\": \"1\", \"Signature\": \"EXAMPLEpH+..\", \"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem\"}","receiptHandle":"exexifyylldwuznxlicibcanaqvcplpaeoztcdlltkzbsuvwiifvlyixrxwuzrmumlmkggofmiencdxilzoaluyreszdppsbycpxcowwvmeiieeplulkitfztfxzkjazucucauhuobpvlzdcnjdcmygqvbrouxkxoggcfryzqtibyquhikawczuif","md5OfBody":"d96df71c445e9282ed4c2fefbf4c8ca1","eventSourceARN":"arn:aws:sqs:ap-southeast-2:000000000000:demo-queue","eventSource":"aws:sqs","awsRegion":"ap-southeast-2","messageId":"a59f7c57-651b-54f6-70bd-a2933fa57099","attributes":{"SenderId":"AIDAIT2UOQQY3AUEKVGXU","SentTimestamp":"1609061572241","ApproximateReceiveCount":"1","ApproximateFirstReceiveTimestamp":"1609061572312"},"messageAttributes":{},"md5OfMessageAttributes":null,"sqs":true}]
localstack    | > END RequestId: ce5ae5ff-054d-16e0-dc62-71161118d3bd
localstack    | > REPORT RequestId: ce5ae5ff-054d-16e0-dc62-71161118d3bd	Init Duration: 3381.65 ms	Duration: 13.13 ms	Billed Duration: 100 ms	Memory Size: 1536 MB	Max Memory Used: 55 MB

About the use of Lambda in LocalStack

Creating a Lambda runtime environment locally is one of the more troublesome services in my opinion, and the following is the official configuration for Lambda creation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

STEPFUNCTIONS_LAMBDA_ENDPOINT: URL to use as the Lambda service endpoint in Step Functions. By default this is the LocalStack Lambda endpoint. Use default to select the original AWS Lambda endpoint.

LAMBDA_EXECUTOR: Method to use for executing Lambda functions. Possible values are:

	- local: run Lambda functions in a temporary directory on the local machine
	- docker: run each function invocation in a separate Docker container
	- docker-reuse: create one Docker container per function and reuse it across invocations
	For docker and docker-reuse, if LocalStack itself is started inside Docker, then the docker command needs to be available inside the container (usually requires to run the container in privileged mode). Default is docker, fallback to local if Docker is not available.

LAMBDA_REMOTE_DOCKER: determines whether Lambda code is copied or mounted into containers. Possible values are:

	- true (default): your Lambda function definitions will be passed to the container by copying the zip file (potentially slower). It allows for remote execution, where the host and the client are not on the same machine.
	- false: your Lambda function definitions will be passed to the container by mounting a volume (potentially faster). This requires to have the Docker client and the Docker host on the same machine.

LAMBDA_DOCKER_NETWORK: Optional Docker network for the container running your lambda function.

LAMBDA_DOCKER_DNS: Optional DNS server for the container running your lambda function.

LAMBDA_CONTAINER_REGISTRY: Use an alternative docker registry to pull lambda execution containers (default: lambci/lambda).

LAMBDA_REMOVE_CONTAINERS: Whether to remove containers after Lambdas finished executing (default: true).

FAQ

Why do I always get some client’s credentials error when I use SNS, KMS?

Because the endpoint-url required for the local environment is not overridden, refer to the explanation in this article

Why did the message to the SNS succeed but did not trigger Lambda?

qing check your creation script to make sure that your SQS is subscribed to the corresponding topic of the SNS and that the SQS has a Mapping that can trigger Lambda

This example code?

Fatezhang/aws-localstack-demo

Reference http://zhangjiaheng.cn/blog/20201227/%E4%BD%BF%E7%94%A8localstack-%E6%90%AD%E5%BB%BA-AWS-%E6%9C%AC%E5%9C%B0%E5%BC%80%E5%8F%91%E7%8E%AF%E5%A2%83/