capsule.adrianhesketh.com

go-sqlite3 on AWS Lambda with CDK

On a recent project, I chose to use `sqlite` as an embededed database engine for running SQL statements written by a data team as part of an API.

By using `sqlite`, I could horizontally scale the excution of the SQL statements across multiple execution environments instead of centralising traffic onto a database server cluster like RDS running Postgres.

`sqlite` is also mature and widely deployed, so I was reasonably confident that I wouldn't run into any strangeness on the SQL expression handling.

Initially, my team ran the API running the embedded `sqlite` database as a Docker container in Fargate, however, we wanted to benefit from Lambda's faster scale out.

We also saw a few instances where a single Fargate container was running slowly for a minute or so. This had an effect on multiple requests at the same time, but since Lambda only processes a single request per container, it would have impacted fewer customers.

However, my first attempt at getting `sqlite` running inside Lambda with Go weren't successful.

Skip to the end if you want to copy/paste the code, or stick around for the explanation.

gosqlite-3

A popular Go library for `sqlite` is `go-sqlite3` [0], so we went with that.

[0]

However, CDK's default Go Lambda function building process doesn't work out of the box for this library, for various reasons...

CGO

Firstly, that `gosqlite-3` uses `CGO`, which means that the Go code relies on C code to function. This makes it slightly more complex to package, since the C code must also be built.

CDK's default build process disables CGO by setting the `CGO_ENABLED` environment variable to `0`.

The compilation won't throw any errors, but at runtime, when you attempt to use the library, you'll get the error:

Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub

To resolve it, you'll need to customise the build to set `CGO_ENABLED=1`.

f := awslambdago.NewGoFunction(stack, jsii.String("sqlite-function"), &awslambdago.GoFunctionProps{
	Architecture: awslambda.Architecture_ARM_64(),
	LogRetention: awslogs.RetentionDays_ONE_WEEK,
	MemorySize:   jsii.Number(1024),
	Timeout:      awscdk.Duration_Seconds(jsii.Number(15)),
	Entry:        jsii.String("function"),
	Runtime:      awslambda.Runtime_PROVIDED_AL2(),
	Bundling: &awslambdago.BundlingOptions{
		CgoEnabled: jsii.Bool(true),
	},
})

Cross-compilation problems

CDK's build process also attempts to cross-compile from your local machine to Linux (`GOOS=linux`) if you have the `go` commands installed.

I got some complaints relating to Linux cross compilation at the C side of things.

# runtime/cgo
linux_syscall.c:67:13: error: implicit declaration of function 'setresgid' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
linux_syscall.c:67:13: note: did you mean 'setregid'?
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/unistd.h:593:6: note: 'setregid' declared here
linux_syscall.c:73:13: error: implicit declaration of function 'setresuid' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
linux_syscall.c:73:13: note: did you mean 'setreuid'?
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/unistd.h:595:6: note: 'setreuid' declared here
panic: Failed to bundle asset go-sqlite-test/sqlite-function/Code/Stage, bundle output is located at /Users/adrian/github.com/a-h/go-sqlite3-lambda/cdk.out/bundling-temp-bcc29fdcb2630f3ec194dbc9145ef91638669da99916efd14e45602452cdb9a0-error: Error: bash exited with status 2

To get around that, I tried forcing it to run in Docker by setting `ForcedDockerBundling`.

f := awslambdago.NewGoFunction(stack, jsii.String("sqlite-function"), &awslambdago.GoFunctionProps{
	Architecture: awslambda.Architecture_ARM_64(),
	LogRetention: awslogs.RetentionDays_ONE_WEEK,
	MemorySize:   jsii.Number(1024),
	Timeout:      awscdk.Duration_Seconds(jsii.Number(15)),
	Entry:        jsii.String("function"),
	Runtime:      awslambda.Runtime_PROVIDED_AL2(),
	Bundling: &awslambdago.BundlingOptions{
		CgoEnabled: jsii.Bool(true),
		ForcedDockerBundling: jsii.Bool(true),
	},
})

Docker cross-compilation

It got further, but now the compilation failed for another reason. Possibly the default CDK build image (it uses SAM's build images at the point of writing) doesn't support ARM.

Bundling asset go-sqlite-test/sqlite-function/Code/Stage...
go: downloading github.com/mattn/go-sqlite3 v1.14.16
go: downloading github.com/a-h/awsapigatewayv2handler v0.0.0-20220723235946-c45b98eb1b9e
go: downloading github.com/aws/aws-lambda-go v1.36.0
# runtime/cgo
gcc_arm64.S: Assembler messages:
gcc_arm64.S:28: Error: no such instruction: `stp x29,x30,[sp,'
gcc_arm64.S:32: Error: too many memory references for `mov'
gcc_arm64.S:34: Error: no such instruction: `stp x19,x20,[sp,'
gcc_arm64.S:37: Error: no such instruction: `stp x21,x22,[sp,'
gcc_arm64.S:40: Error: no such instruction: `stp x23,x24,[sp,'
gcc_arm64.S:43: Error: no such instruction: `stp x25,x26,[sp,'
gcc_arm64.S:46: Error: no such instruction: `stp x27,x28,[sp,'

There's an issue that talks about it at [1].

[1]

But I figured I'd use a different base Docker image instead, and decided to use the `amazonlinux:2` Docker image, since that's the Linux distributation that Lambda actually runs.

To do that, I created a `Dockerfile` and installed sqlite and golang into it.

from amazonlinux:2

RUN yum install -y sqlite sqlite-devel golang

Then updated the CDK code to use this as the build image.

f := awslambdago.NewGoFunction(stack, jsii.String("sqlite-function"), &awslambdago.GoFunctionProps{
	Architecture: awslambda.Architecture_ARM_64(),
	LogRetention: awslogs.RetentionDays_ONE_WEEK,
	MemorySize:   jsii.Number(1024),
	Timeout:      awscdk.Duration_Seconds(jsii.Number(15)),
	Entry:        jsii.String("function"),
	Runtime:      awslambda.Runtime_PROVIDED_AL2(),
	Bundling: &awslambdago.BundlingOptions{
		CgoEnabled: jsii.Bool(true),
		DockerImage:          awscdk.DockerImage_FromBuild(jsii.String("function"), nil),
		ForcedDockerBundling: jsii.Bool(true),
	},
})

Could not create module cache

Next, I got the error:

go: could not create module cache: mkdir /go: permission denied

This was a head scratcher. By using the `CommandHooks` inside the CDK construct, I was able to find out that the user that CDK uses to run the build didn't have a `$HOME` directory, and had no permission to write to `/`.

I tried a few things to get around it, like creating the directories outside of the CDK build, before realising that `/tmp` was probably OK to be written to.

In Go, it's possible to set the `GOMODCACHE` and `GOCACHE` environment variables to control where Go writes to during builds, so I set those in the CDK construct.

Success

The final CDK Go Function configuration ended up as this:

f := awslambdago.NewGoFunction(stack, jsii.String("sqlite-function"), &awslambdago.GoFunctionProps{
	Architecture: awslambda.Architecture_ARM_64(),
	LogRetention: awslogs.RetentionDays_ONE_WEEK,
	MemorySize:   jsii.Number(1024),
	Timeout:      awscdk.Duration_Seconds(jsii.Number(15)),
	Entry:        jsii.String("function"),
	Runtime:      awslambda.Runtime_PROVIDED_AL2(),
	Bundling: &awslambdago.BundlingOptions{
		CgoEnabled: jsii.Bool(true),
		DockerImage:          awscdk.DockerImage_FromBuild(jsii.String("function"), nil),
		ForcedDockerBundling: jsii.Bool(true),
		Environment: &map[string]*string{
			"GOMODCACHE": jsii.String("/tmp/"),
			"GOCACHE":    jsii.String("/tmp/"),
		},
	},
})

Bear in mind that I use an ARM64 Mac, and I use ARM64 as my default architecture for Lambda functions, so you might need to switch architecture by removing the `Architecture` setting from the CDK configuration.

A complete working example is available at [2]:

[2]

DynamoDB Stream to Lambda Filtering With Go CDK

Alerting on AWS Security Hub notifications with OpsGenie