Using DynamoDB in Go Unit Tests

by: tobias
2021-07-28 2:00pm tech learnings

If you’re like me, you’re really into all of the great technologies that our overlords at the US west coast give us. Of course, I’m talking about Google and Amazon, who have graciously provided us with the Go programming language and Amazon Web Services, respectively.

In our research project, we use mostly Go code to implement our fog-native data management solution. To work with bleeding edge tech in the cloud, we also provide an interface to work with DynamoDB, AWS' popular NoSQL Database-as-a-Service.

When I write code, it is often full of bugs and unwanted side-effects, let alone technical debt¹. That’s why I use tests to ~~shut the voices in my head that constantly tell me how terrible my code is~~ prove new implementations before I merge them into our codebase.

In Go, writing tests is super easy² because it’s part of the standard toolchain. Let’s say you have a simple Go package like this:

package hello

// Greet says hello to you.
func Greet(name string) string {
    return "Hello " + name + "!"
}

All it does is take a name and say hello to that name. How nice! To write tests for that package, simply create a file hello_test.go and write down your tests:

package hello

import (
    "testing"
)

func TestGreet(t *testing.T) {
    out := Greet("Theresa")

    if out != "Hello Theresa!" {
        t.Fatalf("output \"%s\" is wrong!", out)
    }
}

No you just run go test and you’re done:

$ go test
PASS
ok      theresa-tobias.website/hello    0.011s

Easy as that! But what if your package has external dependencies? Namely, what if your package requires access to a cloud database such as DynamoDB?

package hello

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/service/dynamodb"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbiface"
)

// GreetKeys says hello to your database table keys.
func GreetKeys(table string, svc dynamodbiface.DynamoDBAPI) (string, error) {

    desc, err := svc.DescribeTable(&dynamodb.DescribeTableInput{
        TableName: aws.String(table),
    })

    if err != nil {
        return "", nil
    }

    return "Hello " + *(desc.Table.KeySchema[0].AttributeName) + "!", nil
}

The main issue is that in your local unit tests, you don’t want to rely on remote databases. You need something local.

One option is to use the go-dynamock library that provides a mocked version of DynamoDB. This is great but has two issues:

it doesn’t cover all of the DynamoDB interface: e.g., TTL configuration is missing (which, by the way, is a deal-breaker for us)
it uses these weird Expectations where you need to say in your tests what will happen to Dynamo before your code does it: this means your tests will need to be adapted when you change code (which is not a good practice)

So that’s out of the question. Thankfully, there is another way: using a local instance of DynamoDB in a Docker container. You can deploy DynamoDB locally on your computer with Docker, which is nice.³ Thanks, Amazon!

So all we have to do is start up a local copy of DynamoDB, point our tests to that local endpoint, and start? Yes! But also, no.

You see, that would require all of our developers, contributors, and our CI to perform additional steps before we can run go test (which is all we want anyway). What if there was a way to have DynamoDB start from within our Go tests?

That’s where testcontainers-go comes in⁴ This library wraps the Docker Go SDK to provide a simple way to start up containers for your tests. All we have to do is to run some additional setup in our TestMain function that runs before every test to create the DynamoDB instance and the tables or data we need:

package hello

import (
    "context"
    "fmt"
    "os"
    "testing"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/dynamodb"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbiface"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/wait"
)

var svc dynamodbiface.DynamoDBAPI

// the TestMain functions runs before any test we execute
func TestMain(m *testing.M) {
    // get a context for our request
    ctx := context.Background()
    // create a container request for DynamoDB
    req := testcontainers.ContainerRequest{
        // we're using the latest version of the image provided by Amazon
        Image: "amazon/dynamodb-local:latest",
        // be sure to use the commands as described in the documentation, but
        // an in-memory version is good enough for us
        Cmd: []string{"-jar", "DynamoDBLocal.jar", "-inMemory"},
        // by default, DynamoDB runs on port 8000
        ExposedPorts: []string{"8000/tcp"},
        // testcontainers let's us block until the port is available, i.e.,
        // DynamoDB has started
        WaitingFor: wait.NewHostPortStrategy("8000"),
    }

    // let's start the container!
    d, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: req,
        Started:          true,
    })

    if err != nil {
        panic(err)
    }

    // be sure to stop the container before we're done!
    defer d.Terminate(ctx)

    // now all we need is the IP and port of our DynamoDB instance to connect
    // to the right endpoints
    ip, err := d.Host(ctx)

    if err != nil {
        panic(err)
    }

    port, err := d.MappedPort(ctx, "8000")

    if err != nil {
        panic(err)
    }

    // create a new session with our custom endpoint
    // we need to specify a region, otherwise we get a fatal error
    sess := session.Must(session.NewSession(&aws.Config{
        Endpoint: aws.String(fmt.Sprintf("http://%s:%s", ip, port)),
        Region:   aws.String("eu-central-1"),
    }))

    // and now we have our service!
    svc = dynamodb.New(sess)

    // now we just need to tell go-test that we can run the tests
    os.Exit(m.Run())
}

func TestGreetKeys(t *testing.T) {
    // in this test, we first create a new table with a key, then see if the
    // greeting works

    table := "greetingtable"

    _, err := svc.CreateTable(&dynamodb.CreateTableInput{
        AttributeDefinitions: []*dynamodb.AttributeDefinition{
            {
                // by the way: note how aws uses custom strings?!
                AttributeName: aws.String("theresa"),
                AttributeType: aws.String("S"),
            },
        },
        BillingMode:            nil,
        GlobalSecondaryIndexes: nil,
        KeySchema: []*dynamodb.KeySchemaElement{
            {
                AttributeName: aws.String("theresa"),
                KeyType:       aws.String(dynamodb.KeyTypeHash),
            },
        },
        LocalSecondaryIndexes: nil,
        ProvisionedThroughput: &dynamodb.ProvisionedThroughput{
            ReadCapacityUnits:  aws.Int64(1),
            WriteCapacityUnits: aws.Int64(1),
        },
        SSESpecification:    nil,
        StreamSpecification: nil,
        TableName:           aws.String(table),
        Tags:                nil,
    })

    if err != nil {
        t.Fatalf("could not create table: %s", err.Error())
    }

    out, err := GreetKeys(table, svc)

    if err != nil {
        t.Fatalf("could not greet table: %s", err.Error())
    }

    if out != "Hello theresa!" {
        t.Fatalf("output \"%s\" is wrong!", out)
    }
}

Sure, it’s a bit long, but it works! Now all we have to do is run go test to see how it works:

$ go test
2021/07/28 15:24:48 Starting container id: 61ed04672831 image: quay.io/testcontainers/ryuk:0.2.3
2021/07/28 15:24:49 Waiting for container id 61ed04672831 image: quay.io/testcontainers/ryuk:0.2.3
2021/07/28 15:24:50 Container is ready id: 61ed04672831 image: quay.io/testcontainers/ryuk:0.2.3
2021/07/28 15:24:50 Starting container id: 2f7fdba07467 image: amazon/dynamodb-local:latest
2021/07/28 15:24:51 Waiting for container id 2f7fdba07467 image: amazon/dynamodb-local:latest
2021/07/28 15:24:58 Container is ready id: 2f7fdba07467 image: amazon/dynamodb-local:latest
PASS
ok      theresa-tobias.website/hello    11.662s

Of course, this takes significantly longer. But it just works.

but who cares, it’s a prototype, proof-of-concept, research implementation anyway ↩︎
unlike in, say, Java ↩︎
Unfortunately, it is written in Java… ↩︎
also based on a previous Java implementation… ↩︎