This post will be more like a tutorial about how to use Google Datastore emulator and Google Pub/Sub emulator for testing. And how to make it possible to run multiple tests in parallel by using Docker containers for each emulator.
TLDR: full working code snippet can be found here.
`docker ps` while running integration tests
Motivation
I’m not a big fan of mocking things around. Or having different implementations of something for unit tests and production. What’s the point of using an in-memory DB for unit tests that will have some different behavior in the most critical moment!? Plus an overhead of maintaining two implementations for testing and production. I personally can’t be 100% confident in the code I deploy after such testing. I’d like to test my application in an environment as similar as possible to production.
Recently I’ve been really into Google Cloud Platform. At the moment I’m building something that consist of huge amount of micro-services written in Kotlin which are using GRPC for communication. All internal services are living inside Kubernetes and the only external services are some GCP products like Datastore and Pub/Sub.
If for testing communication between internal service I can use in-process channels for GRPC which can be a good theme for another post. I faced an issue what to use for services like Datastore, Pub/Sub and Blobstore in my tests.
Implementation
In the very beginning I was simply running emulators in my terminal tabs with commands like:
gcloud beta emulators datastore start --no-legacy \ --project unit-testing-project-name \ --consistency=1
I even found not very well documented http://localhost:<port>/resetendpoint that resets an emulator to a clean state. I was calling it before every test to make sure an emulator is clean before running a test. It worked quite well for Datastore emulator but Pub/Sub emulator wasn’t that reliable and sometime spuriously was erroring. And it was getting worst with growing amount of tests. Pub/Sub emulator was erroring more and more frequently while running tests. Another problem was that I was limited to run tests sequentially. There was no way to share an emulator between tests! The only way was to run multiple instances of emulators with different --host-port flag.
Around that time I learned about an awesome project called Test Containers. The idea is simple: provide clean API to use Docker containers in tests.
And I though why not to use it and just start new containers with necessary emulators for each test!? Yes, it will be slower, but it proved to be more reliable especially for Pub/Sub emulator. And I believe in the long term it will scale much better. I’ve also splitted the project in 60+ Gradle subprojects. With parallel execution and Gradle caching test task it didn’t affect local workflow that much.
In order to start using emulators in tests first we need a container with Google Cloud SDK:
class GoogleCloudContainer :
GenericContainer<GoogleCloudContainer>("google/cloud-sdk:latest")And now we simply define a field on a companion object(basically a static field in terms of Kotlin) for a test class we want to use containers in:
val projectName = "testing" val emulatorPort = 8888
@ClassRule @JvmField
public val datastoreContainer: GoogleCloudContainer =
GoogleCloudContainer()
.withExposedPorts(emulatorPort)
.withCommand("/bin/sh", "-c",
"""
gcloud beta emulators datastore start --no-legacy \
--project $projectName \
--host-port=0.0.0.0:$emulatorPort \
--consistency=1
""")Note: using localhost for --host-port is not working.
And that’s all! Datastore service can be now build to use that running container:
val datastoreService: Datastore by lazy {
val host = datastoreContainer.containerIpAddress
val port = datastoreContainer.getMappedPort(emulatorPort)
DatastoreOptions.newBuilder()
.setProjectId(projectName)
.setHost("$host:$port")
.setCredentials(NoCredentials.getInstance())
.build()
.service
}Simple as that! Full code snippet for Datastore can be found here. Pub/Sub snippet is also available here.
Killing stale containers automatically
One caveat working with Test Containers is that if you debug a test and kill a JVM then containers won’t be cleaned up. Thankfully Test Containers label all containers with org.testcontainers=true . I simply added a clean-up task to Gradle to remove any containers before running tests:
task stopTestContainers(type: Exec) {
executable "sh"
args "-c", 'docker ps -q -f "label=org.testcontainers=true" ' +
'| xargs docker rm -f'
}
test.dependsOn stopTestContainersConclusion
Test Containers are great for having something production like in tests. They proved to be very reliable. Give it a try!