Android Studio Unit Testing Support by compscigrad is licensed under CC BY-NC-SA 2.0

Kotest is a Kotlin multiplatform testing framework and assertions library. It is a powerful framework, that can completely replace the default kotlin test framework, which is based on the popular JUnit and TestNG. Furthermore, Kotest supports some more advanced features, such as property-based testing and its own assertions framework.

An interesting question may be why would anyone use kotest over the standard, popular JUnit (for example) framework? The main reasons are the following:

  • kotest has built-in support for coroutines
  • it offers its own expressive assertions library
  • it has a multitude of extensions that bring all sorts of goodies in
  • it supports many different testing styles

Configuring Kotest in your Android project

I had been using JUnit5 in one of my hobby projects, the metador library, and I wanted to replace it with kotest. The instructions found in the Android section of the kotest quickstart are pretty well written, can even guide into installing some of the most advanced features. I opted for the assertions library and the mockserver extension, in order to replace the okhttp mock webserver implementation I had been using.

Therefore, the following dependencies were added in the module’s build.gradle:

    testImplementation "io.kotest:kotest-runner-junit5:$kotestVersion"
    testImplementation "io.kotest:kotest-assertions-core:$kotestVersion"
    testImplementation "io.kotest.extensions:kotest-extensions-mockserver:1.0.0"

together with a testOptions in the android section of the module’s build gradle:

    testOptions {
        unitTests.all {
            useJUnitPlatform()
        }
    }

to instruct Kotest to use the underlying junit platform.

Choosing a Kotest test style - Using Behavior Specs

This post discusses the use of the Behaviour Spec testing style, one of Kotest’s many supported styles. Its greatest advanced is that it enforces an order in the tests, and makes the test output readable and almost documentation-like. It is the tools used for Behaviour Driven Development. The tests follow a specific scenario-driven narrative using the following structure:

Given: the initial context at the beginning of the scenario, in one or more clauses;
When: the event that triggers the scenario;
Then: the expected outcome, in one or more clauses.

Kotest offers the Given, And, When, Then functions that support the above structure. These functions are used in a nested fashion (And has been added to offer an additional level of nesting if required). There are also corresponding lowercase functions, but since when is a Kotlin-reserved keyword and requires backticks, the uppercase functions seem to be a better alternative.

The nesting functionality of the BehaviorSpec impacts how certain features have been implemented. If you’ve been using JUnit (4 or 5), then you should be familiar with the @Before, @After, @BeforeEach, @AfterEach annotations. They are used for marking specific functions to be called before and after the unit tests. Kotest offers similar functionality using before and after prefixed function blocks. These functions are:

  • beforeAny/afterAny - executed before/after any container or a test (any of the Given, When, And, Then functions). Any such block will be executed before/after the function that follows
  • beforeEach/afterEach - executed before/after a test case (the Then function). Any such block will be executed before/after the test case that follows
  • beforeContainer/afterContainer - executed before/after a container (one of the Given, When, And functions). Any such block will be executed before/after the container that follows
  • beforeTest/afterTest - executed before/after a test case (the Then function). Any such block will be executed before/after the test case that follows
  • beforeSpec/afterSpec - executed before/after each spec (e.g. after the execution of a Spec subclass)

To better understand how the before/after functions work (and when they are executed), you can have a go at running the following example BehaviorSpec, which includes all the before/after functions at various levels:

class ContainerExampleTest : BehaviorSpec({
    beforeContainer {
        println("Inside the root beforeContainer")
    }
    afterContainer {
        println("After the root beforeContainer")
    }
    println("Before the root Given")

    beforeSpec {
        println("Inside the root beforeSpec")
    }

    afterSpec {
        println("Inside the root afterSpec")
    }

    beforeAny {
        println("Inside the root beforeAny")
    }

    afterAny {
        println("Inside the root afterAny")
    }

    beforeEach {
        println("Inside the root beforeEach")
    }

    afterEach {
        println("Inside the root afterEach")
    }

    Given("a String") {
        val aString = "asdf"
        beforeContainer {
            println("Inside the beforeContainer for: And the String is 4 characters long")
        }
        afterContainer {
            println("Inside the afterContainer for: And the String is 4 characters long")
        }
        println("Inside the: Given a String")

        And("the String is 4 characters long") {
            beforeContainer {
                println("Inside the beforeContainer for: And the toUpperCase method is called")
            }
            afterContainer {
                println("Inside the afterContainer for: And the toUpperCase method is called")
            }
            println("Inside the: And the String is 4 characters long")

            When("the toUpperCase method is called") {
                beforeTest {
                    println("Inside the beforeTest for: Then it returns the string in all caps")
                }
                afterTest {
                    println("Inside the afterTest for:  Then it returns the string in all caps")
                }
                println("Inside the: When the toUpperCase method is called")
                val result = aString.toUpperCase()
                Then("it returns the string in all caps") {
                    println("Inside the: Then it returns the string in all caps")

                    result shouldBe "ASDF"
                }
            }
            When("length property is called") {
                beforeTest {
                    println("Inside the beforeTest for: Then it returns 4")
                }
                afterTest {
                    println("Inside the afterTest for: Then it returns 4")
                }
                println("Inside the: When length property is called")

                val result = aString.length
                Then("it returns 4") {
                    println("Inside the: Then it returns 4")
                    result shouldBe 4
                }
            }
        }
    }
})

Running the above test produces the following output:


Before the root Given
Inside the root beforeSpec

Inside the root beforeContainer
Inside the root beforeAny

Inside the: Given a String

Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the: And the String is 4 characters long

Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the beforeContainer for: And the toUpperCase method is called
Inside the: When the toUpperCase method is called

Inside the root beforeAny
Inside the root beforeEach
Inside the beforeTest for: Then it returns the string in all caps
Inside the: Then it returns the string in all caps
Inside the root afterAny
Inside the root afterEach
Inside the afterTest for:  Then it returns the string in all caps

After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long
Inside the afterContainer for: And the toUpperCase method is called


Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the beforeContainer for: And the toUpperCase method is called
Inside the: When length property is called

Inside the root beforeAny
Inside the root beforeEach
Inside the beforeTest for: Then it returns 4
Inside the: Then it returns 4
Inside the root afterAny
Inside the root afterEach
Inside the afterTest for: Then it returns 4

After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long
Inside the afterContainer for: And the toUpperCase method is called

After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long

After the root beforeContainer
Inside the root afterAny

Inside the root afterSpec

In any container, where you would want to execute the same code before each of the following containers of the same level, you would want to use a beforeContainer function. Be careful with the nesting of the beforeContainer and the afterContainer functions. All beforeContainer methods found in the tree leading towards the root node will be executed, as can be seen by the preceeding exaxmple. To group code that will be executed before the actual test cases (the Then function calls) use beforeTest and afterTest. These two sets of functions should cover the majority of your needs, in regards to executing code before and after tests.

Migrating existing tests

The process of migrating existing unit tests to Kotest is pretty straight forward. Just select the testing style that works best for you, and apply it to all of your unit tests. Migrating to the BehaviorSpec style however, requires a bit more work especially in the beginning, in order to figure out how Kotest works and what is the approach to use.

If you’ve been writing tests for some time, you should have heard about the simple arrange, act, assert pattern. This is a very simplistic “design pattern” which suggests that you should be splitting your unit tests in three distinct parts:

  • the arrange part: contains all statements required for setting up the test (e.g. setting the internal state of the test object and its dependencies)
  • the act part: contains all the statements required for executing the test. This should be limited to a few statements (usually no more than one or two)
  • the assert part: contains all the statements required for verifying the result of the test

Tests adhering to the above pattern can be easily converted. The structure of a BehaviorSpec test usually consists of one or more Given function calls, which contain one or more And or When nested function calls. And function calls may include nested When or Then function calls. When function calls may in turn contain And or Then function calls. Finally, Then function calls do not contain any other nested functions - they are at the bottom of the chain. Thus the following are valid examples of nesting function calls in BehaviorSpec tests:

Given {
    When {
        Then {

        }
    }
}
Given {
    And {
        When {
            Then {

            }
        }
    }
}
Given {
    And {
        Then {

        }
    }
}
Given {
    And {
        When {
            And {
                Then {

                }
            }
        }
    }
}

Then blocks are always the inner-most nested elements, and are thus the natural containers of all the assert-related statements. All act-related statements must be placed in the exact preceding And or When block of the corresponding Then. Thus, arrange-related statements will be placed inside Given, When or And blocks, which must precede the act-related and assert-related statements. Because of the nesting implementation of BehaviorSpec tests, and because we want to avoid duplicating the initialisation parts of our tests, we should be placing our arrange-related statements inside beforeContainer blocks, such that they are executed before each of the nested When, And blocks.

Basic example

Let’s have a look at a few actual examples of unit tests converted to BehaviorSpec. The code (and conversion history) can be found here. The example below presents a very basic unit test. The class to be tested, does not use any external dependencies (hence no mocking is required) and has only a single function that is exposed with very simple signature.

@ExperimentalCoroutinesApi
class HtmlMetaExtractorTest {
    lateinit var htmlMetaExtractorInTest: HtmlMetaExtractor

    @BeforeEach
    fun setup() {
        htmlMetaExtractorInTest = HtmlMetaExtractor()
    }

    @Test
    fun `empty string inputs, produce a success with an empty map`() = runBlockingTest {
        val response = htmlMetaExtractorInTest.parseResource("")

        assertThat(response, _is(emptyMap()))
    }

    @Test
    fun `an HTML input where the meta key attribute cannot be determined, produces an empty map`() =
        runBlockingTest {
            val response =
                htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_UNKNOWN_ATTRIBUTE_META)

            assertThat(response, _is(emptyMap()))
        }

    @Test
    fun `an HTML input with all supported the meta key attributes, produces a map with entries for each meta element`() =
        runBlockingTest {
            val response =
                htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_ALL_SUPPORTED_TYPES_META)

            assertThat(
                response,
                _is(
                    mapOf(
                        "key-1" to "value",
                        "key-2" to "value",
                        "key-3" to "value"
                    )

                )
            )
        }
}

Let’s dissect the code of the above unit test class, and discuss how it can be converted to BehaviorSpec. First of all, when porting our code to the Kotest framework, we stop worrying about coroutines - no need for @ExperimentalCoroutinesApi and for littering all our test cases with runBlocking, as it supports coroutines natively. We may need to use the TestCoroutineDispatcher, if we require passing the coroutine dispatcher as a parameter to on of our method calls.

Next, (and since the code will be ported to BehaviorSpec) the @BeforeEach setup function will be replaced with an appropriate beforeContainer method call, which will still be responsible for reinitialising the object in test, before the test is executed. These tests have a merged arrange-act section, which will be directly moved inside a When block. The assertions will be moved inside a Then block, and be replaced with the appropriate Kotest assertions.

The converted test is the following:

import io.kotest.core.spec.style.BehaviorSpec
import io.kotest.matchers.shouldBe

class HtmlMetaExtractorTest : BehaviorSpec({
    lateinit var htmlMetaExtractorInTest: HtmlMetaExtractor

    beforeContainer {
        htmlMetaExtractorInTest = HtmlMetaExtractor()
    }

    Given("an HtmlMetaExtractor") {
        When("it attempts to parse an empty string resource") {
            val response = htmlMetaExtractorInTest.parseResource("")

            Then("it returns an empty map") {
                response shouldBe emptyMap()
            }
        }

        When("the HTML input contains an unknown META key attribute") {
            val response =
                htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_UNKNOWN_ATTRIBUTE_META)

            Then("it returns an empty map") {
                response shouldBe emptyMap()
            }
        }

        When("the HTML input contains supported META key attributes") {
            val response =
                htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_ALL_SUPPORTED_TYPES_META)
            Then("it returns a map containing all META key-value pairs") {
                response shouldBe mapOf("key-1" to "value", "key-2" to "value", "key-3" to "value")
            }
        }
    }
})

A more advanced example

A more complex example unit test conversion is presented below:


@ExperimentalCoroutinesApi
class ResourceParserTest {
    @MockK
    lateinit var mockResourceParserDelegate: ResourceParserDelegate

    private lateinit var objectInTest: ResourceParser

    private val testCoroutineDispatcher = TestCoroutineDispatcher()

    @BeforeEach
    fun setup() {
        MockKAnnotations.init(this)
        objectInTest = ResourceParser(testCoroutineDispatcher)
    }

    @Test
    fun `the resource parser delegates it's work to the supplied ResourceParserDelegate and returns the result to the caller`() =
        runBlockingTest {
            val fakeResource = "fake_resource"
            val expectedResult = mapOf("key" to "value")
            every { mockResourceParserDelegate.parseResource(fakeResource) } returns expectedResult

            val result = objectInTest.parseResource(mockResourceParserDelegate, fakeResource)

            verify { mockResourceParserDelegate.parseResource(fakeResource) }
            assertThat(result, _is(expectedResult))
        }

    @Test
    fun `exceptions thrown by the ResourceParserDelegate are propagated to the caller`() =
        runBlockingTest {
            val fakeResource = "fake_resource"
            every { mockResourceParserDelegate.parseResource(fakeResource) } throws RuntimeException(
                "Error"
            )

            assertThrows<RuntimeException> {
                objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
            }
        }
}

This test requires a few more advanced features. Firstly, it requires the use of the TestCoroutineDispatcher - because it has an external dependency on the CoroutineDispatcher to be used for executing the code. It also requires using a mock to establish that a particular scenario leads to executing a specific method of the mock. Finally, it makes use of an assertion that determines whether a specific code path throws an exception and how that is handled.

A potential conversion of the tests shown above, to the BehaviorSpec in Kotest is the following:

class ResourceParserTest : BehaviorSpec({
    lateinit var mockResourceParserDelegate: ResourceParserDelegate
    lateinit var objectInTest: ResourceParser
    val testCoroutineDispatcher = TestCoroutineDispatcher()

    beforeContainer {
        mockResourceParserDelegate = mockk()
        objectInTest = ResourceParser(testCoroutineDispatcher)
    }

    Given("a ResourceParser") {
        val fakeResource = "fake_resource"
        When("it is requested to parse a resource") {
            val expectedResult = mapOf("key" to "value")
            every { mockResourceParserDelegate.parseResource(fakeResource) } returns expectedResult

            val result = objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
            Then("it produces the result by delegating its work to the supplied ResourceParserDelegate") {
                verify { mockResourceParserDelegate.parseResource(fakeResource) }
                result shouldBe expectedResult
            }
        }

        When("an exception is thrown by the ResourceParserDelegate") {
            every { mockResourceParserDelegate.parseResource(fakeResource) } throws RuntimeException(
                "Error"
            )
            Then("it is propagated to the caller") {
                shouldThrow<RuntimeException> {
                    objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
                }
            }
        }
    }
})

In the Kotest version, you can no longer make use of annotations for initialising your mocks (as supported by popular mocking libraries, such as mockk and mockito). This is because the test is written in a lambda that is passed as a parameter in the BehaviorSpec constructor.Therefore, annoatations cannot be used for this purpose (at least with mockk, although this will also be the case for other libraries as well). The object in test is initialised before any containers are initialised (using a beforeContainer block), as in the previous case. The two tests share no common initialisation code (apart from a common variable), and there is no need to add another beforeContainer before the two When blocks. The arrange-related statements are placed at the beginning of the When blocks. In the first test, this is followed by the act-related statement which also stores the result, that is used inside the assert-related statements placed in the Then block. In the second test, the act-related and the assert-related statements are joined in a single shouldThrow block, because it tests whether an exception is thrown.

CI-support

Kotest integrates very well with Intellij (and Android Studio) IDE, via its official plugin. However, since its structure is a radical departure from the standard JUnit format, the default JUnit report generated by the Gradle test plugin becomes unusable (the test count and the test names are wrong). This may be irrelevant when running your tests locally - it completely breaks potential (or current) CI integrations you may have. To overcome this limitation, the Kotest framework offers the JUnit XML extension, which writes a separate report file that correctly reports the tests executed and their results.

To set it up, you need to add the following line to your module’s build gradle ($kotestVersion refers to the kotest version you use in your project):

testImplementation "io.kotest:kotest-extensions-junitxml:$kotestVersion"

To use the extension, you will need to create a subclass (either a class or object will do) of the AbstractProjectConfig class, which is automatically used by Kotest to configure the overall test project. You can see all available configuration options here. In the subclass, you override the listeners function such that it returns a list with a new instance of the JunitXmlReporter. The subclass can reside anywhere in your test source set - Kotest will find and use it. An example is the following:

object MyKotestProjectConfig : AbstractProjectConfig() {
    override fun listeners(): List<Listener> = listOf(
        JunitXmlReporter(
            includeContainers = false,
            useTestPathAsName = true
        )
    )
}

The constructor of the JunitXmlReporter takes two parameters:

  • includeContainers - a boolean which indicates whether the XML report will include intermediate containers as separate line entries, not just Then blocks. Defaults to false. When set to true, it artificially increases the number of tests included (without much benefit).
  • useTestPathAsName - a boolean which indicates whether the name of the test in the report will include the complete path (the sequence of Given, When, And, Then) of the test, or just the final Then block. This defaults to true. When set to false, all test names will only include the leaf-Then blocks names.

After running your tests, the correct JUnit XML reports (and only those, not the HTML report) will be found in your module’s build directory, under the test-results/test subfolder. Note that the default, incorrect XML version of the test results will also be available under the test-results/testXYZUnitText (XYZ being the combination of flavor/build type used in running the tests).

Example code demonstrating the usage of the JUnit XML extension can be found here.