15 minutes
Using kotest in your Android project
Kotest is a Kotlin multiplatform testing framework and assertions library. It is a powerful framework, that can completely replace the default kotlin test framework, which is based on the popular JUnit and TestNG. Furthermore, Kotest supports some more advanced features, such as property-based testing and its own assertions framework.
An interesting question may be why would anyone use kotest over the standard, popular JUnit (for example) framework? The main reasons are the following:
- kotest has built-in support for coroutines
- it offers its own expressive assertions library
- it has a multitude of extensions that bring all sorts of goodies in
- it supports many different testing styles
Configuring Kotest in your Android project
I had been using JUnit5 in one of my hobby projects, the metador library, and I wanted to replace it with kotest. The instructions found in the Android section of the kotest quickstart are pretty well written, can even guide into installing some of the most advanced features. I opted for the assertions library and the mockserver extension, in order to replace the okhttp mock webserver implementation I had been using.
Therefore, the following dependencies were added in the module’s build.gradle:
testImplementation "io.kotest:kotest-runner-junit5:$kotestVersion"
testImplementation "io.kotest:kotest-assertions-core:$kotestVersion"
testImplementation "io.kotest.extensions:kotest-extensions-mockserver:1.0.0"
together with a testOptions
in the android
section of the module’s build gradle:
testOptions {
unitTests.all {
useJUnitPlatform()
}
}
to instruct Kotest to use the underlying junit platform.
Choosing a Kotest test style - Using Behavior Specs
This post discusses the use of the Behaviour Spec testing style, one of Kotest’s many supported styles. Its greatest advanced is that it enforces an order in the tests, and makes the test output readable and almost documentation-like. It is the tools used for Behaviour Driven Development. The tests follow a specific scenario-driven narrative using the following structure:
Given: the initial context at the beginning of the scenario, in one or more clauses;
When: the event that triggers the scenario;
Then: the expected outcome, in one or more clauses.
Kotest offers the Given
, And
, When
, Then
functions that support the above structure. These functions are used in a nested fashion (And
has been added to offer an additional level of nesting if required). There are also corresponding lowercase functions, but since when
is a Kotlin-reserved keyword and requires backticks, the uppercase functions seem to be a better alternative.
The nesting functionality of the BehaviorSpec impacts how certain features have been implemented. If you’ve been using JUnit (4 or 5), then you should be familiar with the @Before, @After, @BeforeEach, @AfterEach annotations. They are used for marking specific functions to be called before and after the unit tests. Kotest offers similar functionality using before
and after
prefixed function blocks. These functions are:
- beforeAny/afterAny - executed before/after any container or a test (any of the
Given
,When
,And
,Then
functions). Any such block will be executed before/after the function that follows - beforeEach/afterEach - executed before/after a test case (the
Then
function). Any such block will be executed before/after the test case that follows - beforeContainer/afterContainer - executed before/after a container (one of the
Given
,When
,And
functions). Any such block will be executed before/after the container that follows - beforeTest/afterTest - executed before/after a test case (the
Then
function). Any such block will be executed before/after the test case that follows - beforeSpec/afterSpec - executed before/after each spec (e.g. after the execution of a Spec subclass)
To better understand how the before/after functions work (and when they are executed), you can have a go at running the following example BehaviorSpec, which includes all the before/after functions at various levels:
class ContainerExampleTest : BehaviorSpec({
beforeContainer {
println("Inside the root beforeContainer")
}
afterContainer {
println("After the root beforeContainer")
}
println("Before the root Given")
beforeSpec {
println("Inside the root beforeSpec")
}
afterSpec {
println("Inside the root afterSpec")
}
beforeAny {
println("Inside the root beforeAny")
}
afterAny {
println("Inside the root afterAny")
}
beforeEach {
println("Inside the root beforeEach")
}
afterEach {
println("Inside the root afterEach")
}
Given("a String") {
val aString = "asdf"
beforeContainer {
println("Inside the beforeContainer for: And the String is 4 characters long")
}
afterContainer {
println("Inside the afterContainer for: And the String is 4 characters long")
}
println("Inside the: Given a String")
And("the String is 4 characters long") {
beforeContainer {
println("Inside the beforeContainer for: And the toUpperCase method is called")
}
afterContainer {
println("Inside the afterContainer for: And the toUpperCase method is called")
}
println("Inside the: And the String is 4 characters long")
When("the toUpperCase method is called") {
beforeTest {
println("Inside the beforeTest for: Then it returns the string in all caps")
}
afterTest {
println("Inside the afterTest for: Then it returns the string in all caps")
}
println("Inside the: When the toUpperCase method is called")
val result = aString.toUpperCase()
Then("it returns the string in all caps") {
println("Inside the: Then it returns the string in all caps")
result shouldBe "ASDF"
}
}
When("length property is called") {
beforeTest {
println("Inside the beforeTest for: Then it returns 4")
}
afterTest {
println("Inside the afterTest for: Then it returns 4")
}
println("Inside the: When length property is called")
val result = aString.length
Then("it returns 4") {
println("Inside the: Then it returns 4")
result shouldBe 4
}
}
}
}
})
Running the above test produces the following output:
Before the root Given
Inside the root beforeSpec
Inside the root beforeContainer
Inside the root beforeAny
Inside the: Given a String
Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the: And the String is 4 characters long
Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the beforeContainer for: And the toUpperCase method is called
Inside the: When the toUpperCase method is called
Inside the root beforeAny
Inside the root beforeEach
Inside the beforeTest for: Then it returns the string in all caps
Inside the: Then it returns the string in all caps
Inside the root afterAny
Inside the root afterEach
Inside the afterTest for: Then it returns the string in all caps
After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long
Inside the afterContainer for: And the toUpperCase method is called
Inside the root beforeContainer
Inside the root beforeAny
Inside the beforeContainer for: And the String is 4 characters long
Inside the beforeContainer for: And the toUpperCase method is called
Inside the: When length property is called
Inside the root beforeAny
Inside the root beforeEach
Inside the beforeTest for: Then it returns 4
Inside the: Then it returns 4
Inside the root afterAny
Inside the root afterEach
Inside the afterTest for: Then it returns 4
After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long
Inside the afterContainer for: And the toUpperCase method is called
After the root beforeContainer
Inside the root afterAny
Inside the afterContainer for: And the String is 4 characters long
After the root beforeContainer
Inside the root afterAny
Inside the root afterSpec
In any container, where you would want to execute the same code before each of the following containers of the same level, you would want to use a beforeContainer
function. Be careful with the nesting of the beforeContainer
and the afterContainer
functions. All beforeContainer
methods found in the tree leading towards the root node will be executed, as can be seen by the preceeding exaxmple. To group code that will be executed before the actual test cases (the Then
function calls) use beforeTest
and afterTest
. These two sets of functions should cover the majority of your needs, in regards to executing code before and after tests.
Migrating existing tests
The process of migrating existing unit tests to Kotest is pretty straight forward. Just select the testing style that works best for you, and apply it to all of your unit tests. Migrating to the BehaviorSpec style however, requires a bit more work especially in the beginning, in order to figure out how Kotest works and what is the approach to use.
If you’ve been writing tests for some time, you should have heard about the simple arrange, act, assert pattern. This is a very simplistic “design pattern” which suggests that you should be splitting your unit tests in three distinct parts:
- the arrange part: contains all statements required for setting up the test (e.g. setting the internal state of the test object and its dependencies)
- the act part: contains all the statements required for executing the test. This should be limited to a few statements (usually no more than one or two)
- the assert part: contains all the statements required for verifying the result of the test
Tests adhering to the above pattern can be easily converted. The structure of a BehaviorSpec test usually consists of one or more Given
function calls, which contain one or more And
or When
nested function calls. And
function calls may include nested When
or Then
function calls. When
function calls may in turn contain And
or Then
function calls. Finally, Then
function calls do not contain any other nested functions - they are at the bottom of the chain. Thus the following are valid examples of nesting function calls in BehaviorSpec tests:
Given {
When {
Then {
}
}
}
Given {
And {
When {
Then {
}
}
}
}
Given {
And {
Then {
}
}
}
Given {
And {
When {
And {
Then {
}
}
}
}
}
Then
blocks are always the inner-most nested elements, and are thus the natural containers of all the assert-related statements. All act-related statements must be placed in the exact preceding And
or When
block of the corresponding Then
. Thus, arrange-related statements will be placed inside Given
, When
or And
blocks, which must precede the act-related and assert-related statements. Because of the nesting implementation of BehaviorSpec tests, and because we want to avoid duplicating the initialisation parts of our tests, we should be placing our arrange-related statements inside beforeContainer
blocks, such that they are executed before each of the nested When
, And
blocks.
Basic example
Let’s have a look at a few actual examples of unit tests converted to BehaviorSpec. The code (and conversion history) can be found here. The example below presents a very basic unit test. The class to be tested, does not use any external dependencies (hence no mocking is required) and has only a single function that is exposed with very simple signature.
@ExperimentalCoroutinesApi
class HtmlMetaExtractorTest {
lateinit var htmlMetaExtractorInTest: HtmlMetaExtractor
@BeforeEach
fun setup() {
htmlMetaExtractorInTest = HtmlMetaExtractor()
}
@Test
fun `empty string inputs, produce a success with an empty map`() = runBlockingTest {
val response = htmlMetaExtractorInTest.parseResource("")
assertThat(response, _is(emptyMap()))
}
@Test
fun `an HTML input where the meta key attribute cannot be determined, produces an empty map`() =
runBlockingTest {
val response =
htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_UNKNOWN_ATTRIBUTE_META)
assertThat(response, _is(emptyMap()))
}
@Test
fun `an HTML input with all supported the meta key attributes, produces a map with entries for each meta element`() =
runBlockingTest {
val response =
htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_ALL_SUPPORTED_TYPES_META)
assertThat(
response,
_is(
mapOf(
"key-1" to "value",
"key-2" to "value",
"key-3" to "value"
)
)
)
}
}
Let’s dissect the code of the above unit test class, and discuss how it can be converted to BehaviorSpec. First of all, when porting our code to the Kotest framework, we stop worrying about coroutines - no need for @ExperimentalCoroutinesApi
and for littering all our test cases with runBlocking
, as it supports coroutines natively. We may need to use the TestCoroutineDispatcher
, if we require passing the coroutine dispatcher as a parameter to on of our method calls.
Next, (and since the code will be ported to BehaviorSpec) the @BeforeEach
setup
function will be replaced with an appropriate beforeContainer
method call, which will still be responsible for reinitialising the object in test, before the test is executed. These tests have a merged arrange-act
section, which will be directly moved inside a When
block. The assertions will be moved inside a Then
block, and be replaced with the appropriate Kotest assertions.
The converted test is the following:
import io.kotest.core.spec.style.BehaviorSpec
import io.kotest.matchers.shouldBe
class HtmlMetaExtractorTest : BehaviorSpec({
lateinit var htmlMetaExtractorInTest: HtmlMetaExtractor
beforeContainer {
htmlMetaExtractorInTest = HtmlMetaExtractor()
}
Given("an HtmlMetaExtractor") {
When("it attempts to parse an empty string resource") {
val response = htmlMetaExtractorInTest.parseResource("")
Then("it returns an empty map") {
response shouldBe emptyMap()
}
}
When("the HTML input contains an unknown META key attribute") {
val response =
htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_UNKNOWN_ATTRIBUTE_META)
Then("it returns an empty map") {
response shouldBe emptyMap()
}
}
When("the HTML input contains supported META key attributes") {
val response =
htmlMetaExtractorInTest.parseResource(HTML_DOCUMENT_WITH_ALL_SUPPORTED_TYPES_META)
Then("it returns a map containing all META key-value pairs") {
response shouldBe mapOf("key-1" to "value", "key-2" to "value", "key-3" to "value")
}
}
}
})
A more advanced example
A more complex example unit test conversion is presented below:
@ExperimentalCoroutinesApi
class ResourceParserTest {
@MockK
lateinit var mockResourceParserDelegate: ResourceParserDelegate
private lateinit var objectInTest: ResourceParser
private val testCoroutineDispatcher = TestCoroutineDispatcher()
@BeforeEach
fun setup() {
MockKAnnotations.init(this)
objectInTest = ResourceParser(testCoroutineDispatcher)
}
@Test
fun `the resource parser delegates it's work to the supplied ResourceParserDelegate and returns the result to the caller`() =
runBlockingTest {
val fakeResource = "fake_resource"
val expectedResult = mapOf("key" to "value")
every { mockResourceParserDelegate.parseResource(fakeResource) } returns expectedResult
val result = objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
verify { mockResourceParserDelegate.parseResource(fakeResource) }
assertThat(result, _is(expectedResult))
}
@Test
fun `exceptions thrown by the ResourceParserDelegate are propagated to the caller`() =
runBlockingTest {
val fakeResource = "fake_resource"
every { mockResourceParserDelegate.parseResource(fakeResource) } throws RuntimeException(
"Error"
)
assertThrows<RuntimeException> {
objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
}
}
}
This test requires a few more advanced features. Firstly, it requires the use of the TestCoroutineDispatcher
- because it has an external dependency on the CoroutineDispatcher
to be used for executing the code. It also requires using a mock to establish that a particular scenario leads to executing a specific method of the mock. Finally, it makes use of an assertion that determines whether a specific code path throws an exception and how that is handled.
A potential conversion of the tests shown above, to the BehaviorSpec in Kotest is the following:
class ResourceParserTest : BehaviorSpec({
lateinit var mockResourceParserDelegate: ResourceParserDelegate
lateinit var objectInTest: ResourceParser
val testCoroutineDispatcher = TestCoroutineDispatcher()
beforeContainer {
mockResourceParserDelegate = mockk()
objectInTest = ResourceParser(testCoroutineDispatcher)
}
Given("a ResourceParser") {
val fakeResource = "fake_resource"
When("it is requested to parse a resource") {
val expectedResult = mapOf("key" to "value")
every { mockResourceParserDelegate.parseResource(fakeResource) } returns expectedResult
val result = objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
Then("it produces the result by delegating its work to the supplied ResourceParserDelegate") {
verify { mockResourceParserDelegate.parseResource(fakeResource) }
result shouldBe expectedResult
}
}
When("an exception is thrown by the ResourceParserDelegate") {
every { mockResourceParserDelegate.parseResource(fakeResource) } throws RuntimeException(
"Error"
)
Then("it is propagated to the caller") {
shouldThrow<RuntimeException> {
objectInTest.parseResource(mockResourceParserDelegate, fakeResource)
}
}
}
}
})
In the Kotest version, you can no longer make use of annotations for initialising your mocks (as supported by popular mocking libraries, such as mockk and mockito). This is because the test is written in a lambda that is passed as a parameter in the BehaviorSpec constructor.Therefore, annoatations cannot be used for this purpose (at least with mockk, although this will also be the case for other libraries as well). The object in test is initialised before any containers are initialised (using a beforeContainer
block), as in the previous case. The two tests share no common initialisation code (apart from a common variable), and there is no need to add another beforeContainer
before the two When
blocks. The arrange-related
statements are placed at the beginning of the When
blocks. In the first test, this is followed by the act-related
statement which also stores the result, that is used inside the assert-related
statements placed in the Then
block. In the second test, the act-related
and the assert-related
statements are joined in a single shouldThrow
block, because it tests whether an exception is thrown.
CI-support
Kotest integrates very well with Intellij (and Android Studio) IDE, via its official plugin. However, since its structure is a radical departure from the standard JUnit format, the default JUnit report generated by the Gradle test plugin becomes unusable (the test count and the test names are wrong). This may be irrelevant when running your tests locally - it completely breaks potential (or current) CI integrations you may have. To overcome this limitation, the Kotest framework offers the JUnit XML extension, which writes a separate report file that correctly reports the tests executed and their results.
To set it up, you need to add the following line to your module’s build gradle ($kotestVersion refers to the kotest version you use in your project):
testImplementation "io.kotest:kotest-extensions-junitxml:$kotestVersion"
To use the extension, you will need to create a subclass (either a class or object will do) of the AbstractProjectConfig
class, which is automatically used by Kotest to configure the overall test project. You can see all available configuration options here. In the subclass, you override the listeners
function such that it returns a list with a new instance of the JunitXmlReporter
. The subclass can reside anywhere in your test source set - Kotest will find and use it. An example is the following:
object MyKotestProjectConfig : AbstractProjectConfig() {
override fun listeners(): List<Listener> = listOf(
JunitXmlReporter(
includeContainers = false,
useTestPathAsName = true
)
)
}
The constructor of the JunitXmlReporter
takes two parameters:
- includeContainers - a boolean which indicates whether the XML report will include intermediate containers as separate line entries, not just
Then
blocks. Defaults tofalse
. When set totrue
, it artificially increases the number of tests included (without much benefit). - useTestPathAsName - a boolean which indicates whether the name of the test in the report will include the complete path (the sequence of
Given
,When
,And
,Then
) of the test, or just the finalThen
block. This defaults totrue
. When set tofalse
, all test names will only include the leaf-Then
blocks names.
After running your tests, the correct JUnit XML reports (and only those, not the HTML report) will be found in your module’s build directory, under the test-results/test
subfolder. Note that the default, incorrect XML version of the test results will also be available under the test-results/testXYZUnitText
(XYZ being the combination of flavor/build type used in running the tests).
Example code demonstrating the usage of the JUnit XML extension can be found here.