Test Engineering

Writing tests is a common practice, especially for firms that leverage continuous delivery. With this article I’d like to explain some methods that can be used to generate test cases. The following methods are covered in the scope of the post

• Equivalence partitioning and boundary values
• State transitions
• Path testing

Let’s dive into the details of each method.

Partitions and Boundaries

Similar to mathematical functions, in programming a function maps inputs from a given domain X to a codomain Y. Equivalence Partitioning (EP) is a method to split the domain X into multiple partitions that exhibit the same behaviour. For each partition a single test case is created, thus creating a kind of Partition Coverage.

An example will shed light on the methodology

func CalcPrice(amount uint) {
    if amount > 100 {
        return 10
    }

    if amount > 50 {
        return 11
    }

    if amount > 10 {
        return 15
    }

    return 20
}

There are four partitions to this CalcPrice function

1. The range from 100 to infinity (100, ∞)
2. The range 50 to 100 (50, 100]
3. The range 10 to 50 (10, 50]
4. The range up to 10 [0, 10]

EP now recommends to cover each of the partitions in one test. Boundary Value Analysis (BVA) extends this definition to also include the lower and upper values of the boundaries. The reasoning behind this is simple. There are usually a lot of issues with the boundaries, since the ranges can either be openend or closed on either side. BVA adds another 8 tests to the list: UINT_MAX, 101, 100, 51, 50, 11, 10, 0. Together with EP that results in 12 tests, giving a reasonable coverage.

State Transitions

Virtually all systems that we design contain some kind of state and necessarily also some mechanism to alter that state. In state trainsition analysis, all possible states of a single component and their transitions from on to another are graphed. Let’s take the example of a traffic light. There are four different states: Yellow Flahsing, Red, Yellow, Green. Yellow Flashing depicts the base state. Together with these states, there are six allowed transitions:

• Yellow Flashing → Red
• Red → Yellow
• Yellow → Green
• Green → Yellow
• Yellow → Red
• Red → Yellow Flashing

When testing state transitions, there should be tests for each transition, be it valid or invalid. The tests are to assert that no illegal transitions take place and that the system behaves according to the graph.

Path Testing

Path testsing is one possibility of making sure that the code we’ve written works as expected. Let’s first have a look on how we can measure the completeness of our tests. A measurement of the code that has been tested is commonly referred to as Code Coverage. There are three different strategies of analyzing code coverage:

• Statement Coverage - percentage of statements reached
• Branch Coverage - percentage of conditional decisions reached
• Path Coverage - percentage of paths covered

The further down we go in the list, the more tests there are to write. This simple example illustrates the three types

func CalculateValue(withVat bool, amount uint) {
    var calcPrice

    if amount > 10 {
        calcPrice = 50
    } else if amount > 20 {
        // Unreachable code
    } else {
        calcPrice = 55
    }

    if withVat {
        calcPrice *= 1.08
    }

    return amount * calcPrice
}

Now 100% statement coverage would require merely two tests. One for amount > 10 and one for amount < 10, while keeping withVat = true for both. These two tests effectively call 100% of the statements.

100% branch coverage requires that possible scenario in a branch is tested at least once. We’d also write two tests. One for amount > 10 & withVat = true and one amount < 10 & withVat = false. We’d also notice that the branch amount > 20 in the above code can never be reached, thus potentially finding a bug.

100% path coverage is more exact and thus requires more testing effort. For this scenario we’d need four tests (ignoring the amount > 20 problem we’ve already detected)

1. amount > 10 & withVat = true
2. amount < 10 & withVat = true
3. amount > 10 & withVat = false
4. amount < 10 & withVat = false

The measurement of path coverage is thus the most thorough and accurate, yet also the most costly to reach. Path testing is the practice of reaching a high path coverage. In this methodology, tests are extracted according to all the paths that lead through our code.

Conclusion

The three methods above should give the reader a starting point of how tests can be engineered. The list is by no means complete.