Testing Your Procedures with RackUnit

Summary

As you develop procedures and collections of procedures, you have a responsibility to make sure that they work correctly. One mechanism for checking your procedures is a comprehensive suite of tests. In this reading, we consider the design and use of tests. We also consider RackUnit, the testing framework that comes with DrRacket.

Introduction

Most computer programmers strive to write clear, efficient, and correct code. It is (usually) easy to determine whether code is clear. With some practice and knowledge of the correct tools, one can determine how efficient code is. However, believe it or not, it is often difficult to determine whether code is correct.

The gold standard of correctness is a formal proof that the procedure or program is correct. However, in order to prove a program or procedure correct, one must develop a rich mathematical toolkit and devote significant effort to writing the proof. Such effort is worth it for life-critical applications. However, for many programs, the effort and time required for a formal proof are often more than can be reasonably expected.

There is also a disadvantage of formal proof: Code often changes and the proof must therefore also change. Why does code change? At times, the requirements of the code change (e.g., a procedure that was to do three related things is now expected to do four related things). At other times, with experience, programmers realize that they can improve the code by making a few changes. If we require that all code be proven correct, and if changing code means rewriting the proof, then we discourage programmers from changing their code.

Hence, we need other ways to have some confidence that our code is correct. A typical mechanism is a test suite, a collection of tests that are unlikely to all succeed if the code being tested is erroneous. One nice aspect of a test suite is that when you make changes, you can simply re-run the test suite and see if all the tests succeed. To many, test suites encourage programmers to experiment with improving their code, since good suites will tell them immediately whether or not the changes they have made are successful.

But when and how do you develop tests? These questions are the subject of this reading.

What is a Test?

As the introduction suggested, you should write tests when you write code. But what is a test? Put simply, a test is a bit of code that reveals something about the correctness of a procedure or a set of procedures. Most typically, we express tests in terms of expressions and their expected values.

For example, suppose we’ve written a procedure, irgb-add-16-blue, that takes an integer-encoded RGB color as input and adds 16 to its blue component. Given a starting color, color, we might expect that

• (irgb-red (irgb-add-16-blue color)) is (irgb-red color)
• (irgb-green (irgb-add-16-blue color)) is (irgb-green color)
• (irgb-blue (irgb-add-16-blue color)) is (+ 16 (irgb-blue color))

Do these look a lot like our postconditions? Yes. They should. Postconditions are one of the ways we express clear expectations for our programs.

We could express those expectations in a variety of ways. The simplest strategy is to execute each expression, in turn, and see if the result is what we expected. You should have be using this form of testing regularly in your coding. (We often call this “experimenting with your code” to distinguish it from the kinds of testing we introduce in this reading.)

> (irgb-red (irgb-add-16-blue fave1))
150
> (irgb-red fave1)
150
> (irgb-green (irgb-add-16-blue fave1))
25
> (irgb-green fave1
25
> (irgb-blue (irgb-add-16-blue fave1))
190
> (irgb-blue fave1)
174
> (+ 16 174)
190

Of course, one disadvantage of this kind of testing is that you have to manually look at the results to make sure that they are correct. You also have to know what the correct answers should be. But reading isn’t always a good strategy. There’s some evidence that you don’t always catch errors when you have to do this comparison, particularly when you have a lot of tests. We know that we’ve certainly missed a number of errors this way. An appendix to this document presents an interesting historical anecdote about the dangers of writing a test suite in which you must manually read all of the results.

Since reading the results is tedious and perhaps even dangerous, it is often useful to have the computer do the comparison for you. For example, we might write a procedure, check, that checks to make sure that two expressions are equal. (This procedure uses if, which you might not have learned yet. But you should be able understand the general approach.)

(define check
  (lambda (exp1 exp2)
     (if (equal? exp1 exp2)
         "OK"
         "FAILED")))

We can then use this procedure for the tests above, as follows.

> (check (irgb-red (irgb-add-16-blue fave1)) 
         (irgb-red fave1))
"OK"
> (check (irgb-green (irgb-add-16-blue fave1)) 
         (irgb-green fave1))
"OK"
> (check (irgb-blue (irgb-add-16-blue fave1)) 
         (irgb-blue fave1))
"FAILED"
> (check (irgb-blue (irgb-add-16-blue fave1)) 
         (+ 16 (irgb-blue fave1)))
"OK"

Note that in the penultimate test, the test itself, rather than irgb-add-16-blue, was incorrect.

Confirming that our code is correct is now simply a matter of scanning through the results and seeing if any say "FAILED". And, as importantly, we’ve specified the expected result along with each expression, so we don’t need to look it up manually.

Of course, there are still some disadvantages with this strategy. For example, if we put the tests in a file to execute one by one, it may be difficult to tell which ones failed. Also, for a large set of tests, it seems a bit excessive to print OK every time. Finally, we get neither OK nor FAILED when there’s an error in the original expression.

In fact, if an error occurs in the middle of a group of tests, the whole thing may come to a screeching halt.

> (check (irgb-red (irgb-add-16-blue fave1)) (irgb-red (fave1)))
Error! application: not a procedure;
Error! expected a procedure that can be applied to arguments
Error! given: 9836718
Error! arguments...: [none]
> (check (irgb-red (irgb-add-16-blue "red")) (irgb-red "red"))
Error! irgb-red: expects argument of type <integer-encoded rgb color>; given: "red"

Testing Frameworks

To handle all of these additional issues, many program development environments now include some form of testing framework. And even when they don’t, languages often have accompanying testing frameworks. While testing frameworks differ, they tend to share some commonalities.

First, most testing frameworks provide a way for you to check expected results. That’s a lot like our check procedure above. That is, we have a value we expect and an expression; we evaluate the expression; and we compare the result to the expected value. We will refer to procedures used to check expected results as “checks”.

Second, most testing frameworks provide a way to group checks into tests. Why would we need more than one check in a test? Sometimes, it’s because we need to check multiple things about a result. For example, if we want to make sure that (irgb-redder (irgb 100 100 100)) has the right return value, we might want to check that (a) the result is a integer-encoded RGB value, (b) the red component is larger than 100, (c) the green component is no larger than 100, and (d) the blue component is no larger than 100. . But other times, the tester feels it’s natural to put a lot of related checks into a single test—if any of them fail, the whole test fails. How do you divide checks into tests? In some sense, it’s a matter of taste. Some testers like just a few checks per test. Others like a lot.

Finally, most testing frameworks provide a way to group individual tests into test suites. Why do we need groups of tests? Because we often have multiple procedures to test, both individually and in groups, and we want to run all of the tests to ensure that everything works together. For example, we might provide a library of a variety of related functions and want to test the whole library en masse.

So, when you first encounter a new testing framework, you should ask yourself three questions: How do you check for expected results? How do you group checks into tests? And how do you group tests into suites? (You should also ask a few related questions, such as how you run the tests.)

RackUnit: A Testing Framework for Racket

The designers of the Racket programming language designed a testing framework to go with the language. They call this framework “RackUnit”. As you might expect, RackUnit framework is well integrated in DrRacket.

By default, RackUnit is not available to your program. (Why not? Because there are many libraries that Racket programmers might use, and the designers of Racket decided that the default should be to only include those that the programmer deems necessary.) To load RackUnit, add the following to the top of the definitions pane.

(require rackunit)
(require rackunit/text-ui)

RackUnit provides a variety of procedures that we can use to check results. Most of them take an expected result, an input expression, and an optional message.

(check-= expression expected epsilon), (check-= expression expected epsilon optional-message) RackUnit procedure.

Evaluate expression ... expected and then compare them for numeric equality (within *epsilon). If they are equal, do nothing. If they are not equal, print an error message. If the optional message is included, also print that message.

(check-equal? expression expected), (check-equal? expression expected optional-message) RackUnit procedure.

Evaluate expression ... expected and then compare them for equality. If they are equal, do nothing. If they are not equal, print an error message. If the optional message is included, also print that message.

(check-not-equal? expression expected), (check-not-equal? expression expected optional-message)
RackUnit procedure. Evaluate expression ... expected and then compare them. If they are not equal, do nothing. If they are equal, print an error message. If the optional message is included, also print that message.

Although we will typically put checks into tests, we can run them on their own. When they succeed, they print no result. When they fail, they print an error message. For example,

> (check-= 4 4 0)
> (check-= 4 (* 2 2) 0 "two times two is four")
> (check-= 2 (* (sqrt 2) (sqrt 2)) 0.00001 "sqrt 2 squared, approximate")
> (check-= 2 (* (sqrt 2) (sqrt 2)) 0 "sqrt 2 squared, exact")
Error! --------------------
Error! FAILURE
Error! name:       check-=
Error! location:   (stdin #f #f 230 59)
Error! expression: (check-= 2 (* (sqrt 2) (sqrt 2)) 0)
Error! params:     (2 2.0000000000000004 0)
Error! message:    "sqrt 2 squared, exact"
Error! 
Error! Check failure
Error! --------------------

We group checks into tests with test-case.

(test-case description check-1 ... check-n) RackUnit procedure.

Create a new test case by running a series of checks.

Note that test-case runs the test immediately. Sometimes that’s useful; sometimes we’d like to build up a bunch of tests for running later. And that’s where test suites come into play.

(test-suite description check-or-test-or-suite-1 ... check-or-test-or-suite-n) RackUnit procedure.

Create a new test suite that groups together a variety of checks, tests, and other suites. Unlike tests and checks, which are executed immediately, test suites are objects that can be run separately.

Note that, unlike test-case, test-suite does not run the tests. Instead, it builds a suite that we can later run with run-tests. (Why make the distinction? Sometimes it ends up being easier to have the tests grouped so that you can easily redefine them.) So, whenever you make a test suite, you’ll probably have to name it with define.

Let’s return to our color modification example and consider how we might put together our simple checks. Here’s one approach: We can just type them directly.

> (check-= (irgb-red fave1) (irgb-red (irgb-add-16-blue fave1)) 0)
> (check-= (irgb-green fave1) (irgb-green (irgb-add-16-blue fave1)) 0)
> (check-= (+ 16 (irgb-blue fave1)) (irgb-blue (irgb-add-16-blue fave1)) 0)

But we’re better off grouping them into a suite.

(define fave1 (irgb 150 24 174))
(define irgb-add-16-blue-tests
  (test-suite
   "Tests of irgb-add-16-blue"
   (test-case
    "red"
    (check-= (irgb-red fave1) (irgb-red (irgb-add-16-blue fave1)) 0))
   (test-case
    "green"
    (check-= (irgb-green fave1) (irgb-green (irgb-add-16-blue fave1)) 0))
   (test-case
    "blue"
    (check-= (+ 16 (irgb-blue fave1)) (irgb-blue (irgb-add-16-blue fave1)) 0))))

We can then run the tests.

> (run-tests irgb-add-16-blue-tests)
Output! 3 success(es) 0 failure(s) 0 error(s) 3 test(s) run
0

What happens if a test fails? Let’s insert a broken test to see.

(test-case
 "broken"
 (check-equal? 0 1))

> (run-tests irgb-add-16-blue-tests)
Error! --------------------
Error! Tests of irgb-add-16-blue > broken
Error! broken
Error! FAILURE
Error! name:       check-equal?
Error! location:   unsaved-editor142508:29:4
Error! actual:     0
Error! expected:   1
Error! Check failure
Error! --------------------
Output! 3 success(es) 1 failure(s) 0 error(s) 4 test(s) run
1

When to Write Tests

To many programmers, testing is much like documentation. That is, it’s something you add after you’ve written the majority of the code. However, testing, like documentation, can make it much easier to write the code in the first place.

As we suggested in the reading on documentation, by writing documentation first, you develop a clearer sense of what you want your procedures to accomplish. Taking the time to write documentation can also help you think through the special cases. For some programmers, writing the formal postconditions can give them an idea of how to solve the problem.

If you design your tests first, you can accomplish similar goals. For example, if you think carefully about what tests might fail, you make sure the special cases are handled. Also, a good set of tests of the form “this expression should have this value” can serve as a form of documentation for the reader, explaining through example what the procedure is to do. There is even a popular style of software engineering, called test-driven development (TDD), in which you always write the tests first. Test-driven development is a key part of Extreme Programming and a variety of so-called “agile software development strategies”.

Self Checks

Starting DrRacket and add (require rackunit) and (require rackunit/text-ui) to your definitions pane and click Run.

Check 1: Testing the Tester

RackUnit provides a variety of procedures to help you write tests:

(check-equal? expression expected), (check-equal? expression expected optional-message) RackUnit procedure.

Evaluate expression ... expected and then compare them for equality. If they are equal, do nothing. If they are not equal, print an error message. If the optional message is included, also print that message.

(check-not-equal? expression expected), (check-not-equal? expression expected optional-message) RackUnit procedure.

Evaluate expression ... expected and then compare them. If they are not equal, do nothing. If they are equal, print an error message. If the optional message is included, also print that message.

(check-= expression expected epsilon), (check-= expression expected epsilon optional-message) RackUnit procedure.

Evaluate expression ... expected and then compare them for numeric equality (within *epsilon). If they are equal, do nothing. If they are not equal, print an error message. If the optional message is included, also print that message.

(test-case description check-1 ... check-n) RackUnit procedure.

Create a new test case by running a series of checks.

(test-suite description check-or-test-or-suite-1 ... check-or-test-or-suite-n) RackUnit procedure.

Create a new test suite that groups together a variety of checks, tests, and other suites. Unlike tests and checks, which are executed immediately, test suites are objects that can be run separately.

(run-tests tests) RackUnit procedure.

Run the given test suite, printing a summary and producing the number of unsuccessful tests.

In the Interactions pane, try each of the operations a few times to make sure you understand its operation. (Yes, this instruction is intentionally vague.)

Check 2: Testing invert

Write a test suite for the inverse procedure you wrote for the reading on procedures and documented for the reading on documentation.

Appendix: An Historical Tale

Many of us are reminded of the need for unit testing by the following story by Doug McIlroy, posted to The Risks Digest: Forum on Risks to the Public in Computers and Related Systems:

Sometime around 1961, a customer of the Bell Labs computing center questioned a value returned by the sine routine. The cause was simple: a card had dropped out of the assembly language source. Bob Morris pinned down the exact date by checking the dutifully filed reversions tests for system builds. Each time test values of the sine routine (and the rest of the library) had been printed out. Essentially the acceptance criterion was that the printout was the right thickness; the important point was that the tests ran to conclusion, not that they gave right answers. The trouble persisted through several deployed generations of the system.

If Bell Labs had arranged for a count of successes and a list of failures rather than a thick printout they (and their customers) would have have been in much better shape.