Testing Weasel Words

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to cut away. -- Antoine de Saint-Exupéry

Manual and automated software tests often include redundant and unnecessary words which can distract the tester from the real purpose of the test. We're so accustomed to seeing them that they often pass unnoticed.

Consider the following generic test description:

Verify that users should be allowed to login to the system with the correct username and password.

This seemingly innocuous sentence contains several unnecessary words. Clearly, this test is related to something that requires authentication, and the test is supposed to check that legitimate users are granted access.

Let's start with the word "verify". In my experience, this is one of the most heavily-abused words in software testing. Why do we need this word? All testing is about verification. There's no reason to state the fact that you're verifying something; that's what tests are all about. If a test is not verifying anything, then it's a futile exercise. You don't need to come out and say "verify this" or "verify that". Can we just assume that all tests (and all steps within those tests) are verifying, and leave that bit out?

Users should be allowed to login to the system with the correct username and password.

This sentence is just as clear as it was before, and two words shorter. Similar sentiments could apply to words like "test", "ensure", "check", "confirm" and so on--they're all synonyms for "verify".

Next up is the word "should". This one is up for debate; it's very common in automated systems like RSpec and Cucumber, and it seems to express something important. By asserting that the software should have some behavior, we're saying that this behavior is deliberate, designed in, and considered correct. But really, what kind of assertion are we making when we say that "X should Y"? How is this assertion different from saying "X does Y"?

More concretely, if it is true that users should be allowed to login, then what we want to test is whether users actually are allowed to login. We cannot test the truth of a statement like "X should Y"; we are already assuming that statement is true within the context of the test. If instead we say "X does Y", or "X can Y", we're making an assertion about the system that is either true or false. If X actually does Y, then "X does Y" is true. If X does something other than Y, then "X does Y" is false.

We might say "The Earth should go around the Sun," but that sounds ridiculous to anyone versed in the scientific method. Does it or doesn't it? Our test is a hypothesis that the system works correctly; by observing reality, we determine whether the hypothesis is true.

In short, it's a lot like the word "verify". We can say "should" of every assertion that a test makes; the test should be correct, the system should do what it's supposed to. That's a given in testing. What we care about in testing is whether it does.

If we take out the word "should", and make a grammatical adjustment, we now have:

Users are allowed to login to the system with the correct username and password.

This sentence expresses everything it did before, only now the assertion is falsifiable. Users are, or are not, allowed to login. Our test says they can login, and our job now is to confirm or refute that hypothesis.

OK, so what's up with "are allowed to" or "be allowed to"? Isn't there a much shorter phrase that means the same thing? How many parents have corrected children who ask "can I" when they mean "may I"? In this case, it's more about permission than physical ability, so why not substitute "may" for "are allowed to"?

Users may login to the system with the correct username and password.

Personally, I think "can" would work just as well here; we're testing the authentication system, rather than the mental and physical faculties of our users. But if you're a "can" vs. "may" stickler, choose the one that seems most correct to you. Either is miles better than "are allowed to". The same goes for "are able to" or "are capable of".

The next unnecessary word is "system". We often refer to the "system under test" (SUT), so it seems natural to refer to what it is you're testing. But here, the usage is so vague as to be meaningless. If the person running this test doesn't already know what SUT is being tested, they're not going to get any useful information from the phrase "the system". Let's take it out:

Users may login with the correct username and password.

What about "correct"? We're using that here to disambiguate from the case where a user tries to login using an incorrect username or password. Obviously this is an important distinction in testing an authentication process, but could it be phrased more succinctly?

It could be argued that "user" implies someone who has valid credentials, and that without those, they're not really a user. It's possible that we may have some former users whose account has expired or been deactivated, and those users should not be granted access. Maybe we should be saying "Users with an active account" or "Users with an active account and a membership role" or any number of other qualifiers, depending on what we're testing. Those qualifiers, while legitimate uses of extra words, were not part of the original description, and that description is our reference point for this exercise. We're trying to cull unnecessary words without losing meaning; increased specificity is not the goal here.

If we take it as given that each user has one username and one password belonging to them, we could use the possessive article:

Users may login with their username and password.

Has any information been lost from the original description? It doesn't seem so. We might go as far as:

Users may login.

But here we've reduced the original meaning. May users login with any username and password? Do users need a password? We don't know anymore. This version is too short, because we've taken away important words.

All of this leads up to a core principle that I've come to believe strongly in my years of writing and running software tests. Tests should be all about assertion--any verification they do should be stated as factual. This makes them inherently falsifiable. Don't beat around the bush with words that distract from the assertions your test is making. State your hypothesis (usually some form of "it works") as if you are certain that it is true. Running the test confirms or refutes that hypothesis. Writing your tests this way will make them not just good science, but good communication as well.