Home / Team / Eric Pierce

Eric Pierce

I'm a software tester, developer, and graphic designer living in Colorado Springs with my wife and son. I have five years of experience with automated testing, ten years of software engineering and development, and fifteen years in illustration and 3D graphics.

In my spare time, I enjoy programming in Python and learning new languages such as Haskell. I'm a strong proponent of open source and open content. I created Kelp, Cukable, Grinder Webtest, CSVSee, tovid, PySci and other software, and have contributed to several open-source projects. I've also added many illustrations and articles to Wikipedia.

I think any IT professional should have a large repertoire of skills and be open to learning new technologies; nevertheless, I do have some clear favorites:

My web presences:

  • "Share

My Blog

Testing Weasel Words
Tue, 07/19/2011 - 02:43

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to cut away. -- Antoine de Saint-Exupéry

Manual and automated software tests often include redundant and unnecessary words which can distract the tester from the real purpose of the test. We're so accustomed to seeing them that they often pass unnoticed.

Consider the following generic test description:

Verify that users should be allowed to login to the system with the correct username and password.

This seemingly innocuous sentence contains several unnecessary words. Clearly, this test is related to something that requires authentication, and the test is supposed to check that legitimate users are granted access.

Let's start with the word "verify". In my experience, this is one of the most heavily-abused words in software testing. Why do we need this word? All testing is about verification. There's no reason to state the fact that you're verifying something; that's what tests are all about. If a test is not verifying anything, then it's a futile exercise. You don't need to come out and say "verify this" or "verify that". Can we just assume that all tests (and all steps within those tests) are verifying, and leave that bit out?

Users should be allowed to login to the system with the correct username and password.

This sentence is just as clear as it was before, and two words shorter. Similar sentiments could apply to words like "test", "ensure", "check", "confirm" and so on--they're all synonyms for "verify".

Next up is the word "should". This one is up for debate; it's very common in automated systems like RSpec and Cucumber, and it seems to express something important. By asserting that the software should have some behavior, we're saying that this behavior is deliberate, designed in, and considered correct. But really, what kind of assertion are we making when we say that "X should Y"? How is this assertion different from saying "X does Y"?

More concretely, if it is true that users should be allowed to login, then what we want to test is whether users actually are allowed to login. We cannot test the truth of a statement like "X should Y"; we are already assuming that statement is true within the context of the test. If instead we say "X does Y", or "X can Y", we're making an assertion about the system that is either true or false. If X actually does Y, then "X does Y" is true. If X does something other than Y, then "X does Y" is false.

We might say "The Earth should go around the Sun," but that sounds ridiculous to anyone versed in the scientific method. Does it or doesn't it? Our test is a hypothesis that the system works correctly; by observing reality, we determine whether the hypothesis is true.

In short, it's a lot like the word "verify". We can say "should" of every assertion that a test makes; the test should be correct, the system should do what it's supposed to. That's a given in testing. What we care about in testing is whether it does.

If we take out the word "should", and make a grammatical adjustment, we now have:

Users are allowed to login to the system with the correct username and password.

This sentence expresses everything it did before, only now the assertion is falsifiable. Users are, or are not, allowed to login. Our test says they can login, and our job now is to confirm or refute that hypothesis.

OK, so what's up with "are allowed to" or "be allowed to"? Isn't there a much shorter phrase that means the same thing? How many parents have corrected children who ask "can I" when they mean "may I"? In this case, it's more about permission than physical ability, so why not substitute "may" for "are allowed to"?

Users may login to the system with the correct username and password.

Personally, I think "can" would work just as well here; we're testing the authentication system, rather than the mental and physical faculties of our users. But if you're a "can" vs. "may" stickler, choose the one that seems most correct to you. Either is miles better than "are allowed to". The same goes for "are able to" or "are capable of".

The next unnecessary word is "system". We often refer to the "system under test" (SUT), so it seems natural to refer to what it is you're testing. But here, the usage is so vague as to be meaningless. If the person running this test doesn't already know what SUT is being tested, they're not going to get any useful information from the phrase "the system". Let's take it out:

Users may login with the correct username and password.

What about "correct"? We're using that here to disambiguate from the case where a user tries to login using an incorrect username or password. Obviously this is an important distinction in testing an authentication process, but could it be phrased more succinctly?

It could be argued that "user" implies someone who has valid credentials, and that without those, they're not really a user. It's possible that we may have some former users whose account has expired or been deactivated, and those users should not be granted access. Maybe we should be saying "Users with an active account" or "Users with an active account and a membership role" or any number of other qualifiers, depending on what we're testing. Those qualifiers, while legitimate uses of extra words, were not part of the original description, and that description is our reference point for this exercise. We're trying to cull unnecessary words without losing meaning; increased specificity is not the goal here.

If we take it as given that each user has one username and one password belonging to them, we could use the possessive article:

Users may login with their username and password.

Has any information been lost from the original description? It doesn't seem so. We might go as far as:

Users may login.

But here we've reduced the original meaning. May users login with any username and password? Do users need a password? We don't know anymore. This version is too short, because we've taken away important words.

All of this leads up to a core principle that I've come to believe strongly in my years of writing and running software tests. Tests should be all about assertion--any verification they do should be stated as factual. This makes them inherently falsifiable. Don't beat around the bush with words that distract from the assertions your test is making. State your hypothesis (usually some form of "it works") as if you are certain that it is true. Running the test confirms or refutes that hypothesis. Writing your tests this way will make them not just good science, but good communication as well.

Declare or Impair
Thu, 06/09/2011 - 21:24

A topic that crops up often on the Cucumber mailing list is whether it's best to write BDD tests in imperative or declarative style. Back in 2008, a blog post by Ben Mabey described some weaknesses of the imperative style, especially its tendency toward brittleness. Mike Swieton echoes this argument in Never say "Click", while others such as Zach Moazeni disagree, preferring to keep steps as generic and reusable as possible.

I won't rehash the examples and arguments for both approaches here; the links above explain it better than I could, anyway. I would like to explore a little about why the distinction is important, and why programmers and automated testers should know the difference.

When I was starting out with BDD, I didn't think much about imperative vs. declarative test scenarios. Programmers who have only worked with imperative programming languages don't often consider that a declarative programming approach is possible. The first time I heard the term "imperative" in the context of computer programming, it confused me a little, because I had only done imperative programming and didn't realize there was any other way to program. As they say, fish don't know they're wet.

At the lowest levels, computer languages are imperative. Assembly languages are about as imperative as you can get; assembler pseudocode might read something like this:

Load the value at address X into register A
Load the value at address Y into register B
Add register B to register A
Store the contents of register A to address Z

Imperative programs are all about how to do it. They are, by necessity, long and detailed, like the multilingual instructions that come with an unassembled furniture kit. Insert tab A into slot B, insert screw C into hole D, tighten with screwdriver.

The higher-level a programming language is, the more declarative it tends to be. The most common operations such as memory-address-tracking, register-loading, loop-variable-incrementing and other complexities are wrapped up in convenient abstractions like variables, expressions, and iteration. Rather than give the machine step-by-step instructions on how to load X and Y into registers A and B, add them together, and store the result in memory location Z, we just say:

Z = X + Y

We declare Z's relationship to X and Y, instead of explicitly describing how to calculate Z. Yes, technically this is still an imperative statement (unless it's a functional language, in which case you're declaring a function called Z that returns the sum of functions X and Y). The point is, we've taken a big step away from the low-level details, toward a more natural, condensed expression that is less mentally taxing. It reads more like a statement of fact than a command to do.

Declarative programming is the what, rather than the how. Declarative instruction is, if I may stretch the analogy, more like the illustrations that depict the spatial relationships between the unassembled furniture pieces, instead of trying to explain all of the steps using words alone. The higher-level pictures tend to be more intuitive and easier to follow than the verbose, low-level imperative instructions. I don't know about you, but I usually just follow the pictures when I'm assembling furniture, or Lego models. And I certainly prefer writing Python, Haskell, or Ruby to writing Java, C or assembler.

How does this translate to writing business-oriented test scenarios? Well, as the articles linked above demonstrate, and as is often mentioned in places like the Cucumber mailing list, a person familiar with the test domain can usually get by with:

Given I am logged into the website

and doesn't need:

When I go to the login page
And I fill in the username with "admin"
And I fill in the password with "spork"
And I press "Login"

Such low-level minutiae can be a major distraction to someone reading (or writing) test cases. True, occasionally those low-level details are important, like when you're thoroughly testing the various ways that the login process can succeed or fail. But when the login process is not what you're testing, there's no reason to bore the viewer (or author) with the mundane stuff.

Of course, at some point, any declarative steps you write in a BDD scenario need to be translated into imperative directions for the computer to follow; that's pretty much inevitable (and is one of the reasons I created kelp). The beauty of Cucumber is that you get to define the language the tests are written in. The imperative details can often be hidden away in the step definitions, where they can be maintained by the programmers who need to concern themselves with such things. Ideally, the test scenarios themselves should be comprehensible to the business domain experts, and to the average user.

Programmers may find it hard to break out of the imperative frame of mind, but declarative writing can lead to more compact, intelligible tests that need less maintenance in the long run.

The Perils of Trying to be Clever
Sat, 06/04/2011 - 00:13
  • Narrator: Tyler, you are by far the most interesting single-serving friend I've ever met... see I have this thing: everything on a plane is single-serving...
  • Tyler: Oh I get it, it's very clever.
  • Narrator: Thank you.
  • Tyler: How's that working out for you?
  • Narrator: What?
  • Tyler: Being clever.
  • Narrator: Great.

While working on Vittles today, I had a facepalm moment, preceded by an hour of desperate confusion. The feeling will be familiar to many programmers--testing, debugging, head-scratching, more testing and debugging, then finally the moment of realization: "Duh!"

What went wrong? Well, as is the case with many software bugs, I tried too hard to be clever. I have some Recipes, and each Recipe has some Ingredients. Both Recipes and Ingredients keep track of their nutritional information in a related model called NutritionInfo. Whenever a Recipe changes (say, it gets another Ingredient), I want to recalculate the NutritionInfo; I do that by first recalculating all the NutritionInfo for the Ingredients.

I wanted to keep track of the success or failure of each Ingredient's recalculation (indicated by a True/False status). If all Ingredient recalculations were successful, I wanted to know about it. If any one of them failed, I wanted to know that too.

At first, I considered doing something like this:

    results = [ 
        ingredient.nutrition_info.recalculate()
        for ingredient in all_ingredients
    ]   
    success = all(results)

But to my cleverness-infested brain, this seemed like one step too many. Why not just write this?

    success = all(
        ingredient.nutrition_info.recalculate()
        for ingredient in all_ingredients
    )

It seemed like a good idea at the time. But I soon discovered that many ingredients were mysteriously not getting their nutrition info recalculated. What could be wrong? Was some object not getting saved correctly? Was I missing some piece of data that would allow grams to be converted to cups? Finally it dawned on me--something that should have been obvious to a veteran Pythonista like myself--the all method short-circuits on the first occurrence of False, which means that as soon as any ingredient failed the recalculation, none of the ones after that would even be attempted.

After switching back to my initial, more-verbose-but-also-more-correct approach, it works as intended.

Moral: Cleverness doesn't always work out so well.