Automated Test Confessions

My life as a tester is evolving and I’m feeling less like a newbie.  I’ve also had yet another “James Bach” moment.  This time, a friend of mine forwarded me an article her husband had read and passed along to her.  He’s a developer who, I guess, is going through the whole, “unit testing: what does it all mean?” phase of life.  The email contained a few links.  Among them was James Bach’s paper from 1999, “Test Automation Snake Oil.” As I read through, what I now know is a classic, I realized that I’d been recognizing some of what  Bach writes about in my own tests.  His paper highlighted much of what I’ve come to think about my own tests.

At this point, I’ve been a software tester for about two and a half years.  From my perspective, this is not a very long time. The past year, however, has been insanely intense for me intellectually and academically.  There have been many times during the past year when I have felt myself back in the Interdisciplinary Studies program I took as a Freshman and Sophomore at Appalachian State University.  We were given 100+ pages a night of reading per night which ranged all over the humanities and sometimes sciences.  This reading was in addition to lectures and other “programming” we were expected to attend.  Between the Software Engineering classes, the job as Software Tester and the runaway fascination with Data Visualization, I’ve put myself through a similar gamut of reading and working.  This time my activities have centered around software, computers and testing.  The result of this for my job as a software tester is that I am not the tester I was last year.

At all.

Previously, I was really smitten with HP Quality Center because it gave me structure for which I was desperately searching.  This was a great improvement over the massive, disorganized and growing spreadsheets surrounding me that contained all of my test information.  All of my tests could finally be organized, and thanks to the HP online tutorial I knew my tests were organized well.  I felt liberated!  Now I could stop concentrating on how the tests should be organized and concentrate more on the actual testing itself.

This led to the realization that there was NO WAY I would EVER be able to test EVERYTHING.  I was frustrated.  Why were my test cycles so short?  Why did I always feel like a bottleneck?  Was I not good enough at testing?  Was I not fast enough?  “I must find a way to test faster,” I told myself.

After attending the 2008 Google Test Automation Conference, I turned to unit testing and automation.  I mean, I can write code.  It doesn’t scare me at all.  This doesn’t mean that I’m great at it, but I enjoy it enough to spend significant amounts of time doing it.  I decided to use my coding skills to write repeatable tests that could be run over and over and over again.  After all, I’m pulling my group, by the hair, towards automated builds and smoke tests have to be automated.  Business just LOVES these.  I was told that it was making my group look really good to have automated tests.  I came out with my system test automation framework written with bash shell scripts and awk and felt so “smaht.”  Never mind that I didn’t fully vet my system they way I do the system I test.  Never mind that certain pieces of our system are not stable and can change drastically from one release to the next.  I just knew there was a big green button at the end of the automation tunnel.  I pictured myself pushing  CTRL-T.

Then I started using my creation.  When I realized how fragile my system was, all I could do was sigh and shake my head at several tests my system was telling me had passed even though I knew they had F-A-I-L-E-D.  Not only had they F-A-I-L-E-D, they were false positives.  Maybe you’re thinking, “well this must be what happened to her last year.”  Uh…no.  This was about three months ago.

Now that I realize the fragility of automation, I feel a weight on my back.  Even worse, because this automation is perceived as such a “win,” I have fears that my fragile tests will propagate and turn into the suite of tests Bach describes in Reckless Assumption #8:  tests that maintainers are scared to throw out because they might be important.  I’ve also realized that while I was spending so much time on automation, there was something I forgot.  I forgot that I’m supposed to be TESTING.  This scared me the most.  After all, if I’m not concentrating on assessing my SUT because I’m spending so much time on automating my older tests, how am I really benefitting this project?

Thus, this paper of James Bach’s landed in my mailbox during a very interesting time in my life as a tester.  I feel like I’ve been through this whole evolution over the past year of realizing the power of automation, wanting to automate everything and then realizing that I can’t automate absolutely everything, nor should I.  These realizations triggered an identity crisis.  Am I a developer who is writing tests or am I tester who likes to develop?  I decided that I am definitely the latter, and that I need to back off the hardcore automation for a bit in favor of re-examining my SUT as a manual tester.

My group has recently completed a rather large release, and we’re testing more incrementally.  I have fewer features to test with small releases, so I’ve put down the automation for at least the next couple of cycles in favor of straight-up manual testing.  I printed out every set of testing heuristics I could find, and have been reading through them to find the most appropriate heuristics for my tests.

What has this meant for my testing?  There has been both good and bad.  The worst is that Quality Center utterly breaks with this process.  I am convinced that Quality Center was not designed for a human being engaged in the cognitive process of exploratory analysis for testing.  (My last post was about exploratory analysis.)  I think that Quality Center was designed exclusively for the Waterfall process of software engineering.  To be clear:  that is not a compliment.  Another downside, is that I have had times when I have been looking at the screen thinking, “what’s next?”

The biggest advantage is that, of the bugs I have found, far fewer have been trivial.  Once I removed all thoughts of test automation from my working memory, I have found that much more of my working memory is focused on the process of exploring and testing.  I’ve been living through the observation that, “a person assigned to both duties will tend to focus on one to the exclusion of the other.”

The most memorable paragraph in Bach’s paper is at the end.  He describes an incredibly resilient system of mostly irrelevant tests.  That’s what I was building.  I will probably be automating less, but I’m confident that the automation I write will be more relevant.