Testing and Coding…Concurrently
Posts tagged Software Testing
What is quality? What is art? Part deux
Feb 12th
I’m so appreciative of the discussion that developed from my previous post. I could see that people commenting were really digging deep, so I decided to address some of what was said in this follow-up post.
Here are some of the comments about the definition of quality:
Michael Bolton shared his perspective on Jerry Weinberg’s definition: “To be clear, Jerry’s insight is that quality is not an attribute of something, but a relationship between the person and the thing. This is expressed in his famous definition, ‘quality is value to some person(s).’ ”
Rikard Edgren’s definition: “Quality is more like “good art” than “art”, but anyway: I can tell what “quality to me” is when I see it. I can tell what “quality to others” is when I see it, if I know a lot about the intended usage and users.” Rikard also wrote a post where he clarifies his position a bit.
Andrew Prentice wrote about what he feels is missing from Weinberg’s definition: “I like Weinberg’s definition of quality, but I’m not convinced that it is sufficient for a general definition of quality. Off the top of my head I can think of two concepts that I suspect are important to quality that it doesn’t seem to address: perfection and fulfillment of purpose.”
The definition of quality that I learned is from Stephen Kan’s book, Metrics and Models of Software Quality Engineering. Interesting is that Kan shows a hearty and active disdain for what he says is the “popular” definition of quality. “A popular view of quality,” he writes, “is that it is an intangible trait—it can be discussed, felt, and judged, but cannot be weighed or measured. To many people, quality is similar to what a federal judge once commented about obscenity: ‘I know it when I see it.’ This is sounding familiar, no? Here is where the pretension begins to flow: “This view is in vivid contrast to the professional view held in the discipline of quality engineering that quality can, and should, be operationally defined, measured, monitored, managed, and improved.’ ” Easy, tiger. We’ll look at this again later.
Jean-Leon Gerome’s painting of Pygmalion and Galatea brings this discussion to mind. This is a link to themyth of Pygmalion and Galatea.
Pygmalion and Galatea
I’ve seen this painting in person, at the Met. Interesting to note is that the artist was painting himself as Pygmalion in this painting. (and I like listening to “Fantasy” by the Xx while I look at this.)
The relationship in this painting is not limited to the one between Pygmalion and Galatea, the viewer is drawn into the relationship as well and the artist, himself is also participating. In this painting, Pygmalion has been completely drawn in by his own creation. The artist was so drawn in by the story that he painted himself into it. I was and am still so drawn in by the painting that it is simply painful for me to tear my eyes away from it. It slays me. When I see it, I feel the painting. I guess you could say that emotion is an attribute of this painting, but in this case, I think it’s more. In this case, the emotion is the painting. Why else does the painting exist? Would this painting work at all if the chemistry were missing? I don’t think it would. What Gerome has accomplished here is the wielding of every technique at his disposal to produce a painting with emotion as raw, basic and tantalizing as the finest sashimi.
But there is more to this relationship than just the fact that Gerome has painted himself as Pygmalion. Let’s examine the relationships that exist in this painting and what they tell us. Starting with just the painting, itself, we have the man and the woman locked in their embrace. They are surrounded with many objects. (I encourage all readers to click through to the Met’s web site. Looking at their web-site, if you double click on the painting, you can move around and zoom in and out to get a closer, more focused look.) What do you notice about all of the objects in the room? I’ve no doubt that some of you are wondering if these objects take away from the focus in the painting. If that were the case, if the painting consisted of only the man and the woman, how would we know that the man was an artist? So why do we need these particular objects? The painting could be restricted to just the hammer and chisel so what’s with all the stuff? This is where our relationship with the painting deepens should we choose to follow the breadcrumbs…
An overview of Gerome’s life, clarifies his choices. As a young artist, he spent a year in Rome which he felt was one of the happiest years of his life. At the time that Pygmalion and Galatea was painted, Gerome was grieving over the deaths of several relatives and friends. By surrounding himself with artifacts from his youth, the artist is traveling back in time to a younger, more “Roman”-tic time in his life. However depressed he may have been when he painted this, Gerome was also experiencing an artistic breakthrough in his sculpting career. Notice the breakthrough in the painting? Now that you know a bit more history, how do you feel about the painting? Does it change your perspective? This has made the painting very introspective for me. The emotion that flows from this depiction of romantic love is one of vitality and power. Perhaps Gerome is evoking these feelings as a way of tapping into his own creative powers. I remember thinking to myself when I first saw this painting at the Met, before I knew anything at all about it, “She is rescuing him.”
To describe quality as a relationship gives it a larger meaning and captures something neglected and dismissed by the literature of the “software crisis” era e.g. books such as Stephen Kan’s. Is quality as a relationship mutally exclusive to quality being an attribute of software? I don’t agree with describing quality as just an attribute. To say that quality is an attribute de-emphasizes the holistic approach to quality I try to take and for which I’m assuming Michael, Jerry Weinberg (going by his definition here only), agile, context, et. al are striving. (Full disclosure: I haven’t read any of Jerry Weinberg’s books. That does NOT mean they are not on my list. I just got out of school and the only thing I’m reading lately is visa paperwork so give me a break here.)
The software we test has its creators and has an audience of users as well. Just as Gerome had his own relationship with this painting, developers know what they want to see which leads to the building of their own relationship with the software they make. How does this affect the relationship between the software and its audience
How does value fit into this? I value the painting because of how it makes me feel when I look at it. After the examination I did, I now understand why I value the painting. As someone who is constantly seeking artistic inspiration, I am happy to go where Gerome and his muse take me. What does this say for value in software? Does the relationship between an audience of users and software create value for the audience members whether they are paying guests or not? The more I dig into this definition, the more I like it because it allows for gatecrashers, those who we did not think would be using our software, but who may find it so invaluable, they become our software’s greatest fans.
I’m going to marinate on this while I think about the 2nd part of Andrew’s comment, namely, that Mr. Weinberg’s definition of quality does not address perfection and fulfillment of purpose. After all, Kan’s two definitions of quality of “fitness for use” and “conformance to requirements” are fairly widely accepted in software.
What are you thinking? Is there something missing from Jerry Weinberg’s definition? How does measurement fit into what I’ve been writing about if it fits at all?
I leave you to think about this and the painting above. If you haven’t already, take a few minutes to click through and take a good, honest, langorous look. Put down the twitter, the kid, the spreadsheet, the reality tv show. Take some deep breaths and give yourself a few moments alone with Pygmalion and Galatea.
to be continued…
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=41f1664d-b55f-43c9-9229-d57fae8e64f0)
Look Up, Don’t Look Down: Testing in 2010
Jan 2nd
- Image by -Alina- via Flickr
This post reflects what I’d like to see for software testing in 2010. It is a purely selfish list. Most of what I’ve written about below will find its way into my blog over the next year. The list is not in a particular order, that’s why I excluded numbers for each item. I’m just so damn excited about all of it. (and yes, I stole the title from TonchiDot)Btw, I’ve changed my template, my “about” page and my blogroll.
How does my list compare with what you would like to see?
Testers get fed up with their massive tables of data and turn to visualization
Ok, so no surprise here, but I wouldn’t have picked it for a thesis if I didn’t think it was important. Testing meta-data is all around us, and we’ve yet to fully make sense of it. What is it trying to tell us? If we don’t want to boil everything down to a metric number, that doesn’t mean that the meta-data or the secrets it keeps is going away. In reality, we will only have more meta-data. The challenge lies not only in getting our data into a visualization but also in knowing what and how to explore without wasting time. When should we use a scatterplot vs. treemap vs. plain-and-simple bar graph? This goes way beyond anything the Excel wizard will tell us, but that doesn’t mean we won’t need a little magic.
Functional Programming Shows Up on Our Doorstep
I’ve been seeing devs tweet about FP all year, and I’m quite jealous. If a dev gave you unit tests written in Haskell or Erlang, what would you do? Testers aren’t the only ones with meta-data overdrive. Our massively connected world is producing too much info to be processed serially. Get ready for an FP invasion. Personally, I’m looking at Scala.
Weekend Testing Spreads
Indie rock fans will smell BS if they see an indie rock countdown for 2009 without Grizzly Bear (had to work it in somehow). Weekend Testing is obviously the Grizzly Bear of Software Testing for 2009 and their momentum sets a blistering pace. Markus Gaertner has just announced that it’s expanding to Europe and I’m certain it will spread across the Pacific as well. This is a bottom up method for learning how to test, and I hope that instructors of testing take note. I am no expert at testing and want to do whatever I can to set the bar as high as possible. Hey Weekend Testers, count me in!
Testers who don’t blog start to care about their writing skills
With an emphasis on tools that get software process out of our frakking way, we’ll be left with our writing. Ouch. What’s a comma splice? Hey, I’m going for my Strunk & White
Links Between the Arts and Software Testing Will Be Strengthened
Chris McMahon started us off with his chapter in Beautiful Testing
More testers start to care about understanding the fundamentals of measurement and the basics of statistics
Think fast: What is the difference between ratio and proportion? When does the mean not tell an accurate story about a set of numbers? It’s very clear that there are some serious pitfalls in the usage of metrics. What I haven’t seen is lots of testers that have a thorough understanding of basics such as levels of measurement or what a distribution will tell you. I wonder how many testers back away from using these because they don’t understand exactly how they can be harmful or because they just don’t understand exactly how they work in the first place. One assignment I’ve given my blog for the year, is to tackle some basics as applied to testing. Rejecting metrics because you see how they can harm is one thing, rejecting metrics because you don’t understand them is unfortunate. If you count yourself as a tester who is not totally comfortable with math, you’re not alone and, believe me, I understand how you feel.
Collective Intelligence Comes into Play
If I had my way, this list would be vote-able and each reader would have the ability to vote items to the top or bottom. Wouldn’t that be interesting? Unfortunately, I don’t have that…today ;o) But we’re so close! If we’ve got the technology together to analyze the hell out of our blogs through web analytics, what about our tests? I’m picturing myself writing out tests in a wiki with a zemanta-like tool suggesting tests from similar stories that have previously caught bugs. I might not always use these suggested tests, but it would be a great help for brainstorming.
I’ll have an Open Source Project Up and Running for Visualizations to be Used with Testing
This is not a resolution, it’s something I didn’t finish from last year. I am just so late on this. Oh well, giving myself a conduct cut. Seems I had a little conference talk to deal with which quickly morphed into a little talk at Adobe, followed by a little talk at Microsoft. Needless to say, I’ve got some unfinished business that has to do with treemaps. The PNSQC experience was a semester in and of itself. Time to get back into the visualizations.
It’s not lost on me that my last few posts have been sort of personal and high-level. I’ve had big changes and events happening in my life, which has made maintaining focus, well, difficult. You’ll hear all about it soon enough. Trust me, it’ll be good.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=ea549488-64ee-4cd3-bc1d-41c3d68dd0c2)
PNSQC Slides and Paper Are Up
Dec 10th
My thesis defense is tomorrow which is why I haven’t posted in a couple of weeks. If all goes well, I’ll be posting a link to that in the next week.
This is just a quick post to say that my paper and presentation have been added to the PNSQC web-site, along with everyone else’s paper and presentation: click here and have fun exploring.
Since I don’t read from powerpoint slides, you won’t find much in the way of explanatory verbiage in the slides, but I’m happy to answer questions if there’s something you’d like me to clarify. Ideally, I would download the paper, and read the paper while you have the slides up. They told me to make all of the pictures for the paper extremely small and grayscale…sigh. Kind of kills the whole visualization aspect of my paper, but I understand they had good reasons for asking. This is exactly why Edward Tufte took out a 2nd mortgage on his house, and self-published his first book. I would write more about that because it’s a post in and of itself, but my recursion ain’t workin’, gotta go!
Visualizing Defect Percentages with Parallel Sets
Sep 24th
Prof. Robert Kosara’s visualization tool, Parallel Sets (Parsets) fascinates me. If you download it and play with the sample datasets, you will likely be fascinated as well. It shows aggregations of categorical data in an interactive way.
I am so enamored with this tool, in particular, because it hits the sweet spot between beauty and utility. I’m a real fan of abstract and performance art. I love crazy paintings, sculptures and whatnot that force you to question their very existence. This is art that walks the line between brilliant and senseless.
When I look at the visualizations by Parsets, I’m inclined to print them off and stick them on my cube wall just because they’re “purty.” However, they are also quite utilitarian as every visualization should be. I’m going to show you how by using an example set of defects. Linda Wilkinson’s post last week was the inspiration for this. You can get some of the metrics she talks about in her post with this tool.
For my example, I created a dataset for a fictitious system under test (SUT). The SUT has defects broken down by operating system (Mac or Windows), who reported them (client or QA) and which part of the system they affect (UI, JRE, Database, Http, Xerces, SOAP).
Keeping in mind that I faked this data, here is the format:
DefectID,Reported By,OS,Application Component
Defect1,QA,MacOSX,SOAP
Defect2,Client,Windows,UI
Defect3,Client,MacOSX,Database
The import process is pretty simple. I click a button, choose my csv file, it’s imported. More info on the operation of Parsets is here. A warning: I did have to revert back to version 2.0. Maybe Prof. Kosara could be convinced to allow downloads of 2.0.
I had to check and recheck the boxes on the left to get the data into the order I wanted. Here is what I got:
So who wants to show me their piechart that they think is perfectly capable of showing this??? Oh wait, PIE CHARTS WON’T DO THIS. Pie Charts can only show you one variable. This one has 4.
This is very similar to the parallel coordinate plot described by Stephen Few in Now You See It and shows Wilkinson’s example of analyzing who has reported defects. She was showing how to calculate a percentage for defects. See how the QA at the top is highlighted? There’s your percentage. Aside from who has reported the defects, Parsets makes it incredibly easy to see which OS has more defects and how the defects are spread out among the components. If I had more time, I would add a severity level to each defect. Wouldn’t that tell a story.
Parallel Sets is highly interactive. I can reorder the categories by checking and unchecking boxes. I can remove a category by unchecking a box if I wish.
By moving the mouse around, I can highlight and trace data points. Here I see that Defect 205 is a database defect for Mac OS X. Although I didn’t do it here, I bet that I could merge the Defect ID with a Defect Description and see both in the mouse over.
Parallel Sets is still pretty young, but is just so promising. I’m hoping that eventually, it will be viewable in a browser and easier to share. Visualizations like this one keep me engaged while providing me with useful information for exploratory analysis. That’s the promise of data viz, and Parallel Sets delivers.
Automated Test Confessions
Sep 15th
My life as a tester is evolving and I’m feeling less like a newbie. I’ve also had yet another “James Bach” moment. This time, a friend of mine forwarded me an article her husband had read and passed along to her. He’s a developer who, I guess, is going through the whole, “unit testing: what does it all mean?” phase of life. The email contained a few links. Among them was James Bach’s paper from 1999, “Test Automation Snake Oil.” As I read through, what I now know is a classic, I realized that I’d been recognizing some of what Bach writes about in my own tests. His paper highlighted much of what I’ve come to think about my own tests.
At this point, I’ve been a software tester for about two and a half years. From my perspective, this is not a very long time. The past year, however, has been insanely intense for me intellectually and academically. There have been many times during the past year when I have felt myself back in the Interdisciplinary Studies program I took as a Freshman and Sophomore at Appalachian State University. We were given 100+ pages a night of reading per night which ranged all over the humanities and sometimes sciences. This reading was in addition to lectures and other “programming” we were expected to attend. Between the Software Engineering classes, the job as Software Tester and the runaway fascination with Data Visualization, I’ve put myself through a similar gamut of reading and working. This time my activities have centered around software, computers and testing. The result of this for my job as a software tester is that I am not the tester I was last year.
At all.
Previously, I was really smitten with HP Quality Center because it gave me structure for which I was desperately searching. This was a great improvement over the massive, disorganized and growing spreadsheets surrounding me that contained all of my test information. All of my tests could finally be organized, and thanks to the HP online tutorial I knew my tests were organized well. I felt liberated! Now I could stop concentrating on how the tests should be organized and concentrate more on the actual testing itself.
This led to the realization that there was NO WAY I would EVER be able to test EVERYTHING. I was frustrated. Why were my test cycles so short? Why did I always feel like a bottleneck? Was I not good enough at testing? Was I not fast enough? “I must find a way to test faster,” I told myself.
After attending the 2008 Google Test Automation Conference, I turned to unit testing and automation. I mean, I can write code. It doesn’t scare me at all. This doesn’t mean that I’m great at it, but I enjoy it enough to spend significant amounts of time doing it. I decided to use my coding skills to write repeatable tests that could be run over and over and over again. After all, I’m pulling my group, by the hair, towards automated builds and smoke tests have to be automated. Business just LOVES these. I was told that it was making my group look really good to have automated tests. I came out with my system test automation framework written with bash shell scripts and awk and felt so “smaht.” Never mind that I didn’t fully vet my system they way I do the system I test. Never mind that certain pieces of our system are not stable and can change drastically from one release to the next. I just knew there was a big green button at the end of the automation tunnel. I pictured myself pushing CTRL-T.
Then I started using my creation. When I realized how fragile my system was, all I could do was sigh and shake my head at several tests my system was telling me had passed even though I knew they had F-A-I-L-E-D. Not only had they F-A-I-L-E-D, they were false positives. Maybe you’re thinking, “well this must be what happened to her last year.” Uh…no. This was about three months ago.
Now that I realize the fragility of automation, I feel a weight on my back. Even worse, because this automation is perceived as such a “win,” I have fears that my fragile tests will propagate and turn into the suite of tests Bach describes in Reckless Assumption #8: tests that maintainers are scared to throw out because they might be important. I’ve also realized that while I was spending so much time on automation, there was something I forgot. I forgot that I’m supposed to be TESTING. This scared me the most. After all, if I’m not concentrating on assessing my SUT because I’m spending so much time on automating my older tests, how am I really benefitting this project?
Thus, this paper of James Bach’s landed in my mailbox during a very interesting time in my life as a tester. I feel like I’ve been through this whole evolution over the past year of realizing the power of automation, wanting to automate everything and then realizing that I can’t automate absolutely everything, nor should I. These realizations triggered an identity crisis. Am I a developer who is writing tests or am I tester who likes to develop? I decided that I am definitely the latter, and that I need to back off the hardcore automation for a bit in favor of re-examining my SUT as a manual tester.
My group has recently completed a rather large release, and we’re testing more incrementally. I have fewer features to test with small releases, so I’ve put down the automation for at least the next couple of cycles in favor of straight-up manual testing. I printed out every set of testing heuristics I could find, and have been reading through them to find the most appropriate heuristics for my tests.
What has this meant for my testing? There has been both good and bad. The worst is that Quality Center utterly breaks with this process. I am convinced that Quality Center was not designed for a human being engaged in the cognitive process of exploratory analysis for testing. (My last post was about exploratory analysis.) I think that Quality Center was designed exclusively for the Waterfall process of software engineering. To be clear: that is not a compliment. Another downside, is that I have had times when I have been looking at the screen thinking, “what’s next?”
The biggest advantage is that, of the bugs I have found, far fewer have been trivial. Once I removed all thoughts of test automation from my working memory, I have found that much more of my working memory is focused on the process of exploring and testing. I’ve been living through the observation that, “a person assigned to both duties will tend to focus on one to the exclusion of the other.”
The most memorable paragraph in Bach’s paper is at the end. He describes an incredibly resilient system of mostly irrelevant tests. That’s what I was building. I will probably be automating less, but I’m confident that the automation I write will be more relevant.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=c65ab5b3-a838-4144-b49c-2455d201ee13)
Underpants Gnomes Among Us: Exploratory Analysis for Visualization and Testing
Sep 9th
Here’s a picture of tester dog, Laika, with Dr. James Whittaker’s new book, Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design.
Tester Dog
In reading through Stephen Few’s new book, Now You See It,
Stephen Few does a great job of explaining exploratory analysis with pictures:
Exploratory Analysis
Half of the people reading this now understand the underpants gnome tie-in. For those who don’t get it, here’s a link to the original South Park clip (NSFW).
Jokes aside, I’m going to start with the picture, and discuss what this says to me about testing and see if it meshes with what JW’s definition of exploratory testing. I will then look at how this applies to visualization. At the end, the two will either come together or not. At this point, I’m not sure if they will. I’ll just have to keep exploring until I have an answer or a comment telling me why my answer is crap (which is fine with me if you have a good point).
Starting with the picture and testing. I’m assuming the “?” means “write tests.” The eyeball means analyze. The light bulb is the decision of pass or fail. The illustration of directed analysis looks like the process HP Quality Center assumes. QC assumes you’ve primarily written tests and test steps before testing based on written requirements. Then you test. After you’ve tested, you have an outcome.
The second line for “exploratory” analysis looks like a much more cognitive and iterative process. This says that the tester has the opportunity to interact with the system-under-test (SUT) before formulating any tests(eyeball). After playing with the SUT, the tester pokes it with a few tests (“?”). At this point the tester may decide some stuff works and keep poking or decide that some stuff has failed and write defects(light bulb.) Chapter 2 of Exploratory Testing describes how JW defines exploratory testing: “Testers may interact with the application in whatever way they want and use the information the application provides to react, change course and generally explore the application’s functionality without restraint (16).” So far this is looking very similar.
Now that I’ve looked at how the exploratory analysis paradigm applies to testing, here’s how it applies to visualization. As an example visualization, I’m looking at a New York Times graphic, How Different Groups Spend their Day. When I open this graphic, I can see that it’s interactive, so I immediately slide my mouse across the screen. I notice the tool tips. Reading these gets me started reading the labels and eventually the description at the top. Then I start clicking. The boxes on the top right act as a filter. There is a also a filter that engages when a particular layer is clicked.
Few’s point in describing directed analysis vs. exploratory analysis is that in the wild, when we look at visualizations, we use exploratory analysis. It’s not like I knew what I was going to see before I opened the visualization. Few describes the process known as “Schneiderman’s mantra” (for Ben Schneiderman of treemap fame) in more detail saying that we make an overall assessment (eyeball), take a few specific actions (“?”), then reassess (eyeball). Although Few doesn’t say that there is a decision made at some point in this process, I’m assuming there is because of the light bulb in the picture (84).
Recently, Stephen Few asked for industry examples of people using visualization to do their work. Some of the replies were from the airline industry, a mail order warehouse and a medical center. Software engineers should be included in this mix and apparently from page 130 in JW’s book showing a treemap of Vista code complexity, already are. Given that both use the same form of exploratory analysis, I can see why.
Exploratory analysis of software testing and visualization diverge, however, when you look at the scale of data for which each is effective. Visualization requires a large dataset. This could be multiple runs of a set of tests or, as in JW’s example, analysis of large amounts of source code. Exploratory testing as JW describes can occur at a high level such as in the case of a visualization or at the level of an individual test.
One thing my exercise has shown me for sure is that I have to read more of Exploratory Testing.
Training without a Net
Aug 17th
- Image by StarrGazr via Flickr
Those of us who like to be actively involved in the meetings we attend surely notice the effect a giant flat screen presentation has on meeting conversation. It can be stultifying. In the case of training, the presence of slides on a flat screen is the equivalent of showing a really bad talk show like Jerry Springer. Nobody learns or remembers anything unless there is a fight or petty squabble.
This week I had to train some of our system’s users on how to write usable bug reports. I had an outline and an example that I thought was interesting enough to keep people awake and focused on the topic. In order to make the information stick, I decided to go without slides, and see what it got me.
Preparation
You would think there would be less to prepare, but in truth, you have to come up with something that will keep your audience occupied. In my case, I put together a group exercise by creating a scenario involving a bug.
Tip #1: don’t create a trivial scenario
My scenario was too far removed from our daily situation to ring true. I could tell that some of the users felt I was wasting their time by having them work with an example they felt was “too simple.” Once I noticed this, I threw it out and said, “let’s just talk in terms of our system.” This seemed to make people more comfortable.
Tip #2: have an outline IN REALLY BIG LETTERS
Chris McMahon suggested this over twitter (thx!) and it did help. The only problem with my outline, is that I couldn’t see it very well without picking it up. There wasn’t much on it anyway so if I could have easily enlarged the font.
Keeping Your Audience Awake
Chris also suggested that I move around and stay animated. Since I have natural talent as a drama queen, this is typically not a problem for me, but is worth a mention. Raise your hand if you’ve seen a speaker able to put you to sleep merely with the narcoleptic power of their voice. There are also speakers who mumble in which case you won’t be able to understand what they are saying even if their voice keeps you awake because its “nails on a blackboard.” In that case, maybe you really do need the slides. Am I getting too off-topic with this?
Keeping the Focus on Topic
Once the flat screen has been removed, you will find that people have stuff they want to get off of their chest. If you are the only person holding meetings without slides: Guess what? They will choose your meeting to unload. I noticed people communicating more about our defect process than I had anticipated, and not necessarily in ways that I had planned.
Tip #3 Be flexible and open to some change in the agenda
Mid-way through our exercise, we had chucked my example and were discussing the pieces of our system that should be documented in describing the environment of a crash. The users were talking to developers about the challenges they have in reporting their environment and I noticed some holes in our defect process. We were still on topic, but I let the users talk to us about what they typically see when they have problems running the system.
Tip #4 Don’t let meeting participants change your whole agenda
Since people were talking to each other and sharing information about our software process, the discussion was pretty intense. I found myself circling back to my outline a number of times. Some discussion was worthwhile, but obviously needed to happen in a separate meeting. Sometimes attendees will resist moving on, but I find that a quick, “we’ll schedule another meeting, moving on to <next point goes here>…” will get the job done.
Would I do this again?
Absolutely. Even though I write about and study visualization, there are times when we really do need to sit in a circle with the talking stick and communicate with each other. In fact, even Prof. Edward Tufte recognizes that there’s no need to have the monitor on all the time. In his lectures, he shows you the graphic, tells you what to look at and then TURNS IT OFF.
Training without slides is not for the faint of heart, but, in the end, I think my work colleagues respected the fact that I wanted them thinking through the material and not just gaping at a flat screen.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=a455cc98-2fba-4096-b463-f155187ce3ac)
Plagues aren’t just for blog posts
Aug 3rd
- Image via Wikipedia
For the past couple of months, James Whittaker has been writing about the “plagues of testing.” As he’s been posting, I’ve been reading through a book about a real plague.
As software testers, we see a system from a perspective that developers and business types rarely and may never see. We know our tests, we know how well they ran. We know our system under test and which components are picky. If you are like me, and have access to the code base, you also know the code. In the case of both the system and it’s code you know what should be better. Sometimes this is not a big deal, but sometimes it is a warning that it’s time to polish the resume. I have not seen this, personally, but I know that there are testers who find themselves in this position.
John Snow was a doctor in this position. He was an expert in the new art of anesthesiology in 1850′s London. He was also deeply involved in the study of cholera and concluded that it was a water born illness. This was very much counter to the prevailing theory of the time that cholera was passed through a sheer volume of stench or miasma (I can hear Dr. Evil saying this word). Health authorities in London, were so convinced that miasma, or extreme smellyness, was the reason for disease that they passed a law in 1848 requiring Londoners to drain their waste into a sewage system that would deposit into the Thames river. Unfortunately, the Thames was also a main source of drinking water for the city. Snow knew all of this and could see a health crisis in the making.
In 1854 when a major outbreak of cholera erupted in a neighborhood very close to where Snow lived, he conducted a thorough investigation in order to prove this theory that cholera was passed through water. He had a list that detailed the names and addresses of 83 people who had died from cholera. He also had an invaluable resource of detailed information about the neighborhood’s residents in the form of local clergyman Henry Whitehead. While Whitehead tracked down and questioned everyone he possibly could about their drinking habits, Snow analyzed this data to form patterns of who had been drinking the water, who had died and, just as importantly, who had NOT died. Not only were Snow and Whitehead able to convince the local parish board to remove the handle of a contaminated water pump, they knew their data so well that they were able to figure out the index, or original, case that had started the outbreak.
After the outbreak had subsided, Snow put the analysis from his investigation together with a very famous map into a monograph that circulated among London’s health professionals. His monograph slowly but effectively turned the tide of thinking among health professionals away from the miasma theory. This and a very smelly Thames convinced authorities to build a new sewer system that drained into the sea.
Frequently, when I talk to people about data visualization, they always ask me how I know what to visualize. The Ghost Map
Johnson and Whitehead could not drag the dead people’s relatives in front of London’s public health community and force the doctors to listen. These days doctor’s short attention span is because of insurance, but I’m sure there were other reasons back in the day. A good visualization does not take a long time for the viewer to process. That is their special power. Snow’s map is much more concise than a table of 83 names and addresses along with their individual stories. Visualization can quickly show your groups of tests that are failing. It can show that severe defects are increasing, and not decreasing, over time. Business may drive the ship/do not ship decision, but a good tester will know why a seriously ailing system is in so much trouble. A great tester can effectively communicate this to a business team.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=ae7fa8d8-583d-4320-a177-75aa39da53f0)
Test Patterns
Jul 5th
This will be the next-to-last week of my design patterns class, and I’m working on my final project. We were told to pick some category of design pattern and to do write-ups of the patterns in our category. Some of the example categories were security patterns, anti-patterns and concurrency patterns. I chose test patterns so it would be reusable for work.
So far, what I’ve found is that “test pattern” can mean just about anything in testing. In fact, I question whether there is really a difference between “test heuristic” and “test pattern.” It’s all just ways of categorizing abstract testing concepts that can reapplied in difference scenarios, right?
I looked up test patterns in How We Test Software at Microsoft
This book not only has test patterns, but categorizes the test patterns into several chapters. Included are Results-Oriented Test Strategy, Classes, Reusable Components, Subsystems, Integration, Application Systems and Regression Testing. As an example, the Integration chapter contains the patterns Big-Bang Integration, Bottom-Up Integration, Top-down Integration, Collaboration Integration, Client/Server Integration, and a few more.
As I’ve been schlepping through this huge book, I’ve noticed just how technical and detailed it is. This leads to my next question, how many people use test patterns knowingly as test patterns? It’s not like most of us in testing trained for this, and the only place I’ve found straight up definitions of test patterns aside from the microsoft post is in this particular book. When I use Quality Center, it’s not like I’m separating out my tests by pattern or heuristic. Should I be? I’ve also read of testers who felt that their success was due to the fact that they weren’t following a pattern, but acting as a user. But then, isn’t that a pattern too?
I’ll post some of the stuff I’ve done for this project in a week or so. Very interested in what people think about using test patterns for testing.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=aa3cbd46-233c-46b3-89c0-0be2478431e8)
Submitted an Abstract to PNSQC
May 1st
I’m posting the abstract I just submitted to PNSQC. It’s also the abstract of the thesis I’m writing for my Masters. I’ve submitted a poster to the Grace Hopper Conference, but never before have I submitted a full-on paper requiring a full-on presentation. I chose PNSQC for 2 reasons: the focus is more on the practical side, unlike some of the ACM conferences and the conference is in Portland, Oregon. God, I love Portland.
Anyway, here is what I submitted:
Visualizing Software Quality
Moving quality forward will require better methods of assessing quality more quickly for large software systems. Assessing how much to test a software application is consistently a challenge for software testers especially when requirements are less than clear and deadlines are constrained.
For my graduate research and my job as a software tester, I have been looking at how visualization can benefit software testing. In assessing the quality of large-scale software systems, data visualization can be used as an aid. Visualizations can show complexity in a system, coverage of system or unit tests, where tests are passing vs. failing and which areas of a system contain the most frequent and severe defects.
In order to create visualizations for testing with a high level of utility and trustworthiness, I studied the principles of good data visualizations vs. visualizations with compromised integrity. Reading about these lead me to change some of the graphs that I had been using for my qa assessment and to adopt newer types of visualizations such as treemaps to show me where I should be testing and which areas of source code are more likely to have defects.
This paper will describe the principles of visualization I have been using, the visualizations I have created and how they are used as well as anecdotal evidence of their effectiveness for testing.