Marlena's Blog

Previous to my attendance at PNSQC, I had a couple of “Bach moments.”  PNSQC was no exception.  I took his class and believe I was able to distinguish myself as thoroughly mediocre.  Is it just me or does that man have “laser focus” when someone says something that interests him?  This changed dramatically, however, once I hit the first poster session.

I knew people would look at my poster because it had pictures on it, but I wasn’t quite prepared for the volume of interest.  It started right away.  As the last pushpin was going into place, I heard someone behind me say, “um…where’s the quality.”  I turned around to see a) baseball cap b) glasses c) beard.

OMG…JAMES BACH!!!!!!!

“Excuse me, but I’m looking and I don’t see any quality here.”  He glanced all around my poster from the top left corner to the bottom right corner. “Nope, still no quality.”

Meanwhile, I’m still processing who the hell this is.  Love him or hate him, you have to admit that he sits among the “Monsters O’ Test.”  As he keeps drilling me, I notice more people showing up behind him.  When Monsters O’ Test speak, people listen.  I felt like Peter Billingsly, frozen on Santa’s lap in The Christmas Story.  The crowd turned into a gang of elves.  I know that Mr. Bach was asking me about software quality, but really, all I heard was, “HO! HO! HO!”  I think he realized that I was completely petrified, because he kind of backed off.  My reviewer had made his way over to my poster and mentioned that Mr. Bach should read my paper.

I honestly think I started babbling, because I don’t remember a word I said.  There are a few things I know about myself including the fact that I have a complete inability to handle meeting people I respect. My fangrrl tendencies are directly proportional to how much I respect someone.   This is because my respect can only be earned.  (Hey, Atlas Shrugged fans, y’all know what I mean.)  I never respect someone just because I hear they are “important.”  The flip side is that people I respect mean a lot to me.  As an example, I am such a fangrrl of the book How We Test Software at Microsoft since I read it last winter.  I learned so much about testing from this book, and what I learned, I apply every day.  Last summer, Alan Page posted that he likes my blog and was looking forward to seeing me at PNSQC.  The post showed up in my reader, and I swear I nearly had a heart attack.  It was so flattering and unexpected.  I’ve now met Alan (Hiiiiii!), and have managed to calm my fangrrl tendencies somewhat.  This is not to say that there wasn’t some maniacal giggling on my part as Alan showed me around the Microsoft campus last week.  Unfortunately, fangrrl behavior is a dimmer and not an on/off switch.

But back to the matter at hand: there is an unanswered question that was asked of my poster.  Where’s the quality?  This is an excellent question and is, in fact, a question I can’t answer.  My inability to answer this question highlights the backstory of my conference presentation.  When I submitted my abstract to PNSQC I did not have a treemap of tests or any idea if I would be successful at making one.  I also didn’t know if this would be a valid way to look at tests.  Titles are important.  They are a succinct description for a project.  In my case, I knew that I needed a good title if I had a snowball’s chance in hell for my presentation to be accepted.  I decided to go big.  “Visualizing Software Quality” was the biggest title I could think of for what I was (and still am) attempting.  That is, visualizing all aspects of the software tester’s experience.  I don’t want to explore through lists, I want to explore through visualization.

So where’s the quality?  I no longer feel that this is the best question or the best title for my investigations.  I’m focusing on visualizing aspects of software testing.  What is the best way to get a “big picture” view of tests, defects, customer annoyance, wtf’s per minute or any other multi-variate information that testers must face?  What information do we really care about and how is it best presented in an interactive and graphic way for us to explore?  If we are looking at a “big picture” view of tests and defects, what are looking for and where do we want to go after we find something interesting in our data?  This is entering the realm of “exploratory data analysis.”   I can’t answer Mr. Bach’s question, but I’m quite flattered that he was intrigued enough to ask me.  And I don’t want a football or a Red Rider BB Gun for Christmas.  I think I’d much rather find John Tukey’s book, Exploratory Data Analysis under my Solstice tree.

This post is about feedback I received regarding my presentation at PNSQC and my other presentations at Adobe and Microsoft.

Before I mention my feedback, though, I would like to thank Lanette Creamer and Alan Page. They welcomed me to their companies with such hospitality and respect. These are both extremely busy people who took significant amounts of time out of their day to show me around and make me feel at home. This kind of attention is extremely motivating for me, and shows that they lead from the heart.

On to feedback:

What I didn’t know about PNSQC prior to arrival is that they have a feedback system. Whenever an audience member leaves a talk, they can submit a feedback card. The cards are green, yellow and red. Green indicates a good talk with no problems. Yellow means that the presentation was pretty good and may have had some issues. Red means that the presentation was a waste of time.

What did I think of my presentation? I was a nervous wreck, which is extremely out-of-character for me in speaking to people. The moment I started speaking I forgot to breathe. Once that happens, it takes me a good 5 to 10 minutes to recover. My content was challenging to present because I didn’t have bullet points in my slides. In retrospect, I would have spent more time memorizing an outline since I wasn’t going to have that type of aid on screen. My presentations at Adobe and Microsoft were far better.

Here’s a bar graph of the cards I received.

When people turn in cards, they can also write comments on the cards.

Comments on green cards:
1. Good talk: a) should all metrics be visualized? b) need clear goals or don’t bother c)green card mostly for thought production.
2. Great content. Practice your presentation skills

Comments on yellow cards:
1. Too slow
2. Great potential, but message didn’t come across clearly. How to read new visual organizations of data is very important instruction to being able to interpret the pics.

Comments on red card:
Visualization shown was a hard way to view list of failures and counts grouped by component.

I also had some comments from my talks at Microsoft and Adobe. These are comments I remember and are as accurate as possible. Since this post is my bucket for feedback, I’m not answering criticism in this particular post. Don’t worry…you’ll be hearing PLENTY from me later, but this post is about what I heard, not what I think. For anyone who’s been waiting to add their $2, the comments for this post would be a good place.

Alas…the criticism:

The process I’m using is static, although at Microsoft there is more of an interest in real time data.

The purpose of these visualizations is unclear.

Someone on twitter mentioned that presentation started slowly for him.

There were two guys in the back who seemed to have read some stuff about visualization and both were highly critical of my work. I didn’t get to talk to them afterward which is a shame because the only way for me to refine my work is by hearing good criticism. If either of you two dudes is reading, please get in touch.

My example with Parallel Sets is using fake data

None of my examples is being used IRL.

There are not enough configuration options

Some took issue with the title of my paper and presentation. “Where’s the quality? I’m not seeing it.”

Why is the white on the treemap of tests at all?

where is the zooming?

Someone did find a bug in the strip treemap algorithm. This is an internationalization problem. A very nice man was trying to understand the order of the strip treemap. The items in this type of layout are ordered horizontally, and this man was concerned with the vertical ordering. He was Asian and I know that Asian writing is vertically ordered and not horizontally ordered. There is currently no strip layout algorithm I know of that will order the items vertically. Since we live in a global economy, I’m considering this a bug of the strip layout as currently implemented.

I’ve just given my talk on Visualizing Software Quality at the Pacific Northwest Software Quality Conference.  The following links are resources from my talk.  If necessary, I’ll add more to them later.  I’m not posting my slides because, well, most of it’s already on my blog.  Don’t feel you’ve lost much from not seeing them because I didn’t use any bullet points at all.  I’m happy to address any questions or issues in the comments.

Update: Today I presented at Adobe and showed off the really awesome Newsmap This treemap has a very interesting “about” page that I hope you will also peruse. Playing with the filters on the newsmap is also a fascinating exploration of what types of news stories are popular in the media of different countries.

Books
Edward Tufte – The Visual Display of Quantitative Information
Stephen B. Johnson – The Ghost Map

Parallel Sets

Treemap Software

The following are available for free:
JTreeMap @benoitx
Sonar
JavaScript InfoVis Toolkit
Treemap 4.1.2 – not for commercial users

The following are commercial:
Panopticon (Don’t know the price for this one.)
The Hive Group  (They called me and said, “it’s really cheap at $20,000.”  I said, “ha, ha.”)

Horizon Graphs
Panopticon invented this visualization, so I’d ask them about it.
Here is Stephen Few’s blogpost on the subject.

In the previous post, I discussed the app in Matt Heusser’s latest test challenge and questions I asked about it.  In this post, I will show my overall test strategy and plan.  I know he says there’s a prize, but I’d much rather have some feedback regarding what about this test plan sucks and what doesn’t suck.  I have the big girl shoes (with laces) on so there’s no need to hold back.
From the questions and answers Matt provided, I’ve made a few categories that will influence the focus of my testing.

Prioritized Testing:
Since the primary goal is testing that users can see changes in signals and changes, this will be what I focus on first and most.  Before I saw that this was the primary goal, I questioned whether I should really plan to test every filter combination.

Explicitly described as something to be included in testing:

Every time there is a filter change it must be checked in 3 places:
1.  activities widget
2.  google reader feed
3.  reader widget feed

Reply to signal from activities widget with post
Reply to signal from activities widget with private message

I know I will NOT be testing:
Posting a message or private mail through the text box.

Not prioritized, but still important:
Opening a page through a link in a signal and opening a person’s profile through a link in a signal.  I see one possible defect for this case.  If you look at the picture, everyone but Gabe Wachob’s name is underlined.  I would ask why or log a defect.

Navigating through newer, newest, older pages of posts.

Using the wrench icon to display 5, 10, 15, 20 or 25 activities (queue The Phenomenal Handclap Band)

All tests listed by the product owner.

Overall Test Strategy:
I want to automate the bulk of this testing because you will see below that I have several user scenarios and 3 different views.  The setup and teardown of automated testing such as that used with SeleniumRC + TestNG means I could have tests logging in as my 3 different users and checking each of the three different views as filters are changed in different browsers.  Since I’m using code, I would expect to create tests for each combination of filters.

This doesn’t mean that I wouldn’t do any manual testing.  Markus Gaertner suggested in his blog that he would include time-boxed exploratory testing, and I think this is an important activity that cannot be skipped.   In fact, there are a couple of tests that I’m not sure would be suitable for automation.  I typically do some of this on any feature I’m testing before I finish with my test planning as a way to inform my testing and shake loose a few more questions.  One great aspect of Selenium IDE which I recently discovered is that the IDE will create code for use in TestNG projects.  I can easily see myself simultaneously testing manually and creating code from those manual tests for Selenium RC/TestNG automation.

Risks
My performance testing coverage will be minimal.  It’s just not something I know that much about.  I’m not saying that this app shouldn’t be performance tested, I’m just not the one to tell you how to do it.  I would have to discuss this with the PM and work something out.  Maybe get someone from another project to help and show me some basics.

In the case of the rss feed, usage of Google reader requires a good account.  I guess you can fake these, but I’m pretty sure it’s against their EULA.

Since I don’t do testing of web app user interfaces and funtionality on a regular basis, my estimate for how long this testing would take is a WAG, at best.  I would prefer to have time factored in for creating automation that might, at first, take a little longer to write, but would be reusable.  Time that is not given to me for automation is time that I would add into testing the app for subsequent iterations.  Matt mentions that this testing should take 4 hours, I would give it 8 including automation and setup of test environment.  Again, that’s a total WAG.

Test Environment
Because I’ve defined different types of users and different types of views, I would set up an environment for this test with a set of networks available to these 4 users:

1.  The user who doesn’t have many feeds or friends, maybe they only have 1 of each
2.  The user who has many people on their network but only 1 network
3.  The user who has many people spread across many different networks.
4.  The extreme user who has extreme values of people spread across and extreme number of networks.
5.  User with 1 signal
6.  User with lots of signals

I would also set up a reader widget for each user.  If possible, I would set up a google reader for each user, but I’m not sure this is possible, so there would likely only be limited testing on the google reader feed.

Once users, networks and rss feeds are set up, I would build test fixture objects for these users and each of the views that will need to be checked.  (This is an area where one could really go nuts with design patterns and create some extremely re-usable classes and interfaces for modeling user behaviors, BUT, I have yet to see this in the wild, and I’m assuming I won’t have time to get into this type of elaborate set-up.  I sincerely hope, though, that there are places that do this and that this is where automated testing is headed.)

Test Plan
Matt suggests in his blog that these can be brief one-liners.  This ain’t a novel, so I’m keeping each test short.

User scenario based tests:

Filter testing:
For each user, filter for every available combination and verify that only the correct signals appear from the correct network.  I would automate this and have it run in the background while I do other testing.  I would also perform a manual test for each user with up to 3 different types of filters. I would do this test first because it’s the priority for testing.

For users with 1 network:
Verify “show/from/with” appears with the filter
Verify that there is no drop down box for the users with 1 network

For users with more than one network:
Verify that there is a drop down box for users with more than 1 network
Verify that each network appears in the drop-down box

For all users:
Verify consistent choices in “showing edits” and “from everyone” boxes.
Verify resetting number of visible signals to each value 5 – 25
Make several choices in the filter drop down boxes and verify that page reflects pages in reasonable amount of time.
Check how long it takes for changes made in the widget to be reflected in the user interface and rss feeds.

For user with lots of signal messages:
Verify that newer/older command works with display set for each choice of 5 – 25 messages
Verify that navigation works if user thrashes back and forth in the navigation

For user with no signal messages:
Verify that newer/older commands are not highlighted

User-independent tests:
Verify widget is resizable
Verify widget can be minimized, expanded when minimized
Verify widget can be closed
Verify “post message” text box can be collapsed/expanded with button at top left.
NOTE: I had problems with this one IRL.  If you collapse, the button disappears altogether so you have to reply to someone to get it to re-expand.  What does someone with no messages do?  Would they be able to re-expand? I have a screen shot if you want it.
Verify user profiles can be accessed by clicking on their name, clarify difference between underlined and not underlined names.
Verify wiki pages can be accessed by clicking on them.  Possibly, using automation,  select     signals on a page randomly and perform this test (stick ’em in an array and do random select)
Verify signal can be replied to using the button on the right
Verify signal poster can be privately messaged using button on the right
Verify that a signal can be trashed.
Verify x minutes ago for signals and that timing is passable.
Verify changes in filter are reflected when when navigating between newer and older signals
Verify that choice of 5 – 25 messages reflected when navigating between newer and older signals

This is what I would add to the tests Matt posted which was a list produced by the product owners in pretty much the order I would test.  I don’t see the point in re-typing the product owner’s list here, but I would be sure to test each item on the list.

Matt Heusser recently posted a testing challenge on his blog. The challenge is to create a test plan and strategy for the Socialtext’s “Activities” widget. Since I’m very curious and interested in the testing of these types of applications, I decided to put one together.

The picture is a screenshot of the app in question. This post shows a list of “tests” from the product owners. My strategy and plan are in the next post. I’m posting these concurrently, but felt the need to break them up for easier reading.

Before I plan anything, I like to have a full understanding of WHAT I’m supposed to test. The devs I work with will tell you that I ask lots of questions before I test almost anything. Maybe my questions get annoying for them, but I’d rather ask a question and be a nuisance than test something created incorrectly because of too many assumptions and not enough requirements understanding. I try to read through as much relevant documentation as possible before asking, but sometimes, in fact, many times, there is no relevant documentation.

I read through all of the blog posts and comments surrounding this test challenge before I asked questions of my own:

Is there any particular functionality the customer is looking forward to using or functionality that was given particular focus in development?

[Sure. Arguably, the main use of the app will be to see signals and changes in wiki pages. Secondarily, users will want to see things they care about (‘conversations’) and people they care about.]

I’m curious about the rss functionality. It appears that the product owners included this in the test plan. Is there more information about this?

[There’s a little RSS icon in the bottom-right. When you press the product owner, she wants it to “just work” in RSS readers. Finally, you get an exemplar reader: Google Reader. Also, you make a RSS reader widget that supports RSS 1.0, 2.0 and ATOM standards, and it needs to work on that. Does that help?]

Is there a specific reader that this connects with or is it there a list of supported readers?

[The big concern is Google Reader.]

Is the feed supposed to reflect changes in the filters or do you create a feed permanently fixed on current filter settings?

[Yes, the RSS feed link changes depending on what filters you have selected.]

The list from the product owners was fairly specific, but they still left off several items. For example, there are several items regarding being able to re-size the widget but no information about what I assume to be the settings for the widget (wrench in the upper right hand corner.) Is this test round supposed to address EVERYTHING like that or just the specifically mentioned items like re-sizing the widget height.

[Good catch! Yes, monkey-wrench is ‘settings’ and allows you to show 5, 10, 15, 20, or 25 ‘activities’ at a time.]

I also didn’t see anything mentioned about posting signals. Is posting a signal excluded from the testing?

[Yes, you can post signals from the widget. If you want to limit that out of scope to let your test plan be a reasonable size, I’d accept that. You can also REPLY to signals or Reply with a private message; and you’ll need to test that.]

“You can also REPLY to signals or Reply with a private message; and you’ll need to test that.”

In the boxes that show signals, it looks like the arrow on the right is used to reply and the envelope is used to send a message. Is that correct?

[Yes, correct]

I see names in the signals boxes are underlined. What is that and what’s the difference between that and just having a person’s name in blue. Is this another way to initiate a reply?

[A name in blue is a link to that person’s profile. (Think: Facebook ‘info’ page). A page in blue is a link to a page in the wiki; it will open in a new browser tab. Reply and Private reply are initiated at right of an activity in the activities widget. There are other ways to do it, but not on the widget, and so they are out of scope for this exercise.]

In the boxes “showing edits” I see the name of docs are in blue and are bolded. Is something special supposed to happen if I click that?