In the previous post, I discussed the app in Matt Heusser’s latest test challenge and questions I asked about it. In this post, I will show my overall test strategy and plan. I know he says there’s a prize, but I’d much rather have some feedback regarding what about this test plan sucks and what doesn’t suck. I have the big girl shoes (with laces) on so there’s no need to hold back.
From the questions and answers Matt provided, I’ve made a few categories that will influence the focus of my testing.
Prioritized Testing:
Since the primary goal is testing that users can see changes in signals and changes, this will be what I focus on first and most. Before I saw that this was the primary goal, I questioned whether I should really plan to test every filter combination.
Explicitly described as something to be included in testing:
Every time there is a filter change it must be checked in 3 places:
1. activities widget
2. google reader feed
3. reader widget feed
Reply to signal from activities widget with post
Reply to signal from activities widget with private message
I know I will NOT be testing:
Posting a message or private mail through the text box.
Not prioritized, but still important:
Opening a page through a link in a signal and opening a person’s profile through a link in a signal. I see one possible defect for this case. If you look at the picture, everyone but Gabe Wachob’s name is underlined. I would ask why or log a defect.
Navigating through newer, newest, older pages of posts.
Using the wrench icon to display 5, 10, 15, 20 or 25 activities (queue The Phenomenal Handclap Band)
All tests listed by the product owner.
Overall Test Strategy:
I want to automate the bulk of this testing because you will see below that I have several user scenarios and 3 different views. The setup and teardown of automated testing such as that used with SeleniumRC + TestNG means I could have tests logging in as my 3 different users and checking each of the three different views as filters are changed in different browsers. Since I’m using code, I would expect to create tests for each combination of filters.
This doesn’t mean that I wouldn’t do any manual testing. Markus Gaertner suggested in his blog that he would include time-boxed exploratory testing, and I think this is an important activity that cannot be skipped. In fact, there are a couple of tests that I’m not sure would be suitable for automation. I typically do some of this on any feature I’m testing before I finish with my test planning as a way to inform my testing and shake loose a few more questions. One great aspect of Selenium IDE which I recently discovered is that the IDE will create code for use in TestNG projects. I can easily see myself simultaneously testing manually and creating code from those manual tests for Selenium RC/TestNG automation.
Risks
My performance testing coverage will be minimal. It’s just not something I know that much about. I’m not saying that this app shouldn’t be performance tested, I’m just not the one to tell you how to do it. I would have to discuss this with the PM and work something out. Maybe get someone from another project to help and show me some basics.
In the case of the rss feed, usage of Google reader requires a good account. I guess you can fake these, but I’m pretty sure it’s against their EULA.
Since I don’t do testing of web app user interfaces and funtionality on a regular basis, my estimate for how long this testing would take is a WAG, at best. I would prefer to have time factored in for creating automation that might, at first, take a little longer to write, but would be reusable. Time that is not given to me for automation is time that I would add into testing the app for subsequent iterations. Matt mentions that this testing should take 4 hours, I would give it 8 including automation and setup of test environment. Again, that’s a total WAG.
Test Environment
Because I’ve defined different types of users and different types of views, I would set up an environment for this test with a set of networks available to these 4 users:
1. The user who doesn’t have many feeds or friends, maybe they only have 1 of each
2. The user who has many people on their network but only 1 network
3. The user who has many people spread across many different networks.
4. The extreme user who has extreme values of people spread across and extreme number of networks.
5. User with 1 signal
6. User with lots of signals
I would also set up a reader widget for each user. If possible, I would set up a google reader for each user, but I’m not sure this is possible, so there would likely only be limited testing on the google reader feed.
Once users, networks and rss feeds are set up, I would build test fixture objects for these users and each of the views that will need to be checked. (This is an area where one could really go nuts with design patterns and create some extremely re-usable classes and interfaces for modeling user behaviors, BUT, I have yet to see this in the wild, and I’m assuming I won’t have time to get into this type of elaborate set-up. I sincerely hope, though, that there are places that do this and that this is where automated testing is headed.)
Test Plan
Matt suggests in his blog that these can be brief one-liners. This ain’t a novel, so I’m keeping each test short.
User scenario based tests:
Filter testing:
For each user, filter for every available combination and verify that only the correct signals appear from the correct network. I would automate this and have it run in the background while I do other testing. I would also perform a manual test for each user with up to 3 different types of filters. I would do this test first because it’s the priority for testing.
For users with 1 network:
Verify “show/from/with” appears with the filter
Verify that there is no drop down box for the users with 1 network
For users with more than one network:
Verify that there is a drop down box for users with more than 1 network
Verify that each network appears in the drop-down box
For all users:
Verify consistent choices in “showing edits” and “from everyone” boxes.
Verify resetting number of visible signals to each value 5 – 25
Make several choices in the filter drop down boxes and verify that page reflects pages in reasonable amount of time.
Check how long it takes for changes made in the widget to be reflected in the user interface and rss feeds.
For user with lots of signal messages:
Verify that newer/older command works with display set for each choice of 5 – 25 messages
Verify that navigation works if user thrashes back and forth in the navigation
For user with no signal messages:
Verify that newer/older commands are not highlighted
User-independent tests:
Verify widget is resizable
Verify widget can be minimized, expanded when minimized
Verify widget can be closed
Verify “post message” text box can be collapsed/expanded with button at top left.
NOTE: I had problems with this one IRL. If you collapse, the button disappears altogether so you have to reply to someone to get it to re-expand. What does someone with no messages do? Would they be able to re-expand? I have a screen shot if you want it.
Verify user profiles can be accessed by clicking on their name, clarify difference between underlined and not underlined names.
Verify wiki pages can be accessed by clicking on them. Possibly, using automation, select signals on a page randomly and perform this test (stick ’em in an array and do random select)
Verify signal can be replied to using the button on the right
Verify signal poster can be privately messaged using button on the right
Verify that a signal can be trashed.
Verify x minutes ago for signals and that timing is passable.
Verify changes in filter are reflected when when navigating between newer and older signals
Verify that choice of 5 – 25 messages reflected when navigating between newer and older signals
This is what I would add to the tests Matt posted which was a list produced by the product owners in pretty much the order I would test. I don’t see the point in re-typing the product owner’s list here, but I would be sure to test each item on the list.