This is one of those posts that simmered in the back of my mind for months until I read Scott Barber’s blog post 10 Things about Testing that Should Die. Scott’s post is one of the most memorable posts I’ve read about testing in a long time and reading it created the mental tipping point I needed to write things down. (and, yes, I suggest you read it too. If you are a conference organizer, it would make a great keynote.)
One of the awesome contacts I’ve made through the Writing About Testing group is the editor for Techwell, Joey McAllister. When Joey saw me tweeting that I might have to blog some stuff, he asked if I wouldn’t mind writing it up for Techwell.
Last week, I isolated a performance problem that started with the failure of an automated test. I’m blogging it because the bug has an interesting story which highlights some of the weirdness I’ve typically found when isolating performance bugs.
A teammate of mine, Alin Trif, noticed one of our automated checks failing. It writes a review for an add-on. He replicated the failing test manually and wrote up the bug. Since a few others in our group couldn’t replicate the problem, it was subsequently closed. In true tester fashion, Alin mentioned it to me anyway. I thought “what could it hurt to try reproducing it.” So, after opening our staging site, addons-dev.allizom.org, I went through the steps…and reproduced the problem.
Note that I did not just re-open the bug. Personally, I think re-opening bugs is a great way to alienate people who are ready to move on from a particular issue and make testing more irrelevant on your team. This doesn’t mean dropping the ball on a problem. It means that it’s time to uncover more or better information which would likely result in a different bug being opened anyway. If you read to the bottom, you’ll find that this was, indeed, something else entirely.
Two of my favorite web testing tools are Firebug and Skitch. Although I generally use Firebug for inspecting web pages and finding the locators I need for writing Selenium-Webdriver checks, it will also show metrics for requests and responses. To access this, open Firebug and click on “net”
Firebug -> Net
In this case, while the request was off in never-never land, I opened Firebug, switched to the net tab and took a snapshot. When the request eventually finished, I took another snapshot. This time, the culprit is pretty clear.
Although I was testing the addons staging site, the bug is actually for Browser-ID, Mozilla’s new solution for single sign-on.
With the screenshot in hand I logged this bug against browser-id.
Interesting take aways from this bug:
Unless you are actively looking for performance bugs, they are extremely easy to dismiss. In this case, it looked like it wasn’t reproducible. These are also easy to miss unless you are hyper-sensitive to slowness. (Every tester ought to know their physical triggers of impatience for slowness. Do you tap your fingers and/or roll your eyes and/or cuss at the screen?)
If you’re frustrated about one of your bugs getting closed or marked invalid, it’s time to talk to someone about it vs. only leaving a comment.
Using Firebug and your screenshot tool of choice makes it fairly painless to document these bugs. I’ve caught bugs like this before and ended up installing a third party tool for looking at the time taken for requests and responses. This is now much easier if you download the Firebug add-on because the tool is right in your browser.
The next time, something “feels slow” or “slower” to you, give this a try. You might find something you weren’t expecting.
Hat tip to Alin Trif for finding the bug and asking about it even after it was initially closed. That’s good testing.
During my halcyon days of working in the basement at a large financial services company, I noticed a group that frequently had “parties” in one corner of our dark and server-cold basement workspace. Unfortunately, these weren’t celebratory parties with beer, wings and bad karaoke. They were actually oh-my-god-our-site-is-getting-throttled-and-we’re-losing-shitloads-of-money-there-goes-our-bonus parties. They always started with one guy getting tons of instant messages. He’d start complaining, “there goes my lunch” or “that can’t be good.” Then the messages would turn into phone calls. A developer would start pounding on our locked door. Once he was in, everyone who worked with him would quickly follow. All of them gathered around one poor guy’s computer, followed by a chorus of “did you try…” or “are you sure…”
Since I worked on a completely different project, I stayed as far away from these gatherings as I could. The people gathering seemed to form a stressful knot which would become tighter and tighter. The air seemed to contract as the waves of stress would roll off of the group. During the worst of these, I packed up my laptop and went home to work.
Watching Curtis Koenig (nice template, dude!) give a Mozilla brownbag talk last week on “The Neurobiology of Decision Making or Knowing Where One’s Towel Is” reminded me of these parties. While I’ve read about the science behind “fight, flight or freeze” before, it was in the context of a conversation between two people. As a reminder, back in the day, we used the amygdala when we literally had to outrun our enemies or fight them to the death. The amygdala kicks off a rush of blood and adrenaline to the muscles, starving our brains of oxygen and turning us into, as Curtis says, hairless apes.
Protip: When your brain is starved of oxygen, you will not make the best decisions ever.
For this reason, a phrase Curtis mentioned in his talk resonated with me:
“Don’t just do something, stand there.”
Aside from describing the fight-flight-freeze reaction, Curtis kept talking about “amydala-driven-meetings.” These sound very similar to the basement parties I remember so well, although now that it’s a few years later, I realize that they can take other forms as well. When I see fists pounding on a table, hear raised voices or the metaphors go all military and we’re marching against the enemy until “we can see the whites of their eyes”…I know that there is panic and that no good can now come out of the meeting.
These meetings happen to all of us, and it’s worth considering what we, as individuals, can control in these situations. Here’s what I try to do:
1. Recognize that there is panic in the air
2. Refrain from contributing to the stress level. Now is not the time to judge others, make assumptions or pass along 2nd hand information. (Well, it is never time for these, really)
3. If it’s possible, diffuse some stress by introducing forgiveness if someone or another group is being blamed It can help change the tone of people’s thoughts.
4. Use the crucial conversation trick of saying out loud, “I want x and I don’t want y…is there a middle ground here?”
5. Make every effort to avoid commitment as it will be a commitment made as the result of an oxygen-starved decision.
6. Sadly, I’ve also seen meetings where it’s best to just not participate in the party at all and stay quiet. If this happens, it’s an indication that there is some serious dysfunction happening in a group which is usually based in fear and insecurity. In groups where individuals are empowered, this shouldn’t ever happen, but in the real world, even the best groups have their bad days and bad meetings.
It’s easy to blame people who participate in these amygdala driven meetings or to beat ourselves up if we find ourselves participating, but it’s worth remembering that most of us don’t have good stress coping skills modeled for us. In fact, even if we work on this in our personal lives, most workplaces do little to encourage the management of stress in meetings. In fact there are plenty of places where the panic is encouraged. Even though I found a comic element to the “parties in the basement”, I also knew that our company routinely did layoffs at the end of the fourth quarter just to make their bottom line look better. I’d love to see a study of how much revenue is lost from bad decisions made in oxygen-starved meetings, but I’m not, uh, holding my breath on that one. Corporate America…for the loss.
If you’ve ever wanted to see what it looks like to switch tests from the Selenium-RC api to the Selenium-Webdriver api, this github pull request will give you a very good idea of the changes that have to be made.
There are a few reasons why these tests are a particularly good example of what happens.
We use page objects. If there’s ever been proof that page objects help ease the pain of refactoring, this would be it. We made significant changes to our page objects, but the changes made to tests are relatively minor.
We have over 100 tests covering many of the areas in addons.mozilla.org. It’s all open source, so you can see the tests, the website they are testing AND the code of the website they are testing.
Our tests are written in python which is a pretty easy language to read and get to know.
A few highlights and bogans:
When you’ve got css locators for list items and you are using “nth” you have to bump all of your numbers up by one because rc starts at 0 but webdriver starts at 1. (Insert expletives here.)
When assigning locators, the format changes and makes the flavor of your locator much more obvious than it was in Selenium-RC. It’s not that the information is COMPLETELY MISSING in Selenium-RC, it just isn’t in my favorite font-type OF ALL TIME. (<3 <3 <3)
Native events don’t work as well on mac, so we had to set up windows vms in our selenium grid. This was not a trivial task and deserves it’s own post which I will write if I get over the vm trauma of the past couple of weeks.
You can see Webdriver navigating across the different elements of a page which is really freaking cool!
You can write tests for hover elements. We’ve got lots of this in addons at present and Selenium-RC was preventing us from tests that incorporated these. (Open the diff view and search for “hover!”
Instead of working mainly with locators, Selenium-Webdriver uses Web elements. If you don’t know, these are dom elements and have a bit more meat to them than locators. Whereas a locator is a string describing a location in a web page, a web element encompasses tags and everything within them.
Currently, we have to run the tests sequentially in Firefox even though they can be run concurrently in other browsers. This is, apparently, not a new problem. Fortunately, after talking with the Firefox devs, they are working hard on a fix for us which should benefit anyone wanting to run webdriver tests in Firefox.
There are lots of lovely methods which can be chained together to mimic user actions. I know it’s a machine and not a person with a brain running these tests, but personally, I can’t wait to write a mobile test that uses the “double tap” method. Cue Zombies!!!!
There’s much more to this change, but I’m dog tired and it’s Friday. Knock yourselves out with that pull request!
Special thanks to the team that wrote all of that code, Teodosia Pop, Florin Strugariu, Zac Campbell with special guest-star appearances made by Dave Hunt.
The organizers of this year’s GTAC, are attempting to get us nerds at the conference to socialize with each other. When I checked into the conference this morning, a fistful of buttons with a yellow “A” were shoved at me. Before the keynote, James Whittaker let all of us know that we were given the buttons specifically so that we would trade them and “be social.” If we get all of them, they spell out “GTAC” and then there’s one button which has a picture of the cloud. Rumor has it, that if we want the cloud button we should find bugs in the GTAC android app and tweet them.
The gtac app is intended for use by conference attendees for viewing the schedule, making tweets, taking notes about the conference etc.
My former Atlassian partner in testing crime, Mark Hrynczak and I decided to pair on looking for bugs. We had a great time testing Confluence together so we decided to pair on breaking the GTAC app. If this a test automation conference, did we run off and make a bunch of automated mobile tests? Um, no. It didn’t take us much playing with the app to find a couple of crashes and a few oddities.
Here’s Mark with our first crash:
Really GTAC app?
Steps to reproduce:
1. open the app
2. tilt the phone sideways.
Here I am with our 2nd crash:
Just don't install it on a tablet.
Steps to reproduce:
1. install the app on an android tablet
2. try to open it.
After Mark and I finished playing with the app together, my phone started tossing up errors from, guess what…the gtac app!
Aside from crashes, we found some other issues. While I wouldn’t consider these as serious or impactful as a crash, I’d probably raise issues for these. (Just don’t ask if they’d be raised in JIRA or Bugzilla.)
1. We couldn’t figure out where to add notes initially. When click on notes, it opens a page where there’s no way to add a note.
2. Notes don’t link back to sessions.
3. Notes are saved in email as an xml file. What am I going to do with notes in xml format? I don’t know about you, but I’m not in the habit of reading through my notes as an xml file.
4. Why have notes as separate choice if they are attached to sessions?
5. Opening tweets and hitting back doesn’t take the user back to the gtac app. It’s like the app is expecting you to press the one button at the bottom of the interface (o wait, that’s the iphone).
6. There is a “sandbox” tab in the “starred items” window. Why?
7. None of the sessions, including keynotes, list the speaker
After we found the first crash, I rushed up to James Whittaker with Mark and said, “Hey you! We broke your fancy gtac app!! Can we get a pin!” James, who was on the way to the podium looked at me and said, “This is the last time I ask for bugs at a testing conference…you should tweet your find.” So I did. A few minutes later, a googler I had been sitting with at lunch came over and gave me a cloud pin.
Cloud...we haz
This leaves me with a dilemma. While the pin was given specifically to me, Mark and I worked on these bugs together so it’s only fair to work something out especially since a cloud pin means you get a prize at the end of the conference. If you’re a Googler at GTAC reading this post and you still have a cloud pin, please consider giving it to Mark (or if you can’t find him, I’m happy to pass it along.)
Update: Supposedly bugs in the app have been fixed. Hmm…
A couple of weeks ago, I presented at PNSQC on not being a testing asshole. Although my presentation was the end of a lengthy personal journey for me, there was a moment in the middle of the journey which is preserved in a blog post.
You see, I had to find my way through a wall of my own emotional trauma in order to stand up in front of a paying audience and say that IT IS NOT OK TO BE A TESTING ASSHOLE. We all have moments like this when we reach the bottom of something we feel will be endless. For me this moment is preserved in my post, “Let’s Destroy the World.” There’s no way you would know, but I was a weepy mess when I pressed publish on that one as a I was letting go of some really awful things. I found myself at a bottom.
When you’re at the bottom of a tectonic shift happening in your life, it is not uncommon to question. Where am I? Who am I? Am I going the right way or will I spiral back down again? I asked myself all of these questions and more as I pressed publish on Let’s Destroy the World.”
Birth of Stars (NASA, Chandra, 10/7/08)
This is the point where you start looking for signs. I once had a beautiful friend who decided to join a monastery, and for him, the sign was a white rose.
For me “the sign” was this year’s announcement of the Google Test Automation Conference, “Test is Dead” which was serendipitously posted 5 days after “Let’s Destroy the World.” My friend and fellow testing blogger, Chris McMahon called it Zeitgeist. It should come as no surprise that I will be at GTAC this week. (Hey attendees…us Mozillians are throwing y’all a party. I’ll be the girl in the elevator between 7:00 and 8:00, wearing the red leopard print skirt. ;) )
There is plenty of criticism that can be heaped on those of us writing and presenting on the theme “testing is dead,” but as I wrote in my last post, I feel it as more of a transformation or rebirth. There are people in this world who have no problem dealing with the messiness, chaos and defying of logic that come with birth and transformation. I suggest that if you’re so attached to logic that change is inconceivable, you suspend your belief for a few moments and play with what “could be” instead of what you insist upon as “the way.” Not everything we dream of will remain, but it’s the best way I’ve found of clearing a path into the maze of the unknown.
With GTAC, I once more find myself racing into a labryinth of unknowns and uncertainties, but I now know that this is where I live and feel most comfortable. These are the people I work with, the people I play with and the people who feed my dreams. You see, I don’t live in the future or the past, but when I look in the mirror I see them. This week my mirror is GTAC.
My great friend and fellow testing blogger Trish Khoo recently wrote a post inspired by Goranka Bjedov’s keynote at STANZ 2011. In her post, Trish reacts to some news delivered by Goranka at STANZ: “Quality is Dead.”
This morning I’ve had the pleasure of listening to Julian Harty play understudy for an ailing Goranka with his opening keynote for PNSQC. I am guessing that although Julian gave a talk that seemed very much his own, the heart of what he was saying is similar to what Goranka was exposing at STANZ. (You can bet that as you are reading this, I have absconded to watch the video of her keynote.)
What I noticed not just in Julian’s keynote, but the subsequent presentations I attended today, was not a death so much as a rebirth, a liberation and even a tearing down of walls. At the end of the keynote, I tweeted that I can feel another paradigm shift happening not only testing but also in software.
In the keynote, Julian talked about how Dr. Barry Boehm has moved on from saying that bugs are found more cheaply earlier in the software process. Instead, Boehm has been writing that it is now just as cheap to fix bugs in production. This was the perfect setup for Keith Stobie’s presentation, “Testing Services in Production”. Keith drove home the point that he is currently much more comfortable testing in production than testing in any type of “staging” environment.
Aside from moving testing into a production environment, there is some testing that Keith doesn’t even call testing anymore. He calls it “monitoring.” This is where the wall between testing and production really crumbles away. He talked about moving asserts from tests into the production code. I’m sure that this is nothing especially new. What’s new is that now there are people relying on this code as an indication of system health.
In his presentation, “An Introduction to Customer Focused Test Design“, Alan Page (who got us to lunch on time) talked about how his team has been getting away from even doing functional tests by focusing on “ility” testing instead of functional test design. This in essence, means that his team is much more focused on performance testing.
What does all of this mean for those of us who consider ourselves “QA” or “Testers” or some similar combination. It means that our jobs are changing and will probably look very different in a few years. I lost count of the number of times Julian mentioned “testing not being around,” “automation experts being out of a job,” “testers being out of a job” or even questioning the validity of a testing conference.
As someone who is currently employed to primarily work with test automation and test infrastructure, dId this make me uncomfortable? Not at all. In fact, it’s very exciting. You see, I’m here because I love change and innovation. If there is something better, I’m all for tearing away the cruft and exposing the new. It might mean that I start adding code to the app I work with for web timings or that I end up with a job title that has no trace of “test” or “qa.” The focus is undoubtedly shifting away from looking for bugs and presuming to know what customer’s want to making sure real data is available about system health and that we know how to interpret it.
How does all of this apply to my talk on “Hard Lessons About Soft Skills?” All of this means that we’ll be more embedded with the devs than ever before. In fact, it might even mean that we will be the devs. If they don’t want us on their team anymore because we’ve burned our bridges with them and we are sheltering behind the wall of QA, what’s gonna happen when the wall comes down?
No matter how things shift, I can guarantee that I’ll still be around and giving a shit about how well the software is working and whether or not customers are happy with it. Are you ready to be liberated?
I’ll leave this guerilla testing post with a classic from LEGENDARY rockers, “The Scorpions.”
There are testers I follow, there are developers I follow…and then there’s Elisabeth Hendrickson. Aside from being a consultant for agile testing, Elisabeth is the founder of Entaggle. When it comes to software and building it in a collaborative way, she knows. When I was getting started in testing, her cheat sheet helped me out many times. (I think everyone in testing must use it at one time or another.) When I met her at the first Writing About Testing conference I didn’t tell her, but I was in a state of fangirl awe. Elisabeth is a pragmatic wonderwoman of software as evidenced in posts such as:
Next Thursday, October 6 at 12:30 pm (PDT), Mozilla is lucky to have Elisabeth give her talk Lessons learned from 100+ Simulated Agile Transitions which she previously gave at Agile testing days in Berlin. The talk will be broadcast on air.mozilla.com which means you are also invited. If you are deliberating whether or not this will be worth an hour of your time, I suggest you read this blog post of hers. Oh…and then there are the stunt hamsters. (I’m not joking!!!)
By the way, Elisabeth is giving a rare public 3-day class on Agile Testing at Agilistry Studios which includes the word count simulation. Believe me, if I weren’t giving a talk at PNSQC, I would be going.
Update… Fellow tweep @mubbashir put together this world clock for the talk so you can find the time closest to your locale.
A flashpoint that I don’t see much discussion about is the moment when developers realize they need more of a stake in the quality of an application.
I know quite a few developers who care about quality, and I’m sure they each have their own story about the moment they realized how important it can be. I’ve seen devs realize the difference a focus on testing can make when they’ve outpaced their testers and the testers had no hope of ever catching up.
In a meeting I attended awhile back, a dev said to me that he wanted to have more ownership of testing. Now, it’s not like he wants to wrestle it away from the testers, but he would like to feel like he is taking a more active role in the quality of his code and in the application.
These are just a few suggestions from me:
Don’t make it too cumbersome to write tests
I’ve worked with test infrastructure that was so old, there was a lot of new functionality lacking and old bugs made some testing a real pain. A frequently heard dialog was, “did you write tests for that?” “Uh, no, the framework doesn’t support it.”
Don’t be too strict about having perfect tests
Especially if you have developers who are just getting use to writing unit tests or functional tests, make it a positive experience. It should be ok to play around with what works well in the tests. It means that there will be refactoring later, but having tests to refactor and developers interested in refactoring them is better than having devs who are scared of ridicule when they write tests and no tests at all.
Use short developer checklists
– have an awareness of stuff that you tend to miss and make a short list to check for them
– are there certain types of problems your team is trying to avoid?
Make sure the interns are testing
We all know that testing education is hit or miss in CS programs. If you work around interns make sure they are writing tests. If they don’t know how to write tests, you’re looking at someone who will be a developer that doesn’t own tests or the quality of their code unless you intervene. This isn’t to say that I haven’t met interns who know how test, just that there are plenty who don’t.
But this is just me…the tester talking about owning quality. I went in search of some Mozilla web devs interested in sharing their perspective on this topic.
Kumar McMillan (@kumar303) a developer for addons.mozilla.org had these tips for developers:
test each component in isolation so that it works
Integrate components early and often because there are usually surprises, especially when integration spans multiple teams (via APIs / services, etc)
Test at different layers of integration. For example: write tests that combine lots of backend components but not no frontend components.
Developers should deploy code directly or be present when deploying code and should know where to look to monitor the health of the deploy. Without this direct responsibility it’s easy for developers to lose track of quality.
It’s also important for developers not to over-test components. It’s tempting to boost code coverage graphs or attempt a 3:1 test to code ratio but generally these practices are not helpful. It will slow you down. It’s more important to get code into the hands of users with crucial edge cases covered by tests. The “happy paths” are not so important to test because it is more likely for those to be explored.
Whenever there’s a bug it’s important to write a test to prevent regressions before the bug is fixed.
Always use continuous integration and make every developer own up to their mistakes when they break the test suite.
All tests should be automated in CI otherwise that test is useless — someone will forget to run it.
Dave Dash (@davedash) who recently gave the workshop “Test Anything, Test Everything” at DjangoCon took a more philosophical approach. Dave’s perspective is that “Quality is more than finding bugs. It’s about having a high-quality product.” He mentioned it’s about having pride in your work and caring about the website you build. Dave’s attitude is one that I’ve consistently found in the best developers I’ve worked with. As a tester, I’ve noticed that it’s always 10 times easier to work with developers who care about quality and testing because they want their code and the applications they produce to be great. Quality is, indeed, about more than finding bugs.
Thanks to Dave and Kumar for sharing :)
If you have your own opinion about how developers can own quality, I invite you to share it in the comments.
Today, I’m blogging from the HTML5Dev Conference. The house is packed, and I’m breaking out of my shell as a tester to have a look at all of this from the dev perspective. Of course, the tester in me has tagged along. What’s great about conferences in the age of twitter is that you are never in a bubble during these things. One of my co-workers, Greg Koberger tweeted about some performance testing win for the lastest Firefox release, Firefox 7.
Greg's tweet
In taking a look at the lifehacker article he tweeted, I noticed that there was a category for css performance tests. Since I spend all day, every day looking at selenium tests that have css locators and I sit next to a css dev (who has excellent taste in rap), css has been on the brain lately. “Hmm…css peformance…LET’S GOOGLE.” If you google for css peformance, the first link is for an article titled, Performance Impact of CSS Selectors, by Steve Souders. Given, the article is talking about Firefox 3.0 so it’s obviously a bit long in the tooth, but it’s interests me for not one, but two reasons:
I’m waiting for the talk “High Performance HTML5” to begin. It’s being given by Steve Souders. (Serendipity or Coincidence? I shall leave you, dear reader, to ponder.)
The article has a list of different websites and the number of DOM elements in each one.
Since this is a guerilla blog post I’m finishing up as a talk starts, I have no intention of delving deeply into the guts of the article, at the moment. What I can share, however, is a new Firefox addon I’ve been playing with called Tilt. The picture you see above is my Facebook page, turned on it’s side in tilt. Tilt visualizes a web page’s dom elements in 3d and was developed by a Mozilla intern who has recently been hired. So if we filter the list in the article, Google has the least number of elements while Facebook has the most. It should be no surprise that the Google home page has had its performance tweaked to oblivion. Here we can compare the dom of Google visually with the Dom of Face book.
Facebook:
Facebook Tilt
Google:
Google tilt
I’m definitely forming a hypothesis about the CSS performance of these two pages based on their tilt results.
And that’s your guerilla blog post from the HTML5Dev Conf. They really ought to make this thing 2 days next year. It’s pretty cool.
