Last week, I isolated a performance problem that started with the failure of an automated test. I’m blogging it because the bug has an interesting story which highlights some of the weirdness I’ve typically found when isolating performance bugs.
A teammate of mine, Alin Trif, noticed one of our automated checks failing. It writes a review for an add-on. He replicated the failing test manually and wrote up the bug. Since a few others in our group couldn’t replicate the problem, it was subsequently closed. In true tester fashion, Alin mentioned it to me anyway. I thought “what could it hurt to try reproducing it.” So, after opening our staging site, addons-dev.allizom.org, I went through the steps…and reproduced the problem.
Note that I did not just re-open the bug. Personally, I think re-opening bugs is a great way to alienate people who are ready to move on from a particular issue and make testing more irrelevant on your team. This doesn’t mean dropping the ball on a problem. It means that it’s time to uncover more or better information which would likely result in a different bug being opened anyway. If you read to the bottom, you’ll find that this was, indeed, something else entirely.
Two of my favorite web testing tools are Firebug and Skitch. Although I generally use Firebug for inspecting web pages and finding the locators I need for writing Selenium-Webdriver checks, it will also show metrics for requests and responses. To access this, open Firebug and click on “net”
In this case, while the request was off in never-never land, I opened Firebug, switched to the net tab and took a snapshot. When the request eventually finished, I took another snapshot. This time, the culprit is pretty clear.
Although I was testing the addons staging site, the bug is actually for Browser-ID, Mozilla’s new solution for single sign-on.
With the screenshot in hand I logged this bug against browser-id.
Interesting take aways from this bug:
- Unless you are actively looking for performance bugs, they are extremely easy to dismiss. In this case, it looked like it wasn’t reproducible. These are also easy to miss unless you are hyper-sensitive to slowness. (Every tester ought to know their physical triggers of impatience for slowness. Do you tap your fingers and/or roll your eyes and/or cuss at the screen?)
- If you’re frustrated about one of your bugs getting closed or marked invalid, it’s time to talk to someone about it vs. only leaving a comment.
- Using Firebug and your screenshot tool of choice makes it fairly painless to document these bugs. I’ve caught bugs like this before and ended up installing a third party tool for looking at the time taken for requests and responses. This is now much easier if you download the Firebug add-on because the tool is right in your browser.
The next time, something “feels slow” or “slower” to you, give this a try. You might find something you weren’t expecting.
Hat tip to Alin Trif for finding the bug and asking about it even after it was initially closed. That’s good testing.
