Tag: Selenium

Isolating a performance bug

English: Firebug's logo Français : Logo de Firebug
Image via Wikipedia

Last week, I isolated a performance problem that started with the failure of an automated test.  I’m blogging it because the bug has an interesting story which highlights some of the weirdness I’ve typically found when isolating performance bugs.

 

A teammate of mine, Alin Trif, noticed one of our automated checks failing.  It writes a review for an add-on. He replicated the failing test manually and wrote up the bug.  Since a few others in our group couldn’t replicate the problem, it was subsequently closed. In true tester fashion, Alin mentioned it to me anyway.  I thought “what could it hurt to try reproducing it.”  So, after opening our staging site, addons-dev.allizom.org, I went through the steps…and reproduced the problem.

 

Note that I did not just re-open the bug.  Personally, I think re-opening bugs is a great way to alienate people who are ready to move on from a particular issue and make testing more irrelevant on your team.  This doesn’t mean dropping the ball on a problem.  It means that it’s time to uncover more or better information which would likely result in a different bug being opened anyway.  If you read to the bottom, you’ll find that this was, indeed, something else entirely.

Two of my favorite web testing tools are Firebug and Skitch.  Although I generally use Firebug for inspecting web pages and finding the locators I need for writing Selenium-Webdriver checks, it will also show metrics for requests and responses.  To access this, open Firebug and click on “net”

Firebug -> Net

In this case, while the request was off in never-never land, I opened Firebug, switched to the net tab and took a snapshot.  When the request eventually finished, I took another snapshot.  This time, the culprit is pretty clear.

 

Although I was testing the addons staging site, the bug is actually for Browser-ID, Mozilla’s new solution for single sign-on.

 

With the screenshot in hand I logged this bug against browser-id.

 

Interesting take aways from this bug:

  • Unless you are actively looking for performance bugs, they are extremely easy to dismiss.  In this case, it looked like it wasn’t reproducible.  These are also easy to miss unless you are hyper-sensitive to slowness.  (Every tester ought to know their physical triggers of impatience for slowness.  Do you tap your fingers and/or roll your eyes and/or cuss at the screen?)
  •  If you’re frustrated about one of your bugs getting closed or marked invalid, it’s time to talk to someone about it vs. only leaving a comment.
  • Using Firebug and your screenshot tool of choice makes it fairly painless to document these bugs.  I’ve caught bugs like this before and ended up installing a third party tool for looking at the time taken for requests and responses.  This is now much easier if you download the Firebug add-on because the tool is right in your browser.

The next time, something “feels slow” or “slower” to you, give this a try.  You might find something you weren’t expecting.

 

Hat tip to Alin Trif for finding the bug and asking about it even after it was initially closed.  That’s good testing.

Enhanced by Zemanta

Anatomy of Switching Selenium from RC to Webdriver

Selenium logo

If you’ve ever wanted to see what it looks like to switch tests from the Selenium-RC api to the Selenium-Webdriver api, this github pull request will give you a very good idea of the changes that have to be made.

 

 

 

 

 

There are a few reasons why these tests are a particularly good example of what happens.

  • We use page objects.  If there’s ever been proof that page objects help ease the pain of refactoring, this would be it.  We made significant changes to our page objects, but the changes made to tests are relatively minor.
  • We have over 100 tests covering many of the areas in addons.mozilla.org.  It’s all open source, so you can see the tests, the website they are testing AND the code of the website they are testing.
  • Our tests are written in python which is a pretty easy language to read and get to know.

A few highlights and bogans:

  • When you’ve got css locators for list items and you are using “nth” you have to bump all of your numbers up by one because rc starts at 0 but webdriver starts at 1.  (Insert expletives here.)
  • When assigning locators, the format changes and makes the flavor of your locator much more obvious than it was in Selenium-RC.  It’s not that the information is COMPLETELY MISSING in Selenium-RC, it just isn’t in my favorite font-type OF ALL TIME. (<3 <3 <3)
  • Instead of using get_css_count, you have to get the list of items instead and test the length.
  • Native events don’t work as well on mac, so we had to set up windows vms in our selenium grid.  This was not a trivial task and deserves it’s own post which I will write if I get over the vm trauma of the past couple of weeks.
  • You can see Webdriver navigating across the different elements of a page which is really freaking cool!
  • You can write tests for hover elements.  We’ve got lots of this in addons at present and Selenium-RC was preventing us from tests that incorporated these.  (Open the diff view and search for “hover!”
  • Instead of working mainly with locators, Selenium-Webdriver uses Web elements.  If you don’t know, these are dom elements and have a bit more meat to them than locators.  Whereas a locator is a string describing a location in a web page, a web element encompasses tags and everything within them.
  • Currently, we have to run the tests sequentially in Firefox even though they can be run concurrently in other browsers.  This is, apparently, not a new problem.  Fortunately, after talking with the Firefox devs, they are working hard on a fix for us which should benefit anyone wanting to run webdriver tests in Firefox.
  • There are lots of lovely methods which can be chained together to mimic user actions.   I know it’s a machine and not a person with a brain running these tests, but personally, I can’t wait to write a mobile test that uses the “double tap” method.  Cue Zombies!!!!

There’s much more to this change, but I’m dog tired and it’s Friday.  Knock yourselves out with that pull request!

Special thanks to the team that wrote all of that code, Teodosia Pop, Florin Strugariu, Zac Campbell with special guest-star appearances made by Dave Hunt.

Enhanced by Zemanta