Twitter github

In search of clouds…

A guerrilla blog post from GTAC 2011.

 

The organizers of this year’s GTAC, are attempting to get us nerds at the conference to socialize with each other.  When I checked into the conference this morning, a fistful of buttons with a yellow “A” were shoved at me.  Before the keynote, James Whittaker let all of us know that we were given the buttons specifically so that we would trade them and “be social.”  If we get all of them, they spell out “GTAC” and then there’s one button which has a picture of the cloud.  Rumor has it, that if we want the cloud button we should find bugs in the GTAC android app and tweet them.

 

The gtac app is intended for use by conference attendees for viewing the schedule, making tweets, taking notes about the conference etc.

 

My former Atlassian partner in testing crime, Mark Hrynczak and I decided to pair on looking for bugs.  We had a great time testing Confluence together so we decided to pair on breaking the GTAC app.  If this a test automation conference, did we run off and make a bunch of automated mobile tests?  Um, no.  It didn’t take us much playing with the app to find a couple of crashes and a few oddities.

 

Here’s Mark with our first crash:

Really GTAC app?

 

 

 

 

 

 

 

 

 

 

 

Steps to reproduce:

1. open the app

2. tilt the phone sideways.

 

Here I am with our 2nd crash:

Just don't install it on a tablet.

Steps to reproduce:

1. install the app on an android tablet

2. try to open it.

 

After Mark and I finished playing with the app together, my phone started tossing up errors from, guess what…the gtac app!

 

Aside from crashes, we found some other issues.  While I wouldn’t consider these as serious or impactful as a crash, I’d probably raise issues for these.  (Just don’t ask if they’d be raised in JIRA or Bugzilla.)

 

1. We couldn’t figure out where to add notes initially.  When click on notes, it opens a page where there’s no way to add a note.

2. Notes don’t link back to sessions.

3. Notes are saved in email as an xml file.  What am I going to do with notes in xml format?  I don’t know about you, but I’m not in the habit of reading through my notes as an xml file.

4. Why have notes as separate choice if they are attached to sessions?

5. Opening tweets and hitting back doesn’t take the user back to the gtac app.  It’s like the app is expecting you to press the one button at the bottom of the interface (o wait, that’s the iphone).

6. There is a “sandbox” tab in the “starred items” window.  Why?

7. None of the sessions, including keynotes, list the speaker

After we found the first crash, I rushed up to James Whittaker with Mark and said, “Hey you!  We broke your fancy gtac app!!  Can we get a pin!”  James, who was on the way to the podium looked at me and said, “This is the last time I ask for bugs at a testing conference…you should tweet your find.”  So I did.   A few minutes later, a googler I had been sitting with at lunch came over and gave me a cloud pin.

 

Cloud...we haz

 

This leaves me with a dilemma.  While the pin was given specifically to me, Mark and I worked on these bugs together so it’s only fair to work something out especially since a cloud pin means you get a prize at the end of the conference.  If you’re a Googler at GTAC reading this post and you still have a cloud pin, please consider giving it to Mark (or if you can’t find him, I’m happy to pass it along.)

 

Update:  Supposedly bugs in the app have been fixed.  Hmm…

 

 

Zeitgeist

Seal birth 13 (2:32pm) by nutmeg66, on Flickr

Seal birth 13 (2:32pm) by nutmeg66, on Flickr

(If you like to listen to music while you read my posts, I suggest “Violent Dreams” by Crystal Castles.)

 

A couple of weeks ago, I presented at PNSQC on not being a testing asshole.  Although my presentation was the end of a lengthy personal journey for me, there was a moment in the middle of the journey which is preserved in a blog post.

 

You see, I had to find my way through a wall of my own emotional trauma in order to stand up in front of a paying audience and say that IT IS NOT OK TO BE A TESTING ASSHOLE.  We all have moments like this when we reach the bottom of something we feel will be endless.  For me this moment is preserved in my post, “Let’s Destroy the World.”  There’s no way you would know, but I was a weepy mess when I pressed publish on that one as a I was letting go of some really awful things.  I found myself at a bottom.

 

When you’re at the bottom of a tectonic shift happening in your life, it is not uncommon to question.  Where am I? Who am I? Am I going the right way or will I spiral back down again?  I asked myself all of these questions and more as I pressed publish on Let’s Destroy the World.”

Birth of Stars (NASA, Chandra, 10/7/08)

Birth of Stars (NASA, Chandra, 10/7/08)

 

This is the point where you start looking for signs.  I once had a beautiful friend who decided to join a monastery, and for him, the sign was a white rose.

 

For me “the sign” was this year’s announcement of the Google Test Automation Conference, “Test is Dead” which was serendipitously posted  5 days after “Let’s Destroy the World.”  My friend and fellow testing blogger, Chris McMahon called it Zeitgeist.  It should come as no surprise that I will be at GTAC this week.  (Hey attendees…us Mozillians are throwing y’all a party.  I’ll be the girl in the elevator between 7:00 and 8:00, wearing the red leopard print skirt.  ;)  )

 

There is plenty of criticism that can be heaped on those of us writing and presenting on the theme “testing is dead,” but as I wrote in my last post, I feel it as more of a transformation or rebirth.  There are people in this world who have no problem dealing with the messiness, chaos and defying of logic that come with birth and transformation.  I suggest that if you’re so attached to logic that change is inconceivable, you suspend your belief for a few moments and play with what “could be” instead of what you insist upon as “the way.”  Not everything we dream of will remain, but it’s the best way I’ve found of clearing a path into the maze of the unknown.

 

With GTAC, I once more find myself racing into a labryinth of unknowns and uncertainties, but I now know that this is where I live and feel most comfortable.  These are the people I work with, the people I play with and the people who feed my dreams.  You see, I don’t live in the future or the past, but when I look in the mirror I see them.  This week my mirror is GTAC.

 

Hey testers…how soon is now?

 

Apart from the upcoming GTAC, this post was inspired by an interview with William Gibson tweeted by @chris_blain

The Liberation of Testing

A slide from Alan's presentation at PNSQC

 

This is a guerilla blog post from PNSQC.

 

My great friend and fellow testing blogger Trish Khoo recently wrote a post inspired by Goranka Bjedov’s keynote at STANZ 2011.  In her post, Trish reacts to some news delivered by Goranka at STANZ:  “Quality is Dead.”

 

This morning I’ve had the pleasure of listening to Julian Harty play understudy for an ailing Goranka with his opening keynote for PNSQC.  I am guessing that although Julian gave a talk that seemed very much his own, the heart of what he was saying is similar to what Goranka was exposing at STANZ.  (You can bet that as you are reading this, I have absconded to watch the video of her keynote.)

 

What I noticed not just in Julian’s keynote, but the subsequent presentations I attended today, was not a death so much as a rebirth,  a liberation and even a tearing down of walls.  At the end of the keynote, I tweeted that I can feel another paradigm shift happening not only testing but also in software.

In the keynote, Julian talked about how Dr. Barry Boehm has moved on from saying that bugs are found more cheaply earlier in the software process.  Instead, Boehm has been writing that it is now just as cheap to fix bugs in production. This was the perfect setup for Keith Stobie’s presentation, “Testing Services in Production”.  Keith drove home the point that he is currently much more comfortable testing in production than testing in any type of “staging” environment.

 

Aside from moving testing into a production environment, there is some testing that Keith doesn’t even call testing anymore.  He calls it “monitoring.” This is where the wall between testing and production really crumbles away.   He talked about moving asserts from tests into the production code.  I’m sure that this is nothing especially new.  What’s new is that now there are people relying on this code as an indication of system health.

 

In his presentation, “An Introduction to Customer Focused Test Design“, Alan Page (who got us to lunch on time) talked about how his team has been getting away from even doing functional tests by focusing on “ility” testing instead of functional test design.  This in essence, means that his team is much more focused on performance testing.

 

What does all of this mean for those of us who consider ourselves “QA” or “Testers” or some similar combination.  It means that our jobs are changing and will probably look very different in a few years.  I lost count of the number of times Julian mentioned “testing not being around,” “automation experts being out of a job,” “testers being out of a job” or even questioning the validity of a testing conference.

 

As someone who is currently employed to primarily work with test automation and test infrastructure, dId this make me uncomfortable?  Not at all.  In fact, it’s very exciting.  You see, I’m here because I love change and innovation.  If there is something better, I’m all for tearing away the cruft and exposing the new.  It might mean that I start adding code to the app I work with for web timings or that I end up with a job title that has no trace of “test” or “qa.”  The focus is undoubtedly shifting away from looking for bugs and presuming to know what customer’s want  to making sure real data is available about system health and that we know how to interpret it.

 

How does all of this apply to my talk on “Hard Lessons About Soft Skills?“  All of this means that we’ll be more embedded with the devs than ever before.  In fact, it might even mean that we will be the devs.  If they don’t want us on their team anymore because we’ve burned our bridges with them and we are sheltering behind the wall of QA, what’s gonna happen when the wall comes down?

 

No matter how things shift, I can guarantee that I’ll still be around and giving a shit about how well the software is working and whether or not customers are happy with it.  Are you ready to be liberated?

 

I’ll leave this guerilla testing post with a classic from LEGENDARY rockers, “The Scorpions.”

Stunt Hamster Alert: Elisabeth Hendrickson will be giving a talk at Mozilla next Thursday!

© Quality Tree Software Inc.

There are testers I follow, there are developers I follow…and then there’s Elisabeth Hendrickson.  Aside from being a consultant for agile testing, Elisabeth is the founder of Entaggle.  When it comes to software and building it in a collaborative way, she knows.  When I was getting started in testing, her cheat sheet helped me out many times. (I think everyone in testing must use it at one time or another.)  When I met her at the first Writing About Testing conference I didn’t tell her, but I was in a state of fangirl awe.  Elisabeth is a pragmatic wonderwoman of software as evidenced in posts such as:

 

Testing is a whole team activity.

Agile Backlash? Or Career Wakeup Call?

Specialized Test Management Systems Are an Agile Impediment

Why Test Automation Costs Too Much

 

Next Thursday, October 6 at 12:30 pm (PDT), Mozilla is lucky to have Elisabeth give her talk Lessons learned from 100+ Simulated Agile Transitions which she previously gave at Agile testing days in Berlin.  The talk will be broadcast on air.mozilla.com which means you are also invited.  If you are deliberating whether or not this will be worth an hour of your time, I suggest you read this blog post of hers.  Oh…and then there are the stunt hamsters.  (I’m not joking!!!)

 

By the way, Elisabeth is giving a rare public 3-day class on Agile Testing at Agilistry Studios which includes the word count simulation.  Believe me, if I weren’t giving a talk at PNSQC, I would be going.

 

Update… Fellow tweep @mubbashir put together this world clock for the talk so you can find the time closest to your locale.

If you are a developer and you would like to own the quality of your code…

Operation Sock Monkey! by moon angel, on Flickr

A flashpoint that I don’t see much discussion about is the moment when developers realize they need more of a stake in the quality of an application.

 

I know quite a few developers who care about quality, and I’m sure they each have their own story about the moment they realized how important it can be.  I’ve seen devs realize the difference a focus on testing can make when they’ve outpaced their testers and the testers had no hope of ever catching up.

 

In a meeting I attended awhile back, a dev said to me that he wanted to have more ownership of testing.  Now, it’s not like he wants to wrestle it away from the testers, but he would like to feel like he is taking a more active role in the quality of his code and in the application.

 

 

These are just a few suggestions from me:

Don’t make it too cumbersome to write tests
I’ve worked with test infrastructure that was so old, there was a lot of new functionality lacking and old bugs made some testing a real pain.  A frequently heard dialog was, “did you write tests for that?”  “Uh, no, the framework doesn’t support it.”

Don’t be too strict about having perfect tests
Especially if you have developers who are just getting use to writing unit tests or functional tests, make it a positive experience.  It should be ok to play around with what works well in the tests.  It means that there will be refactoring later, but having tests to refactor and developers interested in refactoring them is better than having devs who are scared of ridicule when they write tests and no tests at all.

 

Use short developer checklists

- have an awareness of stuff that you tend to miss and make a short list to check for them
- are there certain types of problems your team is trying to avoid?

 

Make sure the interns are testing
We all know that testing education is hit or miss in CS programs.  If you work around interns make sure they are writing tests.  If they don’t know how to write tests, you’re looking at someone who will be a developer that doesn’t own tests or the quality of their code unless you intervene. This isn’t to say that I haven’t met interns who know how test, just that there are plenty who don’t.

 

But this is just me…the tester talking about owning quality.  I went in search of some Mozilla web devs interested in sharing their perspective on this topic.

 

Kumar McMillan  (@kumar303)  a developer for addons.mozilla.org had these tips for developers:
test each component in isolation so that it works

  •  Integrate components early and often because there are usually surprises, especially when integration spans multiple teams (via APIs / services, etc)
  •  Test at different layers of integration.  For example: write tests that combine lots of backend components but not no frontend components.
  • Developers should deploy code directly or be present when deploying code and should know where to look to monitor the health of the deploy.  Without this direct responsibility it’s easy for developers to lose track of quality.
  • It’s also important for developers not to over-test components.  It’s tempting to boost code coverage graphs or attempt a 3:1 test to code ratio but generally these practices are not helpful.  It will slow you down.  It’s more important to get code into the hands of users with crucial edge cases covered by tests.  The “happy paths” are not so important to test because it is more likely for those to be explored.
  • Whenever there’s a bug it’s important to write a test to prevent regressions before the bug is fixed.
  • Always use continuous integration and make every developer own up to their mistakes when they break the test suite.
  • All tests should be automated in CI otherwise that test is useless — someone will forget to run it.

 

Dave Dash (@davedash)  who recently gave the workshop “Test Anything, Test Everything” at DjangoCon took a more philosophical approach.  Dave’s perspective is that “Quality is more than finding bugs.  It’s about having a high-quality product.”  He mentioned it’s about having pride in your work and caring about the website you build.  Dave’s attitude is one that I’ve consistently found in the best developers I’ve worked with.  As a tester,  I’ve noticed that it’s always 10 times easier to work with developers who care about quality and testing because they want their code and the applications they produce to be great.  Quality is, indeed, about more than finding bugs.

Thanks to Dave and Kumar for sharing :)

 

If you have your own opinion about how developers can own quality, I invite you to share it in the comments.

Tilt Visualization and CSS Performance

Facebook City

Today, I’m blogging from the HTML5Dev Conference.  The house is packed, and I’m breaking out of my shell as a tester to have a look at all of this from the dev perspective.  Of course, the tester in me has tagged along.  What’s great about conferences in the age of twitter is that you are never in a bubble during these things.  One of my co-workers, Greg Koberger tweeted about some performance testing win for the lastest Firefox release, Firefox 7.

Greg's tweet

In taking a look at the lifehacker article he tweeted, I noticed that there was a category for css performance tests.  Since I spend all day, every day looking at selenium tests that have css locators and I sit next to a css dev (who has excellent taste in rap), css has been on the brain lately.  “Hmm…css peformance…LET’S GOOGLE.”  If you google for css peformance, the first link is for an article titled, Performance Impact of CSS Selectors, by Steve Souders.  Given, the article is talking about Firefox 3.0 so it’s obviously a bit long in the tooth, but it’s interests me for not one, but two reasons:

  1. I’m waiting for the talk “High Performance HTML5″ to begin.  It’s being given by Steve Souders.  (Serendipity or Coincidence?  I shall leave you, dear reader, to ponder.)
  2. The article has a list of different websites and the number of DOM elements in each one.

Since this is a guerilla blog post I’m finishing up as a talk starts, I have no intention of delving deeply into the guts of the article, at the moment.  What I can share, however, is a new Firefox addon I’ve been playing with called Tilt.  The picture you see above is my Facebook page, turned on it’s side in tilt.  Tilt visualizes a web page’s dom elements in 3d and was developed by a Mozilla intern who has recently been hired.  So if we filter the list in the article, Google has the least number of elements while Facebook has the most.  It should be no surprise that the Google home page has had its performance tweaked to oblivion.  Here we can compare the dom of Google visually with the Dom of Face book.

Facebook:

Facebook Tilt

 

Google:

Google tilt

 

I’m definitely forming a hypothesis about the CSS performance of these two pages based on their tilt results.

And that’s your guerilla blog post from the HTML5Dev  Conf.   They really ought to make this thing 2 days next year.  It’s pretty cool.

Talking at PNSQC about being a testing asshole

Portland by diebmx

A theme has been running through my blog on the topic of being a testing asshole.  It began with the post “The Tester’s Paradox” and continued with the posts, “Is it necessary for testers to care?,  “Are you a testing asshole?“  and  “Let’s destroy the World.”

 

In these posts I worked through my own experiences with my behavior on a software team.  These posts also came from my processing on some of the books I was reading at the time.  My bookshelf for these posts includes:

  • The No Asshole Rule, by Bob Sutton
  • The Sociopath Next Door, by Martha Stout
  • Crucial Conversations by Kerry Patterson
  • Good Boss, Bad Boss also by Bob Sutton
  • Artful Making by Rob Austin and Lee Devin

 

After all of the reading, all of the testing, lots of asshole screw-ups of my own and dealing with some people in my life who are pretty big assholes in general,  I’ve drawn some conclusions:  Testers are automatically placed in the most socially awkward position on any software team which means we need more social skills and awareness than anyone else.  We need more emotional fluency.  We are not assholes and because of the work we do, we must learn how to minimize our asshole behavior as much as possible.  As I’ve been reading and learning about this, I’ve found enough compelling information that I decided to put it into the hands of testers by giving a talk at this year’s Pacific Northwest Software Quality Conference (PNSQC).

 

The talk is meant as a means of self-reflection just as the examination of this topic has been an exercise in self-reflection for me.  I’m hoping that people who come to my talk walk away with more questions than answers.  I hope it gets people thinking about how they act with others on their team.  There is so much in testing that we have absolutely no control over.  My hope for this talk is that it gives people enough information to work on controlling the things we can control.

 

Since I’m not a psychologist, I teamed up with my friend, Gordon Shippey, who is a licensed counselor, to write the paper.  We didn’t feel comfortable calling the talk, “Are you a testing asshole?” so we settled on, “Hard Lessons about Soft Skills — Understanding the Psyche of the Software Tester.”

 

I’ll be talking about:

  • Asshole behavior in the context of testing
  • Crucial Conversations
  • Recovery and Repair
  • Testing in a Safe Environment

The paper Gordon and I wrote in conjunction with the talk is intended as a resource for times of testing distress when testers feel they’ve backed themselves into a corner or find themselves handling situations in ways that don’t make them proud.

 

Another conclusion that I’ve drawn in all of this is that anyone can be an asshole when put in the right situation.  Personally, I now have enough confidence to know that 95% of the time, I’m not one of them.  This realization has made all of the effort and hard work I’ve put into this paper and talk worth the time spent.  I hope others find it useful as well.

 

On Blogging

Magic sofa is magic.

Unless you write a blog, there’s no way to understand how random the different aspects of it can be.  Within the blogosphere, random takes the form of:
* Bloggers who inspire me and keep me blogging
* Who the comments come from, what they say and when they show up
* The different places I go while writing posts
* The posts that show up in front of the public and the posts that don’t
Recently Justin Wehr,  a blogger I’ve followed, like for years, posted on his blog writing habits.  I’ve also been talking to a few people who are interested in starting their own blogs. This post of mine copies the format of his post with my answers.  Justin: this post is for you. Blog on.

 

Why I got into it
The reason I started blogging was because I was going through a Masters of Software Engineering program and wanted to have something I could easily show others when I finished.

Why I stay(ed) in it

I found that it helped me think through my schoolwork and that I enjoy writing more than I ever understood.  I’ve always had a journal in one form or another.  In fact, several years worth of entries exist on the 286 pc clone running Norton commander that I think might still be in my mom’s basement.

 

My blog has been a metamorphosis for my personal and professional life.  It’s taken me around the world.  It’s introduced me to some very good friends I otherwise would never have met.  It’s kept me connected during some dark times and has continued to help me process the world around me.  I process my life through writing, some of which makes it onto this blog.

 

What *is* blogging for me?

Blogging is the most purely selfish habit I have.  Whenever I sit down to write a post, the priority is always on me and what I want to say.  There are no ads.  There is no genuflecting.  I have never written a post because someone asked me to or taken money for anything that has shown up on this blog.

Although my content was pretty narrowly focused in the beginning towards school projects, it’s turned into quite a mixed bag.  Anyone remember the Kangaroo post?  What about the Desert post?  Then there’s the dumpster post.  (We’ll just leave the “Twilight” post in the ether, shall we?)

 

How hard is it to keep chugging?
For most of my blogging life, I’ve never, ever had a problem coming up with posts.  There was only one period in time when I worried that my blog was withering and the fact that I wasn’t writing freely was a very strong smell of things going on in my life.

 

Do I read and/or revise old posts?
There is always a small mistake or two that I miss in a post.  There are very few posts where I haven’t hit publish then looked at the post and thought (oh shit…edit!  Edit now!)  Although, to be fair, the edits are always quite small.  I do, on occasion, read my old posts.  My thinking is circular and I enjoy looking at how my opinions change over time.

 

What keeps me from sucking? 

There are a few guidelines I have for myself.

  •  Be kind to myself.  Since the blog is by me, for me, I practice self-love.  There is no reason why I should ever not be nice and overly forgiving to myself on my blog.  This includes comments.  I’ve been lucky to only ever have a couple of comments that I just wouldn’t post.  (Protip: RTFM has no place here.)
  • Posts shouldn’t be a reaction to someone else’s negative energy or merely a regurgitation.  The blog is mine so it should be as 100% me as possible.  The posts I write shouldn’t be posts that anyone else would or could write.  This post is a notable exception, but I think that’s in the best possible way.
  • Be patient.  There are posts that sit in my backlog for months before I finish them.  Some of my best posts including the last post on continuous deployment might sit for a while.  Sometimes this is because I’m feeling cranky about something and it comes out in a bad way.  Sometimes it’s because of, well…life.  I’ve found that the good ideas for posts have a certain timelessness so it’s ok if I have to put it down for a few weeks or even months.
  • Capture the ideas as soon as possible.  I know to keep some paper in the bathroom for ideas I have in the shower.  I have evernote on my phone.  I’ve got religion about it.  If an idea comes, I write it down somewhere.

What do I think other people think of this blog?

Umm…I don’t freaking care.  I just don’t.

 

Where do the ideas come from?
I love Justin’s answer to this question, so I’m reprinting it here:
“All over the place.

All over the damn place.

Not uncommonly: Books, blogs, field observations, porch ruminations, pesky bedtime thoughts, car rides, and conversations.”

 

How do I decide what’s post-worthy?
The content has to be interesting for me and usually relates to a problem I’m thinking through or a challenge I’ve faced.  Sometimes I blog about trips that I take or conferences I’ll be attending.

 
Which posts are most salient to me?

Be Gaga-riffic, Be yourself
Testing in a Throwaway Culture
A Desert Tale of Rocks and Ruins
Picasso Ate My Metrics Paper: Visualizing Software Metrics with Treemaps
Tossing out the map
A twist in the plot
What is Quality, What is Art

What is Quality, What is Art – Part Deux
Continuous Deployment and Data Visualization
Owning my Celeb-u-tester
The Tester’s Paradox

My post about gender and diversity

Are you a testing Asshole

Let’s Destroy the World

Bi-Testual:  Coming out of the Software Closet

 

Where/when is the writing done?

Where:  Currently, on a $50 goodwill sleeper/sofa in a one bedroom apartment located in Mountain View, California.  There is usually a dog or a man also sitting on the sofa.
When: Usually either early on Saturday morning or late at night (note…it’s 10:54 on Thursday night and I have to be working at 6:00am)

 

How long does it take, typically?
It’s so dependent on the post.  Some are eked out over lengthy periods of time and some come in a great whoosh.

 

What’s my 5 year plan? What are my aspirations for this mofo?
Uh…who knows.

Continuous Deployment and Data Visualization

Mozilla-firefox-usage-data

Image via Wikipedia

A phrase I hear a lot around Mozilla is “continuous deployment.”  I hear there’s this product Mozilla makes that’s competing with some other product that has rapid release cycles.  So, yeah, we’re working on continuous deployment.

 

I’ve noticed that a main resource around our office for information about continuous deployment is this video from Etsy.  Hearing, “We’re moving to continuous deployment,” is nothing new for me.  This is the 2nd job I’ve had where it’s been a major focus.  Since I’ve  heard of the Flickr version, I decided to watch this Etsy video.

 

Picture yourself at your computer about to hit the big button and deploy a feature you’ve been working on.  You are fairly confident that nothing catastrophic will happen, but you don’t know.  (I’m writing this from a dev perspective, but even if you’re a tester…come on…you never know, even if you’ve tested the hell out of something).  In the talk, this is what is referred to frequently as, “the fear.”  It’s actually referred to as either, “THE FEAR” or “the fear.”

 

“Fear for startups is the biggest no-no.”

“Fear is what keeps you from deleting your database.”

“Fear doesn’t go with creative work.”

 

This rings true for me because I frequently deploy selenium tests for addons.mozilla.org.  My teammates and I have talked about “THE FEAR.”  We have strategies for coping with it such as holding one’s breath, saying a prayer or running the 90+ tests one more time.  When Etsy talks about “The Fear” I know exactly what they mean.

 

Etsy’s video fascinates me because of how they have conquered “The Fear.”  It’s been on my mind every day since I watched the video.  What’s the special-continuous-deployment-sekrit-sauce-that-makes-everything-all-better?

 

Etsy combats “the fear” with visibility.  You see, at Etsy, EVERYTHING IS GRAPHED ALL THE TIME.

 

Here are some of the things they mentioned graphing in the video:
How many visitors are using this thing?
Can we deploy that to 100%?
Did we make it faster?
Did I just break something?
How long is it taking to generate a page?
How many users are logged in?
How is the bandwidth?
What’s the database load?
What’s the requests per second?

 

If you look at the graphs, they are simple bar or line graphs.  They are not exceptionally fancy but they are numerous and the maintenance admittedly takes work.  They are not, however maintained by specialists working in a silo.  The graphs are created by an engineer.  Here are some numbers:

 

20,000 lines/second is their log traffic, at times
16,000 is the number of “metrics” they have organized through dashboards
25 engineers committing code to dashboards
20 dashboards

 

I doubt that when Etsy decided to start graphing everything they woke up one day with 25 dashboards.  It sounded very much like they put the tools in the developers hands and lovingly nudged them along.

 

This is a serious commitment to data.
Data doesn’t just happen.  It takes a persistent effort to include log messages in your code. It takes servers and databases capable of handling the traffic created by the log messages and staff to maintain them.  It takes investing in huge monitors all around the office and giving people the bandwidth to figure out how to work with the data & graphics stack.  Most importantly, it takes trust so that employees are allowed to see the data without making them jump through hoops.

So how can a team move closer to the graphing part of continuous deployment?
According to Etsy:

  • Give people access to production data — without making them wait months for a special password or even log in every time.
  • Make the data real time instead of daily.  When I say access, I mean feeds.  This goes well beyond a spreadsheet.
  • Create copious amounts of log messages.  If someone clicks a link, goes full screen or downloads something…log it.
  • once you have the data, make graphs for features before you release them

 

I love data, but will be the first to admit that it is not pretty.  The plain truth about data is that it takes patience because combing through and refining  it can be tedious, monotonous work.  It is very easy to buy a bunch of monitors and put them on a wall showing an inst-o-matic graph that came with your bug tracker (I’ve seen this done.  O hai, expensive wallpaper!).  It takes more time to ask deeper, meaningful questions.  It takes even more time to filter the data into something graph-able.  After that, you have to find the right way to share it.  Note, that even if you do all of this and the data successfully tells a story, you’ll have to spend time dealing with, “and why did you use those colors.”  What was I saying? Oh yes, data is not pretty.

Now that I’m working every day with tests I visualized a couple of years ago, I’m continuing my quest for deeper questions about tests.  In my context, the tests are the selenium tests I work with day in and day out, so besides coming to grips with “THE FEAR,” I’ve also been thinking about, “THE FAIL.”  But wait!  That’s another blog post.

 

If you want to read more about Etsy’s graphs and data, they have written their own post about it.

Enhanced by Zemanta

Running AMO Tests with moz-grid-config

This post contains the instructions for a workshop that is part of the addons.mozilla.org automation testday.

 

For running the AMO tests, it’s good to use our moz-grid-config repo to run selenium because it allows you to get around some issues like setting up a separate firefox profile for accepting certificates and it contains both the standalone version of selenium rc as well as what you’ll need to run selenium grid.

 

If you’d like to know more about how grid works, you can check out this page.

 

Here is the url for the repo which includes a readme.

 

Once you’ve cloned the moz-grid-config repo using
git clone git@github.com:mozilla/moz-grid-config.git

You can try starting it up.  There are 2 commands for this which will each need to be running in their own terminal window:
ant launch-hub
ant launch-remote-control  -Drc.environment=”environment name”

 

The value from the -Drc.environment parameter comes from the grid_configurations.yaml file.  If you have a look in the file, you’ll see options for other environments including Firefox 6 and the Aurora channel.

 

It’s also worth noting that the selenium jars are in the vendor directory.

 

Once you’ve got grid up and running, and forked the AMO test repository, you can run some AMO tests with this type of command line

 

py.test -n 5 –browser-name=”Firefox 5 on Mac OS X” -q test_search.py -k test_that_searching_with_substrings_returns_results

-n 5 is xdist telling py.test to run 5 instances at a time.  Note that you need to have multiple terminals going with ant launch-remote on different ports to use this option.  If you don’t specify a port number when you run “ant launch-remote” it will launch on 5555
–browser-name=”Firefox 5 on Mac OS X” is specified in the grid_configuration.yaml of moz-grid-config
-q test_search.py is telling py.test to run the file test_search.py
-k test_that_searching_with_substrings_returns_results

 

-k is a keyword parameter.  In this example it’s used for a testname, but we could also specify -k substrings to run any tests with substring in the testname.

If you get stuck or if something isn’t working with your environment, feel free to stop by the testday irc channel:  irc.mozilla.org #testday.  (Note, I probably won’t be watching twitter as much today since I’ll be in the irc channel)