On to the keynote:
In preparation for this keynote, I read JW’s blog posts about the future of testing. It’s an 8 part (yeah, I said OCHO) blog post about what he sees in the future of testing. He seems to be somewhat of an oracle on this subject, so here is what <a href=”http://blogs.msdn.com/james_whittaker/default.aspx”>his blog</a> says the future holds for test in summary:
Outsourcing: vendors will provide crowdsourcing…think TopCoder for testing
Testsourcing: hmm, really have no idea what he means with this one. I’ll be looking for clarity in his speech. Apparently has something to do with the next innovation, which is…
Virtualization: Test environments will be available anywhere for anyone
Reusability of software development artifacts: Test cases are reusable for others and more independent from their platform.
Information: He says testers will be given more information from which to produce tests. (I’m not laughing, I just swallowed a developer, really). He also introduces the idea of a Tester’s Heads Up Display which unfortunately carries the acronym THUD. This is a great concept which would show the application in context as it’s running.
Bug creation/bug detection gap narrows: more testing will happen earlier in the process
Visualization: specifically for testing
Testing culture: developers and architects will include some idea of quality within their ideas and designs…”quality will be everyone’s job…one role to rule them all”
Testers as Designers: when testing moves earlier into the process, test design will be much more important.
Testing beyond release: when your app goes down, testing will be notified, fix will be produced, deployed and will regress itself on the spot.
So now we’ll see how much his speech matches up with his blog post. He’s written these posts over several months, so I’m not sure how much will have changed.
Opening joke: Be nice to me or I’ll install vista on your laptop.
The theme for his talk is “the magic of software.”
Some examples, going from the structure of DNA to mapping the Human Genome
Alternative Energy
His most relevant example for me is the modeling of global finance
“But what will save us from software” is his next question.
and now it’s an awesome version of Daily WTF…The Microsoft Version WOO-HOO!
“If quality is this low w/ today’s software what about tomorrow?”
Now on to his 8 points…virtualization is his favorite. He’s talking about testsourcing 1st.
It’s mainly about vendors. 1st testing is insourced, 2nd came outsourcing (for testing), next is crowdsourcing he specifically mentions utest…I think Microsoft already does this with people using their products?
Testsourcing – venders provide the tests. “Surely something like this has already been tested.” So all of the testcases, etc. would be created by the vendor. To get there we need 1. reusable test assets 2. shareable environments.
This leads to the next topic….Virtualization. Bugs need to be transferable from machine to machine. Dang, now he’s talking about Vista, “I’ve never been part of a product that got more testing than vista…” He seems to think virtualization really would have helped. He wants to see libraries of VM’s. That would be really cool. Also virtualized test libraries and you add the app to it.
His third point is Visualization. He’s comparing software testing with the QA of automobiles. You can see if a car is missing a bumper.
He has this whole list of stuff he’d like to see visualized. Basic stuff like i/o, modules, etc. Shows a treemap of Vista’s source code each square represents size, the darker the square the more complex the code. The bigger and greener the square, the more testing it needed. He used this when he started testing Vista and it really helped him out.
Also showing a map of overnight changes and where the breaks are. He also put this on top of a visual from a game and it showed which areas of the game the testers were visiting more often.
Wrapping up…the point of all of this is that he wants the image of the tester “performing late cycle heroics” to go away.
Q&A
Q: What kind of metrics would you use for visualization?
A: How many testcases didn’t get run, stuff that’s “countably infinate.”
Q: static vs. dynamic testing..
A: as you’re testing what’s going on with the software…he’s starting to talk about the THUD. He wants to hover over web-components and see the code
Q: Star Wars initiative…how can this be applied to the real world
A: He’s desiging the THUD for use in games testing.
Q: state changes
A: “State really sucks” testing data flow is not in a good place.
Q: Showed map of vista what is that?
A: Heat map from Cambridge
Q: How do you preach the possibility of perfection
A:
Q: What is the difference in education requirements?
A: It’s not the same path as development. He wants education about testing in high school. Microsoft actually has someone who teaches new testers how to test.