Marlena's Blog

This involves the testing of an application created with <a href=” http://code.google.com/webtoolkit/”> Google Web Toolkit</a> 

 

GWT, as I hear it’s called, is being used extensively at my work.  Therefore, seeing this type of application tested will really help me out.  The session abstract says that they are focusing on testing for GWT that goes “beyond the gui.” 

 They asked how many people had heard of GWT and got a big vacuuous silence. 

The example shows how many emploees they have in different locations doing different tasks.  

use cases for testing the app:

apply/remove filter

increaste/decrese number of items

querying the current balance

they asked the audience for more use cases.  My favorite is having a user update the app in one location then have another user check in another location.  Another good one, fill one field with quotes and add a sql script.  

Now they are asking how many test cases you would have and it ends up being hundreds of tests to automate.  How much time would it take you?  How long does it take the tests to run?  

The next set of questions is for the audience.  How many tests are you responsible for? How often do you run them?  How many are flaky?   Most people are running daily sets of tests.  But I think they were mostly asking about webapps. 

They’re actually going beyond unit tests.  After talking about the architecture of a webapp, they’re showing pictures of the architecture and talking about tests that excercise all levels of the app from ui to datastore and how these tests can be too big.  There are also medium tests that test certain sets of layers. 

Questions for testing the backend?  Sending invalid requests.  Removing the connection to the database and using a mock that will, at times, not answer requests or send a malformed response.  What happnes if there is no table?  Load testing.  Each of these test would take much less time which creates an improvement over tests that would take about 15 minutes to run. 

How would you want to test RPC?  In htis type of test would want to mock out the rpc call/rpc backend.  Use DOM objects.  Use fake messages, could also mock out the servlet.  

Testing in layer-pairs, communication between layers.  Use lots of mocks for servelets, tables, etc. 

After the integration tests, still have to system tests.  Not hundreds, but maybe 20 or 15.  “One test per use-case is probably sufficient”

Q: how does 20 use cases placate the customer?

A:  have a discussion with customer about the other testing.

from the audience:  how many people have customers who would take their word for it?

(laughter)

Presenters tell the guy it’s an interesting topic, but they need to move on in the slides (read:  EPIC FAIL)

hmm…the presenters are wrapping up, but leaving some unanswered questions.  They should have just chucked the rest of their presentation and gone with the discussion.  They did get across their main point which is that AJAX testing should not be completed solely through the GUI.  Let’s see how the Q&A goes…

Q:  running 50,000 individual tests for GMail…how can they say they are reducing test steps?

A:  the presenter is in the middle of shortening the steps now

Q: if in interface lots of functions that are not being used, won’t testing yield false positives?

A:  you should drop the functions.

Q: but what about 3rd party api’s

A: have to draw the line somewhere about how deep you’ll go with testsing

Q: are the products in google tested in the way you are advocated

A: he won’t give a name

Based on the session abstract, these two really are trying to automate everything:  testcases, testdata, traceability matrices(um, what is that), compare utilities, log parsers. 

 I googled Interface Design Language and found nothing

 

 Count o’ software tools mentioned: 19

What is meant by automated software testing?  Using software to implement testing is applied throughout software testing lifecycle.  “Software Testing is the Stepchild of Innovation”  Companies focus on research and development but not how to test it.  James Whittiker has also mentioned this on his blog.  

They used hardware testing as a paradigm for automating testing.  

Component reuse, Autonomous computing, self-testable components they want to create a header that will generate a test.  

Component Reuse:  looked at sites www.koders.com and www.krugle.com .  They also like the idea of libraries of components for tests

Autonomous computing:  self-protection, self-healing.  IBM is also researching this, google has implemented.  Automate the automation: test generators, log file analyzers, Fireeye, 

Staf, stax – available open source software test automation engine.  testcase distribution and execution.  

Automated test data generator – Fireeye allows all types of combinations of test paramters

Matlab:  

Test Vector Generator

Automated Test Case generation: lots of papers but only vendor specific products.  She is developing an open source test case generator.  

Software Interface Test Code: provides glue be/t interface code and test framework.  

test case code, have testcases in standard format w/ csv parser

Test case generation tools: Antlr, Stringtemplate, jaxbe

start with a spreadsheet that gets translated to xml.  Use jaxb to parse the xml file and create java code using stringtemplate allows for definition of template to define output. template goes through java and a java test is built.  

They are saying this is 70% reduction in effort for testcase generation.  They had a demo which showed the basics of the system they are designing.  It’s mostly java based and started with a testcase in a spread sheet.  I’ll be honest, they lost me after they showed the their testcase object in java.  I’m also questioning what they mean by testcase.  Their system seems very promising.  

I like that they are emphasizing open source tools.  They started their research process by assessing which areas of test automation are already being handled by open source tools.  ooh…just mentioned Bugzilla. 

Eggplant R/P, VNC Robot, Quality Center, IBM Rational 

for their system also used Rich Client Platform, Postgres, Hibernate, Apache POI, Eclipse, Subversion

Q:  comments in code generation were conflicting.  what if someone edits a test while someone else is generating it?

A:  Testcases should go through review.  Testcases will be static.  

Q: have you conidered run-time automation instead of static generation? 

A: yes

Q: test parameters are in spreadsheet then generate testcode could write library for parameters

A  in message based testing no forehand knowledge of customer’s data structure, so because of changing data structure could not generate library.  

 

 

On to the keynote:

In preparation for this keynote, I read JW’s blog posts about the future of testing.  It’s an 8 part (yeah, I said OCHO) blog post about what he sees in the future of testing.  He seems to be somewhat of an oracle on this subject, so here is what <a href=”http://blogs.msdn.com/james_whittaker/default.aspx”>his blog</a> says the future holds for test in summary:

 

Outsourcing: vendors will provide crowdsourcing…think TopCoder for testing

Testsourcing: hmm, really have no idea what he means with this one. I’ll be looking for clarity in his speech.  Apparently has something to do with the next innovation, which is…

Virtualization: Test environments will be available anywhere for anyone

Reusability of software development artifacts:  Test cases are reusable for others and more independent from their platform.

Information:  He says testers will be given more information from which to produce tests.  (I’m not laughing, I just swallowed a developer, really). He also introduces the idea of a Tester’s Heads Up Display which unfortunately carries the acronym THUD.  This is a great concept which would show the application in context as it’s running. 

 

Bug creation/bug detection gap narrows:  more testing will happen earlier in the process

 

Visualization:  specifically for testing

 

Testing culture:  developers and architects will include some idea of quality within their ideas and designs…”quality will be everyone’s job…one role to rule them all”

Testers as Designers: when testing moves earlier into the process, test design will be much more important. 

Testing beyond release:  when your app goes down, testing will be notified, fix will be produced, deployed and will regress itself on the spot.

So now we’ll see how much his speech matches up with his blog post.  He’s written these posts over several months, so I’m not sure how much will have changed. 

Opening joke:  Be nice to me or I’ll install vista on your laptop.  

The theme for his talk is “the magic of software.”  

Some examples, going from the structure of DNA to mapping the Human Genome

Alternative Energy

His most relevant example for me is the modeling of global finance

“But what will save us from software” is his next question.

and now it’s an awesome version of Daily WTF…The Microsoft Version WOO-HOO!

“If quality is this low w/ today’s software what about tomorrow?” 

Now on to his 8 points…virtualization is his favorite.  He’s talking about testsourcing 1st.

It’s mainly about vendors.  1st testing is insourced, 2nd came outsourcing (for testing), next is crowdsourcing he specifically mentions utest…I think Microsoft already does this with people using their products?

Testsourcing – venders provide the tests.  “Surely something like this has already been tested.”  So all of the testcases, etc. would be created by the vendor.  To get there we need 1. reusable test assets 2. shareable environments.

This leads to the next topic….Virtualization.  Bugs need to be transferable from machine to machine. Dang, now he’s talking about Vista, “I’ve never been part of a product that got more testing than vista…”  He seems to think virtualization really would have helped.  He wants to see libraries of VM’s.  That would be really cool.  Also virtualized test libraries and you add the app to it. 

His third point is Visualization. He’s comparing software testing with the QA of automobiles.  You can see if a car is missing a bumper.  

He has this whole list of stuff he’d like to see visualized.  Basic stuff like i/o, modules, etc.  Shows a treemap of Vista’s source code each square represents size, the darker the square the more complex the code.  The bigger and greener the square, the more testing it needed.  He used this when he started testing Vista and it really helped him out.

Also showing a map of overnight changes and where the breaks are. He also put this on top of a visual from a game and it showed which areas of the game the testers were visiting more often. 

Wrapping up…the point of all of this is that he wants the image of the tester “performing late cycle heroics” to go away.  

Q&A

Q: What kind of metrics would you use for visualization?

A: How many testcases didn’t get run, stuff that’s “countably infinate.”

Q: static vs. dynamic testing..

A: as you’re testing what’s going on with the software…he’s starting to talk about the THUD.  He wants to hover over web-components and see the code

Q: Star Wars initiative…how can this be applied to the real world

A:  He’s desiging the THUD for use in games testing.

Q: state changes

A: “State really sucks”  testing data flow is not in a good place.  

Q: Showed map of vista what is that?

A:  Heat map from Cambridge 

Q: How do you preach the possibility of perfection

A: 

Q:  What is the difference in education requirements?

A:  It’s not the same path as development.  He wants education about testing in high school.  Microsoft actually has someone who teaches new testers how to test.  

 

 

This is a lengthy read, but worth it if you are picking out a C++ Test Framework

When I am writing code for anything, I MUST unit test. For me, writing code without unit testing is like walking a tightrope without a net. The developers whose code I test are another story entirely. I’ve written about them previously in my post “Why developers balk at testing.”

Although I write most of my code in Java, at work, I mostly use C++. My testing is for a distributed system and many distributed systems are written in C++. When I started looking for a c++ unit testing framework, I found that unit testing frameworks for C++ are like the wild west when compared to Java. In Java, if you are unit testing, you use JUnit. In C++, there are choices, lots of choices. What I found problematic was that most of the choices were either not quite robust enough or were no longer being actively supported or just lacked enough documentation to be useful.

When I started exploring C++ unit testing with the developers in our group, we decided to try CxxTest since it has a facility for mock objects. I was a little bothered that the framework is for C++ but written in Perl, however, it was working well and I loved the choice of asserts. The big snag was when I wanted to start including mock objects. With CxxTest, you have to include headers for CxxTest in your source code. This turned me off enough that I started looking at what other frameworks had to offer.

At about this time, and quite serendipitously, Google came out with their C++ unit testing framework called Googletest (note that it’s one word). Although the developers were cranky when I said we were switching, what choice did they have. In my group, QA means Queen of it All…Queen of All testing, that is.

The install for Googletest has a couple of known bugs for Microsoft Visual Studio 2008, but since they are known, it just took some searching on the google site to find the fix. Once we had those fixed, we were able to run the sample tests.

This framework does a much better job of keeping the code and the unit tests separate than CxxTest did. It also keeps the unit tests in a *.cpp file which seems much more logical to me than having everything in a header file, like CxxTest. I haven’t found anything about mock objects, but I’m assuming they could be implemented with an interface if a stub is not adequate.

One approach Google has taken with their framework that I find extremely helpful is the way they have you group your tests. A string is included in the parameters for each test which becomes that test’s test suite. When the *.cpp file containing the tests is executed, the output displays the tests according to the grouping and shows which have passed, which have failed, etc. This makes grouping tests so easy, and developers seem more comfortable with it since all they are doing is copying and pasting a tag as a parameter.

They have also created a syntax that separates tests needing a fixture from those that do not. Any test that calls data set in the SetUp routine starts with TEST_F instead of TEST. This is awesome because I can immediately see which of my tests are using fixtures.

Since Googletest is supported, has documentation (that could be better) and allows for extensions, I will be sticking with it. As of today, its been downloaded over 7,000 times so it looks like I am not alone in my decision.

Please feel welcome to leave comments about Googletest or your C++ unit testing experience.

I’ve been blogging using Google’s blogger, but decided it was time for a change.  The content will still be the same nerdy, technical content about automated testing, semantic web, concurrency among other topics.

If you want to see the old posts, they are here.