Category: Uncategorized

Testing event driven software

Oh boy…AM starts talking about different states of event driven systems and is referring to state S-0 but he’s not saying zero. He is saying naught. S-naught…otherwise known as Snot. I can’t decide which is better. Snot or THUD.

Ok…back to testing. His problem domain is for testing event driven systems in all of their various states. He’s using a tool in Windows XP called computer management. He gives it a large number that’s obviously invalid and watches it fail. Then he goes to another dialog and it fails. He seems mainly concerned with generating all possible combinations of events for a web app. Elfriede Dustin mentoned a tool for this yesterday.

He’s about to crash united airlines site but too many people (including me) on the wifi. Tip: always use slides for demos.

1st part of this talk is a basic summary of the current state of web testing. He talks about creating lots of tests through the UI although yesterday, this strategy was somewhat debunked in the session by Markus Clermont. MC pointed out that building up lots of these end to end tests running through a gui can take a really long time to run if you have too many.

He wants to see some type of zoom in for these tests which would be really cool. This implies a hierarchical structure within these event driven tests.

GUI Model Based Testing
Event Flow Graph: He uses paint as an example to show how event interactions are diagrammed. Fellow blogger, David Burns has done a tutorial on this type of testing using graphs.

His app will rip the gui, generate an event flow graph and use this to generate test cases. Has a demo.
After test cases are run his app will analyze the run time to track the interactions. This is all done on desktop gui apps and he’s working on the web app side of it. He’s expecting some audience feed back for this.

His student is also making a presentation of an independant study she did. She put together a basic page with typical webapp inputs which would generate different states. She’s generated a state machine model of this basic web page. Her final state is getting to the next page. She took as test cases event sequences + expected state. Once she has possible states she creates a table. Most of her work was centered around getting the webripping and generation of test cases to work.

He’s asking for ideas on how to test this type

Q: Oh this is what I wanted to hear, they are asking about the test case explosion problem I mentioned.
A: He’s expecting large state machines that can be executed in a distributed manner. They don’t expect testers to use this and automate their process, they want them to generate the graph so that they can find more interesting test cases.

Q: (Really this is a suggestion) Have a best practice for a web app to generate a graph from which a more testable webapp can be created.

Q: They aren’t considering data dependant states. How do you get data into test cases?
A: We use partitioning method and come up with constraints between field elements. This problem however isn’t solved.

6 common tasks that mattered for their testing
test distribution and run control
test case set up
test case execution
test case evaluation
test case tear down
results reporting
They had writtin this code 11 different times and 11 different places. PS’s group was in charge of consolidating these.

They were in agreement until they started talking about how each task should be implemented and its importance. What they realized was thatthey all had different priorities for their approach to automated testing.

To resolve the conflict, PS started asking questions: Who is writing the tests? Who looks at results? They started grouping the tools that they had

They came up w/ different contexts: individual dev, dev team, project, and product line. When most people talk about automation, they are talking about project context. The product line context is when a produt has been released. Test can be reused in different contexts. Difference is tthe framework being used.

Dev Context: unit tests, xUnit Test Patterns by Gerard Meszaro.

Dev Team Context: focused on a subsystem of the product. These test don’t rquire knoledge of internals. Ex. tests his team wrote to catch race conditions.

Object Context: are builds more or less stable? focus on user functionality. Infrastructure more complex and you have to think about depdenancies, etc.

Project context: this where graphical tools are useful

Product Line context: long term stability test…when release approved they test for backward compatibility. Ex. have run something for 97 days and then found an out of memory error.

Case Studies of Their Tools
ITE (Integrated Test Environment) built on top of STAF/STAX. Built by testers for testers. Lots of code written in python. Primary design criteria for this was stability. All tests have metadata describing intended hardware and version of the product. Wanted to reduce setup and tear down.

How 6 tasks addressed in ITE
tests and framework distributed as a linux chroot. Verification left to test writer, does health checks. Store results of runs in a database accessible via web page.

XBVT perl based system for developer use. In desiging wanted tests to run inside or outside the tool with little overhead.
Dist/Runtime control: stored in source control. Runtime conrolled by test manifests.
Results Verification: left to writer
Teardown: left to writer
Reporting: text file w/ pass/fail is generated and stored on a web server. These results are not stored in a database. Have to search for the file.

Instead of building “one framework to test them all” they are focusing on modules.

He sees the next step as some visualization and some type of THUD.

Ask yourself:
who is goin gto rite and maintain the framework?
who will build and maintain the tests?
how are the test going to be used?
how long will the test live?

I wish my company would take this kind of initiative, but F5 looks a lot smaller than mine.

Q: What is meant by test target…tests useful across multiple contexts
A: Liked the idea yesterday of small, medium and large tests because is breaking it up in a similar way team context, project context and product line context. Tests can be identical but different stakeholders want to see results in a different way. For ex. Management wants a graph

Q: Discover lessons about retaining people writing tests for wrong context
A: We didn’t run into the case where people writing tests for wrong context

Q: You mentioned 2 tools
A: One team addresses stargegy all 4 contexts. 1 tool for individual dev context. Devs use that tool. Those tests accessible to testers but end-to-end tests work through diff tool. Tools have one way direction. Testers tool can pick up developer tests but won’t go the other

Q: ITE stores results in database is this shared
A: One database for everyone. Right now difficult to look at results multiple runs. Want ability to aggregate runs to generate report for a build on all platforms.

Q: Was choice of chroot environment because of virtualization
A: Decision was made 2 years ago when vmware wasn’t there. He wasn’t with the company when decision was made. They did this because of the heavy packet switching they work with.
F: If that weren’t the case would you be looking at virtualization
A: yes and they are looking at it. Have tests harnesses consisting multiple boxes and looking at how to virtualize that as one chunk. They are trying to come up with schema for test harness in the cloud.

Using Django for the reporting UI and for data driven tests. Use code review tool called Review Board. I also have it on good authority that Fogbugz when used with WebSVN and Cruise Control works really well and has similar functionality.

Mark is presenting a project called ETICS which is a European project funded by CERN. CERN has most recently been in the news for their Large Hedron Collider.

Automating System Tests
They use machines to investigate and have problems getting the machines allocated which means that by the time he can run the test, the system has been deployed and the problem exists in production. This was partly becase the release dynamics are spread across stakeholders which creates complexity for automation. In order to get around this he started using Amazon’s Web Services to run some of the tests.

ETICS’ goal was to show vision of automated testing. Mark wanted to “capture people’s worries in a common automated platform.” To do this his team built a testing framework on top of a backend grid system called Condor. This is where problems started to appear. Challenges included limited cm, limited sys admin, limited resource to implment automated testing. People were using this to do tests but their tests kept crashing because of inconsistencies in the complex grid environment. In order to simplify management of the system the decision was made to move the backend from Condor to Amazon’s Web Services.

Lessons Learned
Tools MUST be simple or people won’t use them. ROI must be obvious.

The use case Mark presented was for the Diligent project which used ETICS to automate testing. Diligent goes through large set of documents for queries. Used ETICS for deployment tests.
Full process automation is not required for ROI
If you don’t have access to machine, troubleshooting becomes difficult. I can see why they chose to use Web Services in this instance. Amazon allows an amazing amount of control for the boxes being used.

Cloud Computing
build -> package -> install-> deploy-> test
cloud helps with building lots of test beds.

He prefers using RESTful Web Services because of their ease of use

Mark touched on the hardware virtualization runtime environment called Xen for this and says Microsoft is about to announce their cloud solution. I remember reading about the different ways to use AWS and they really seem to push using it through REST

The next step, he says is kvm which goes from para virtualization (xen) to full virtualization(kvm).

Public cloud vs. Private cloud – they are seeing a need for clouds that work with senstive services.

Q&A
Q: are there projects being worked on with this tech?
A: “I wish I could be more open about this” that means YES.
Q: I looked into cost of 24×7 AWS and iwas cost prohibitive. If you use AWS for 10 minutes they charge for an hour how do you keep costs down
A: Its like mobile phone contracts It’s a problem. personally he’d like to see this go away.
A..from audience
had dinner with a friend doing performance testing using aws to attack his system. He uses it for an hour and then turns it off
Q: performance testing?
A: wrote study between grid and cloud it’s on the CERN web-site. Amazon uses compute unit to guide user in their choices. We have little information on how Amazon puts their back end together.

OMG the crowd RAN OUT of questions.

This involves the testing of an application created with <a href=” http://code.google.com/webtoolkit/”> Google Web Toolkit</a> 

 

GWT, as I hear it’s called, is being used extensively at my work.  Therefore, seeing this type of application tested will really help me out.  The session abstract says that they are focusing on testing for GWT that goes “beyond the gui.” 

 They asked how many people had heard of GWT and got a big vacuuous silence. 

The example shows how many emploees they have in different locations doing different tasks.  

use cases for testing the app:

apply/remove filter

increaste/decrese number of items

querying the current balance

they asked the audience for more use cases.  My favorite is having a user update the app in one location then have another user check in another location.  Another good one, fill one field with quotes and add a sql script.  

Now they are asking how many test cases you would have and it ends up being hundreds of tests to automate.  How much time would it take you?  How long does it take the tests to run?  

The next set of questions is for the audience.  How many tests are you responsible for? How often do you run them?  How many are flaky?   Most people are running daily sets of tests.  But I think they were mostly asking about webapps. 

They’re actually going beyond unit tests.  After talking about the architecture of a webapp, they’re showing pictures of the architecture and talking about tests that excercise all levels of the app from ui to datastore and how these tests can be too big.  There are also medium tests that test certain sets of layers. 

Questions for testing the backend?  Sending invalid requests.  Removing the connection to the database and using a mock that will, at times, not answer requests or send a malformed response.  What happnes if there is no table?  Load testing.  Each of these test would take much less time which creates an improvement over tests that would take about 15 minutes to run. 

How would you want to test RPC?  In htis type of test would want to mock out the rpc call/rpc backend.  Use DOM objects.  Use fake messages, could also mock out the servlet.  

Testing in layer-pairs, communication between layers.  Use lots of mocks for servelets, tables, etc. 

After the integration tests, still have to system tests.  Not hundreds, but maybe 20 or 15.  “One test per use-case is probably sufficient”

Q: how does 20 use cases placate the customer?

A:  have a discussion with customer about the other testing.

from the audience:  how many people have customers who would take their word for it?

(laughter)

Presenters tell the guy it’s an interesting topic, but they need to move on in the slides (read:  EPIC FAIL)

hmm…the presenters are wrapping up, but leaving some unanswered questions.  They should have just chucked the rest of their presentation and gone with the discussion.  They did get across their main point which is that AJAX testing should not be completed solely through the GUI.  Let’s see how the Q&A goes…

Q:  running 50,000 individual tests for GMail…how can they say they are reducing test steps?

A:  the presenter is in the middle of shortening the steps now

Q: if in interface lots of functions that are not being used, won’t testing yield false positives?

A:  you should drop the functions.

Q: but what about 3rd party api’s

A: have to draw the line somewhere about how deep you’ll go with testsing

Q: are the products in google tested in the way you are advocated

A: he won’t give a name

Based on the session abstract, these two really are trying to automate everything:  testcases, testdata, traceability matrices(um, what is that), compare utilities, log parsers. 

 I googled Interface Design Language and found nothing

 

 Count o’ software tools mentioned: 19

What is meant by automated software testing?  Using software to implement testing is applied throughout software testing lifecycle.  “Software Testing is the Stepchild of Innovation”  Companies focus on research and development but not how to test it.  James Whittiker has also mentioned this on his blog.  

They used hardware testing as a paradigm for automating testing.  

Component reuse, Autonomous computing, self-testable components they want to create a header that will generate a test.  

Component Reuse:  looked at sites www.koders.com and www.krugle.com .  They also like the idea of libraries of components for tests

Autonomous computing:  self-protection, self-healing.  IBM is also researching this, google has implemented.  Automate the automation: test generators, log file analyzers, Fireeye, 

Staf, stax – available open source software test automation engine.  testcase distribution and execution.  

Automated test data generator – Fireeye allows all types of combinations of test paramters

Matlab:  

Test Vector Generator

Automated Test Case generation: lots of papers but only vendor specific products.  She is developing an open source test case generator.  

Software Interface Test Code: provides glue be/t interface code and test framework.  

test case code, have testcases in standard format w/ csv parser

Test case generation tools: Antlr, Stringtemplate, jaxbe

start with a spreadsheet that gets translated to xml.  Use jaxb to parse the xml file and create java code using stringtemplate allows for definition of template to define output. template goes through java and a java test is built.  

They are saying this is 70% reduction in effort for testcase generation.  They had a demo which showed the basics of the system they are designing.  It’s mostly java based and started with a testcase in a spread sheet.  I’ll be honest, they lost me after they showed the their testcase object in java.  I’m also questioning what they mean by testcase.  Their system seems very promising.  

I like that they are emphasizing open source tools.  They started their research process by assessing which areas of test automation are already being handled by open source tools.  ooh…just mentioned Bugzilla. 

Eggplant R/P, VNC Robot, Quality Center, IBM Rational 

for their system also used Rich Client Platform, Postgres, Hibernate, Apache POI, Eclipse, Subversion

Q:  comments in code generation were conflicting.  what if someone edits a test while someone else is generating it?

A:  Testcases should go through review.  Testcases will be static.  

Q: have you conidered run-time automation instead of static generation? 

A: yes

Q: test parameters are in spreadsheet then generate testcode could write library for parameters

A  in message based testing no forehand knowledge of customer’s data structure, so because of changing data structure could not generate library.  

 

 

On to the keynote:

In preparation for this keynote, I read JW’s blog posts about the future of testing.  It’s an 8 part (yeah, I said OCHO) blog post about what he sees in the future of testing.  He seems to be somewhat of an oracle on this subject, so here is what <a href=”http://blogs.msdn.com/james_whittaker/default.aspx”>his blog</a> says the future holds for test in summary:

 

Outsourcing: vendors will provide crowdsourcing…think TopCoder for testing

Testsourcing: hmm, really have no idea what he means with this one. I’ll be looking for clarity in his speech.  Apparently has something to do with the next innovation, which is…

Virtualization: Test environments will be available anywhere for anyone

Reusability of software development artifacts:  Test cases are reusable for others and more independent from their platform.

Information:  He says testers will be given more information from which to produce tests.  (I’m not laughing, I just swallowed a developer, really). He also introduces the idea of a Tester’s Heads Up Display which unfortunately carries the acronym THUD.  This is a great concept which would show the application in context as it’s running. 

 

Bug creation/bug detection gap narrows:  more testing will happen earlier in the process

 

Visualization:  specifically for testing

 

Testing culture:  developers and architects will include some idea of quality within their ideas and designs…”quality will be everyone’s job…one role to rule them all”

Testers as Designers: when testing moves earlier into the process, test design will be much more important. 

Testing beyond release:  when your app goes down, testing will be notified, fix will be produced, deployed and will regress itself on the spot.

So now we’ll see how much his speech matches up with his blog post.  He’s written these posts over several months, so I’m not sure how much will have changed. 

Opening joke:  Be nice to me or I’ll install vista on your laptop.  

The theme for his talk is “the magic of software.”  

Some examples, going from the structure of DNA to mapping the Human Genome

Alternative Energy

His most relevant example for me is the modeling of global finance

“But what will save us from software” is his next question.

and now it’s an awesome version of Daily WTF…The Microsoft Version WOO-HOO!

“If quality is this low w/ today’s software what about tomorrow?” 

Now on to his 8 points…virtualization is his favorite.  He’s talking about testsourcing 1st.

It’s mainly about vendors.  1st testing is insourced, 2nd came outsourcing (for testing), next is crowdsourcing he specifically mentions utest…I think Microsoft already does this with people using their products?

Testsourcing – venders provide the tests.  “Surely something like this has already been tested.”  So all of the testcases, etc. would be created by the vendor.  To get there we need 1. reusable test assets 2. shareable environments.

This leads to the next topic….Virtualization.  Bugs need to be transferable from machine to machine. Dang, now he’s talking about Vista, “I’ve never been part of a product that got more testing than vista…”  He seems to think virtualization really would have helped.  He wants to see libraries of VM’s.  That would be really cool.  Also virtualized test libraries and you add the app to it. 

His third point is Visualization. He’s comparing software testing with the QA of automobiles.  You can see if a car is missing a bumper.  

He has this whole list of stuff he’d like to see visualized.  Basic stuff like i/o, modules, etc.  Shows a treemap of Vista’s source code each square represents size, the darker the square the more complex the code.  The bigger and greener the square, the more testing it needed.  He used this when he started testing Vista and it really helped him out.

Also showing a map of overnight changes and where the breaks are. He also put this on top of a visual from a game and it showed which areas of the game the testers were visiting more often. 

Wrapping up…the point of all of this is that he wants the image of the tester “performing late cycle heroics” to go away.  

Q&A

Q: What kind of metrics would you use for visualization?

A: How many testcases didn’t get run, stuff that’s “countably infinate.”

Q: static vs. dynamic testing..

A: as you’re testing what’s going on with the software…he’s starting to talk about the THUD.  He wants to hover over web-components and see the code

Q: Star Wars initiative…how can this be applied to the real world

A:  He’s desiging the THUD for use in games testing.

Q: state changes

A: “State really sucks”  testing data flow is not in a good place.  

Q: Showed map of vista what is that?

A:  Heat map from Cambridge 

Q: How do you preach the possibility of perfection

A: 

Q:  What is the difference in education requirements?

A:  It’s not the same path as development.  He wants education about testing in high school.  Microsoft actually has someone who teaches new testers how to test.  

 

 

This is a lengthy read, but worth it if you are picking out a C++ Test Framework

When I am writing code for anything, I MUST unit test. For me, writing code without unit testing is like walking a tightrope without a net. The developers whose code I test are another story entirely. I’ve written about them previously in my post “Why developers balk at testing.”

Although I write most of my code in Java, at work, I mostly use C++. My testing is for a distributed system and many distributed systems are written in C++. When I started looking for a c++ unit testing framework, I found that unit testing frameworks for C++ are like the wild west when compared to Java. In Java, if you are unit testing, you use JUnit. In C++, there are choices, lots of choices. What I found problematic was that most of the choices were either not quite robust enough or were no longer being actively supported or just lacked enough documentation to be useful.

When I started exploring C++ unit testing with the developers in our group, we decided to try CxxTest since it has a facility for mock objects. I was a little bothered that the framework is for C++ but written in Perl, however, it was working well and I loved the choice of asserts. The big snag was when I wanted to start including mock objects. With CxxTest, you have to include headers for CxxTest in your source code. This turned me off enough that I started looking at what other frameworks had to offer.

At about this time, and quite serendipitously, Google came out with their C++ unit testing framework called Googletest (note that it’s one word). Although the developers were cranky when I said we were switching, what choice did they have. In my group, QA means Queen of it All…Queen of All testing, that is.

The install for Googletest has a couple of known bugs for Microsoft Visual Studio 2008, but since they are known, it just took some searching on the google site to find the fix. Once we had those fixed, we were able to run the sample tests.

This framework does a much better job of keeping the code and the unit tests separate than CxxTest did. It also keeps the unit tests in a *.cpp file which seems much more logical to me than having everything in a header file, like CxxTest. I haven’t found anything about mock objects, but I’m assuming they could be implemented with an interface if a stub is not adequate.

One approach Google has taken with their framework that I find extremely helpful is the way they have you group your tests. A string is included in the parameters for each test which becomes that test’s test suite. When the *.cpp file containing the tests is executed, the output displays the tests according to the grouping and shows which have passed, which have failed, etc. This makes grouping tests so easy, and developers seem more comfortable with it since all they are doing is copying and pasting a tag as a parameter.

They have also created a syntax that separates tests needing a fixture from those that do not. Any test that calls data set in the SetUp routine starts with TEST_F instead of TEST. This is awesome because I can immediately see which of my tests are using fixtures.

Since Googletest is supported, has documentation (that could be better) and allows for extensions, I will be sticking with it. As of today, its been downloaded over 7,000 times so it looks like I am not alone in my decision.

Please feel welcome to leave comments about Googletest or your C++ unit testing experience.

I’ve been blogging using Google’s blogger, but decided it was time for a change.  The content will still be the same nerdy, technical content about automated testing, semantic web, concurrency among other topics.

If you want to see the old posts, they are here.