At this point, the work I’m doing on my master’s thesis is starting to congeal. I’ve been reading about data visualization and how it can assist quality. There seem to be several levels at work. There is the source code level where developers and, to some extent, qa examine individual pieces of code. This level is addressed by unit tests generally written by a developer or specialized white box tester. Higher up is the system test level which may or may not be automated. Recently, frameworks and tools such as STAX/STAF and Selenium have helped to automate and provide more consistency for some level of system test. At the highest level, quality is less about tests and more about metrics, in particular, lines of code.
In my research, I have found many papers about providing analysis for source code. There are also plenty of papers which address the production of metrics. Software and User Interfaces have developed to the point where the idea of zooming is going to bring these layers together. If you think about Google Maps and the zooming capability that it has, think about how that type of zooming could be applied to the Software Development process. James Whittaker, the architect of Microsoft Team Test certainly has. His vision is that this type of interaction will result in a Heads Up Display for quality.
Assuming that this is where we are going. I can see some challenges in how projects are organized. The main challenge will be mapping from one level to the next. Moving from a system test level to a unit test level will highlight how system tests are mapped to unit tests. This goes back to requirements and design which I know is still an issue at not all but plenty of software shops. Apart from unit to system test mapping, there is also the issue of how LOC is examined and how that maps to source code and maybe even system tests. What if two developers have worked on the same code? How would management assess the LOC count?
I know that, technically, system tests, unit tests and source code are all supposed to be organized based on requirements, but some of us live in the real world!!!! I know that there are places where requirements are maybe not great, but usable. That said, there are plenty of testers who don’t work in shops where we have the benefit of usable requirements or they may be usable in a very loose sense of the word. If a tester is working in a group with bad requirements, they are still responsible for the bugs they didn’t catch.
In this case, complexity is not just about source code anymore, but about how well tests cover the system.
Really good article Marlena. I like the idea of taking the T.H.U.D and being able to zoom through the system.
I never took that away from James’ talk about the future of testing but i do like it.
Thanks for the compliment!
Initially, I hadn’t paid as much attention to his idea of a T.H.U.D, but the more I look into using visualizations, the more I think it will be necessary to have interactive visualizations.
As testers, we’re so responsible for having exact reasons about why we think something is wrong that we need the capability to go from high level to low level detail. It’s also beneficial to have multiple views available at a time so that we can quickly compare. This way, if a tester is making a high level comparison across multiple views and sees something funky, they can investigate the relevant pieces.
I like model based testing for simplifying the complexity of the code but find I need “pop out models” for the more complex areas. It helps me to figure out some of the detail and where the code is more complex but I wonder about an overall view showing areas of high change as the product goes on. Are there any tools that do basically a heat image showing the amount of change happening in what area of code as a project goes on? I’m curious about more ways to visually represent not only code complexity, but risk and justification for more test coverage and focus in one area versus another.
Hmmm….a heat map of change in areas of code. This is interesting because it brings together a couple of areas I’ve been looking into. The first is heat maps of source code. I’ve seen a heat map showing areas of source code with more complexity. The second area is the idea of software changing over time, or “software evolution.” If you can have a tree map showing complexity, it seems like producing a treemap that shows the amount of change is equally possible.
There are software packages such as tableau that will produce these types of maps if provided the right data. If you have access to good charting software, it’s probably a matter of acquiring the right data from your source code repository.
I totally hear what you are saying about assessing risk for test coverage. “How many tests” seems like the million dollar question for each release. Visualization could be very helpful.