Update: If you’re in reader, the formatting just crapped out on me this time. Click through and it looks much better.
Seeing a sign that says 30% off in a store might make me stop and take a look. Of course, it depends on the store. I am much more likely to stop and look if the store is Target vs. oh, let’s say, Versace. Why is that? Because I already know that anything Versace has on 30% probably started out at thousands of dollars.
Look at the pie chart on the right. Aside from the fact that it’s adding up to more than 100%, how many people were involved in the survey? It could have been 3 or it could have been 300.
How is this relevant for testing? Let’s look at some bugs. 30% of them are user interface bugs. Are you thinking this is good or bad? Actually, it doesn’t tell you much of anything at all. Why? Because, for starters, there’s no way to tell how many bugs we are talking about.
If there are 9 bugs total, that means 3 of them are user interface bugs. In this case, the percentage doesn’t mean very much because the number is so small. You can make a fancy pie chart out of that (it’s more likely your boss will) but it will be a meaningless pie chart that wastes everyone’s time. If you’ve got a number that’s lower than about 30, there’s no point in using a percentage at all. Just use a table. There won’t be any flash, but your data will be clear.
But that’s not the only problem in this scenario. The larger problem is that 30% says nothing about the size or complexity of:
-each bug
-the application
-the testing effort
What if I’ve got 49 bugs that are cosmetic and 1 bug that is causing memory, leaks, data loss or volcanic eruptions in Iceland? (Think of the children!!!!!)
A percentage about a count that is too small is unnecessary and a percentage used to obscure large numbers deserving of attention in and of themselves is an oversimplification. I’ve been seeing a lot of percentages lately when the count is really low or when there is no count at all. When I see this I immediately mis-trust any other data included with the percentage.
Percentages are still very useful, but they must be used with care. Just because they are highly scannable and easily processed by the human brain, doesn’t give them any meaning. I think I once read a blogpost or something of James Bach’s where he was referring to testcases as a briefcase. It might be big or small but there’s really no way to tell what’s inside. The same can be said for percentages.
You have a good point. My experience in market research shows that clients love pie charts. But as with an actual pie, we might just be eating a lot of empty calories. The visual appeal of a percentage can be misleading and, ultimately, meaningless.