Don't Count Test Cases. Make Test Cases that Count. Part I

One of the most cringe-worthy questions I get as QA manager is "How many automated tests do you have?" I dread it because the number of test cases could be a meaningless metric. It's possible to have hundreds of tests around non-critical functionality like tooltips, input box characters, or banners.  

I've seen test case inflation due to logging into an application, verifying one element on the page, and then another test performing the same login, same navigation, and verifying another trivial element on the same page.  Data-driven testing makes it a snap to add more and more meaningless test cases.  This wouldn't be so bad if the critical functions were receiving extensive coverage as well, but all too often they aren't.  

Another horrible stat is test executions.  Some use the number of test executions as proof of a robust testing practice.  Puh-leaze! It doesn't take much to set up a Jenkins job that just executes the same worthless tests continuously.  It's like saying one book is better than another because it has more words.

That's why it's important for stakeholders to understand that more test cases don't mean better testing.  In fact, the number by itself means nothing. What's important is the quality of the test cases themselves, a characteristic that defies a metric.  

So forget the numbers, please, and focus on quality test cases.  What makes a quality test case?  No easy answer, sorry, but there are some ways to ensure that you are executing the test cases that matter.

One great source of real test cases is in production logs. Production logs have a wealth of data.  They can tell you what is happening in real life: what pages are being hit the most, what functions users are executing on them, and how they are getting there.  This gold mine of information will yield the traversals that should be focused on in your tests.  For example, in one application that I support with more than 300 pages, approximately 50 pages account for most of the hits.  Yes, the 80/20 rule is alive and well.  Pound away at the pages at the core of your business. 

Production usage provides another important dimension:  Who is using the application?  It's convenient to talk about the "app," but the app, depending upon who is using it, can provide a multiplicity of tailored experiences.  One app can really be dozens of apps because the demographics of the user can drive the functionality of a page or determine the set of pages available.  We can't condense our testing down to one or several typical users anymore.  There is no typical user, only many typical users. Slice and dice the user data to ensure that user-specific functionality will be covered. 

In Part II, I will discuss more ways in how to create the tests that count.