Don't Count Test Cases. Make Test Cases that Count. Part I
One of the most cringe-worthy questions I get as QA manager is "How many automated tests do you have?" I dread it because the number of test cases could be a meaningless metric. It's possible to have hundreds of tests around non-critical functionality like tooltips, input box characters, or banners. I've seen test case inflation due to logging into an application, verifying one element on the page, and then another test performing the same login, same navigation, and verifying another trivial element on the same page. Data-driven testing makes it a snap to add more and more meaningless test cases. This wouldn't be so bad if the critical functions were receiving extensive coverage as well, but all too often they aren't. Another horrible stat is test executions. Some use the number of test executions as proof of a robust testing practice. Puh-leaze! It doesn't take much to set up a Jenkins job that just executes ...