Performance Zone is brought to you in partnership with:

JJ is a developer advocate for YouTube APIs. His goal is to foster a rich set of third-party applications built on YouTube APIs. He's a well-known member of the Python community. He blogs at jjinux.blogspot.com on topics such as Python, Ruby, Linux, open source software, the Web, and lesser-known programming languages. Shannon is a DZone MVB and is not an employee of DZone and has posted 18 posts at DZone. You can read more from them at their website. View Full User Profile

Unit Tests Don't Find Bugs: the Death of QA

10.27.2012
| 28487 views |
  • submit to reddit

Unit tests don't find bugs. They find regressions. This is a painful lesson I learned when I first started doing TDD (test-driven development), and it's well known among most TDD circles.

TDD's goal is to prevent programmers from introducing new bugs into working code. However, when you're writing code from scratch, your tests won't help you find all the bugs in your code. That's because you can't possibly write tests for all the ways your software will be used (or abused). When I first started doing TDD, I had really good tests, but I was too tired to do much exploratory QA. However, my boss wasn't, and I was very embarrassed to find that my software had lots of bugs. Simply put, he used my software in ways that I hadn't intended.

I've seen a lot of companies that don't bother writing any tests or doing any QA. They just let their users find all the bugs. Needless to say, I've never had respect for those companies.

However, it's growing more and more popular to destaff the QA department and just require engineers to write lots and lots of tests. Often, these are in the form of unit tests. Even though integration tests can conceivably catch more bugs, they take much longer to run. Hence, even integration tests are often deprioritized.

What I'm discovering is that a lot of projects have both lots of unit tests and lots of bugs. These are bugs that could have been found manually by a QA engineer, but it seems that manual QA testing (i.e. exploratory testing) has gone out of vogue.

I used to think that code that was well-documented, well-styled, well-tested, and code reviewed would rarely have bugs. Sadly, I no longer believe that to be the case. I think we need to go back to the days when we had decently-sized QA departments, perhaps in addition to all the other things we do.

To tweak what Knuth said, "Beware of the above code. I have only tested that it works. I haven't actually tried it."

Published at DZone with permission of Shannon Behrens, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Dean Schulze replied on Mon, 2012/10/29 - 8:46am

Good article.  I hope you have your flame suit on.  The TDD zealots won't like this at all.

Shouldn't the title be about the need to resurrect QA, though?  I've found that QA testers also are the best people to turn to when trying to understand the use cases on a project that I've just joined.

I believe that integration tests that cover the known use cases should be written first.  In addition to finding bugs in the code they find problems with integration which is where a lot of bugs are.

 

 

Marc Stock replied on Mon, 2012/10/29 - 10:20am

I was going to post a relevant comment but since the site now requires me to use IE instead of Chrome, I won't bother.  Please fix.

Dean Schulze replied on Mon, 2012/10/29 - 11:16am in response to: Marc Stock

I was able to post my comment and this reply using chrome.  What problem did you have with chrome?

Marc Stock replied on Mon, 2012/10/29 - 12:16pm in response to: Dean Schulze

The text area is set to "display:none" for some reason.  If I try and just override that, it's still not happy.  There's numerous js errors on the page though.

Tom Hicks replied on Tue, 2012/10/30 - 2:48pm in response to: Marc Stock

How did you manage to post these comments if Chrome doesn't allow you to post comments?

Just kidding, but I posted this easily with Chrome.  Looks like you might have a great career testing software :)


Marc Stock replied on Tue, 2012/10/30 - 4:14pm in response to: Tom Hicks

I blasted my cache and cookies away...all is well.  Strange stuff...I wasn't having problems with any other sites.

Nicholas Wright replied on Wed, 2012/10/31 - 5:32am

I agree with the article.  We have both JUnit tests and a GUI-based tester.  We also write tests first, before writing code.  But I agree it will never be perfect.

Recently someone added code and wrote tests using PowerMock.  OMG, how many hours did I waste on that.  "On no - you're now calling a method 3 times instead of the predicted 2." "Unexpected call to method... with an object ID which is different to the one predicted..."  Not to mention the way the Classes and Methods are mocked to do death, so in the end you're only testing the code, not the context as well.

Arthur Gorr replied on Wed, 2012/10/31 - 7:59am

There are more benefits to TDD besides regression testing.

Practicing TDD encourages separation of concerns, I could go on.

I consider the biggest benefit of TDD to be that the tests are "first consumer" of the code.

This is not a flame, just a reminder that TDD is not strictly about preventing regressions.  In my opinion the regression testing benefit is secondary.

Sarah Goff-dupont replied on Wed, 2012/10/31 - 10:57am

Agreed, Shannon.  Unit tests are a great sanity-check, but most bugs live in the integration points between different modules of your application - or integration points between you and a 3rd party service/app. Viva la integration tests!  And we still need skilled QA engineers to identify those places and put their creative-thinking skills to use imagining & writing tests for the ways they will get exercised. (...and abused, as you say.) Yes, integration tests take longer to run, but that's where test multi-threading and/or parallel-running test jobs really save the day.  My team was able to reduce a 30-minute job to 8 minutes this way. 

Nilesh Thali replied on Wed, 2012/10/31 - 1:41pm

you're absolutely right about TDD not helping you find all the bugs in your software, but i'm flogging a dead TDD horse when i say that TDD is primarily meant to improve the design of your software, not provide test coverage (a useful by-product), so when (not if) you find those bugs in your code, you will hopefully be able to isolate it to the 5-line method in a small, well-designed single responsibility (more or less) class, which caused it, instead of a 500 line method in a 2000 line class doing everything.

Arthur Gorr replied on Wed, 2012/10/31 - 2:43pm in response to: Nilesh Thali


Yes! 

Nilesh Thali == preacher

Arthur == choir

Marc Stock replied on Wed, 2012/10/31 - 3:32pm in response to: Nilesh Thali

If the only way you can keep from writing 500 line methods and 2000+ line class files is via TDD, you have much bigger problems on your hands.  Ironically, the people who are really inclined to do TDD are the ones that need it the least.

Nilesh Thali replied on Wed, 2012/10/31 - 4:01pm in response to: Arthur Gorr

thanks. you're saved! :) tell all the people that you see, "follow me".

Ben Emmett replied on Thu, 2012/11/01 - 1:51pm

Agreed - unit tests and testers find different sorts of problem. We write a lot of unit tests, but we also have a lot of testers (about 1 tester for every 1 – 2 developers) who are essential for finding ui glitches, workflow problems, etc. We separately use an automated error reporting system to let users send in details if an application crashes, which helps us find the bugs which unit testing is unlikely to catch and which get missed by QA. It means we usually find out about any nasty bugs within a few days of shipping and can fix it a lot faster than waiting for customers to complain. A tester I work with wrote about it at http://www.simple-talk.com/blogs/2011/09/08/smartassembly-error-reporting-with-the-bug-finding-power-of-more-than-20-testers/ 

Nilesh Thali replied on Thu, 2012/11/01 - 2:05pm in response to: Marc Stock

not the only way, but a really great way, Marc. 

and by your second statement, if you're inferring that there are people who are already writing really, really good cohesive do-one-thing-well kind of code, then i can still think of no good reason not to do it in TDD fashion. 

best,

Nilesh

Lund Wolfe replied on Sun, 2012/11/04 - 12:47am

If you have a problem with failing unit tests then you need unit tests.  If QA testers are finding real bugs or customers are reporting bugs then you need QA testers.  If the quality of the design/code is a problem, then deal with it at that level.  Do what works in your environment to catch bugs early.

Guillermo Cruz replied on Sun, 2012/11/04 - 5:45am

I can see the point in each of these postings, IMHO what I dont see is a mention of UML. Starting projects from way back at a use case level and bringing it to an Activity Diagram and so on in UML, will help flush out the most overlooked potential bugs. Yes even as porgrammers we should be UML savvy, it's not just for analysts anymore. I love TDD but even UML simple class diagrams help you see the BIG picture before writting a unit test case. Taking a Class, Activity or Object diagram and stitching it together with Scotch Tape and hanging it in your cubicle wall will work wonders. Being able to see and confer with your collegues gives everyone a better understanding of the System under design. Now with Unit Testing, Integration testing, Developer load testing(JMeter), and QA testing everyone can go home and enjoy the weekend.

Jean Said replied on Tue, 2012/11/06 - 8:39am

TDD is not the absolute solution to detect bugs. The best solution so far is to integrate QA into the developer team. QA can then do the exploratory testing, and the developer can assist them really early in the development cycle. And when the reproducible datasets are already there, it's a breeze to automate the tests scenarios. Of course you cannot detect every possible issue, but you can try to detect most.

A Scrum teacher told people in my company that the developers are the most qualified persons to test their own code. I say bullshit. A pressured developer is focused on achieving the main, written features. Most developers I've been working with are not focused on details and we cannot always foresee what the end users, or the external systems will do in real life. We imagine secondary scenarios, but these are not often the right ones.Any external, non-developer early feedback and ideas are highly welcomed and valuable.

Jean Said replied on Tue, 2012/11/06 - 8:52am in response to: Guillermo Cruz

 UML is mostly a Mighty Architect thing, not a developer thing. And yes, you get the big picture, the usually big, utterly wrong and unrealistic picture.

UML is a communication tool, not a programming tool. If you need to communicate design patterns, ideas and complicated processes or activities, it's the right tool. If you need UML to develop your classes, I'd rather hire another coder who does actual code rather than do beautiful pictures.

UML is no silver bullet and unless my UML schemas are about components, processes or deployment, QA will not care about it.

UML does not cure cancer and will not erase your bugs from earth.

Nicholas Wright replied on Tue, 2012/11/06 - 9:05am

I don't mean to whinge, but how the heck are you meant to unsubscribe from a thread?  When I click on the link it says "Access Denied", and I've had no reply from the DZone email address :(

Rickywhore Ricky replied on Thu, 2012/11/15 - 4:07am

Combining tests can conceivably snap statesman bugs, they traverse much individual to run Thence steady desegregation tests are oft prioritized  http://www.testbells.com/000-978.html

 

Bryan Taylor replied on Sat, 2012/11/24 - 3:52pm

OMG, NO! Manual QA is a monstrosity that systematically cripples your ability to create a high frequency feedback loop with customers. Aggressive continuous deployment pipelines will destroy the manual testing model in a side-by-side competition every time. When you start piling up manual tests, you'll be forced you into longer and longer release cycles. If you aren't capable of releasing multiple times a day, you are inevitably heading down a road to producing crappy, expensive software that would wilt under real competition. 

It sounds like your problem is that you aren't doing TDD well. Unit tests are once facet of TDD, but if that's all you are relying on, you're crippling your efforts. The TDD strategy that high velocity continuous delivery shops use is a mixed BDD and unit testing strategy. BDD tests the user stories as stated in the voice of the customer. Unit tests are a secondary form of testing which have an internal developer focus: they assure APIs are clean, they prevent regressions, and they enable refactoring. Fundamentally, though, it's the BDD tests that achieve quality. 

Most "bugs" actually don't violate any perviously stated acceptance criteria, and so we should view fixing them as a prioritization decision against competing features. The actual users of the software will make better decisions than QA people who live in a bubble every time on this. There should only be one voice of the customer giving input on prioritization and if that isn't literally the customer, you've created a strategic deficiency in your software development process. 

If you ship lots of tiny releases and have good feedback mechanisms from users, and use feature flags to confine exposure of new functionality, you'll have an experience sooner or later where you ship something that breaks unstated needs, and you add the acceptance criteria or story and it gets prioritized to "right now" and you fix it same day while the feature flag is still set to the beta users group. After this happens, you'll look on big QA as the wasteful monstrosity that it is. Nobody that I know that has tried this model ever goes back. 



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.