Sarah is an Agile Delivery evangelist with Atlassian, a former test automation engineer, and a fan of anything that makes life easier for the nerds. Like continuous integration and automation, for example. Because manual testing gets boring. Her heros include Margret Thatcher, MacGyver, and her mom. Sarah has posted 2 posts at DZone. You can read more from them at their website. View Full User Profile

How I Learned to Stop Worrying and Love TestNG

03.02.2013
| 4200 views |
  • submit to reddit

Think back to your early 20′s. Remember going out on Halloween or New Year’s Eve? No sooner do you walk into a bar than someone in your group starts lobbying to go to some other bar. ‘Cuz it’s gonna be way better. (In your exasperation – you just ordered a drink for pete’s sake! –  you vow that next year you’re gonna find a house party instead.)

Apologies in advance, but… I’m gonna be “that someone”. Replace New Year’s Eve with every Monday through Friday. The bar you just walked into is JUnit. That other bar you should, like, totally make a b-line for is TestNG - an open-source automated testing framework.

Now, where this analogy breaks down (assuming it ever had legs to begin with), is that unlike the human confines of singular spacio-temporal existence™, you can use both JUnit and TestNG at once. Here are 3 reasons you should.

Groups

This is hands-down my favorite thing about TestNG because it’s the most flexible way to split your tests into sub-suites. Let’s jump right to an example, inspired by Bamboo (a CI server from Atlassian). Here we see two tests around ordering sub-steps, or "Tasks", within a build job:


Each test has been assigned to groups representing functional area of the application, level of technology stack, maturity level, and whether it should be run as a post-deploy smoke test. This is incredibly powerful because you can use these assignments at run time to pull tests from disparate classes or packages and run them inside a single job. A job that runs all the API-level tests having to do with configuration functionality, for example. Or a job that runs all the tests for features that are still in development and not really expected to pass yet (TDD, anyone??).

And you guessed it: the logical next step is to create jobs that cover the fully body of tests you wish to execute, and run those jobs in parallel inside a stage. Check out this post for more implementation details.

@DataProvider

Let’s say you’re developing, oh I dunno… a continuous integration server. Let’s also say that your CI server needs to support switching between branches from build to build. There are loads of permutations you want to test, but you’re verifying the same functionality for each of them.

What’s a dev to do? Duplicate the hell outta their test code, making explicit sets of tests for each repo type/branch type combination? Oh, the humanity! But wait: up there in the sky! It’s a bird… no, it’s a plane… no! it’s @DataProvider! Here’s a simple example from TestNG’s site:


The output of that test will be: 


But wait. Junit supports parameterized tests as of v.4.4, which is essentially what this is. So why bother with TestNG? First, TestNG’s implementation is slicker and easier than JUnit’s. Second, and more importantly, the reporting in TestNG is much clearer. TestNG tells you exactly which inputs were used for each execution, making it much easier to diagnose failures.


JUnit just tells you which array the parameters came from, and their positions therein – 'bout as clear as mud.

Skipped Tests

Adherents to the D.R.Y principal (and that’s most of us) rely on @BeforeTest or @BeforeClass set-up methods. But if something goes awry in set-up, there’s no point in trying to run the actual test. We know it just gonna fail. With TestNG, the default behavior is to simply skip a test if its upstream dependency barfs. I love this for two reasons.

First, executing the actual test is a waste of time and just means you wait that much longer to find out this revision of your code no es bueno. Second, when you look at your test results, your count of skipped tests makes it immediately clear that something fundamental to your application is borked. Much faster than diving into 350 test failures and, somewhere around number 20, noticing that they’ve all failed on exactly the same call.

And it’s not just set-up methods that can trigger a skip. If you’ve declared TestA as being dependent upon TestB, and TestA fails, TestB will be skipped. Here’s what that looks like in your build logs:


Two Can Play at This Game…

Incorporating TestNG into your arsenal is easy, and again: you can use it in concert with JUnit. Add it as a dependency in your Maven POM, or as an Ant task. Then get the TestNG plugin for Eclipse or IDEA and start sprucing up your tests. Visit testng.org for details.

Among the goodies in Bamboo’s latest release is native support for parsing TestNG results - @DataProvider, skipped tests… the whole works. Just add the TestNG Parser as a final task in the jobs that execute tests. If you’re running the tests by way of Maven, Ant or Gradle, be sure to disable test reporting in those tasks so there are no conflicts. The TestNG plugin for Jenkins also supports all three features I've talked about. 

Try it out and see if you love TestNG, too!


ps: For more tips and best practices, be sure to check out our new automated testing page with all sorts of helpful info. Even automation gurus are sure to find at least one new idea )

Published at DZone with permission of its author, Sarah Goff-dupont.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Nicolas Frankel replied on Sat, 2013/03/02 - 4:21am

 Hi Sarah,

I'm also a TestNG proponent and I agree with most of your article. Where I heartily disagree is to use both. What's the point? TestNG can do everything JUnit does!

Remember the KISS principle...

Fabrizio Giudici replied on Sat, 2013/03/02 - 3:41pm

Agreed with Nikolas. It's easy to port JUnit test to TestNG as there's a compatibile assert() method (not trivial: the native assert() for TestNG has got reversed arguments). There's also a plugin for NetBeans and Hudson/Jenkins.

Sarah Goff-dupont replied on Mon, 2013/03/04 - 12:00pm

Hey guys! I too feel that you can easily use TestNG exclusively and have all your needs covered (and then some). Its ability to co-habitate with JUnit is most useful when you are first introducing it - no need to take the plunge and do a full switch-over just to try TestNG out.   Ultimately, switching over 100% will make for easier maintenance in the long run. But for teams with thousands of tests using JUnit, it's a bit project to go in and flip the contents of your assert() statements around. So co-habitation removes that (rather significant) barrier to entry. 

Thanks for reading!

Nicolas Frankel replied on Sun, 2013/03/10 - 9:32am

 The main argument against using both in the same project is Fabrizio's: assert arguments are opposite in JUnit & TestNG (actual vs expected). Just migrate and be done with it.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.