TestNG vs JUnit 5 battle

Sergey Pirogov

Senior Automation QA

@s_pirogov

http://automation-remarks.com

Java is on top

Java - it is reliable

49%

Python - easy and fast

21%

Same as main project

17%

Other

10%

http://automated-testing.info/t/opros-na-kakom-yazyke-delat-avtomatizacziyu/12252

People choose TestNG

65%

35%

TestNG

JUnit

http://automated-testing.info/t/junit-vs-testng-nuzhna-pomoshh-ot-kommyuniti/11988

JUnit 4 kung fu

@Test
public void pushTest() {
   System.out.println("Parameterized Number is : " + number);
}
@RunWith(value = Parameterized.class)
public class JunitTest6 {

	 private int number;

	 public JunitTest(int number) {
	    this.number = number;
	 }

	 @Parameters
	 public static Collection<Object[]> data() {
	   Object[][] data = new Object[][] { { 1 }, { 2 } };
	   return Arrays.asList(data);
	 }
        
         @Rule
         public ExpectedException expectedEx = ExpectedException.none();
         @Rule         
         public VideoRule video = new VideoRule();

         @Test
         @Video
         public void pushTest() {
            expectedEx.expect(RuntimeException.class);
            expectedEx.expectMessage("Employee ID is null");
            System.out.println("Parameterized Number is : " + number);
         }
}
  • Parametrize test
  • Check exception type and message
  • Add Video Recorder Support
@Listeners(VideoListener.class)
public class TestNGParametrizedTest {

  @DataProvider
  public static Object[][] number() {
    return new Object[][]{{1}, {2}, {3}};
  }

  @Video
  @Test(expectedExceptions = RuntimeException.class,
          expectedExceptionsMessageRegExp = "Employee ID is null",
          dataProvider = "number")
  public void thisIsParametrized(int number) {
    System.out.println("Parameterized Number is : " + number);
  }
}

TestNG kung fu

TestNG is widely used because...

  • Beloved by a lot of AQA  
  • Easier test parametrization
  • Parallel test run support    
  • Have a lot of point to extend
  • A lot of training materials     

Is there any chance Junit 5 beat TestNG?

Public is not required! 

class JUnit5Test {

    @Test
    void someTest() {
        assertTrue(true);
    }
}

New annotation names

@BeforeClass
@Before
@AfterClass
@After

Before

Now

@BeforeAll
@BeforeEach
@AfterEach
@AfterAll

Composed Annotations

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Tag("Fast")
@Tag("UI")
@Test
public @interface FastUiTest {}
@Test
@Tag("UI")
@Tag("Fast")
void fastTest(){
    assertTrue(false);
}
@FastUiTest
void fastTest(){
    assertTrue(false);
}

Before

Now

Soft Assertions

@Test
void testWithSoftAssert() {
    User user = new User("Dima", 27);

    assertAll(
            () -> assertEquals("Dma", user.getName()),
            () -> assertEquals(26, user.getAge())
    );
}

org.opentest4j.MultipleFailuresError: Multiple Failures (2 failures)
    expected: <Dma> but was: <Dima>
    expected: <26> but was: <27>

private SoftAssert softAssert = new SoftAssert();

@Test
public void testForSoftAssertionFailure() {
  softAssert.assertTrue(false);
  softAssert.assertEquals(1, 2);
  softAssert.assertAll();
}

Extension  points

Before

Now

  • RunWith
  • Rule
  • ClassRule
  • TestWatcher
  • Test Instance Post Processor
  • BeforeAll Callback
  • Test Execution Condition
  • BeforeEach Callback
  • Parameter Resolution
  • Before Test Execution
  • After Test Execution
  • Exception Handling
  • AfterEach Callback
  • AfterAll Callback

JUnit DisableIfOS

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(DisableIfCondition.class)
public @interface DisableIfOS {
  String value();
}
public class DisableIfCondition implements TestExecutionCondition {
  @Override
  public ConditionEvaluationResult evaluate(TestExtensionContext context) {
     // logic here
  }
}

Junit DisableIfOs

public class DisabledTest {

  @Test
  @DisableIfOS(WINDOWS)
  void thisIsDisabledOnWindows() {
    assert false;
  }
}

TestNG DisableIfOs

@Listeners(DisableTransformer.class)
public class DisabledTest {

  @Test
  @DisableIfOS(WINDOWS)
  void thisIsDisabledOnWindows() {
    assert false;
  }
}
public class DisableTransformer implements IAnnotationTransformer {
  @Override
  public void transform(ITestAnnotation annotation, 
                        Class testClass, 
                        Constructor testConstructor, 
                        Method testMethod) {
    if (isDisabled(testMethod)) {
      annotation.setEnabled(false);
    }
  }

  private boolean isDisabled(Method testMethod) {
    DisableIfOs annotation = testMethod
                               .getDeclaredAnnotation(DisableIfOs.class);
    return annotation != null && System.getProperty("os.name")
                               .startsWith(annotation.value());
  }
}

Junit 5 Parameter injection

@ExtendsWith(UserParameterResolver.class)
class Parametrized{

    @Test
    void canRegister(User user) {
    // do something with `server`
    }
}
public class UserParameterResolver implements ParameterResolver {
  @Override
  public boolean supports(ParameterContext parameterContext, 
                          ExtensionContext extensionContext){
    return parameterContext.getParameter().getType().equals(User.class);
  }

  @Override
  public Object resolve(ParameterContext parameterContext, 
                        ExtensionContext extensionContext){
    return new User("Ivan");
  }
}

TestNG parameter injection

  @Test(dataProvider = "user", 
        dataProviderClass = UserDataProvider.class)
  public void register(User user) {
    assert user.getName().equals("Ivan");
  }
public class ParameterTransformer implements IAnnotationTransformer {
  @Override
  public void transform(ITestAnnotation annotation, 
                        Class testClass,
                        Constructor testConstructor, 
                        Method testMethod) {

    Class<?>[] parameterTypes = testMethod.getParameterTypes();

    if (Arrays.asList(parameterTypes).contains(User.class)) {
      annotation.setDataProviderClass(UserDataProvider.class);
      annotation.setDataProvider("user");
    }

  }

TestNG smart injection

@Listeners(ParameterTransformer.class)
class Test{  

  @Test
  public void register(User user) {
    assert user.getName().equals("Ivan");
  }

}

Test Execution Listeners

public class VideoExtension implements BeforeTestExecutionCallback, 
                                       AfterTestExecutionCallback {

  private IVideoRecorder recorder;

  @Override
  public void beforeTestExecution(TestExtensionContext context) throws Exception {
    recorder = RecorderFactory.getRecorder(VideoRecorder.conf().getRecorderType());
    recorder.start();
  }

  @Override
  public void afterTestExecution(TestExtensionContext context) throws Exception {
    File video = stopRecording(fileName);
  }

JUnit 5 Apply Listeners

@ExtendWith(VideoExtension.class)
public class AutomaticDriverManagerTest {

     @Test
     void testCanDoMagic(){

     }
}
public class AutomaticDriverManagerTest {

     @Test
     @ExtendWith(VideoExtension.class)
     void testCanDoMagic(){

     }
}
public class AutomaticDriverManagerTest {

     @Test
     @Video
     void testCanDoMagic(){

     }
}

JUnit 5 Extensions

Summary

  • flexible because of many extension points
  • extensions compose well
  • customizable due to meta-annotations

TestNG listeners bug

@Listeners(VideoListener.class)
public class TestNgVideoTest {

    @Test
    @Video
    public void shouldFail() {
        Thread.sleep(1000);
        assert false;
    }
}

https://github.com/cbeust/testng/issues/88

JUnit 5 Build tools support

apply plugin: 'org.junit.platform.gradle.plugin'

junitPlatform {
    // filter tests to run
    selectors {
        packages 'com.acme.foo', 'com.acme.bar'
        methods 'com.acme.Foo#a', 'com.acme.Foo#b'
        method 'com.example.app.Application#run(java.lang.String[])'
    }
    //

    tags {
        include 'fast', 'smoke'
        exclude 'slow', 'ci'
    }

    packages {
        include 'com.sample.included1', 'com.sample.included2'
        exclude 'com.sample.excluded1', 'com.sample.excluded2'
    }   
}

TestNG Build tools support

apply plugin: 'java'

test {
     useTestNG {
         suites 'src/main/resources/testng.xml'
     }
}

Boilerplate is gone

public class JUnit5Parametrized {
  
  @Video
  @ParametrizedTest
  public void pushTest(Integer number) {
    Throwable exception = assertThrows(RuntimeException.class, () -> {
      System.out.println(getUserById(number));
    });
    assertEquals("Employee ID is null",exception.getMessage());
  }
}
@RunWith(value = Parameterized.class)
public class Junit4Parametrized {

	 private int number;

	 public JunitTest(int number) {
	    this.number = number;
	 }

	 @Parameters
	 public static Collection<Object[]> data() {
	   Object[][] data = new Object[][] { { 1 }, { 2 } };
	   return Arrays.asList(data);
	 }
        
         @Rule
         public ExpectedException expectedEx = ExpectedException.none();
         @Rule    
         public VideoRule video = new VideoRule();

         @Video
         @Test
         public void pushTest() {
            expectedEx.expect(RuntimeException.class);
            expectedEx.expectMessage("Employee ID is null");
            System.out.println("Parameterized Number is : " + number);
         }
}

Summary

JUnit5​

Extension

TestNG

Parameters

Parallel testes

Build tools

support

Test tags

Production

JUnit 5 is a great alternative for TestNG !

Thank you!

@s_pirogov

http://automation-remarks.com

TestNG vs JUnit battle

By Sergey Pirogov

Made with Slides.com

TestNG vs JUnit battle

  • 15,969

More from Sergey Pirogov