TestNG vs JUnit 5 battle

Sergey Pirogov

Senior Automation QA

@s_pirogov

http://automation-remarks.com

Java is on top

Java - it is reliable

49%

Python - easy and fast

21%

Same as main project

17%

Other

10%

http://automated-testing.info/t/opros-na-kakom-yazyke-delat-avtomatizacziyu/12252

People choose TestNG

65%

35%

TestNG

JUnit

http://automated-testing.info/t/junit-vs-testng-nuzhna-pomoshh-ot-kommyuniti/11988

JUnit 4 kung fu

@Test
public void pushTest() {
   System.out.println("Parameterized Number is : " + number);
}

@RunWith(value = Parameterized.class)
public class JunitTest6 {

	 private int number;

	 public JunitTest(int number) {
	    this.number = number;
	 }

	 @Parameters
	 public static Collection<Object[]> data() {
	   Object[][] data = new Object[][] { { 1 }, { 2 } };
	   return Arrays.asList(data);
	 }
        
         @Rule
         public ExpectedException expectedEx = ExpectedException.none();

         @Rule         
         public VideoRule video = new VideoRule();

         @Test
         @Video
         public void pushTest() {
            expectedEx.expect(RuntimeException.class);
            expectedEx.expectMessage("Employee ID is null");
            System.out.println("Parameterized Number is : " + number);
         }
}

Parametrize test

Check exception type and message

Add Video Recorder Support

@Listeners(VideoListener.class)
public class TestNGParametrizedTest {

  @DataProvider
  public static Object[][] number() {
    return new Object[][]{{1}, {2}, {3}};
  }

  @Video
  @Test(expectedExceptions = RuntimeException.class,
          expectedExceptionsMessageRegExp = "Employee ID is null",
          dataProvider = "number")
  public void thisIsParametrized(int number) {
    System.out.println("Parameterized Number is : " + number);
  }
}

TestNG kung fu

TestNG is widely used because...

Beloved by a lot of AQA

Easier test parametrization

Parallel test run support

Have a lot of point to extend

A lot of training materials

Is there any chance Junit 5 beat TestNG?

Public is not required!

class JUnit5Test {

    @Test
    void someTest() {
        assertTrue(true);
    }
}

New annotation names

@BeforeClass

@Before

@AfterClass

@After

Before

Now

@BeforeAll

@BeforeEach

@AfterEach

@AfterAll

Composed Annotations

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Tag("Fast")
@Tag("UI")
@Test
public @interface FastUiTest {}

@Test
@Tag("UI")
@Tag("Fast")
void fastTest(){
    assertTrue(false);
}

@FastUiTest
void fastTest(){
    assertTrue(false);
}

Before

Now

Soft Assertions

@Test
void testWithSoftAssert() {
    User user = new User("Dima", 27);

    assertAll(
            () -> assertEquals("Dma", user.getName()),
            () -> assertEquals(26, user.getAge())
    );
}

org.opentest4j.MultipleFailuresError: Multiple Failures (2 failures)
expected: <Dma> but was: <Dima>
expected: <26> but was: <27>

private SoftAssert softAssert = new SoftAssert();

@Test
public void testForSoftAssertionFailure() {
  softAssert.assertTrue(false);
  softAssert.assertEquals(1, 2);
  softAssert.assertAll();
}

Extension points

Before

Now

RunWith
Rule
ClassRule
TestWatcher

Test Instance Post Processor
BeforeAll Callback
Test Execution Condition
BeforeEach Callback
Parameter Resolution
Before Test Execution
After Test Execution
Exception Handling
AfterEach Callback
AfterAll Callback

JUnit DisableIfOS

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(DisableIfCondition.class)
public @interface DisableIfOS {
  String value();
}

public class DisableIfCondition implements TestExecutionCondition {
  @Override
  public ConditionEvaluationResult evaluate(TestExtensionContext context) {
     // logic here
  }
}

Junit DisableIfOs

public class DisabledTest {

  @Test
  @DisableIfOS(WINDOWS)
  void thisIsDisabledOnWindows() {
    assert false;
  }
}

TestNG DisableIfOs

@Listeners(DisableTransformer.class)
public class DisabledTest {

  @Test
  @DisableIfOS(WINDOWS)
  void thisIsDisabledOnWindows() {
    assert false;
  }
}

public class DisableTransformer implements IAnnotationTransformer {
  @Override
  public void transform(ITestAnnotation annotation, 
                        Class testClass, 
                        Constructor testConstructor, 
                        Method testMethod) {
    if (isDisabled(testMethod)) {
      annotation.setEnabled(false);
    }
  }

  private boolean isDisabled(Method testMethod) {
    DisableIfOs annotation = testMethod
                               .getDeclaredAnnotation(DisableIfOs.class);
    return annotation != null && System.getProperty("os.name")
                               .startsWith(annotation.value());
  }
}

Junit 5 Parameter injection

@ExtendsWith(UserParameterResolver.class)
class Parametrized{

    @Test
    void canRegister(User user) {
    // do something with `server`
    }
}

public class UserParameterResolver implements ParameterResolver {
  @Override
  public boolean supports(ParameterContext parameterContext, 
                          ExtensionContext extensionContext){
    return parameterContext.getParameter().getType().equals(User.class);
  }

  @Override
  public Object resolve(ParameterContext parameterContext, 
                        ExtensionContext extensionContext){
    return new User("Ivan");
  }
}

TestNG parameter injection

  @Test(dataProvider = "user", 
        dataProviderClass = UserDataProvider.class)
  public void register(User user) {
    assert user.getName().equals("Ivan");
  }

public class ParameterTransformer implements IAnnotationTransformer {
  @Override
  public void transform(ITestAnnotation annotation, 
                        Class testClass,
                        Constructor testConstructor, 
                        Method testMethod) {

    Class<?>[] parameterTypes = testMethod.getParameterTypes();

    if (Arrays.asList(parameterTypes).contains(User.class)) {
      annotation.setDataProviderClass(UserDataProvider.class);
      annotation.setDataProvider("user");
    }

  }

TestNG smart injection

@Listeners(ParameterTransformer.class)
class Test{  

  @Test
  public void register(User user) {
    assert user.getName().equals("Ivan");
  }

}

Test Execution Listeners

public class VideoExtension implements BeforeTestExecutionCallback, 
                                       AfterTestExecutionCallback {

  private IVideoRecorder recorder;

  @Override
  public void beforeTestExecution(TestExtensionContext context) throws Exception {
    recorder = RecorderFactory.getRecorder(VideoRecorder.conf().getRecorderType());
    recorder.start();
  }

  @Override
  public void afterTestExecution(TestExtensionContext context) throws Exception {
    File video = stopRecording(fileName);
  }

JUnit 5 Apply Listeners

@ExtendWith(VideoExtension.class)
public class AutomaticDriverManagerTest {

     @Test
     void testCanDoMagic(){

     }
}

public class AutomaticDriverManagerTest {

     @Test
     @ExtendWith(VideoExtension.class)
     void testCanDoMagic(){

     }
}

public class AutomaticDriverManagerTest {

     @Test
     @Video
     void testCanDoMagic(){

     }
}

JUnit 5 Extensions

Summary

flexible because of many extension points

extensions compose well

customizable due to meta-annotations

TestNG listeners bug

@Listeners(VideoListener.class)
public class TestNgVideoTest {

    @Test
    @Video
    public void shouldFail() {
        Thread.sleep(1000);
        assert false;
    }
}

https://github.com/cbeust/testng/issues/88

JUnit 5 Build tools support

apply plugin: 'org.junit.platform.gradle.plugin'

junitPlatform {
    // filter tests to run
    selectors {
        packages 'com.acme.foo', 'com.acme.bar'
        methods 'com.acme.Foo#a', 'com.acme.Foo#b'
        method 'com.example.app.Application#run(java.lang.String[])'
    }
    //

    tags {
        include 'fast', 'smoke'
        exclude 'slow', 'ci'
    }

    packages {
        include 'com.sample.included1', 'com.sample.included2'
        exclude 'com.sample.excluded1', 'com.sample.excluded2'
    }   
}

TestNG Build tools support

apply plugin: 'java'

test {
     useTestNG {
         suites 'src/main/resources/testng.xml'
     }
}

Boilerplate is gone

public class JUnit5Parametrized {
  
  @Video
  @ParametrizedTest
  public void pushTest(Integer number) {
    Throwable exception = assertThrows(RuntimeException.class, () -> {
      System.out.println(getUserById(number));
    });
    assertEquals("Employee ID is null",exception.getMessage());
  }
}

@RunWith(value = Parameterized.class)
public class Junit4Parametrized {

	 private int number;

	 public JunitTest(int number) {
	    this.number = number;
	 }

	 @Parameters
	 public static Collection<Object[]> data() {
	   Object[][] data = new Object[][] { { 1 }, { 2 } };
	   return Arrays.asList(data);
	 }
        
         @Rule
         public ExpectedException expectedEx = ExpectedException.none();

         @Rule    
         public VideoRule video = new VideoRule();

         @Video
         @Test
         public void pushTest() {
            expectedEx.expect(RuntimeException.class);
            expectedEx.expectMessage("Employee ID is null");
            System.out.println("Parameterized Number is : " + number);
         }
}

Summary

JUnit5

Extension

TestNG

Parameters

Parallel testes

Build tools

support

Test tags

Production

JUnit 5 is a great alternative for TestNG !

Thank you!

@s_pirogov

http://automation-remarks.com