К сожалению, обработка исключений, генерируемых тестируемыми методами, всегда была громоздкой. В последние несколько месяцев произошло некоторое облегчение, поскольку JUnit и TestNG предприняли попытку улучшить обработку ошибок:

• JUnit 4.7 добавил правила с ExpectedException в качестве одной реализации

• TestNG 5.10 добавил атрибут Ожидаемое исключениеMessageRegExp к аннотации @Test

Однако, несмотря на эти усилия, накладные расходы на обработку исключений все еще довольно высоки.

Визуальный подход, введенный MagicTest, с другой стороны, делает обработку исключений проще простого

В этой статье будут представлены функции, реализованные в JUnit и TestNG, и проведено сравнение с новым визуальным подходом, представленным MagicTest.

пример

Чтобы сравнить различные подходы, мы рассмотрим пример метода, который возвращает значение конфигурации. Тестируемый метод должен соответствовать следующим требованиям:

• Если пространство имен и ключ верны, необходимо вернуть значение конфигурации

• Если пространство имен допустимо, но ключ неизвестен, должна быть выдана ошибка, содержащая неверный ключ

• Если пространство имен недопустимо, должна быть выдана ошибка, содержащая недопустимое пространство имен

Так что простая реализация может выглядеть так:

public class Config {
	public static Object getValue(String namespace, String key) {
		if (!namespace.equals("database")) {
			throw new IllegalArgumentException("Unknown namespace: " + namespace);
		}
		if (!key.equals("name")) {
			throw new IllegalArgumentException("Unknown key: " + key);
		}
		return "db1";
	}
}

Наш метод испытаний должен просто проверить три требования. В общем, нам просто нужно написать следующее:

// Successful tests 
Config.getValue("database", "name");	// result: "db1"

// Tests with errors
Config.getValue("database", "UNKNOWN");	// error: "Unknown key: UNKNOWN"
Config.getValue("UNKNOWN", "UNKNOWN");	// error: "Unknown namespace: UNKNOWN"

As we will see, MagicTest allows us to use exactly these three lines of code to cover all testing needs.

But how would we implement testing these requirements using JUnit and TestNG?

Traditional approach

Before TestNG entered the stage, you probably used JUnit 3 for testing – which offered no support at all for testing error cases. So you had to catch the exceptions manually and ended up with code like this:

public class ConfigTest_Traditional {
	@Test
	public void testGetValue() {
		assertEquals(Config.getValue("database", "name"), "db1");

		try {
			Config.getValue("database", "UNKNOWN");
			fail("Exception must be thrown");
		} catch (IllegalArgumentException e) {
			// expected exception
		}

		try {
			Config.getValue("UNKNOWN", "UNKNOWN");
			fail("Exception must be thrown");
		} catch (IllegalArgumentException e) {
			assertEquals(e.getMessage(), "Unknown namespace: UNKNOWN");
		}
	}
}

While this approach forced you to type a lot, you had the possibilty to check the exception messages for correctness and also check several exceptions in one test method.

TestNG

TestNG added the annotation expectedExceptions which allowed you to easily check whether an exception of the given type was thrown. This could save you quite some lines of code — if you had only to check a single expection case in your test method and did not have to check the exception message.

TestNG 5.10 then added the attribute expectedExceptionsMessageRegExp, which allows checking the content of the exception message.

public class ConfigTest_TestNG510 {
	@Test
	public void testConcat() {
		assertEquals(Config.getValue("database", "name"), "db1");
	}
	
	@Test(expectedExceptions = IllegalArgumentException.class)
	public void testConcatErr1() {
		Config.getValue("database", "UNKNOWN");
	}

	@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = ".*null.*")
	public void testConcatErr2() {
		Config.getValue("UNKNOWN", "UNKNOWN");
	}
}

JUnit

With version 4, JUnit adapted the annotation based configuration approach of TestNG and the ability to easily check for an expected exception. As the approach is the same, so are the drawbacks: no checking of exception message and only one exceptionper test method can be handled.

JUnit 4.7 added a concept called rules. One implementation of a rule is called ExpectedException and allows you to specify the expected exception with its message programmatically.

public class ConfigTest_Junit47 {
	@Test
	public void testGetValue() {
		assertEquals(Config.getValue("database", "name"), "db1");
	}
	
	@Test(expected = IllegalArgumentException.class)
	public void testGetValueErrJUnit40() {
		Config.getValue("database", "UNKNOWN");
	}

	@Rule
	public ExpectedException errJUnit47 = ExpectedException.none();

	@Test
	public void testGetValueErrJUnit47() {
		errJUnit47.expect(IllegalArgumentException.class);
		errJUnit47.expectMessage("Unknown namespace: UNKNOWN");
		Config.getValue("UNKNOWN", "UNKNOWN");
	}
}

Using ExpectedException it is possible to check the message of the exception, but if we compare the lines of code used for this approach with the traditional try-catch approach, we can conclude that both need five lines, so the benefit of this feature seems to remain quite low.

MagicTest

The visual approach introduced by MagicTest makes testing error cases as easy as sucessful ones. So we really just need to code three lines to test our three requirements:

public class ConfigTest {
	@Trace
	public void testGetValue() {
		// Successful tests 
		Config.getValue("database", "name");	// result: "db1"
		
		// Tests with errors
		Config.getValue("database", "UNKNOWN");	// error: "Unknown key: UNKNOWN"
		Config.getValue("UNKNOWN", "UNKNOWN");	//  error: "Unknown namespace: UNKNOWN"
	}
}

As MagicTest automatically catches and reports exceptions but then normally continues processing, we can have several exception cases in a row – something which is not possible using JUnit or TestNG without having to explicitly use try-catch clauses.

If we run this test, the test report shows us the result of the three method calls. As the test is run the first time, there is no reference result yet and the steps are therefore considered as failed.

If we then confirm the actual result to be correct by clicking on the Save link, it is saved as new reference result and therefore the test is successful.

Behind the scenes

How does this work? The magic lies in the @Trace annotation. It tells MagicTest to trace all calls to the method getValue and reports parameters and result.

To generate the needed output without having to manually code these statements, the byte code is instrumented before the test method is executed: So for each call to the getValue() method parameters, return value, and thrown exception are automatically traced.

So a call to the method under test will roughly look like this:

try {
   printParameters("database", "name");
   String result = Config.getValue("database", "name");
   printResult(result);
} catch (Throwable t) {
   printError(t);
}

The data traced out with the print-methods is then collected and visualized as HTML report.

Conclusion

We can try to collect the bare facts of the different approaches in a table:

Obviously the approach featured by MagicTest looks very promising.

Additionally the possibility to check successful and error cases in a single test method can greatly reduce the number of test methods needed – which may be easier to handle even if theory says that one test should just test a single thing.

Have a look at http://www.magicwerk.org/magictest to find out more about MagicTest.