Вот несколько заметок.
Эта проблема
TestNG позволяет вам объявлять зависимости между вашими методами тестирования. Вот простой пример:
@Test
public void a1() {}
@Test(dependsOnMethods = "a1")
public void b1() {}
@Test
public void x() {}
@Test
public void y() {}
В этом примере b1 () не будет работать, пока a1 () не завершится и не пройдет. В случае сбоя a1 (), b1 () будет помечен как «Пропущено». Для целей этих статей я называю оба метода a1 () и b1 () «зависимыми», а x () и y () — «свободные» методы.
Все становится интереснее, если вы хотите запустить эти четыре метода тестирования параллельно. Когда вы указываете, что эти методы должны выполняться в пуле из трех потоков, TestNG по-прежнему необходимо поддерживать порядок a1 () и b1 (). Это достигается путем запуска всех зависимых методов в одном потоке, гарантируя, что они не только не будут перекрываться, но и будет строго соблюдаться порядок.
Текущий алгоритм поэтому прост:
- Разбейте все методы тестирования на две категории: бесплатные и зависимые.
- Свободные методы добавляются в пул потоков и выполняются Исполнителем, по одному методу на каждого работника, что гарантирует полный параллелизм.
- Зависимые методы сортируются и запускаются в исполнителя, который содержит только один поток.
Это был алгоритм планирования в течение более пяти лет. Работает отлично, но не оптимально.
улучшения
Зависимые методы являются очень популярной функцией TestNG, особенно в таких средах веб-тестирования, как Selenium, где тестирование страниц очень зависит от порядка выполнения операций на этих страницах. Эти тесты, как правило, сделаны из большинства зависимых методов, что означает, что текущий алгоритм планирования делает очень трудным использование какого-либо параллелизма вообще в этих ситуациях.
Например, рассмотрим следующий пример:
Поскольку все четыре метода являются зависимыми, все они будут работать в одном потоке, независимо от размера пула потоков. Очевидным улучшением будет запуск a1 () и b1 () в одном потоке, а a2 () и b2 () в другом потоке.
Но почему бы не продвинуться дальше и посмотреть, не можем ли мы просто выбросить все эти четыре метода в основной пул потоков и при этом уважать их порядок?
Эта мысль заставила меня поближе взглянуть на параллельные примитивы, доступные в JDK, а точнее, на Executors .
Мой первый вопрос состоял в том, можно ли добавлять работников в Исполнителя, не обязательно готовя их к немедленному запуску, но вскоре эта идея показалась мне противоречащей принципу Исполнителей, поэтому я отказался от нее.
The other idea was to see if it was possible to start with only a few workers and then add more workers to an Executor as it’s running, which turns out to be legal (or at least, not explicitly prohibited). Looking through the existing material, it seems to me that Executors typically do not modify their own set of workers. They get initialized and then external callers can add workers to them with the execute() method.
At this point, the solution was becoming fairly clear in my mind, but before I get into details, we need to take a closer look at sorting.
Topological sort
In the example shown at the beginning, I said that TestNG was sorting the methods before executing them but I didn’t explain exactly how this was happening. As it turns out, we need a slightly different sorting algorithm than the one you are used to.
Looking back at this first example, it should be obvious that there are more than one correct way to order the methods:
- a1() b1() x() y()
- x() a1() b1() y()
- y() a1() x() b1()
- …
In short, any ordering that executes a1() before b1() is legal. What we are doing here is trying to sort a set of elements that cannot all be compared to each other. In other words, if I pick two random methods “f” and “g” and I ask you to compare them, your answer will be either “f must run before g”, “g must run before f” or “I cannot compare these methods” (for example if I give you a1() and x()).
This is called a Topological sorting. This link will tell you all you need to know about topological sorting, but if you are lazy, suffice to say that there are basically two algorithms to achieve this kind of sort.
Let’s see the execution of a topological sort on a simple example.
The following graphs represent test methods and how they depend on
each other. Methods in green are “free methods”: they don’t depend on any other methods. Arrows represent dependencies and dotted arrows are dependencies that have been satisfied. Finally, grey nodes are methods that have been executed.
First iteration, we have four free methods. These four methods are ready to be run.
Result so far: { a1, a2, x, y }
The four methods have been run and they “freed” two new nodes, b1 and b2, which become eligible for the next wave of executions. Note that while one of d’s dependencies has been satisfied (a1), d still depends on b1 so it’s not free yet.
Result so far: { a1, a2, x, y, b1, b2 }
b2 and b1 have run and they free three additional methods.
The last methods have run, we’re done.
Final result: { a1, a2, x, y, b1, b2, c1, c2, d }
Again, note that this is not the only valid topological sort for this example: you can reorder certain elements as long as the dependencies are respected. For example, a result that would start with {a2, a1} would be as correct as the one above, which starts with {a1, a2}.
This is a pretty static, academic example. In the case of TestNG, things are a lot more dynamic and the entire running environment needs to be re-evaluated each time a method completes. Another important aspect of this algorithm is that all the free methods need to be added to the thread pool as soon as they are ready to run, which means that the ExecutorService will have workers added to its pool even as it is running other workers.
For example, let’s go back to the following state:
At this stage, we have two methods that get added to the thread pool and run on two different threads: b1 and b2. We can then have two different situations depending on which one completes first:
b1 finishes first and frees both c1 and d.
Or…
b2 finishes first but doesn’t free any new node.
A new kind of Executor
Since the early days, TestNG’s model has always been very dynamic: what methods to run and when is being decided as the test run progresses. One of the improvements I have had on my mind for a while was to create a “Test Plan” early on. A Test Plan means that the engine would look at all the TestNG annotations inside the classes and it would come up with a master execution plan: a representation of all the methods to run, which I can then hand over to a runner that would take care of it.
Understanding the scenario above made me realize that the idea of a “Test Plan” was doomed to fail. Considering the dynamic aspect of TestNG, it’s just plain impossible to look at all the test classes during the initialization and come up with an execution plan, because as we saw above, the order in which the methods are run will change depending on which methods finish first. A Test Plan would basically make TestNG more static, while we need the exact opposite of this: we want to make it even more dynamic than it is right now.
The only way to effectively implement this scenario is basically to reassess the entire execution every time a test method completes. Luckily, Executors allow you to receive a callback each time a worker completes, so this is the perfect place for this. My next question was to wonder whether it was legal to add workers to an Executor when it’s already running (the answer is: yes).
Here is an overview of what the new Executor looks like.
The Executor receives a graph of test methods to run in its constructor and then simply revolves around two methods:
/**
* Create one worker per node and execute them.
*/
private void runNodes(Set<ITestNGMethod> nodes) {
List<IMethodWorker> runnables = m_factory.createWorkers(m_xmlTest, nodes);
for (IMethodWorker r : runnables) {
m_activeRunnables.add(r);
setStatus(r, Status.RUNNING);
try {
execute(r);
}
catch(Exception ex) {
// ...
}
}
}
The second part is to reassess the state of the world every time a method completes:
@Override
public void afterExecute(Runnable r, Throwable t) {
m_activeRunnables.remove(r);
setStatus(r, Status.FINISHED);
synchronized(m_graph) {
if (m_graph.getNodeCount() == m_graph.getNodeCountWithStatus(Status.FINISHED)) {
shutdown();
} else {
Set<ITestNGMethod> freeNodes = m_graph.getFreeNodes();
runNodes(freeNodes);
}
}
}
Когда работник заканчивает работу, Исполнитель обновляет свой статус на графике. Затем он проверяет, выполнили ли мы все узлы, а если нет, то запрашивает у графа новый список свободных узлов и планирует запуск этих узлов.
Завершение
This is basically a description of the new TestNG scheduling engine. I tried to focus on general concepts and I glossed over a few TestNG specific features that made this implementation more complex than I just described, but overall, implementing this new engine turned out to be fairly straightforward thanks to TestNG’s layered architecture.
With this new implementation, TestNG is getting as close to possible to offering maximal parallelism for test running, and a few Selenium users have already reported tremendous gains in test execution (from an hour down to ten minutes).
When I ran the tests with this new engine, very few tests failed and the ones that did were exactly the ones that I wanted to fail (such as on that verified that dependent methods execute in the same thread, which is exactly what this new engine is fixing). Similarly, I added new tests to verify that dependent methods are now sharing the thread pool with free methods, which turned out to be trivial since I already have plenty of support for this kind of testing.
This new engine will be available in TestNG 5.11, which you can beta test here.