Статьи

Тестирование производительности WSO2 Enterprise Service Bus (ESB) — Раунд 3

Декабрь 1, 2017

Эта статья является третьей в серии статей, которые измеряют и сравнивают производительность WSO2 Enterprise Service Bus (ESB) с другими ведущими реализациями — как с открытым исходным кодом, так и с частной собственностью.

Две предыдущие статьи доступны в качестве  WSO2 ESB тестирования производительности 1 -го раунда и WSO2 ESB Performance Testing Round 2

Содержание

  1. Вступление
  2. Примеры запросов сообщений
  3. Полнота реализации сценария и возникшие проблемы
  4. Конфигурации и настройки
  5. Полученные результаты
  6. Выводы
  7. Ресурсы

Вступление

В этом цикле тестирования мы сравнили WSO2 Enterprise Service Bus (ESB) [6] v1.7 с ведущей проприетарной ESB (PROP-S / PROP-N), проприетарной версией ESB с открытым исходным кодом (PROP-OS). ), альтернативы с открытым исходным кодом — Apache Service Mix 3.2.1 и Mule Community Edition 2.0.1 (Mule CE). WSO2 Enterprise Service Bus (ESB) — это проект с открытым исходным кодом (по лицензии Apache License v2.0), разработанный WSO2 на основе популярной Apache Synapse Enterprise Service Bus (ESB) [7]. Поскольку WSO2 ESB v1.7 встраивает выпуск Apache Synapse ESB v1.2, показатели производительности между ними будут идентичны. Однако WSO2 ESB поставляется с улучшенной поддержкой управления и разработки с графическим веб-интерфейсом, расширенной поддержкой JMX и встроеннымЭкземпляр реестра WSO2 .

Хотя изначально мы планировали включить OpenESB и JBoss ESB в раунды тестирования, из-за сложности понимания этих продуктов для создания простых сценариев, которые нам нужны, мы решили обойтись без них и вместо этого включить последующую статью. Из первого и второго раундов тестирования вы помните, что у нас было три конкретных сценария, предназначенных для тестирования производительности обработки посредника Enterprise Service Bus (ESB). Хотя эти сценарии, очевидно, не являются исчерпывающими, и мы понимаем, что могут быть и другие сценарии, в которых альтернативные тесты могут показывать разные результаты, это некоторые из наиболее распространенных сценариев использования, основанных на ситуациях, с которыми мы столкнулись. Три сценария:

 

1. Виртуализация, где ESB скрывает реальный сервис и выполняет маршрутизацию сообщений.

В этом сценарии мы создаем прокси-сервис (определенный ProxyWSDL) для фактического сервиса Echo (определенного EchoWSDL), который просто передает сообщения реальному сервису. Этот шаблон позволяет пользователям скрывать реальные сервисы за прокси-серверами, предоставляемыми через ESB, и тем самым предотвращает создание прямых ссылок между приложениями, приложениями к сервисам и / или сервисами на предприятии для наведения порядка в Сервисе. Развертывание ориентированной архитектуры (SOA). Это дополнительно позволяет пользователям выполнять аутентификацию, авторизацию, валидацию, WS-Security, WS-Reliable Messaging, SSL-дешифрование и т. Д. На уровне ESB, так что реальные услуги в организации могут быть упрощены. Это также позволяет отображать реальные услуги, доступные через различные транспорты или схемы, через другие транспорты и / или интерфейсы,а также разрешать вложения WS-Policy для согласованного применения общеорганизационных политик.

 

2. Content-based Routing (CBR), where the ESB routes on data within the message

The Content Based Routing (CBR) proxy service performs an evaluation of an XPath expression over the payload of the messages, before they are routed to the real service. For this example, we use an XML payload with a list of ‘order’ elements, and check if the first order element is for the symbol «IBM». If this condition is satisfied, we forward it to the actual service implementation, else return an error. Typically payloads are routed on transport/SOAP headers and/or user defined header elements, or payload body elements.

 

3. Transformation, where the ESB transforms the request and response messages using XSLT-based transformations

In this scenario, we expose a different interface at the proxy service that expects all elements in reverse order. Thus a client querying the WSDL of the proxy service is presented a completely different WSDL with a different schema. The messages received from the client is in reverse order, and are then transformed as expected by the real service. The response from the real service is then transformed again (i.e. reversed) and sent back to the client. This is a typical use case when newer versions of a service are exposed, and a subset of its users now require backwards compatibility with the previous schemas etc.

 

Sample Request Messages

  • A sample request message used for the Direct Proxy and CBR Proxy scenarios is given below. Refer[4] for actual requests used.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://services.samples/xsd">
    <soapenv:Body>
        <xsd:buyStocks>
            <order><symbol>IBM</symbol><buyerID>asankha</buyerID><price>140.34</price><volume>2000</volume></order>
            <order><symbol>MSFT</symbol><buyerID>ruwan</buyerID><price>23.56</price><volume>8030</volume></order>
            <order><symbol>SUN</symbol><buyerID>indika</buyerID><price>14.56</price><volume>5000000</volume></order>
        </xsd:buyStocks>
    </soapenv:Body>
</soapenv:Envelope>
  • A sample request message used for the XSLT Proxy scenario is given below. Refer[4] for actual requests used.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://services.samples/xsd">
    <soapenv:Body>
        <xsd:skcotSyub>
            <redro><lobmys>IBM</lobmys><DIreyub>asankha</DIreyub><ecirp>140.34</ecirp><emulov>2000</emulov></redro>
            <redro><lobmys>MSFT</lobmys><DIreyub>ruwan</DIreyub><ecirp>23.56</ecirp><emulov>8030</emulov></redro>
            <redro><lobmys>SUN</lobmys><DIreyub>indika</DIreyub><ecirp>14.56</ecirp><emulov>500</emulov></redro>
        </xsd:skcotSyub>
    </soapenv:Body>
</soapenv:Envelope>

 

Completeness of the Scenario Implementations and Issues Encountered

We did not encounter any issues in implementing the scenarios within the WSO2 Enterprise Service Bus (ESB) or the proprietary ESB. However, Mule CE was not able to create any real proxy services (i.e. a service that can ‘expose’ a different interface from that of a real service they hide behind, and have the ability to forward those requests to the real service), without having to write a full service implementation class in Java. The proprietary version of the open source ESB was unable to expose the WSDL for the XSLT transformation scenario due to an error. Apache ServiceMix couldn’t proxy the incoming SOAPAction to the real service implementation, and we found this to be a known issue with ServiceMix 3.2.1

 

Figure 1 — Ability to create real proxy services for the scenarios

Implementation Comments
WSO2 Enterprise Service Bus (ESB) No errors — all scenarios completed successfully with default settings
Proprietary ESB No errors — all scenarios completed successfully with ‘Production’ settings

 
Proprietary version of the open source ESB Without tuning, this ESB was unable to handle even a small load of concurrency. After tuning its maximum active threads parameter to 100, we could test upto 80 concurrent users. With this setting, anything over 80 users caused this ESB to crash

 
Mule CE 2.0.1 Without tuning, Mule CE 2.0.1 was unable to handle even a small load of concurrency. After tuning the ‘maxThreadsActive’ parameter to 100, we could test upto 80 concurrent users. With this setting, anything over 80 users caused Mule CE to crash

Consistently, we saw 1% of all requests sent to Mule CE 2.0.1 failing. This was seen for every single scenario and test conducted
Apache ServiceMix 3.2.1 Apache ServiceMix 3.2.1 was unable to propagate the incoming SOAPAction to the real service. This is a known bug with the ServiceMix 3.2.1 as per this conversation on the mailing list.

The default settings failed to handle the concurrency levels above 320. After tuning the ‘max-threads’ parameter to 100 (from 20) we were able to test all scenarios successfully without any errors

Table 1 — Issues encountered with each ESB

 

Benchmark Configurations and Adjustments Made

Implementation Comments
WSO2 Enterprise Service Bus (ESB) Sun JDK 1.5.0_15 with 512 MB Heap allocated

We tested the out-of-the box configuration of the WSO2 ESB, only increasing its default socket timeout value to 120 seconds

For the transformation scenario alone, we turned on the Apache Xalan translet compilation flag on
Proprietary ESB Custom JDK 1.5.0 with 512 MB Heap allocated

The ‘Production’ mode deployment of this ESB was used

We tested with and without additional performance improvements introduced by this ESB
Proprietary version of the open source ESB Sun JDK 1.5.0_15 with 512 MB Heap allocated

We set the maximum active threads to 100, as without this setting, we were unable to effectively benchmark this ESB at all
Mule CE 2.0.1 Sun JDK 1.5.0_15 with 512 MB Heap allocated

We set the maximum active threads to 100, as without this setting, we were unable to effectively benchmark Mule CE 2.0.1 at all

<threading-profile doThreading=»true» maxThreadsActive=»100″ maxThreadsIdle=»50″ poolExhaustedAction=»WAIT»/>
Apache ServiceMix 3.2.1 Sun JDK 1.5.0_15 with 512 MB Heap allocated

ServiceMix was tuned by setting the following values into the conf/servicemix.properties, without these settings we were unable to test for concurrency levels above ~320

servicemix.corePoolSize = 20

servicemix.maximumPoolSize = 100

servicemix.queueSize = -1

Table 2 — Tuning of each of the systems

 

Backend Service Implementation

We developed a mock EchoService as a servlet which simulated a 10ms processing time, and deployed this on an Apache Tomcat 5.5.26 instance.We soon noticed Tomcat running out of resources to support the load test, with a message «SEVERE: All threads (150) are currently busy, waiting. Increase maxThreads (150) or check the servlet status». We then increased the maxThreads and acceptCount to 600, and allocated 512M of heap memory. The MockService WAR and its source code can be found at[2]

Client implementation

We used the excellent Apache HttpComponents projects’ Benchmark code, which is a good replacement for the ApacheBench load generator, with support for HTTP 1.1 and SSL. This code has now been exposed as a simple to use binary JavaBench, and its source code/binary is available at[1]

Benchmarking methodology followed

We used the shell scripts available at[3] to drive each scenario. A warm up run of each of the three scenarios were executed by the script before the actual tests for each ESB began. We also re-started the Apache Tomcat backend before each test run. The output of the shell scripts were re-directed to a text file, and was fed through a utility to convert into a CSV file, which was then used to generate the tables and graphs.

 

Results

Important: In the following tables and charts, the proprietary ESB used, and the proprietary version of the open source ESB are indicated with the following keys:

  • «PROP-S» — The proprietary ESB with its additional performance improvements turned on, and started in ‘Production’ mode
  • «PROP-N» — The proprietary ESB without the additional performance improvements, and started in ‘Production’ mode
  • «PROP-OS» — The proprietary version of an open source ESB

1. Direct Proxy Service — Virtualization

Observations:

  1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
  2. Mule CE 2.0.1 consistently failed on ~1% of the requests for each and every scenario tested, even for the low concurrency tests.
  3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
  4. The proprietary ESB demonstrated a clear leadership over all scenarios, with or without additional performance improvements turned on
  5. The proprietary ESB performed 1.6~1.9 times TPS on average (with and without additional performance improvements) than the WSO2 ESB
  6. WSO2 Enterprise Service Bus (ESB) v1.7 with its default configuration beat all other Open Source alternatives consistently
  7. The WSO2 ESB average TPS was over 4 times than that of Mule CE and the proprietary version of the open source ESB, and over twice that of Apache ServiceMix

2. Content Based Routing

Observations:

  1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
  2. Mule CE 2.0.1 consistently failed on ~1% of requests for each and every scenario tested, even for the low concurrency tests.
  3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
  4. The WSO2 ESB performed better than the proprietary ESB (even with its additional performance improvements turned on) for 1/2K and 1K messages.
  5. For 5K messages, the proprietary ESB did slightly better than the WSO2 ESB.
  6. WSO2 Enterprise Service Bus (ESB) v1.7 with its default configuration beat all other Open Source alternatives consistently
  7. The WSO2 ESB average TPS was roughly over 3 times than that of Mule CE, the proprietary version of the open source ESB, and Apache ServiceMix

3. XSLT Transformations

Notes:

  • We turned on the default Apache Xalan translet transformation flag with the WSO2 ESB and this result is shown as «WSO2-*»
  • The proprietary ESB did not have any additional performance optimizations for this scenario, where the message was being transformed during proxying

Observations:

  1. Mule CE 2.0.1 and the proprietary version of the open source ESB crashed with a concurrency level over 80 (Also see [5])
  2. Mule CE 2.0.1 consistently failed on ~1% of requests for each and every scenario tested, even for the low concurrency tests.
  3. Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction to the real service
  4. The WSO2 ESB performed better than the proprietary ESB for 1/2K messages with Xalan translet compilation enabled, but was beaten by it for the 1K and 5K messages.
  5. WSO2 Enterprise Service Bus (ESB) v1.7 with Xalan translet compilation enabled, beat Mule CE and the proprietary version of the open source ESB for all scenarios.
  6. Apache ServiceMix beats WSO2 ESB for XSLT transformation of 5K messages
  7. Considering the average TPS, the WSO2 ESB with the Xalan translet compilation enabled, performed better than all ESB’s with a close tie from the proprietary ESB, but with a clear lead against the open source options.

Conclusions

  • WSO2 Enterprise Service Bus (ESB) displays excellent stability and performance on all scenarios
  • WSO2 ESB v1.7 leads in performance over all other open source alternatives, as well as the proprietary version of the open source ESB benchmarked
  • The performance of the proprietary ESB has improved drastically over the past year, as compared to numbers it gained during the rounds 1[8] and 2[9]

Notes

  • All performance numbers were captured while running a single instance of each ESB, on a single JVM.
  • All software (JavaBench, Tomcat, WSO2 ESB, Mule CE, the proprietary version of the open source ESB and ServiceMix) were run on the Sun JDK 1.5.0_15 on Red Hat Enterprise Linux Server release 5 (Tikanga)
  • The proprietary ESB was run on its own JVM on the same Red Hat Enterprise Linux Server release 5 (Tikanga) server
  • Hardware configuration
  • Client / Load generator — Intel(R) Xeon(TM) 3.20GHz 2MB Cache Dual Core — 2 CPU system / 2GB RAM / 1G NW
  • Enterprise Service Bus — Intel(R) Xeon(TM) 3.20GHz 2MB Cache Dual Core — 2 CPU system / 2GB RAM / 1G NW
  • Backend service simulator — Intel(R) Xeon(R) 2.00GHz Dual core — 2 CPU system / 2 GB RAM / 1G NW
  • All three systems were tuned at the Operating System level as follows:
/etc/sysctl.conf 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_fin_timeout = 30 
fs.file-max = 2097152 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_tw_reuse = 1 

/etc/security/limits.conf 
* soft nofile 4096 
* hard nofile 65535

 

Resources

[1] JavaBench — a Java clone of the popular Apache Bench load generator, implemented by Apache HttpComponents project

Source available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/Client, with build instructions and a Maven project file

A complimentary binary build is available here: java-bench-0.1.zip

[2] MockServices — a high performance Echo Servlet that acts as a service simulator.

Source available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/MockService/mock-services with build instructions and a Maven project file

A complimentary binary build is available here: MockServices.war

[3] Resources used for the load testing, including XSLT transformations, WSDL’s, and shell scripts etc. available at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Round_3/Resources

[4] Sample client requests of 0.5K, 1K and 5K can be found at https://wso2.org/repos/wso2/trunk/commons/performance/esb/Tools/Client/requests

[5] Mule CE 2.0.1 encountered the following exception when the number of concurrent users exceeded the maximum thread count configured

ERROR 2008-06-20 16:42:05,283 [HttpConnector.receiver.1] org.mule.providers.http.HttpConnector: Work caused exception on 'workCompleted'. 
   Work being executed was: HttpMessageReceiver{this=160c4b0, receiverKey=http://testb.wso2.com:8090/services/CBRProxy, endpoint=http://testb.wso2.com:8090/services/CBRProxy}


ERROR 2008-06-20 16:42:05,285 [HttpConnector.receiver.1] org.mule.impl.DefaultExceptionStrategy: Caught exception in Exception Strategy: null
edu.emory.mathcs.backport.java.util.concurrent.RejectedExecutionException
        at org.mule.util.concurrent.WaitPolicy.rejectedExecution(WaitPolicy.java:56)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:765)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1288)
        at org.mule.impl.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:39)
        at org.mule.impl.work.MuleWorkManager.executeWork(MuleWorkManager.java:276)
        at org.mule.impl.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:243)
        at org.mule.providers.tcp.TcpMessageReceiver.run(TcpMessageReceiver.java:177)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
        at java.lang.Thread.run(Thread.java:595)

[6] WSO2 Enterprise Service Bus (ESB) — http://wso2.org/esb

[7] Apache Synapse Enterprise Service Bus (ESB) — http://synapse.apache.org

[8] Performance Testing Round 1 = http://wso2.org/library/1721

[9] Performance Testing Round 2 = http://wso2.org/library/2259

 

 

Author

Asankha Perera, WSO2 Inc.

Project Management Commitee member and comitter of Apache Synapse Enterprise Service Bus (ESB), Apache Web Services and Apache HttpComponents projects

Architect and Product Manager of the WSO2 Enterprise Service Bus (ESB)

asankha at wso2 dot com

http://esbmagic.blogspot.com

52
Share: