Непрерывная доставка с использованием Maven

В настоящее время я работаю над системой непрерывной доставки, в которой я работаю, поэтому я решил написать кое-что о том, что я делаю. Вкратце, система непрерывной доставки выглядит примерно так:

Я начал с карт-бланша в отношении того, какие инструменты использовать, но вот список того, что уже использовалось, в той или иной форме, когда я начал свое приключение:

• Муравей (основной инструмент сборки)
• Maven (используется для управления зависимостями)
• Круиз-контроль
• CruiseControl.Net
• Идти
• монит
• JUnit
• JS-тест-драйвер
• Селен
• Artifactory
• насильственный

Решение о том, какой из этих инструментов использовать для моей системы, зависело от ряда факторов. Сначала я объясню, почему я решил использовать Maven в качестве инструмента для сборки (шок !!).

Я большой поклонник Ant, я обычно выбирал его (или, возможно, Gradle сейчас) вместо Maven в любой день недели, но уже существовала система сборки Ant, которая стала немного монолитной (это мое вежливый способ сказать, что это был огромный беспорядок), поэтому я не хотел идти туда! Кроме того, первым проектом, который войдет в новую систему непрерывной доставки, был простой Java-проект — слишком простой, чтобы оправдать переписывание всей системы ant с нуля и ее усовершенствование, поэтому я пошел на Maven. Кроме того, поскольку проект был (с точки зрения сборки) довольно простым, я подумал, что Maven справится с этим без особых хлопот. Я уже пользовался Maven, так что у меня были свои заезды, и я знаю, как тяжело может быть, если вы захотите сделать что-нибудь за пределами «The Maven Way». Но, как я уже сказал,проект, над которым я работал, выглядел довольно простым, поэтому Мейвен кивнул.

GO был последним и лучшим в использовании CI-сервером, а системы CruiseControl уже были немногочисленными, поэтому я выбрал GO (также я никогда не использовал его раньше, поэтому я подумал, что это будет круто, и это от Thoughtworks Studios , так что я подумал, что это может быть довольно хорошо). Мне особенно понравилась функция конвейера, и то, как он управляет каждым из своих агентов. Мой коллега, Энди Берри, уже проделал немалую работу над системой GO CI, поэтому уже было с чего начать. Я бы пошел на Дженкинса, если бы компания еще не сделала значительные инвестиции в GO до моего приезда.

Я решил использовать Artifactory в качестве менеджера репозитория артефактов просто потому, что там уже был установлен экземпляр, и он вроде как уже настроен. Существующая система сборки на самом деле не использовала ее, так как большинство артефактов / зависимостей обслуживалось из общих сетевых ресурсов. Я бы рассмотрел Nexus, если бы Artifactory еще не был установлен.

Я настроил Sonar для работы в качестве инструмента для анализа сборки / составления отчетов, потому что мы начинали с проекта Java. Мне действительно нравится то, что делает Sonar, я думаю, что информация, которую он представляет, может использоваться очень эффективно. Больше всего мне просто нравится, как она доставляет информацию. Плагин для сайта Maven может генерировать почти всю информацию, которую делает Sonar, но я думаю, что способ, которым Sonar представляет информацию, намного лучше — об этом позже.

Perforce была действующей системой контроля над источниками, и поэтому было несложно заниматься этим. На самом деле, изменение системы SC никогда не было под вопросом. Тем не менее, я бы выбрал Subversion, если бы это был вариант, просто потому, что он совершенно свободен!

Это было об инструментах, которые я хотел использовать. Оставшаяся часть проектной команды должна была определить, какие инструменты использовать для тестирования и разработки. Все, что мне было нужно для системы, которую я настраивал, — это различие между юнит-тестами, приемочными тестами и интеграционными тестами. В конце концов, команда позаботилась о тестировании с Junit, Mockito и парой собственных приложений.

Maven Build и радости релизного плагина!

Идея моей системы непрерывной доставки заключалась в следующем:

• Каждая регистрация запускает множество юнит-тестов.
• Если они проходят, он проходит нагрузку приемочных испытаний
• Если они пройдут, мы запустим больше тестов — тесты интеграции, сценариев и производительности
• Если все они пройдут, мы запустим кучу статического анализа, создадим красивые отчеты и в конечном итоге развернем кандидата в репозиторий «Release Candidate», где QA и другие единомышленники смогут посмотреть на него, подтолкнуть его и в конечном итоге дать ему печать утверждение.

Это базовая схема конвейера сборки:

Maven не совсем подходит для встраивания в процесс конвейера. Для начала мы запускаем несколько этапов тестирования, и Maven следует процессу «жизненного цикла», то есть каждый раз, когда вы вызываете конкретную фазу конвейера, он снова запускает все предыдущие этапы. Наш конвейер должен запустить плагин maven Surefire дважды, потому что это плагин, который мы используем для выполнения наших различных тестов. При первом запуске мы хотим выполнить все модульные тесты. Во второй раз, когда мы запускаем его, мы хотим выполнить приемочные тесты — но мы не хотим, чтобы он снова запускал модульные тесты, очевидно.

You probably need some familiarity with the maven build lifecycle at this point, because we’re going to be binding the Surefire plugin to two different phases of the maven lifecycle so that we can run it twice and have it run different tests each time. Here is the maven lifecycle, (for a more detailed description check out the Maven’s own lifecycle page)

Clean Lifecycle

• pre-clean
  
• clean
• post-clean

Default Lifecycle

• validate
• initialize
  
• generate-sources
  
• process-sources
  
• generate-resources
• process-resources
• compile
• process-classes
  
• generate-test-sources
  
• process-test-sources
  
• generate-test-resources
  
• process-test-resources
  
• test-compile
  
• process-test-classes
  
• test
  
• prepare-package
• package
  
• pre-integration-test
  
• integration-test
  
• post-integration-test
• verify
• install
• deploy

Site Lifecycle

• pre-site
  
• site
• post-site
  
• site-deploy

So, we want to bind our Surefire plugin to both the test phase to execute the UTs, and the integration-test phase to run the ATs, like this:

<plugin>
<!-- Separates the unit tests from the integration tests. -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
  <configuration>
  -Xms256m -Xmx512m
  <skip>true</skip>
  </configuration>
  <executions>
    <execution>
      <id>unit-tests</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
  <configuration>
    <testClassesDirectory>
      target/test-classes
    </testClassesDirectory>
    <skip>false</skip>
    <includes>
      <include>**/*Test.java</include>
    </includes>
    <excludes>
      <exclude>**/acceptance/*.java</exclude>
      <exclude>**/benchmark/*.java</exclude>
      <include>**/requestResponses/*Test.java</exclude>
    </excludes>
  </configuration>
</execution>
<execution>
  <id>acceptance-tests</id>
  <phase>integration-test</phase>
  <goals>
    <goal>test</goal>
  </goals>
  <configuration>
    <testClassesDirectory>
      target/test-classes
    </testClassesDirectory>
    <skip>false</skip>
    <includes>
      <include>**/acceptance/*.java</include>
      <include>**/benchmark/*.java</include>
      <include>**/requestResponses/*Test.java</exclude>
    </includes>
  </configuration>
</execution>
</executions>
</plugin>

Now in the first stage of our pipeline, which polls Perforce for changes, triggers a build and runs the unit tests, we simply call:

mvn clean test

This will run the surefire test phase of the maven lifecycle. As you can see from the Surefire plugin configuration above, during the “test” phase execution of Surefire (i.e. this time we run it) it’ll run all of the tests except for the acceptance tests – these are explicitly excluded from the execution in the “excludes” section. The other thing we want to do in this phase is quickly check the unit test coverage for our project, and maybe make the build fail if the test coverage is below a certain level. To do this we use the cobertura plugin, and configure it as follows:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>cobertura-maven-plugin</artifactId>
  <version>2.4</version>
  <configuration>
    <instrumentation>
      <excludes><!-- this is why this isn't in the parent -->
        <exclude>**/acceptance/*.class</exclude>
        <exclude>**/benchmark/*.class</exclude>
        <exclude>**/requestResponses/*.class</exclude>
      </excludes>
    </instrumentation>
    <check>
      <haltOnFailure>true</haltOnFailure>
      <branchRate>80</branchRate>
      <lineRate>80</lineRate>
      <packageLineRate>80</packageLineRate>
      <packageBranchRate>80</packageBranchRate>
      <totalBranchRate>80</totalBranchRate>
      <totalLineRate>80</totalLineRate>
    </check>
    <formats>
      <format>html</format>
      <format>xml</format>
    </formats>
  </configuration>
  <executions>
    <execution>
      <phase>test</phase>
      <goals>
        <goal>clean</goal>
        <goal>check</goal>
      </goals>
    </execution>
  </executions>
</plugin>

To get the cobertura plugin to execute, we need to call “mvn cobertura:cobertura”, or run the maven “verify” phase by calling “mvn verify”, because the cobertura plugin by default binds to the verify lifecycle phase. But if we delve a little deeper into what this actually does, we see that it actually runs the whole test phase all over again, and of course the integration-test phase too, because they precede the verify phase, and cobertura:cobertura actually invokes execution of the test phase before executing itself. So what I’ve done is to change the lifecycle phase that cobertura binds to, as you can see above. I’ve made it bind to the test phase only, so that it only executes when the unit tests run. A consequence of this is that we can now change the maven command we run, to something like this:

mvn clean cobertura:cobertura

This will run the Unit Tests implicitly and also check the coverage!

In the second stage of the pipeline, which runs the acceptance tests, we can call:

mvn clean integration-test

This will again run the Surefire plugin, but this time it will run through the test phase (thus executing the unit tests again) and then execute the integration-test phase, which actually runs our acceptance tests.

You’ll notice that we’ve run the unit tests twice now, and this is a problem. Or is it? Well actually no it isn’t, not for me anyway. One of the reasons why the pipeline is broken down into sections is to allow us to separate different tasks according to their purpose. My Unit Tests are meant to run very quickly (less than 3 minutes ideally, they actually take 15 seconds on this particular project) so that if they fail, I know about it asap, and I don’t have to wait around for a lifetime before I can either continue checking in, or start fixing the failed tests. So my unit test pipeline phase needs to be quick, but what difference does an extra few seconds mean for my Acceptance Tests? Not too much to be honest, so I’m actually not too fussed about the unit tests running for a second time.  If it was a problem, I would of course have to somehow skip the unit tests, but only in the test phase on the second run. This is doable, but not very easy. The best way I’ve thought of is to exclude the tests using SkipTests, which actually just skips the execution of the surefire plugin, and then run your acceptance tests using a different plugin (the Antrun plugin for instance).

The next thing we want to do is create a built artifact (a jar or zip for example) and upload it to our artifact repository. We use 4 artifact repositories in our continuous delivery system, these are:

1. A cached copy of the maven central repo
2. A C.I. repository where all builds go
3. A Release Candidate (RC) repository where all builds under QA go
4. A Release repository where all builds which have passed QA go

Once our build has passed all the automated test phases it gets deployed to the C.I. repository. This is done by configuring the C.I. repository in the maven pom file as follows:

<distributionManagement>
<repository>
<id>CI-repo</id>
<url>http://artifactory.mycompany.com/ci-repo</url>
</repository>
</distributionManagement>

and calling:

mvn clean deploy

Now, since Maven follows the lifecycle pattern, it’ll rerun the tests again, and we don’t want to do all that, we just want to deploy the artifacts. In fact, there’s no reason why we shouldn’t just deploy the artifact straight after the Acceptance Test stage is completed, so that’s what exactly what we’ll do. This means we need to go back and change our maven command for our Acceptance Test stage as follows:

mvn clean deploy

This does the same as it did before, because the integration-test phase is implicit and is executed on the way to reaching the “deploy” phase as part of the maven lifecycle, but of course it does more than it did before, it actually deploys the artifact to the C.I. repository.

One thing that is worth noting here is that I’m not using the maven release plugin, and that’s because it’s not very well suited to continuous delivery, as I’ve noted here. The main problem is that the release plugin will increment the build number in the pom and check it in, which will in turn kick off another build, and if every build is doing this, then you’ll have an infinitely building loop. Maven declares builds as either a “release build” which uses the release plugin, or a SNAPSHOT build, which is basically anything else. But I want to create releases out of SNAPSHOT builds, but I don’t want them to be called SNAPSHOT builds, because they’re releases! So what I need to do is simply remove the word SNAPSHOT from my pom. Get rid of it entirely. This will now build a normal “snapshot” build, but not add the SNAPSHOT label, and since we’re not running the release plugin, that’s fine (WARNING: if you try removing the word snapshot from your pom and then try to run a release build using the release plugin, it’ll fail).

Ok, let’s briefly catch up with what our system can now do:

• We’ve got a build pipeline with 2 stages
• It’s executed every time code is checked-in
• Unit tests are executed in the first stage
• Code coverage is checked, also in the first stage
• The second stage runs the acceptance tests
• The jar/zip is built and deployed to our ci repo, this also in the second stage of our pipeline

So we have a jar, and it’s in our “ci” repo, and we have a code coverage report. But where’s the rest of our static analysis? The build should report a lot more than just the code coverage. What about coding styles & standards, rules violations, potential defect hot spots, copy and pasted code etc and so forth??? Thankfully, there’s a great tool which collects all this information for us, and it’s called Sonar.

I won’t go into detail about how to setup and install Sonar, because I’ve already detailed it here.

Installing Sonar is very simple, and to get your builds to produce Sonar reports is as simple as adding a small amount of configuration to your pom, and adding the Sonar plugin to you plugin section. To produce the Sonar reports for your project, you can simply run:

mvn sonar:sonar

So that’s exactly what we’ll do in the next section of our build pipeline.

So we now have 3 pipeline sections and were producing Sonar reports with every build. The Sonar reports look something like this:

As you can see, Sonar produces a wealth of useful information which we can pour over and discuss in our daily stand-ups. As a rule we try to fix any “critical” rule violations, and keep the unit test coverage percentage up in the 90s (where appropriate). Some people might argue that unit test coverage isn’t a valuable metric, but bear in mind that Sonar allows you to exclude certain files and directories from your analysis, so that you’re only measuring the unit test coverage of the code you want to have covered by unit tests. For me, this makes it a valuable metric.

Moving on from Sonar now, we get to the next stage of my pipeline, and here I’m going to run some Integration Tests (finally!!). The ITs have a much wider scope than the Unit Test, and they also have greater requirements, in that we need an Integration Test Environment to run them in. I’m going to use Ant to control this phase of the pipeline, because it gives me more control than Maven does, and I need to do a couple of funky things, namely:

• Provision an environment
• Deploy all the components I need to test with
• Get my newly built artifact from the ci repository in Artifactory
• Deploy it to my IT environment
• Kick of the tests

The Ant script is fairly straightforward, but I’ll just mention that getting our artifact from Artifactory is as simple as using Ant’s own “get” task (you don’t need to use Ivy juts to do this):

<get src=”${artifactory.url}/${repo.name}/${namespace}/${jarname}-${version}” dest=”${temp.dir}/${jarname}-${version}” />

The Integration Test stage takes a little longer than the previous stages, and so to speed things up we can run this stage in parallel with the previous stage. Go allows us to do this by setting up 2 jobs in one pipeline stage. Our Sonar stage now changes to “Reports and ITs”, and includes 2 jobs:

<jobs>
          <job name="sonar">
            <tasks>
              <exec command="mvn" args="sonar:sonar" workingdir="JavaDevelopment" />
            </tasks>
            <resources>
              <resource>windows</resource>
            </resources>
          </job>
 <job name="ITs">
            <tasks>
              <ant buildfile="run_ITs.xml" target="build" workingdir="JavaDevelopment" />
            </tasks>
            <resources>
              <resource>windows</resource>
            </resources>
          </job>
</jobs>

Once this phase completes successfully, we know we’ve got a half decent looking build! At this point I’m going to throw a bit of a spanner into the works. The QA team want to perform some manual exploratory tests on the build. Good idea! But how does that fit in with our Continuous Delivery model? Well, what I did was to create a separate “Release Candidate” (RC) repository, also known as a QA repo. Builds that pass the IT stage get promoted to the RC repo, and from there the QA team can take them and do their exploratory testing.

Does this stop us from practicing “Continuous Delivery”? Well, not really. In my opinion, Continuous Delivery is more about making sure that every build creates a potentially releasable artifact, rather that making every build actually deploy an artifact to production – that’s Continuous Deployment.

Our final stage in the deployment pipeline is to deploy our build to a performance test environment, and execute some load tests. Once this stage completes we deploy our build to the Release Repository, as it’s all signed off and ready to handover to customers. At this point there’s a manual decision gate, which in reality is a button in my CI system. At this point, only the product owner or some such responsible person, can decide whether or not to actually release this build into the wild. They may decide not to, simply because they don’t feel that the changes included in this build are particularly worth deploying. On the other hand, they may decide to release it, and to do this they simply click the button. What does the button do? Well, it simply copies the build to the “downloads” repository, from where a link is served and sent to customers, informing them that a new release is available – that’s just the way things are done here. In a hosted environment (like a web-based company), this button-press could initiate the deploy script to deploy this build to the production environment.

A Word on Version Numbers

This system is actually dependant on each build producing a unique artifact. If a code change is checked in, the resultant build must be uniquely identifiable, so that when we come to release it, we know we’re releasing theexact same build that has gone through the whole pipeline, not some older previous build. To do this, we need to version each build with a unique number. The CI system is very useful for doing this. In Go, as with most other CI systems, you can retrieve a unique “counter” for your build, which is incremented every time there’s a build. No two builds of the same name can have the same counter. So we could add this unique number to our artifact’s version, something like this (let’s say the counter is 33, meaning this is the 33rd build):

myproject.jar-1.0.33

This is good, but it doesn’t tell us much, apart from that this is the 33rd build of “myproject”. A more meaningful version number is the source control revision number, which relates to the code commit which kicked off the build. This is extremely useful. From this we can cross reference every build to the code in our source control system, and this saves us from having to “tag” the source code with every build. I can access the source control revision number via my CI system, because Go sets it as an environment variable at build time, so I simply pass it to my build script in my CI system’s xml, like this:

cobertura:cobertura -Dp4.revision=${env.GO_PIPELINE_LABEL}
-Dbuild.counter=${env.GO_PIPELINE_COUNTER"

p4.revision and build.counter are used in the maven build script, where I set the version number:

<groupId>com.mycompany</groupId>
<artifactId>myproject</artifactId>
<packaging>jar</packaging>
<version>${main.version}-${build.number}-${build.counter}</version>
<name>myproject</name>

<properties>
<build.number>${p4.revision}</build.number>
<major.version>1</major.version>
<minor.version>0</minor.version>
<patch.version>0</patch.version>
<main.version>${major.version}.${minor.version}.${patch.version}</main.version>
</properties>

If my Perforce check-in number was 1234, then this build, for example, will produce:

myproject.jar-1.0.0-1234-33

And that just about covers it. I hope this is useful to some people, especially those who are using Maven and are struggling with the release plugin!

From http://jamesbetteley.wordpress.com/2012/02/21/continuous-delivery-using-maven/