Начало работы с Spring Batch 2.0

Возможно, вам будет полезно взглянуть на файлы данных игрока и игры, просто чтобы посмотреть, что случилось. Файлы данных находятся внутри src / main / resources / data / footballjob / input .

Давайте запустим это. Мы можем запустить работу, выполнив тест JUnit

org.springframework.batch.sample.FootballJobFunctionalTests

в папке src / test / java . Идите и попробуйте это сейчас. Тесты JUnit должны пройти.

По умолчанию тест использует базу данных HSQLDB в памяти. Хотя это и делает быстрый тест, он не так полезен, чтобы попытаться увидеть, что на самом деле делает работа. Поэтому вместо этого давайте запустим пакетное задание для постоянной базы данных. Я использую MySQL, хотя вы можете использовать все, что вам нравится. Вот что нам нужно сделать.

Шаг 1. Создать базу данных; например, CREATE DATABASE spring_batch_samples .

Шаг 2. Внутри src / main / resources вы увидите различные файлы batch-xxx.properties . Откройте ту, которая соответствует выбранной вами СУБД, и при необходимости измените свойства. Убедитесь, что значение batch.jdbc.url соответствует имени базы данных, которое вы выбрали на шаге 1.

Шаг 3. При запуске тестов нам нужно переопределить СУБД по умолчанию, указанную в src / main / resources / data-source-context.xml . Если вы ленивы, вы можете просто найти bean-компонент среды в этом файле и изменить значение свойства defaultValue с hsql на mysql или sqlserver или любым другим . (Параметры соответствуют файлам batch-xxx.properties, которые мы упоминали выше.) Однако правильный способ сделать это — установить

org.springframework.batch.support.SystemPropertyInitializer.ENVIRONMENT

системное свойство. Есть разные способы сделать это. Если вы используете Eclipse, перейдите в диалоговое окно «Выполнить»> «Выполнить настройки» и в конфигурации выполнения для FootballJobFunctionalTests перейдите на вкладку «Аргументы». Затем добавьте следующее к аргументам VM:

-Dorg.springframework.batch.support.SystemPropertyInitializer.ENVIRONMENT = MySQL

(Я разбил это на две строки для целей форматирования, но это должна быть одна строка.)

Шаг 4. Просто для того, чтобы сделать эту пакетную работу более интересной (т.е. сделать ее намного больше), откройте файл контекста приложения src / main / resources / jobs / footballJob.xml и найдите bean-компонент footballProperties . Измените его свойства с

<beans:value>
        games.file.name=games-small.csv
        player.file.name=player-small1.csv
        job.commit.interval=2
    </beans:value>

 to

    <beans:value>
        games.file.name=games.csv
        player.file.name=player.csv
        job.commit.interval=100
    </beans:value>

Step 5. Run FootballJobFunctionalTests again. It will run for a while depending on how fast your computer is. Mine is pretty slow but the job still finishes in a couple of minutes.

Assuming everything runs as it should, step 5 creates several tables in your database. Here’s what it looks like in MySQL:

mysql> show tables;

+--------------------------------+

| Tables_in_spring_batch_samples |

+--------------------------------+

| batch_job_execution |

| batch_job_execution_context |

| batch_job_execution_seq |

| batch_job_instance |

| batch_job_params |

| batch_job_seq |

| batch_staging |

| batch_staging_seq |

| batch_step_execution |

| batch_step_execution_context |

| batch_step_execution_seq |

| customer |

| customer_seq |

| error_log |

| games |

| player_summary |

| players |

| trade |

| trade_seq |

+--------------------------------+

19 rows in set (0.00 sec)

Spring Batch uses the batch_xxx tables to manage job execution. These are part of Spring Batch itself, not part of the samples, and so the SQL scripts that generate them are inside the org.springframework.batch.core-2.0.0.RC2.jar. On the other hand, the other tables are sample business tables. These are defined in the src/main/resources/business-schema-xxx.sql scripts. As you can see, there are some extra tables here—these support some of the other sample apps—but the only business tables we care about are players, games and player_summary.

There’s a lot of data in the tables. Here’s what it looks like:

mysql> select count(*) from players;

+----------+

| count(*) |

+----------+

| 4320 |

+----------+

1 row in set (0.00 sec)


mysql> select count(*) from games;

+----------+

| count(*) |

+----------+

| 56377 |

+----------+

1 row in set (0.06 sec)


mysql> select count(*) from player_summary;

+----------+

| count(*) |

+----------+

| 5931 |

+----------+

1 row in set (0.01 sec)

If you want to check out some of the data itself without having to pull down the entire dataset, you can use the following queries:

select * from players limit 10;

select * from games limit 10;

select * from player_summary limit 10;

Just for kicks, you might find it entertaining to investigate the batch_xxx tables too. For instance:

mysql> select * from batch_job_execution;

+------------------+---------+-----------------+---------------------+

| JOB_EXECUTION_ID | VERSION | JOB_INSTANCE_ID | CREATE_TIME |

+------------------+---------+-----------------+---------------------+

| 1 | 2 | 1 | 2009-03-22 20:31:40 |

+------------------+---------+-----------------+---------------------+


+---------------------+---------------------+-----------+-----------+

| START_TIME | END_TIME | STATUS | EXIT_CODE |

+---------------------+---------------------+-----------+-----------+

| 2009-03-22 20:31:40 | 2009-03-22 20:33:44 | COMPLETED | COMPLETED |

+---------------------+---------------------+-----------+-----------+


+--------------+---------------------+

| EXIT_MESSAGE | LAST_UPDATED |

+--------------+---------------------+

| | 2009-03-22 20:33:44 |

+--------------+---------------------+

1 row in set (0.00 sec)

This will give you some visibility into how Spring Batch keeps track of job executions, but we’re not going to worry about that here. (Consult the Spring Batch 2.0 reference manual for more information on that.)

It’s time to take a closer look at what’s going on behind the scenes.

Understanding The Football Sample Application 

Let’s start from FootballJobFunctionalTests and work backwards.

Normally we wouldn’t launch batch jobs from JUnit tests, but that’s what we’re doing here so let’s look at that. The sample app uses the Spring TestContext framework, and without going into the gory details (they’re not directly relevant to Spring Batch), it turns out that the TestContext framework provides a default application context file for FootballJobFunctionalTests; namely

org/springframework/batch/sample/FootballJobFunctionalTests-context.xml

inside the src/test/resources folder. Listing 1 shows what it contains.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

    <import resource="classpath:/simple-job-launcher-context.xml" />
    <import resource="classpath:/jobs/footballJob.xml" />
</beans>

In listing 1 we can see that the app context provides a couple of things: first, it provides via the simple-job-launcher-context.xml import a JobLauncher bean so we can run jobs; second, it provides via the jobs/footballJob.xml import an actual job to run. Both of these live in the src/main/resources folder. Once you have a JobLauncher, a Job and a JobParameters (we’re using an empty JobParameters bean for this sample app), all we have to do is this:

jobLauncher.run(job, jobParameters);

That’s exactly what the FootballJobFunctionalTests class does, though you have to navigate up its inheritance hierarchy to AbstractBatchLauncherTests.testLaunchJob() to see it.

Anyway, let’s look first at the JobLauncher.

Defining A JobLauncher

As noted above, the sample app defines the JobLauncher bean in simple-job-launcher-context.xml. We can see some of the bean dependencies in figure 3, courtesy of Spring IDE:

Listing 2 shows the corresponding application context file.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
        http://www.springframework.org/schema/aop
        http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
        http://www.springframework.org/schema/tx
        http://www.springframework.org/schema/tx/spring-tx-2.5.xsd">

    ... some imports ...

    <bean id="jobLauncher"
        class="org.springframework.batch.core.launch.support.
            SimpleJobLauncher"
        p:jobRepository-ref="jobRepository" />

    <bean id="jobRepository"
        class="org.springframework.batch.core.repository.support.
            JobRepositoryFactoryBean"
        p:dataSource-ref="dataSource"
        p:transactionManager-ref="transactionManager" />

    ... other bean definitions ...

</beans>

I’ve obviously suppressed some of the beans from the application context. The two beans we need to know about here are the JobLauncher itself and its JobRepositoryFactoryBean dependency, which is a factory for SimpleJobRepository instances. I already mentioned that the JobLauncher allows us to run jobs. The point of the JobRepository is to store and retrieve job metadata of the sort stored in the batch_xxx tables we saw earlier. Again we’re not going to cover that here, but the basic idea is that the JobRepository contains information on which jobs we ran when, which steps succeeded and failed, and that sort of thing. That kind of metadata allows Spring Batch to support, for example, job retries.

We’ll now consider the job definition itself.

Defining A Job

The footballJob.xml application context defines the football job. It’s a long file, so let’s digest it in pieces. First, here are the namespace declarations:

<beans:beans xmlns="http://www.springframework.org/schema/batch"
    xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
        http://www.springframework.org/schema/batch
        http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
        http://www.springframework.org/schema/aop
        http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
        http://www.springframework.org/schema/tx
        http://www.springframework.org/schema/tx/spring-tx-2.0.xsd">

As we discussed above, we’re using batch namespace elements like job, step and tasklet. You can see that it certainly cleans up the configuration as compared to defining everything using bean elements.

Our football job has three steps. Individual steps can use the next attribute to point to the next step in the flow. Each step has some internal tasklet details (more on these in a minute), and we can define steps either internally to the job (see, e.g., playerLoad and gameLoad) or else they can be externalized (see, e.g., playerSummarization). I don’t think there’s a good reason to externalize the playerSummarization step here other than simply to show that it can be done and to show how to do it.

Earlier in the article we noted that each step reads items from some source, optionally processes them in some way and finally writes them out somewhere. Our three steps fit that general pattern. None of them includes an explicit processing step, but they all read items and write them back out.

You may recall that we said that the first two steps read player and game data from flat files. They do this using a class from the Spring Batch infrastructure module called FlatFileItemReader. Let’s see how that works.

Loading Items From a Flat File

We’ll focus on the playerload step, since it’s essentially the same as the gameLoad step. Here’s the definition for the playerFileItemReader bean we reference from the playerLoad step:

<beans:bean id="playerFileItemReader"
    class="org.springframework.batch.item.file.FlatFileItemReader">
    <beans:property name="resource"
        value="classpath:data/footballjob/input/${player.file.name}" />
    <beans:property name="lineMapper">
        <beans:bean class="org.springframework.batch.item.file.mapping.
            DefaultLineMapper">
            <beans:property name="lineTokenizer">
                <beans:bean class="org.springframework.batch.item.file.
                    transform.DelimitedLineTokenizer">
                    <beans:property name="names" value=
                      "ID,lastName,firstName,position,birthYear,debutYear" />
                </beans:bean>
            </beans:property>
            <beans:property name="fieldSetMapper">
                <beans:bean class="org.springframework.batch.sample.domain.
                    football.internal.PlayerFieldSetMapper" />
            </beans:property>
        </beans:bean>
    </beans:property>
</beans:bean>

The player.file.name property resolves to player.csv since that’s what we set it to just before we ran the job. Anyway, there are a couple of dependencies the reader needs. First it needs a Resource to represent the file we want to read. (See the Spring 2.5.6 Reference Documentation, chapter 4, for more information about Resources). Second it needs a LineMapper to help tokenize and parse the file. We won’t dig into all the details of the LineMapper dependencies—you can check out the Javadocs for the various infrastructure classes involved—but it’s worth taking a peek at the PlayerFieldSetMapper class, since that’s a custom class. Listing 3 shows what it does.

package org.springframework.batch.sample.domain.football.internal;

import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.batch.sample.domain.football.Player;

public class PlayerFieldSetMapper implements FieldSetMapper<Player> {

    public Player mapFieldSet(FieldSet fs) {
        if (fs == null) { return null; }
        
        Player player = new Player();
        player.setId(fs.readString("ID"));
        player.setLastName(fs.readString("lastName"));
        player.setFirstName(fs.readString("firstName"));
        player.setPosition(fs.readString("position"));
        player.setDebutYear(fs.readInt("debutYear"));
        player.setBirthYear(fs.readInt("birthYear"));
        
        return player;
    }
}

PlayerFieldSetMapper carries a FieldSet (essentially a set of tokens) to a Player domain object. If you don’t want to do this kind of mapping manually, you might check out the Javadocs for BeanWrapperFieldSetMapper, which uses reflection to accomplish the mapping automatically.

We’ll now turn to the topic of storing records to a database table.

Storing Items Into a Database

Here’s the playerWriter bean we referenced from the playerLoad step:

<beans:bean id="playerWriter" class="org.springframework.batch.sample.domain.
    football.internal.PlayerItemWriter">
    <beans:property name="playerDao">
        <beans:bean class="org.springframework.batch.sample.domain.football.
            internal.JdbcPlayerDao">
            <beans:property name="dataSource" ref="dataSource" />
        </beans:bean>
    </beans:property>
</beans:bean>

The PlayerItemWriter is a custom class, though it turns out that it’s pretty trivial as listing 4 shows.

package org.springframework.batch.sample.domain.football.internal;

import java.util.List;

import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.sample.domain.football.Player;
import org.springframework.batch.sample.domain.football.PlayerDao;

public class PlayerItemWriter implements ItemWriter<Player> {
    private PlayerDao playerDao;

    public void setPlayerDao(PlayerDao playerDao) {
        this.playerDao = playerDao;
    }

    public void write(List<? extends Player> players) throws Exception {
        for (Player player : players) {
            playerDao.savePlayer(player);
        }
    }

There isn’t anything special happening here. The step will use the FlatFileItemReader to pull items from a flat file and will pass them in chunks to the PlayerItemWriter, which dutifully saves them to the database.

The examples we’ve seen so far are among the simplest possible, but the general idea behind ItemReaders and ItemWriters should be clear now: readers pull items from an arbitrary data source and map them to domain objects, whereas writers map domain objects to items in a data sink. But just for good measure, let’s take a look at one more ItemReader.

JdbcCursorItemReader

The JdbcCurstorItemReader allows us to pull items from a database. In the case of the football job, we’re using the JdbcCursorItemReader to pull player and game data from the database so that we can synthesize them into PlayerSummary domain objects, which we’ll subsequently write. At any rate here’s the definition for our playerSummarizationSource, which is part of the job’s third step:

<beans:bean id="playerSummarizationSource"
    class="org.springframework.batch.item.database.JdbcCursorItemReader">
    <beans:property name="dataSource" ref="dataSource" />
    <beans:property name="rowMapper">
        <beans:bean class="org.springframework.batch.sample.domain.football.
            internal.PlayerSummaryMapper" />
    </beans:property>
    <beans:property name="sql">
        <beans:value>
            SELECT games.player_id, games.year_no, SUM(COMPLETES),
            SUM(ATTEMPTS), SUM(PASSING_YARDS), SUM(PASSING_TD),
            SUM(INTERCEPTIONS), SUM(RUSHES), SUM(RUSH_YARDS),
            SUM(RECEPTIONS), SUM(RECEPTIONS_YARDS), SUM(TOTAL_TD)
            from games, players where players.player_id =
            games.player_id group by games.player_id, games.year_no
        </beans:value>
    </beans:property>
</beans:bean>

The sql property as you might guess provides the SQL used to pull data from the data source. Here we’re using both the players and games tables to compute player stats. The result of that query is a JDBC ResultSet, which this particular ItemReader implementation passes to a RowMapper implementation. The PlayerSummaryMapper is a custom implementation, and it essentially takes a row in a ResultSet and carries it to a PlayerSummary domain object.

Summary

With that we conclude our introductory tour of Spring Batch 2.0. We’ve only scratched the surface, showing how to create simple jobs with simple sequential steps, and how to run them.

Once you feel comfortable with simple jobs, it makes sense to spend a little time with the introductory chapters of the Spring Batch Reference Documentation to learn more about the execution environment, including the difference between Jobs, JobInstances and JobExcecutions. You can use Spring Batch in conjunction with a scheduler (such as Quartz) to run batch jobs on a recurring basis in an automated fashion.

More advanced topics include non-sequential step flow, such as conditional flows and parallel flows, and support for partitioning individual steps across multiple threads or even servers.

Enjoy!

Willie is an IT director with 12 years of Java development experience. He and his brother John are coauthors of the upcoming book Spring in Practice by Manning Publications (www.manning.com/wheeler/). Willie also publishes technical articles (including many on Spring) to wheelersoftware.com/articles/.