Помимо JSON: внедрение протокола Spearal Serialization

Spearal — это новый протокол сериализации с открытым исходным кодом, который призван обеспечить быструю замену JSON — в качестве первого шага — HTML и собственных мобильных приложений, подключенных к бэкэнду Java.

Основная цель Spearal состоит в том, чтобы предложить протокол сериализации, который просто работает независимо от сложности структур данных, которыми обмениваются конечные точки: некоторые ограничения JSON, как мы увидим очень скоро, отравляют разработчиков проблемами, которых не было бы при хорошем и универсальный формат сериализации.

Помимо этой основной цели, Spearal также предлагает расширенные функции, которых нет в стандартном JSON, такие как частичная сериализация объектов, встроенная поддержка неинициализированных ассоциаций JPA, согласование расходящихся моделей, фильтрация свойств объектов и т. Д. На ранней стадии разработки Spearal уже можно использовать для приложений HTML, подключенных к бэкэнду Java. Поддержка Android в основном готова, а iOS скоро появится.

Содержание:

1. Немного фона
2. Что не так с JSON?
3. Каким должен быть хороший протокол сериализации?
4. Что такое протокол сериализации Spearal?
5. Хватит разговоров, посмотрим Spearal в действии!
6. Пример приложения AngularJS / JAX-RS / JPA под управлением Spearal
7. Сбой с JSON, на стороне сервера
8. Работа с устаревшими клиентскими приложениями
9. Заключение и будущая работа

Для нетерпеливых, которые хотят немедленно увидеть Spearal в действии, вы можете пропустить первые четыре страницы этой статьи и перейти непосредственно к разговору «Достаточно», давайте посмотрим Spearal в действии! стр.

Немного фона

Сериализация стала основной проблемой с появлением веб-приложений и мобильных приложений, которые получают свои данные через асинхронные запросы. В прежние времена чистого шаблона навигации по страницам HTML никому не приходилось иметь дело с форматами сериализации, DTO, цикличностью графических данных и ссылками, транскрипцией между Java (или любой другой серверной серверной технологией) и JavaScript и т. Д.

Еще в начале 2000-х годов сериализация объектных графов независимо от HTML-страницы стала возможной благодаря введению XMLHttpRequest. Microsoft впервые создала объект Microsoft.XmlHttp с помощью Internet Explorer 5.0 в 1999 году. Вскоре после этого Mozilla и Apple реализовали совместимый объект XMLHttpRequest, который стал рекомендацией W3C в 2006 году.

Как следует из названия «XMLHttpRequest», он изначально был разработан для использования XML в качестве формата сериализации. После того, как JSON был создан Дугласом Крокфордом в 2002 году, он быстро стал успешной альтернативой XML, главным образом потому, что он был намного легче XML (см., Например, эту интересную дискуссию о JSON против XML здесь ).

JSON теперь является стандартом де-факто, используемым для обмена данными между конечными точками приложения, а не только теми, которые основаны на HTML / JavaScript. Некоторые другие форматы, такие как Google Protocol Buffers , предлагают интересные альтернативы, но ни один из них не достиг повсеместного состояния JSON.

Теперь, когда спецификация XMLHttpRequest допускает стандартный способ сериализации и обработки двоичных данных через ArrayBuffer (точнее, начиная с рабочего проекта W3C от 16 августа 2011 года ), пришло время пересмотреть ограничения JSON и что может быть лучшим протоколом сериализации , В этом вся цель этой статьи и цель проекта Spearal .

Далее: что не так с JSON?

Что не так с JSON?

Давайте проясним это: JSON — это отличный, очень читаемый, простой в освоении формат сериализации, и его компактность сложно превзойти, когда вы работаете только с отдельными объектами. Однако у него есть одно непреодолимое (по крайней мере, в стандартном JSON) и хорошо известное ограничение: оно терпит неудачу с любым циклическим графом объектов:

var selfish = {};
selfish.friend = selfish;
JSON.stringify(selfish);

Если вы запустите приведенный выше код в браузере, вы обязательно получите следующую ошибку:

Uncaught TypeError: Converting circular structure to JSON

Хотя этот пример, конечно, немного искусственный, совсем не необычно проектировать модели данных с циклическими ссылками и языками программирования (в частности, JavaScript), базы данных и механизмы JPA успешно работают с циклическими структурами.

Другое связанное ограничение связано с отсутствием ссылок на объекты в стандартном формате JSON. Если вы сериализуете массив из двух объектов, каждый из которых имеет ссылку на другой третий объект, ссылка будет полностью сериализована дважды. Это не только неоптимально с точки зрения использования полосы пропускания (ссылочный индекс вместо полного состояния третьего объекта будет меньше): это также нарушает согласованность вашего графа данных, как показано в следующем фрагменте кода:

var child = { age: 1 };
var parents = [{ child: child }, { child: child }];

parents[0].child.age = 2;
console.log("Should be 2: " + parents[1].child.age);

parents = JSON.parse(JSON.stringify(parents));

parents[0].child.age = 3;
console.log("Should be 3: " + parents[1].child.age);

Выполнение приведенного выше кода даст вам следующий вывод:

 Should be 2: 2 Should be 3: 2 
 
The child object, after being serialized and deserialized, has been duplicated and the two parent objects are not referencing the same instance anymore. There is no standard way to deal with references (hence cyclic graphs) in JSON, such feature isn't part of the specification. See the ECMA specification ("JSON does not support cyclic graphs, at least not directly") and the cycle.js non standard extension.
 
 
Another issue with JSON is that it doesn't retain the class of serialized objects. Let's run the following code:
 
 
function BourneIdentity(name) {
    this.name = name;
}

var obj = new BourneIdentity("Jason");
console.log("Should be a BourneIdentity: " + (obj instanceof BourneIdentity));

obj = JSON.parse(JSON.stringify(obj));

console.log("Should be a BourneIdentity: " + (obj instanceof BourneIdentity));
This is what you get in your console:
  Should be a BourneIdentity: true Should be a BourneIdentity: false 
  
You can run into the same kind of issue with JavaScript Date objects:
 
  
var date = new Date()
console.log("Should be a Date: " + (date instanceof Date));
date = JSON.parse(JSON.stringify(date));
console.log("Should be a Date: " + (date instanceof Date));
Look at your console:
   Should be a Date: true Should be a Date: false 
   
Again, you can find many non standard ways to overcome this kind of issues (see, for example, here). But the more you are relying on non-standard tricks with replacers and revivers, the more your code is likely going to decrease the native performance of the JSON encoding / decoding process.
 
   
So far, I have been only speaking about limitations that are inherent to the standard JSON format. Things are getting much more tricky when it comes to exchanging data between, for example, a Java server and a HTML application.
 
   
Several JSON libraries exist for Java, among them Jackson and Gson, and provide a great help in recreating Java objects from JSON serialized data. JAX-RS has also a built-in support for the JSON format (often based on Jackson). It all works fine when you only deal with rather trivial models. But, as soon as you want to handle more complex structures, it starts failing as we will see later with a basic Java EE / AngularJS application using JSON.
 
   
More generally, from my experience and feedbacks from many other developers, serialization has become very challenging with real-world applications. While almost everybody is using JSON because of its de-facto standard status, serialization is simply poisoning our life with tedious and uninteresting snags. I'm not at all saying that JSON is bad or that you can't realistically work with it. But people spend much too much time and energy to solve issues that simply shouldn't happen with a good serialization format. Witness the fact that there are now about a thousand articles on DZone about Jackson!
 
   
Next: What should be a good serialization protocol?
 
   
   
What should be a good serialization protocol?
 
   
In no particular order, here are the key features I'd like to find in an ideal serialization protocol:
 
   
1. Compact, fast and pretty-printable:
 
   
Performance, in terms of both output size and execution speed, is always a key feature when it comes to send data over a network and especially a mobile network. Serialized data, if they are encoded using a binary format, should also be human readable with the help of some easy to use printer tool.
 
   
2. Handling arbitrary complex graphs of objects:
 
   
The serialization format shouldn't force us to arbitrary lower the complexity of our data structures. Any data model should be serializable, with or without cyclic references, and shared instances should be kept as is, not duplicated.
 
   
3. Able to serialize only few selected properties instead of whole objects:
 
   
If an application only displays a few properties of a complex object, there must be a way to request only these few properties instead of the whole object. Conversely, if only some few properties of a complex object have been updated, there must be a way to send only the few modified properties back to the server.
 
   
4. Dealing correctly with outdated data models, without DTOs:
 
   
It is not always possible to update all client applications when the server-side model evolves with additional properties. Outdated clients, using a previous version of the data model, should continue to work by ignoring unknown properties sent by the server. Moreover, incomplete data received by the server from these outdated clients shouldn't nullify any existing data because new properties are missing.
 
   
5. Retaining the class name of non anonymous structures:
 
   
If an instance of a class X is serialized, it must be deserialized as an X instance, unless there is no class X available in the deserialization context. This requirement implies that the class name of objects is carried with serialized data.
 
   
6. Preventing the serialization of object not intended to be serialized:
 
   
There must be a way to restrict the scope of serializable objects. Trying to serialize, for example, an HTTP session object should fail immediately.
 
   
7. Aware of JPA uninitialized properties:
 
   
The serialization protocol should provide an extension that deals correctly with uninitialized JPA properties, preventing LazyInitializationException's or unwanted collection initialization and serialization. More importantly, uninitialized associations must be serialized in a way that distinguishes them, unambiguously, from null values: failing to do that would eventually result in nullifying lazy associations in your database.
 
   
8. Available for Java, HTML / JavaScript and all major mobile platforms (at least):
 
   
The protocol should offer encoding / decoding libraries for all platforms, with the same level of functionalities. As a first step, being able to develop HTML, Android and iOS applications connected to a Java backend would be a good fit for my needs.
 
   
9. Optional code generation tools to replicate data models for all supported platforms:
 
   
Most of the time the data model has to be replicated by hands for each supported client platform: Java and Android (we don't want a dependency to JPA in an Android application), iOS, JavaScript (working with anonymous objects isn't a good practice), etc. Having a tool that can do this job quickly and reliably would be at least a time-saver.
 
   
___
 
   
Some of these 9 key features are clearly coming from my Java background, especially the 7th one about JPA. However, we certainly don't want a serialization protocol which couldn't be implemented in other languages or platforms because of its Java origin. Hence a 10th requirement:
 
   
10. Able to be ported in any other language or platform:
 
   
The serialization protocol must be implementable in other languages such as PHP, C, C++, etc. Features like being able to deal with JPA uninitialized properties should be implemented as extensions, for platforms where similar use cases exist (e.g.. a PHP ORM).
 
   
Next: What is the Spearal serialization protocol?
 
   
   
What is the Spearal serialization protocol?
 
   
Spearal is an ongoing effort to provide a set of libraries that implement the 10 requirements of the previous section. As of today, it is already usable with HTML / JavaScript applications connected to a Java server backend. As we will see in the next section, enabling Spearal in an existing AngularJS / Java EE application doesn't require much effort and can provide a great deal of improvements as soon as the data model doesn't stay trivial.
 
   
Spearal is open-source, released under the Apache Licence Version 2.0. Help and contributions are very (I mean VERY) welcomed. It is currently actively developed by two people, William Drai and I. We have both a long experience in developing Java EE applications and frameworks, acquired in particular through the development of the GraniteDS platform. More specifically, we are quite familiar with the implementation and usage of serialization formats such as AMF3 and JMF.
 
   
Without going into depth about the Spearal serialization format, let's highlight some of its key characteristics:
 
   
It is available for Java, JavaScript, Android and (soon) iOS:
 
   
The Java and JavaScript implementations are mostly complete and already usable. We are currently working on Android, some of the Java implementation requirements not being available on the Android platform (such as Javassist proxies for incomplete objects). An iOS port, using the new Apple's Swift language, is in preliminary development. Again, help and contributions would be very welcome.
 
   
It takes care of objects identity and strings equality:
 
   
When serializing an object graph, Spearal constructs a dictionary of objects (an identity map) and a dictionary of strings (based on equality). Whenever an object or string is encountered twice, a reference is encoded instead of the whole item. This feature enables the Spearal format (just like other serialization format such as AMF3 or ProtoBuf) to deal correctly with cyclic graphs, redundancy and allows to save a great deal of bandwidth with highly redundant data.
 
   
It integrates with JAX-RS and Spring:
 
   
Spearal provides an out-of-the-box JAX-RS provider and a Spring message converter. Enabling Spearal in an existing application that uses one of these frameworks is just a matter of a few lines of code or configuration. Writing a custom servlet for handling Spearal encoding and decoding should be trivial if you use none of these two frameworks.
 
   
It allows partial serialization and handles JPA uninitialized associations or outdated client models:
 
   
When Spearal encodes an object, it adds some information about its class and properties. If a client uses an outdated model (typically, some new properties of a class are missing), the unknown properties are ignored and the missing properties are marked as undefined when received by the other endpoint. In the Java implementation for example, an incomplete (or partial) object is decoded as a proxy that will throw an UndefinedPropertyException if one tries to access a property that wasn't supplied by the client.
 
   
JPA uninitialized properties are treated just the same: they are simply ignored, missing in the encoded output stream and marked as undefined for the receiver. The Spearal JPA2 extension, on the server-side, takes care of these undefined properties when merging a partial entity with already persisted data: missing properties, regardless if they are representing uninitialized associations or not, are never nullifying existing values.
 
   
Even if the client and server models are identical, Spearal offers a way to filter which properties are encoded. For example, given a bean of class X which has a lot of properties, you can ask the encoder to serialize only a few of them. For example, in Java, you will configure the encoder as follow:
 
   
SpearalFactory factory = new DefaultSpearalFactory();
SpearalEncoder encoder = factory.newEncoder(outputStream);
encoder.getPropertyFilter().add(X.class, "property1", "property2");
encoder.writeAny(new X());

According to the above code, the encoder will ignore all properties of the X instance but "property1" and "property2". Moreover, Spearal offers a way to ask the server, from the client and with specific HTTP headers, to return only some properties of X instances (we will soon see this feature in action with the Java EE / AngularJS demonstration).
It is a binary serialization format with parameterizable types:
While binary formats are not directly human-readable, they provide a compactness and typing richness which is hardly achievable with a textual format. Without going into depth about how Spearal serializes data, let's quickly exemplify the idea of parameterizable types with integral values.
Spearal has 15 primitive types:
NULL         : 0x00
TRUE         : 0x01
FALSE        : 0x02

INTEGRAL     : 0x10
BIG_INTEGRAL : 0x20
FLOATING     : 0x30
BIG_FLOATING : 0x40
STRING       : 0x50
BYTE_ARRAY   : 0x60
DATE_TIME    : 0x70
COLLECTION   : 0x80
MAP          : 0x90
ENUM         : 0xa0
CLASS        : 0xb0
BEAN         : 0xc0
The first 3 types are respectively bytes values for null, boolean true and false. All other types stand for more complex types and are parameterizable: the half byte of the type identifier can carry up to 4 bits of parameters. For example, the integral value of 65,535 (0xffff) is encoded using 3 bytes:
0x11 0xff 0xff
The first byte is the INTEGRAL identifier (0x10), plus the number of significant bytes in the integer (0xffff uses 2 significant bytes), minus 1. Hence we have 0x10 + (2 - 1) = 0x11, followed by the 0xff and 0xff bytes. If the integral value is negative, it is first converted to its absolute value and a negative sign flag (0x08) is used to save this information. Thus, all integral values are encoded with a type equal to 0x10 (identifier) | 0x08 (if negative) | 0x00...0x07 (length of the significant bytes minus 1). For the fast and worried minds here, Long.MIN_VALUE is encoded in a way that doesn't lead to an overflow.
This encoding is (IMHO) much simpler and readable than, for example, the ZigZag encoding used in ProtoBuf, while it gives the same advantages: the ability to use less than 8 bytes for the encoding 64-bits integral values (or less than 4 bytes for 32-bits values), which can save a great deal of bandwidth when you serialize a lot of small integers.
One more thing about Spearal JavaScript:
The JavaScript implementation is ECMAScript 6 ready: it uses the excellent Google's traceur compiler to convert JS 6 code into compatible JS 5 code.
The Java implementation of Spearal only allows to encode or decode java.io.Serializable objects. This is clearly a very basic security system, but it prevents many unwanted and highly dangerous operations. This security mechanism is however fully configurable and you can adopt a more restrictive and different approach.
Next: Enough talk, let's see Spearal in action!
   
Enough talk, let's see Spearal in action!
Before going to a less trivial Java EE / AngularJS sample application, we're going to start with a short offline test demonstrating how Spearal succeeds where JSON fails (see the "What's wrong with JSON?" section). You can download this sample as a Github Gist here.
First, lets' initialize Spearal and write an helper function for cloning an object through encoding / decoding:
var factory = new SpearalFactory();

function encodeDecode(obj) {
    var encoder = factory.newEncoder();
    encoder.writeAny(obj);
    return factory.newDecoder(encoder.buffer).readAny();
}
The SpearalFactory class is the entry point of the Spearal API and it lets you create encoders and decoders. It is also holding a shared context that allows you to customize the default behavior of the API. Here, we don't need anything specific and just accept all default settings. The encodeDecode() function creates an encoder, serialized its obj parameter in memory, creates a decoder with the encoded data and returns what it has read from the buffer.
Let's see what happens with cyclic structures:
var selfish = {};
selfish.friend = selfish;
console.log("Should be true: " + (selfish === selfish.friend));
selfish = encodeDecode(selfish);
console.log("Should be true: " + (selfish === selfish.friend));
The console output is as expected:
Should be true: true
Should be true: true
As we have seen before, JSON would immediately fail with such self-referent object. Next, let's make a test with shared instances:
var child = { age: 1 };
var parents = [{ child: child }, { child: child }];
parents[0].child.age = 2;
console.log("Should be 2: " + parents[1].child.age);
parents = encodeDecode(parents);
parents[0].child.age = 3;
console.log("Should be 3: " + parents[1].child.age);
Which results in this correct output:
Should be 2: 2
Should be 3: 3
Unlike JSON, Spearal doesn't duplicate the child object and the shared reference is preserved. Let's now see what happens with a JavaScript qualified object:
function BourneIdentity(name) {
    this.name = name;
}
var obj = new BourneIdentity("Jason");
console.log("Should be a BourneIdentity: " + (obj instanceof BourneIdentity));
obj = encodeDecode(obj);
console.log("Should be a BourneIdentity: " + (obj instanceof BourneIdentity));
This is what you get:
Should be a BourneIdentity: true
Should be a BourneIdentity: true
Here, JSON wouldn't keep track of the object class and the decoded copy would be an unqualified object. Finally, with JavaScript Date objects:
var date = new Date()
console.log("Should be a Date: " + (date instanceof Date));
date = encodeDecode(date);
console.log("Should be a Date: " + (date instanceof Date));
Here is the output:
Should be a Date: true
Should be a Date: true
JSON would serialize Date objects as strings, with no standard way to get a Date back after deserialization. While it is already nice to have Spearal working with these simple test cases, we have just scratched the surface of its possibilities.
Next: An AngularJS / JAX-RS / JPA sample application running Spearal
   
An AngularJS / JAX-RS / JPA sample application running Spearal
We're going to start with an existing and excellent sample application created by Roberto Cortez with AngularJS and Java EE. We have forked and slightly modified this demo to use AngularJS 1.3.0-rc.0, to run an embedded Wildfly instance and, of course, to include and enable the use of the Spearal libraries in the client and server applications.
Cloning, building and starting the demo
You just need to run the following three lines of command (you need Git and Maven):
git clone https://github.com/spearal-examples/javaee7-angular.git
cd javaee7-angular
mvn clean install wildfly:run
When Wildfly is started, point your browser to http://localhost:8080/javaee7-angular-3.1/. The application is self-explanatory, you can play with basic CRUD operations with your favorite manga heroes. Here is a screenshot of the interface:

So far, JSON is used for serialization. If you activate XMLHttpRequest logging in your browser (follow this procedure with a Chrome browser), you will notice this kind of output:

Switching from JSON to Spearal
We can switch to Spearal serialization very easily: just open a new browser tab and point it to http://localhost:8080/javaee7-angular-3.1/index-spearal.html. After the page is loaded, you won't notice any change with the JSON version: everything just works like before. To make sure that you're actually using Spearal instead of JSON, look at your XMLHttpRequest logs:

Let's have a quick look at the code: the only difference between the Spearal index page and the JSON are the following:
<script src="lib/dependencies/traceur-runtime.js"></script>
<script src="lib/dependencies/spearal.js"></script>
These two lines import the traceur runtime and the compiled ECMAScript 6 Spearal classes. Next come a block of inline JavaScript which configures the AngularJS $resourceProvider so it uses Spearal instead of JSON:
app.config(['$resourceProvider', function ($resourceProvider) {

    function encode(data, headersGetter) {
        if (data !== undefined) {
            var encoder = spearalFactory.newEncoder();
            encoder.writeAny(data);
            return new Uint8Array(encoder.buffer);
        }
    }

    function decode(arraybuffer, headersGetter) {
        if (arraybuffer && arraybuffer.byteLength > 0)
            return spearalFactory.newDecoder(arraybuffer).readAny();
    }

    var spearalFactory = new SpearalFactory();
    var contentTypeHeaders = {
        "Accept": Spearal.APPLICATION_SPEARAL,
        "Content-Type": Spearal.APPLICATION_SPEARAL
    };

    var actions = $resourceProvider.defaults.actions;
    for (method in actions) {
        var action = actions[method];
        action.responseType = 'arraybuffer';
        action.transformRequest = [ encode ];
        action.transformResponse = [ decode ];
        action.headers = angular.copy(contentTypeHeaders);
        /*
        if (method === "query") {
            action.headers[Spearal.PROPERTY_FILTER_HEADER] = Spearal.filterHeader(
               "com.cortez.samples.javaee7angular.data.Person",
               "id", "name", "description"
            );
        }
        */
    }
}]);
Basically, each REST action (get, save, query, remove and delete) is configured so it gets binary data (arraybuffer), uses the encode / decode function for (de)serialization and sets the "Accept" and "Content-Type" headers to "application/spearal". This block of code could be a good start for a spearal-angular library: remember, contributors are welcome.
Playing with Spearal partial serialization
So far so good, but using Spearal, even if the response size is about 20% smaller, doesn't seem to be a must to have. Let's now use property filtering. Uncomment the following part of the code in the src/main/webapp/index-spearal.html file:
if (method === "query") {
    action.headers[Spearal.PROPERTY_FILTER_HEADER] = Spearal.filterHeader(
        "com.cortez.samples.javaee7angular.data.Person",
        "id", "name", "description"
    );
}
Stop Wildfly, rebuild and restart the application with this command:
mvn clean install wildfly:run
Refresh the index-spearal.html page: everything is still working as before, but here is what you see if you look at your console:

With a few lines of code and without altering the application functionalities, we have just reduced the size of encoded data from 1.9KB to 449B (about 4 times smaller) and the execution of the request is also significantly faster (from 14ms to 10ms). A less trivial data model, with more properties and associations could certainly lead to more dramatic gains in terms of performance.
Some explanations about the code above: the "query" method is used when the application fetches a list of Persons, which is displayed in the data grid of the application. Because the data grid doesn't display the URL of the image, we don't need to query this field when populating the grid. The new code above, accordingly, adds a header to request only the id, name and description properties of each Person. However, when you click on a character in the grid, the entire Person bean is fetched through the "get" method, which is not configured with a property filter. That's why the image URL is still correctly displayed in the form.
Note: the configuration of the "query" method shouldn't be done at the $resourceProvider level. It should rather be enabled in the personService resource only, without affecting all other resources as well. It doesn't matter here since we are only using one resource and this global configuration is just a quick way of demonstrating how you can benefit from the Spearal partial serialization feature.
Next: Failing with JSON, server-side
   
Failing with JSON, server-side

Cyclic structures
Let's introduce some circularity in our Person entity. We're going to add a ManyToOne relationship pointing to the worst enemy of our manga characters. Edit the src/main/java/com/cortez/samples/javaee7angular/data/Person.java file and uncomment the following lines:
@ManyToOne
private Person worstEnemy;

public Person getWorstEnemy() {
    return worstEnemy;
}

public void setWorstEnemy(Person worstEnemy) {
    this.worstEnemy = worstEnemy;
}
Given the fact that it is quite usual to be the worst enemy of our worst enemy, let's also modify the src/main/resources/sql/load.sql script to introduce a circular reference. Uncomment this part of the script:
UPDATE PERSON SET WORSTENEMY_ID = 13 WHERE ID = 1;
UPDATE PERSON SET WORSTENEMY_ID = 1 WHERE ID = 13;
Uzumaki Naruto's worst enemy is now Rock Lee and vice-versa (with absolutely no respect to the Naruto story). Let's now stop, rebuild and restart the application. As soon as you reload the JSON index.html page, you get this error in your server console:
Caused by: com.fasterxml.jackson.databind.JsonMappingException:
Infinite recursion (StackOverflowError)
On the other hand, refreshing the Spearal page just works as before, the circular reference has been detected and successfully encoded. If you want to make sure that the client application actually received the worst enemy association, uncomment the worst enemy column definition in src/main/webapp/script/person.js:
columnDefs: [
    { field: 'id', displayName: 'Id' },
    { field: 'name', displayName: 'Name' },
    { field: 'description', displayName: 'Description' },
    { field: 'worstEnemy.name', displayName: 'Worst Enemy' },
    { field: '', width: 30, cellTemplate: [...] }
]
We also need to modify our filter to include the worstEnemy property in src/main/webapp/index-spearal.html:
if (method === "query") {
    action.headers[Spearal.PROPERTY_FILTER_HEADER] = Spearal.filterHeader(
        "com.cortez.samples.javaee7angular.data.Person",
        "id", "name", "description", "worstEnemy"
    );
}
You should see this result after rebuilding and restarting the application:


JPA uninitialized associations
Let's modify again our Person entity and configure our worst enemy association so it is lazily fetched:
@ManyToOne(fetch=FetchType.LAZY)
private Person worstEnemy;
As usual, stop, rebuild and restart the application. Refreshing the JSON index page in your browser will immediately trigger this error in your server console:
Caused by: com.fasterxml.jackson.databind.JsonMappingException:
could not initialize proxy - no Session:
...
Caused by: org.hibernate.LazyInitializationException:
could not initialize proxy - no Session
Server-side, JSON (Jackson here) is unable to detect a JPA uninitialized association and trying to serialize the content of a detached proxy, hence the LazyInitializationException.
Now, refresh the index-spearal.html page: everything works just fine, Spearal has detected and skipped the uninitialized association. If you have enabled the displaying of the worst enemy name (see above), you will just notice that nothing is displayed anymore in this column. This careful handling of uninitialized associations is not limited to single associations (proxies) but works as well with collections.
Could you explain that magic?
JPA handling with Spearal is activated by default in this application. It wouldn't work without a bit of setup and it is time to see how you enable it. Here is what you find in the src/main/java/com/cortez/samples/javaee7angular/rest/PersonApplication.java class:
@ApplicationPath("/resources")
public class PersonApplication extends Application {

    @PersistenceUnit
    private EntityManagerFactory entityManagerFactory;

    @PersistenceContext
    private EntityManager entityManager;
    
    @Produces @ApplicationScoped
    public SpearalFactory getSpearalFactory() {
        SpearalFactory spearalFactory = new DefaultSpearalFactory();
        SpearalConfigurator.init(spearalFactory, entityManagerFactory);
        return spearalFactory;
    }
    
    @Produces
    public EntityManager getEntityManager() {
        return new EntityManagerWrapper(entityManager);
    }
}
First, an EntityManagerFactory and an EntityManager are injected in our REST application: there is nothing special here, this is just standard Java EE injection. Then, a SpearalFactory is created, configured with the injected entityManagerFactory and finally outjected through the @Produces CDI annotation. The Spearal JAX-RS module used in this sample application needs this outjected, application scoped, SpearalFactory for encoding / decoding operations (this setup would be much simpler with the Spring message converter).
The SpearalConfigurator.init(...) instruction, which is part of the Spearal JPA2 module, does two things: it constructs a set of all persistent classes present in the current JPA context, so the Spearal encoder is aware of classes that can have uninitialized properties; it also adds the id and version properties of these classes to the set of UnfilterableProperty's, so we can be sure that they are always serialized, regardless of which properties were requested by the client. In the JavaScript code used in the previous section to filter Person's properties, the following two expressions are strictly equivalent:
Spearal.filterHeader(
    "com.cortez.samples.javaee7angular.data.Person",
    "id", "name", "description"
)
Spearal.filterHeader(
    "com.cortez.samples.javaee7angular.data.Person",
    "name", "description"
)
Entity's ids (and versions if any) are always encoded, you never need to request them explicitly.
Next is the wrapping and outjection of the EntityManager. Without going into depth about how this wrapper works, here are some highlights: when Spearal receives a partial object on the server-side (i.e. not all properties of a given class have been serialized), it creates a proxy keeping track of what was present and absent. As we will see in the next section, the EntityManagerWrapper is responsible of unproxifying these special objects before giving them back to the underlying JPA engine.
If you give a quick look at the src/main/java/com/cortez/samples/javaee7angular/rest/PersonResource.java class, you will see that the entity manager is injected with a CDI @Inject annotation:
@Inject
private EntityManager entityManager;
The entity manager injected in the person resource is the one outjected by the person application. It means that we have been transparently using this wrapped entity manager from the beginning, with JSON and Spearal: the wrapper doesn't do anything if there are no Spearal proxies and such proxies are only created during a Spearal deserialization. Using an unwrapped entity manager wouldn't change anything to what we have experienced with JSON so far.
Note: to be fair, Jackson has an extension to deal with uninitialized associations and we will use it in the next section. But, unlike the Spearal module that works with all JPA 2 engines, the jackson-datatype-hibernate, as its name implies, only works with Hibernate. More importantly, it only prevents from LazyInitializationException's (or unwanted initialization if a session is active): an uninitialized association is serialized as null and, when you send it back from the client for update, the server has no way to recreate an uninitialized association in place of this null value. Without a careful handling of this case in your code, you're likely going to end with the deletion of this association in your database!
Next: Dealing with outdated client applications
   
Dealing with outdated client applications
Now, we want to simulate what would happen with an outdated client application that is unaware of the new worst enemy property we added above. With a classical Web application like the one we are using here, outdated clients are usually not an issue at all: the HTML application is loaded from the server each time it is accessed and reflects the last state of your deployment. However, if the client application is installed on the user's terminal (eg. an HTML mobile application created with PhoneGap), you are likely going to deal with outdated clients that are unaware of the evolution of your data model.
First, we want to revert the client application to its original state. Comment out the WorstEnemy column in src/main/webapp/script/person.js:
columnDefs: [
    { field: 'id', displayName: 'Id' },
    { field: 'name', displayName: 'Name' },
    { field: 'description', displayName: 'Description' },
    // { field: 'worstEnemy.name', displayName: 'Worst Enemy' },
    { field: '', width: 30, cellTemplate: [...] }
]
Because we want the JSON serialization to work with uninitialized associations, we need to edit our pom.xml file and uncomment the Jackson's Hibernate extension dependency:
<dependency>
    <groupId>com.fasterxml.jackson.datatype</groupId>
    <artifactId>jackson-datatype-hibernate4</artifactId>
    <version>2.4.0</version>
</dependency>
Next, we need a new JAX-RS provider to initialize this Jackson's extension: move the jackson-ext/JacksonContextResolver.java class to the src/main/java/com/cortez/samples/javaee7angular/rest directory, next to the PersonApplication.java file.
Finally, edit the src/main/java/com/cortez/samples/javaee7angular/rest/PersonResource.java file and make the following changes in the findPersons method:
...
// return query.getResultList();

List<Person> persons = query.getResultList();
System.out.println("-------------");
for (Person p : persons) {
    System.out.printf(
        "%d) %s's worst enemy: %s\n",
        p.getId(),
        p.getName(),
        (p.getWorstEnemy() != null ? p.getWorstEnemy().getClass().getName() : "null")
    );
}
return persons;
Instead of just returning the query result, we will now also print the class of the worst enemy association to see if it is null or not. Since the association is lazily fetched, the class of the worst enemy entity is going to be a Javassist Hibernate proxy, as we will see soon.
Stop, rebuild and restart the application. Refreshing the JSON index page should now work, the Jackson Hibernate extension preventing the serialization of the uninitialized association. The Spearal page works fine as well and looks exactly the same.
If you have a look at the Wildfly console, you can see this kind of output:
1) Uzumaki Naruto's worst enemy: com.cortez.samples.javaee7angular.data.Person_$_jvst153_0
2) Hatake Kakashi's worst enemy: null
...
Now, from the Spearal page, let's modify the Uzumaki Naruto record: click on the corresponding line in the grid, enter another name in the "Name" field and save it. Everything works just fine and the Java console looks exactly the same: Uzumaki Naruto (or the new name you entered) has still a worst enemy, updating this record from the outdated Spearal page didn't break this association.
Go to the JSON page and refresh it: you should see the modification you just made. Edit the name of the first Person (ex Uzumaki Naruto) and save it. Have a look again at the Java console:
1) Uzumaki Naruto's worst enemy: null
2) Hatake Kakashi's worst enemy: null
...
Editing the Uzumaki Naruto record from the outdated JSON application nullified the existing worst enemy association. While Spearal detected an incomplete object and only updated defined properties, JSON received and saved a null value for the worst enemy association.
Let's have a look at the savePerson method in the src/main/java/com/cortez/samples/javaee7angular/rest/PersonResource.java file:
@POST
public Person savePerson(Person person) {
    return entityManager.merge(person);

    /*        
    if (person.getId() == null) {
        Person personToSave = new Person();
        personToSave.setName(person.getName());
        personToSave.setDescription(person.getDescription());
        personToSave.setImageUrl(person.getImageUrl());
        entityManager.persist(person);
    } else {
        Person personToUpdate = getPerson(person.getId());
        personToUpdate.setName(person.getName());
        personToUpdate.setDescription(person.getDescription());
        personToUpdate.setImageUrl(person.getImageUrl());
        person = entityManager.merge(personToUpdate);
    }

    return person;
    */
}
The part of the code in comments is the one which was originally used for creating or updating persons. We have intentionally replaced it by a simple entityManager.merge(person) instruction. Re-enabling this code would of course prevent from the issue we just had with JSON: the worst enemy association wouldn't be nullified, because the code above only updates the name, description and imageUrl properties.
However, a property added to a data model is of course meant to be used somewhere and the most straightforward evolution of the above code would be:
if (person.getId() == null) {
    ...
    personToSave.setWorstEnemy(person.getWorstEnemy());
    entityManager.persist(person);
} else {
    ...
    personToUpdate.setWorstEnemy(person.getWorstEnemy());
    person = entityManager.merge(personToUpdate);
}
We don't need to test this code to figure out that it wouldn't change anything with an outdated JSON application: the worst enemy association would be broken as well. You can imagine several (bad) solutions to this problem. The safest one (IMHO) is to create a new resource, say PersonResourceV2, which is used by your new client, while the outdated one still uses the old version, without the setWorstEnemy calls.
Whatever is the adopted solution, it will never be as simple as the one you can use with Spearal: use a simple merge and do nothing. Missing properties will be detected and ignored, there is no risk of messing up new properties with outdated clients. And, if you really want to use the get/set pattern above, you can do it in a much simpler way with Spearal, without writing a new PersonResourceV2:
try {
    personToUpdate.setWorstEnemy(person.getWorstEnemy());
}
catch (UndefinedPropertyException e) {
    // deal with your outdated client.
}

Differential updates
This ability to deal with incomplete data has another interesting application: the client can send for update only the few properties that have been actually modified. We can quickly illustrate this feature with our angular application. The form in the src/main/webapp/index-spearal.html page already passes the personForm object as a parameter in the call of the updatePerson method:
<form name="personForm" ng-submit="updatePerson(personForm)" novalidate>
Now, edit src/main/webapp/script/person.js and uncomment the following code in src/main/webapp/script/person.js:
delete person._class;
if (!personForm.name.$dirty)
    delete person.name;
if (!personForm.description.$dirty)
    delete person.description;
if (!personForm.imageUrl.$dirty)
    delete person.imageUrl;
alert('Person to save: ' + angular.toJson(person, true));
Stop, rebuild and restart the application as usual. Then, refresh the Spearal index page in your browser and edit the name (only the name) of one of the persons. When you click on the "Save" button, you can see this alert popup:

The popup reflects exactly what the server is going to receive: the id and the name property, without any other field. However, after clicking on "OK", you will see that the grid contains your modification and that the others properties weren't nullified by this partial (or differential) update. It wouldn't work with JSON, even with the old version of server side code: missing properties would result in null properties.
Next: Conclusion and future work
   
Conclusion and future work
With this long article (thank you if you have reached this last page!) we have been demonstrating how Spearal overcomes the limitations of JSON and introduces interesting new features:

It correctly deals with graph references, even circular.
It retains the identity of objects (aka classes).
It allows data graph filtering.
It correctly deals with outdated client applications.
It allows differential updates.

Through its integration with JAX-RS and JPA2, we have been able to port quite easily an existing Angular JS / Java EE application using JSON and show the full benefits of Spearal as soon as the data model doesn’t stay trivial. Spearal is still in an early development stage and we plan to have the following client technology scope in the near future:

JavaScript / HTML (beta)
Android (beta)
iOS (early development stage)

And, server-side (Java):

JAX-RS integration (beta)
Spring integration (beta)
JPA 2 integration (beta)

Feedback would be very appreciated, especially if you plan using or working with us on Spearal. Don’t hesitate to comment on this article or to contact us in the Spearal forum!
Follow us on Twitter: @Spearal.