Skip to content

APISerialization

Jump to bottom Edit New page
Andrew Byrd edited this page Feb 24, 2014 · 9 revisions

This page contains developer-oriented notes about how OTP REST API responses are converted to a textual format that is sent across the wire to the client.

Definitions

This Stack Overflow response clarifies the relationship: http://stackoverflow.com/a/17981041/778449 The JAX-RS implementation needs to receive and send objects (request and response bodies) over HTTP. When JAX-RS wants to return a response with the application/xml media type, it uses JAXB to figure out how to write objects out as XML.

Serialization to XML vs. JSON

JAXB entails adding annotations to classes indicating what fields should be serialized and how, such as @XmlRootElement and @XmlElement (actually, the original intent was to generate annotated class files from an XML schema). If we want to also provide other formats such as JSON, the JSON library could re-use the JAXB annotations to provide equivalent serialized responses for the same annotated objects. So the JAXB annotations (despite including the letters XML) can be applied to an XML or JSON serialization process.

However, there are ugly points in using JAXB to produce JSON. For example. the attributes vs. content distinction in XML does not naturally exist in JSON. When going through JAXB to produce JSON, HashMaps will also become strangely verbose. It is therefore often cleaner to simply use a library that converts objects directly to JSON. See: http://stackoverflow.com/questions/6001817/jackson-with-jaxb

While it is theoretically possible to use the same model classes and annotations to produce exactly equivalent responses in multiple formats (XML and JSON) in practice it is difficult to keep the two in sync, because special cases will be described and handled differently by the two libraries in use. Therefore it is likely that we will drop XML support in version 1.0 of the OTP API to avoid future maintenance concerns. Research has also shown that JSON is much less resource intensive on mobile devices.

Enabling JSON output from JAX-RS

It appears that Jersey does not automatically include a JSON MessageWriter, because if you remove the Maven dependency com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider from the POM, then requests for Content-type:appication/json will fail with a:

javax.ws.rs.WebApplicationException: com.sun.jersey.api.MessageException: 
A message body writer for Java class org.opentripplanner.api.model.transit.RouteList, and Java type class org.opentripplanner.api.model.transit.RouteList, and MIME media type application/json was not found.

You need a JSON library that implements the extension points MessageBodyReader/Writer of JAX-RS, then declares its implementations to Java's ServiceLoader mechanism, allowing Jersey (and other software) to automatically detect and use it for JSON parsing/writing. OTP is using the Jackson library to supply JSON capabilities to Jersey.

Some wisdom gleaned from Stack Overflow and wikis on choosing and using a JSON library:

  • "Jackson and Gson are the most complete Java JSON packages regarding actual data binding support; many other packages only provide primitive Map/List (or equivalent tree model) binding." That is, Jackson and Gson allow marshalling and unmarshalling objects, other libraries may just parse the JSON into a tree of strings.
  • Jackson 1.x and 2.x live in different Java packages (1.x under org.codehaus.jackson, 2.x under com.fasterxml.jackson), be sure to import Jackson objects from matching versions.
  • Do not use Maven dependencies jersey-media-json (doesn't exist anymore in Jersey 2.x) and jersey-json (only for Jersey 1.x). Use either jersey-media-moxy (for JAXB) or jersey-media-json-jackson (for POJO). The Jersey User Guide has information on these modules in the JSON section at https://jersey.java.net/documentation/latest/media.html#json . Note that in OTP we are using yet another module from Jackson itself, I have no idea what the difference is with the jersey-media module.
  • "As of v1.18 of Jersey you do NOT need to write your own MessageBodyWriter and the return type of your @GET methods can be POJO-objects." That is, your Jersey resource methods can just return any old object, and Jackson will still be used to serialize it according to the usual rules.
  • "Jackson 1.7 added ability to register serializers and deserializes via Module interface. This is the recommended way to add custom serializers -- all serializers are considered "generic", in that they are used for subtypes unless more specific binding is found. The simplest way is to extend SimpleModule, add serializer(s), and register module with ObjectMapper."

POJO Mapping

We are using "POJO mapping" rather than JAXB. POJO mapping means just reading the public fields and getters of objects and outputting them to JSON rather than requiring annotations. We were enabling a the "POJO mapping feature", which avoids the need for annotations but why this was necessary is not clear to me; perhaps for XML output.

The Jackson FAQ states that "Jackson 1.0 supports Bean method - based binding, where serialization requires 'getter' methods, and deserialization 'setters' (with optional 'creator' methods). Version 1.1 also supports direct access of member fields (public instance fields, annotated instance fields)."

However, "With version Jackson 1.1, it is also possible to use subset of JAXB annotations to denote getter and setter methods and fields." See http://wiki.fasterxml.com/JacksonJAXBAnnotations

Badgerfish Style

We initially had problems in OTP with the "Badgerfish" JSON mapping which is designed to perfectly reproduce the semantics of XML but is bizarrely non-idiomatic in JSON. At one point we had to disable this, specifying a more natural mapping from objects to JSON. However, the current Jackson FAQ asserts that Jackson does not implement Badgerfish or any other mapping, as it is "100% JSON and does not try to imitate or emulate XML. Property name mapping is based on standard Java Bean naming convention (although can be overridden using annotations or custom naming strategy)."

Custom JSON response serialization using Jackson and Jersey

The Jackson API provides multiple ways to do many things. Here are a few ways to specify a custom serialization method for a class:

The first three require annotations. In OTP we have a third-party class (AgencyAndId) that we want to serialize in a nonstandard way. There is always the possibility of wrapping the third-party class in a custom class we have control over, but imagine how many small modifications that will entail throughout the code. It would also be possible to add @JsonSerialize to every field of the third-party type, but some of those fields might be found in other third-party types (e.g. the One Bus Away model of GTFS which contains AgencyAndId fields we want to serialize in OTP Transit Index API responses).

The fourth option seems to be the best way to blanket-redefine how all instances of a third-party class are serialized. http://wiki.fasterxml.com/JacksonHowToCustomSerializers says that "Jackson 1.7 added ability to register serializers and deserializes via Module interface. This is the recommended way to add custom serializers."

Most of these methods require a separate serializer class, defined like so:

public class ThingSerializer extends JsonSerializer<Thing> {

    @Override
    public void serialize(Thing thing, JsonGenerator jgen,
        SerializerProvider provider) throws IOException, JsonProcessingException {
        jgen.writeStartObject();
        jgen.writeNumberField("x", thing.getNumber());
        jgen.writeEndObject();
    }

    @Override
    // Gets around type erasure, allowing 
    // module.addSerializer(new ThingSerializer()) to correctly associate this 
    // serializer with the proper type
    public Class<Thing> handledType() {
        return Thing.class;
    }

}

Once we have a custom serializer defined for AgencyAndId, we have to register it with Jackson. This involves making or getting an ObjectMapper, combining some serializers/deserializers into a Module, then registering that Module with the ObjectMapper, as described at http://wiki.fasterxml.com/JacksonHowToCustomSerializers. The ObjectMapper then constructs corresponding ObjectWriters which are used to output JSON. Unfortunately we are not calling ObjectMapper.writer().writeValue() manually. Instead JAX-RS (Jersey) is automatically writing out our result objects. Therefore need to somehow register our custom serializer Module with the specific ObjectMapper instance used by Jersey.

The only way I have found to do this is via a ContextResolver (thanks to this post http://jersey.576304.n2.nabble.com/Customizing-ObjectMapper-tp6234597p6234646.html). This ContextResolver is annotated with @Provider and detected by Jersey in package-scanning mode, in much the same way it detects REST resource classes or parameter classes. The purpose of a ContextResolver is to provide a specific ObjectMapper as a function of the type we are serializing. For example:

@Provider
@Produces(MediaType.APPLICATION_JSON)
public class OTPObjectMapperProvider implements ContextResolver<ObjectMapper> {

    private final ObjectMapper mapper;

    /**
     * Pre-instantiate a Jackson ObjectMapper that will be handed off to all incoming Jersey
     * requests, and used to construct the ObjectWriters that will produce JSON responses.
     */
    public OTPObjectMapperProvider() {
        // Create a module, i.e. a group of one or more Jackson extensions.
        // Our module includes a single class-serializer relationship.
        // Constructors are available for both unnamed, unversioned throwaway modules
        // and named, versioned reusable modules.
        SimpleModule module = new SimpleModule("OTP", Version.unknownVersion());
        module.addSerializer(AgencyAndId.class, new AgencyAndIdSerializer());
        mapper = new ObjectMapper();
        mapper.registerModule(module);
        mapper.setSerializationInclusion(Include.NON_NULL); // skip null fields
    }

    /** 
     * When serializing any kind of result, use the same ObjectMapper.
     * The "type" parameter will be the type of the object being serialized,  
     * so you could provide different ObjectMappers for different result types.
     */
    @Override
    public ObjectMapper getContext(Class<?> type) {
        return mapper;
    }

}

Tracing execution, we see that the Jersey IOC component provider instantiates the ContextResolver at startup, after the package scanning is finished and it logs "Initiating Jersey application". The specific ObjectMapper used to serialize a response is determined by calling getContext(Class type) while handling each incoming Jersey request. The "type" parameter contains the type of the response object being serialized, so you could provide differently configured ObjectMappers for different result types. The repeated calls to getContext are why we create the ObjectMapper once and return the pre-constructed instance in the getContext method.

Another option exists that allows specifying Jackson serialization annotations for third-party classes, which are called mix-In Annotations: http://wiki.fasterxml.com/JacksonMixInAnnotations This provides an annotation-driven alternative to Serializer, and the mix-in class or interface must be registered with an ObjectWriter in much the same way. However, they do not need to be grouped into Modules to be registered, which makes the code simpler.

Using Jackson to serialize XML

Jersey has its own JAXB-annotation-driven MessageWriters, as required by the JAX-RS standard it implements. However, since we are using Jackson to produce JSON, especially if we make use of its POJO abilities the JSON it produces will get out of sync with the JAXB XML.

Jackson (which we are using to make JSON in Jersey) can now also produce/consume XML: https://github.com/FasterXML/jackson-jaxrs-xml-provider

Jackson gives priority to plain-old-java-objects rather than tagging everything up with annotations (of course annotations can be added when clarification is needed). It is intended to be "code-first", not work on classes generated from an XML schema. If we use Jackson for both XML and JSON, we can ensure equivalent output in both formats.

However, if you want to add Jackson XML be sure to match the version with that of the other Jackson modules, or you will get the traditional malfunctions and cryptic stacktraces (see commits 4a46f399f8fa16869cc9d5dd9f4430bf53b16ead and 9d4fd1600353fb6bf338cd914d179e501d9dfe8b).

If you want to use Jackson's XML serialization instead of Jersey's in-built system, you just add a Maven dependency for the Jackson XML writer, which implements some extension points and mysteriously makes it possible for Jersey to detect and use them. It's not immediately clear to me why or how Jackson automatically overrides the default XML (JAXB) serialization library. The selection order is "clarified" at https://jersey.java.net/documentation/latest/message-body-workers.html#providers-selection and is related to XMLJAXBElementProvider but I still can't grasp how it will behave.

Doc Links

Add a custom footer

The documentation on this wiki is outdated and should not be used

unless you are intentionally working with legacy versions of OpenTripPlanner. Please consult the current documentation at readthedocs

Clone this wiki locally