Performance Benchmarks
Performance (i.e., fast parsing of JSON or XML) is important to us, since it has a direct effect on the amount of time a user has to wait before they can see results fetched from the internet - in our case, real-time transit information from a SIRI REST API. Slower parsing also typically results in longer CPU, screen, and radio usage, which all negatively effect battery life.
We can optimize our SIRI client in many different ways. In this article, we'll look at the following items related to Jackson, since that's what we're using as the JSON and XML parser in our SIRIRestClient library:
- HTTP connection - should we use an Android-specific HTTP connection object, or the default internal Jackson HTTP object (carried over from J2SE)?
- Should we use the Jackson ObjectMapper or the ObjectReader for JSON parsing?
- Should we choose XML or JSON as the response data format?
Note: You can perform your own benchmarks using the SiriRestClientUI app, just like we did for this article!
We have two basic options for HTTP connections when using Jackson:
- Android HttpURLConnection - Instantiate our own HTTP connection via Android's HttpURLConnection, and pass the InputStream from that connection into the Jackson object.
- Use embedded Jackson HTTP connection - Simply pass the URL into the Jackson object (e.g., ObjectMapper, ObjectReader, or XmlMapper) and Jackson handles establishing an HTTP connection and retrieving the data.
An article on the Android Developer Blog article states that HttpURLConnection should be the most efficient connection type on Android. According to Jackson Best Practices, we should just pass in the URL and let Jackson handle the connection. But, we should keep in mind that these are suggestions targeted at desktop and server developers, and not mobile developers. As of v2.1, Jackson seems to just use the default FileInputStream connection from the normal Java platform. Given that Android is a specialized platform, it seems that the Android HttpURLConnection would be more efficient and would override any existing Jackson best practices that aren't targeted directly at the mobile platform.
For more discussion of Android's HTTP clients, please see http://android-developers.blogspot.com/2011/09/androids-http-clients.html
We hope to add future benchmarking results to this section evaluating which connection type is more efficient using the SiriRestClientUI app.
You can perform your own benchmarks using the SiriRestClientUI app, as the settings menu allows you to pick between the two HTTP connection types:
The code for both HTTP connection types is implemented in the SiriRestClient library.
There are two options for Jackson objects that can be used when parsing a JSON-formatted response:
- Use the
ObjectMapperdirectly - the core class that must be instantiated before parsing JSON content in Jackson. - Use the
ObjectReader- this object is retrieved from theObjectMapperafter theObjectMapperis instantiated, and can be used to parse JSON instead of using theObjectMapperdirectly.
According to Jackson Best Practices, using the ObjectReader should be more efficient than the ObjectMapper, especially for Jackson versions 2.1 and later. Therefore, The ObjectMapper configuration is included only for benchmarking purposes.
We hope to add future benchmarking results to this section evaluating which connection type is more efficient using the SiriRestClientUI app.
You can perform your own benchmarks using the SiriRestClientUI app, as the settings menu allows you to pick between the ObjectMapper or ObjectReader classes:
Check out the class SiriJacksonConfig included in this project to see the instantiation and configuration of the ObjectMapper and ObjectReader, and the SiriRestClient.makeRequest() method for how the ObjectMapper and ObjectReader and then used to actually parse the JSON response.
Note: For parsing XML, there is only one option: XmlMapper.
Many web services include the option of sending a response in either JSON or XML formatting. If you're not familiar with the differences between the two formats, see a detailed discussion on the Parsing JSON and XML on Android page.
In short, JSON response tend to include far fewer characters than XML response, and look like this:
{
Siri: {
ServiceDelivery: {
ResponseTimestamp: "2012-08-21T12:06:21.485-04:00",
VehicleMonitoringDelivery: [
{
VehicleActivity: [
{
MonitoredVehicleJourney: {
LineRef: "MTA NYCT_S40",
DirectionRef: "0",
FramedVehicleJourneyRef: {
DataFrameRef: "2012-08-21",
DatedVehicleJourneyRef: "MTA NYCT_20120701CC_072000_S40_0031_S4090_302"
},
JourneyPatternRef: "MTA NYCT_S400031",
PublishedLineName: "S40",
OperatorRef: "MTA NYCT",
OriginRef: "MTA NYCT_200001"
}
}
]
}
]
}
}
XML tends to be more verbose, and looks like this:
<Siri xmlns:ns2="http://www.ifopt.org.uk/acsb" xmlns:ns4="http://datex2.eu/schema/1_0/1_0" xmlns:ns3="http://www.ifopt.org.uk/ifopt" xmlns="http://www.siri.org.uk/siri">
<ServiceDelivery>
<ResponseTimestamp>2012-09-12T09:28:17.213-04:00</ResponseTimestamp>
<VehicleMonitoringDelivery>
<VehicleActivity>
<MonitoredVehicleJourney>
<LineRef>MTA NYCT_S40</LineRef>
<DirectionRef>0</DirectionRef>
<FramedVehicleJourneyRef>
<DataFrameRef>2012-09-12</DataFrameRef>
<DatedVehicleJourneyRef>MTA NYCT_20120902EE_054000_S40_0031_MISC_437</DatedVehicleJourneyRef>
</FramedVehicleJourneyRef>
<JourneyPatternRef>MTA NYCT_S400031</JourneyPatternRef>
<PublishedLineName>S40</PublishedLineName>
<OperatorRef>MTA NYCT</OperatorRef>
<OriginRef>MTA NYCT_200001</OriginRef>
</MonitoredVehicleJourney>
</VehicleActivity>
</VehicleMonitoringDelivery>
<ServiceDelivery>
</Siri>
The extra characters contained in the XML response, plus the additional complexity that XML parsers must negotiate to bind an XML document to a Java object, result in substantial negative impacts on mobile device performance, including slower responses and less battery life.
Our goal with the following experiments is to demonstrate the benefits of using JSON over XML, and also discuss some optimizations that can further improve the user experience of apps that fetch real-time transit data from web services such as a SIRI REST API.
Note: You can perform your own benchmarks using the SiriRestClientUI app, as the settings menu allows you to pick between JSON or XML responses:
The below section discuss the performance of the SiriRestClient (using Jackson for response parsing) for JSON and XML. Special optimizations to further improve the user experience are discussed in the next section.
We used the SiriRestClientUI app to perform benchmarks of JSON vs. XML parsing using the SiriRestClient library on a Samsung Galaxy S3 (32GB USA Sprint CDMA) with Android 4.1.1, 1.5 GHz dual core processor, 2GB RAM (Power saving mode off). Jackson 2.1.2 with Aalto 0.9.8 was used. The Jackson Internal HTTP connection was used, as well as the ObjectReader for JSON parsing. These tests were performed on the USF WiFi network. Results from SpeedTest.NET Android app at time of test were 51353 kbps down, 49554 kbps up, and ping of 16 ms.
50 requests were performed back-to-back using the MTA BusTime SIRI StopMonitoring API, without any time between requests. We implemented a timestamp recording feature to the SiriRestClientUI app and SiriRestClient library, which automatically captures how long a request took, from when the request was issued to when a Siri object became available from Jackson.
The detailed steps of the methodology we used can be found on the Detailed Test Methodology - Normal Benchmarks page, if you'd like to replicate these tests yourself.
XML vs. JSON
The elapsed time from request to parsed response for 50 sequential requests is shown in Figure 1.
Figure 1 - XML vs. JSON Parsing - Elapsed Time from Request to Parsed Response for 50 Requests
There is a substantial difference between the "cold start" times for both JSON and XML (i.e., the first request). The time for the XML cold start response is almost 18 seconds, over 4 times as long as the time for the JSON cold start response (a little over 4 seconds). After the cold start, the differences between the response times for JSON and XML are much smaller.
Figure 2 shows the summary statistics for XML and JSON parsing time for this test.
Figure 2 - XML vs. JSON Parsing - Summary Statistics for the Elapsed Time of 50 Requests
JSON edges out XML in average response time of 401ms, vs. the XML average response time of 625ms. 95th percentile of elapsed times is closer, with JSON having a 95th percentile of 501ms and XML having a 95th percentile of 626ms. The increase in standard deviation for XML response time reflects the initial large cold start value that is substantially larger than the following warm starts.
As mentioned above, the first execution of a request to the server (i.e., a cold start) typically takes much longer than subsequent requests (i.e., warm starts). This is because Jackson will dynamically construct Class model from Java class definitions the first time deserializers are needed. This typically occurs first time the readValue() Jackson method is called (i.e., when the application initiates the request). Here, JSON yields significant better performance than XML - JSON performance is over 4 times faster than XML with a time difference of 14 seconds. After this initial cold start, Jackson will typically parse subsequent responses (i.e., warm starts) more quickly for both JSON and XML. As stated above, JSON has a slight performance advantage for warm starts too - an average of 224ms faster than XML. Since recent human-computer interaction studies have indicated that users can perceive time differences of 100ms when waiting for a response [1], JSON still yields a noticeable performance increase for warm starts from the user's perspective.
On Android, the warm start state can persistent even if the app is closed by the user (i.e., the "Back" button is pressed) and re-opened. Android devices will typically keep recently closed apps in memory as a cached background process to enhance future startup performance. In this case, when the user "starts" the app, it is actually loading the application from the cached process in memory, which loads all the Jackson data structures needed to perform quick parsing on warm starts without needing to re-initialize them from a cold start state. As a result, when comparing different app configurations (e.g., XML parsing time vs. JSON), its important to note that the first execution of the app (i.e., when the app is truly starting up for the first time without being restored from a cached process) will typically show significantly worse performance than subsequent warm starts.
While the warm start state provides a significant advantage in response time performance, it unfortunately cannot be reliable upon for consistent performance increases after an app is started on the device. Android may remove a cached process from memory if the operating system/framework is running low on memory. Given the multitasking that typically occurs on most cell phones, especially in a scenario where the user is waiting for a bus to arrive, performing tasks such as checking email, internet browsing, or using social networking apps may result in the real-time transit app being removed from the app cache. At this point, the app will revert to the cold start state and the there may be a significant delay in retrieving transit data. The size of the SiriRestClientUI cached process was observed to typically be between 24-31MB, which is large for a frequently used app. For comparison, on a Samsung Galaxy S3 following apps had the following cached sizes - Calendar app = 7.2MB, Clock = 16MB, Google+ = 17MB, Maps = 13MB. Since the largest non-system processes are typically the first targets for process cache eviction, it is likely that this app would frequently be restored to a cold start state.
The above observations lead us to pursue a manual caching strategy for Jackson objects on Android in an attempt to consistently reduce the cold start penalty for response times. The follow section discusses the results of these Jackson object manual caching optimizations.
To improve cold start performance of retrieving and parsing the SIRI responses using Jackson, we must gain further control over the caching process. To recap, warm starts have substantially shorter elapsed response/parsing times than cold starts. Our definition of cold and warm starts is again detailed below:
- Cold start = Android app process is not cached and no requests have been executed during this app execution (e.g., first start up of app).
- Warm start = The Android app already contains an initialized Jackson object from a previous request. This state occurs when the app is already running and has made previous requests during this execution session, or when the Android app process was cached from a previous execution and restored from that cached state by the Android OS.
One method to possibly improve cold start performance is to gain further control over the caching mechanism that makes warm starts performance substantially better than cold starts. Since an Android app can't directly control when its own process is cached (or evicted from the process cache) by the Android OS, we must look at alternative caching mechanisms that can save and retrieve this application state even after the application process is completely terminated and evicted from the process cache by the Android OS. We call this mechanism a "Pseudo-warm start," which we define as follows:
- Pseudo-warm start = An artificial warm-start by the app, where the Jackson object is manually cached to persistent memory by the app and read from memory during app startup (in what would otherwise be a cold start).
Figure 3 shows the stages of application execution for A) Cold Starts, B) warm starts, and C) Pseudo-Warm Starts.
Figure 3 - The Stages of App Execution for Cold, Warm, and Pseudo-warm Starts
The red-shaded blocks (i.e., the first block in A) and C)) indicate the overhead required to initialize Jackson (note that warm starts do not have this overhead). In C), we replace the normal Jackson initialization process with reading the cached Jackson objects that were initialized in a previous application execution and then written to a persistent cache - this is the Pseudo-Warm Start.
Our initial goal with our first set of tests, discussed next, was to evaluate whether the cache read used in Pseudo-Warm starts is faster than the normal Jackson initialization process in cold starts.
A final note on Figure 3 - With C) Pseudo-Warm Starts, we can actually start the cache read immediately on application start-up. Unlike the normal Jackson initialization on a cold start, we don't need to wait for the user to initiate a request before starting the initialization process via the cache read. This flexibility is indicated in Figure 3 via the "Request initiated" call out over the entire cache read period, as the request may be initiated by the user at any point during the cache read. The flexibility of when the cache read can be started is important in later discussions of this optimization.
Android cache implementation details - We implement cache writing in Android by serializing the Jackson objects that contain the state necessary to re-create a warm start to a private file in the application's package via the Context.openFileOutput() method and an ObjectOutputStream. We implement cache reading by reading the file from the cache and deserialize the Jackson objects on a cold-start via the Context.openFileInput() method and the ObjectInputStream. For JSON formatting, either the Jackson ObjectMapper or ObjectReader is serialized and deserialized, depending on the configuration of the SIRI Rest Client via the "Settings" menu. For XML formatting, the Jackson XMLMapper is serialized and deserialized.
We again used the SiriRestClientUI app to perform benchmarks of JSON vs. XML parsing using the SiriRestClient library on a Samsung Galaxy S3 (32GB USA Sprint CDMA) with Android 4.1.1, 1.5 GHz dual core processor, 2GB RAM (Power saving mode off). Jackson 2.1.2 with Aalto 0.9.8 was used. The Jackson Internal HTTP connection was used, as well as the ObjectReader for JSON parsing. These tests were performed on the USF WiFi network. Results from SpeedTest.NET Android app at time of test were 52176 kbps down, 78721 kbps up, and ping of 15 ms.
30 requests were performed using the MTA BusTime SIRI StopMonitoring API, taking steps outlined in the methodology below to reset to a cold start state between each cold start test, and to the pseudo-warm start state for the pseudo warm start tests.
An overview of the process of testing -
A) A cold start response time is measured,
B) We turn on caching in the app and make another request so the app caches the Jackson object used to make requests to persistent memory - we measure the cache write time as well as cached Jackson object size,
C) We manually remove the Android cached process from memory via the Application Manager to reset to a cold start state,
D) A pseudo-warm start response time is measured (=Time to read cached object + elapsed response time).
E) We turn off caching in the app, and we manually remove the Android cached process from memory via the Application Manager to reset to a cold start state, and then repeat again starting at Step A.
The detailed steps of the methodology we used can be found on the Detailed Test Methodology - Cold Start Benchmarks page, if you'd like to replicate these tests yourself.
XML - Cold vs. Pseudo-warm starts
First, we will look at the performance of cold vs. pseudo-warm starts when using XML to format the server response. Figure 4 shows the elapsed time of 30 cold and pseudo-warm start tests, and Figure 5 shows the summary of the results from these tests.
Figure 4 - Cold vs. Pseudo-Warm Start performance for XML responses from 30 requests
Figure 5 - Cold vs. Pseudo-Warm Start performance for XML responses - Summary
Because of the large cold start penalties, we are able to substantially improve the performance of XML parsing via pseudo-warm starts and caching. Cold starts had an average elapsed time of 17.7 seconds, and pseudo-warm starts had an average of only 9.7 seconds, an 8 second improvement. Overall, pseudo-warm starts thereby yield an average of a 44% performance increase over cold starts, a dramatic improvement.
JSON - Cold vs. Pseudo-warm starts
Next, we will look at the performance of cold vs. pseudo-warm starts when using JSON to format the server response. Figure 7 shows the elapsed time of 30 cold and pseudo-warm start tests, and Figure 8 shows the summary of the results from these tests.
Figure 7 - Cold vs. Pseudo-Warm Start performance for JSON responses from 30 requests
Figure 8 - Cold vs. Pseudo-Warm Start performance for JSON responses - Summary Results
Pseudo-warm starts perform slightly better than cold starts, with an average elapsed time of 3,785ms vs. the cold start average of 3,963ms - a difference of 178ms on average, which may still be noticeable to the user, but is not nearly as dramatic as the improvement of the XML pseudo-warm start.
Overall, for JSON the pseudo-warm start amounted to an average 3.96% performance increase over the cold start.
The dramatic improvement in XML parsing performance using pseudo-warm starts indicates that the time required to read a cached Jackson object from mobile device persistent memory is far less than the time required to newly instantiate that same object. Therefore, a huge 44% performance increase on average can be produced by using pseudo-warm starts.
JSON pseudo-warm start performance improvements (3.96% on average), however, are not nearly as impressive as their XML counterparts. This difference is partially due to the fact that XML cold starts are significantly larger than JSON cold starts to begin with (by a factor of 4), which gives the XML pseudo-warm start a much greater margin of potential improvement.
After reviewing these results, one may think that a pseudo-warm start implementation for JSON parsing isn't worth the effort. However, we haven't yet discussed a significant advantage of pseudo-warm starts over cold starts - the potential to hide part of the delay from the user by beginning the cache read when the app is first started.
Figure 9 shows how user interactions occur in context of A) cold starts compared with B) pseudo-warm starts.
Figure 9 - With A) cold starts, the user always observes the fully overhead delay, but with B) pseudo-warm starts the overhead delay can be hidden from the user
As mentioned earlier, the cache read used in pseudo-warm starts can be initiated immediately upon application start up and can execute in the background while the user is browsing within the application. Therefore, using pseudo-warm starts, a considerable amount of the overhead time may pass before the user actually triggers a request to the server (e.g., for real-time arrival data). In the best-case scenario, shown in Figure 9B i, the entire cache read finishes prior to the user initiating a request, and the user does not observe any delay. In this case, pseudo-warm starts are equivalent to warm starts in performance.
An alternate pseudo-warm start scenario is shown in Figure 9B ii. Here, the user performs a few UI actions while the cache read starts, but initiates the request to the server before the cache read can complete. In this situation, the user would observe a partial delay, depending on how much time has elapsed and how long the cache read will take to complete.
Cold starts, on the other hand, cannot hide any of this overhead wait time from the user - Jackson initialization must always start when the user initiates a server request, as shown in Figure 9A.
When hiding the cache read time from the user, the seemingly small improvements in JSON pseudo-warm start performance suddenly become quite large. Figure 10 shows the differences between pseudo-warm starts and cold starts if the user spends the entire cache read time browsing through the app.
Figure 10 - If user interactions with the app hide the cache read latency, JSON pseudo-warm starts perform significantly better than cold starts
Assuming that this cache read time is hidden from the user, the average JSON pseudo-warm parsing start elapsed time is 469ms, compared to nearly 4 seconds of cold start time. The average cache read time in this test was 3.3 seconds, meaning the user would need to spend 3.3 seconds in the the app before requesting real-time transit information for the cache read to remain completely hidden. Given the initial app start-up process and other activity of the user before they may retrieve real-time transit information, the expectation that at least some of the cache read will remain hidden seems reasonable.
Revisiting XML pseudo-warm starts while considering the potential to hide the cache read yields even greater performance increases over cold starts. Figure 11 shows the difference between pseudo-warm starts and cold starts when the cache read is completely hidden.
Figure 11 - XML pseudo-warm start performance increases are even more significant if the cache read time can be hidden from the user
Average pseudo-warm starts improve to 473ms, versus average cold starts of 17.7 seconds. However, one must also consider that to fully hide the cache read for XML parsing, the user would need to spend an average of 9.3 seconds in the app before triggering a request to the server. This is a much longer amount of time than that required for JSON, and therefore less of the XML cache read time will be hidden from the user on average. The longer cache read time for XML when compared to JSON may be explained by the larger cache file size for the respective Jackson objects (XmlMapper.cache = ~1,200kB, and ObjectReader = ~260kB).
One drawback to the caching strategy of pseudo-warm starts over cold starts is the persistent memory consumed by the cache file. However, the sizes of 1,200kB and 260kB for XML and JSON cache files, respectively, should be negligible for most smart phones.
It should be noted that the strategy of pseudo-warm starts is just one potential avenue for improving user experience in mobile applications that use real-time information from a server. Our focus here is on reducing the amount of time necessary to retrieve new real-time information from a server, when an update needs to be retrieved. Response caches, available on Android 4.0 and up, can be used to avoid retrieving a full response to the mobile device when no change in the data has occurred since the last request. Compression can also be used to reduce the size of HTTP data transfers (although one must also be careful to take into account the computational costs of device-side decompression). The refresh interval on the device can also be adjusted to avoid querying the server too frequently. For example, if the real-time information isn't updated server-side any more frequently than once per minute, there isn't much of a reason to query the server every 10 seconds for new real-time information.
Another strategy for hiding the latency for cold starts from the user is to initiate a "dummy" read of a small bit of data on startup (e.g., { }). This dummy read will force initialization of many of the internal Jackson data structures used for deserialization. While the dummy read isn't expected to increase the general initialization performance, it does offer the advantage of being able to hide some of the latency from the user vs. a normal cold start. In this paper we compare the cold start initialization against the cache read pseudo-warm start technique as the cache read as it is a different method that did yield an actual performance increase, in addition to hiding the latency from the user. Future work could compare the performance of dummy reads against cache read times to quantify the differences.
Tests discussed in this article used Jackson v2.1.2. Newer versions of Jackson are now available with additional performance improvements, and these new versions should be used in future benchmarking tests.
When using real-time transit information services, mobile app users must often wait for their phone to retrieve the latest real-time arrival information from a server. This article focused on performance evaluations of different data formats and their effect on the elapsed time required to retrieve updated information from a server using the Jackson JSON and XML Processor.
In general, JSON is a preferred data transfer format over XML for mobile devices. Average performance for cold starts (i.e., when the user first starts the mobile app) was over 4 times faster for JSON than XML, with an average time difference of 14 seconds. JSON also had a noticeable performance advantage in warm starts too, being an average of 224ms faster than XML.
We also examined an optimization strategy, pseudo-warm starts, aimed at reducing the large performance penalty of cold starts. Pseudo-warm starts, which use cached Jackson objects instead of re-initializing the objects on app startup, provide a dramatic increase in performance for XML parsing, reducing elapsed parsing time from an average of 17.7 seconds to an average of only 9.7 seconds, an 8 second improvement. Pseudo-warm start improvements for JSON were more modest, with an average elapsed time of 3,785ms vs. the cold start average of 3,963ms - a difference of 178ms on average.
Finally, we discuss the potential impact of pseudo-warm starts vs. cold starts in the context of user-observable performance. Since the cache read used in pseudo-warm starts can be initiated on application startup, unlike the cold start initialization overhead which must being upon user-triggered requests to the server, the pseudo-warm start provides the opportunity to hide a portion of the overhead wait time from the user. In other words, while the user is initially browsing through the app, the cache read can be executing in parallel in the background. Assuming that the entire cache read time is hidden from the user, the average JSON pseudo-warm parsing start elapsed time improves to 469ms, compared to nearly 4 seconds of cold start time. Similarly, assuming a hidden cache read, average XML pseudo-warm starts further improve to 473ms, versus average cold starts of 17.7 seconds. However, given that XML cache reads take an average of 9.3 seconds, versus average JSON cache read of 3.3 seconds, it may not be realistic to completely hide XML cache reads from the user.
A big thanks to Tatu Saloranta from FasterXML, LLC for his work on various Jackson issues, as well as implementing the ability to (de)serialize Jackson objects, which made much of this work possible. The research described in this paper was funded by the National Center for Transit Research.
[1] R. Jota, A. Ng, P. Dietz, and D. Wigdor. "How Fast is Fast Enough? A Study of the Effects of Latency in Direct-Touch Pointing Tasks," CHI 2013, Paris, France. April 27 - May 2, 2013.