Technical Overview Of Pinpoint
Pinpoint is a platform that analyzes large-scale distributed systems and provides a solution to handle large collections of trace data. It has been developed since July 2012 and was launched as an open-source project on January 9, 2015.
This article introduces Pinpoint; it describes what motivates us to start it up, which technologies are used, and how the Pinpoint agent can be optimized.
Compared to nowadays, the number of Internet users was relatively small and the architecture of Internet services was less complex. Web services were generally configured using a 2-tier (web server and database) or 3-tier (web server, application server, and database) architecture. However, today, supporting a large number of concurrent connections is required and functionalities and services should be organically integrated as the Internet has grown, resulting in much more complex combinations of software stack. That is, n-tier architecture more than 3 tiers has become more widespread. A service-oriented architecture (SOA) or the microservices architecture is now a reality.
The system's complexity has consequently increased. The more complex it is, the more difficult you solve problems such as system failure or performance issues. It is not that hard to find a solution in such a 3-tier architecture. You are required to analyze only 3 components such as web server, application server, and database and a number of servers are small. While, if a problem occurs in an n-tier architecture, a large number of components and servers should be investigated. Another problem is that it is difficult to see the big picture only with an analysis of individual components; a low visibility issue is raised. The higher the degree of system complexity is, the longer it takes time to find out the reasons. Even worse, there increase situations in which we may not even find them out.
Such problems had occurred in the systems at NAVER. A variety of tools like Application Performance Management (APM) had been used but they were not enough to handle the problems effectively; so we finally ended up developing a new tracing platform for n-tier architecture, which can provide solutions to systems with an n-tier architecture.
Pinpoint, which has been developed since July 2012 and was launched as an open-source project on January 2015, is an n-tier architecture tracing platform for large-scale distributed systems. The characteristics of Pinpoint are as follows:
- Distributed transaction tracing to trace messages across distributed applications
- Automatically detecting the application topology that helps you to figure out the configurations of an application
- Horizontal scalability to support large-scale server group
- Providing code-level visibility to easily identify points of failure and bottlenecks
- Adding a new functionality without code modifications, using the bytecode instrumentation technique
In this article, we describe the Pinpoint's techniques such as transaction tracing and bytecode instrumentation. And we explain the optimization method applied in the Pinpoint agent, which modifies bytecode and records performance data.
Pinpoint traces distributed requests in a single transaction, modelled on Google's Dapper.
The core of a distributed tracing system is to identify relationships between messages processed at Node 1 and messages processed at Node 2 in a distributed system when a message is sent from Node 1 to Node 2 (Figure 1).
Figure 1. Message relationship in a distributed system
The problem is that there is no way to identify relationships between messages. For example, we cannot recognize relationships between N messages sent from Node 1 and N' messages received in Node 2. In other words, when a X-th message is sent from Node 1, the X-th message cannot be identified among N' messages received in Node 2. An attempt was made to trace messages at TCP or operating system level. However, implementation complexity was high with low performance because it should be implemented separately for each protocol. In addition, it was difficult to accurately trace messages.
However, a simple solution to resolve such issues has been implemented in Google's Dapper. The solution is to add application-level tags that can be a link between messages when sending a message. For example, it includes tag information for a message in the HTTP header at an HTTP request and traces the message using this tag.
Google's Dapper
For more information on Google's Dapper, see "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure."
Pinpoint is modelled on the tracing technique of Google's Dapper but has been modified to add application-level tag data in the call header so as to trace distributed transactions at a remote call. The tag data consists of a collection of keys, which is defined as a TraceId.
In Pinpoint, the core of a data structure consists of Spans, Traces, and TraceIds.
- Span: The basic unit of RPC (remote procedure call) tracing; it indicates work processed when an RPC arrives and contains trace data. To ensure the code-level visibility, a Span has children labeled SpanEvent as a data structure. Each Span contains a TraceId.
- Trace: A collection of Spans; it consists of associated RPCs (Spans). Spans in the same trace share the same TransactionId. A Trace is sorted as a hierarchical tree structure through SpanIds and ParentSpanIds.
- TraceId: A collections of keys consisting of TransactionId, SpanId, and ParentSpanId. The TransactionId indicates the message ID, and both the SpanId and the ParentSpanId represent the parent-child relationship of RPCs.
- TransactionId (TxId): The ID of the message sent/received across distributed systems from a single transaction; it must be globally unique across the entire group of servers.
- SpanId: The ID of a job processed when receiving RPC messages; it is generated when a RPC arrives at a node.
- ParentSpanId (pSpanId): The SpanId of the parent span which generated the RPC. If a node is the starting point of a transaction, there will not be a parent span - for these cases, we use a value of -1 to denote that the span is the root span of a transaction.
Differences in terms between Google's Dapper and NAVER's Pinpoint
The term "TransactionId" in Pinpoint has the same meaning as the term "TraceId" in Google's Dapper and the term "TraceId" in Pinpoint refers to a collection of keys.
The figure below illustrates the behavior of a TraceId in which RPCs were made 3 times within 4 nodes.
Figure 2. Example of a TraceId behavior
A TransactionId (TxId) represents that three different RPCs are associated with each other as a single transaction in Figure 2. However, a TransactionId itself cannot describe explicitly the relationships between RPCs. To identify the relationships between RPCs, a SpanId and a ParentSpanId (pSpanId) are required. Suppose that a node is Tomcat. You can think of a SpanId as a thread which handles HTTP requests. A ParentSpanId indicates the SpainId of a parent that makes RPC calls.
Pinpoint can find associated n Spans using a TransactionId and can sort the n Spans as a hierarchical tree structure using a SpanId and a ParentSpanId.
A SpanId and a ParentSpanId are 64-bit long integers. Conflict might arise because the number is generated arbitrarily, but this is unlikely to happen, considering the value can range from -9223372036854775808 to 9223372036854775807. If there is a conflict between keys, Pinpoint as well as Google's Dapper lets developers know what happens, instead of resolving the conflict.
A TransactionId consists of AgentIds, JVM (Java virtual machine) startup time, and SequenceNumbers.
- AgentId: A user-created ID when JVM starts; it must be globally unique across the entire group of servers where Pinpoint has been installed. The easiest way to make it unique is to use a hostname ($HOSTNAME) because the hostname is not duplicate in general. If you need to run multiple JVMs within the server group, add a postfix to the hostname to avoid duplicates.
- JVM startup time: Required to guarantee a unique SequenceNumber which starts with zero. This value is useful to prevent ID conflicts in case when a user created a duplicate AgentId by mistake.
- SequenceNumber: ID issued by the Pinpoint agent, with sequentially increasing numbers that start with zero; it is issued per message.
Dapper and Zipkin, a distributed systems tracing platform at Twitter, generate random TraceIds (TransactionIds in Pinpoint) and consider conflict situations as normal. However, we wanted to avoid the possibility of conflict in Pinpoint so implemented the systems as described above. There were two options available; one is a method in which the amount of data is small but the probability of conflict is high; the other is a method in which the amount of data is large but the probability of conflict is low; the second option was one chosen.
There may be a better way to handle transactions. We came up with ideas in which keys are issued by a central key server. If this model was implemented, it could have caused performance issues and network errors. Therefore, issuing keys in bulk was considered as an alternative. Later such methods may be developed; for now, a simple method is adopted. In Pinpoint, a TransactionId is regarded as changeable data.
As earlier, we explain distributed transaction tracing. One way for implementing this is that developers modify code by themselves. Allow developers to add tag information when an RPC is made. However, it could be a burden to modify code even though such functionality is useful to developers.
Twitter's Zipkin provides the functionality of distributed transaction tracing using modified libraries and its container (Finagle). However, it requires that code be modified if needed. We wanted the functionality to work without code modifications and desired to ensure code-level visibility. To solve such problems, the bytecode instrumentation technique was adopted in Pinpoint. The Pinpoint agent intervenes code to make RPCs so as to automatically handle tag information.
Bytecode instrumentation belongs to an automatic method between manual and automatic methods.
- Manual method: Developers develop code that records data at important points using APIs provided by Pinpoint.
- Automatic method: Developers do not involve code modifications because Pinpoint decides which API is to be intervened and developed.
Advantages and disadvantages of each method are as follows:
Table 1 Advantages and disadvantage of each method
|Advantage |Disadvantage
---------|----------|------------
Manual Tracing| - Requires less development resources.
- An API can become simpler and consequently the number of bugs can be reduced. |- Developers must modify code.
- Tracing level is low.
Automatic Tracing|- Developers are not required to modify code.
- More precise data can be collected because of much information in bytecode.|- Would cost 10 times more to adopt an automatic method than to adopt a manual method in developing Pinpoint.
- Requires highly competent developers who can instantly recognize the library code to be traced and make decisions on the tracing points.
- Can increase the likelihood of a bug occurring because high level development development skill like bytecode instrumentation is used.
Bytecode instrumentation is a technique in which levels of difficulty and risks are high. However, using this technique has many benefits, considering development resources and difficulty levels.
Although it requires a large number of development resources, it requires few resources in developing services. For example, the following shows the costs between an automatic method which uses bytecode instrumentation and a manual method which uses libraries (in this context, costs are random numbers assumed for clarity).
- Automatic method: Total of 100
- Cost of Pinpoint development: 100
- Cost of services applied: 0
- Manual method: Total of 30
- Cost of Pinpoint development: 20
- Cost of services applied: 10
The data above tells us that a manual method is more cost-effective than an automatic one. However, it is not applied in our environment at NAVER. We have thousands of services at NAVER so it needs to modify the cost of service applied in the data above. If we have 10 services which require to be modified, the total cost can be calculated as follows:
- Cost of Pinpoint development 20 + Cost of services applied 10 x 10 = 120
As a result of that, an automatic method is a more cost-efficient way.
We are lucky to have many developers who are highly competent and specialized in Java in the Pinpoint team. Therefore, we believe it is only a matter of time to overcome the technical difficulties in Pinpoint development.
The reason we chose bytecode instrumentation is that it has the following strong points, besides those described earlier.
Hidden API
If an API is exposed so that developers can use, we, as API providers, are restricted to modify an API as we desire. Such a restriction can impose stress on us.
We may modify an API to correct wrong design or add new functionalities. However, if there is a restriction to do them, it is difficult for us to improve an API. The best answer for solving such a problem is a scalable system design, which everybody knows this is not an easy option. It is almost impossible to create perfect API design as we can't predict the future.
As with bytecode instrumentation, we don't have to worry about exposing the tracing APIs and can continuously improve design, without considering dependency relationship. For those who are planning to develop their applications using Pinpoint, in other words, it means an API is changeable by the Pinpoint developers. For now, we will maintain the idea hiding an API because improving performance and design is our first priority.
The disadvantage of using bytecode instrumentation is that it could affect your applications when a problem occurs in the profiling code of a library or Pinpoint itself. However, you can solve it by enabling or disabling Pinpoint with easy because you don't have to change code.
You can easily enable Pinpoint to your applications by adding the three lines (associated with the configuration of the Pinpoint agent) below into your JVM startup script.
- javaagent:$AGENT_PATH/pinpoint-bootstrap-$VERSION.jar
- Dpinpoint.agentId=<Agent's UniqueId>
- Dpinpoint.applicationName=<The name indicating a same service (AgentId collection)>
If a problem occurs due to Pinpoint, you just delete the configuration data in the JVM startup script.
Since the bytecode instrumentation technique deals with Java bytecode, it tends to increase the risk for development while it decreases productivity with this technique. In addition, developers are prone to make mistakes. In Pinpoint, we made it abstract in an interceptor to improve productivity and accessibility. Pinpoint injects the tracing code necessary for distributed transactions and performance information by intervening application code at class loading time, which results in increasing performance as code injection is carried out in the application code directly.
Figure 3. Behavior of bytecode instrumentation
In Pinpoint, the interceptor APIs are separated from where performance data is recorded. We added the interceptor to a target method for tracing to make the before () and after () methods called and implemented a part of recording performance data in the before() and after() methods. With bytecode instrumentation, the Pinpoint agent can record the data of necessary method only so the size of profiling data is small.
Finally, we describe the ways used to optimize performance of the Pinpoint agent.
You can increase encoding speed by using a binary format (Thrift) although it is difficult to use and debug. It can also be beneficial to reduce network usage because the size of data generated is small.
If you convert a long integer into a fixed-length string, the data size is usually 8 bytes. However, if you use variable-length encoding, the data size can vary from 1 to 10 bytes, depending on the size of a given number. To reduce data size, Pinpoint uses the CompactProtocol protocol of Thrift to encode data as a variable-length string and records data that can be optimized for the encoding format. The Pinpoint agent can reduce data size by converting the rest of the time in a vector based on the traced root method.
Variable-length encoding
For more information on the variable-length encoding, see "Base 128 Varints" in Google Developers.
Figure 4 illustrates the idea described in the paragraph above.
Figure 4. Comparison between fixed-length encoding and variable-length encoding
You have to measure time in different 6 points to get data about when three different methods are called (Figure 4); it requires 48 bytes (6 × 8) with fixed-length encoding.
In the meantime, the Pinpoint agent uses variable-length encoding and records data according to its corresponding format. And it calculates a value (in a vector) of time at another point by comparing it with a reference point; the reference point is determined by the start time of a root method. It consumes a small number of bytes because a vector takes small numbers; thirteen bytes are consumed in Figure 4.
If it takes more time to execute a method, it will increase the number of bytes even though variable-length encoding is used. However, it is still more efficient than fixed-length encoding.
We wanted Pinpoint to enable code-level tracing. However, it has a problem in terms of increasing data size. Every time data with a high degree of precision is sent to a server, it will increase the size of the data; it results in increasing network usage.
To solve such a problem, we adopted a strategy to create a constant table in a remote HBase server. For example, every time information called "method A" is sent to Pinpoint Collector, the size of the data will be large; the Pinpoint agent replaces the "method A" with an ID, stores information on the ID and the "method A" as a constant table in HBase, and generates trace data with the ID. And when a user retrieves trace data on the Website, the Pinpoint web searches for the method information of the corresponding ID in the constant table and combines them. The same way is used to reduce data size in SQLs or frequently-used strings.
The requests to our online portal services are huge. A single service handles more than 20 billion requests a day. It is easy to trace such requests; the way is that adding network infrastructure and servers as much as the number of request to be traced and expanding servers to collect data. However, it is not a cost-effective way to handle such situations; it just wastes money and resources.
In Pinpoint, you can collect sampling data without tracing every request. In a development environment where request is small, every data is collected; while in a product environment where request is large, data with small rate as much as 1~5% is collected, which is enough for checking the status of entire applications. With sampling, you can minimize network overhead in applications and reduce costs of infrastructure such as networks and servers.
Sampling method in Pinpoint
Pinpoint supports a Counting sampler, which collects data only for one of 10 requests if it is set to 10. We plan to add new samplers that can collect data more effectively.
Pinpoint does not block application threads because encoded data or remote messages are transferred asynchronously by another thread.
Unlike Google's Dapper, Pinpoint transfers data through a network to ensure the data speed. As a common infrastructure used along with services, a network can be an issue when data traffic becomes increasingly bursty. In such a situation, the Pinpoint agent uses the UDP protocol to give the network connection priority to services.
Note
The data transfer APIs can be replaced because it is separated as an interface. It can be changed into an implementation that stores data in a different way, like local files.
We give you an example of how to get data from applications so that you can comprehensively understand the content described earlier.
Figure 5 shows data when you install Pinpoint in TomcatA and TomcatB. You can see the trace data of an individual node as a single traction, which represents the flow of distributed transaction tracing.
Figure 5. Example 1: Pinpoint applied
The following describes what Pinpoint does for each method.
- When a request arrives at TomcatA, the Pinpoint agent issues a TraceId.
- TX_ID: TomcatA^TIME^1
- SpanId: 10
- ParentSpanId: -1(Root)
- Records data from Spring MVC controllers.
- Intervenes the calls of the HttpClient.execute() method and configures the TraceId in HttpGet.
a. Creates a child TraceId.
- TX_ID: TomcatA^TIME^1 -> TomcatA^TIME^1
- SPAN_ID: 10 -> 20
- PARENT_SPAN_ID: -1 -> 10 (parent SpanId)
b. Configures the child TraceId in the HTTP header.
- HttpGet.setHeader(PINPOINT_TX_ID, "TomcatA^TIME^1")
- HttpGet.setHeader(PINPOINT_SPAN_ID, "20")
- HttpGet.setHeader(PINPOINT_PARENT_SPAN_ID, "10")
- Transfers the tagged request to TomcatB.
a. TomcatB checks the header of a transferred request.
- HttpServletRequest.getHeader(PINPOINT_TX_ID)
b. TomcatB behaves like a child node as it identifies a TraceId in the header.
- TX_ID: TomcatA^TIME^1
- SPAN_ID: 20
- PARENT_SPAN_ID: 10
- Records data from Spring MVC controllers and completes the request.
Figure 6. Example 2: Pinpoint applied
-
When the request from TomcatB is complete, the Pinpoint agent sends trace data to Pinpoint Collector so as to store it in HBase.
-
After the HTTP calls from TomcatB is terminated, the request from TomcatA is complete. The Pinpoint agent sends trace data to Pinpoint Collector so as to store it in HBase.
-
UI reads the trace data from HBase and creates a call stack by sorting trees.
Pinpoint is another application that runs along with your applications. Using bytecode instrumentation makes Pinpoint seem like that it does not require code modifications. In general, the bytecode instrumentation technique makes applications vulnerable to risks; if a problem occurs in Pinpoint, it will affect your applications as well. For now, we are focusing on improving performance and design of Pinpoint, instead of getting rid of such threats, because we think such things make Pinpoint more valuable. So it's your decision on whether or not to use Pinpoint.
We still have a large amount of work to be done to improve Pinpoint. Despite of its incompleteness, Pinpoint was released as an open-source project; we are continuously trying to develop and improve Pinpoint so as to meet your expectations.
Written by Woonduk Kang
In 2011, I wrote about myself like this—As a developer, I would like to make a software program that users are willing to pay for, like those of Microsoft or Oracle. As Pinpoint was launched as an open-source project, it seems that my dreams somewhat come true. For now, my desire is to make Pinpoint more valuable and attractive to users.