Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Golang] Not able to store tracker in integration #5171

Closed
niraj-elastic opened this issue Feb 2, 2023 · 1 comment
Closed

[Golang] Not able to store tracker in integration #5171

niraj-elastic opened this issue Feb 2, 2023 · 1 comment

Comments

@niraj-elastic
Copy link
Contributor

Problem statement

We are developing Golang integration. Metrics of golang are exposed on the http endpoint in JSON format. We are fetching 'PauseNs' metric which is an array. Each value of this array shows time spent by a single garbage collection process. Initially, all the values in this array are “0”.

"PauseNs": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]

As garbage collection gets executed these “0” are overwritten by the time it took for garbage collection. For example, if garbage collection ran 5 times, the array would represent the time of all 5 garbage collections as follows.

"PauseNs": [
98323,
53765,
43486,
123122,
86370,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]

We want to derive some fields by processing 'PauseNs'. For example, we need to calculate the average of all the newly added values in ‘PauseNs’ since the last document was ingested, I.E. If the average of 1st to 5th value is calculated while indexing document and after some time if 5 more garbage collections are performed (total count 10) then while indexing a new document, average from 6th to 10th values should be calculated only. For that, we need to keep track of how many values of ‘PauseNs’ are already processed. Currently, there is no cleaner way to achieve this.

"PauseNs": [
98323,
53765,
43486,
123122,
86370,
122549,
122086,
43954,
62701,
53028,
0,
0,
0,
0,
0
]

We thought of storing the index of the last processed ‘PauseNs’ in the cursor of httpjson input. But, the Problem with developing that approach is that index can be of ‘int64’. If we try to store a large value, the cursor changes the format of the numeric value to exponential notation format which we can not use further.
For example:- If the index is “1234567”, it is automatically converted to exponential notation format “1.234567e+06” by a cursor.

Following is our configuration:-

filebeat.inputs:
  - type: httpjson
    enabled: true
    request.url: http://<host>:<port>/<endpoint>
    interval: 10s
    response.transforms:
      - set:
          target: body.last_index
          value: '[[.cursor.last_index]]'
          default: "0"
    cursor:
      last_index:
        value: '[[.last_response.body.index]]'

Related PR

PR

@botelastic
Copy link

botelastic bot commented Feb 2, 2024

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Feb 2, 2024
@botelastic botelastic bot closed this as completed Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant