Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[O11y][GoLang] Resolve the conflict in host.ip field #7494

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions packages/golang/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,101 @@ You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommen

For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.

### Troubleshooting

Conflicts in any field in any data stream can be solved by reindexing the data.
If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Heap`` and ``Expvar`` data stream's indices.
To reindex the data, the following steps must be performed.

1. Stop the data stream by going to `Integrations -> Golang -> Integration policies` open the configuration of Golang and disable the `Collect Golang metrics` toggle to reindex metrics data stream and save the integration.

2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.

```
POST _reindex
{
"source": {
"index": "<index_name>"
},
"dest": {
"index": "temp_index"
}
}
```
Example:
```
POST _reindex
{
"source": {
"index": "logs-golang.heap-default"
},
"dest": {
"index": "temp_index"
}
}
```

```
DELETE /_data_stream/<data_stream>
```
Example:
```
DELETE /_data_stream/logs-golang.heap-default
```

```
DELETE _index_template/<index_template>
```
Example:
```
DELETE _index_template/logs-golang.heap
```
3. Go to `Integrations -> Golang -> Settings` and click on `Reinstall Golang`.

4. Copy data from temporary index to new index by performing the following steps in the Dev tools.

```
POST _reindex
{
"conflicts": "proceed",
"source": {
"index": "temp_index"
},
"dest": {
"index": "<index_name>",
"op_type": "create"

}
}
```
Example:
```
POST _reindex
{
"conflicts": "proceed",
"source": {
"index": "temp_index"
},
"dest": {
"index": "logs-golang.heap-default",
"op_type": "create"

}
}
```

5. Verify data is reindexed completely.

6. Start the data stream by going to the `Integrations -> Golang -> Integration policies` and open configuration of integration and enable the `Collect Golang metrics` toggle and save the integration.

7. Delete temporary index by performing the following step in the Dev tools.

```
DELETE temp_index
```

More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).

## Logs reference

### expvar
Expand Down
5 changes: 5 additions & 0 deletions packages/golang/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.4.1"
changes:
- description: Resolve the conflict in host.ip field.
type: bugfix
link: https://github.com/elastic/integrations/pull/7494
- version: "0.4.0"
changes:
- description: Rename ownership from obs-service-integrations to obs-infraobs-integrations
Expand Down
2 changes: 2 additions & 0 deletions packages/golang/data_stream/expvar/fields/ecs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,7 @@
name: event.original
- external: ecs
name: event.type
- external: ecs
name: host.ip
- external: ecs
name: service.address
2 changes: 2 additions & 0 deletions packages/golang/data_stream/heap/fields/ecs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,7 @@
name: event.original
- external: ecs
name: event.type
- external: ecs
name: host.ip
- external: ecs
name: service.address
97 changes: 97 additions & 0 deletions packages/golang/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,101 @@ You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommen

For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.

### Troubleshooting

Conflicts in any field in any data stream can be solved by reindexing the data.
If host.ip is shown conflicted under ``logs-*`` data view, then this issue can be solved by reindexing the ``Heap`` and ``Expvar`` data stream's indices.
To reindex the data, the following steps must be performed.

1. Stop the data stream by going to `Integrations -> Golang -> Integration policies` open the configuration of Golang and disable the `Collect Golang metrics` toggle to reindex metrics data stream and save the integration.

2. Copy data into the temporary index and delete the existing data stream and index template by performing the following steps in the Dev tools.

```
POST _reindex
{
"source": {
"index": "<index_name>"
},
"dest": {
"index": "temp_index"
}
}
```
Example:
```
POST _reindex
{
"source": {
"index": "logs-golang.heap-default"
},
"dest": {
"index": "temp_index"
}
}
```

```
DELETE /_data_stream/<data_stream>
```
Example:
```
DELETE /_data_stream/logs-golang.heap-default
```

```
DELETE _index_template/<index_template>
```
Example:
```
DELETE _index_template/logs-golang.heap
```
3. Go to `Integrations -> Golang -> Settings` and click on `Reinstall Golang`.

4. Copy data from temporary index to new index by performing the following steps in the Dev tools.

```
POST _reindex
{
"conflicts": "proceed",
"source": {
"index": "temp_index"
},
"dest": {
"index": "<index_name>",
"op_type": "create"

}
}
```
Example:
```
POST _reindex
{
"conflicts": "proceed",
"source": {
"index": "temp_index"
},
"dest": {
"index": "logs-golang.heap-default",
"op_type": "create"

}
}
```

5. Verify data is reindexed completely.

6. Start the data stream by going to the `Integrations -> Golang -> Integration policies` and open configuration of integration and enable the `Collect Golang metrics` toggle and save the integration.

7. Delete temporary index by performing the following step in the Dev tools.

```
DELETE temp_index
```

More details about reindexing can be found [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).

## Logs reference

### expvar
Expand Down Expand Up @@ -174,6 +269,7 @@ An example event for `expvar` looks as following:
| golang.expvar.obtained.total.bytes | The total bytes of memory obtained from the OS. | long | byte | gauge |
| golang.expvar.pointer.lookups | The number of pointer lookups performed by the runtime. | long | | gauge |
| golang.expvar.stack.bytes | Bytes in stack spans. | long | byte | gauge |
| host.ip | Host ip addresses. | ip | | |
| input.type | Type of Filebeat input. | keyword | | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
| tags | List of keywords used to tag each event. | keyword | | |
Expand Down Expand Up @@ -318,6 +414,7 @@ An example event for `heap` looks as following:
| golang.heap.system.released.bytes | Bytes of physical memory returned to the OS. | long | byte | gauge |
| golang.heap.system.stack.bytes | Bytes of stack memory obtained from the OS. | long | byte | gauge |
| golang.heap.system.total.bytes | Bytes of heap memory obtained from the OS. | long | byte | gauge |
| host.ip | Host ip addresses. | ip | | |
| input.type | Type of Filebeat input. | keyword | | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
| tags | List of keywords used to tag each event. | keyword | | |
Expand Down
2 changes: 1 addition & 1 deletion packages/golang/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 2.0.0
name: golang
title: Golang
version: "0.4.0"
version: "0.4.1"
description: This Elastic integration collects metrics from Golang applications.
type: integration
categories:
Expand Down