Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove index management, always use data streams #6606

Merged
merged 6 commits into from
Nov 16, 2021
Merged

Remove index management, always use data streams #6606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7b822d6
Remove index management, always use data streams
axw Oct 27, 2021
00cfa0f
Update notice
axw Nov 9, 2021
498b68d
Merge branch 'master' into no-index-management
axw Nov 15, 2021
f988600
Update README
axw Nov 15, 2021
a91ba22
Merge branch 'master' into no-index-management
axw Nov 16, 2021
ae0b3bd
Fix merge
axw Nov 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,21 +79,13 @@ bench:
@$(GO) test -benchmem -run=XXX -benchtime=100ms -bench='.*' ./...

##############################################################################
# Rules for updating config files, fields.yml, etc.
# Rules for updating config files, etc.
##############################################################################

update: fields go-generate add-headers build-package notice $(MAGE)
update: go-generate add-headers build-package notice $(MAGE)
@$(MAGE) update
@go mod download all # make sure go.sum is complete

fields_sources=\
$(shell find model -name fields.yml) \
$(shell find x-pack/apm-server/fields -name fields.yml)

fields: include/fields.go x-pack/apm-server/include/fields.go
include/fields.go x-pack/apm-server/include/fields.go: $(MAGE) magefile.go $(fields_sources)
@$(MAGE) fields

config: apm-server.yml apm-server.docker.yml
apm-server.yml apm-server.docker.yml: $(MAGE) magefile.go _meta/beat.yml
@$(MAGE) config
Expand Down
6,646 changes: 1,618 additions & 5,028 deletions NOTICE.txt
View file
Open in desktop

Large diffs are not rendered by default.

39 changes: 5 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ in the same directory with the name apm-server.
make
```

You also need to create all files needed by the APM Server by running the additional command below.
If you make code changes, you may also need to update the project by running the additional command below:

```
make update
Expand All @@ -60,51 +60,22 @@ To run APM Server with debugging output enabled, run:
./apm-server -c apm-server.yml -e -d "*"
```

APM Server expects index templates, ILM policies, and ingest pipelines to be set up externally.
This should be done by [installing the APM integration](https://www.elastic.co/guide/en/fleet/current/fleet-quick-start-traces.html#add-apm-integration).
When running APM Server directly, it is only necessary to install the integration and not to run an Elastic Agent.

### Testing

For Testing check out the [testing guide](TESTING.md)

### Update
simitt marked this conversation as resolved.
Show resolved Hide resolved

Each beat has a template for the mapping in elasticsearch and a documentation for the fields
which is automatically generated based on `fields.yml`.
To generate required configuration files and templates run:

```
make update
```

### Generate package

APM-Server includes a script to generate an integration package to run with Fleet.
To Generate a package run:

```
make fields gen-package
```

That command takes the existing `fields.yml` files and split them into `ecs.yml` and `fields.yml` files for each data stream type.
It also generates a `README.md` with a field reference that will be shown in the integration package.

After generating a package, `apmpackage/apm` should be manually copied to `elastic/integrations`.
Then follow instructions in https://github.com/elastic/integrations/blob/master/CONTRIBUTING.md.

### Cleanup

To clean APM Server source code, run the following commands:

```
make fmt
```

To clean up the build directory and generated artifacts, run:

```
make clean
```

For further development, check out the [beat developer guide](https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html).

### Contributing

See [contributing](CONTRIBUTING.md) for details about reporting bugs, requesting features,
Expand Down
148 changes: 0 additions & 148 deletions _meta/beat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,75 +261,6 @@ apm-server:
# Configure curve types for ECDHE based cipher suites.
#ssl.curve_types: []

#---------------------------- APM Server - ILM Index Lifecycle Management ----------------------------

#ilm:
# Supported values are `auto`, `true` and `false`.
# `true`: Make use of Elasticsearch's Index Lifecycle Management (ILM) for APM indices. If no Elasticsearch output is
# configured or the configured instance does not support ILM, APM Server cannot apply ILM and must create
# unmanaged indices instead.
# `false`: APM Server does not make use of ILM in Elasticsearch.
# `auto`: If an Elasticsearch output is configured with default index and indices settings, and the configured
# Elasticsearch instance supports ILM, `auto` will resolve to `true`. Otherwise `auto` will resolve to `false`.
# Default value is `auto`.
#enabled: "auto"

#setup:
# Only disable setup if you want to set up everything related to ILM on your own.
# When setup is enabled, the APM Server creates:
# - aliases and ILM policies if `apm-server.ilm.enabled` resolves to `true`.
# - An ILM specific template per event type. This is required to map ILM aliases and policies to indices. In case
# ILM is disabled, the templates will be created without any ILM settings.
# Be aware that if you turn off setup, you need to manually manage event type specific templates on your own.
# If you simply want to disable ILM, use the above setting, `apm-server.ilm.enabled`, instead.
# Defaults to true.
#enabled: true

# Configure whether or not existing policies and ILM related templates should be updated. This needs to be
# set to true when customizing your policies.
# Defaults to false.
#overwrite: false

# Set `require_policy` to `false` when policies are set up outside of APM Server but referenced here.
# Default value is `true`.
#require_policy: true

# Customized mappings will be merged with the default setup, so you only need to configure mappings for the
# event types, policies, and index suffixes that you want to customize.
# Indices are named in this way: `apm-%{[observer.version]}-%{[event.type]}-{index_suffix}`,
# e.g., apm-7.9.0-span-custom*. The `index_suffix` is optional.
# NOTE: When configuring an `index_suffix`, ensure that no previously set up templates conflict with the
# newly configured ones. If an index matches multiple templates with the same order, the settings of
# the templates will override each other. Any conflicts need to be cleaned up manually.
# NOTE: When customizing `setup.template.name` and `setup.template.pattern`, ensure they still match the indices.
#mapping:
#- event_type: "error"
# policy_name: "apm-rollover-30-days"
# index_suffix: ""
#- event_type: "span"
# policy_name: "apm-rollover-30-days"
# index_suffix: ""
#- event_type: "transaction"
# policy_name: "apm-rollover-30-days"
# index_suffix: ""
#- event_type: "metric"
# policy_name: "apm-rollover-30-days"
# index_suffix: ""

# Configured policies are added to pre-defined default policies.
# If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
#policies:
#- name: "apm-rollover-30-days"
#policy:
#phases:
#hot:
#actions:
#rollover:
#max_size: "50gb"
#max_age: "30d"
#set_priority:
#priority: 100

#================================= General =================================

# Data is buffered in a memory queue before it is published to the configured output.
Expand Down Expand Up @@ -357,42 +288,6 @@ apm-server:
# default is the number of logical CPUs available in the system.
#max_procs:

#================================= Template =================================

# A template is used to set the mapping in Elasticsearch.
# By default template loading is enabled and the template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones.

# Set to false to disable template loading.
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
#setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
#setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
#setup.template.fields: "${path.config}/fields.yml"

# Overwrite existing template.
#setup.template.overwrite: false

# Elasticsearch template settings.
#setup.template.settings:

# A dictionary of settings to place into the settings.index dictionary
# of the Elasticsearch template. For more details, please check
# https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
#index:
#number_of_shards: 1
#codec: best_compression
#number_of_routing_shards: 30
#mapping.total_fields.limit: 2000

#============================= Elastic Cloud =============================

# These settings simplify using APM Server with the Elastic Cloud (https://cloud.elastic.co/).
Expand Down Expand Up @@ -439,49 +334,6 @@ output.elasticsearch:
# Number of workers per Elasticsearch host.
#worker: 1

# By using the configuration below, APM documents are stored to separate indices,
# depending on their `processor.event`:
# - error
# - transaction
# - span
# - sourcemap
#
# The indices are all prefixed with `apm-%{[observer.version]}`.
# To allow managing indices based on their age, all indices (except for sourcemaps)
# end with the information of the day they got indexed.
# e.g. "apm-7.3.0-transaction-2019.07.20"
#
# Be aware that you can only specify one Elasticsearch template.
# If you modify the index patterns you must also update these configurations accordingly,
# as they need to be aligned:
# * `setup.template.name`
# * `setup.template.pattern`
#index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
#indices:
# - index: "apm-%{[observer.version]}-sourcemap"
# when.contains:
# processor.event: "sourcemap"
#
# - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "error"
#
# - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "transaction"
#
# - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "span"
#
# - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "metric"
#
# - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "onboarding"

# Optional HTTP Path.
#path: "/elasticsearch"

Expand Down
Loading