feat: sync Docker Compose files with Beats Integrations #84

mdelapenya · 2020-02-17T11:16:58Z

What does this PR do?

This PR syncs the docker compose files from the project with the existing ones in Beats for metricbeat's integrations.

To achieve that, we added a new command to the CLI (sync integrations [--remove, --remote]) which fetches the default Beats repo (elastic/beats:master) to the tool's workspace: ~/.op/git/beats. It's possible to instrument this command with two flags:

[-R | --remove]: to remove the existing local repository before cloning the repo
[-r | --remote]: to use a different remote, in the $USER:$BRANCH format (i.e. "mdelapenya:my-branch")

Once the Beats repo is cloned, we will copy the compose files into the tool's workspace, alongside the other compose files: ~/.op/compose/services.

To make this possible, we had to change the layout of the services and stacks. In the past, there was a YML file representing the service/stack to deploy, where its name was the service name. Now we are going to use the service name as a directory, in which we will copy the docker-compose.yml file coming from the Beats repo. Besides that, we downgraded the spec version of the files from '3.0' to '2.3', to match the spec version Beats files use.

Alongside with the compose files, we are copying the _meta directory from Beats repo, where there exists a supported-versions.yml file, which is responsible for the versions to support (as obvious). This will be mapped as environment variables into the running services to be able to define properly the variants for those services needing it. These files will be important, because ONLY those services with the file will be copied to the tool.

Talking about variants, we had to create two different Gherkin steps to cover this scenario, which is used by MySQL. It was super easy to create them and reuse common functionality.

Maybe we need to refactor those two new steps to share code with the old ones. I intentionally did not do it, as they are just two. Another occurrence and we must do it for sure.

Once the docker-compose files have been fetched from the git repo, they will be discoverable by the CLI tool, so any subcommand (i.e. run, stop, deploy, undeploy will know about them.

$ op run service dropwizard

A follow-up for this is asking the user for the versioning an interactive manner: reading the supported files on runtime and offering the user a list with the supported versions.

Because we want to match Beats integrations files, we also removed the default services that came bundled with the CLI. They will not be present for first time, and a sync must be done if we want to use them.

Another improvement we did in this PR is to address #36. With each run of the CLI will create a state file under tool's workspace (i.e. ~/.op/apache-service.run. These state files will hold information about the run: which environment variables were added to the run, including its version, and the services in the run in the case we deployed services to a stack. This file will be in YML format, and will contain a suffix in its name setting if the run if for a service or a stack. The CLI will update the state when needed, destroying it with the Down. On the other hand, the CLI will read the state to simplify its usage, and not forcing the user to pass again more variables (i.e. the version) with each command. As an example: if you run an apache service v2.4, then it's not needed to pass the version again when stopping it.

As a consequence of above requirements, we created several specialised Go files: io.go, shell.go, git.go, state.go, which will represent internal functionality for the CLI.

To finish, we used 7.6.0 as the default version for the stack and metricbeat, to match with current release (remember it's possible to modify it using environment variables: OP_STACK_VERSION and OP_METRICBEAT_VERSION.

Author's checklist

Tests for Apache passing
Tests for MySQL and ALL variants passing
Tests for Redis passing
Tests for vSphere passing

Follow-up

As a result of elastic/beats#16311, we are going to massage the fetched docker-compose files to remove the build section, which requires accessing a relative path which was not present at the first docker-compose file. This will be done in next commits in this PR, but I wanted to publish this work for you to review it 🙏

Related issues

We'll create adirectory per service/integration, which will need a modification of the current behavior for fetching services: instead of using its name as compose file, we will use that name as directory in where a docker-compose.yml file will be located

We are going to copy the docker-compose.yml file AND the _meta directory for each service

We downgraded from 3 to 2.3 because Beats' iintegratioins use that version. It's not possible to upgrade theirs, because they leverage 'depends_on' and 'extends', which are noot supported by 3.x

Those service in the file system take precedence over those boxed in the binary

Let's use the default ones, until the user syncs the integrations with the `sync integrations` command

…config

We do not want to read from the packr box: use the available services struct

Retrieving a docker-compose file from the configuration should not expose whether it's located in the file system or boxed in the binary

A run is an execution of the tool, be it a stack or a service. With the tool we can add or remove services to a run The .run file will contain the stack (optional) and the services in the run, including the environment used for the ran, which will allow to restore that state for future updates in the run. We also support recovering and desttroying the state for a run

mdelapenya · 2020-02-17T12:03:40Z

.ci/scripts/functional-test.sh

@@ -23,6 +23,9 @@ make -C metricbeat-tests fetch-binary
 # Build runtime dependencies
 STACK_VERSION=${STACK_VERSION} make -C metricbeat-tests run-elastic-stack

+# Sync integrations
+make -C metricbeat-tests sync-integrations


This is forcing the CI to get the compose files from Beats

mdelapenya · 2020-02-17T12:05:45Z

cli/config/config.go

-func GetPackedCompose(isStack bool, composeName string) (string, error) {
-	composeFileName := composeName + ".yml"
+func GetComposeFile(isStack bool, composeName string) (string, error) {
+	composeFileName := "docker-compose.yml"


Because of the layout change, we can keep the name as the default one

mdelapenya · 2020-02-17T12:07:23Z

cli/config/config.go

+// PutServiceEnvironment puts the environment variables for the service, replacing "SERVICE_"
+// with service name in uppercase. The variables are:
+//  - SERVICE_VERSION: where it represents the version of the service (i.e. APACHE_VERSION)
+//  - SERVICE_PATH: where it represents the path to its compose file (i.e. APACHE_PATH)


Follow-up: we won't need the *_PATH after updating the fetched compose files to remove the build section.

mdelapenya · 2020-02-17T12:07:31Z

cli/config/config.go

+			"service": service,
+		}).Warn("Could not find compose file")
+	} else {
+		env[serviceUpper+"_PATH"] = filepath.Dir(srv.Path)


Follow-up: we won't need the *_PATH after updating the fetched compose files to remove the build section.

jsoriano

Great job here, I have added some comments, they can probably be follow ups.

cli/cmd/sync.go

jsoriano · 2020-02-17T17:48:20Z

cli/config/config.go

+//  - SERVICE_VARIANT: where SERVICE is the name of the service (i.e. APACHE_VARIANT)
+//  - SERVICE_VERSION: where it represents the version of the service (i.e. APACHE_VERSION)
+func PutServiceVariantEnvironment(env map[string]string, service string, serviceVariant string, serviceVersion string) map[string]string {
+	type EnvVar interface{}


I think you can safely assume that this is going to be an string to string map.

Suggested change

type EnvVar interface{}

type EnvVar map[string]string

I guess that would affect https://github.com/elastic/metricbeat-tests-poc/pull/84/files/96f571c258cb5f13c2ebea54c6c377529e29da29#diff-4826362d61b88bc316da9d728b970b7aR203

metricbeat-tests/metricbeat_test.go

cli/config/config.go

cli/internal/io.go

cli/internal/state.go

metricbeat-tests/runner_test.go

cli/README.md

cli/cmd/sync.go

cli/config/config.go

cli/internal/shell.go

metricbeat-tests/runner_test.go

mdelapenya added 30 commits February 11, 2020 16:22

feat: support cloning git repositories

10388e0

feat: add a subcommand to download elastic/beats using its master branch

71dac4d

chore: add a VSCode configuration for debugging new command

0e720cb

feat: support using another remote different than upstream

a840bbd

feat: support removing the existing repository before cloning

0792353

chore: migrate existing services and stacks to the new format

0c40b76

chore: add compose files for all integrations from Beats

b00d92c

feat: store new compose and related files into the packr box

7cd468f

We are going to copy the docker-compose.yml file AND the _meta directory for each service

chore: sync docker-compose version with Beats

dfdb525

We downgraded from 3 to 2.3 because Beats' iintegratioins use that version. It's not possible to upgrade theirs, because they leverage 'depends_on' and 'extends', which are noot supported by 3.x

fix: add services and stacks that are present at tool's workspace

9429684

Those service in the file system take precedence over those boxed in the binary

chore: do not bundle the tool with any service

d4c7b69

Let's use the default ones, until the user syncs the integrations with the `sync integrations` command

docs: update CLI docs

30370f8

chore: sync integration services using the build

1782e10

chore: run go mod tidy

9176313

feat: support calculating a service's environment variables from CLI …

8ee8a1d

…config

chore: do not pollute io file with compose business logic

68c57e1

fix: only copy services including a supported-versions.yml file

a2d490c

fix: simplify calculating service environment variables

7920118

We do not want to read from the packr box: use the available services struct

chore: move exists method to io file

f44ec0b

chore: rename method to be more descriptive

d70a5c4

Retrieving a docker-compose file from the configuration should not expose whether it's located in the file system or boxed in the binary

chore: use io's mkdirall everywhere

57a9dd4

chore: move writeFile to io

3e2bc59

chore: move readDir to io

b06ce15

chore: move func close to its struct

55d042c

chore: update vscode debug configurations

b740aa2

fix: retrieve service environment when running or deploying a service

715174a

feat: support updating state after docker-compose execution

464a947

chore: use same compose version in existing services

e67d92a

feat: create specific steps for service variants

eae1881

mdelapenya self-assigned this Feb 17, 2020

mdelapenya added the automation label Feb 17, 2020

mdelapenya requested review from a team and jsoriano February 17, 2020 11:17

mdelapenya added 2 commits February 17, 2020 12:43

fix: remove duplicated test

a181898

fix: mkdirall must create the dirs for a path, not its parent

244540e

mdelapenya commented Feb 17, 2020

View reviewed changes

fix: handle errors for mkdirall when needed

5dc4cb0

mdelapenya force-pushed the 79-sync-integrations branch from c9e2719 to 5dc4cb0 Compare February 17, 2020 12:48

fix: do not capitalise variable (gocritic)

96f571c

jsoriano reviewed Feb 17, 2020

View reviewed changes

chore: excludes from golint

9121592

mdelapenya commented Feb 18, 2020

View reviewed changes

cli/internal/state.go Show resolved Hide resolved

v1v reviewed Feb 18, 2020

View reviewed changes

metricbeat-tests/runner_test.go Show resolved Hide resolved

cli/README.md Outdated Show resolved Hide resolved

cachedout reviewed Feb 18, 2020

View reviewed changes

mdelapenya added 3 commits February 18, 2020 16:40

fix: rename flag to a more descriptive name

330d262

fix: typos in docs

fc79996

docs: document execute method

83d192b

This was referenced Feb 18, 2020

Get latest version of the stack in an automated manner #85

Closed

Use Beats utilities to copy files and directories #86

Closed

mdelapenya marked this pull request as ready for review February 18, 2020 17:21

jsoriano approved these changes Feb 18, 2020

View reviewed changes

cachedout approved these changes Feb 18, 2020

View reviewed changes

mdelapenya merged commit 81be30b into elastic:master Feb 19, 2020

mdelapenya deleted the 79-sync-integrations branch February 19, 2020 09:07

mdelapenya mentioned this pull request Feb 21, 2020

Remove build context YAML section from fetched docker-compose files #87

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: sync Docker Compose files with Beats Integrations #84

feat: sync Docker Compose files with Beats Integrations #84

mdelapenya commented Feb 17, 2020 •

edited

Loading

mdelapenya Feb 17, 2020

mdelapenya Feb 17, 2020

mdelapenya Feb 17, 2020

mdelapenya Feb 17, 2020

jsoriano left a comment

jsoriano Feb 17, 2020

mdelapenya Feb 17, 2020

feat: sync Docker Compose files with Beats Integrations #84

feat: sync Docker Compose files with Beats Integrations #84

Conversation

mdelapenya commented Feb 17, 2020 • edited Loading

What does this PR do?

Author's checklist

Follow-up

Related issues

mdelapenya Feb 17, 2020

Choose a reason for hiding this comment

mdelapenya Feb 17, 2020

Choose a reason for hiding this comment

mdelapenya Feb 17, 2020

Choose a reason for hiding this comment

mdelapenya Feb 17, 2020

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Feb 17, 2020

Choose a reason for hiding this comment

mdelapenya Feb 17, 2020

Choose a reason for hiding this comment

mdelapenya commented Feb 17, 2020 •

edited

Loading