v2 refactor #1008

adam-stokes · 2021-04-09T03:47:51Z

Signed-off-by: Adam Stokes [email protected]

What does this PR do?

This PR refactors a lot of the internal code into more organized packages under internal top level directory.

Why is it important?

As we grow the project it makes it a bit easier to reason about what code is doing what and provides a central location to find function references and definitions.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

Tag current master as 1.99.0 prior to merging
Verify current e2e tests pass

How to test this PR locally

The standard godog test instructions apply here:

cd e2e/_suites/fleet
godog -t '@fleed_mode_agent'

Related issues

Supersedes consolidate kibana api to internal project structure #992

elasticmachine · 2021-04-09T03:54:59Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #1008 updated
Start Time: 2021-04-21T14:55:45.956+0000
Duration: 21 min 33 sec
Commit: f11ef39

Test stats 🧪

Test	Results
Failed	0
Passed	93
Skipped	0
Total	93

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	93
Skipped	0
Total	93

e2e/_suites/fleet/features/backend_processes.feature

e2e/_suites/helm/helm_charts_test.go

Signed-off-by: Adam Stokes <[email protected]>

There were 2 steps identical in both the stand-alone and fleet test suites. Running the stand-alone test suite was picking up the step from the fleet test suite and trying to reference the FleetTestSuite structure which did not hold any of the agent information (like the hostname) for the stand alone tests. This fixes it so that the standalone test step is being referenced in the correct test suite. Signed-off-by: Adam Stokes <[email protected]>

Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Signed-off-by: Adam Stokes <[email protected]>

Signed-off-by: Adam Stokes <[email protected]>

e2e/_suites/fleet/fleet.go

adam-stokes · 2021-04-20T13:33:49Z

Another bug hit: elastic/fleet-server#245 putting here to track

Signed-off-by: Adam Stokes <[email protected]>

internal/kibana/agents.go

e2e/_suites/fleet/fleet.go

Signed-off-by: Adam Stokes <[email protected]>

In some cases such as attempting to re-enroll with a revoked token, the elastic-agent will retry indefinitely. This fix adds a safeguard utilizing 'timeout' command prepended to the elastic-agent command so that it will timeout after TimeoutFactor Signed-off-by: Adam Stokes <[email protected]>

mdelapenya · 2021-04-21T15:16:32Z

internal/kibana/agents.go

-	if force {
-		reqBody = `{"force": true}`
-	}
+	reqBody := `{"revoke": true}`


* master: v2 refactor (elastic#1008) fix: use a version of kibana with the fix for fleet-server validations (elastic#1055) feat: add first scenario for Fleet Server (elastic#900) fix: do not use GT_REPO variable, use fixed repo name instead (elastic#1049)

* Move kibana into internals, update fleet test suite * migrate docker-compose related code to internal layout * move docker related code to internal layout * move git related code to internal layout * move common attributes into internal common file system layout * move elasticsearch specifics into its own filesystem layout * move installer based code to internal layout * move shell related code to internal layout * move sanitizer code to internal layout * move io related code to internal layout * move utils into internal layout * Update package integration querying/altering * move curl to internal layout * move helm to internal layout * move kubectl into internal layout * move state internal filesystem * cleanup config in stand-alone * remove unused files * Uniquify the stand-alone step for checking agent status (#993) There were 2 steps identical in both the stand-alone and fleet test suites. Running the stand-alone test suite was picking up the step from the fleet test suite and trying to reference the FleetTestSuite structure which did not hold any of the agent information (like the hostname) for the stand alone tests. This fixes it so that the standalone test step is being referenced in the correct test suite. * Remove the agent config file parameters for stand alone (#983) * Update helm/metricbeat tests to use new layout * Fix policy endpoint update * fix panic on helm init * Fix step reference as this being merged seperately * Update function call to correct standalone step * Fix merge conflict * update ProfileEnv query/set for KibanaVersion * More fixes to agent endpoint security checks * update backend feature to call out endpoint in step * use common.TimeoutFactor in docker checkprocess state * Update adding endpoint integration * enable features for fleet server * not necessary to enroll after install * wait for filebeat/metricbeat before restarts * clear out fts.CurrentToken during beforeScenario * attach system integration on deploy * enroll if rpm * dont store fleet policy * update kibana config for latest fleet server * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update .pre-commit-config.yaml * Update e2e/Makefile * rename apt -> deb for installer type * execute docker start/stop with timeout between * fixes fleet_server scenario * Utilize fleet server in all tests * Fix enrollment url for fleet server * Query elasticsearch logs for endpoint security event changes * Increase search result size for ES * Fix issue with fleet server restarting continuously * unpin kibana pr now that most major breakage is resolved * force unenroll * for new fleet bootstrap on re-enrollment * Fix unenrollment * Add timeout safeguard to elastic-agent execution In some cases such as attempting to re-enroll with a revoked token, the elastic-agent will retry indefinitely. This fix adds a safeguard utilizing 'timeout' command prepended to the elastic-agent command so that it will timeout after TimeoutFactor Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 5f59670) # Conflicts: # e2e/Makefile # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/fleet/stand-alone.go # e2e/_suites/fleet/world.go

* Move kibana into internals, update fleet test suite * migrate docker-compose related code to internal layout * move docker related code to internal layout * move git related code to internal layout * move common attributes into internal common file system layout * move elasticsearch specifics into its own filesystem layout * move installer based code to internal layout * move shell related code to internal layout * move sanitizer code to internal layout * move io related code to internal layout * move utils into internal layout * Update package integration querying/altering * move curl to internal layout * move helm to internal layout * move kubectl into internal layout * move state internal filesystem * cleanup config in stand-alone * remove unused files * Uniquify the stand-alone step for checking agent status (#993) There were 2 steps identical in both the stand-alone and fleet test suites. Running the stand-alone test suite was picking up the step from the fleet test suite and trying to reference the FleetTestSuite structure which did not hold any of the agent information (like the hostname) for the stand alone tests. This fixes it so that the standalone test step is being referenced in the correct test suite. * Remove the agent config file parameters for stand alone (#983) * Update helm/metricbeat tests to use new layout * Fix policy endpoint update * fix panic on helm init * Fix step reference as this being merged seperately * Update function call to correct standalone step * Fix merge conflict * update ProfileEnv query/set for KibanaVersion * More fixes to agent endpoint security checks * update backend feature to call out endpoint in step * use common.TimeoutFactor in docker checkprocess state * Update adding endpoint integration * enable features for fleet server * not necessary to enroll after install * wait for filebeat/metricbeat before restarts * clear out fts.CurrentToken during beforeScenario * attach system integration on deploy * enroll if rpm * dont store fleet policy * update kibana config for latest fleet server * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update .pre-commit-config.yaml * Update e2e/Makefile * rename apt -> deb for installer type * execute docker start/stop with timeout between * fixes fleet_server scenario * Utilize fleet server in all tests * Fix enrollment url for fleet server * Query elasticsearch logs for endpoint security event changes * Increase search result size for ES * Fix issue with fleet server restarting continuously * unpin kibana pr now that most major breakage is resolved * force unenroll * for new fleet bootstrap on re-enrollment * Fix unenrollment * Add timeout safeguard to elastic-agent execution In some cases such as attempting to re-enroll with a revoked token, the elastic-agent will retry indefinitely. This fix adds a safeguard utilizing 'timeout' command prepended to the elastic-agent command so that it will timeout after TimeoutFactor Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 5f59670) # Conflicts: # e2e/Makefile # e2e/_suites/fleet/ingest_manager_test.go # e2e/_suites/fleet/stand-alone.go # e2e/_suites/fleet/world.go Co-authored-by: Adam Stokes <[email protected]>

This reverts commit d757abf. Signed-off-by: Adam Stokes <[email protected]>

* Move kibana into internals, update fleet test suite * migrate docker-compose related code to internal layout * move docker related code to internal layout * move git related code to internal layout * move common attributes into internal common file system layout * move elasticsearch specifics into its own filesystem layout * move installer based code to internal layout * move shell related code to internal layout * move sanitizer code to internal layout * move io related code to internal layout * move utils into internal layout * Update package integration querying/altering * move curl to internal layout * move helm to internal layout * move kubectl into internal layout * move state internal filesystem * cleanup config in stand-alone * remove unused files * Uniquify the stand-alone step for checking agent status (#993) There were 2 steps identical in both the stand-alone and fleet test suites. Running the stand-alone test suite was picking up the step from the fleet test suite and trying to reference the FleetTestSuite structure which did not hold any of the agent information (like the hostname) for the stand alone tests. This fixes it so that the standalone test step is being referenced in the correct test suite. * Remove the agent config file parameters for stand alone (#983) * Update helm/metricbeat tests to use new layout * Fix policy endpoint update * fix panic on helm init * Fix step reference as this being merged seperately * Update function call to correct standalone step * Fix merge conflict * update ProfileEnv query/set for KibanaVersion * More fixes to agent endpoint security checks * update backend feature to call out endpoint in step * use common.TimeoutFactor in docker checkprocess state * Update adding endpoint integration * enable features for fleet server * not necessary to enroll after install * wait for filebeat/metricbeat before restarts * clear out fts.CurrentToken during beforeScenario * attach system integration on deploy * enroll if rpm * dont store fleet policy * update kibana config for latest fleet server * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update e2e/_suites/fleet/fleet.go * Update .pre-commit-config.yaml * Update e2e/Makefile * rename apt -> deb for installer type * execute docker start/stop with timeout between * fixes fleet_server scenario * Utilize fleet server in all tests * Fix enrollment url for fleet server * Query elasticsearch logs for endpoint security event changes * Increase search result size for ES * Fix issue with fleet server restarting continuously * unpin kibana pr now that most major breakage is resolved * force unenroll * for new fleet bootstrap on re-enrollment * Fix unenrollment * Add timeout safeguard to elastic-agent execution In some cases such as attempting to re-enroll with a revoked token, the elastic-agent will retry indefinitely. This fix adds a safeguard utilizing 'timeout' command prepended to the elastic-agent command so that it will timeout after TimeoutFactor Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Signed-off-by: Adam Stokes <[email protected]>

adam-stokes mentioned this pull request Apr 9, 2021

consolidate kibana api to internal project structure #992

Closed

4 tasks

mdelapenya reviewed Apr 12, 2021

View reviewed changes

e2e/_suites/fleet/features/backend_processes.feature Outdated Show resolved Hide resolved

mdelapenya reviewed Apr 12, 2021

View reviewed changes

e2e/_suites/helm/helm_charts_test.go Outdated Show resolved Hide resolved

adam-stokes added the enhancement New feature or request label Apr 12, 2021

adam-stokes and others added 24 commits April 12, 2021 12:35

Move kibana into internals, update fleet test suite

1806d81

Signed-off-by: Adam Stokes <[email protected]>

migrate docker-compose related code to internal layout

ac1ad48

Signed-off-by: Adam Stokes <[email protected]>

move docker related code to internal layout

c6d685c

Signed-off-by: Adam Stokes <[email protected]>

move git related code to internal layout

790fe23

Signed-off-by: Adam Stokes <[email protected]>

move common attributes into internal common file system layout

ce361ed

Signed-off-by: Adam Stokes <[email protected]>

move elasticsearch specifics into its own filesystem layout

47f3d1e

Signed-off-by: Adam Stokes <[email protected]>

move installer based code to internal layout

e23bafe

Signed-off-by: Adam Stokes <[email protected]>

move shell related code to internal layout

6e193f8

Signed-off-by: Adam Stokes <[email protected]>

move sanitizer code to internal layout

caae73c

Signed-off-by: Adam Stokes <[email protected]>

move io related code to internal layout

2feb193

Signed-off-by: Adam Stokes <[email protected]>

move utils into internal layout

95792f5

Signed-off-by: Adam Stokes <[email protected]>

Update package integration querying/altering

50b691d

Signed-off-by: Adam Stokes <[email protected]>

move curl to internal layout

2f7d8a2

Signed-off-by: Adam Stokes <[email protected]>

move helm to internal layout

23617ef

Signed-off-by: Adam Stokes <[email protected]>

move kubectl into internal layout

2c5165d

Signed-off-by: Adam Stokes <[email protected]>

move state internal filesystem

5bbb197

Signed-off-by: Adam Stokes <[email protected]>

cleanup config in stand-alone

b5ed125

Signed-off-by: Adam Stokes <[email protected]>

remove unused files

2a47d9d

Signed-off-by: Adam Stokes <[email protected]>

Remove the agent config file parameters for stand alone (#983)

9bcc3a7

Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Signed-off-by: Adam Stokes <[email protected]>

Update helm/metricbeat tests to use new layout

edcfd24

Signed-off-by: Adam Stokes <[email protected]>

Fix policy endpoint update

16d0167

Signed-off-by: Adam Stokes <[email protected]>

fix panic on helm init

bdbc8e0

Signed-off-by: Adam Stokes <[email protected]>

Fix step reference as this being merged seperately

9842175

Signed-off-by: Adam Stokes <[email protected]>

adam-stokes force-pushed the feat-kibana-refactor branch from bc65320 to 9842175 Compare April 12, 2021 17:50

mdelapenya reviewed Apr 20, 2021

View reviewed changes

e2e/_suites/fleet/fleet.go Show resolved Hide resolved

adam-stokes added 7 commits April 20, 2021 10:49

Merge branch 'master' into feat-kibana-refactor

6715695

Query elasticsearch logs for endpoint security event changes

b43d3e9

Signed-off-by: Adam Stokes <[email protected]>

Increase search result size for ES

9b77822

Signed-off-by: Adam Stokes <[email protected]>

Fix issue with fleet server restarting continuously

4b395d2

Signed-off-by: Adam Stokes <[email protected]>

unpin kibana pr now that most major breakage is resolved

728d927

Signed-off-by: Adam Stokes <[email protected]>

force unenroll

76605b5

Signed-off-by: Adam Stokes <[email protected]>

for new fleet bootstrap on re-enrollment

4a2f459

Signed-off-by: Adam Stokes <[email protected]>

mdelapenya reviewed Apr 21, 2021

View reviewed changes

internal/kibana/agents.go Show resolved Hide resolved

mdelapenya reviewed Apr 21, 2021

View reviewed changes

e2e/_suites/fleet/fleet.go Show resolved Hide resolved

adam-stokes added 2 commits April 21, 2021 10:50

Fix unenrollment

3b5a727

Signed-off-by: Adam Stokes <[email protected]>

mdelapenya reviewed Apr 21, 2021

View reviewed changes

internal/kibana/agents.go

if force {

reqBody = `{"force": true}`

}

reqBody := `{"revoke": true}`

Copy link

Contributor

mdelapenya Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!!

mdelapenya mentioned this pull request Apr 21, 2021

Kubernetes autodiscover suite #1064

Merged

13 tasks

adam-stokes merged commit 5f59670 into master Apr 21, 2021

mdelapenya mentioned this pull request Apr 21, 2021

fix: run unit tests after refactor #1067

Merged

8 tasks

mdelapenya added the v7.13.0 label Apr 21, 2021

mergify bot mentioned this pull request Apr 21, 2021

v2 refactor (backport #1008) #1069

Merged

adam-stokes added a commit that referenced this pull request Apr 21, 2021

Revert "v2 refactor (#1008) (#1069)"

17a80f3

This reverts commit d757abf. Signed-off-by: Adam Stokes <[email protected]>

mdelapenya deleted the feat-kibana-refactor branch April 22, 2021 08:43

adam-stokes mentioned this pull request Apr 22, 2021

work towards a standard project layout for go #959

Closed

This was referenced Apr 23, 2021

Kibana master and 7.x is not able to start #1048

Closed

chore: initialise stale agent only when used #1084

Merged

mdelapenya mentioned this pull request May 11, 2021

chore: update 7.x versions #1157

Merged

8 tasks

mdelapenya mentioned this pull request Oct 26, 2021

chore: bump elastic-package to v0.25.0 #1721

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2 refactor #1008

v2 refactor #1008

adam-stokes commented Apr 9, 2021 •

edited

Loading

elasticmachine commented Apr 9, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

adam-stokes commented Apr 20, 2021

mdelapenya Apr 21, 2021

v2 refactor #1008

v2 refactor #1008

Conversation

adam-stokes commented Apr 9, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Apr 9, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

adam-stokes commented Apr 20, 2021

mdelapenya Apr 21, 2021

Choose a reason for hiding this comment

adam-stokes commented Apr 9, 2021 •

edited

Loading

elasticmachine commented Apr 9, 2021 •

edited

Loading