Fix fleet resources not having ID set on import #447

taylor-swanson · 2023-10-13T19:27:58Z

Fixed fleet objects not having an ID set when importing.
When creating a new resource, if the ID was not explicitly specified, then the *_id value is used from the schema. For all other modes (update, read, delete), the existing/provided ID is used. This resolves the issue where the *_id value (even if it was empty) would override the provided ID value (i.e., during import).
Update fleet documentation to remove kibana space id. Fleet objects aren't scoped by spaces, but rather use their own IDs.

Related issues:

Fixes [Bug] Cannot import Fleet policy #396
Fixes [Bug] Cannot import Fleet output #369

- Fixed fleet objects not having an ID set when importing. - Update fleet documentation to remove kibana space id

taylor-swanson · 2023-10-13T19:38:00Z

Tested importing agent policy, fleet server host, and output:

$ TF_LOG=INFO terraform import elasticstack_fleet_agent_policy.docker_fleet_agent elastic-agent-managed-ep
2023-10-13T14:13:49.507-0500 [INFO]  Terraform version: 1.4.6
2023-10-13T14:13:49.508-0500 [INFO]  Go runtime version: go1.19.6
2023-10-13T14:13:49.508-0500 [INFO]  CLI args: []string{"terraform", "import", "elasticstack_fleet_agent_policy.docker_fleet_agent", "elastic-agent-managed-ep"}
2023-10-13T14:13:49.508-0500 [INFO]  Loading CLI configuration from /Users/tswanson/.terraformrc
2023-10-13T14:13:49.508-0500 [INFO]  CLI command args: []string{"import", "elasticstack_fleet_agent_policy.docker_fleet_agent", "elastic-agent-managed-ep"}
2023-10-13T14:13:49.518-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:13:49.868-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:13:49.868-0500
2023-10-13T14:13:49.912-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:13:49.935-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:13:49.935-0500
elasticstack_fleet_agent_policy.docker_fleet_agent: Importing from ID "elastic-agent-managed-ep"...
elasticstack_fleet_agent_policy.docker_fleet_agent: Import prepared!
  Prepared elasticstack_fleet_agent_policy for import
elasticstack_fleet_agent_policy.docker_fleet_agent: Refreshing state... [id=elastic-agent-managed-ep]
2023-10-13T14:13:50.051-0500 [WARN]  Provider "registry.terraform.io/hashicorp/elasticstack" produced an unexpected new value for elasticstack_fleet_agent_policy.docker_fleet_agent during refresh.
      - .name: was null, but now cty.StringVal("Elastic-Agent (elastic-package)")
      - .namespace: was null, but now cty.StringVal("default")
      - .monitor_logs: was null, but now cty.True
2023-10-13T14:13:50.052-0500 [INFO]  Writing state output to:

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
                                                                                            
$ TF_LOG=INFO terraform import elasticstack_fleet_server_host.default_host fleet-default-fleet-server-host
2023-10-13T14:14:21.337-0500 [INFO]  Terraform version: 1.4.6
2023-10-13T14:14:21.337-0500 [INFO]  Go runtime version: go1.19.6
2023-10-13T14:14:21.337-0500 [INFO]  CLI args: []string{"terraform", "import", "elasticstack_fleet_server_host.default_host", "fleet-default-fleet-server-host"}
2023-10-13T14:14:21.337-0500 [INFO]  Loading CLI configuration from /Users/tswanson/.terraformrc
2023-10-13T14:14:21.338-0500 [INFO]  CLI command args: []string{"import", "elasticstack_fleet_server_host.default_host", "fleet-default-fleet-server-host"}
2023-10-13T14:14:21.346-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:14:21.379-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:14:21.379-0500
2023-10-13T14:14:21.417-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:14:21.438-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:14:21.438-0500
elasticstack_fleet_server_host.default_host: Importing from ID "fleet-default-fleet-server-host"...
elasticstack_fleet_server_host.default_host: Import prepared!
  Prepared elasticstack_fleet_server_host for import
elasticstack_fleet_server_host.default_host: Refreshing state... [id=fleet-default-fleet-server-host]
2023-10-13T14:14:21.499-0500 [WARN]  Provider "registry.terraform.io/hashicorp/elasticstack" produced an unexpected new value for elasticstack_fleet_server_host.default_host during refresh.
      - .name: was null, but now cty.StringVal("Default")
      - .default: was null, but now cty.True
      - .hosts: was null, but now cty.ListVal([]cty.Value{cty.StringVal("https://fleet-server:8220")})
2023-10-13T14:14:21.501-0500 [INFO]  Writing state output to:

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ TF_LOG=INFO terraform import elasticstack_fleet_output.default_output fleet-default-output              
2023-10-13T14:14:27.709-0500 [INFO]  Terraform version: 1.4.6
2023-10-13T14:14:27.709-0500 [INFO]  Go runtime version: go1.19.6
2023-10-13T14:14:27.709-0500 [INFO]  CLI args: []string{"terraform", "import", "elasticstack_fleet_output.default_output", "fleet-default-output"}
2023-10-13T14:14:27.709-0500 [INFO]  Loading CLI configuration from /Users/tswanson/.terraformrc
2023-10-13T14:14:27.710-0500 [INFO]  CLI command args: []string{"import", "elasticstack_fleet_output.default_output", "fleet-default-output"}
2023-10-13T14:14:27.719-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:14:27.796-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:14:27.795-0500
2023-10-13T14:14:27.834-0500 [INFO]  provider: configuring client automatic mTLS
2023-10-13T14:14:27.868-0500 [INFO]  provider.terraform-provider-elasticstack: configuring server automatic mTLS: timestamp=2023-10-13T14:14:27.868-0500
elasticstack_fleet_output.default_output: Importing from ID "fleet-default-output"...
elasticstack_fleet_output.default_output: Import prepared!
  Prepared elasticstack_fleet_output for import
elasticstack_fleet_output.default_output: Refreshing state... [id=fleet-default-output]
2023-10-13T14:14:27.929-0500 [INFO]  Writing state output to:

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

taylor-swanson · 2023-10-13T20:10:05Z

@tobio,

I noticed when testing this that there seem to be some breaking changes around the provider schema, specifically when providing username and password for the kibana or fleet sections. This used to work, but not anymore, possibly due to #343?

provider "elasticstack" {
  fleet {
    endpoint = "https://localhost:5601"
    username = "elastic"
    password = "abcd1234"
    insecure = true
  }
}

Yields:

│ Error: Invalid Path Expression for Schema
│ 
│   with provider["registry.terraform.io/hashicorp/elasticstack"],
│   on /Users/tswanson/Workarea/terraform/elasticstack/test8/main.tf line 1, in provider "elasticstack":
│    1: provider "elasticstack" {
│ 
│ The Terraform Provider unexpectedly provided a path expression that does not match the current schema. This can happen if the path expression does not correctly follow the schema in structure or types. Please report this to the provider developers.
│ 
│ Path Expression: fleet.0.username
╵

╷
│ Error: Invalid Path Expression for Schema
│ 
│   with provider["registry.terraform.io/hashicorp/elasticstack"],
│   on /Users/tswanson/Workarea/terraform/elasticstack/test8/main.tf line 1, in provider "elasticstack":
│    1: provider "elasticstack" {
│ 
│ The Terraform Provider unexpectedly provided a path expression that does not match the current schema. This can happen if the path expression does not correctly follow the schema in structure or types. Please report this to the provider developers.
│ 
│ Path Expression: fleet.0.password
╵

Kibana has the same problem. If I move those to elasticsearch, then everything works fine.

Not sure if this was intentional or if you've seen anything with this, yet.

tobio

LGTM

breml · 2023-10-27T05:16:09Z

@tobio, @taylor-swanson would it be possible to have a new release including this fix, maybe v0.9.1?

taylor-swanson · 2023-10-27T13:12:10Z

@breml, that's what I was thinking, a v0.9.1 release for this and include #448 as well, which was just merged.

@tobio, would we be able to cut a new release soon? It'd be good to get these bug fixes out before the Protocol 6 work gets merged.

tobio · 2023-10-30T23:41:18Z

Yea, we get these two released. It feels worthwhile waiting for #454 as well unless you think that one might get delayed?

taylor-swanson · 2023-10-31T13:03:35Z

@tobio, I'll see if I can get the changes turned around for #454. If it takes me more than 24 hours, feel free to cut the v0.9.1 release as is and #454 could go into a v0.9.2 or something equivalent. I mainly don't want to hold up the bug fixes as we have a number of people waiting on them.

Fix fleet objects not having ID set on import

d84f917

- Fixed fleet objects not having an ID set when importing. - Update fleet documentation to remove kibana space id

taylor-swanson self-assigned this Oct 13, 2023

taylor-swanson added the bug Something isn't working label Oct 13, 2023

taylor-swanson mentioned this pull request Oct 13, 2023

[Bug] Cannot import Fleet policy #396

Closed

taylor-swanson added 2 commits October 13, 2023 14:38

Update changelog

427f027

Merge branch 'main' into bug/fleet-import

909a80b

taylor-swanson marked this pull request as ready for review October 13, 2023 20:06

tobio mentioned this pull request Oct 16, 2023

Fix schema validation for kibana and fleet configuration #449

Merged

Merge branch 'main' into bug/fleet-import

3b8fa28

taylor-swanson mentioned this pull request Oct 17, 2023

Switch to protocol 6 #451

Merged

tobio approved these changes Oct 24, 2023

View reviewed changes

tobio merged commit 3ea2662 into elastic:main Oct 24, 2023
13 of 14 checks passed

taylor-swanson deleted the bug/fleet-import branch October 27, 2023 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix fleet resources not having ID set on import #447

Fix fleet resources not having ID set on import #447

taylor-swanson commented Oct 13, 2023

taylor-swanson commented Oct 13, 2023

taylor-swanson commented Oct 13, 2023

tobio left a comment

breml commented Oct 27, 2023

taylor-swanson commented Oct 27, 2023

tobio commented Oct 30, 2023

taylor-swanson commented Oct 31, 2023

Fix fleet resources not having ID set on import #447

Fix fleet resources not having ID set on import #447

Conversation

taylor-swanson commented Oct 13, 2023

taylor-swanson commented Oct 13, 2023

taylor-swanson commented Oct 13, 2023

tobio left a comment

Choose a reason for hiding this comment

breml commented Oct 27, 2023

taylor-swanson commented Oct 27, 2023

tobio commented Oct 30, 2023

taylor-swanson commented Oct 31, 2023