Plugin process not exited properly after server process exit with error

[lapwingcloud]

Describe the bug

To Reproduce
See above screenshot

When the database not initialized, run sudo -u boundary /usr/bin/boundary server -config /etc/boundary/controller.hcl twice

config

disable_mlock = true

controller {
  name = "vm-tools-stg-boundary-main000000"
  database {
    url = "[redacted]"
  }
}

listener "tcp" {
  purpose = "api"
  address = "10.45.100.5"

  tls_disable = true
}

listener "tcp" {
  purpose = "cluster"
  address = "127.0.0.1"

  tls_disable = true
}

plugins {
  execution_dir = "/var/run/boundary/plugin-exec"
}

kms "azurekeyvault" {
  purpose        = "root"
  tenant_id      = "[redacted]"
  client_id      = "[redacted]"
  vault_name     = "[redacted]"
  key_name       = "boundary-root"
}

Expected behavior
The second time it should return the same error

The database has not been initialized. Please run 'boundary database init'.

instead of

Error parsing KMS configuration: error writing out plugin for execution: open /var/run/boundary/plugin-exec/boundary-plugin-kms-azurekeyvault.gz: text file busy

Additional context
Add any other context about the problem here.

[lapwingcloud]

seems like it only happens in certain scenario, if the boundary server is fully started , then stopping the server process will stop the plugin process as well

[louisruch]

@jchenship thanks for reporting this issue. I was able to reproduce it locally and I have a PR up that fixes this and some other situations that would produce the same/similar error. You might notice that we also did not cleanup the plugins after running database init even without an error.