device or resource busy when attempting to load elastic-agent config in docker

[adam-stokes]

In some instances like stand-alone testing and running from the cli we hit this error:

Performing setup of Fleet in Kibana
The Elastic Agent is currently in BETA and should not be used in production

Error: could not save enrollment information: could not backup /usr/share/elastic-agent/elastic-agent.yml: rename /usr/share/elastic-agent/elastic-agent.yml /usr/share/elastic-agent/elastic-agent.yml.2021-04-01T12-38-10.4257.bak: device or resource busy
Error: enrollment failed: exit status 1

If we remove that volume mount everything works as intended, however, this makes it a little difficult to include the default system integration

[adam-stokes]

/cc @jalvz I'm hitting this as well in my tests, investigating to see what's going on

[adam-stokes]

From my research, anything that modfies the inode will fail after docker 1.10 so we may need to look into attempting to mount a parent directory.

Or don't use volumes and just copy in the required configuration files within the step and cli provisioning

[jalvz]

Thanks for digging @adam-stokes !

anything that modfies the inode will fail after docker 1.10

What would that be? Or in other words, why this wasn't a problem before #978?

[adam-stokes]

I'm not sure why this wasn't a problem before, I think this is a pretty good description of what's happening when elastic-agent.yml is getting updated https://stackoverflow.com/a/60289765. So I think during the operation of making a backup of elastic.agent.yml the reference inode is being replaced causing the resource busy error. Maybe that backup feature was introduced after March 31?

[mdelapenya]

As we discussed online, maybe we do not need the volume anymore, as we are already consuming the configuration file in the latest state:

• for Beats PRs, where we downloaded the file from the PR, the config file for that PR should be already bundled into the docker image, so no need to download it twice.
• for Beats branches, where we consume packaged images from a GCP bucket, the config file should be already bundled into the docker image, so ditto.
• for the development life cycle of the tests, where we consume released images, the config file is already bundled. Ditto again.
  UPDATE:
• for running tests for a local Beats repository, where we get the filepath to the Docker image tar file from Beats' distributions after a local build. Again, the config file should be bundled. Ditto again.

I propose removing that fetch part, as it is duplicating efforts. Therefore, we could remove the volume to the config file in the agent compose file.

[mdelapenya]

BTW, maybe @michalpristas knows about this: is the config file present/bundled into the installers (tar, rpm, deb, docker)?

When we extract a TAR file:

[2021-04-06T14:12:27.904Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:28.166Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/heartbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:28.166Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/elastic-agent.yml
[2021-04-06T14:12:28.166Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:28.166Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:28.428Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:28.428Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/apm-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:28.428Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/heartbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:28.428Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/elastic-agent
[2021-04-06T14:12:28.689Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:28.689Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/apm-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:28.949Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/fleet-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:29.209Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/fleet-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/LICENSE.txt
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/endpoint-security-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/fleet-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/README.md
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/apm-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:29.210Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/heartbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
[2021-04-06T14:12:29.470Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/.build_hash.txt
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/elastic-agent.reference.yml
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/.elastic-agent.active.commit
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/NOTICE.txt
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/data/elastic-agent-e8aa6d/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
[2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/elastic-agent

We do see [2021-04-06T14:12:29.471Z] elastic-agent-8.0.0-SNAPSHOT-linux-x86_64/elastic-agent.reference.yml. Will that file be used by the agent?

[jalvz]

AFAIK, no, that file is not used.

[mdelapenya]

This should be fixed with #983

[mdelapenya]

What would that be? Or in other words, why this wasn't a problem before #978?

Could it be the case an agent with fleet-server performed an operation on the file that the simple enrollment of the agent was not performing?