Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xenstore dirs in network backend are not cleaned up correctly #5369

Closed
marmarek opened this issue Oct 4, 2019 · 16 comments · Fixed by QubesOS/qubes-vmm-xen#92
Closed

xenstore dirs in network backend are not cleaned up correctly #5369

marmarek opened this issue Oct 4, 2019 · 16 comments · Fixed by QubesOS/qubes-vmm-xen#92
Assignees
@marmarek
Labels
C: Xen P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.0-bullseye-stable r4.0-buster-stable r4.0-centos7-stable r4.0-centos8-stable r4.0-dom0-stable r4.0-fc31-stable r4.0-fc32-stable r4.0-fc33-stable r4.0-stretch-stable r4.1-bullseye-stable r4.1-buster-stable r4.1-centos8-stable r4.1-dom0-stable T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Milestone
Release 4.0 updates

Comments

@marmarek
Copy link
Member

marmarek commented Oct 4, 2019

Qubes OS version
R4.0

Affected component(s) or functionality
xen

Brief summary
Xenstore directories of network backends are not cleaned up after removing actual backend devices. Empty directories named after previously connected VM's IDs remains. After fair amount of VM startups, empty dirs trigger per-vm entries quota (1000 entries). User visible effect is failure to connect/start further VMs and E2BIG messages in network backend VM kernel log.

To Reproduce

Start and stop a VM connected to sys-firewall a lot of times. Happens during integration tests.

Expected behavior
Old entries are removed.

Actual behavior
Actual entries are removed, but containing directory stays.

Additional context

I vaguely remember a call to remove this directory done by the backend, failing because of missing permission (would have worked with network backend in dom0). I can't find it right now.

Related, non-duplicate issues

Similar to #5343.
@marmarek marmarek added T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists. P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Oct 4, 2019
@marmarek marmarek self-assigned this Oct 4, 2019
@marmarek marmarek added this to the Release 4.0 updates milestone Oct 4, 2019
@marmarek
Copy link
Member Author

marmarek commented Oct 4, 2019

Found it: there is xen-hotplug-cleanup script, patched in our package to specifically remove this cleanup as it didn't worked because of lack of permissions. And the script is gone in upstream Xen 4.12, as it wasn't called by anything anymore (and the actual commit removing it is mine...).

@talex5
Copy link

talex5 commented Dec 22, 2019

Aha - this is the cause of the mirage-firewall crashes in mirage/qubes-mirage-firewall#35 too. We have to monitor all these empty directories in case new interfaces get added to them, but each one costs us a XenStore watch and if you ask for too many then XenStore returns E2BIG.

@xaki23 xaki23 mentioned this issue Nov 16, 2020
Merged
marmarek added a commit to marmarek/qubes-vmm-xen that referenced this issue Jan 3, 2021
@marmarek
Cleanup parent xenstore dir after device remove
10ae10c 
Driver domain is responsible for xenstore cleanup, but it doesn't have
access to remove (empty) parent directory. Do it in toolstack domain as
the final step.

Fixes QubesOS/qubes-issues#5369

(cherry picked from commit df0f102)
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The package xen_4.8.5-29 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package python2-xen-4.8.5-29.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.0 testing repository for the CentOS centos8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component vmm-xen (including package python2-xen-4.8.5-29.fc32) has been pushed to the r4.0 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.1 testing repository for the CentOS centos8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package python3-xen-4.14.1-1.fc32 has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.0 stable repository for the CentOS centos8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component vmm-xen (including package python2-xen-4.8.5-29.fc32) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package xen_2001:4.8.5-29+deb10u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package python2-xen-4.8.5-29.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package vmm-xen has been pushed to the r4.1 stable repository for the CentOS centos8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package xen_4.14.1-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package python3-xen-4.14.1-1.fc32 has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marmarek marmarek

Labels
C: Xen P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.0-bullseye-stable r4.0-buster-stable r4.0-centos7-stable r4.0-centos8-stable r4.0-dom0-stable r4.0-fc31-stable r4.0-fc32-stable r4.0-fc33-stable r4.0-stretch-stable r4.1-bullseye-stable r4.1-buster-stable r4.1-centos8-stable r4.1-dom0-stable T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Projects
None yet
Milestone
Release 4.0 updates
Development

Successfully merging a pull request may close this issue.

Cleanup parent xenstore dir after device remove marmarek/qubes-vmm-xen
4 participants
@talex5 @marmarek @andrewdavidwong @qubesos-bot