Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Salt fails with TypeError: cannot pickle 'generator' object and AttributeError: 'Pillar' object has no attribute '_closing' #9532

Closed
ben-grande opened this issue Oct 23, 2024 · 1 comment
Closed

Salt fails with TypeError: cannot pickle 'generator' object and AttributeError: 'Pillar' object has no attribute '_closing' #9532

ben-grande opened this issue Oct 23, 2024 · 1 comment
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: mgmt P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. R: self-closed Voluntarily closed by the person who opened it before another resolution occurred. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.

Comments

@ben-grande
Copy link

How to file a helpful issue

Qubes OS release

R4.2

Brief summary

I was editing a lot of files in /srv/, specially /srv/salt/_grains/ for a new grain and I also updated dom0. On the next salt-call, the error started appearing. I am not sure it is related to the updates as I waited for some days to post this to see if anyone faces it also. Last update was qubes-mgmt-salt-dom0-update, which did not have anything related to Pillar in the commit, so I dicarded it.

Software versions:

qubes-mgmt-salt.noarch                        4.2.2-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-admin-tools.noarch            4.2.2-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-base.noarch                   4.1.7-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-base-config.noarch            4.1.2-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-base-topd.noarch              4.2.1-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-config.noarch                 4.2.2-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-dom0.noarch                   4.2.2-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-dom0-qvm.noarch               4.2.1-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-dom0-update.noarch            4.2.0-1.fc37   @qubes-dom0-cached
qubes-mgmt-salt-dom0-virtual-machines.noarch  4.2.18-1.fc37  @qubes-dom0-cached
salt.noarch                                   3006.9-2.fc37  @qubes-dom0-cached
salt-minion.noarch                            3006.9-2.fc37  @qubes-dom0-cached

What I have tried:

  • Reinstall all salt packages with sudo qubes-dom0-update --action=reinstall "*salt*"
  • Remove /srv/ and reinstall all salt packages
  • Restart the computer
  • Remove __pycache__/*.pyc and restart the computer

Steps to reproduce

Call any salt module such as:

sudo qubesctl saltutil.sync_utils refresh=True -l debug

Expected behavior

States are applied succesfully.

Actual behavior

[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/lib/python3.11/site-packages/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Determining pillar cache
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
TypeError: cannot pickle 'generator' object
Traceback (most recent call last):
  File "/usr/bin/qubesctl", line 130, in <module>
    salt_call()
  File "/usr/lib/python3.11/site-packages/salt/scripts.py", line 444, in salt_call
    client.run()
  File "/usr/lib/python3.11/site-packages/salt/cli/call.py", line 40, in run
    caller = salt.cli.caller.Caller.factory(self.config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 42, in factory
    return ZeroMQCaller(opts, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 299, in __init__
    super().__init__(opts)
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 63, in __init__
    self.minion = salt.minion.SMinion(opts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/minion.py", line 942, in __init__
    self.gen_modules(initial_load=True, context=context)
  File "/usr/lib/python3.11/site-packages/salt/minion.py", line 445, in gen_modules
    self.opts["pillar"] = salt.pillar.get_pillar(
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 86, in get_pillar
    return ptype(
           ^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 556, in __init__
    self.opts = self.__gen_opts(opts, grains, saltenv=saltenv, pillarenv=pillarenv)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 654, in __gen_opts
    opts = copy.deepcopy(opts_in)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'generator' object
Traceback (most recent call last):
  File "/usr/bin/qubesctl", line 130, in <module>
    salt_call()
  File "/usr/lib/python3.11/site-packages/salt/scripts.py", line 444, in salt_call
    client.run()
  File "/usr/lib/python3.11/site-packages/salt/cli/call.py", line 40, in run
    caller = salt.cli.caller.Caller.factory(self.config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 42, in factory
    return ZeroMQCaller(opts, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 299, in __init__
    super().__init__(opts)
  File "/usr/lib/python3.11/site-packages/salt/cli/caller.py", line 63, in __init__
    self.minion = salt.minion.SMinion(opts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/minion.py", line 942, in __init__
    self.gen_modules(initial_load=True, context=context)
  File "/usr/lib/python3.11/site-packages/salt/minion.py", line 445, in gen_modules
    self.opts["pillar"] = salt.pillar.get_pillar(
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 86, in get_pillar
    return ptype(
           ^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 556, in __init__
    self.opts = self.__gen_opts(opts, grains, saltenv=saltenv, pillarenv=pillarenv)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 654, in __gen_opts
    opts = copy.deepcopy(opts_in)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'generator' object
Exception ignored in: <function Pillar.__del__ at 0x7b854d8c7240>
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 1379, in __del__
    self.destroy()
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 1363, in destroy
    if self._closing:
       ^^^^^^^^^^^^^
AttributeError: 'Pillar' object has no attribute '_closing'
@ben-grande ben-grande added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists. labels Oct 23, 2024
@ben-grande ben-grande mentioned this issue Oct 23, 2024
Closed
@andrewdavidwong andrewdavidwong added C: mgmt needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. affects-4.2 This issue affects Qubes OS 4.2. labels Oct 24, 2024
@ben-grande
Copy link
Author

I have not found the cause of the issue, probably some bad pillars that were cached and broke salt.

I fixed with:

sudo rm -rf /var/cache/salt/minion
sudo qubes-dom0-update --action=reinstall "*salt*"

The last command might not even be needed, it possibly can be substituted by:

sudo qubesctl saltutil.clear_cache
sudo qubesctl saltutil.sync_all

@andrewdavidwong andrewdavidwong added R: self-closed Voluntarily closed by the person who opened it before another resolution occurred. and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: mgmt P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. R: self-closed Voluntarily closed by the person who opened it before another resolution occurred. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andrewdavidwong @ben-grande