From 4b096c1e8626ecde13fcb2874fbde23ec6627ebe Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 00:07:30 -0500 Subject: [PATCH 01/26] Move all the module-development documentation into a single folder --- doc/topics/development/{ => modules}/dunder_dictionaries.rst | 0 doc/topics/development/{ => modules}/external_pillars.rst | 0 doc/topics/development/{modular_systems.rst => modules/index.rst} | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename doc/topics/development/{ => modules}/dunder_dictionaries.rst (100%) rename doc/topics/development/{ => modules}/external_pillars.rst (100%) rename doc/topics/development/{modular_systems.rst => modules/index.rst} (100%) diff --git a/doc/topics/development/dunder_dictionaries.rst b/doc/topics/development/modules/dunder_dictionaries.rst similarity index 100% rename from doc/topics/development/dunder_dictionaries.rst rename to doc/topics/development/modules/dunder_dictionaries.rst diff --git a/doc/topics/development/external_pillars.rst b/doc/topics/development/modules/external_pillars.rst similarity index 100% rename from doc/topics/development/external_pillars.rst rename to doc/topics/development/modules/external_pillars.rst diff --git a/doc/topics/development/modular_systems.rst b/doc/topics/development/modules/index.rst similarity index 100% rename from doc/topics/development/modular_systems.rst rename to doc/topics/development/modules/index.rst From 5bfc6c6eeb51d7a8d2fd830ba475972623dabb65 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 02:58:56 -0500 Subject: [PATCH 02/26] Update reference to modules/index --- doc/topics/development/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/topics/development/index.rst b/doc/topics/development/index.rst index c4a341c0404b..02db13212b34 100644 --- a/doc/topics/development/index.rst +++ b/doc/topics/development/index.rst @@ -12,6 +12,7 @@ Developing Salt raet/index git/index conventions/index + modules/index ../../ref/internals/index ../projects/index ../tutorials/writing_tests From 125e46ed65cef7ecb3edf56a8296287280b303b6 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 02:59:16 -0500 Subject: [PATCH 03/26] Add section on how modules get loaded, including giant table of names --- doc/topics/development/modules/index.rst | 110 +++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 6060ceea88cc..97664f4538a2 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -10,6 +10,116 @@ The most commonly used modular systems are execution modules and states. But the modular systems extend well beyond the more easily exposed components and are often added to Salt to make the complete system more flexible. +Loading Modules +=============== + +Modules come primarily from several sources: + +* The Salt package itself +* The Salt File Server +* The extmods directory +* Secondary packages installed + +Using one source to override another is not supported. + +The Salt Package +---------------- + +Salt itself ships with a large number of modules. These are part of the Salt +package itself and don't require the user to do anything to use them. (Although +a number of them have additional dependencies and/or configuration.) + +The Salt File Server +-------------------- + +The user may add modules by simply placing them in special directories in their +:ref:`fileserver `. + +These will be eventually picked up by minions, or users can use +:py:mod:```saltutil.sync_*`` execution functions ` and +:py:mod:```saltutil.sync_*`` runner functions ` to force +it immediately. + +The name of the directory inside of the file server is the directory name prepended +by ``_``. + +Using saltenvs besides ``base`` may not work in all contexts. + +The extmods Directory +--------------------- + +Any files places in the directory set by ``extension_modules`` settings +(:conf_minion:`minion `, +:conf_master:`master `, default +``/var/cache/salt/*/extmods``) can also be loaded as modules. Note that these +directories are also used by the ``saltutil.sync_*`` functions (mentioned +above). + +Secondary Packages +------------------ + +Third-party packages may also add modules to Salt if they are installed in the +same system and Python environment as the Salt Minion or Master. + +This is done via setuptools entry points: + +.. code-block:: python + setup( + # ... + entry_points={ + 'salt.loader': [ + 'module_dirs=spirofs.loader:module', + ], + }, + # ... + ) + +Note that these are not synced from the Salt Master to the Minion. They must be +installed indepdendently on the Minion. + +Names +----- + +The specific names for each of these methods are as follows. See sections below +for a short summary of each of these systems. + +============ ================================================================ ========================= ===================== +Module Type Salt Package Name Directory Name Entry Point +============ ================================================================ ========================= ===================== +Auth ``salt.auth`` (:ref:`index `) ``auth`` [#no-fs]_ ``auth_dirs`` +Beacon ``salt.beacons`` (:ref:`index `) ``beacons`` ``beacons_dirs`` +Cache ``salt.cache`` (:ref:`index `) ``cache`` ``cache_dirs`` +Cloud ``salt.cloud.clouds`` (:ref:`index `) ``clouds`` ``cloud_dirs`` +Engine ``salt.engines`` (:ref:`index `) ``engines`` ``engines_dirs`` +Executor ``salt.executors`` (:ref:`index `) ``executors`` [#no-fs]_ ``executor_dirs`` +Execution ``salt.modules`` (:ref:`index `) ``modules`` ``module_dirs`` +File Server ``salt.fileserver`` (:ref:`index `) ``fileserver`` [#no-fs]_ ``fileserver_dirs`` +Grain ``salt.grains`` (:ref:`index `) ``grains`` ``grains_dirs`` +Log Handler ``salt.log.handlers`` (:ref:`index `) ``log_handlers`` ``log_handlers_dirs`` +Net API ``salt.netapi`` (:ref:`index `) ``netapi`` [#no-fs]_ ``netapi_dirs`` +Outputter ``salt.output`` (:ref:`index `) ``output`` ``outputter_dirs`` +Pillar ``salt.pillar`` (:ref:`index `) ``pillar`` ``pillar_dirs`` +Proxy ``salt.proxy`` (:ref:`index `) ``proxy`` ``proxy_dirs`` +Queue ``salt.queues`` (:ref:`index `) ``queues`` ``queue_dirs`` +Renderer ``salt.renderers`` (:ref:`index `) ``renderers`` ``render_dirs`` +Returner ``salt.returners`` (:ref:`index `) ``returners`` ``returner_dirs`` +Roster ``salt.roster`` (:ref:`index `) ``roster`` ``roster_dirs`` +Runner ``salt.runners`` (:ref:`index `) ``runners`` ``runner_dirs`` +SDB ``salt.sdb`` (:ref:`index `) ``sdb`` ``sdb_dirs`` +Search ``search`` [#no-fs]_ ``search_dirs`` +Serializer ``salt.serializers`` (:ref:`index `) ``serializers`` [#no-fs]_ ``serializers_dirs`` +SPM pkgdb ``pkgdb`` [#no-fs]_ ``pkgdb_dirs`` +SPM pkgfiles ``pkgfiles`` [#no-fs]_ ``pkgfiles_dirs`` +SSH Wrapper ``wrapper`` [#no-fs]_ ``wrapper_dirs`` +State ``salt.states`` (:ref:`index `) ``states`` ``states_dirs`` +Thorium ``salt.thorium`` (:ref:`index `) ``thorium`` [#no-fs]_ ``thorium_dirs`` +Top ``salt.tops`` (:ref:`index `) ``tops`` ``top_dirs`` +Util ``salt.utils`` ``utils`` ``utils_dirs`` +Wheel ``salt.wheels`` (:ref:`index `) ``wheel`` ``wheel_dirs`` +============ ================================================================ ========================= ===================== + +.. [#no-fs] This module cannot be loaded from the Salt File Server. + Execution Modules ================= From d44f5ce0fd25230e573b8016489aeb9269e462f1 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 03:00:11 -0500 Subject: [PATCH 04/26] Rephrase footnote --- doc/topics/development/modules/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 97664f4538a2..504d089a9b1e 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -118,7 +118,7 @@ Util ``salt.utils`` `` Wheel ``salt.wheels`` (:ref:`index `) ``wheel`` ``wheel_dirs`` ============ ================================================================ ========================= ===================== -.. [#no-fs] This module cannot be loaded from the Salt File Server. +.. [#no-fs] These modules cannot be loaded from the Salt File Server. Execution Modules ================= From 65e4199c254520f716db4547a215663c1e117362 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 03:21:47 -0500 Subject: [PATCH 05/26] I feel like it should be up there --- doc/topics/development/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/topics/development/index.rst b/doc/topics/development/index.rst index 02db13212b34..65a7e6ee861f 100644 --- a/doc/topics/development/index.rst +++ b/doc/topics/development/index.rst @@ -7,12 +7,12 @@ Developing Salt :glob: * + modules/index extend/index tests/* raet/index git/index conventions/index - modules/index ../../ref/internals/index ../projects/index ../tutorials/writing_tests From d4a8f7e47dbac8a7a1d4d2dc731e6ca0a8f70f25 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 03:27:51 -0500 Subject: [PATCH 06/26] Remove some code from some links --- doc/topics/development/modules/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 504d089a9b1e..501216c26c46 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -36,8 +36,8 @@ The user may add modules by simply placing them in special directories in their :ref:`fileserver `. These will be eventually picked up by minions, or users can use -:py:mod:```saltutil.sync_*`` execution functions ` and -:py:mod:```saltutil.sync_*`` runner functions ` to force +:py:mod:`saltutil.sync_* execution functions ` and +:py:mod:`saltutil.sync_* runner functions ` to force it immediately. The name of the directory inside of the file server is the directory name prepended From 205fea9cc30053aed70d075dd4cae77c37cbf858 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 03:36:20 -0500 Subject: [PATCH 07/26] Some more revising --- doc/topics/development/modules/index.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 501216c26c46..dfc763cedb0a 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -36,12 +36,11 @@ The user may add modules by simply placing them in special directories in their :ref:`fileserver `. These will be eventually picked up by minions, or users can use -:py:mod:`saltutil.sync_* execution functions ` and -:py:mod:`saltutil.sync_* runner functions ` to force -it immediately. +the ``saltutil.sync_*`` :py:mod:`execution functions ` +and :py:mod:`runner functions ` to force it immediately. -The name of the directory inside of the file server is the directory name prepended -by ``_``. +The name of the directory inside of the file server is the directory name +prepended by ``_``. Using saltenvs besides ``base`` may not work in all contexts. From 26468dfc751be51a6567840f54cd0fed422ccfc2 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 18:21:16 -0500 Subject: [PATCH 08/26] Add missing blank line --- doc/topics/development/modules/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index dfc763cedb0a..d0803fb5440e 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -63,6 +63,7 @@ same system and Python environment as the Salt Minion or Master. This is done via setuptools entry points: .. code-block:: python + setup( # ... entry_points={ From f647c1d74c1f36a672158dccd561b3652fd3dc38 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 19:33:12 -0500 Subject: [PATCH 09/26] Fill in some missing modules --- doc/topics/development/modules/index.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index d0803fb5440e..31ad805f8223 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -106,11 +106,11 @@ Returner ``salt.returners`` (:ref:`index `) `` Roster ``salt.roster`` (:ref:`index `) ``roster`` ``roster_dirs`` Runner ``salt.runners`` (:ref:`index `) ``runners`` ``runner_dirs`` SDB ``salt.sdb`` (:ref:`index `) ``sdb`` ``sdb_dirs`` -Search ``search`` [#no-fs]_ ``search_dirs`` +Search ``salt.search`` ``search`` [#no-fs]_ ``search_dirs`` Serializer ``salt.serializers`` (:ref:`index `) ``serializers`` [#no-fs]_ ``serializers_dirs`` -SPM pkgdb ``pkgdb`` [#no-fs]_ ``pkgdb_dirs`` -SPM pkgfiles ``pkgfiles`` [#no-fs]_ ``pkgfiles_dirs`` -SSH Wrapper ``wrapper`` [#no-fs]_ ``wrapper_dirs`` +SPM pkgdb ``salt.spm.pkgdb`` ``pkgdb`` [#no-fs]_ ``pkgdb_dirs`` +SPM pkgfiles ``salt.spm.pkgfiles`` ``pkgfiles`` [#no-fs]_ ``pkgfiles_dirs`` +SSH Wrapper ``salt.client.ssh`` ``wrapper`` [#no-fs]_ ``wrapper_dirs`` State ``salt.states`` (:ref:`index `) ``states`` ``states_dirs`` Thorium ``salt.thorium`` (:ref:`index `) ``thorium`` [#no-fs]_ ``thorium_dirs`` Top ``salt.tops`` (:ref:`index `) ``tops`` ``top_dirs`` From bed35095bee7c2a58773a716d594498242412a34 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 19:48:58 -0500 Subject: [PATCH 10/26] Refactor, add a TOC --- doc/topics/development/modules/developing.rst | 60 ++++++++++++++++ doc/topics/development/modules/index.rst | 72 ++++--------------- 2 files changed, 73 insertions(+), 59 deletions(-) create mode 100644 doc/topics/development/modules/developing.rst diff --git a/doc/topics/development/modules/developing.rst b/doc/topics/development/modules/developing.rst new file mode 100644 index 000000000000..75ec48e79cfa --- /dev/null +++ b/doc/topics/development/modules/developing.rst @@ -0,0 +1,60 @@ +====================== +Developing New Modules +====================== + +Interactive Debugging +===================== + +Sometimes debugging with ``print()`` and extra logs sprinkled everywhere is not +the best strategy. + +IPython is a helpful debug tool that has an interactive python environment +which can be embedded in python programs. + +First the system will require IPython to be installed. + +.. code-block:: bash + + # Debian + apt-get install ipython + + # Arch Linux + pacman -Syu ipython2 + + # RHEL/CentOS (via EPEL) + yum install python-ipython + + +Now, in the troubling python module, add the following line at a location where +the debugger should be started: + +.. code-block:: python + + test = 'test123' + import IPython; IPython.embed_kernel() + +After running a Salt command that hits that line, the following will show up in +the log file: + +.. code-block:: text + + [CRITICAL] To connect another client to this kernel, use: + [IPKernelApp] --existing kernel-31271.json + +Now on the system that invoked ``embed_kernel``, run the following command from +a shell: + +.. code-block:: bash + + # NOTE: use ipython2 instead of ipython for Arch Linux + ipython console --existing + +This provides a console that has access to all the vars and functions, and even +supports tab-completion. + +.. code-block:: python + + print(test) + test123 + +To exit IPython and continue running Salt, press ``Ctrl-d`` to logout. diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 31ad805f8223..0def3b9a270b 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -10,6 +10,17 @@ The most commonly used modular systems are execution modules and states. But the modular systems extend well beyond the more easily exposed components and are often added to Salt to make the complete system more flexible. +Additional Information +---------------------- + +.. toctree:: + :maxdepth: 1 + :glob: + + developing + dunder_dictionaries + + Loading Modules =============== @@ -140,65 +151,6 @@ For a list of all built in execution modules, click :ref:`here For information on writing execution modules, see :ref:`this page `. - -Interactive Debugging -===================== - -Sometimes debugging with ``print()`` and extra logs sprinkled everywhere is not -the best strategy. - -IPython is a helpful debug tool that has an interactive python environment -which can be embedded in python programs. - -First the system will require IPython to be installed. - -.. code-block:: bash - - # Debian - apt-get install ipython - - # Arch Linux - pacman -Syu ipython2 - - # RHEL/CentOS (via EPEL) - yum install python-ipython - - -Now, in the troubling python module, add the following line at a location where -the debugger should be started: - -.. code-block:: python - - test = 'test123' - import IPython; IPython.embed_kernel() - -After running a Salt command that hits that line, the following will show up in -the log file: - -.. code-block:: text - - [CRITICAL] To connect another client to this kernel, use: - [IPKernelApp] --existing kernel-31271.json - -Now on the system that invoked ``embed_kernel``, run the following command from -a shell: - -.. code-block:: bash - - # NOTE: use ipython2 instead of ipython for Arch Linux - ipython console --existing - -This provides a console that has access to all the vars and functions, and even -supports tab-completion. - -.. code-block:: python - - print(test) - test123 - -To exit IPython and continue running Salt, press ``Ctrl-d`` to logout. - - State Modules ============= @@ -243,6 +195,8 @@ function to execute. The default system outputter is the ``nested`` module. Pillar ====== +See also: :ref:`External Pillars ` + Used to define optional external pillar systems. The pillar generated via the filesystem pillar is passed into external pillars. This is commonly used as a bridge to database data for pillar, but is also the backend to the libvirt From 0dcd3419af066a0616e138eb34fd5f665f2f8320 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 20:02:46 -0500 Subject: [PATCH 11/26] Possible horrific use of toctrees? maybe --- doc/topics/development/modules/index.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 0def3b9a270b..005808449352 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -195,7 +195,11 @@ function to execute. The default system outputter is the ``nested`` module. Pillar ====== -See also: :ref:`External Pillars ` +.. toctree:: + :maxdepth: 1 + :glob: + + external_pillars Used to define optional external pillar systems. The pillar generated via the filesystem pillar is passed into external pillars. This is commonly used From 53154a0fe04e9750ec1474d4550f9cca447d78ea Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 20:43:40 -0500 Subject: [PATCH 12/26] Oops, missed that --- doc/topics/development/modules/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 005808449352..1868118a11bc 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -121,7 +121,7 @@ Search ``salt.search`` `` Serializer ``salt.serializers`` (:ref:`index `) ``serializers`` [#no-fs]_ ``serializers_dirs`` SPM pkgdb ``salt.spm.pkgdb`` ``pkgdb`` [#no-fs]_ ``pkgdb_dirs`` SPM pkgfiles ``salt.spm.pkgfiles`` ``pkgfiles`` [#no-fs]_ ``pkgfiles_dirs`` -SSH Wrapper ``salt.client.ssh`` ``wrapper`` [#no-fs]_ ``wrapper_dirs`` +SSH Wrapper ``salt.client.ssh.wrapper`` ``wrapper`` [#no-fs]_ ``wrapper_dirs`` State ``salt.states`` (:ref:`index `) ``states`` ``states_dirs`` Thorium ``salt.thorium`` (:ref:`index `) ``thorium`` [#no-fs]_ ``thorium_dirs`` Top ``salt.tops`` (:ref:`index `) ``tops`` ``top_dirs`` From 172d8c2902fa24dd62513ed1f72acc763b442756 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 20:56:43 -0500 Subject: [PATCH 13/26] Refactor headings --- doc/topics/development/modules/index.rst | 30 ++++++++++++------------ 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 1868118a11bc..c2962bcbbd1c 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -88,10 +88,10 @@ This is done via setuptools entry points: Note that these are not synced from the Salt Master to the Minion. They must be installed indepdendently on the Minion. -Names ------ +Module Types +============ -The specific names for each of these methods are as follows. See sections below +The specific names used by each loading method above are as follows. See sections below for a short summary of each of these systems. ============ ================================================================ ========================= ===================== @@ -132,7 +132,7 @@ Wheel ``salt.wheels`` (:ref:`index `) `` .. [#no-fs] These modules cannot be loaded from the Salt File Server. Execution Modules -================= +----------------- Execution modules make up the core of the functionality used by Salt to interact with client systems. The execution modules create the core system @@ -152,7 +152,7 @@ For information on writing execution modules, see :ref:`this page `. State Modules -============= +------------- State modules are used to define the state interfaces used by Salt States. These modules are restrictive in that they must follow a number of rules to @@ -165,35 +165,35 @@ function properly. state. Auth -==== +---- The auth module system allows for external authentication routines to be easily added into Salt. The `auth` function needs to be implemented to satisfy the requirements of an auth module. Use the ``pam`` module as an example. Fileserver -========== +---------- The fileserver module system is used to create fileserver backends used by the Salt Master. These modules need to implement the functions used in the fileserver subsystem. Use the ``gitfs`` module as an example. Grains -====== +------ Grain modules define extra routines to populate grains data. All defined public functions will be executed and MUST return a Python dict object. The dict keys will be added to the grains made available to the minion. Output -====== +------ The output modules supply the outputter system with routines to display data in the terminal. These modules are very simple and only require the `output` function to execute. The default system outputter is the ``nested`` module. Pillar -====== +------ .. toctree:: :maxdepth: 1 @@ -207,32 +207,32 @@ as a bridge to database data for pillar, but is also the backend to the libvirt state used to generate and sign libvirt certificates on the fly. Renderers -========= +--------- Renderers are the system used to render sls files into salt highdata for the state compiler. They can be as simple as the ``py`` renderer and as complex as ``stateconf`` and ``pydsl``. Returners -========= +--------- Returners are used to send data from minions to external sources, commonly databases. A full returner will implement all routines to be supported as an external job cache. Use the ``redis`` returner as an example. Runners -======= +------- Runners are purely master-side execution sequences. Tops -==== +---- Tops modules are used to convert external data sources into top file data for the state system. Wheel -===== +----- The wheel system is used to manage master side management routines. These routines are primarily intended for the API to enable master configuration. From 40f9b16530daca08ee96047daf3e26533238e2b2 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 21:22:57 -0500 Subject: [PATCH 14/26] Add missing module type sections --- doc/topics/development/modules/index.rst | 129 +++++++++++++++++------ 1 file changed, 97 insertions(+), 32 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index c2962bcbbd1c..a12365ab03a4 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -95,15 +95,15 @@ The specific names used by each loading method above are as follows. See section for a short summary of each of these systems. ============ ================================================================ ========================= ===================== -Module Type Salt Package Name Directory Name Entry Point +Module Type Salt Package Name FS/Directory Name Entry Point ============ ================================================================ ========================= ===================== Auth ``salt.auth`` (:ref:`index `) ``auth`` [#no-fs]_ ``auth_dirs`` Beacon ``salt.beacons`` (:ref:`index `) ``beacons`` ``beacons_dirs`` Cache ``salt.cache`` (:ref:`index `) ``cache`` ``cache_dirs`` Cloud ``salt.cloud.clouds`` (:ref:`index `) ``clouds`` ``cloud_dirs`` Engine ``salt.engines`` (:ref:`index `) ``engines`` ``engines_dirs`` -Executor ``salt.executors`` (:ref:`index `) ``executors`` [#no-fs]_ ``executor_dirs`` Execution ``salt.modules`` (:ref:`index `) ``modules`` ``module_dirs`` +Executor ``salt.executors`` (:ref:`index `) ``executors`` [#no-fs]_ ``executor_dirs`` File Server ``salt.fileserver`` (:ref:`index `) ``fileserver`` [#no-fs]_ ``fileserver_dirs`` Grain ``salt.grains`` (:ref:`index `) ``grains`` ``grains_dirs`` Log Handler ``salt.log.handlers`` (:ref:`index `) ``log_handlers`` ``log_handlers_dirs`` @@ -131,8 +131,33 @@ Wheel ``salt.wheels`` (:ref:`index `) `` .. [#no-fs] These modules cannot be loaded from the Salt File Server. -Execution Modules ------------------ +Auth +---- + +The auth module system allows for external authentication routines to be easily +added into Salt. The `auth` function needs to be implemented to satisfy the +requirements of an auth module. Use the ``pam`` module as an example. + +Beacon +------ + +Cache +----- + +Cloud +----- + +Engine +------ + +Execution +--------- + +.. toctree:: + :maxdepth: 1 + :glob: + + /ref/modules/index Execution modules make up the core of the functionality used by Salt to interact with client systems. The execution modules create the core system @@ -145,46 +170,31 @@ detecting information about the system. The only restraint in execution modules is that the defined functions always return a JSON serializable object. -For a list of all built in execution modules, click :ref:`here -` - -For information on writing execution modules, see :ref:`this page -`. - -State Modules -------------- +Executor +-------- -State modules are used to define the state interfaces used by Salt States. -These modules are restrictive in that they must follow a number of rules to -function properly. - -.. note:: - - State modules define the available routines in sls files. If calling - an execution module directly is desired, take a look at the `module` - state. - -Auth ----- - -The auth module system allows for external authentication routines to be easily -added into Salt. The `auth` function needs to be implemented to satisfy the -requirements of an auth module. Use the ``pam`` module as an example. - -Fileserver ----------- +File Server +----------- -The fileserver module system is used to create fileserver backends used by the +The file server module system is used to create file server backends used by the Salt Master. These modules need to implement the functions used in the fileserver subsystem. Use the ``gitfs`` module as an example. Grains ------ +* :ref:`writing-grains` + Grain modules define extra routines to populate grains data. All defined public functions will be executed and MUST return a Python dict object. The dict keys will be added to the grains made available to the minion. +Log Handler +----------- + +Net API +------- + Output ------ @@ -206,6 +216,9 @@ the filesystem pillar is passed into external pillars. This is commonly used as a bridge to database data for pillar, but is also the backend to the libvirt state used to generate and sign libvirt certificates on the fly. +Proxy +----- + Renderers --------- @@ -220,17 +233,69 @@ Returners are used to send data from minions to external sources, commonly databases. A full returner will implement all routines to be supported as an external job cache. Use the ``redis`` returner as an example. +Roster +------ + Runners ------- +.. toctree:: + :maxdepth: 1 + :glob: + + /ref/runners/index + Runners are purely master-side execution sequences. +SDB +--- + +Search +------ + +Serializer +---------- + +State +----- + +.. toctree:: + :maxdepth: 1 + :glob: + + /ref/states/index + +State modules are used to define the state interfaces used by Salt States. +These modules are restrictive in that they must follow a number of rules to +function properly. + +.. note:: + + State modules define the available routines in sls files. If calling + an execution module directly is desired, take a look at the `module` + state. + +SPM pkgdb +--------- + +SPM pkgfiles +------------ + +SSH Wrapper +----------- + +Thorium +------- + Tops ---- Tops modules are used to convert external data sources into top file data for the state system. +Util +---- + Wheel ----- From 705e7e460f31a7dabf39c0da496c66e626082600 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 21:37:32 -0500 Subject: [PATCH 15/26] Add a bunch of cross linking --- doc/ref/file_server/dynamic-modules.rst | 11 +++-------- doc/topics/beacons/index.rst | 6 ++++-- doc/topics/development/modules/index.rst | 10 ++++++++++ 3 files changed, 17 insertions(+), 10 deletions(-) diff --git a/doc/ref/file_server/dynamic-modules.rst b/doc/ref/file_server/dynamic-modules.rst index 17bf149685fd..6913824c6045 100644 --- a/doc/ref/file_server/dynamic-modules.rst +++ b/doc/ref/file_server/dynamic-modules.rst @@ -16,19 +16,14 @@ be used. The directories are prepended with an underscore: -- :file:`_beacons` -- :file:`_clouds` -- :file:`_engines` - :file:`_grains` - :file:`_modules` -- :file:`_output` -- :file:`_proxy` -- :file:`_renderers` -- :file:`_returners` - :file:`_states` -- :file:`_tops` +- :file:`_runners` - :file:`_utils` +See :ref:`Modular Systems ` for the complete list. + The contents of these directories need to be synced over to the minions after Python modules have been created in them. There are a number of ways to sync the modules. diff --git a/doc/topics/beacons/index.rst b/doc/topics/beacons/index.rst index bc9ba7a5355d..d6996834c13f 100644 --- a/doc/topics/beacons/index.rst +++ b/doc/topics/beacons/index.rst @@ -283,6 +283,8 @@ All beacons are configured using a similar process of enabling the beacon, writing a reactor SLS (and state SLS if needed), and mapping a beacon event to the reactor SLS. +.. _writing-beacons: + Writing Beacon Plugins ====================== @@ -353,5 +355,5 @@ new execution modules and functions to back specific beacons. Distributing Custom Beacons --------------------------- -Custom beacons can be distributed to minions using ``saltutil``, see -:ref:`Dynamic Module Distribution `. +Custom beacons can be distributed to minions via the standard methods, see +:ref:`Modular Systems `. diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index a12365ab03a4..5d6ac97ddd32 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -1,3 +1,5 @@ +.. _modular-systems: + =============== Modular Systems =============== @@ -43,6 +45,8 @@ a number of them have additional dependencies and/or configuration.) The Salt File Server -------------------- +* :ref:`Dynamic Module Distribution ` + The user may add modules by simply placing them in special directories in their :ref:`fileserver `. @@ -94,6 +98,8 @@ Module Types The specific names used by each loading method above are as follows. See sections below for a short summary of each of these systems. +.. _module-name-table: + ============ ================================================================ ========================= ===================== Module Type Salt Package Name FS/Directory Name Entry Point ============ ================================================================ ========================= ===================== @@ -141,6 +147,10 @@ requirements of an auth module. Use the ``pam`` module as an example. Beacon ------ +* :ref:`Writing Beacons ` + +Beacons are polled event generators. + Cache ----- From d06739a6474a18667037377f776f02a8b4616334 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Mon, 26 Nov 2018 22:25:23 -0500 Subject: [PATCH 16/26] Add short writeups for a bunch of systems. --- doc/topics/development/modules/index.rst | 25 +++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 5d6ac97ddd32..4f08b368760f 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -149,17 +149,27 @@ Beacon * :ref:`Writing Beacons ` -Beacons are polled event generators. +Beacons are polled by the Salt event loop to monitor non-salt processes. See +:ref`Beacons ` for more information about the beacon system. Cache ----- +The minion cache is used by the master to store various information about +minions. See :ref:`Minion Data Cache ` for more information. + Cloud ----- +Cloud modules are backend implementations used by :ref:`Salt Cloud `. + Engine ------ +Engines are open-ended services managed by the Salt daemon (both master and +minion). They may interact with event loop, call other modules, or a variety of +non-salt tasks. See :ref:`Salt Engines ` for complete details. + Execution --------- @@ -183,6 +193,9 @@ object. Executor -------- +Executors control how execution modules get called. The default is to just call +them, but this can be customized. + File Server ----------- @@ -202,9 +215,14 @@ dict keys will be added to the grains made available to the minion. Log Handler ----------- +Log handlers allows the logs from salt (master or minion) to be sent to log +aggregation systems. + Net API ------- +Net API modules are the actual server implementation used by Salt API. + Output ------ @@ -229,6 +247,9 @@ state used to generate and sign libvirt certificates on the fly. Proxy ----- +:ref:`Proxy Minions ` are a way to manage devices that cannot run +a full minion directly. + Renderers --------- @@ -246,6 +267,8 @@ external job cache. Use the ``redis`` returner as an example. Roster ------ +The :ref:`Roster system ` is used by Salt SSH to enumerate devices. + Runners ------- From 3a9ab1b5f541cb67b2eb587d6ce8437b1d3369ec Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Wed, 28 Nov 2018 17:53:14 -0500 Subject: [PATCH 17/26] Merge dunders into developing, expand to cover all dunders. --- doc/topics/development/modules/developing.rst | 175 ++++++++++++++++++ .../modules/dunder_dictionaries.rst | 119 ------------ 2 files changed, 175 insertions(+), 119 deletions(-) delete mode 100644 doc/topics/development/modules/dunder_dictionaries.rst diff --git a/doc/topics/development/modules/developing.rst b/doc/topics/development/modules/developing.rst index 75ec48e79cfa..2609eeda2ffe 100644 --- a/doc/topics/development/modules/developing.rst +++ b/doc/topics/development/modules/developing.rst @@ -58,3 +58,178 @@ supports tab-completion. test123 To exit IPython and continue running Salt, press ``Ctrl-d`` to logout. + +Special Module Contents +======================= + +These are things that may be defined by the module to influence various things. + +__virtual__ +----------- + +__virtual_aliases__ +------------------- + +__virtualname__ +--------------- + +__init__ +-------- + +Called before ``__virtual__()`` + +__proxyenabled__ +---------------- +grains and proxy modules + +__proxyenabled__ as a list containing the names of the proxy types that the module supports. + +__load__ +-------- + +__func_alias__ +-------------- + +__outputter__ +------------- + +Dunder Dictionaries +=================== + +Salt provides several special "dunder" dictionaries as a convenience for Salt +development. These include ``__opts__``, ``__context__``, ``__salt__``, and +others. This document will describe each dictionary and detail where they exist +and what information and/or functionality they provide. + +The following dunder dictionaries are always defined, but may be empty + +* ``__context__`` +* ``__grains__`` +* ``__pillar__`` +* ``__opts__`` + + +__opts__ +-------- + +Defined in: All modules + +The ``__opts__`` dictionary contains all of the options passed in the +configuration file for the master or minion. + +.. note:: + + In many places in salt, instead of pulling raw data from the __opts__ + dict, configuration data should be pulled from the salt `get` functions + such as config.get, aka - ``__salt__['config.get']('foo:bar')`` + The `get` functions also allow for dict traversal via the *:* delimiter. + Consider using get functions whenever using ``__opts__`` or ``__pillar__`` + and ``__grains__`` (when using grains for configuration data) + +The configuration file data made available in the ``__opts__`` dictionary is the +configuration data relative to the running daemon. If the modules are loaded and +executed by the master, then the master configuration data is available, if the +modules are executed by the minion, then the minion configuration is +available. Any additional information passed into the respective configuration +files is made available + +__salt__ +-------- + +Defined in: Auth, Beacons, Engines, Execution, Executors, Outputters, Pillars, +Proxies, Renderers, Returners, Runners, SDB, SSH Wrappers, State, Thorium + +``__salt__`` contains the execution module functions. This allows for all +functions to be called as they have been set up by the salt loader. + +.. code-block:: python + + __salt__['cmd.run']('fdisk -l') + __salt__['network.ip_addrs']() + +.. note:: + + When used in runners or outputters, ``__salt__`` references other + runner/outputter modules, and not execution modules. + +__grains__ +---------- + +Filled in for: Execution, Pillar, Renderer, Returner, SSH Wrapper, State. + +The ``__grains__`` dictionary contains the grains data generated by the minion +that is currently being worked with. In execution modules, state modules and +returners this is the grains of the minion running the calls, when generating +the external pillar the ``__grains__`` is the grains data from the minion that +the pillar is being generated for. + +While ``__grains__`` is defined for every module, it's only filled in for some. + +__pillar__ +----------- + +Filled in for: Execution, Returner, SSH Wrapper, State + +The ``__pillar__`` dictionary contains the pillar for the respective minion. + +While ``__pillar__`` is defined for every module, it's only filled in for some. + +__context__ +----------- + +During a state run the ``__context__`` dictionary persists across all states +that are run and then is destroyed when the state ends. + +When running an execution module ``__context__`` persists across all module +executions until the modules are refreshed; such as when +:py:func:`saltutil.sync_all ` or +:py:func:`state.apply ` are executed. + +A great place to see how to use ``__context__`` is in the cp.py module in +salt/modules/cp.py. The fileclient authenticates with the master when it is +instantiated and then is used to copy files to the minion. Rather than create a +new fileclient for each file that is to be copied down, one instance of the +fileclient is instantiated in the ``__context__`` dictionary and is reused for +each file. Here is an example from salt/modules/cp.py: + +.. code-block:: python + + if not 'cp.fileclient' in __context__: + __context__['cp.fileclient'] = salt.fileclient.get_file_client(__opts__) + + +.. note:: Because __context__ may or may not have been destroyed, always be + sure to check for the existence of the key in __context__ and + generate the key before using it. + +__utils__ +--------- +Defined in: Cloud, Engine, Execution, File Server, Pillar, Proxy, Runner, SDB. + +__proxy__ +--------- +Defined in: Beacon, Engine, Execution, Executor, Proxy, Renderer, Returner, State, Util + +__runners__ +----------- +Defined in: Engine, Roster, Thorium + +__ret__ +------- +Defined in: Proxy, Search + +__thorium__ +----------- +Defined in: Thorium + +__states__ +---------- +Defined in: Renderers, State + +__serializers__ +--------------- +Defined in: State + +__sdb__ +------- +Defined in: SDB diff --git a/doc/topics/development/modules/dunder_dictionaries.rst b/doc/topics/development/modules/dunder_dictionaries.rst deleted file mode 100644 index 64df937a1523..000000000000 --- a/doc/topics/development/modules/dunder_dictionaries.rst +++ /dev/null @@ -1,119 +0,0 @@ -=================== -Dunder Dictionaries -=================== - -Salt provides several special "dunder" dictionaries as a convenience for Salt -development. These include ``__opts__``, ``__context__``, ``__salt__``, and -others. This document will describe each dictionary and detail where they exist -and what information and/or functionality they provide. - - -__opts__ --------- - -Available in -~~~~~~~~~~~~ - -- All loader modules - -The ``__opts__`` dictionary contains all of the options passed in the -configuration file for the master or minion. - -.. note:: - - In many places in salt, instead of pulling raw data from the __opts__ - dict, configuration data should be pulled from the salt `get` functions - such as config.get, aka - __salt__['config.get']('foo:bar') - The `get` functions also allow for dict traversal via the *:* delimiter. - Consider using get functions whenever using __opts__ or __pillar__ and - __grains__ (when using grains for configuration data) - -The configuration file data made available in the ``__opts__`` dictionary is the -configuration data relative to the running daemon. If the modules are loaded and -executed by the master, then the master configuration data is available, if the -modules are executed by the minion, then the minion configuration is -available. Any additional information passed into the respective configuration -files is made available - -__salt__ --------- - -Available in -~~~~~~~~~~~~ - -- Execution Modules -- State Modules -- Returners -- Runners - -``__salt__`` contains the execution module functions. This allows for all -functions to be called as they have been set up by the salt loader. - -.. code-block:: python - - __salt__['cmd.run']('fdisk -l') - __salt__['network.ip_addrs']() - -.. note:: - - When used in runners, ``__salt__`` references other runner modules, and not - execution modules. - -__grains__ ----------- - -Available in -~~~~~~~~~~~~ - -- Execution Modules -- State Modules -- Returners -- External Pillar - -The ``__grains__`` dictionary contains the grains data generated by the minion -that is currently being worked with. In execution modules, state modules and -returners this is the grains of the minion running the calls, when generating -the external pillar the ``__grains__`` is the grains data from the minion that -the pillar is being generated for. - -__pillar__ ------------ - -Available in -~~~~~~~~~~~~ - -- Execution Modules -- State Modules -- Returners - -The ``__pillar__`` dictionary contains the pillar for the respective minion. - -__context__ ------------ - -``__context__`` exists in state modules and execution modules. - -During a state run the ``__context__`` dictionary persists across all states -that are run and then is destroyed when the state ends. - -When running an execution module ``__context__`` persists across all module -executions until the modules are refreshed; such as when -:py:func:`saltutil.sync_all ` or -:py:func:`state.apply ` are executed. - -A great place to see how to use ``__context__`` is in the cp.py module in -salt/modules/cp.py. The fileclient authenticates with the master when it is -instantiated and then is used to copy files to the minion. Rather than create a -new fileclient for each file that is to be copied down, one instance of the -fileclient is instantiated in the ``__context__`` dictionary and is reused for -each file. Here is an example from salt/modules/cp.py: - -.. code-block:: python - - if not 'cp.fileclient' in __context__: - __context__['cp.fileclient'] = salt.fileclient.get_file_client(__opts__) - - -.. note:: Because __context__ may or may not have been destroyed, always be - sure to check for the existence of the key in __context__ and - generate the key before using it. From bd3796a9c2f781ff68c61d3f89f8c5b0d3a85223 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Wed, 28 Nov 2018 18:20:46 -0500 Subject: [PATCH 18/26] Update TOC --- doc/topics/development/modules/index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 4f08b368760f..d3656f25e428 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -16,11 +16,10 @@ Additional Information ---------------------- .. toctree:: - :maxdepth: 1 + :maxdepth: 2 :glob: developing - dunder_dictionaries Loading Modules From 1bdd49d43acb38f487d7d7d55637743a717951df Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Thu, 29 Nov 2018 22:49:39 -0500 Subject: [PATCH 19/26] Add more links and write some system summaries --- doc/topics/development/modules/index.rst | 16 ++++++++++++++++ doc/topics/sdb/index.rst | 1 + 2 files changed, 17 insertions(+) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index d3656f25e428..8237f233d24e 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -143,6 +143,9 @@ The auth module system allows for external authentication routines to be easily added into Salt. The `auth` function needs to be implemented to satisfy the requirements of an auth module. Use the ``pam`` module as an example. +See :ref:`External Authentication System ` for more about +authentication in Salt. + Beacon ------ @@ -202,6 +205,8 @@ The file server module system is used to create file server backends used by the Salt Master. These modules need to implement the functions used in the fileserver subsystem. Use the ``gitfs`` module as an example. +See :ref:`File Server Backends ` for more information. + Grains ------ @@ -211,6 +216,8 @@ Grain modules define extra routines to populate grains data. All defined public functions will be executed and MUST return a Python dict object. The dict keys will be added to the grains made available to the minion. +See :ref:`Grains ` for more. + Log Handler ----------- @@ -282,12 +289,21 @@ Runners are purely master-side execution sequences. SDB --- +* :ref:`Writing SDB Modules ` + +SDB is a way to store data that's not associated with a minion. See +:ref:`Storing Data in Other Databases `. + Search ------ +A system for indexing the file server and pillars. Removed in 2018.3. + Serializer ---------- +Primarily used with :py:func:`file.serialize `. + State ----- diff --git a/doc/topics/sdb/index.rst b/doc/topics/sdb/index.rst index c4d94120e2c7..4c670613ef4e 100644 --- a/doc/topics/sdb/index.rst +++ b/doc/topics/sdb/index.rst @@ -154,6 +154,7 @@ When writing Salt modules, it is not recommended to call ``sdb.get`` directly, as it requires the user to provide values in SDB, using a specific URI. Use ``config.get`` instead. +.. _sdb-writing-modules: Writing SDB Modules =================== From 98da110be264408c394a2ea9a590353a28c3e174 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Fri, 30 Nov 2018 18:21:12 -0500 Subject: [PATCH 20/26] Migrate Dynamic Module Distribution to Modular Systems. --- doc/ref/file_server/dynamic-modules.rst | 52 ------------------------ doc/topics/development/modules/index.rst | 44 ++++++++++++++------ 2 files changed, 32 insertions(+), 64 deletions(-) delete mode 100644 doc/ref/file_server/dynamic-modules.rst diff --git a/doc/ref/file_server/dynamic-modules.rst b/doc/ref/file_server/dynamic-modules.rst deleted file mode 100644 index 6913824c6045..000000000000 --- a/doc/ref/file_server/dynamic-modules.rst +++ /dev/null @@ -1,52 +0,0 @@ -.. _module-sync: -.. _dynamic-module-distribution: - -=========================== -Dynamic Module Distribution -=========================== - -.. versionadded:: 0.9.5 - -Custom Salt execution, state, and other modules can be distributed to Salt -minions using the Salt file server. - -Under the root of any environment defined via the :conf_master:`file_roots` -option on the master server directories corresponding to the type of module can -be used. - -The directories are prepended with an underscore: - -- :file:`_grains` -- :file:`_modules` -- :file:`_states` -- :file:`_runners` -- :file:`_utils` - -See :ref:`Modular Systems ` for the complete list. - -The contents of these directories need to be synced over to the minions after -Python modules have been created in them. There are a number of ways to sync -the modules. - -Sync Via States -=============== - -The minion configuration contains an option ``autoload_dynamic_modules`` -which defaults to ``True``. This option makes the state system refresh all -dynamic modules when states are run. To disable this behavior set -:conf_minion:`autoload_dynamic_modules` to ``False`` in the minion config. - -When dynamic modules are autoloaded via states, modules only pertinent to -the environments matched in the master's top file are downloaded. - -This is important to remember, because modules can be manually loaded from -any specific environment that environment specific modules will be loaded -when a state run is executed. - -Sync Via the saltutil Module -============================ - -The saltutil module has a number of functions that can be used to sync all -or specific dynamic modules. The saltutil module function ``saltutil.sync_all`` -will sync all module types over to a minion. For more information see: -:mod:`salt.modules.saltutil` diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 8237f233d24e..b494d5a9cb58 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -12,9 +12,6 @@ The most commonly used modular systems are execution modules and states. But the modular systems extend well beyond the more easily exposed components and are often added to Salt to make the complete system more flexible. -Additional Information ----------------------- - .. toctree:: :maxdepth: 2 :glob: @@ -44,19 +41,42 @@ a number of them have additional dependencies and/or configuration.) The Salt File Server -------------------- -* :ref:`Dynamic Module Distribution ` - The user may add modules by simply placing them in special directories in their :ref:`fileserver `. -These will be eventually picked up by minions, or users can use -the ``saltutil.sync_*`` :py:mod:`execution functions ` -and :py:mod:`runner functions ` to force it immediately. - The name of the directory inside of the file server is the directory name -prepended by ``_``. +prepended by underscore, such as: + +- :file:`_grains` +- :file:`_modules` +- :file:`_states` + +Modules must be synced before they can be used. This can happen a few ways, +discussed below. + +.. note: + Using saltenvs besides ``base`` may not work in all contexts. + +Sync Via States +~~~~~~~~~~~~~~~ + +The minion configuration contains an option ``autoload_dynamic_modules`` +which defaults to ``True``. This option makes the state system refresh all +dynamic modules when states are run. To disable this behavior set +:conf_minion:`autoload_dynamic_modules` to ``False`` in the minion config. + +When dynamic modules are autoloaded via states, only the modules defined in the +same saltenvs as the states currently being run. + +Sync Via the saltutil Module +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The saltutil module has a number of functions that can be used to sync all +or specific dynamic modules. The ``saltutil.sync_*`` +:py:mod:`execution functions ` and +:py:mod:`runner functions ` can be used to sync modules +to minions and the master, respectively. -Using saltenvs besides ``base`` may not work in all contexts. The extmods Directory --------------------- @@ -66,7 +86,7 @@ Any files places in the directory set by ``extension_modules`` settings :conf_master:`master `, default ``/var/cache/salt/*/extmods``) can also be loaded as modules. Note that these directories are also used by the ``saltutil.sync_*`` functions (mentioned -above). +above) and files may be overwritten. Secondary Packages ------------------ From 374d0e9745d65404bade25f007c35b8b660dde50 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Fri, 30 Nov 2018 19:14:35 -0500 Subject: [PATCH 21/26] Finish the last sections --- doc/topics/development/modules/developing.rst | 2 ++ doc/topics/development/modules/index.rst | 17 ++++++++++++++++- doc/topics/spm/dev.rst | 3 +++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/doc/topics/development/modules/developing.rst b/doc/topics/development/modules/developing.rst index 2609eeda2ffe..a01137d744a7 100644 --- a/doc/topics/development/modules/developing.rst +++ b/doc/topics/development/modules/developing.rst @@ -93,6 +93,8 @@ __func_alias__ __outputter__ ------------- +.. _dunder-dictionaries: + Dunder Dictionaries =================== diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index b494d5a9cb58..dc12e97d2ec7 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -172,7 +172,7 @@ Beacon * :ref:`Writing Beacons ` Beacons are polled by the Salt event loop to monitor non-salt processes. See -:ref`Beacons ` for more information about the beacon system. +:ref:`Beacons ` for more information about the beacon system. Cache ----- @@ -346,15 +346,27 @@ function properly. SPM pkgdb --------- +* :ref:`SPM Development Guide: Package Database ` + +pkgdb modules provides storage backends to the package database. + SPM pkgfiles ------------ +* :ref:`SPM Development Guide: Package Database ` + +pkgfiles modules handle the actual installation. + SSH Wrapper ----------- +Replacement execution modules for :ref:`Salt SSH `. + Thorium ------- +Modules for use in the :ref:`Thorium ` event reactor. + Tops ---- @@ -364,6 +376,9 @@ the state system. Util ---- +Just utility modules to use with other modules via ``__utils__`` (see +:ref:`Dunder Dictionaries `). + Wheel ----- diff --git a/doc/topics/spm/dev.rst b/doc/topics/spm/dev.rst index 11ad83364c1d..d6ed99aea372 100644 --- a/doc/topics/spm/dev.rst +++ b/doc/topics/spm/dev.rst @@ -20,6 +20,7 @@ marked as required are crucial to SPM's core functionality, while arguments that are marked as optional are provided as a benefit to the module, if it needs to use them. +.. _spm-development-pkgdb: Package Database ---------------- @@ -146,6 +147,8 @@ The only argument that is expected is ``db_``, which is the package database file. +.. _spm-development-pkgfiles: + Package Files ------------- By default, package files are installed using the ``local`` module. This module From 678958e55972181d272a881f60d4b2c7e7823222 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Fri, 30 Nov 2018 19:21:54 -0500 Subject: [PATCH 22/26] Editing --- doc/topics/development/modules/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index dc12e97d2ec7..f1a27d239b08 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -108,8 +108,8 @@ This is done via setuptools entry points: # ... ) -Note that these are not synced from the Salt Master to the Minion. They must be -installed indepdendently on the Minion. +Note that these are not synced from the Salt Master to the Minions. They must be +installed indepdendently on each Minion. Module Types ============ From 318ca48127fae0e9fdf2fb52bd0eff83e9f688be Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Tue, 4 Dec 2018 01:35:27 -0500 Subject: [PATCH 23/26] Make a note to not import modules --- doc/topics/development/modules/index.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index f1a27d239b08..7ebf21829bbb 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -156,6 +156,13 @@ Wheel ``salt.wheels`` (:ref:`index `) `` .. [#no-fs] These modules cannot be loaded from the Salt File Server. +.. note: + While it is possible to import modules directly with the import statement, + it is strongly recommended that the appropriate + :ref:`dunder dictionary ` is used to access them + instead. This is because a number of factors affect module names, module + selection, and module overloading. + Auth ---- From 7cb1a87fe6cca9a386fb848948f83d99b97619a1 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Tue, 4 Dec 2018 02:08:39 -0500 Subject: [PATCH 24/26] Add a quick index of loader-related settings --- .../development/modules/configuration.rst | 25 +++++++++++++++++++ doc/topics/development/modules/index.rst | 1 + 2 files changed, 26 insertions(+) create mode 100644 doc/topics/development/modules/configuration.rst diff --git a/doc/topics/development/modules/configuration.rst b/doc/topics/development/modules/configuration.rst new file mode 100644 index 000000000000..154e76695201 --- /dev/null +++ b/doc/topics/development/modules/configuration.rst @@ -0,0 +1,25 @@ +===================== +Configuration Options +===================== + +A number of configuration options can affect the load process. This is a quick +list of them: + +* ``autoload_dynamic_modules`` (:conf_minion:`Minion `) +* ``cython_enable`` (:conf_minion:`Minion `, :conf_master:`Master `) +* ``disable_modules`` (:conf_minion:`Minion `) +* ``disable_returners`` (:conf_minion:`Minion `) +* ``enable_zip_modules`` (:conf_minion:`Minion `) +* ``extension_modules`` (:conf_master:`Master `) +* ``extmod_whitelist`` (:conf_minion:`Minion `, :conf_master:`Master `) +* ``extmode_blacklist`` (:conf_minion:`Minion `) +* ``whitelist_modules`` (:conf_minion:`Minion <>`) +* ``grains_dirs`` (:conf_minion:`Minion `) +* ``module_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `) +* ``outputter_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `) +* ``providers`` (:conf_minion:`Minion `) +* ``render_dirs`` (:conf_minion:`Minion `) +* ``returner_dirs`` (:conf_minion:`Minion `) +* ``runner_dirs`` (:conf_master:`Master `) +* ``states_dirs`` (:conf_minion:`Minion `) +* ``utils_dirs`` (:conf_minion:`Minion `) diff --git a/doc/topics/development/modules/index.rst b/doc/topics/development/modules/index.rst index 7ebf21829bbb..e421c91f3130 100644 --- a/doc/topics/development/modules/index.rst +++ b/doc/topics/development/modules/index.rst @@ -17,6 +17,7 @@ and are often added to Salt to make the complete system more flexible. :glob: developing + configuration Loading Modules From 5ca1f8e2a53984eb56714aaddf9c11cbbfe761b4 Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Tue, 4 Dec 2018 04:20:43 -0500 Subject: [PATCH 25/26] Typos --- doc/topics/development/modules/configuration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/topics/development/modules/configuration.rst b/doc/topics/development/modules/configuration.rst index 154e76695201..42ee84535222 100644 --- a/doc/topics/development/modules/configuration.rst +++ b/doc/topics/development/modules/configuration.rst @@ -12,8 +12,8 @@ list of them: * ``enable_zip_modules`` (:conf_minion:`Minion `) * ``extension_modules`` (:conf_master:`Master `) * ``extmod_whitelist`` (:conf_minion:`Minion `, :conf_master:`Master `) -* ``extmode_blacklist`` (:conf_minion:`Minion `) -* ``whitelist_modules`` (:conf_minion:`Minion <>`) +* ``extmod_blacklist`` (:conf_minion:`Minion `, :conf_master:`Master `) +* ``whitelist_modules`` (:conf_minion:`Minion `) * ``grains_dirs`` (:conf_minion:`Minion `) * ``module_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `) * ``outputter_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `) From 03e31688a185605f7e92e4daac6badf4e0734aeb Mon Sep 17 00:00:00 2001 From: Jamie Bliss Date: Tue, 4 Dec 2018 04:31:47 -0500 Subject: [PATCH 26/26] whitelist_modules is linked via enable_whitelist_modules for some reason? idk --- doc/topics/development/modules/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/topics/development/modules/configuration.rst b/doc/topics/development/modules/configuration.rst index 42ee84535222..c794830f8027 100644 --- a/doc/topics/development/modules/configuration.rst +++ b/doc/topics/development/modules/configuration.rst @@ -13,7 +13,7 @@ list of them: * ``extension_modules`` (:conf_master:`Master `) * ``extmod_whitelist`` (:conf_minion:`Minion `, :conf_master:`Master `) * ``extmod_blacklist`` (:conf_minion:`Minion `, :conf_master:`Master `) -* ``whitelist_modules`` (:conf_minion:`Minion `) +* ``whitelist_modules`` (:conf_minion:`Minion `) * ``grains_dirs`` (:conf_minion:`Minion `) * ``module_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `) * ``outputter_dirs`` (:conf_minion:`Minion `, :conf_master:`Master `)