Merge remote-tracking branch 'upstream/develop' into feature_plugin_t…

…asks_files_gitlab

.travis.yml

@@ -23,7 +23,7 @@ deploy:
   script: poetry publish
   on:
     tags: true
-    condition: "$TRAVIS_PYTHON_VERSION == 3.7"
+    condition: "$PYTHON == 3.7"
 after_success:
 - coveralls
 env:

CHANGELOG.rst

@@ -1,14 +1,54 @@
+2.1.1 - March 19 2019
+=====================
+
+* [MISC] Workaround to sdispater/poetry#743 #358
+* [MISC] Fix automated deployment to pypi #358
+
+Thanks to the following people for their contributions:
+
+* `dbarrosop <https://github.com/dbarrosop>`_
+
+2.1.0 - March 18 2019
+=====================
+
+* [CORE_ENHANCEMENTS] inventory's transform function supports options #292
+* [CORE_ENHANCEMENTS] minor improvements to tests #293 #296 #306 #307 #312 #337
+* [CORE_ENHANCEMENTS] mypy improvements #308
+* [CORE_ENHANCEMENTS] expand user home when deserializing configuration #304
+* [CORE_ENHANCEMENTS] fix order of preference when deserializing config #309
+* [CORE_ENHANCEMENTS] fix and deprecate dict() function #314
+* [CORE_ENHANCEMENTS] migrate to poetry #315
+* [CORE_ENHANCEMENTS] Improve logging #316
+* [CORE_BUGFIX] (windows only) fix issue #319 - ascii color codes appear instead of color in output #320 #323
+* [PLUGIN_IMPROVEMENT] napalm and netmiko plugins support now reading ssh configuration from file #298
+* [PLUGIN_BUGFIX] fix paramiko chan.recv_exit_status() call order #313
+* [PLUGIN_BUGFIX] temporary fix for enum34 and netmiko-poetry issue #322
+* [PLUGIN_IMPROVEMENT] Print OrderDicts nicely in print_result #345
+* [DOCS] Various improvements #303 #305 #310 #318 #331 #335 #340
+
+Thanks to the following people for their contributions:
+
+* `bradh11 <https://github.com/bradh11>`_
+* `fallenarc <https://github.com/fallenarc>`_
+* `floatingstatic <https://github.com/floatingstatic>`_
+* `jimmelville <https://github.com/jimmelville>`_
+* `optiz0r <https://github.com/optiz0r>`_
+* `wdesmedt <https://github.com/wdesmedt>`_
+* `dmfigol <https://github.com/dmfigol>`_
+* `ktbyers <https://github.com/ktbyers>`_
+* `dbarrosop <https://github.com/dbarrosop>`_
+
 2.0.0 - December 17 2018
 ========================
 
-For details about upgrading to 2.0.0 see the [https://nornir.readthedocs.io/en/2.0.0-beta/upgrading/1_to_2.html](notes).
+For details about upgrading to 2.0.0 see the `notes <https://nornir.readthedocs.io/en/2.0.0-beta/upgrading/1_to_2.html>`_.
 
 + [CORE_ENHANCEMENTS] Lots of core enhancements, too many to document
 + [CORE_ENHANCEMENTS] Changes on how the inventory
-+ [CORE_ENHANCEMENTS] New ``F`` object for advanced filtering of hosts [docs](file:///Users/dbarroso/workspace/nornir/docs/_build/html/howto/advanced_filtering.html)
++ [CORE_ENHANCEMENTS] New ``F`` object for `advanced filtering of hosts <https://nornir.readthedocs.io/en/stable/howto/advanced_filtering.html>`_
 + [CORE_ENHANCEMENTS] Improvements on how to serialize/deserialize user facing data like the configuration and the inventory
 + [CORE_ENHANCEMENTS] Connections are now their own type of plugin
-+ [CORE_ENHANCEMENTS] Ability to handle connections manually [docs](file:///Users/dbarroso/workspace/nornir/docs/_build/html/howto/handling_connections.html)
++ [CORE_ENHANCEMENTS] Ability to `handle connections manually <https://nornir.readthedocs.io/en/stable/howto/handling_connections.html>`_
 + [CORE_BUGFIX] Lots
 + [PLUGIN_BUGFIX] Lots
 + [PLUGIN_NEW] netmiko_save_config

Makefile

@@ -55,7 +55,7 @@ mypy:
 .PHONY: _nbval_docker
 _nbval_docker:
 	/root/.poetry/bin/poetry install
-	pytest --nbval \
+	pytest --nbval --sanitize-with docs/nbval_sanitize.cfg \
 		docs/plugins \
 		docs/howto \
 		docs/tutorials/intro/initializing_nornir.ipynb \
@@ -67,7 +67,7 @@ nbval:
 		run nornir \
 			make _nbval_docker
 
-PHONY: tests
+.PHONY: tests
 tests: build_test_container black sphinx pylama mypy nbval
 	make pytest PYTEST=3.6
 	make pytest PYTEST=3.7

docs/_data_templates/configuration-parameters.j2

@@ -15,7 +15,7 @@
    * - **Type**
      - ``{{ v["type"] }}``
    * - **Default**
-     - {{ "``{}``".format(v["default"]) if v["default"] else "" }}
+     - {{ "``{}``".format(v.get("default")) if v.get("default") != "" else "" }}
    * - **Required**
      - ``{{ v["required"] or false }}``
    * - **Environment Variable**

docs/conf.py

@@ -83,7 +83,7 @@ def extract_version():
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "**/.ipynb_checkpoints"]
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = "sphinx"

docs/configuration/index.rst

@@ -15,6 +15,12 @@ A similar example using a ``yaml`` file:
 .. include:: ../howto/advanced_filtering/config.yaml
    :code: yaml
 
+Logging
+------------
+
+| By default, Nornir automatically configures logging when ``InitNornir`` is called. Logging configuration can be modified and available options are described in the section below. If you want to use Python logging module to configure logging, make sure to set ``logging.enabled`` parameter to ``False`` in order to avoid potential issues.
+| In some situations Nornir will detect previous logging configuration and will emit :obj:`nornir.core.exceptions.ConflictingConfigurationWarning`
+
 Next, you can find each section and their corresponding options.
 
 .. include:: generated/parameters.rst

docs/howto/advanced_filtering.ipynb

@@ -153,7 +153,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "As you can see we have built ourselves a collection of animals with different properties. The ``F`` object let's you access the magic methods of each typesby just prepeding two underscores and the the name of the magic method. For instance, if you want to check if a list contains a particular element you can just prepend ``__contains``. Let's use this feature to retrieve all the animals that belong to the group ``bird``:"
+    "As you can see we have built ourselves a collection of animals with different properties. The ``F`` object let's you access the magic methods of each types by just prepeding two underscores and the the name of the magic method. For instance, if you want to check if a list contains a particular element you can just prepend ``__contains``. Let's use this feature to retrieve all the animals that belong to the group ``bird``:"
    ]
   },
   {

docs/howto/handling_connections.ipynb

@@ -136,40 +136,40 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "dev1.group_1:\r\n",
-      "    port: 22\r\n",
-      "    hostname: dev1.group_1\r\n",
-      "    username:\r\n",
-      "    password: a_password\r\n",
-      "    platform: eos\r\n",
-      "    data:\r\n",
-      "        my_var: comes_from_dev1.group_1\r\n",
-      "        www_server: nginx\r\n",
-      "        role: www\r\n",
-      "        nested_data:\r\n",
-      "            a_dict:\r\n",
-      "                a: 1\r\n",
-      "                b: 2\r\n",
-      "            a_list: [1, 2]\r\n",
-      "            a_string: asdasd\r\n",
-      "    groups:\r\n",
-      "        - group_1\r\n",
-      "    connection_options:\r\n",
-      "        paramiko:\r\n",
-      "            port: 22\r\n",
-      "            hostname:\r\n",
-      "            username: root\r\n",
-      "            password: docker\r\n",
-      "            platform: linux\r\n",
-      "            extras: {}\r\n",
-      "        dummy:\r\n",
-      "            hostname: dummy_from_host\r\n",
-      "            port:\r\n",
-      "            username:\r\n",
-      "            password:\r\n",
-      "            platform:\r\n",
-      "            extras:\r\n",
-      "                blah: from_host\r\n",
+      "dev1.group_1:\n",
+      "    port: 22\n",
+      "    hostname: dev1.group_1\n",
+      "    username:\n",
+      "    password: a_password\n",
+      "    platform: eos\n",
+      "    data:\n",
+      "        my_var: comes_from_dev1.group_1\n",
+      "        www_server: nginx\n",
+      "        role: www\n",
+      "        nested_data:\n",
+      "            a_dict:\n",
+      "                a: 1\n",
+      "                b: 2\n",
+      "            a_list: [1, 2]\n",
+      "            a_string: asdasd\n",
+      "    groups:\n",
+      "        - group_1\n",
+      "    connection_options:\n",
+      "        paramiko:\n",
+      "            port: 22\n",
+      "            hostname:\n",
+      "            username: root\n",
+      "            password: docker\n",
+      "            platform: linux\n",
+      "            extras: {}\n",
+      "        dummy:\n",
+      "            hostname: dummy_from_host\n",
+      "            port:\n",
+      "            username:\n",
+      "            password:\n",
+      "            platform:\n",
+      "            extras:\n",
+      "                blah: from_host\n",
       "\u001b[0m\u001b[0m"
      ]
     }
@@ -195,7 +195,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

docs/howto/transforming_inventory_data.ipynb

@@ -55,7 +55,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can modify inventory data (regardless of the plugin you are using) on the fly easily by password a ``transform_function`` like this:"
+    "You can modify inventory data (regardless of the plugin you are using) on the fly easily by passing a ``transform_function`` like this:"
    ]
   },
   {
@@ -154,7 +154,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now everything we have to do is put the import path as the ``transform_function`` configuration option:"
+    "Now the only thing left to do is put the import path as the ``transform_function`` configuration option:"
    ]
   },
   {
@@ -265,9 +265,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "rtr00.password: None\n",
+      "rtr01.password: None\n"
+     ]
+    }
+   ],
    "source": [
     "from nornir import InitNornir\n",
     "\n",
@@ -295,6 +304,59 @@
     "for name, host in nr.inventory.hosts.items():\n",
     "    print(f\"{name}.password: {host.password}\")"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using ConnectionOptions\n",
+    "\n",
+    "Additionally, you might come across the need to include certain `ConnectionOptions` to be able to connect to your devices.  \n",
+    "Documentation specific to `nornir` can be found [here](../plugins/connections/index.rst) and for `napalm` can be found [here](https://napalm.readthedocs.io/en/latest/support/index.html#optional-arguments).\n",
+    "\n",
+    "\n",
+    "The following example shows how to correctly do so in the `helpers.py`:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from nornir.core.inventory import ConnectionOptions\n",
+    "\n",
+    "def adapt_host_data(host):\n",
+    "    \n",
+    "    # This function receives a Host object for manipulation\n",
+    "    host.username = host[\"user\"]\n",
+    "    host.password = host[\"password\"]\n",
+    "    host.connection_options[\"napalm\"] = ConnectionOptions(\n",
+    "        extras={\n",
+    "            \"optional_args\": {\n",
+    "                \"secret\":host.password\n",
+    "            }\n",
+    "        }\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also specify them as a `default` for your entire inventory in this manner:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from nornir.core.inventory import ConnectionOptions\n",
+    "\n",
+    "nr.inventory.defaults.connection_options = ConnectionOptions(extras={\"optional_args\":{\"secret\":\"my_secret\"}})"
+   ]
   }
  ],
  "metadata": {
@@ -313,7 +375,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.6.6"
   }
  },
  "nbformat": 4,

docs/howto/writing_a_custom_inventory_plugin.ipynb

@@ -60,7 +60,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "A dynamic inventory basically would have to retrieve the data, rearrange in a similar way as in the example above and cal ``super``. Now, let's see how to use it:"
+    "A dynamic inventory basically would have to retrieve the data, rearrange in a similar way as in the example above and call ``super``. Now, let's see how to use it:"
    ]
   },
   {

docs/index.rst

@@ -8,7 +8,7 @@ Welcome to nornir's documentation!
 
 Nornir is an automation framework written in python to be used with python. Most automation
 frameworks hide the language they are written in by using some cumbersome pseudo-language
-which usually is almost Turing complete but lacks tooling to debug and troubleshoot. Integrating
+which usually is almost Turing complete, but lacks tooling to debug and troubleshoot. Integrating
 with other systems is also usually quite hard as they usually have complex APIs if any at all.
 Some of the other common problems of those pseudo-languages is that are usually quite bad
 at dealing with data and re-usability is limited.

docs/nbval_sanitize.cfg

@@ -0,0 +1,3 @@
+[regex1]
+regex: \r\n
+replace: \n

docs/plugins/tasks/data/echo_data.ipynb

@@ -148,7 +148,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.0"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

docs/tutorials/intro/failed_tasks.ipynb

@@ -409,7 +409,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "To achieve this `nornir` keeps a list failed hosts in it's shared [data](../../ref/api/nornir.rst#nornir.core.state.GlobalState) object:"
+    "To achieve this `nornir` keeps a list of failed hosts in it's shared [data](../../ref/api/nornir.rst#nornir.core.state.GlobalState) object:"
    ]
   },
   {

docs/tutorials/intro/grouping_tasks.ipynb

@@ -210,7 +210,7 @@
     "However, this was a `dry_run`. Let's set the `dry_run` variable to `False` so changes are commited and then run the code again:\n",
     "\n",
     "<div class=\"alert alert-info\">\n",
-    "**Note:** The `dry_run` value is shared between the main nornir objects and its childs so in the snippet below `nr.data.dry_run = False` and `cmh.data.dry_run = False` are equivalent.\n",
+    "**Note:** The `dry_run` value is shared between the main nornir objects and its children so in the snippet below `nr.data.dry_run = False` and `cmh.data.dry_run = False` are equivalent.\n",
     "</div>"
    ]
   },
@@ -300,7 +300,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "As the configuration should've been commited already if we run it again the task \"Loading Configuration on the device\" should tell us that `changed : False` and should return an empty diff. Let's see if that's true:"
+    "As the configuration should've been commited already, if we run it again the task \"Loading Configuration on the device\" should tell us that `changed : False` and should return an empty diff. Let's see if that's true:"
    ]
   },
   {