Add module_utils and filters for quoting and unquoting (#53)

* Move splitting code to own file.

* Move list to dictionary code to quoting as well.

* Add quoting functionality.

* Add quoting filters.

* Add integration tests to CI.

* Fix bugs, increase coverage.

* Make parsing more strict.

* Extract function parse_argument_value from split_routeros_command to make proper parsing of WHERE possible.

* Adjust expected error message in integration tests.

* Simplify code and improve coverage.

* Add changelog fragment for WHERE strictness in api module.

* Add documenation.

* Fix typo.

* Add documentation references.

* Add example to api module which uses quote_argument_value.

* Fix bug, and add tests which prevent this in the future.

* Add more escape sequence tests.

* Make sure all control characters are quoted.

.github/workflows/ansible-test.yml

@@ -112,3 +112,76 @@ jobs:
       - uses: codecov/codecov-action@v1
         with:
           fail_ci_if_error: false
+
+###
+# Integration tests (RECOMMENDED)
+#
+# https://docs.ansible.com/ansible/latest/dev_guide/testing_integration.html
+
+  integration:
+    runs-on: ubuntu-latest
+    name: I (Ⓐ${{ matrix.ansible }}+py${{ matrix.python }})
+    strategy:
+      fail-fast: false
+      matrix:
+        ansible:
+          - stable-2.12
+          - devel
+        python:
+          - 3.8
+          - 3.9
+          - "3.10"
+        include:
+          # 2.9
+          - ansible: stable-2.9
+            python: 2.7
+          - ansible: stable-2.9
+            python: 3.5
+          - ansible: stable-2.9
+            python: 3.6
+          # 2.10
+          - ansible: stable-2.10
+            python: 3.5
+          # 2.11
+          - ansible: stable-2.11
+            python: 2.7
+          - ansible: stable-2.11
+            python: 3.6
+          - ansible: stable-2.11
+            python: 3.9
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v2
+        with:
+          path: ansible_collections/community/routeros
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          # it is just required to run that once as "ansible-test integration" in the docker image
+          # will run on all python versions it supports.
+          python-version: 3.8
+
+      - name: Install ansible-core (${{ matrix.ansible }})
+        run: pip install https://github.com/ansible/ansible/archive/${{ matrix.ansible }}.tar.gz --disable-pip-version-check
+
+      - name: Install collection dependencies
+        run: git clone --depth=1 --single-branch https://github.com/ansible-collections/ansible.netcommon.git ansible_collections/ansible/netcommon
+        # NOTE: we're installing with git to work around Galaxy being a huge PITA (https://github.com/ansible/galaxy/issues/2429)
+        # run: ansible-galaxy collection install ansible.netcommon -p .
+
+      # Run the integration tests
+      - name: Run integration test
+        run: ansible-test integration -v --color --retry-on-error --continue-on-error --diff --python ${{ matrix.python }} --docker --coverage
+        working-directory: ./ansible_collections/community/routeros
+
+        # ansible-test support producing code coverage date
+      - name: Generate coverage report
+        run: ansible-test coverage xml -v --requirements --group-by command --group-by version
+        working-directory: ./ansible_collections/community/routeros
+
+      # See the reports at https://codecov.io/gh/ansible-collections/community.routeros
+      - uses: codecov/codecov-action@v1
+        with:
+          fail_ci_if_error: false

changelogs/fragments/53-api-where.yml

@@ -0,0 +1,2 @@
+minor_changes:
+- "api - make validation of ``WHERE`` for ``query`` more strict (https://github.com/ansible-collections/community.routeros/pull/53)."

changelogs/fragments/53-quoting-filters.yml

@@ -0,0 +1,12 @@
+---
+add plugin.filter:
+  - name: split
+    description: Split a command into arguments
+  - name: quote_argument_value
+    description: Quote an argument value
+  - name: quote_argument
+    description: Quote an argument
+  - name: join
+    description: Join a list of arguments to a command
+  - name: list_to_dict
+    description: Convert a list of arguments to a list of dictionary

docs/docsite/extra-docs.yml

@@ -4,3 +4,4 @@ sections:
     toctree:
       - api-guide
       - ssh-guide
+      - quoting

docs/docsite/rst/quoting.rst

@@ -0,0 +1,14 @@
+.. _ansible_collections.community.routeros.docsite.quoting:
+
+How to quote and unquote commands and arguments
+===============================================
+
+When using the :ref:`community.routeros.command module <ansible_collections.community.routeros.command_module>` or the :ref:`community.routeros.api module <ansible_collections.community.routeros.api_module>` modules, you need to pass text data in quoted form. While in some cases quoting is not needed (when passing IP addresses or names without spaces, for example), in other cases it is required, like when passing a comment which contains a space.
+
+The community.routeros collection provides a set of Jinja2 filter plugins which helps you with these tasks:
+
+- The ``community.routeros.quote_argument_value`` filter quotes an argument value: ``'this is a "comment"' | community.routeros.quote_argument_value == '"this is a \\"comment\\""'``.
+- The ``community.routeros.quote_argument`` filter quotes an argument with or without a value: ``'comment=this is a "comment"' | community.routeros.quote_argument == 'comment="this is a \\"comment\\""'``.
+- The ``community.routeros.join`` filter quotes a list of arguments and joins them to one string: ``['foo=bar', 'comment=foo is bar'] | community.routeros.join == 'foo=bar comment="foo is bar"'``.
+- The ``community.routeros.split`` filter splits a command into a list of arguments (with or without values): ``'foo=bar comment="foo is bar"' | community.routeros.split == ['foo=bar', 'comment=foo is bar']``
+- The ``community.routeros.list_to_dict`` filter splits a list of arguments with values into a dictionary: ``['foo=bar', 'comment=foo is bar'] | community.routeros.list_to_dict == {'foo': 'bar', 'comment': 'foo is bar'}``. It has two optional arguments: ``require_assignment`` (default value ``true``) allows to accept arguments without values when set to ``false``; and ``skip_empty_values`` (default value ``false``) allows to skip arguments whose value is empty.

plugins/filter/quoting.py

@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+
+# Copyright: (c) 2021, Felix Fontein <[email protected]>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+from ansible.errors import AnsibleFilterError
+from ansible.module_utils.common.text.converters import to_text
+
+from ansible_collections.community.routeros.plugins.module_utils.quoting import (
+    ParseError,
+    convert_list_to_dictionary,
+    join_routeros_command,
+    quote_routeros_argument,
+    quote_routeros_argument_value,
+    split_routeros_command,
+)
+
+
+def wrap_exception(fn, *args, **kwargs):
+    try:
+        return fn(*args, **kwargs)
+    except ParseError as e:
+        raise AnsibleFilterError(to_text(e))
+
+
+def split(line):
+    '''
+    Split a command into arguments.
+
+    Example:
+        'add name=wrap comment="with space"'
+    is converted to:
+        ['add', 'name=wrap', 'comment=with space']
+    '''
+    return wrap_exception(split_routeros_command, line)
+
+
+def quote_argument_value(argument):
+    '''
+    Quote an argument value.
+
+    Example:
+        'with "space"'
+    is converted to:
+        r'"with \"space\""'
+    '''
+    return wrap_exception(quote_routeros_argument_value, argument)
+
+
+def quote_argument(argument):
+    '''
+    Quote an argument.
+
+    Example:
+        'comment=with "space"'
+    is converted to:
+        r'comment="with \"space\""'
+    '''
+    return wrap_exception(quote_routeros_argument, argument)
+
+
+def join(arguments):
+    '''
+    Join a list of arguments to a command.
+
+    Example:
+        ['add', 'name=wrap', 'comment=with space']
+    is converted to:
+        'add name=wrap comment="with space"'
+    '''
+    return wrap_exception(join_routeros_command, arguments)
+
+
+def list_to_dict(string_list, require_assignment=True, skip_empty_values=False):
+    '''
+    Convert a list of arguments to a list of dictionary.
+
+    Example:
+        ['foo=bar', 'comment=with space', 'additional=']
+    is converted to:
+        {'foo': 'bar', 'comment': 'with space', 'additional': ''}
+
+    If require_assignment is True (default), arguments without assignments are
+    rejected. (Example: in ['add', 'name=foo'], 'add' is an argument without
+    assignment.) If it is False, these are given value None.
+
+    If skip_empty_values is True, arguments with empty value are removed from
+    the result. (Example: in ['name='], 'name' has an empty value.)
+    If it is False (default), these are kept.
+
+    '''
+    return wrap_exception(
+        convert_list_to_dictionary,
+        string_list,
+        require_assignment=require_assignment,
+        skip_empty_values=skip_empty_values,
+    )
+
+
+class FilterModule(object):
+    '''Ansible jinja2 filters for RouterOS command quoting and unquoting'''
+
+    def filters(self):
+        return {
+            'split': split,
+            'quote_argument': quote_argument,
+            'quote_argument_value': quote_argument_value,
+            'join': join,
+            'list_to_dict': list_to_dict,
+        }