LVM performance, supported services listing, misc fixes and tests

doc/index.rst

@@ -16,6 +16,7 @@ manpages and API documentation. For primary user documentation, see
    qubes
    qubes-vm/index
    qubes-events
+   qubes-features
    qubes-storage
    qubes-exc
    qubes-ext

doc/qubes-features.rst

@@ -0,0 +1,193 @@
+:py:class:`qubes.vm.Features` - Qubes VM features, services
+============================================================
+
+Features are generic mechanism for storing key-value pairs attached to a
+VM. The primary use case for them is data storage for extensions (you can think
+of them as more flexible properties, defined by extensions), but some are also
+used in the qubes core itself. There is no definite list of supported features,
+each extension can set their own and there is no requirement of registration,
+but :program:`qvm-features` man page contains well known ones.
+In addition, there is a mechanism for VM request setting a feature. This is
+useful for extensions to discover if its VM part is present. 
+
+Features can have three distinct values: no value (not present in mapping,
+which is closest thing to :py:obj:`None`), empty string (which is
+interpreted as :py:obj:`False`) and non-empty string, which is
+:py:obj:`True`. Anything assigned to the mapping is coerced to strings,
+however if you assign instances of :py:class:`bool`, they are converted as
+described above. Be aware that assigning the number `0` (which is considered
+false in Python) will result in string `'0'`, which is considered true.
+
+:py:class:`qubes.vm.Features` inherits from :py:class:`dict`, so provide all the
+standard functions to get, list and set values.  Additionally provide helper
+functions to check if given feature is set on the VM and default to the value
+on the VM's template or netvm. This is useful for features which nature is
+inherited from other VMs, like "is package X is installed" or "is VM behind a
+VPN".
+
+Example usage of features in extension:
+
+.. code-block:: python
+
+   import qubes.exc
+   import qubes.ext
+
+   class ExampleExtension(qubes.ext.Extension):
+      @qubes.ext.handler('domain-pre-start')
+      def on_domain_start(self, vm, event, **kwargs):
+         if vm.features.get('do-not-start', False):
+            raise qubes.exc.QubesVMError(vm, 
+               'Start prohibited because of do-not-start feature')
+
+         if vm.features.check_with_template('something-installed', False):
+            # do something
+
+The above extension does two things:
+
+ - prevent starting a qube with ``do-not-start`` feature set
+ - do something when ``something-installed`` feature is set on the qube, or its template
+
+
+qvm-features-request, qubes.PostInstall service
+------------------------------------------------
+
+When some package in the VM want to request feature to be set (aka advertise
+support for it), it should place a shell script in ``/etc/qubes/post-install.d``.
+This script should call :program:`qvm-features-request` with ``FEATURE=VALUE`` pair(s) as
+arguments to request those features. It is recommended to use very simple
+values here (for example ``1``). The script should be named in form
+``XX-package-name.sh`` where ``XX`` is two-digits number below 90 and
+``package-name`` is unique name specific to this package (preferably actual
+package name). The script needs executable bit set.
+
+``qubes.PostInstall`` service will call all those scripts after any package
+installation and also after initial template installation.
+This way package have a chance to report to dom0 if any feature is
+added/removed.
+
+The features flow to dom0 according to the diagram below. Important part is
+that qubes core :py:class:`qubes.ext.Extension` is responsible for handling such request in
+``features-request`` event handler. If no extension handles given feature request,
+it will be ignored. The extension should carefuly validate requested
+features (ignoring those not recognized - may be for another extension) and
+only then set appropriate value on VM object
+(:py:attr:`qubes.vm.BaseVM.features`). It is recommended to make the
+verification code as bulletproof  as possible (for example allow only specific
+simple values, instead of complex structures), because feature requests
+come from untrusted sources. The features actually set on the VM in some cases
+may not be necessary those requested. Similar for values.
+
+.. graphviz::
+
+   digraph {
+
+      "qubes.PostInstall";
+      "/etc/qubes/post-install.d/ scripts";
+      "qvm-features-request";
+      "qubes.FeaturesRequest";
+      "qubes core extensions";
+      "VM features";
+
+      "qubes.PostInstall" -> "/etc/qubes/post-install.d/ scripts";
+      "/etc/qubes/post-install.d/ scripts" -> "qvm-features-request" 
+         [xlabel="each script calls"];
+      "qvm-features-request" -> "qubes.FeaturesRequest" 
+         [xlabel="last script call the service to dom0"];
+      "qubes.FeaturesRequest" -> "qubes core extensions" 
+         [xlabel="features-request event"];
+      "qubes core extensions" -> "VM features" 
+         [xlabel="verification"];
+
+   }
+
+Example ``/etc/qubes/post-install.d/20-example.sh`` file:
+
+.. code-block:: shell
+
+   #!/bin/sh
+
+   qvm-features-request example-feature=1
+
+Example extension handling the above:
+
+.. code-block:: python
+
+   import qubes.ext
+
+   class ExampleExtension(qubes.ext.Extension):
+      # the last argument must be named untrusted_features
+      @qubes.ext.handler('features-request')
+      def on_features_request(self, vm, event, untrusted_features):
+         # don't allow TemplateBasedVMs to request the feature - should be
+         # requested by the template instead
+         if hasattr(vm, 'template'):
+            return
+
+         untrusted_value = untrusted_features.get('example-feature', None)
+         # check if feature is advertised and verify its value
+         if untrusted_value != '1':
+            return
+         value = untrusted_value
+
+         # and finally set the value
+         vm.features['example-feature'] = value
+
+Services
+---------
+
+`Qubes services <https://www.qubes-os.org/doc/qubes-service/>`_ are implemented
+as features with ``service.`` prefix. The
+:py:class:`qubes.ext.services.ServicesExtension` enumerate all the features
+in form of ``service.<service-name>`` prefix and write them to QubesDB as
+``/qubes-service/<service-name>`` and value either ``0`` or ``1``.
+VM startup scripts list those entries for for each with value of ``1``, create
+``/var/run/qubes-service/<service-name>`` file. Then, it can be conveniently
+used by other scripts to check whether dom0 wishes service to be enabled or
+disabled.
+
+VM package can advertise what services are supported. For that, it needs to
+request ``supported-service.<service-name>`` feature with value ``1`` according
+to description above. The :py:class:`qubes.ext.services.ServicesExtension` will
+handle such request and set this feature on VM object. ``supported-service.``
+features that stop being advertised with ``qvm-features-request`` call are
+removed. This way, it's enough to remove the file from
+``/etc/qubes/post-install.d`` (for example by uninstalling package providing
+the service) to tell dom0 the service is no longer supported. Services
+advertised by TemplateBasedVMs are currently ignored (related
+``supported-service.`` features are not set), but retrieving them may be added
+in the future. Applications checking for specific service support should use
+``vm.features.check_with_template('supported-service.<service-name>', False)``
+call on desired VM object. When enumerating all supported services, application
+should consider both the vm and its template (if any).
+
+Various tools will use this information to discover if given service is
+supported. The API does not enforce service being first advertised before being
+enabled (means: there can be service which is enabled, but without matching
+``supported-service.`` feature). The list of well known services is in
+:program:`qvm-service` man page.
+
+Example ``/etc/qubes/post-install.d/20-my-service.sh``:
+
+.. code-block:: shell
+
+   #!/bin/sh
+
+   qvm-features-request supported-service.my-service=1
+
+Services and features can be then inspected from dom0 using
+:program:`qvm-features` tool, for example:
+
+.. code-block:: shell
+
+   $ qvm-features my-qube
+   supported-service.my-service  1
+
+Module contents
+---------------
+
+.. autoclass:: qubes.vm.Features
+   :members:
+   :show-inheritance:
+
+.. vim: ts=3 sw=3 et
+

qubes/api/admin.py

@@ -479,7 +479,7 @@ def vm_volume_import(self):
         if not self.dest.is_halted():
             raise qubes.exc.QubesVMNotHaltedError(self.dest)
 
-        path = self.dest.storage.import_data(self.arg)
+        path = yield from self.dest.storage.import_data(self.arg)
         assert ' ' not in path
         size = self.dest.volumes[self.arg].size
 

qubes/api/internal.py

@@ -68,7 +68,8 @@ def vm_volume_import_end(self, untrusted_payload):
         success = untrusted_payload == b'ok'
 
         try:
-            self.dest.storage.import_data_end(self.arg, success=success)
+            yield from self.dest.storage.import_data_end(self.arg,
+                success=success)
         except:
             self.dest.fire_event('domain-volume-import-end', volume=self.arg,
                 success=False)

qubes/app.py

@@ -725,6 +725,12 @@ class Qubes(qubes.PropertyHolder):
         doc='''Default time in seconds after which qrexec connection attempt is
             deemed failed''')
 
+    default_shutdown_timeout = qubes.property('default_shutdown_timeout',
+        load_stage=3,
+        default=60,
+        type=int,
+        doc='''Default time in seconds for VM shutdown to complete''')
+
     stats_interval = qubes.property('stats_interval',
         default=3,
         type=int,

qubes/exc.py

@@ -101,6 +101,14 @@ def __init__(self, vm, msg=None):
         super(QubesVMNotHaltedError, self).__init__(vm,
             msg or 'Domain is not powered off: {!r}'.format(vm.name))
 
+class QubesVMShutdownTimeoutError(QubesVMError):
+    '''Domain shutdown timed out.
+
+    '''
+    def __init__(self, vm, msg=None):
+        super(QubesVMShutdownTimeoutError, self).__init__(vm,
+            msg or 'Domain shutdown timed out: {!r}'.format(vm.name))
+
 
 class QubesNoTemplateError(QubesVMError):
     '''Cannot start domain, because there is no template'''
@@ -151,7 +159,7 @@ def __init__(self, msg=None):
             msg or 'Backup cancelled')
 
 
-class QubesMemoryError(QubesException, MemoryError):
+class QubesMemoryError(QubesVMError, MemoryError):
     '''Cannot start domain, because not enough memory is available'''
     def __init__(self, vm, msg=None):
         super(QubesMemoryError, self).__init__(

qubes/ext/r3compatibility.py

@@ -80,6 +80,9 @@ def on_firewall_changed(self, vm, event):
 
     def write_iptables_qubesdb_entry(self, firewallvm):
         # pylint: disable=no-self-use
+        # skip compatibility rules if new format support is advertised
+        if firewallvm.features.check_with_template('qubes-firewall', False):
+            return
         firewallvm.untrusted_qdb.rm("/qubes-iptables-domainrules/")
         iptables = "# Generated by Qubes Core on {0}\n".format(
             datetime.datetime.now().ctime())

qubes/ext/services.py

@@ -62,3 +62,40 @@ def on_domain_feature_delete(self, vm, event, feature):
             return
         service = feature[len('service.'):]
         vm.untrusted_qdb.rm('/qubes-service/{}'.format(service))
+
+    @qubes.ext.handler('features-request')
+    def supported_services(self, vm, event, untrusted_features):
+        '''Handle advertisement of supported services'''
+        # pylint: disable=no-self-use,unused-argument
+
+        if getattr(vm, 'template', None):
+            vm.log.warning(
+                'Ignoring qubes.FeaturesRequest from template-based VM')
+            return
+
+        new_supported_services = set()
+        for requested_service in untrusted_features:
+            if not requested_service.startswith('supported-service.'):
+                continue
+            if untrusted_features[requested_service] == '1':
+                # only allow to advertise service as supported, lack of entry
+                #  means service is not supported
+                new_supported_services.add(requested_service)
+        del untrusted_features
+
+        # if no service is supported, ignore the whole thing - do not clear
+        # all services in case of empty request (manual or such)
+        if not new_supported_services:
+            return
+
+        old_supported_services = set(
+            feat for feat in vm.features
+            if feat.startswith('supported-service.') and vm.features[feat])
+
+        for feature in new_supported_services.difference(
+                old_supported_services):
+            vm.features[feature] = True
+
+        for feature in old_supported_services.difference(
+                new_supported_services):
+            del vm.features[feature]

qubes/ext/windows.py

@@ -34,7 +34,7 @@ def qubes_features_request(self, vm, event, untrusted_features):
 
         guest_os = None
         if 'os' in untrusted_features:
-            if untrusted_features['os'] in ['Windows']:
+            if untrusted_features['os'] in ['Windows', 'Linux']:
                 guest_os = untrusted_features['os']
 
         qrexec = None

qubes/storage/__init__.py

@@ -198,6 +198,8 @@ def import_data(self):
             volume data require something more than just writing to a file (
             for example connecting to some other domain, or converting data
             on the fly), the returned path may be a pipe.
+
+            This can be implemented as a coroutine.
         '''
         raise self._not_implemented("import")
 
@@ -207,6 +209,8 @@ def import_data_end(self, success):
 
         This method is called regardless the operation was successful or not.
 
+        This can be implemented as a coroutine.
+
         :param success: True if data import was successful, otherwise False
         '''
         # by default do nothing
@@ -654,24 +658,34 @@ def export(self, volume):
 
         return self.vm.volumes[volume].export()
 
+    @asyncio.coroutine
     def import_data(self, volume):
         ''' Helper function to import volume data (pool.import_data(volume))'''
         assert isinstance(volume, (Volume, str)), \
             "You need to pass a Volume or pool name as str"
         if isinstance(volume, Volume):
-            return volume.import_data()
+            ret = volume.import_data()
+        else:
+            ret = self.vm.volumes[volume].import_data()
 
-        return self.vm.volumes[volume].import_data()
+        if asyncio.iscoroutine(ret):
+            ret = yield from ret
+        return ret
 
+    @asyncio.coroutine
     def import_data_end(self, volume, success):
         ''' Helper function to finish/cleanup data import
         (pool.import_data_end( volume))'''
         assert isinstance(volume, (Volume, str)), \
             "You need to pass a Volume or pool name as str"
         if isinstance(volume, Volume):
-            return volume.import_data_end(success=success)
+            ret = volume.import_data_end(success=success)
+        else:
+            ret = self.vm.volumes[volume].import_data_end(success=success)
 
-        return self.vm.volumes[volume].import_data_end(success=success)
+        if asyncio.iscoroutine(ret):
+            ret = yield from ret
+        return ret
 
 
 class VolumesCollection: