Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Factory PKI docs according to recent changes #699

Merged
merged 1 commit into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 186 additions & 25 deletions source/reference-manual/security/device-gateway.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,27 @@
Managing Factory PKI
====================

LmP devices connect to OTA services via a :ref:`device gateway <ref-ota-architecture>` configured with mutual TLS.
Each Factory uses a default device gateway with certificates owned by Foundries.io™.
We allow—and **encourage**\—you to set up your own PKI infrastructure.
This is so that you are in control of the security of the device gateway.
LmP devices connect to OTA services via a :ref:`Device Gateway <ref-ota-architecture>` configured with mutual TLS.
A set of Digital Certificates used to establish trust between Factory devices and Device Gateway is a Factory Public Key Infrastructure (PKI).

When a new Factory is created, it is configured to use the default shared PKI with Certificates owned by Foundries.io™.
This provides a faster engagement with the FoundriesFactory®, allowing streamlined product development.

FoundriesFactory supports setting up your own Factory PKI via either :ref:`Fioctl® <ref-fioctl>` commands or the API integration.
We recommend setting up your own Factory PKI **before** going to production.
Benefits of owning your Factory PKI are two-fold:

- Controlling which devices can connect to your Factory Device Gateway to fetch configuration and software updates.
- Controlling which servers can deliver configuration and software updates to your Factory devices.

.. warning::
The Factory :ref:`Root of Trust <Root-of-Trust>` **can only be set once**; subsequent attempts will fail.
Other Factory PKI certificates can be updated at any time; having that you own your Factory Root of Trust.

`Contact customer support <https://support.foundries.io>` if you need your Factory PKI being reset.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, so reset Factory PKI and set up your own Factory PKI are two different things. But both rotate the keys, right? This is only a matter of who is performing the rotation. Maybe a link in the "encourage" sentence (line 12) is missing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resetting Factory PKI does not rotate the certificates.
It is like a factory reset - return back to using the shared PKI.

At that point all existing devices will no longer be update to fetch updates, until manual intervention.
We work on rotating the Root of Trust as a separate feature, which is (tentatively) going to be ready in Q4 this year.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this hyperlink is not working, I guess we are missing an _ somewhere

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`Contact customer support <https://support.foundries.io>` if you need your Factory PKI being reset.
`Contact customer support <https://support.foundries.io>`_ if you need your Factory PKI being reset.

Once a reset was performed, all connected devices will lose their connection.
These devices will not be able to connect to the Device Gateway until they are re-provisioned with a new Root of Trust.
On practice that usually means that these devices need to be re-flashed (after the Factory PKI reset).

Terminology
-----------
Expand All @@ -16,51 +33,105 @@ Terminology
Root of Trust: ``factory_ca.key / factory_ca.pem``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The PKI root of trust for your Factory.
You own the private key (NIST P-256 by default).
The corresponding x509 certificate is shared with Foundries.io to define your root of trust.
An X.509 certificate used as a Root Certificate Authority (RCA) for your Factory.
You own the private key (NIST P-256 by default), and share the corresponding certificate with Foundries.io.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You own the private key (NIST P-256 by default), and share the corresponding certificate with Foundries.io.
You own the private key (NIST P-256 by default), and share the corresponding public certificate with Foundries.io.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nixed.

A key can be public or private. A certificate is just a... certificate, bearing a public key. So it is presumed to be public.


All intermediate Certificate Authorities (CAs) and TLS certificates configured in your Factory must be signed by its Root of Trust.
The Root of Trust is preloaded to factory devices, so that they can use it to verify the FoundriesFactory web APIs TLS certificates.

.. warning::
Never lose the private key of your Factory Root of Trust.
By design, Foundries.io only stores a copy of the CA certificate bearing its public key.
We are not able to recover your private key in case of its loss.

We recommend storing your Factory Root of Trust in a cloud-based HSM solution of your choice.
For example, we verified that the `AWS Cloud HSM <https://aws.amazon.com/cloudhsm/>`_ supports `importing EC private keys`_.
That way you get an increased safety of your highly important secret through their redundancy and backup policies.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we? and if so, were do we document how to do it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of our customers do store their Factory root private keys in the Cloud HSM.
I don't know if they used the --hsm-xxx options of Fioctl for that or not.
I'd like to get feedback from customer support on that.

As for documenting the HOW part, there are two options:

  1. Do nothing.
    The user may generate a private key without --hsm-xxx options, and then copy it into the HSM.
    It is up to the HSM provider to document how to do that.
  2. Document --hsm-xxx options in a separate section here.
    Before doing that, I'd like to first verify our integration with e.g. AWS Cloud HSM, and write a blog post about that.

Currently, I have no capacity to do 2, so I decided that 1 is better than nothing.

I do not feel confident with only intructing our users to print private key and store in multiple safes.
The thing is: almost nobody does that, and when they lose the key - they come to blame on us.

A simple sentence to say they need to store a key in a cloud HSM will motivate them to do the research and offload this pain onto a billionaire business.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't recommend something and not provide documentation for something this complicated. It will lead to a support ticket for each new customer.

Copy link
Member Author

@vkhoroz vkhoroz Apr 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, I think it should be enough to refer to this page: https://docs.aws.amazon.com/cloudhsm/latest/userguide/key_mgmt_util-importPrivateKey.html.

I did not mean to document a complete HSM support in this iteration.
But, I want to recommend at least importing the private key into the HSM (use it as a digital safe).

@doanac WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe. You should consult with support. The problem I see with statement around this are people then asking:

  • Why AWS instead of $X
  • How much does it cost?
  • <insert random question for something we can't predict since we haven't even tried ourselves>

Copy link
Member Author

@vkhoroz vkhoroz Apr 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to use vendor neutral sentence, but I think it might be improved like e.g. below:

We recommend storing your Factory Root of Trust in a cloud-based HSM solution of your choice.
For example, we verified that the `AWS Cloud HSM <link to landing page>`_ supports `importing EC private keys <link to manual>`_.

Obviously, I need to verify that it really works, before making that bold statement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can live with that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like AWS Cloud HSM advertisement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrt customer support: Andy has a point here, we will get inquiries if we don't provide information on this topic. I'm ok with pointing to AWS Cloud HSM. We can always revisit this later if this becomes a pain point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rephrased accoring to my suggestion.


All intermediate CA and mutual TLS certificates configured in your Factory must be signed by this keypair.
In particular, the certificates mentioned below.
Additionally, we recommend printing the private key of your Root of Trust on paper and storing it in multiple fire and waterproof safes.

.. _importing EC private keys: https://docs.aws.amazon.com/cloudhsm/latest/userguide/key_mgmt_util-importPrivateKey.html

.. _tls-crt:

Server TLS Certificate: ``tls-crt``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This certificate along with its private key is used by Device Gateway during mTLS handshake/session setup.
Specifically, they are used for Device Gateway identity verification by a device/client and a TLS session's symmetric key setup.
The private key is owned by Foundries.io and the certificate is signed by the root of trust.
An X.509 certificate used by :ref:`Device Gateway <ref-ota-architecture>` during a mutual TLS handshake and session setup.
Foundries.io owns the private key (NIST P-256 by default), and you sign the certificate by the Factory Root of Trust.

When your Factory devices connect to the Device Gateway, they verify the server identity by validating its TLS Certificate.
They use the preloaded Factory Root of Trust to perform that validation.
Once the mutual trust is established, Device Gateway uses its TLS Certificate to setup a session symmetric key.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like TLS cert is the only thing that is needed/involved in the session key setup.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the only one.
A client certificate is involved at a later time during the TLS handshake to for the client authorization after the session was already set up.

That temporary symmetric key is used to encrypt all session traffic between the Device Gateway and the device.

Device Client Certificate
~~~~~~~~~~~~~~~~~~~~~~~~~

An X.509 certificate was issued to your Factory device during the registration process.
The device owns the private key (NIST P-256 by default) and the certificate.

This certificate must be signed by either a :ref:`Local Device CA <local-ca>` or an :ref:`Online Device CA <online-ca>` (see below).
For example, when using the `lmp-device-register`_ to register your device, it generates the Device Client Certificate Signing Request (CSR).
That CSR is then signed by an appropriate Device CA at the registration server (either your own or Foundries.io), and stored on the device.

When connecting to the :ref:`Device Gateway <ref-ota-architecture>`, a device must present its Client Certificate during a TLS handshake.
The device identity is verified at the Device Gateway, and the device is either allowed or denied to connect based on its certificate validity.
Once the mutual trust is established, device uses its Client Certificate to setup a session symmetric key.

.. _lmp-device-register: https://github.com/foundriesio/lmp-device-register/

.. _online-ca:

Certificate: ``online-ca``
~~~~~~~~~~~~~~~~~~~~~~~~~~
Online Device CA: ``online-ca``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order for ``lmp-device-register`` to work, Foundries.io needs the ability to sign client certificates for devices.
If enabled, the root of trust will sign an ``online-ca`` certificate that Foundries.io can use to sign client authentication certificates.
An X.509 certificate used as a CA issuing certificates to devices registered via the FoundriesFactory API.
Foundries.io owns the private key (NIST P-256 by default), and you sign the certificate by the Factory Root of Trust.

When using the "shared" Factory PKI, this is the only CA used to issue Client Certificates to your Factory devices.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"shared" again. I don't know how to interpret this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe "default Factory PKI"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is both default and shared...
Shared better emphasizes on its essense, as it is exactly the same set of cetificates for all factories.

Once you take ownership of your Factory PKI, you may opt out of using the Online Device CA.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe opt out of using the Online Device CA in favor of Local Device CA?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree. That would make a sentence excessively longer providing marginally more value.


.. _local-ca:

Certificate: ``local-ca``
~~~~~~~~~~~~~~~~~~~~~~~~~
Local Device CA: ``local-ca``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Optional pair(s) of a private key and intermediate CA certificate, signed by the root CA.
Can be used by something like your manufacturing process to sign client certificates for devices—without needing access to Foundries.io.
An X.509 certificate used as a CA issuing certificates to devices registered via your offline registration process.
You own the private key (NIST P-256 by default), and share the corresponding certificate with Foundries.io.
It must be signed by the Root of Trust, so that Foundries.io may verify if a user is entitled to upload a Device CA.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here "Root of Trust" is alone. In other places, it's "Factory Root of Trust".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we need to fix it. I prefer to skip.


It is also known and referred to as ``offline CA``, since you own its private key and keep it "offline".
At creation, your Factory only has an Online Device CA and no Local Device CAs.
Your factory may be configured to have one or more Local Device CAs only after you took ownership of your Factory PKI.
You may use the Local Device CA with our :ref:`ref-factory-registration-ref` to register your devices offline.

.. figure:: /_static/ca_certs.png
:align: center
:scale: 90 %
:alt: PKI hierarchy

.. _est-tls-crt:

EST Server TLS Certificate: ``est-tls-crt``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An X.509 certificate used by FoundriesFactory hosted :ref:`ref-cert-rotation` during a mutual TLS handshake and session setup.
Foundries.io owns the private key (NIST P-256 by default), and you sign the certificate by the Factory Root of Trust.

The FoundriesFactory process for rotating device certificates is based on the industry standard `RFC 7030`_ Enrollment over Secure Transport (EST).
Your Factory may be configured to use a FoundriesFactory hosted EST service, your own EST service, or no EST service.

.. _RFC 7030: https://datatracker.ietf.org/doc/html/rfc7030

.. _ref-rm-pki:

Managing Your Factory PKI
-------------------------

Setting Up Your PKI
-------------------
~~~~~~~~~~~~~~~~~~~

:ref:`ref-fioctl` includes a sub-command to set up your PKI:
:ref:`ref-fioctl` includes a command to set up your PKI:

.. warning::
The following command can only be used once.
Expand All @@ -72,7 +143,7 @@ Setting Up Your PKI
A few important things to note about this command:

* Use a PKCS#11 compatible HSM.
This will ensure the safety of your Factory's root of trust private key.
This will ensure the safety of your Factory's Root of Trust private key.

* The "PKI Directory" is important, and should be securely backed up.

Expand All @@ -86,7 +157,97 @@ After running the above command, you can validate the outcome and view the confi

fioctl keys ca show --pretty

Rotating Server TLS Certificate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes, you might need to rotate the TLS certificate used by the Device Gateway to serve your Factory devices.
For example, the corresponding TLS certificate might be close to its expiration date, or it might be compromised.
Foundries.io is not able to perform that task for you, as it requires access to your Factory Root of Trust.

:ref:`ref-fioctl` includes a command to rotate your Server TLS Certificate:

.. code-block::

fioctl keys ca rotate-tls /absolute/path/to/certs/

Adding Device CA
~~~~~~~~~~~~~~~~

Sometimes, you might need to add more than one Device CA to your Factory.
Some use cases when this is needed include (but are not limited to) the following situations:

* You have only initially set up an Online Device CA for your Factory,
and want to also configure a Local Device CA (or vice versa).

* You opened a new manufacturing site,
and want a dedicated Local Device CA to issue Client Certificates to devices manufactured at this site.

* One of your Device CAs was compromised,
and you need to replace it by a new Device CA (either Online or Local).

:ref:`ref-fioctl` includes a command to add one more Device CA to your Factory:

.. code-block::

fioctl keys ca add-device-ca /absolute/path/to/certs/ [--online-ca | --local-ca]

Revoking Device CA
~~~~~~~~~~~~~~~~~~

You may need to revoke or disable a Device CA for your Factory.
Some use cases when this is needed include the following situations:

* One of your Device CAs was compromised,
and you need to deny an ability to register new devices with client certificates issued by this CA.
You may also want to completely deny access to the Device Gateway for already registered devices with such certificates.

* You are closing a manufacturing site,
and want to make sure that a Device CA issued for that manufacturing site can no longer be used to issue new client certificates.

:ref:`ref-fioctl` provides two separate commands: to disable and revoke an existing Device CA.

There is an important difference between disabling and revoking a Device CA:

- When you disable the Device CA,
new devices with client certificates issued by that CA cannot be registered.
- When you revoke the Device CA, in addition to the above,
already registered devices with client certificates issued by that CA cannot connect to your Factory.

Use the below command when you need to disable a Device CA:

.. code-block::

fioctl keys ca disable-device-ca /absolute/path/to/certs/ [--ca-file <filename> | --ca-serial <serial>]

Use the following command when you need to revoke a Device CA:

.. code-block::

fioctl keys ca revoke-device-ca /absolute/path/to/certs/ [--ca-file <filename> | --ca-serial <serial>]

After the Device CA is revoked, devices can no longer update their apps or config.
Therefore, the revocation process needs to be planned properly.
We recommend the following workflow:

1. Disable the Device CA.
This action needs to be taken as soon as you notice that your Device CA was compromised.
This makes sure that an attacker is not able to register new devices with client certificates issued by that CA.

2. Inspect your fleet of already registered devices, and delete those devices which you think are not legitimate.
After this point, you can be sure that an attacker can no longer steal your new Intellectual Property (provided by OTA updates).
FoundriesFactory advices you to also prepare a separate plan how to deal with already compromised devices.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 226 can become a note/important/warning

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to keep as is to avoid fancy markup inside a list.


3. Rotate client certificates on your devices which have a client certificate issued by a Device CA you are revoking.
You may use Foundries.io hosted ref:`ref-cert-rotation` service, or use your own certificate rotation workflow.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref is not working here

Make sure that new device client certificates are issued by one of Device CAs enabled at your Factory.

4. Revoke the Device CA.
At this point a reference to a given Device CA is completely removed from our servers, hence becomes untrusted.

Related Topics
--------------

The Factory PKI is interwoven with the device manufacturing process and device registration.
You can find out more details on this topic in this guide :ref:`ref-factory-registration-ref`.

More details on Factory PKI can be found in this :ref:`guide <ref-device-gateway-pki-details>`.
More details on Factory PKI internals can be found in this :ref:`guide <ref-device-gateway-pki-details>`.
Loading
Loading