Merge pull request #5997 from gilles-peskine-arm/storage-format-doc-2…

…02206-2.28

Backport 2.28: Documentation about storage format compatibility

BRANCHES.md

@@ -16,14 +16,22 @@ API compatibility in the `master` branch between major version changes. We
 also maintain ABI compatibility within LTS branches; see the next section for
 details.
 
-## Backwards Compatibility
+## Backwards Compatibility for application code
 
 We maintain API compatibility in released versions of Mbed TLS. If you have
 code that's working and secure with Mbed TLS x.y.z and does not rely on
 undocumented features, then you should be able to re-compile it without
 modification with any later release x.y'.z' with the same major version
 number, and your code will still build, be secure, and work.
 
+Note that this guarantee only applies if you either use the default
+compile-time configuration (`mbedtls/config.h`) or the same modified
+compile-time configuration. Changing compile-time configuration options can
+result in an incompatible API or ABI, although features will generally not
+affect unrelated features (for example, enabling or disabling a
+cryptographic algorithm does not break code that does not use that
+algorithm).
+
 There are rare exceptions: code that was relying on something that became
 insecure in the meantime (for example, crypto that was found to be weak) may
 need to be changed. In case security comes in conflict with backwards
@@ -42,6 +50,19 @@ increase code size for a security fix.)
 For contributors, see the [Backwards Compatibility section of
 CONTRIBUTING](CONTRIBUTING.md#backwards-compatibility).
 
+## Backward compatibility for the key store
+
+We maintain backward compatibility with previous versions of the
+PSA Crypto persistent storage since Mbed TLS 2.25.0, provided that the
+storage backend (PSA ITS implementation) is configured in a compatible way.
+We intend to maintain this backward compatibility throughout a major version
+of Mbed TLS (for example, all Mbed TLS 3.y versions will be able to read
+keys written under any Mbed TLS 3.x with x <= y).
+
+Mbed TLS 3.x can also read keys written by Mbed TLS 2.25.0 through 2.28.x
+LTS, but future major version upgrades (for example from 2.28.x/3.x to 4.y)
+may require the use of an upgrade tool.
+
 ## Current Branches
 
 The following branches are currently maintained:

include/psa/crypto_types.h

@@ -69,38 +69,70 @@ typedef int32_t psa_status_t;
  */
 
 /** \brief Encoding of a key type.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_KEY_TYPE_xxx`.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint16_t psa_key_type_t;
 
 /** The type of PSA elliptic curve family identifiers.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_ECC_FAMILY_xxx`.
  *
  * The curve identifier is required to create an ECC key using the
  * PSA_KEY_TYPE_ECC_KEY_PAIR() or PSA_KEY_TYPE_ECC_PUBLIC_KEY()
  * macros.
  *
  * Values defined by this standard will never be in the range 0x80-0xff.
  * Vendors who define additional families must use an encoding in this range.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint8_t psa_ecc_family_t;
 
 /** The type of PSA Diffie-Hellman group family identifiers.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_DH_FAMILY_xxx`.
  *
  * The group identifier is required to create an Diffie-Hellman key using the
  * PSA_KEY_TYPE_DH_KEY_PAIR() or PSA_KEY_TYPE_DH_PUBLIC_KEY()
  * macros.
  *
  * Values defined by this standard will never be in the range 0x80-0xff.
  * Vendors who define additional families must use an encoding in this range.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint8_t psa_dh_family_t;
 
 /** \brief Encoding of a cryptographic algorithm.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_ALG_xxx`.
  *
  * For algorithms that can be applied to multiple key types, this type
  * does not encode the key type. For example, for symmetric ciphers
  * based on a block cipher, #psa_algorithm_t encodes the block cipher
  * mode and the padding mode while the block cipher itself is encoded
  * via #psa_key_type_t.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint32_t psa_algorithm_t;
 
@@ -142,6 +174,14 @@ typedef uint32_t psa_algorithm_t;
  * #PSA_KEY_LIFETIME_PERSISTENT is supported if persistent storage is
  * available. Other lifetime values may be supported depending on the
  * library configuration.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_KEY_LIFETIME_xxx`.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint32_t psa_key_lifetime_t;
 
@@ -173,6 +213,11 @@ typedef uint32_t psa_key_lifetime_t;
  * \note Key persistence levels are 8-bit values. Key management
  *       interfaces operate on lifetimes (type ::psa_key_lifetime_t) which
  *       encode the persistence as the lower 8 bits of a 32-bit value.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint8_t psa_key_persistence_t;
 
@@ -209,6 +254,11 @@ typedef uint8_t psa_key_persistence_t;
  * \note Key location indicators are 24-bit values. Key management
  *       interfaces operate on lifetimes (type ::psa_key_lifetime_t) which
  *       encode the location as the upper 24 bits of a 32-bit value.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
  */
 typedef uint32_t psa_key_location_t;
 
@@ -220,9 +270,27 @@ typedef uint32_t psa_key_location_t;
  *   #PSA_KEY_ID_VENDOR_MIN to #PSA_KEY_ID_VENDOR_MAX.
  * - 0 is reserved as an invalid key identifier.
  * - Key identifiers outside these ranges are reserved for future use.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to how values are allocated must require careful
+ *       consideration to allow backward compatibility.
  */
 typedef uint32_t psa_key_id_t;
 
+/** Encoding of key identifiers as seen inside the PSA Crypto implementation.
+ *
+ * When PSA Crypto is built as a library inside an application, this type
+ * is identical to #psa_key_id_t. When PSA Crypto is built as a service
+ * that can store keys on behalf of multiple clients, this type
+ * encodes the #psa_key_id_t value seen by each client application as
+ * well as extra information that identifies the client that owns
+ * the key.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
+*/
 #if !defined(MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER)
 typedef psa_key_id_t mbedtls_svc_key_id_t;
 
@@ -246,7 +314,16 @@ typedef struct
  * @{
  */
 
-/** \brief Encoding of permitted usage on a key. */
+/** \brief Encoding of permitted usage on a key.
+ *
+ * Values of this type are generally constructed as bitwise-ors of macros
+ * called `PSA_KEY_USAGE_xxx`.
+ *
+ * \note Values of this type are encoded in the persistent key store.
+ *       Any changes to existing values will require bumping the storage
+ *       format version and providing a translation when reading the old
+ *       format.
+ */
 typedef uint32_t psa_key_usage_t;
 
 /**@}*/
@@ -375,7 +452,11 @@ typedef uint64_t psa_key_slot_number_t;
  * @{
  */
 
-/** \brief Encoding of the step of a key derivation. */
+/** \brief Encoding of the step of a key derivation.
+ *
+ * Values of this type are generally constructed by macros called
+ * `PSA_KEY_DERIVATION_INPUT_xxx`.
+ */
 typedef uint16_t psa_key_derivation_step_t;
 
 /**@}*/

include/psa/crypto_values.h

@@ -12,6 +12,11 @@
  * designations of cryptographic algorithms, and error codes returned by
  * the library.
  *
+ * Note that many of the constants defined in this file are embedded in
+ * the persistent key store, as part of key metadata (including usage
+ * policies). As a consequence, they must not be changed (unless the storage
+ * format version changes).
+ *
  * This header file only defines preprocessor macros.
  */
 /*
@@ -40,6 +45,18 @@
 
 /* PSA error codes */
 
+/* Error codes are standardized across PSA domains (framework, crypto, storage,
+ * etc.). Do not change the values in this section or even the expansions
+ * of each macro: it must be possible to `#include` both this header
+ * and some other PSA component's headers in the same C source,
+ * which will lead to duplicate definitions of the `PSA_SUCCESS` and
+ * `PSA_ERROR_xxx` macros, which is ok if and only if the macros expand
+ * to the same sequence of tokens.
+ *
+ * If you must add a new
+ * value, check with the Arm PSA framework group to pick one that other
+ * domains aren't already using. */
+
 /** The action was completed successfully. */
 #define PSA_SUCCESS ((psa_status_t)0)
 
@@ -316,6 +333,12 @@
  * @{
  */
 
+/* Note that key type values, including ECC family and DH group values, are
+ * embedded in the persistent key store, as part of key metadata. As a
+ * consequence, they must not be changed (unless the storage format version
+ * changes).
+ */
+
 /** An invalid key type value.
  *
  * Zero is not the encoding of any key type.
@@ -673,6 +696,11 @@
      1u << PSA_GET_KEY_TYPE_BLOCK_SIZE_EXPONENT(type) :                         \
      0u)
 
+/* Note that algorithm values are embedded in the persistent key store,
+ * as part of key metadata. As a consequence, they must not be changed
+ * (unless the storage format version changes).
+ */
+
 /** Vendor-defined algorithm flag.
  *
  * Algorithms defined by this standard will never have the #PSA_ALG_VENDOR_FLAG
@@ -1928,6 +1956,11 @@
  * @{
  */
 
+/* Note that location and persistence level values are embedded in the
+ * persistent key store, as part of key metadata. As a consequence, they
+ * must not be changed (unless the storage format version changes).
+ */
+
 /** The default lifetime for volatile keys.
  *
  * A volatile key only exists as long as the identifier to it is not destroyed.
@@ -2043,6 +2076,11 @@
 
 #define PSA_KEY_LOCATION_VENDOR_FLAG            ((psa_key_location_t)0x800000)
 
+/* Note that key identifier values are embedded in the
+ * persistent key store, as part of key metadata. As a consequence, they
+ * must not be changed (unless the storage format version changes).
+ */
+
 /** The null key identifier.
  */
 #define PSA_KEY_ID_NULL                         ((psa_key_id_t)0)
@@ -2154,6 +2192,11 @@ static inline int mbedtls_svc_key_id_is_null( mbedtls_svc_key_id_t key )
  * @{
  */
 
+/* Note that key usage flags are embedded in the
+ * persistent key store, as part of key metadata. As a consequence, they
+ * must not be changed (unless the storage format version changes).
+ */
+
 /** Whether the key may be exported.
  *
  * A public key or the public part of a key pair may always be exported
@@ -2255,6 +2298,9 @@ static inline int mbedtls_svc_key_id_is_null( mbedtls_svc_key_id_t key )
  * @{
  */
 
+/* Key input steps are not embedded in the persistent storage, so you can
+ * change them if needed: it's only an ABI change. */
+
 /** A secret input for key derivation.
  *
  * This should be a key of type #PSA_KEY_TYPE_DERIVE

scripts/abi_check.py

@@ -1,9 +1,10 @@
 #!/usr/bin/env python3
-"""
-This script compares the interfaces of two versions of Mbed TLS, looking
+"""This script compares the interfaces of two versions of Mbed TLS, looking
 for backward incompatibilities between two different Git revisions within
 an Mbed TLS repository. It must be run from the root of a Git working tree.
 
+### How the script works ###
+
 For the source (API) and runtime (ABI) interface compatibility, this script
 is a small wrapper around the abi-compliance-checker and abi-dumper tools,
 applying them to compare the header and library files.
@@ -20,7 +21,66 @@
 Returns 0 on success, 1 on non-compliance, and 2 if there is an error
 while running the script.
 
-You must run this test from an Mbed TLS root.
+### How to interpret non-compliance ###
+
+This script has relatively common false positives. In many scenarios, it only
+reports a pass if there is a strict textual match between the old version and
+the new version, and it reports problems where there is a sufficient semantic
+match but not a textual match. This section lists some common false positives.
+This is not an exhaustive list: in the end what matters is whether we are
+breaking a backward compatibility goal.
+
+**API**: the goal is that if an application works with the old version of the
+library, it can be recompiled against the new version and will still work.
+This is normally validated by comparing the declarations in `include/*/*.h`.
+A failure is a declaration that has disappeared or that now has a different
+type.
+
+  * It's ok to change or remove macros and functions that are documented as
+    for internal use only or as experimental.
+  * It's ok to rename function or macro parameters as long as the semantics
+    has not changed.
+  * It's ok to change or remove structure fields that are documented as
+    private.
+  * It's ok to add fields to a structure that already had private fields
+    or was documented as extensible.
+
+**ABI**: the goal is that if an application was built against the old version
+of the library, the same binary will work when linked against the new version.
+This is normally validated by comparing the symbols exported by `libmbed*.so`.
+A failure is a symbol that is no longer exported by the same library or that
+now has a different type.
+
+  * All ABI changes are acceptable if the library version is bumped
+    (see `scripts/bump_version.sh`).
+  * ABI changes that concern functions which are declared only inside the
+    library directory, and not in `include/*/*.h`, are acceptable only if
+    the function was only ever used inside the same library (libmbedcrypto,
+    libmbedx509, libmbedtls). As a counter example, if the old version
+    of libmbedtls calls mbedtls_foo() from libmbedcrypto, and the new version
+    of libmbedcrypto no longer has a compatible mbedtls_foo(), this does
+    require a version bump for libmbedcrypto.
+
+**Storage format**: the goal is to check that persistent keys stored by the
+old version can be read by the new version. This is normally validated by
+comparing the `*read*` test cases in `test_suite*storage_format*.data`.
+A failure is a storage read test case that is no longer present with the same
+function name and parameter list.
+
+  * It's ok if the same test data is present, but its presentation has changed,
+    for example if a test function is renamed or has different parameters.
+  * It's ok if redundant tests are removed.
+
+**Generated test coverage**: the goal is to check that automatically
+generated tests have as much coverage as before. This is normally validated
+by comparing the test cases that are automatically generated by a script.
+A failure is a generated test case that is no longer present with the same
+function name and parameter list.
+
+  * It's ok if the same test data is present, but its presentation has changed,
+    for example if a test function is renamed or has different parameters.
+  * It's ok if redundant tests are removed.
+
 """
 
 # Copyright The Mbed TLS Contributors