Add Support For MongoDB Client Side Field Level Encryption (CSFLE)

[dill0wn]

This implementation was guided largely by pymongo's examples on explicit, manual encryption.

These changes add a lot:

• The Document class now supports automatic encryption and decryption of individual fields. It gains this ability by inheriting from ming.encryption.EncryptedDocumentMixin and leveraging the new quasi-field class ming.encryption.DecryptedField.

• Encryption configuration options are handled by ming.encryption.EncryptionConfig

  • ming.configure automatically parses flattened config values like you would see in an ini. For example:

  ming.main.encryption.kms_providers.local.key = ENCR_KEY
  ming.main.encryption.key_vault_namespace = encryption_test.dataKeyVault
  ming.main.encryption.provider_options.local.key_alt_names = ["datakey_test1"]
  

• A new formencode validator ming.validators.EncryptionConfigValidator has been added to validate config dict values.

• For runtime usage, the EncryptionConfig is added to the DataStore instance as a new DataStore.encryption instance property. This is what individual Documents reference when performing encryption/decryption.

• Added new package dependencies: pymongo[decryption] and cachetools

• Added tests for new encryption features

• Added new demo docs/presentations/demo_encryption.py

See pymongo's documentation on encryption for further details on the implementation and expected configuration.

• https://pymongo.readthedocs.io/en/stable/examples/encryption.html
• https://pymongo.readthedocs.io/en/stable/api/pymongo/encryption.html

[codecov]

Codecov Report

Attention: Patch coverage is 95.39474% with 21 lines in your changes missing coverage. Please review.

Project coverage is 91.42%. Comparing base (637a033) to head (f335589).
Report is 22 commits behind head on master.

Files with missing lines	Patch %	Lines
ming/encryption.py	92.55%	7 Missing ⚠️
ming/validators.py	91.30%	6 Missing ⚠️
ming/datastore.py	90.47%	4 Missing ⚠️
ming/tests/test_encryption.py	99.01%	2 Missing ⚠️
ming/odm/mapper.py	87.50%	1 Missing ⚠️
ming/odm/property.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
+ Coverage   90.19%   91.42%   +1.22%     
==========================================
  Files          43       47       +4     
  Lines        6724     7146     +422     
==========================================
+ Hits         6065     6533     +468     
+ Misses        659      613      -46     

Flag	Coverage Δ
tests-3.10	91.60% <95.39%> (+1.21%)	⬆️
tests-3.11	91.60% <95.39%> (+1.21%)	⬆️
tests-3.12	91.60% <95.39%> (?)
tests-3.7	?
tests-3.8	?
tests-3.9	91.41% <95.39%> (+1.22%)	⬆️
tests-pypy3.9	91.54% <95.39%> (+1.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dill0wn]

This work is not ready to merge yet as it only covers Documents. We still need to support MappedClasses.

[brondsem]

can delete this line, since self is the datastore now!

[dill0wn]

Fixed

[brondsem]

in def include_in_repr(self): lets add Binary types to the skip list. I tested repr(u) in one of the encryption tests to see what it'd look like. I don't think we want binary!

[dill0wn]

Fixed

[brondsem]

I'm ok with this for now, to get this MR merged. But we probably should make it work on 4.9 next, before making a release 😐 In case anyone's already using ming with a higher pymongo version

[dill0wn]

Agreed.