Support denormalising data in references

[alcaeus]

Feature Request

Q	A
New Feature	yes
RFC	no
BC Break	no

Summary

When storing references as object (e.g. by using dbref or ref strategies), we can add data to the reference. This is currently only used to store a discriminator field if the reference or target document uses discriminator mappings, but could be used for any other fields from the additional document. For example, one could want to denormalise a username in a user reference:

{
    "title": "Test article",
    "author": {
        "id": <...>,
        "username": "alcaeus"
    },
    <...>
}

Why would users want to do this?

• Users could query articles by username without having to use the $lookup aggregation pipeline stage to filter by username
• Proxy objects generated for this reference would initialise the proxy with the username from the reference, meaning that accessing only the username would not require proxy initialisation
• This can be used to duplicate data for record-keeping purposes (think of storing product details with an order in an e-commerce platform).

One big issue in this whole thing is updating the stored value. In my initial proof of concept (see #1527), I mentioned that the denormalised value is not yet updated for two reasons:

• Tracking incoming references: this is a very big issue that requires an overhaul of the metadata system. In the example above, the user document would have to know that there is an incoming reference from an article document that needs updating. That is not reliably the case at the moment (it can be done using inverse references but there are better ways). One way to solve this would be to introduce an object graph to track metadata: this would allow us to efficiently traverse the metadata graph to find incoming references and update these values (if so specified in the options).
• Query support when updating nested values: at the time the proof of concept was created (a whopping 4 years ago), MongoDB did not support using multiple positional dollar operators in queries, limiting updating such references to non-nested fields. With array filters in update options added in MongoDB 3.6, we could take advantage of the $[<identifier>] operator to update such values when the original document is modified.

Both issues require extensive changes under the hood, especially to Metadata handling which is currently provided by the doctrine/persistence library. For that reason, I wouldn't want to have that as part of the first implementation, meaning that denormalised values are never updated. Updating will be an opt-in feature (due to potential performance implications). Due to potential inconsistencies, we should also hold off on building this opt-in feature until transaction support has been completed (which also requires large-scale changes under the hood).