Transaction support

[Steveb-p]

Feature Request

Introduce transaction support on database level as MongoDB starts to allow since version 4.0.

Q	A
New Feature	yes
RFC	yes/no
BC Break	should not be a BC break

Summary

I'd like to ask for opinion and discuss future solutions regarding transactions in Doctrine ODM.

MongoDB documentation regarding transactions: https://docs.mongodb.com/manual/core/transactions/

There are a few relevant bits of information regarding this feature. Specifically:

In most cases, multi-document transaction incurs a greater performance cost over single document writes, and the availability of multi-document transaction should not be a replacement for effective schema design.

Multi-document transactions are available for replica sets only. Transactions for sharded clusters are scheduled for MongoDB 4.2

What is Doctrine ODM desired approach on this? Introducing beginTransaction(), transactional() methods into DocumentManager? Since MongoDB transactions incur performance penalty, they probably will not be enabled on flush by default?

[jmikola]

@alcaeus: FYI, we have a new session API in the specifications pipeline for MongoDB drivers, which is intended to also improve the API around transactions. ETA for those APIs will be for the driver versions supporting MongoDB 4.2. It may be far off, but it's is something we can keep in mind when brainstorming about how best to integrate transactions into ODM.

On a related note: https://github.com/p-mongo/tests/blob/master/docs/mongoid-sessions-transactions.md

[Ludo444]

If somebody passes around this, it's pretty easy to implement with ODM 2.0 by passing session into flush() method $options

$session = $dm->getClient()->startSession();
$session->startTransaction();

// persist some documents
$dm->flush(['session' => $session]);

$session->commitTransaction();
$session->endSession();

Ideally you can override DocumentManager and store session there.

[alcaeus]

I'll just note here that the above is not suggested: this will lead to the UnitOfWork assuming that all data was written successfully, while the commitTransaction call might still lead to errors. In that case, the UnitOfWork will be left in a very inconsistent state.

[Ludo444]

Fair point, in my case it's used with DEFERRED_EXPLICIT policy and in case of any error I am persisting all documents again.

[DavideTriso]

Hello, I would like to contribute and work on an implementation of this feature. Could you please provide some advice and tell me how the desired implementation should work?

[alcaeus]

@DavideTriso That is great news!

The first step would be basic transaction handling: have the UnitOfWork start a session (if the feature is enabled), do its usual commit operations, then commit the transaction and handle any errors. From then on, we can see what other features we want to add. Note that we can't make this an "always-on" feature, as transactions require replica sets which may not be available. Thus, I'd add this to the configuration and allow users to turn transactions on or off as necessary.

To get started, I recommend forking from the "master" branch, which always points to the upcoming minor release and creating a draft PR as a discussion point. If you need more guidance, also feel free to contact us in the Doctrine Slack via the #mongodb-odm channel and we can go from there. The resources @jcaillot are a good starting point for the non-technical side of things, so go ahead and check those out if you need help.

Thanks to both of you for moving this along!

[DavideTriso]

Tanks for your advice @alcaeus and @jcaillot.

I have given a nearest look at the mongodb-odm code before starting to work at the implementation
This is how I would implement this feature:

1. Add a new enableSession attribute to Doctrine\ODM\MongoDB\Configuration which defaults to false.
2. When enableSession is true, have the DocumentManager create a new session in the constructor and store it.
3. In DocumentManager::flush(), let the user decide if they want the operation to be performed in a multi-document transaction or not (maybe by adding a bool $transactional = false argument to the flush method?)
4. If the user requested the flush operation to be performed in a transaction, start a transaction and commit it in UnitOfWork::commit().

Please let me know your thoughts!

[Steveb-p]

2. When enableSession is true, have the DocumentManager create a new session in the constructor and store it.

Session should be "opened" as late as possible, possibly only before flush call or explicit DocumentManager::beginTransaction. They should be as short-lived as possible. I'm not an expert in MongoDB, but I'm pretty sure for high throughput multi-document transactions you would start getting into deadlocks.

3. In DocumentManager::flush(), let the user decide if they want the operation to be performed in a multi-document transaction or not (maybe by adding a bool $transactional = false argument to the flush method?)

I believe current approach is to leave flush method without arguments (just recently a single entity/document flush was removed - true that it was mostly because of issues with ORM/ODM database desync, but I think it's also against the purpose those tools are for).
Introducing one option argument would also cause issues in cases where another similar option is introduced as second and so on.

An argument can be made that if transactions are enabled in configuration, then it should become the default way of handling flush operations. In this case transaction should be explicitly disabled instead (mostly for batch or simple operations).

Just my two cents ;)

[alcaeus]

To elaborate on what @Steveb-p commented:

Session should be "opened" as late as possible, possibly only before flush call or explicit DocumentManager::beginTransaction.
I agree. If we were to start the transaction in the constructor, this would also directly open the connection to the server, which is not desired. Since there are no plans to use sessions on operations outside of flush, we shouldn't keep an open session around.

I believe current approach is to leave flush method without arguments [...]
There is an $options array, but I'd have to double-check to see if the contents of it are passed through to the persisters performing the individual write commands. If the configuration says to flush with transactions, we do so. If we want to be able to configure this on a per-call basis, I'd take care of this in a separate PR.

IMO, this should happen as follows:

1. If configuration has enableCommitTransaction false, don't change current behaviour
2. If configuration has enableCommitTransaction true, start a session, start a transaction, and make sure the session is used in all commands sent to the server during the commit.

From there, we can expand this. For example, we may want to let the user pass their own session into flush, in which case we start a transaction in the provided session instead of starting a new session. I don't see this as part of this scope though - the most important part is making sure that all commands run in a transaction.

As for testing this, just some notes that might make your life easier:

• Sessions and transactions require a replica set. Using the provided configurations (if you're using mongo-orchestration) or mtools can help you set up a cluster to run tests
• For tests, using command monitoring allows you to inspect the commands UoW generates and to ensure that they all contain an lsid field for the session in question.
• While transactions are available starting MongoDB 4.0, you need to run MongoDB 4.2 for them to work on sharded clusters. To detect what topology the tests are running on, you can copy over the helpers that we use in the PHP library: https://github.com/mongodb/mongo-php-library/blob/474ec83cbe7f9a464b5edfdc4d5ca2ac6fded2f0/tests/FunctionalTestCase.php#L304..L338.

[delta4op]

@DavideTriso Hey,
Any update on transactions support ?

I tried implementing this using this blog.
https://zgadzaj.com/development/mongodb/mongodb-multi-document-transactions-in-symfony-4-with-doctrine-and-mongodb-odm-bundle#symfony-doctrine

Though I get the following error while passing session to flush()
The \"writeConcern\" option cannot be specified within a transaction. Instead, specify it when starting the transaction.

I want to develop my new application based on this package but I am just unsure because of its lack of support for multi-document transactions.

[DavideTriso]

Hello @gitwithravish,

the error you have reported is related to how MongoDB works and not to the ODM itself.
When using multi-document transactions the write concern must be set on the whole transaction, which means you cannot set this on a single operation (e.g. when calling $dm->persist()).

Implementing multi-document transactions in ODM is currently not possible without BCs.
See #2220 for the details.

As far as I know, if you need multi-document transactions you have to handle them yourself, for example by wrapping the DocumentManager methods in another class which handles the transaction (start, commit, rollback). Also remember you cannot use the ODM events when doing this, because the ODM is not aware of the transaction and will fire the postUpdate, postRemove, postPersist events as soon as the DB command is executed, but if you are inside a transaction the operations can still be rolled back if an error occurs.

[delta4op]

@DavideTriso
Thanks for reverting back.

I did not pass session in $dm->persist() method. I passed it in ->flush() method as mentioned here. This implementation is very old and it was demonstrated with version 1 odm package and it worked as well. However its not working in odm 2.2.

Currently I have my entire application built upon doctrine-odm package. But wherever I am required to use transactions, I am using mongodb php driver functions directly to start a session, pass the session to all transactions and then commit/abort.

Issue that I have to use 2 database communication approaches. Some using docrine-odm and some using pure php mongodb driver. Can you suggest me when and where do I pass a session to doctrine-odm package and make use of existing functionalities ?

I understand that there is no straightforward way to this but since you have been onto this since a long time, you will be able to help me out with this i believe.

Can you brief me about the following statement ?

if you need multi-document transactions you have to handle them yourself, for example by wrapping the DocumentManager methods in another class which handles the transaction (start, commit, rollback)

This approach has also done something similar but it didnt do the job for me.

[DavideTriso]

@gitwithravish I will reach you on Slack for further details

[jseparovic1]

@DavideTriso @gitwithravish If you have any conclusions please write it here. Thank you :)

[DavideTriso]

My conclusion is that it is possible to use multi-document transactions with odm by implementing your additional layer to handle them, but with some limitations. This approach is only possible if you do not rely on the built-in events system of doctrine, because odm is not aware of the transaction and fires the “post-flush” events as soon as the db command is executed.
The other problem is that after a rollback the unit of work is in an inconsistent state which must be handled “manually” (e.g. clear or restore the state the uow had before the flush call).

[cousinjoe]

Has there been any movement on this in the past 1.5 years? MongoDB 4.2 has been out since August 2019 supporting multi-document transactions. It would be great to be able to strictly use this library as opposed to having to also use the mongo/php library for this use case.

[alcaeus]

@cousinjoe no, there has not. @DavideTriso summed up the issue quite well, as in that the UnitOfWork ends up in an inconsistent state if a flush does not succeed. In ORM, an error in a transaction leads to a closed entity manager, which is an irrecoverable state (meaning that you have to reset the EM leading to all entities becoming invalid). Due to the way MongoDB works we are a lot more flexible and the driver is able to recover from such errors. However, the way UnitOfWork is implemented means that once a flush fails, it can't be executed again. This makes transactions only semi-useful and introduces this broken state into UnitOfWork. This is something I'd like to avoid at all costs.

Unfortunately, this means that UnitOfWork requires significant changes. To be fair, it would also benefit from these changes even when not using transactions, as it can already end up in a very broken state when writes fail.

[alcaeus]

This was implemented in #2586 and will be released in 2.7.0 🎉