Deprecate Partial Objects

[beberlei]

We plan to remove partial objects in the forseeable future. This doesn't have to be ORM 3, because it depends on a few alternatives being in place.

Partial objects are a bad fix for wanting to avoid loading too much data like text fields /blobs that are not needed in the request. Three solutions are proposed to migrate towards around this deprecation and removal:

• Use data transfer objects with the SELECT new MyDTO()
• Use new fetch=Eager support with Query::setFetchMode to implement multi-step hydration: [GH-1569] Optimize eager fetch for collections to batch query #8391
• Add support for lazy loading some properties or embeddables through a better proxy manager (this is not supported yet) Use ProxyManager #8518

As for use of PARTIAL with array hydration, a potential solution could be the introduction of a dedicated function: PARTIAL(alias, ...) that does the same that PartialExpression previously did. This could encapsulate this logic in ArrayHydrator and a specific function and avoid the polution of the DQL language with this concept.

Update: Deprecation reverted in ORM 2.20.0 and 3.3.0-dev

See #8471 (comment)

[gharlan]

I sometimes use the trick from the section "Coding multi-step hydration in Doctrine ORM" from https://ocramius.github.io/blog/doctrine-orm-optimization-hydration/.

What is the best way to migrate from this?

[pszalko]

I have the same concerns as @gharlan. Is there any way to migrate? @beberlei

[stepozer]

I believe the best way for this will be just allow developers to specify what exactly you want to load as we have in ActiveRecord in Laravel - https://laravel.com/docs/8.x/eloquent-relationships#eager-loading. I believe this feature will be useful to avoid this terrible joins with OneToMany hydration. And doctrine should automatically decide is it leftJoin or SELECT FROM IN request.

@beberlei what do you think?

[AlexisFinn]

Hello, just to be clear, do you plan on removing partial arrays, as in

$qb->selec('partial  user.{id, name}')
->getQuery()->getArrayResult();

Because that's actually really usefull for retrieving a nicely formatted array.

[raziel057]

I don't see any answer for the question raised by AlexisFinn

Can you reconsider the use of partial with Array Hydration ? I understand the issues given for Partial Object Hydration but there is no problem at all working with array. it seems better and less time consuming than having to define a Dto or having to use a scalar hydration with the need to specify each property and having all in a flat array.

Ex.:

$qb->select('d, p, partial f.{id, mimetype, name, size}')
            ->from(Document::class, 'd')
            ->join('d.file', 'f')
            ->leftJoin('d.publications', 'p')
            ->getQuery()->getArrayResult();

[raziel057]

From Doctrine page https://www.doctrine-project.org/projects/doctrine-orm/en/2.16/reference/partial-objects.html it's mentioned:

The partial object problem in general does not apply to methods or queries where you do not retrieve the query result as objects. Examples are: Query#getArrayResult(), Query#getScalarResult(), Query#getSingleScalarResult(), etc.

[beberlei]

@gharlan @pszalko for multi step hydration the new way to go will be #8391 this is essentially what @stepozer mentioned laravel also does.

As for array hydration with partial, this is something we haven't considered before, so its a valid use-case and I had the idea maybe this works with a function that can only be used in combination with array hydration:

SELECT PARTIAL(p, id, name, description) FROM product p

[bakugo]

@beberlei #8391 is not an adequate replacement because it does not give the user the same control over when and how the hydration happens. I need to be able to hydrate the associations I need, when I need them, in the queries I specify, so if PARTIAL is removed I'll be forced to just load already loaded entities multiple times for no reason.

Removing partial objects is understandable but there still should be some way to signal to DQL "only load the ID for this alias for association purposes because the relevant entities were already previously loaded", it could even throw an exception if it encounters an ID that isn't already loaded.

[klkvsk]

After reading all comments and links in this issue, it's still unclear how to load entities partially.
Referencing the example by @raziel057

$qb->select('d, p, partial f.{id, mimetype, name, size}')
            ->from(Document::class, 'd')
            ->join('d.file', 'f')
            ->leftJoin('d.publications', 'p')
            ->getQuery()->  getResult() // objects, not array!

So let's assume we want to get documents as objects with all relations, but there is a file.content field, that can be lots of data we won't ever need in that result, and we want to omit just it.

What @beberlei suggested

1. "SELECT new MyDTO()" -- it would not be possible to hydrate $document->file property, which is of class File, with a some other DTO object, isn't it?
2. "fetch=Eager" -- but fetch mode is applied on a related entity(-ies) as a whole, not their separate properties, isn't it?
3. "Add support for lazy loading some properties" -- well, sounds like that's it, but still not implemented.

So what do you actually suggest, except ignoring the deprecation warning until (3) is merged?

[beberlei]

@klkvsk yes, ignore the deprecation and stay on 2.x - you can call Deprecation::ignoreDeprecation (or similar) with the url identifer to silence it for now

[beberlei]

@bakugo you can set the assoc lazy by default and use Query::setFetchMode to change on a per query basis. This is the same level of granularity that PARTIAL offers

[bakugo]

@beberlei FETCH_EAGER is not adequate because it runs an extra query for each association.

The use case in question is basically this: #4762 (comment). I have one "main" entity that has a lot of fields and a lot of associations. If I use EAGER, I end up with more queries than necessary (and it cannot handle nested associations apparently). If I simply JOIN all the associations in one query, there will be too many of them and the query will be extremely slow and inefficient.

The only approach that reliably works is splitting the JOINs into multiple queries, aka multi-step hydration, where you select the "main" entity by ID multiple times and JOIN a different set of associations each time, such that each individual query is efficient, and you end up with all the associations you need hydrated in a low number of queries. The only real problem here is that each query will waste time selecting the fields of the main entity that were already loaded before. SELECT PARTIAL entity.{id} solves this, and does NOT result in partially loaded objects because those objects were already loaded, so there's no reason for it to not be allowed in this case.

[clesquir]

As for array hydration with partial, this is something we haven't considered before, so its a valid use-case and I had the idea maybe this works with a function that can only be used in combination with array hydration:

SELECT PARTIAL(p, id, name, description) FROM product p

We are using partials a lot in our application in order to return nicely formatted arrays. Converting all of them with DTOs will take weeks of work. Also, we need to include OneToMany in there which I was not able to achieve using a DTO.

Would this function be available in Doctrine 3.0 or is it only an idea?

If this is only an idea, can you provide some pointers on how to implement?

Thanks!

[lowwa132]

Same issue as @clesquir here, the DTO alternative is not suitable for OneToMany relations usecase.

[beberlei]

After long deliberation I see no other way than allowing PARTIAL keyword again in 3.0, but only in ArrayHydrator. We will work on adding this again in ORM 3.2 (as 3.0 and 3.1 are already released).

We will also change the deprecation in 2.x to only trigger when the PARTIAL keyword is used in combination with a hydrator that calls UnitOfWork::createEntity.

This will keep the keyword as long as no alternative API for this use-case is available, such as the Entity Graph API from JPA, https://www.baeldung.com/jpa-entity-graph#2-defining-an-entity-graph-with-the-jpa-api

This issue title and description will be updated accordingly once #11365 and #11366 are merged.

[fmonts]

I use partial in Symfony forms, like this:

->add('allowedUsers', EntityType::class, [
    'class' => User::class,
    'query_builder' => function (EntityRepository $er) use ($options) {
        return $er->createQueryBuilder('u')
            ->select('partial u.{id, username}, partial up.{user}, partial us.{user}')
            ->leftJoin('u.profile', 'up')
            ->leftJoin('u.settings', 'us')
            [...]
            ->orderBy('u.username', 'ASC');
    },
    'label' => 'Allowed users',
    'required' => false,
    'multiple' => true,
    'choice_label' => function (User $choice) {
        return $choice->getId() . ' - ' . $choice->getUsername();
    }
])

(profile and settings are a one-to-one, I need to join them otherwise it will do n queries to fetch them instead of 1, where n is the number of users)

Since I only need ID + username to display them in a dropdown, it's a nice optimization to improve speed and reduce the traffic from the db.

See the difference in performance, 42 ms vs 307:

Why deprecating this? I don't get it

[derrabus]

Since I only need ID + username to display them in a dropdown, it's a nice optimization to improve speed and reduce the traffic from the db.

Why do you hydrate objects anyway if you only need two scalar values?

[idybil]

Since I only need ID + username to display them in a dropdown, it's a nice optimization to improve speed and reduce the traffic from the db.

Why do you hydrate objects anyway if you only need two scalar values?

Is this because of the EntityType which is expecting an entity ? so the persisting of the entity is managed automatically by the form.

[derrabus]

I see. And if you select an entity with this form, you would also get a partially hydrated one as a result, right?

Can you please open an issue on the Symfony tracker and link to this issue here? Maybe we can improve the EntityType so you don't need to use partial hydration anymore. But discussing that in this issue would be at bit out of scope.

[idybil]

I see. And if you select an entity with this form, you would also get a partially hydrated one as a result, right?

Yes

Can you please open an issue on the Symfony tracker and link to this issue here? Maybe we can improve the EntityType so you don't need to use partial hydration anymore. But discussing that in this issue would be at bit out of scope.

@fmonts , do you have time to open an issue on the Symfony tracker ?

[cbichis]

@beberlei maybe we can still keep partial also when using \Doctrine\ORM\Query::HINT_READ_ONLY ? Not just for array hydration?

[kevinpapst]

I was trying to tackle a few deprecated PARTIALS today, but found that the status quo is not yet in feature parity with the PARTIAL construct. Reading through several issues, I see that many others run into the same problems.

My main 3 issues which prevent me from upgrading to ORM v3 are:

• better support for ManyToMany collections (currently O(n) as 1 query per entity is triggered)
• support for multi-level hydration
• support for hydrating collections of already initialized entities (least important, would minimize refactoring time)

Is there any consideration to tackle the missing points?

[beberlei]

@kevinpapst these disclaimers apply to an upgrade to v3 as per our blog posts, so there is no rush if you use Partials and when subselect / eager fetching does not work for you yet:

Current users of ORM 2 should note that there is no urgency right now to update to ORM 3, as we are still working on replacement APIs and forward compatibility, and do not intend to ship them all with ORM 3.0, but with later versions.

For some of the deprecations in ORM, we are still planning replacement APIs, especially:

• There is currently no way to limit the number of entities that the flush operation considers changed. Flush will currently always calculate change sets on all entities that are not read-only.
• As a replacement for removing PARTIAL object hydration, we are looking at making embeddable objects lazy, perhaps improving nesting of the new DTO expression in DQL. We are also looking to introduce subselect or batch loading for collections for more efficient multi-level hydration.

These will be released in 2.x as forward compatible APIs so that you can switch to using them before upgrading to ORM 3.

[beberlei]

@beberlei maybe we can still keep partial also when using \Doctrine\ORM\Query::HINT_READ_ONLY ? Not just for array hydration?

no, because it can still break the assumptions that when you have a reference from the outside, you expect it to be accessible. With the identity map, you could still load that partial object early in the request and then in a totally different query get the partial object returned from the identity map, where you expect a full one to be.

[beberlei]

An update from the current Doctrine core team meetup: With PHP 8.4 Lazy Object RFC, there is way to use proxies in combination with partial objects that would make it possible to keep partial objects in a way, where they lazy load triggers a database fetch of the missing columns.

This might even be possible with versions earlier using symfony/var-exporter lazy ghosts, but we need to see if this is true and possible.

Course of action therefore is that we will fully re-introduce partial objects in 3.x or 4.x in combination with PHP 8.4 and fully undeprecate it in 2.20.x. There are a few details to carve out about the exact roll out plan.