_sqlAliasManager unexpectedly null in SelectExpression.PushdownIntoSubquery() call after previous call of SelectExpression.Update()

[lauxjpn]

An existing SelectExpression is "updated" in SqlNullabilityProcessor.Visit(SelectExpression selectExpression, bool visitProjection) via a call to SelectExpression.Update(...).

On that resulting SelectExpression, we later call SelectExpression.PushdownIntoSubquery().

Because SelectExpression.PushdownIntoSubquery() uses _sqlAliasManager internally, and SelectExpression.Update(...) does not retain the SqlAliasManager instance of the original SelectExpression, _sqlAliasManager is null in the SelectExpression.PushdownIntoSubquery() call and an unexpected/implicit NRE is thrown.

Include provider and version information

EF Core version: 9.0.0
Database provider: Pomelo.EntityFrameworkCore.MySql

[roji]

Hey @lauxjpn 👋

Are you saying you call PushdownIntoSubquery at some point after in SqlNullabilityProcessor or afterwards, in your provider? That's not something we currently do elsewhere - can you provide a bit more context on where/why you need to do that?

To give more context, SelectExpression (currently) has two modes: mutable and immutable; a SelectExpression is only mutable during the translation phase, as more operators get processed and the SelectExpression get mutated; right after translation it is made mutable. This is a design that I think is problematic in several ways, and I hope to be able to change in 10, so that our SQL tree is fully immutable. In any case, the SQL alias manager is currently only available in the mutable phase.

Very concretely: rather than doing PushdownIntoSubquery(), which is meant to be used during the mutable phase, maybe it's possible to simply construct the new SelectExpression on top of the existing one (perform a "manual pushdown")? If not, I'll look into finding a solution for you.

Unfortunately, this entire area is quite messy at the moment, and in a transitional phase... But I'm slowly trying to move us in a more sane direction.

[lauxjpn]

Yeah, we're back baby!

Original post

Yes, we have a couple of expression visitors that we need to run as late as possible. I have now moved the one in question (the one that is doing the pushdown) into the post translation handling. The debugger shows though, that IsMutable is already false.

Since at this point, no call to SelectExpression.Update() seems to have happened yet, the SelectExpression still contains a valid _sqlAliasManager object and we can still successfully perform a pushdown, even though IsMutable is already false (SelectExpression.PushdownIntoSubquery() does not check IsMutable, it just tries to use the _sqlAliasManager object whenever called).

Unfortunately, we can't be 100% sure that nobody has already called SelectExpression.Update() (there is no official way to check), so we are now calling the following extension method before the pushdown, to ensure that there is a valid _sqlAliasManager object available in the SelectExpression:

public static SelectExpression UpdateWithSqlAliasManager(
    this SelectExpression selectExpression,
    SqlAliasManager sqlAliasManager)
    => new(
        selectExpression.Alias,
        selectExpression.Tables.ToList(),
        selectExpression.Predicate,
        selectExpression.GroupBy.ToList(),
        selectExpression.Having,
        selectExpression.Projection.ToList(),
        selectExpression.IsDistinct,
        selectExpression.Orderings.ToList(),
        selectExpression.Offset,
        selectExpression.Limit,
        selectExpression.Tags,
        selectExpression.GetAnnotations().ToDictionary(a => a.Name, a => a),
        sqlAliasManager,
        false);

We are still able to call this method in MySqlQueryTranslationPostprocessor.Process(), because we still have access to a QueryCompilationContext:

// In  MySqlQueryTranslationPostprocessor.Process():
query = new MySqlHavingExpressionVisitor(
    _sqlExpressionFactory,
    ((RelationalQueryCompilationContext)QueryCompilationContext).SqlAliasManager)
    .Visit(query);

// In MySqlHavingExpressionVisitor.VisitSelect:
selectExpression = selectExpression.UpdateWithSqlAliasManager(_sqlAliasManager);
selectExpression.PushdownIntoSubquery();

The question is: Is it a bug that we are able to do this successfully in MySqlQueryTranslationPostprocessor.Process() (and it could break in the future, so we should use a different way to accomplish what we want), or is this still supposed to work there, even though IsMutable is already false (and we can continue using it for now)?

If we should not do this, we will either move the expression visitor to the end of the translation phase (but before MySqlQueryTranslationPostprocessor) or do the pushdown manually (as you suggested).

(For context: The expression visitor in question has to do some weird syntax stuff for queries with HAVING clauses, if the SELECT statement does not contain an aggregate function. In such cases (and only in those), the GROUP BY clause needs to reference aliases (from the projection) for the expressions it uses (it cannot use the expressions themselves). We need to run this late in the pipeline, so that we can be sure that nobody changes or alters the query in a significant way).

I will reevaluate what we are doing with our HAVING expression visitor and come back to this, once I know in which direction we want to go with this.