InvalidOperationException when using dynamically created expressions with projections to derived types

[GeorgDangl]

In LightQuery, I'm dynamically creating expressions for sorting (OrderBy and OrderByDescending) ObjectResults in ASP.NET Core where the Value is a IQueryable. It's basically an attribute that applies sorting and pagination parameters from the Http request to the results.

With netcoreapp3.0 and Entity Framework Core 3.0.0, a regression was reported for a specific use case: When the IQueryable has a projection to a derived class, an InvalidOperationException is thrown.

Below is a complete repro, it's been verified both with SQLite and SQL Server. InMemory works fine. Additionally, there's a project from @binard available that demonstrates the issue in action: https://github.com/binard/testlq

Steps to reproduce

The below sample demonstrates the issue with Microsoft.EntityFrameworkCore.Sqlite version 3.0.0. The code works fine with earlier versions, e.g. 2.1.3.
ExpressionTest.cs

using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using System;
using System.Linq;
using System.Linq.Expressions;
using System.Threading.Tasks;
using Xunit;

namespace LightQuery.EntityFrameworkCore.Tests
{
    public class ExpressionTest
    {
        public class UserBase
        {
            public int Id { get; set; }
            public string Name { get; set; }
        }

        public class User : UserBase { }

        public class AppDbContext : DbContext
        {
            public AppDbContext(DbContextOptions options) : base(options) { }
            public DbSet<UserBase> Users { get; set; }
        }

        [Fact]
        public async Task CanOrderBy()
        {
            var context = new ServiceCollection()
                .AddDbContext<AppDbContext>(options => options
                .UseSqlite(GetSqliteInMemoryConnectionString()))
                .BuildServiceProvider()
                .GetRequiredService<AppDbContext>();
            await context.Database.EnsureCreatedAsync();

            var usersQueryable = context.Users
                .Select(u => new User
                {
                    Name = u.Name
                });

            var orderByExp = CreateExpression(typeof(User), "Name");
            var wrappedExpression = Expression.Call(typeof(Queryable),
                "OrderBy",
                new[] { typeof(User), typeof(string) },
                usersQueryable.Expression,
                Expression.Quote(orderByExp));

            var constructedQueryable = usersQueryable.Provider.CreateQuery(wrappedExpression);
            dynamic dynamicQueryable = constructedQueryable;

            // Throws only in netcoreapp3.0:
            /*
            System.InvalidOperationException : The LINQ expression 'OrderBy<UserBase, string>(
                source: DbSet<UserBase>, 
                keySelector: (u) => new User{ Name = u.Name }
                .Name)' could not be translated. Either rewrite the query in a form that can be translated,
                or switch to client evaluation explicitly by inserting a call to either AsEnumerable(),
                AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
             */

            var result = await EntityFrameworkQueryableExtensions.ToListAsync(dynamicQueryable);
            Assert.NotNull(result);
        }

        public static LambdaExpression CreateExpression(Type type, string propertyName)
        {
            var param = Expression.Parameter(type, "v");
            Expression body = param;
            body = Expression.PropertyOrField(body, propertyName);
            return Expression.Lambda(body, param);
        }

        private static SqliteConnection _connection;
        private static string GetSqliteInMemoryConnectionString()
        {
            var connectionString = new SqliteConnectionStringBuilder
            {
                DataSource = "in-memory.db",
                Cache = SqliteCacheMode.Shared,
                Mode = SqliteOpenMode.Memory
            }.ToString();
            _connection = new SqliteConnection(connectionString);
            _connection.Open();
            return connectionString;
        }
    }
}

Here's the complete stacktrace of the xUnit test:

Message: 
    System.InvalidOperationException : The LINQ expression 'OrderBy<UserBase, string>(
        source: DbSet<UserBase>, 
        keySelector: (u) => new User{ Name = u.Name }
        .Name)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
  Stack Trace: 
    QueryableMethodTranslatingExpressionVisitor.<VisitMethodCall>g__CheckTranslated|8_0(ShapedQueryExpression translated, <>c__DisplayClass8_0& )
    QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
    Database.CompileQuery[TResult](Expression query, Boolean async)
    QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
    <>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
    CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
    CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
    QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
    EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
    EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)
    ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()
    EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
    CallSite.Target(Closure , CallSite , Object )
    UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
    ExpressionTest.CanOrderBy() line 64
    --- End of stack trace from previous location where exception was thrown ---

I'd be very happy to hear about any possible workarounds😊

Further technical details

EF Core version: 3.0.0
Database provider: Microsoft.EntityFrameworkCore.Sqlite
Target framework: .NET Core 3.0
Operating system: Current Windows 10
IDE: Visual Studio 2019 16.3.4

[smitpatel]

Your issue is bad reflection.
In precise terms,

((MemberExpression)((LambdaExpression)((UnaryExpression)((MethodCallExpression)constructedQueryable.Expression).Arguments[1]).Operand).Body).Member == ((MemberAssignment)((MemberInitExpression)((LambdaExpression)((UnaryExpression)((MethodCallExpression)usersQueryable.Expression).Arguments[1]).Operand).Body).Bindings[0]).Member

MemberInfo for Name in projection is different from Name in v.Name which is in order by. If they are same, EF Core simplifies it. Since they are different we could not and since SQL cannot represent user, it throws exception.

Further, when I write the order by manually,

var usersQueryable = context.Users
                .Select(u => new User
                {
                    Name = u.Name
                })
                .OrderBy(v => v.Name)

The MemberInfo for both Name is the same and EF Core translates it correctly.

[GeorgDangl]

Thank you very much, I've been able to locate the error and fix the issue!

[onionhammer]

@GeorgDangl

Can you please share your solution?

[GeorgDangl]

Hey @onionhammer, here's the relevant commit from LightQuery: GeorgDangl/LightQuery@5343116#diff-29ac354021182e6dbc1a86496f11807f

[smitpatel]

We are enabling support for this scenario in #19182