DatabaseFacade.Migrate - please allow --namespace as argument

[Tommigun1980]

Hi.

I have multiple sets of migrations as per #21443, and would like to run auto migration for the local and development environments.

DatabaseFacade.Migrate doesn't accept any arguments, so it will try to run all migrations in the project. Running

dotnet ef database update --namespace=MyApp.MobileAppService.Migrations.Development --context ContextName

allows for the --namespace argument, which will pick the correct "set" of migrations.

I'd like to do the same programmatically, but DatabaseFacade.Migrate() doesn't support it. So please add an overload,

DatabaseFacade.Migrate(string namespace)

so it reaches parity with the command line migration capability.

---

Here is an example of what I'd like to do:

if (this.IsLocalOrDevelopmentEnvironment())
{
    var migrationNamespace = $"MyApp.MobileAppService.Migrations.{this.environment.EnvironmentName}";
    myContext.Database.Migrate(migrationNamespace);
}

which would be analogous to running:

dotnet ef database update --namespace=MyApp.MobileAppService.Migrations.$ENVIRONMENT --context MyContext

[ajcvickers]

Note from triage: we will need to investigate this, since @bricelam indicated that the namespace should not be used for database update, but the assertion here is that it is used. Once we understand this we can consider whether to enable this at least in IMigrator, if not in the Migrate sugar API.

[Tommigun1980]

Note from triage: we will need to investigate this, since @bricelam indicated that the namespace should not be used for database update, but the assertion here is that it is used. Once we understand this we can consider whether to enable this at least in IMigrator, if not in the Migrate sugar API.

Hi @ajcvickers and thanks for the reply. I admit that it gets a bit messy with multiple migration "tracks", but I couldn’t figure out any other way of running multiple environments in different stages of development and possibly running different databases. If this is not the intended way, please advice on the proper procedure? I couldn’t unfortunately find any guidance in the documentation for this.

Thanks so much.

[bricelam]

You need different assemblies to separate migrations sets (but EF6 used namespaces). Switch between them using the ASP.NET environment:

options.UseSqlServer(
    connectionString,
    x => x.MigrationsAssembly(
        $"MyApp.Migrations.{env.EnvironmentName}"));

SET ASPNETCORE_ENVIRONMENT=Production
dotnet ef database update

[bricelam]

Also, you may have been running into an issue with #8332 where parameters (e.g. --namespace) were allowed where they shouldn't have been.

[Tommigun1980]

Hi @bricelam and thanks for the info. With your help I was able to find https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/projects?tabs=dotnet-core-cli which outlines the steps in further detail.

But I don't understand how the steps in that guide could ever work, nor could I get them to work, so I'd really appreciate some further insight into this.
I now have the following projects - MyApp.MobileAppService, MyApp.MobileAppService.Migrations.Local and MyApp.MobileAppService.Migrations.Development.

MyApp.MobileAppService.Migrations.Local and MyApp.MobileAppService.Migrations.Development both have an assembly reference to

    <Reference Include="MyApp.MobileAppService">
      <HintPath>..\MyApp.MobileAppService\bin\$(Configuration)\netcoreapp3.1\MyApp.MobileAppService.dll</HintPath>
    </Reference>

and they build just fine.

However, MyApp.MobileAppService would get a circular dependency no matter what I do in case I add an assembly reference to the migration projects. https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/projects?tabs=dotnet-core-cli has a hint about this but I don't see how that would help at all, nor did it help when I tried it, as the circular dependencies will be there no matter the location where the assembly is built. If I understand that guide correctly it asks me to add assembly references both from the migration projects to the server's project, and from the server's project to the migration projects.

If I build MyApp.MobileAppService I get the following error:

/usr/local/share/dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(5,5): Error MSB4018: The "GenerateDepsFile" task failed unexpectedly.
System.ArgumentException: An item with the same key has already been added. Key: MyApp.MobileAppService
  at System.Collections.Generic.Dictionary`2[TKey,TValue].TryInsert (TKey key, TValue value, System.Collections.Generic.InsertionBehavior behavior) [0x001cc] in <55adae4546cd485ba70e2948332ebe8c>:0 
  at System.Collections.Generic.Dictionary`2[TKey,TValue].Add (TKey key, TValue value) [0x00000] in <55adae4546cd485ba70e2948332ebe8c>:0 
  at System.Linq.Enumerable.ToDictionary[TSource,TKey] (TSource[] source, System.Func`2[T,TResult] keySelector, System.Collections.Generic.IEqualityComparer`1[T] comparer) [0x0000e] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-12/external/bockbuild/builds/mono-x64/external/corefx/src/System.Linq/src/System/Linq/ToCollection.cs:82 
  at System.Linq.Enumerable.ToDictionary[TSource,TKey] (System.Collections.Generic.IEnumerable`1[T] source, System.Func`2[T,TResult] keySelector, System.Collections.Generic.IEqualityComparer`1[T] comparer) [0x00043] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-12/external/bockbuild/builds/mono-x64/external/corefx/src/System.Linq/src/System/Linq/ToCollection.cs:59 
  at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WritePortableTarget (System.Collections.Generic.IReadOnlyList`1[T] runtimeLibraries, System.Collections.Generic.IReadOnlyList`1[T] compilationLibraries) [0x00025] in <fa4af318e8374ad18eef8b73f115183f>:0 
  at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WriteTargets (Microsoft.Extensions.DependencyModel.DependencyContext context) [0x00025] in <fa4af318e8374ad18eef8b73f115183f>:0 
  at Microsoft.Extensions.DependencyModel.DependencyContextWriter.Write (Microsoft.Extensions.DependencyModel.DependencyContext context) [0x00032] in <fa4af318e8374ad18eef8b73f115183f>:0 
  at Microsoft.Extensions.DependencyModel.DependencyContextWriter.Write (Microsoft.Extensions.DependencyModel.DependencyContext context, System.IO.Stream stream) [0x00031] in <fa4af318e8374ad18eef8b73f115183f>:0 
  at Microsoft.NET.Build.Tasks.GenerateDepsFile.WriteDepsFile (System.String depsFilePath) [0x00290] in <ca130b05ecf14a059003fdf2ce5040bc>:0 
  at Microsoft.NET.Build.Tasks.GenerateDepsFile.ExecuteCore () [0x00007] in <ca130b05ecf14a059003fdf2ce5040bc>:0 
  at Microsoft.NET.Build.Tasks.TaskBase.Execute () [0x0002e] in <ca130b05ecf14a059003fdf2ce5040bc>:0 
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00023] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-12/external/bockbuild/builds/msbuild-15/src/Build/BackEnd/TaskExecutionHost/TaskExecutionHost.cs:575 
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x002a9] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-12/external/bockbuild/builds/msbuild-15/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs:818  (MSB4018) (MyApp.MobileAppService)

I have added the dependencies in MyApp.MobileAppService to the migration projects as such:

  <ItemGroup>
    <Reference Include="MyApp.MobileAppService.Migrations.Local">
      <HintPath>bin\$(Configuration)\netcoreapp3.1\MyApp.MobileAppService.Migrations.Local.dll</HintPath>
    </Reference>
    <Reference Include="MyApp.MobileAppService.Migrations.Development">
      <HintPath>bin\$(Configuration)\netcoreapp3.1\MyApp.MobileAppService.Migrations.Development.dll</HintPath>
    </Reference>
  </ItemGroup>

Is there a non-spoken step of splitting MyApp.MobileAppService to multiple assemblies or something? I don't want to do that as they work just fine in one assembly and that would add unnecessary overhead. I just want to run the migrations, not completely revamp my server. But I guess this is not the case as the documents would probably have mentioned something about it.

So yeah, any further insight into this would be greatly appreciated as I don't see how the guide could work, and it feels like I'm doing something wrong as I'm hacking around bad design regarding the assemblies. Is it really the case that both the server and the migration projects need dependencies on each other, and if that really is the case, how can I get around it?

Furthermore, if the main project has an assembly reference to the migration projects, where the migration projects build into the main project's folder (as described in the guide), I will be forced to always manually build the migration projects first and also to rely on a weird weird build step that sounds nightmarish to maintain. Maybe the guide meant a project reference and not an assembly reference (but that wouldn't solve the circular dependency either)?

The only way I think there's a chance this could work is if the documentation meant that if the build script of the migration projects is edited to output the assembly to the build folder of the main project, then skip the step of adding a dependency from the main project to the migration projects, but it doesn't tell one to do that so I have not idea what it's trying to say. But even in that case it sounds to me like a design mistake.

Thanks so much again!

---

Edit: https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/projects?tabs=dotnet-core-cli says to run

dotnet ef migrations add NewMigration --project MyApp.Migrations

but doing that will try to read the DbContexts from the migrations project which of course won't work. dotnet ef add --help says that there's a command line option, --startup-project which I think I'd need, but I'm scared to use it because

1. https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/projects?tabs=dotnet-core-cli doesn't say you need it and one could assume a tutorial to be correct and complete.
2. Last time I added command line arguments to dotnet ef, we ended up with this thread.

---

Edit 2: Tbh I think you guys might need to go back to the drawing board because if the design really relies on circular dependencies and/or editing build scripts to just about get around it, it's not a sustainable design. Also the documentation is almost impossible to follow at this point no matter how correctly you try to do it. My guess is that --startup-project is needed and the guide omits it, and that the section that asks one to edit the build script, omits that when doing so one should no longer add a reference back from the main project to the migration projects. But what I don't understand is when it says "If this causes a circular dependency, update the output path of the class library:" -- how it is a case of if as that will create a circular dependency. And I'd rather not guess if that can be avoided.

[Tommigun1980]

When I followed through with all guesses in Edit 2 it seems to work to a certain extent, but I don't think this is correct in any way.

1. It really seems as one has to add assembly dependencies (as adding a project dependency doesn't seem to work), but I'm not committing the artifacts so if I clean my projects I'll end up with broken link icons in VS, or worse, VS won't display the dependencies at all until they are built and/or VS is restarted after a build.
2. I have to manually build my migration projects in VS before dotnet ef migrations add will work, as I was afraid of. I can't rely on this kind of state.
3. With this approach I can add migrations, but when I run them as part of the server startup code, it complains about not finding the migration assembly. If I then once again add references to the migration assemblies (residing under the server project's bin folder's current configuration, ie where the migration projects build it), I am back at this error when building:

/usr/local/share/dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(5,5): Error MSB4018: The "GenerateDepsFile" task failed unexpectedly.
System.ArgumentException: An item with the same key has already been added. Key: MyApp.MobileAppService

This, in conjunction with the dependency on circular design and lacking/incorrect documentation makes me want to go back to using namespaces. I am exactly where I started, still can't run them as part of server deployment, but with a ton of overhead. Please clarify what parts of the document are wrong or what I am misunderstanding.

Thanks again.

[bricelam]

We definitely need better docs on this. It is very hard to get right and understand. Tracking issue: dotnet/EntityFramework.Docs#933 (comment)

[Tommigun1980]

Is there a reason this is so overly complicated to begin with? It was so much easier with namespaces than with circular dependencies and magical build steps to create the assemblies etc? This is such a complex endeavour that even with proper documentation I think something has gone wrong in the design.

Thanks.

[ajcvickers]

@Tommigun1980 We will continue to look into ways the make the process this easier. For now, the issue mentioned above (dotnet/EntityFramework.Docs#933 (comment)) is tracking documentation updates to provide better guidance in docs.