POC: Remove Azure Identity from Microsoft.Data.SqlClient #2247

ErikEJ · 2023-11-28T17:58:07Z

(to demonstrate where dependencies are used and test a build)

relates to #1108

Following linked files were removed from the .csproj:

    <Compile Include="..\..\src\Microsoft\Data\SqlClient\ActiveDirectoryAuthenticationProvider.cs">
      <Link>Microsoft\Data\SqlClient\ActiveDirectoryAuthenticationProvider.cs</Link>
    </Compile>
    <Compile Include="..\..\src\Microsoft\Data\SqlClient\ActiveDirectoryAuthenticationTimeoutRetryHelper.cs">
      <Link>Microsoft\Data\SqlClient\ActiveDirectoryAuthenticationTimeoutRetryHelper.cs</Link>
    </Compile>
 <Compile Include="..\..\src\Microsoft\Data\SqlClient\AzureAttestationBasedEnclaveProvider.cs">
      <Link>Microsoft\Data\SqlClient\AzureAttestationBasedEnclaveProvider.cs</Link>
    </Compile>

codecov · 2023-11-29T11:36:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (507e8dd) 72.66% compared to head (5c6391d) 72.44%.

❗ Current head 5c6391d differs from pull request most recent head 022bce6. Consider uploading reports for the commit 022bce6 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2247      +/-   ##
==========================================
- Coverage   72.66%   72.44%   -0.22%     
==========================================
  Files         310      309       -1     
  Lines       61877    61963      +86     
==========================================
- Hits        44960    44887      -73     
- Misses      16917    17076     +159

Flag	Coverage Δ
addons	`92.88% <ø> (ø)`
netcore	`76.64% <100.00%> (-0.28%)`	⬇️
netfx	`69.84% <ø> (-0.35%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AraHaan · 2023-12-07T00:32:59Z

src/Microsoft.Data.SqlClient.sln

@@ -206,6 +200,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{4F3CD363-B1E
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.SqlServer.Server", "Microsoft.SqlServer.Server\Microsoft.SqlServer.Server.csproj", "{A314812A-7820-4565-A2A8-ABBE391C11E4}"
 EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.SqlClient.Core", "Microsoft.Data.SqlClient\netcore\src\Microsoft.Data.SqlClient.Core.csproj", "{638A00EA-9648-45F1-B2FF-94E9A111AD73}"


I think a better name for the "Core" package is Microsoft.Data.SqlClient and instead document that Azure people must now reference Microsoft.Data.SqlClient.Azure for the Azure bits which would reference the "Core" package for them.

Agreed that additional Azure functionality would be (far) better integrated via an additional package with Azure in its name. The problem with that, of course, is that it would be a breaking change for existing users; but it may be acceptable given that the mitigation is extremely trivial.

I also agree, but that would be a huge breaking change. I also just did this to explore where the Azure dependencies are and maybe enable adding an additional package over time

Sure, though I'll just say it doesn't seem like a more significant breaking change than e.g. when encryption was set to true by default. Having SqlClient take a dependency on Azure packages was a problematic decision from the very beginning and is the more highly-voted issue on this repro.

SimonCropp · 2023-12-07T00:59:58Z

doesnt this bifurcate the ecosystem? eg if i want to write a lib that extends/uses sql, i now need two packages, one that targets each sql package

AraHaan · 2024-02-06T14:52:07Z

doesnt this bifurcate the ecosystem? eg if i want to write a lib that extends/uses sql, i now need two packages, one that targets each sql package

Not at all, the Azure version of the package could depend on the base version, and the extension libs could use the base version like how it does now, just with user versions not including the azure libs. This way it all acts similar to how creating extendable/extension libs using Microsoft.Extensions.DependencyInjection is currently.

AraHaan · 2024-02-06T14:59:37Z

...soft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlAuthenticationProviderManager.cs

@@ -36,6 +36,7 @@ private static void SetDefaultAuthProviders(SqlAuthenticationProviderManager ins
        {
            if (instance != null)
            {
+#if !CORE


After looking further, can't this file be excluded from "Core" builds and only include this file on the Azure version of the library, then it could reference the "Core" version directly without breaking any functionality at all?

I could try!

Also, this is not a PR, just a demo POC. Needs design!

Yeah, I know, just offering up some ideas that could help contribute to a design.

AraHaan · 2024-02-07T06:18:58Z

src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs

@@ -2312,6 +2312,7 @@ internal bool TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataRead
                            }
                            break;
                        }
+#if !CORE


I feel like a generic type in the "Core" assembly that the Azure assembly could initialize values on (default null without the application "initializing" them to different values from the Azure assembly version) would be better in these internal types (TdsParser, SqlInternalConnectionTds, ADP (AdapterUtil.cs).
The new generic type also being internal as well so only the azure assembly could use it as well.

For the case of TdaParer, it can store the member information needed to Invoke the code for that case (with some null checking first to default values of said internal type to null in cases where the azure assembly is not used directly by the application. CreateSqlException can also fall under this as well, same for the other file where an enum is used to call into Active Directory code (EnclaveDelegate in EnclaveDelegate.Crypto.cs.

SimonCropp · 2024-02-07T08:45:59Z

@AraHaan

Not at all, the Azure version of the package could depend on the base version, and the extension libs could use the base version like how it does now, just with user versions not including the azure libs. This way it all acts similar to how creating extendable/extension libs using Microsoft.Extensions.DependencyInjection is currently.

but "creating an Azure version of the package " is not what is being proposed here. this is slicing off a new Microsoft.Data.SqlClient.Core package.

so it means, to avoid the azure bits, the stack of libraries that currently use/extend Microsoft.Data.SqlClient would need to move to Microsoft.Data.SqlClient.Core. for example Microsoft.EntityFrameworkCore.SqlServer. and that would be considered a breaking change for each of those libs, so would likely have a very long tail until top level apps could remove Microsoft.Data.SqlClient

while breaking out a Microsoft.Data.SqlClient.Azure package would be a breaking change for this project, it does seem like that would have less combine fallout for consumers

unless u misunderstand what is being proposed here

ErikEJ · 2024-02-07T09:27:25Z

while breaking out a Microsoft.Data.SqlClient.Azure package would be a breaking change for this project, it does seem like that would have less combine fallout for consumers

Except it would be breaking for anyone using Azure auth today...

SimonCropp · 2024-02-07T10:32:34Z

@ErikEJ

Except it would be breaking for anyone using Azure auth today...

yeah. no idea which group is more common

ErikEJ · 2024-02-07T11:23:05Z

OK - once 5.2 has been released, I will try this approach (initially .NET only)

Microsoft.Data.SqlClient.Azure = current package, renamed (new .csproj with all current dependencies)

Microsoft.Data.SqlClient = new package without Azure auth dependencies, base functionality will be SQL auth, integrated Windows auth, and token auth (AccessToken, AccessTokenCallback).

This will require anyone using Azure auth methods to use an explict reference to:

<PackageReference Include="Microsoft.Data.SqlClient.Azure" Version="6.0.0" />

Attempts to use Azure auth with Microsoft.Data.SqlClient should result in an actionable error message with links to a suitable docs page

Thoughts? @SimonCropp @roji @AraHaan @David-Engel

AraHaan · 2024-02-07T15:34:18Z

OK - once 5.2 has been released, I will try this approach (initially .NET only)

Microsoft.Data.SqlClient.Azure = current package, renamed (new .csproj with all current dependencies)

Microsoft.Data.SqlClient = new package without Azure auth dependencies

This will require anyone using Azure auth methods to use an explict reference to:

<PackageReference Include="Microsoft.Data.SqlClient.Azure" Version="6.0.0" />

Attempts to use Azure auth with Microsoft.Data.SqlClient should result in an actionable error message with links to a suitable docs page

Thoughts? @SimonCropp @roji @AraHaan @David-Engel

Sounds good, yeah.

David-Engel · 2024-02-07T18:00:43Z

OK - once 5.2 has been released, I will try this approach (initially .NET only)

Microsoft.Data.SqlClient.Azure = current package, renamed (new .csproj with all current dependencies)

Microsoft.Data.SqlClient = new package without Azure auth dependencies

This will require anyone using Azure auth methods to use an explict reference to:

<PackageReference Include="Microsoft.Data.SqlClient.Azure" Version="6.0.0" />

Attempts to use Azure auth with Microsoft.Data.SqlClient should result in an actionable error message with links to a suitable docs page

Thoughts? @SimonCropp @roji Shay Rojansky FTE @AraHaan @David-Engel David Engel (SIMBA TECHNOLOGIES INC) Vendor

We should err on the side of compatibility. The reverse would be far less disruptive. People already expect M.D.SqlClient to support Azure connectivity. Those few who object to the current long dependency tail can move to a new M.D.SqlClient.Core package, which just has the base functionality. Base functionality in my mind is SQL auth, integrated Windows auth, and token auth (AccessToken, AccessTokenCallback).

SimonCropp · 2024-02-07T23:34:32Z

@David-Engel

Those few who object to the current long dependency tail can move to a new M.D.SqlClient.Core package, which just has the base functionality

to achieve that all libs using Microsoft.Data.SqlClient would need to make a breaking change.
i think breaking change decisions should be made based on total impact to the consumers of Microsoft.Data.SqlClient.
ie SqlClient usage is an ecosystem, so decisions on braking changes should take that ecosystem into account

SimonCropp · 2024-02-07T23:35:26Z

@ErikEJ

OK - once 5.2 has been released, I will try this approach (initially .NET only)

Microsoft.Data.SqlClient.Azure = current package, renamed (new .csproj with all current dependencies)

Microsoft.Data.SqlClient = new package without Azure auth dependencies

This will require anyone using Azure auth methods to use an explict reference to:

<PackageReference Include="Microsoft.Data.SqlClient.Azure" Version="6.0.0" />

Yes. i think that is the best compromise of all the options discussed

David-Engel · 2024-02-08T00:00:51Z

Those few who object to the current long dependency tail can move to a new M.D.SqlClient.Core package, which just has the base functionality

to achieve that all libs using Microsoft.Data.SqlClient would need to make a breaking change. i think breaking change decisions should be made based on total impact to the consumers of Microsoft.Data.SqlClient. ie SqlClient usage is an ecosystem, so decisions on braking changes should take that ecosystem into account

@SimonCropp
How would libs need to make a breaking change? I don't think we are thinking the same thing. My goal is not to have breaking changes for any existing users of Microsoft.Data.SqlClient. Libs referencing MDS would not have to change. MDS would have a dependency on MDS.Core. Only users who want the version without Azure support would need to change their dependency from MDS to MDS.Core.

AraHaan · 2024-02-08T00:20:44Z

Those few who object to the current long dependency tail can move to a new M.D.SqlClient.Core package, which just has the base functionality

to achieve that all libs using Microsoft.Data.SqlClient would need to make a breaking change. i think breaking change decisions should be made based on total impact to the consumers of Microsoft.Data.SqlClient. ie SqlClient usage is an ecosystem, so decisions on braking changes should take that ecosystem into account

@SimonCropp How would libs need to make a breaking change? I don't think we are thinking the same thing. My goal is not to have breaking changes for any existing users of Microsoft.Data.SqlClient. Libs referencing MDS would not have to change. MDS would have a dependency on MDS.Core. Only users who want the version without Azure support would need to change their dependency from MDS to MDS.Core.

Basically libs the once depended on the single package would be broken for a bit because they would need to then reference the "Core" package to then remove the azure bits, vs having all the core bits pushed in a new MAJOR version of the current package and split the azure bits to a separate version with .Azure appended to it's name. Either way those who depends on the SqlClient package will be broken anyways, except it limits the breakage to those who ACTUALLY want the azure bits.

As such I feel like from the start the azure bits should have been a separate opt-in .Azure package anyways much like how a lot of things in MSBuild and even the runtime (dotnet/runtime) has opt-in escape hatches put in place.

Also the new major version then could have something like this in as a package msbuild file that would be imported by the tooling system upon resolving it:

<Message Importance="high" Text="The azure bits that was in this package has been migrated to the Microsoft.Data.SqlClient.Azure package, if you depend on these parts change the package reference from Microsoft.Data.SqlClient to Microsoft.Data.SqlClient.Azure. If you do not depend on the azure functionality you can suppress this warning with setting 'DisableAzureSqlClientMessage' to 'true'." Condition="'$(DisableAzureSqlClientMessage)' != 'true'" />

David-Engel · 2024-02-08T00:43:46Z

Basically libs the once depended on the single package would be broken for a bit because they would need to then reference the "Core" package to then remove the azure bits, vs having all the core bits pushed in a new MAJOR version of the current package and split the azure bits to a separate version with .Azure appended to it's name. Either way those who depends on the SqlClient package will be broken anyways, except it limits the breakage to those who ACTUALLY want the azure bits.

This still isn't making sense to me. Why do we need to break anyone? In my scenario, those that currently depend on MDS (Azure or not) should require no change. MDS currently supports Azure and will continue to. The only ones that need to change should be those that really don't want Azure (the "feature" we are removing) because they want a slimmed down dependency tree.

Think of the existing MDS package as the "all in" package. For those who don't want everything, they can make changes.

As such I feel like from the start the azure bits should have been a separate opt-in .Azure package anyways much like how a lot of things in MSBuild and even the runtime (dotnet/runtime) has opt-in escape hatches put in place.

I'm not going to disagree with that. But it doesn't matter. We are here now and we aren't going to make a big breaking change for Azure users (or any other users) that depend on MDS.

SimonCropp · 2024-02-08T01:08:10Z

@David-Engel
lets take a specific example Microsoft.EntityFrameworkCore.SqlServer. IMO for them to move from Microsoft.Data.SqlClient to Microsoft.Data.SqlClient.Core, that would be a breaking change. and that change would need to be done in all similar libraries.

i think we should avoid that long tail of breaking changes. and just do on in this repo

also, to fully remove the azure bits, the fact that "change to different nuget" is significantly more works and more difficult o communicate to all consuming libs. when compared to "do a nuget update" which they are used to regularly doing

also the Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Azure approach means top level apps can remove the azure bits with only a single top level ref update.

where the Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Core approach requires all intermediate nugets to be updated to remove the azure bits

SimonCropp · 2024-02-08T01:17:57Z

let me clarify that specific example

an project using azure references Microsoft.EntityFrameworkCore.SqlServer and not Microsoft.Data.SqlClient.
we do Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Core in a "non breaking way"
to get the value out of "no azure bits" Microsoft.EntityFrameworkCore.SqlServer moves from Microsoft.Data.SqlClient to Microsoft.Data.SqlClient.Core
project updates Microsoft.EntityFrameworkCore.SqlServer
boom because no azure bits

now multiply that over all the libs that extend Microsoft.Data.SqlClient. ie it might not break when they update Microsoft.EntityFrameworkCore.SqlServer. it might break when they update ErikEJ.SqlClient.Extensions, or Serilog.Sinks.MSSqlServer, etc. but evernually they will update the last sql client lib that removes Microsoft.Data.SqlClient (and hence azure) and their project will break

David-Engel · 2024-02-08T01:41:08Z

I think I see the disconnect. You are trying to make it easier for Azure to be an opt-in for everyone, including those who currently only have it via a transitive dependency. But you are proposing breaking Azure for everyone all the way down the line in order to do it.

So in your scenario, everyone (EF, for example) would only depend on/get non-Azure functionality (and all their Azure users would be broken). To get Azure authentication, apps (likely at the tail end unless libs like EF want to offer a ".Azure" variant) would simply need to add a dependency on the .Azure package.

I think you underestimate the disruption this would cause to existing Azure users. Given that Microsoft owns this library, uses it heavily internally (think about all the code that must be behind MS/Azure products and services), heavily promotes MDS in their docs (that's a lot of updating docs - if version >= 6, use M.D.S.Azure), and has an interest in keeping it simple and easy to use against Azure, I can't see how this change would be welcomed.

AraHaan · 2024-02-08T01:56:59Z

let me clarify that specific example
* an project using azure references Microsoft.EntityFrameworkCore.SqlServer and not Microsoft.Data.SqlClient.

* we do Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Core  in a "non breaking way"

* to get the value out of "no azure bits" Microsoft.EntityFrameworkCore.SqlServer moves from Microsoft.Data.SqlClient to Microsoft.Data.SqlClient.Core

* project updates Microsoft.EntityFrameworkCore.SqlServer

* boom because no azure bits
now multiply that over all the libs that extend Microsoft.Data.SqlClient. ie it might not break when they update Microsoft.EntityFrameworkCore.SqlServer. it might break when they update ErikEJ.SqlClient.Extensions, or Serilog.Sinks.MSSqlServer, etc. but evernually they will update the last sql client lib that removes Microsoft.Data.SqlClient (and hence azure) and their project will break

Or do like the Microsoft.Extensions.DependencyInjection approach and allow someone to register the azure bits if they want them, as a service. After all Azure is in a nutshell a service after all.

Another Example: Try installing Microsoft.SqlServer.SqlManagementObjects into a project, yet only use the non-azure bits like:

using var connection = new SqlConnection(this.SqlConnectionString);
var db = new Server(new ServerConnection(connection));
db.ConnectionContext.ExecuteNonQuery(File.ReadAllText(this.SqlOutputFile));

(allows loading sql code from a file on a server with GO statements inside of it).

As such having the Azure stuff split separate from it where the application has to Initialize azure via an explicit api call that initializes an internal static type inside of Microsoft.Data.SqlClient is actually a smart idea as well (default each item in there as nullable so they default as null).

SimonCropp · 2024-02-08T02:00:53Z

@AraHaan

Or do like the Microsoft.Extensions.DependencyInjection approach and allow someone to register the azure bits if they want them, as a service. After all Azure is in a nutshell a service after all.

r u proposing only a runtime api level change ?

if so, this cant be only a run time change. we already have one significant bug cause by sqlclient nuget pulling in azure refs AzureAD/microsoft-authentication-library-for-dotnet#4468 i know because it broke our integration environment

it has to be nuget change

AraHaan · 2024-02-08T02:03:56Z

@AraHaan

Or do like the Microsoft.Extensions.DependencyInjection approach and allow someone to register the azure bits if they want them, as a service. After all Azure is in a nutshell a service after all.

r u proposing only a runtime api level change ?

if so, this cant be only a run time change. we already have one significant bug cause by sqlclient nuget pulling in azure refs AzureAD/microsoft-authentication-library-for-dotnet#4468 i know because it broke our integration environment

it has to be nuget change

Basically yes, have the azure bits after they install the azure version of the library (which has a dependency on the base version) which:

must be explicitly initialized by the application to "register it" as a "service" for the base version to obtain and use.

SimonCropp · 2024-02-08T02:05:13Z

@David-Engel

You are trying to make it easier for Azure to be an opt-in for everyone, including those who currently only have it via a transitive dependency. But you are proposing breaking Azure for everyone all the way down the line in order to do it.

not exactly. i am pointing out that even we move to "Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Core" to "prevent breaking changes for azure users". it is till a braking change for a subset of azure users. ie those that leverage Microsoft.Data.SqlClient identity transitively through something like Microsoft.EntityFrameworkCore.SqlServer.

since both approaches are breaking for at least a subset of azure users. i think we should chose the approach that gets us into the desired state of "Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Azure". and also has less impact for all non-azure users..

SimonCropp · 2024-02-08T02:06:00Z

@AraHaan

Basically yes, have the azure bits after they install the azure version of the library (which has a dependency on the base version) which: must be explicitly initialized by the application to "register it" as a "service" for the base version to obtain and use.

ahh. yes. thats what i want

SimonCropp · 2024-02-08T02:12:03Z

if we do go with the Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Core approach. the doco will be a fun read for everyone moving forward

use Microsoft.Data.SqlClient if you are targeting Azure. otherwise use Microsoft.Data.SqlClient.Core

and that when people even bother to read and understand that doco. because the entire internet history of blogposts, stackoverflow, and training will continue to be pointing to "Microsoft.Data.SqlClient is the one you want"

IMO the technical debt and significant number of people who will continue to pick the incorrect package should be reason enough to rule out that approach

0xced · 2024-02-08T08:28:23Z

Why?

Because any change needs to be approved by the responsible maintainers of this repo…

The why was about the only way to do this is via MDS.Core

Only @David-Engel has argued about this route, all other participants in this thread are arguing towards Microsoft.Data.SqlClient + Microsoft.Data.SqlClient.Azure. And also the 51 upvotes on #1108 (comment)

But David is the lead for SQL Server drivers so I don't have high expectations on the weight of random internet people arguments. ¯\_(ツ)_/¯

roji · 2024-02-08T08:52:50Z

A few notes/thoughts/summary from me...

I don't have the latest figures, but IIRC a very significant number of usages of SqlClient comes through EF Core.
If we go with the SqlClient+SqlClient.Core, EF must react to the change for it to affect its users; otherwise Microsoft.EntityFramework.SqlServer continues to reference SqlClient, and there's no way to opt out of the Azure dependencies while using EF. So at this point there are two options:
1. Do the same at the EF level and introduce a new M.EF.SqlServer.Core, for users who don't want Azure dependencies. This means that library dependencies on top of SqlClient - EF isn't the only one - all need to bifurcate and take on added maintenance/communication burden (we'd need to explain to our users what the difference between the package is etc.). I'd really like to avoid this.
2. Switch M.EF.SqlServer to depend on the new SqlClient.Core package; users who want Azure also add SqlClient.Azure as described above. If this happens, EF Azure users are broken in the exact same way as going the other way in SqlClient and introducing SqlClient.Azure instead (as @SimonCropp argued above). In other words, the breaking change has just been pushed up.
In other words, I'd argue that a more holistic view of the ecosystem is needed here; if SqlClient were a single library used directly, that would lead to a different set of considerations than the current one, where SqlClient is more of an underlying library typically consumed via other 3rd-party layers (such as EF). Either you ask all layers to make a change as well, or the breakage is pushed onto them.
Another pretty obvious point but worth saying - with SqlClient.Core we end up in a place where package naming is simply misleading; SqlClient.Azure clearly expresses what that package is about (adding Azure-specific stuff), but just having SqlClient+SqlClient.Core makes it non-obvious for users to choose the right option.
Thinking ahead a bit, introducing SqlClient.Core blocks that name to mean "SqlClient but without Azure", forever; if we need to do a similar exercise at some point, the name is already taken. I specifically have NativeAOT in mind; if SqlClient has any functionaliy by default that's incompatible with NativeAOT, introducing a SqlClient.Core is one main way to deal with that (this has happened in other places in the ecosystem). But if SqlClient.Core is already used to mean "but without Azure", we can't do that.
Finally, I agree with above arguments that although this a breaking change, it's the "best kind" of breaking change - it's discoverable immediately (not e.g. a subtle behavioral change discovered only after moving to production), and we hopefully can include an informative exception pointing to the exact mitigation ("install SqlClient.Azure"). We have some experience with introducing such changes in EF, and if done right, the actual impact and user dissatisfaction has been minimal.

ErikEJ · 2024-02-08T09:03:36Z

Thanks for joining, @roji - your opinion is valuable to me. So your are in favor of my proposal above ❤️

Let's stand down, try to not get too emotional and think about this one more time...

I will share my proposal and collect more feedback

AraHaan · 2024-02-08T09:27:05Z

A few notes/thoughts/summary from me...

* I don't have the latest figures, but IIRC a very significant number of usages of SqlClient comes through EF Core.

* If we go with the SqlClient+SqlClient.Core, EF must react to the change for it to affect its users; otherwise Microsoft.EntityFramework.SqlServer continues to reference SqlClient, and there's no way to opt out of the Azure dependencies while using EF. So at this point there are two options:
  
  1. Do the same at the EF level and introduce a new M.EF.SqlServer.Core, for users who don't want Azure dependencies. This means that library dependencies on top of SqlClient - EF isn't the only one - all need to bifurcate and take on added maintenance/communication burden (we'd need to explain to our users what the difference between the package is etc.). I'd really like to avoid this.
  2. Switch M.EF.SqlServer to depend on the new SqlClient.Core package; users who want Azure also add SqlClient.Azure as described above. If this happens, EF Azure users are broken in the exact same way as going the other way in SqlClient and introducing SqlClient.Azure instead (as @SimonCropp argued above). In other words, the breaking change has just been pushed up.

* In other words, I'd argue that a more holistic view of the ecosystem is needed here; if SqlClient were a single library used directly, that would lead to a different set of considerations than the current one, where SqlClient is more of an underlying library typically consumed via other 3rd-party layers (such as EF). Either you ask all layers to make a change as well, or the breakage is pushed onto them.

* Another pretty obvious point but worth saying - with SqlClient.Core we end up in a place where package naming is simply misleading; SqlClient.Azure clearly expresses what that package is about (adding Azure-specific stuff), but just having SqlClient+SqlClient.Core makes it non-obvious for users to choose the right option.

* Thinking ahead a bit, introducing SqlClient.Core blocks that name to mean "SqlClient but without Azure", forever; if we need to do a similar exercise at some point, the name is already taken. I specifically have NativeAOT in mind; if SqlClient has any functionaliy by default that's incompatible with NativeAOT, introducing a SqlClient.Core is one main way to deal with that (this has happened in other places in the ecosystem). But if SqlClient.Core is already used to mean "but without Azure", we can't do that.

* Finally, I agree with above arguments that although this a breaking change, it's the "best kind" of breaking change - it's discoverable immediately (not e.g. a subtle behavioral change discovered only after moving to production), and we hopefully can include an informative exception pointing to the exact mitigation ("install SqlClient.Azure"). We have some experience with introducing such changes in EF, and if done right, the actual impact and user dissatisfaction has been minimal.

Not to mention from Microsoft.SqlServer.SqlManagementObjects as well.

ErikEJ · 2024-02-08T09:27:34Z

while i am grateful that this is be re-considered.

I am just re-opening to collect more feedback - and yes, I am pragmatic.

dustinmoris · 2024-02-08T09:38:31Z

there is a noticeable bias on .net OSS project to give priority to who says something over what is being said.

if a community member had made exactly the same comment as @roji would we have had the same outcome?

The answer is no. Discussions are not based on merit. Microsoft has an agenda, people are short sighted and they essentially do what they want to the detriment of .NET if it serves themselves or the progression of Azure. Only if another MSFT employee with internal clout disagrees it carries weight, because now it's a stand-off between two employees and the one who feels inferior pulls his tiny weeny back in and makes space for the other one. It's corporate politics played out in the open, nothing else.

AraHaan · 2024-02-08T09:56:32Z

This is what I feel should happen:

Modify: M.D.SC
- Remove: All Azure SDK refs
- Add: Microsoft.Extensions.DependencyInjection.Abstractions (or the normal one) Most people consuming this library pulls in those dependencies anyways, why not have it utilize this to help simplify it's own code. Even EFCore builds it's own IServiceCollection and provider if I remember it all exactly.
  - With this add a base but internal interface (IAzureSqlService could be it's name) that M.D.SC.Azure can implement itself, interface defining all members that the base library would need to use.
    - With this also an internal type that builds an IServiceProvider from an IServiceCollection, this being in a static type internal to the base library, the azure library then adds the "AzureSqlService" type to this IServiceCollection, that then gets automatically built into an IServiceProvider inside of the base lib, base lib then checks to see if the collection then contains any objects that implements the IAzureSqlService interface and if it returns null, it skips calling those members.
Add: M.D.SC.Azure (Microsoft.Data.SqlClient.Azure for short)
- internally implements AzureSqlService as an internal only type with all of the current azure code within M.D.SC today (the enum case stuffs being an Action, or one of the Action<> variants).
  - Also add a static class that basically "Installs the AzureSqlService" into the base library's IServiceCollection before it automatically builds the service provider internally within the ctors of specific types of the public api (or in some other place).

there is a noticeable bias on .net OSS project to give priority to who says something over what is being said.

if a community member had made exactly the same comment as @roji would we have had the same outcome?

The answer is no. Discussions are not based on merit. Microsoft has an agenda, people are short sighted and they essentially do what they want to the detriment of .NET if it serves themselves or the progression of Azure. Only if another MSFT employee with internal clout disagrees it carries weight, because now it's a stand-off between two employees and the one who feels inferior pulls his tiny weeny back in and makes space for the other one. It's corporate politics played out in the open, nothing else.

It might look that way on the surface, some areas of MSFT are unaware of what pains other areas of MSFT, so that is why sometimes those other areas have to put in their inputs to then have merit because then they would realize that it would make maintaining their other codebases that much more harder (not to mention waste a lot of money for them in terms of wasted developer's time). We all know that happy developers with minimal conflicts with maintaining code makes up the the most amount of revenue that a company can generate 😄.

ErikEJ · 2024-02-08T10:07:09Z

@AraHaan > add: Microsoft.Extensions.DependencyInjection.Abstractions

That will add a new trail of dependencies for .NET Framework, so...

AraHaan · 2024-02-08T10:08:35Z

@AraHaan > add: Microsoft.Extensions.DependencyInjection.Abstractions

That will add a new trail of dependencies for .NET Framework, so...

But won't it be ok for .NET Framework though, even if it is used as an implementation detail internally?

edcsu · 2024-02-08T10:14:13Z

This is great although it will be a breaking change for me. Thanks for pointing this out

ErikEJ · 2024-02-08T10:16:36Z

@AraHaan Making this change and adding new dependencies does not seem right

roji · 2024-02-08T10:21:11Z

Everyone, just to be clear - I work at Microsoft but I don't make any decisions where SqlClient is concerned. @David-Engel is the owner here and has the final say, and I know he has SqlClient's best interests in mind, as well as the best knowledge about who uses SqlClient, how, and what breaking changes make sense in that context. Although I have my opinion, I fully trust @David-Engel will make the right choice here etc.

@dustinmoris your comment really does not reflect the reality of the situation; there's no "corporate politics" or an employee show-down with clout. There really are just two Microsoft engineers here, discussing what to do the best for SqlClient and for the .NET ecosystem as a whole. We regularly engage with the community on important design decisions, and very frequently change our minds based on community feedback that we get. It's very common for people on the outside to assume "corporate politics" and various dark "behind the scenes" dealings at Microsoft - I hear this all the time - but we're really just a bunch of passionate engineers discussing and trying to arrive at the best outcome given the situation.

So again, I'm just voicing my opinion here, with the added weight that I have the EF perspective (and EF is an important factor, being a major way that SqlClient is used).

@AraHaan for the record, I'm against the direction of introducing DI into SqlClient. Unlike EF, SqlClient (and other ADO.NET providers) aren't currently based on DI concepts, and rewriting SqlClient to start using DI internally is a far-reaching change that doesn't really seem related to this discussion in any way. I do agree that better integration with DI is a good idea, but I'd do that in a separate package (much like how Npgsql has Npgsql.DependencyInjection for people who want it). Enabling Azure functionality via an external package (regardless of how we call it) is most probably doable without introducing DI.

…dependencies (to demonstrate where dependencies are used and test a build) relates to dotnet#1108

ErikEJ · 2024-02-08T13:09:38Z

Published payload with .NET 8 target:

robertmclaws · 2024-02-08T17:28:52Z

Make this change, increment the version to 6.0, and break it.

Why would anyone running on-prem SQL Server want all of the Azure bloat in their apps?

It's super annoying that the SqlClient dragged in all these dependencies, and honestly the Microsoft team should have thought of this at the beginning.

This is a great solution. Break it, fix it, move on.

David-Engel · 2024-02-08T18:29:30Z

Thanks for all the input folks. I've just been stating my opinion based on my own experiences. It's good to hear different opinions. I'm not against having my mind changed. After reading all the comments, I'm inclined to agree that MDS + MDS.Azure seems to make more sense. (Implementation details TBD.) I'll have to discuss this with other internal stakeholders to see what their thoughts are. I don't want to make people do a bunch of work that won't ever be merged.

SimonCropp · 2024-02-08T19:02:06Z

@David-Engel thanks so much for re-considering

AraHaan · 2024-04-10T21:09:00Z

Anything new about this yet? It has been a few months since the last comment on this.

cheenamalhotra · 2024-10-01T23:27:33Z

Closing PR in favor of #2247 (comment) - as it will require rework and new PR would be ideal here. We've picked up the work for upcoming milestones, so it will be worked upon by our team.

AraHaan · 2024-10-02T20:07:58Z

@cheenamalhotra mind doing an issue for this and a feature branch so people who are interested can keep track with the progress and potentially submit PR's to help land a hand if needed?

MichaelKetting · 2024-10-03T11:22:30Z

@cheenamalhotra , I'll second @AraHaan's request. Please post this here so we know which issue to watch. Can you also confirm that this will be delivered with .NET 10? For us, this is an extremly important issue because System.Data.SqlClient is no longer officially supported on .NET 10 and Microsoft.Data.SqlClient in its current shape is a significant problem due to the current dependency tree.

AraHaan · 2024-10-03T18:27:04Z

I would recommend delaying the System.Data.SqlClient with not being supported in .NET 10 until it is proven that Microsoft.Data.SqlClient can do this for .NET 10 before it's release so it does not break people who have STRICT security standards on dependencies.

cheenamalhotra · 2024-10-05T05:45:27Z

Would recommend tracking the same issue #1108

ErikEJ mentioned this pull request Nov 28, 2023

Feature | Split Azure dependent functionality in a separate NuGet Package #1108

Open

ErikEJ force-pushed the core-package branch from b4aca64 to 5c6391d Compare November 29, 2023 08:15

AraHaan reviewed Dec 7, 2023

View reviewed changes

simdanne mentioned this pull request Jan 22, 2024

Security: No CVE free Azure.Identity 1.3.x available for consumption in e.g. Microsoft.Data.SqlClient 3.1.x #2312

Closed

AraHaan reviewed Feb 6, 2024

View reviewed changes

AraHaan reviewed Feb 7, 2024

View reviewed changes

ErikEJ mentioned this pull request Jan 29, 2024

SqlClient contribs ErikEJ/SqlClientExtensions#18

Closed

ErikEJ reopened this Feb 8, 2024

ErikEJ changed the title ~~POC: Add Microsoft.Data.SqlClient.Core package that has no Azure SDK …~~ POC: Remove Azure Identity from Microsoft.Data.SqlClient Feb 8, 2024

ErikEJ added 2 commits February 8, 2024 13:45

POC: Add Microsoft.Data.SqlClient.Core package that has no Azure SDK …

ddf7deb

…dependencies (to demonstrate where dependencies are used and test a build) relates to dotnet#1108

Merge with latest main

bcaa92f

Update to latest main

022bce6

ErikEJ force-pushed the core-package branch from 5c6391d to 022bce6 Compare February 8, 2024 13:10

ErikEJ mentioned this pull request Jun 16, 2024

Release a new version with an update ref to 'Azure.Identity' #2568

Closed

cheenamalhotra closed this Oct 1, 2024

POC: Remove Azure Identity from Microsoft.Data.SqlClient #2247

POC: Remove Azure Identity from Microsoft.Data.SqlClient #2247

Conversation

ErikEJ commented Nov 28, 2023 • edited Loading

codecov bot commented Nov 29, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonCropp commented Dec 7, 2023

AraHaan commented Feb 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonCropp commented Feb 7, 2024

ErikEJ commented Feb 7, 2024

SimonCropp commented Feb 7, 2024

ErikEJ commented Feb 7, 2024 • edited Loading

AraHaan commented Feb 7, 2024

David-Engel commented Feb 7, 2024

SimonCropp commented Feb 7, 2024

SimonCropp commented Feb 7, 2024

David-Engel commented Feb 8, 2024

AraHaan commented Feb 8, 2024 • edited Loading

David-Engel commented Feb 8, 2024 • edited Loading

SimonCropp commented Feb 8, 2024

SimonCropp commented Feb 8, 2024 • edited Loading

David-Engel commented Feb 8, 2024

AraHaan commented Feb 8, 2024 • edited Loading

SimonCropp commented Feb 8, 2024

AraHaan commented Feb 8, 2024

SimonCropp commented Feb 8, 2024

SimonCropp commented Feb 8, 2024

SimonCropp commented Feb 8, 2024

0xced commented Feb 8, 2024 • edited Loading

roji commented Feb 8, 2024

ErikEJ commented Feb 8, 2024

AraHaan commented Feb 8, 2024

ErikEJ commented Feb 8, 2024

dustinmoris commented Feb 8, 2024

AraHaan commented Feb 8, 2024 • edited Loading

ErikEJ commented Feb 8, 2024

AraHaan commented Feb 8, 2024

edcsu commented Feb 8, 2024

ErikEJ commented Feb 8, 2024

roji commented Feb 8, 2024 • edited Loading

ErikEJ commented Feb 8, 2024

robertmclaws commented Feb 8, 2024

David-Engel commented Feb 8, 2024

SimonCropp commented Feb 8, 2024

AraHaan commented Apr 10, 2024

cheenamalhotra commented Oct 1, 2024

AraHaan commented Oct 2, 2024

MichaelKetting commented Oct 3, 2024

AraHaan commented Oct 3, 2024

cheenamalhotra commented Oct 5, 2024

ErikEJ commented Nov 28, 2023 •

edited

Loading

codecov bot commented Nov 29, 2023 •

edited

Loading

ErikEJ commented Feb 7, 2024 •

edited

Loading

AraHaan commented Feb 8, 2024 •

edited

Loading

David-Engel commented Feb 8, 2024 •

edited

Loading

SimonCropp commented Feb 8, 2024 •

edited

Loading

AraHaan commented Feb 8, 2024 •

edited

Loading

0xced commented Feb 8, 2024 •

edited

Loading

AraHaan commented Feb 8, 2024 •

edited

Loading

roji commented Feb 8, 2024 •

edited

Loading